mirror of
https://github.com/saymrwulf/nvidia-AI.summerschool17-notebooks.git
synced 2026-11-17 08:55:07 +00:00
1198 lines
252 KiB
Text
1198 lines
252 KiB
Text
|
|
{
|
||
|
|
"metadata": {
|
||
|
|
"name": "14-bayesian-parameter-estimation"
|
||
|
|
},
|
||
|
|
"nbformat": 3,
|
||
|
|
"nbformat_minor": 0,
|
||
|
|
"worksheets": [
|
||
|
|
{
|
||
|
|
"cells": [
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"from scipy import stats\n",
|
||
|
|
"from scipy.stats import norm"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 1
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Parameter Estimation\n",
|
||
|
|
"======================\n",
|
||
|
|
"\n",
|
||
|
|
"Assume we are given a samples from a Gaussian:\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(x) = \\sigma^{-1} (2\\pi)^{-1/2} e^{-\\frac{(x-\\mu)^2}{2\\sigma^2}}$$\n",
|
||
|
|
"\n",
|
||
|
|
"You already know the formulas for estimating means and variances of Gaussians:\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\mu \\approx \\hat{\\mu} = \\bar{x} = {1 \\over n} \\sum_{k=1}^n x_k $$\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\sigma^2 \\approx \\hat{\\sigma}^2 = \\bar{(x-\\bar{x})^2} = {1 \\over n} \\sum_{k=1}^n (x_k-\\bar{x})$$\n",
|
||
|
|
"\n",
|
||
|
|
"We're asking the following questions here:\n",
|
||
|
|
"\n",
|
||
|
|
"- Where do these formulas come from?\n",
|
||
|
|
"- Are they \"correct\"?\n",
|
||
|
|
"- Are they the whole story?"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"The $x_k$ above are an i.i.d. sample from the original sample density:\n",
|
||
|
|
"\n",
|
||
|
|
"$$D=\\{x_1,\\ldots,x_n\\}$$\n",
|
||
|
|
"\n",
|
||
|
|
"We want to estimate the parameters of the density $p(x)$. \n",
|
||
|
|
"Let's call the parameters generically $\\theta$.\n",
|
||
|
|
"For the univariate normal density, $\\theta = (\\mu,\\sigma)$.\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(x_i) = p(x,\\theta) = p(x|\\theta)$$\n",
|
||
|
|
"\n",
|
||
|
|
"Now we can write a density for the entire dataset:\n",
|
||
|
|
"\n",
|
||
|
|
"$$ p(D|\\theta) = p(x_1,\\ldots,x_n|\\theta) = p(x_1|\\theta)\\cdot\\ldots\\cdot p(x_n|\\theta) = \\prod_{i=1}^n p(x_i|\\theta)$$\n",
|
||
|
|
"\n",
|
||
|
|
"We call $\\ell(\\theta) = p(D|\\theta)$ the _likelihood_ of the data.\n",
|
||
|
|
"Note that likelihoods are parameterized densities viewed as functions of the parameters.\n",
|
||
|
|
"\n",
|
||
|
|
"Maximum Likelihood Estimate\n",
|
||
|
|
"============================\n",
|
||
|
|
"\n",
|
||
|
|
"The _maximum likelihood estimate_ is given by:\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\hat{\\theta} = \\arg\\max_\\theta ~ p(D|\\theta)$$\n",
|
||
|
|
"\n",
|
||
|
|
"This seems like a reasonable thing to do: choose the parameter that was\n",
|
||
|
|
"most likely to produce the data set.\n",
|
||
|
|
"\n",
|
||
|
|
"However, in general, there is little reason why the maximum of the likelihood function\n",
|
||
|
|
"should mean anything.\n",
|
||
|
|
"For example, we can easily modify the likelihood function to put a tiny spike in it\n",
|
||
|
|
"that moves the maximum somewhere arbitrary without actually changing the problem much at all.\n",
|
||
|
|
"\n",
|
||
|
|
"Maximum A Posteriori Estimate\n",
|
||
|
|
"=============================\n",
|
||
|
|
"\n",
|
||
|
|
"By analogy to Bayesian methods, we can also ``multiply in'' a prior:\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\hat{\\theta} = \\arg\\max_\\theta ~ p(D|\\theta) p(\\theta)$$\n",
|
||
|
|
"\n",
|
||
|
|
"This is called the _maximum a-posterior estimate_ (MAP) for the parameter $\\theta$.\n",
|
||
|
|
"\n",
|
||
|
|
"MAP vs Bayesian Methods\n",
|
||
|
|
"========================\n",
|
||
|
|
"\n",
|
||
|
|
"Although MAP looks like the derivation may have involved _Bayes rule_, it is\n",
|
||
|
|
"not a Bayesian method at all.\n",
|
||
|
|
"Bayesian methods are *not* methods that involve Bayes rule somewhere.\n",
|
||
|
|
"\n",
|
||
|
|
"**Bayesian methods are methods that result in decisions that minimize expected loss.**\n",
|
||
|
|
"\n",
|
||
|
|
"In order to minimize expected loss, you actually need a loss function.\n",
|
||
|
|
"Maximum likelihood or MAP estimation does not attempt to minimize a loss function,\n",
|
||
|
|
"hence it is not a Bayesian method.\n",
|
||
|
|
"For individual distributions, ML or MAP estimation may implicitly optimize some loss\n",
|
||
|
|
"function, but that loss function may depend on the distribution, sample size, and parameters."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Maximum Likelihood Estimation of the Mean of a Normal Density\n",
|
||
|
|
"=========================================\n",
|
||
|
|
"\n",
|
||
|
|
"Let's derive a maximum likelihood estimator, for the mean of a Gaussian\n",
|
||
|
|
"we want to maximize the likelihood function:\n",
|
||
|
|
"\n",
|
||
|
|
"$p(x_1,...,x_n|\\mu)=\\prod_i p(x_i|\\mu)$\n",
|
||
|
|
"\n",
|
||
|
|
"Let's take logarithms\n",
|
||
|
|
"\n",
|
||
|
|
"$$l(\\mu) = \\sum_i p(x_i|\\mu) = \\sum -{1\\over 2} \\ln((2\\pi)^d |\\Sigma|) - {1\\over 2} (x_i-\\mu)\\Sigma^{-1}(x_i-\\mu)$$\n",
|
||
|
|
"\n",
|
||
|
|
"If $l$ is sufficiently well behaved (it is), then a necessary condition for a local maximum is that the gradient in the parameter to be estimated is 0\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\nabla_\\mu l(\\mu) = 0$$\n",
|
||
|
|
"\n",
|
||
|
|
"The gradient is\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\nabla_\\mu l(\\mu) = \\sum_i \\Sigma^{-1} (x_i-\\mu)$$\n",
|
||
|
|
"\n",
|
||
|
|
"So,\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\nabla_\\mu l(\\hat{\\mu}) = \\sum_i \\Sigma^{-1} (x_i-\\hat{\\mu}) = 0$$\n",
|
||
|
|
"\n",
|
||
|
|
"Multiply by $\\Sigma$\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\sum_i (x_i-\\hat{\\mu}) = 0$$\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\sum_i x_i = n\\cdot\\hat{\\mu}$$\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\hat{\\mu} = {1 \\over n} \\sum_{i=1}^n x_i$$\n",
|
||
|
|
"\n",
|
||
|
|
"This is nice because it says that the maximum likelihood estimate of the mean of a Gaussian is just the arithmetic mean of the samples (the empirical mean or sample mean).\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Example\n",
|
||
|
|
"=======\n",
|
||
|
|
"\n",
|
||
|
|
"Let's try this out in 1D.\n",
|
||
|
|
"\n",
|
||
|
|
"We assume that $\\sigma=1$"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"D = randn(10)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 3
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"mus = linspace(-5,5,10000)\n",
|
||
|
|
"l_of_mu = [-(x-mus)**2 for x in D]\n",
|
||
|
|
"for l in l_of_mu: plot(mus,l)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD9CAYAAACoXlzKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd0FGX3xx8CKqg/IMnuJiSkEDoECNJ7lRKQKlVQBBVB\nRcGGryKI1EBAQOm9tyAg0iFANr333nt207a3me/vjycrL2RnyO5Ldz/n5BxznNm5bGbuPM+933tv\nPQAgVqxYsWLlpcHmWRtgxYoVK1YeL1bHbsWKFSsvGVbHbsWKFSsvGVbHbsWKFSsvGVbHbsWKFSsv\nGVbHbsWKFSsvGU/MsV+9epW0a9eOtG7dmqxfv/5JXcaKFStWrDxEvSehY2cYhrRt25bcvHmTODs7\nkx49epATJ06Q9u3bP+5LWbFixYqVh3giK/awsDDSqlUr4u7uTl555RUyffp0cuHChSdxKStWrFix\n8hANnsSHFhYWEhcXl39+b968OQkNDf3n93r16j2Jy1qxYsXKS09dgixPxLHXxXG/zJ0MVqxYQVas\nWPGszXhicP37AJD0inQSXRxNYktjSWxpLIkrjSMSpYS4NnElHrYepIVtC+L8f85E8Lrgn58mrzUh\nr9Z/ldgU2RA2iiWGBAPRJGuILllHmFKGkOaEGBwMRCVQENRPI6+oEsjrkhRiV5pDHCurSGOtgZTZ\nvUkUTg6knrMr0Te0I2VaG5JWVU2SSotJanEekRn0xLaxE7Ft6ECExJ40NbxOtDZKohJVEYODlJDX\nykk9tpI4aOqThNR65Hvb/yPNynVEWCYnukYNicStGSlzdiT59k1J2ptvklgtIakZpaSgKIdUVOUT\nm1ebE+b1bsTGvjUhb9oSNCAEqCKkaS6pb59F6r2ZR5iGEvKmyoGINM7EhXUhHq+6k85N7InIvprU\ne7OAsA1KCeoXEoZ5lcgULUhZuTsprXAkpVIhKa4QkAq5PVFrGxOloglRVTQm+orXyeuiauJoX0Da\nNEgkAxpFkvGuScSxy5vE0LoZ0bvZEV1TPdFo84lWm080mjyi0xWSI0cak6++GkzeeKMjefPNt0jj\nxr3Jq68Kn/6N9AR42Z+9ui6Kn4hjd3Z2Jvn5+f/8np+fT5o3b/4kLmXlGcKwDIkuiSb3cu+RgLwA\nIs4Tk9dfeZ281ewt0sWhC/mo60eki2MX4t7UndjUezDqB4Ao45Wk0q+SVAdWk+qQakJYQhr3aUya\ndG1CnD5yIm+0eZU0kieTegF3Cbl7lxD/QEIEAkLeeouQgRMJ6dqVaFq2JJdSosmfVy+Q0JBQknv7\nHrFpbENYZ5a4t3Unfcb1IZ8M+In0bd+XRBRHkNs5t8mdnDskvTyddBR1JF5NvIinYRJpUdqCCOOE\npH5ofZJbtJPYOH9OdGNsiXJIE9LETU6ahiSQqkvxxDU4jvTICycL9FISLehOApvMJLdeGUACJQNI\nQ/sK0kQTTV6r9ifV6ghS3yad9GvZlAwdMIAMHbGSuPfpQCLzTpPI/Cskoewuia/0I6dl9YjA8Cbp\n0bAD6e86kIxqu5a0Eb5l8vuWywnJyCAkIYGQiAhCguMMJK7sVSJp4kqqGzUnd0u9yX8uvEle89cS\nN+d8MpjcJQsKDhCvbgJCBnkTMmgQYXq1J1evfksEggFEqUwghYVbSXLyLPLqq0LSuHEf0qTJAGJn\nN4I0bOj+FO4gK0+KJ5I8NRgMpG3btuTWrVvEycmJ9OzZ84Hkab169awr9heUCnUF+eTrT0ijtxuR\nqxlXiegNERnkNoj0d+1PBrgOIC5NXDjPNVQZSPnlclJxrYJUXq8kNq/bELsRdqTJwCakce/GpKF7\nQ1JPKiXk778JuXiRkFu3CHF3J2TQIEIGDyZkwABChEKSlZVFLl26RK5du0YCAgJIx44dyfDhw0nf\nvn1Jr169iJ2dHalUV5KjcUfJqcRTJKYkhij1SmLfyJ70d+1PPvT6kIxuPZq8Wv9Vk3b+/MPPZNGQ\nxSTxaBWpvF1JGhYrSUq9/yMxnW1JcLPXSU5OI9I0q5qMdrxBJjKXyIDS28SWkZF6Hh6EjBhByPz5\nhHTqRAoKCsjt28fJtWunSUBAApHLdaRXLxEZM2YImTBhHnF27k9AXiFJkiQSlB9EAvICyI2sG8S2\noS0Z1WoUGdVqFBnsPpg0bNCQ8ztlGOro/f0JuXOHkDuBLHmjnYr8n6uOlJfZkPKI18lrb+qJZ8ss\nMk39J/ks9Q+y3llAVixdSoi3NyH29gRgiEqVQmSyYFJVdYdUVNwgDRo0IXZ2I4md3Shiazuc2Ni8\n9r/dOE+Jl/nZI6TuvvOJOHZCCLly5Qr56quvCMMwZN68eeSHH34w27gXlTt37pDBgwc/azMeG5Xq\nSuKX7EdOJJwg4YXhxFPlSd4f/z4Z3Wo0cWvqxnuuvlJPyi+UE8lZCam6V0WaDmpK7EbbEbuRdqRR\ny0b0oOJiQk6eJMTPj5D4eELefpuQceMIGT2aECENEWRkZJCzZ8+SM2fOkIKCAjJ27FgyatQoMmzY\nMGJnZ0cIobuAoPwgcirxFLmYepE0sGlAxrcbT8a1GUfaC9uTOzl3yF9pf5Er6VeIc2NnMrXDVDKz\n00zSwrYFIYQ6yYAAQjZvvkNCQgYTh+Ysse8vI3l5BlJ89w3SSEvIUEMZ6dcgm4xi/YjLEAN5Y2Y/\nQkaOpCdfu0Zw5QrB9StE6/4GKR2kJeVD3yRNO0whdnZjSHV1c3Ljxh1y4cIFcu/ePdK3b18yceJE\n8u677xJ7e3tCCCEsWBJTEkOuZlwll9Mvk0RJIhnbZiyZ2mEqGdFyBHmtAb+DZRhCoqLoe/HiRUIK\ny1jSdrSSKOrrSEb4q0Sd2Yg0b3mOLGhWSRYFLSdvdG5NyLRphEydSohIVPM9skSpjCMVFddJefkl\nolTGE3v7MUQonEJsbUeQ+vUbWXwvPWletmfvYZ65Y+e96Evu2F8GtAYtOZ9ynhyLP0bu5t4lb3u8\nTWZ4ziDerb1Jo1f4H2wYQMovl5OS/SWk0r+S2A6zJcJ3hcR+rD1p0Lgm+ieXE/Lnn4QcPUpIeDgh\nEyZQBzNkCCGvUedVWVlJjh8/Tg4cOEAKCgrIpEmTyJQpU8iAAQNIgwb3o4iZFZnkSNwRciTuCHmt\n/mtkZqeZZEK7CaSjsKPJmCTDMiQwP5CcTDhJziSdIc4NWxPbgpkk6fR04mRrTzqNV5DYIh1JuPQ6\nadiQkMFvZZIvy3aRt+OOETJhEqnuNJ1I81xJ2Rkpec3pNeL4gSNpPElFynUnSGnpYWLDvEJc07oR\nuxsy0uByAKnXpQshs2fTf9+bb9b88+Xk2rVr5OzZs+Tq1atkyJAh5IMPPiDe3t7k1Vfv7ySK5cXE\nL9mPnE48TRLKEsiEdhPIR299RPo071OneGt2NiEXLhBy/Dgh+fmE9J+uIqUN1CTqzitElfo66dyv\nhPzQ6CSZemsVqde3DyEzZxIyaRIhb7xx/17QFhOp9E8ikZwlCkUUEQgmk2bN5pLGjftahRBPmTr7\nTjwDntFlrdSBNGkavrn+DYQ+Qgw9NBSHYw6jWlNdp3OVaUpkfp+JwGaBiOwbiaJ9RdDL9A8eFBsL\nzJ8PNG0KvPMOcPIkoFT+879ZlsWtW7cwc+ZMNGnSBNOmTcO1a9dgMBge+BitQYuT8ScxYP8AiDaI\n8OWVLxFRGAGWZetkq1wO7N0L9Oyjg33vv9Fu2Vw0mP876ncpg00TLXpOLsSF6RvBNmkKTJoEXLgA\naDQPfIZBp0XmjT0QH+sB//NNEPzbDBTcvAWGYe4fpFYDf/4JjBsH2NoCn3wCREQ88DnV1dXYt28f\nBg4cCIFAgMWLFyM9Pb2WzYWyQqwXr0ebbW3Q/vf28A3yRZmirE7/XgBISQGWLQNatAC6dAGWbFai\n39dS1HdVopGbClPej0fOqPcBOzvg88+BhIRan6HVFiM3dz1CQ9siJKQNcnPXQastrbMNVv436uo7\nrY7dChiWwV+pf2HYoWEQbRDh
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 11
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"l = sum(array(l_of_mu),0)\n",
|
||
|
|
"plot(mus,l)\n",
|
||
|
|
"print mus[argmax(l)]"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "stream",
|
||
|
|
"stream": "stdout",
|
||
|
|
"text": [
|
||
|
|
"0.398539853985"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD9CAYAAABQvqc9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XtYlNXaBvAbBTxknitzhkRhFDBE3ApoZpTiMYysTHBL\nnvp2mKYdzNztEirBQ2YeQi1BEytNDbBS0qxRcysqqLllbxwFFQawtPTDLTmK7/fHqvk0FYdhZtbM\nO/fvuuYKBoa5pwsf1qx3rWd5KIqigIiI3EI92QGIiMhxWPSJiNwIiz4RkRth0SciciMs+kREboRF\nn4jIjTi86Ofk5CAgIAA6nQ6zZ8929NMTEbk1D0eu06+urkanTp3w7bffQqPRoEePHvjss88QGBjo\nqAhERG7NoSP9vXv3wt/fH76+vvDy8sKIESOQnZ3tyAhERG7N05FPZjQa4ePjY/5cq9UiNzf3uu/x\n8PBwZCQiItWwZOLGoSN9Swu6oiiqvc2YMUN6Br42vj6+PvXdLOXQoq/RaFBSUmL+vKSkBFqt1pER\niIjcmkOLfvfu3WEwGHDixAmYTCasXbsWQ4cOdWQEIiK35tA5fU9PTyxevBgDBgxAdXU1xo0b53Yr\ndyIjI2VHsBs1vzaAr8/Vqf31WcqhSzYt4eHhUav5KSKZLl4EjEbg9GmgogL45Rfgt9/EzWQCGjQA\nGjcWtxYtgLZtxe2eewAvL9npSU0srZ0s+kQWUBTg5Engn/8E9u0D/v1vcfvpJ0CjEUW8TRugVSug\nYUPA21vcTCagqkr8cfjlF6CsTNzOnAHatwfuvx/o3Bno3h3o1Us8nsgaLPpEdfTTT0BODrBpE7B9\nu7ivVy8gLEwU6sBAwNcXqF+/9j/70iXg6FHgyBHgX/8C9u4FcnPFH5A+fYAhQ4C+fcU7BCJLsOgT\nWaGsDFizBli7FigsFIV3yBDg4YdFgbfnNpIrV4DDh4HvvgO++grIywMeegh4+mng8ceBO+6w33OT\n62PRJ7LQpUvA+vXAypXA/v2iwMbGioLr7S0v16+/incZn34K7NoFPPYYMHo0EBlp3z8+5JpY9Ilu\no6wMWLoU+PBDIDgYePZZIDoaaNRIdrIbnT4NfPYZ8NFHQL16wAsvACNHcvqH/h+LPtEtHD8OJCcD\nmZliRD9xopifdwWKAmzbBixYAOzZA0yaBEyeDDRrJjsZyWZp7WQ/fXIbx46J6ZHwcECrFZ9/8IHr\nFHxATOv06wd8+aWY8ikqAvz9gbfeAs6fl52OXAGLPqne2bNiOiQiQiyTPHYMSEoCWraUnaxuOnYU\n1yH++U/x7kWnAxYtAi5flp2MnBmLPqmWyQTMnw8EBADV1WJd/YwZQPPmspPZlk4HfPyxWPXz5ZdA\nly7A5s2yU5Gz4pw+qdKuXeLCbLt2wLx5QFCQ7ESOoShixc9LL4l3Ah98ANx3n+xU5Aic0ye3dP48\nkJAADB8u5rk3bXKfgg+IOf8hQ8R6/7AwoFs3YPFi8U6HCGDRJxXZvFnslK2uFrtcn3zSfdeze3sD\nb7wB7NwpNps9+KDYbEbEok8ur6pKLF187jkgI0Osu2/RQnYq5xAYCOzYAcTFAQ88INb5c/bUvXFO\nn1zawYNik1KXLkBqKot9TQoKRPFv314U/9atZSciW+KcPqmaogDLlgFRUcD06aJVAQt+zYKCRFM3\nf38gNFQs9ST3w5E+uZyqKmDCBNEn54svxJJFqp2vvwbGjAHefBN4/nn3vfahJhzpkyoVFYn2xpcu\niTYELPjWGTIE2L0bWL4ciI8X/f7JPbDok8vYvl0U/DFjgE8+YavhuvLzE1M8Hh7iIm9pqexE5Ags\n+uQSVq0CnnpKFPsXXuB0hK00bix288bFAT17Avn5shORvXFOn5yaoojWCatXi4NF3GmjlaNt2CCW\nvaanixbT5FosrZ2eDshCZBWTSUzlFBWJ+fu775adSN2eeALw8RGHyJw8KVpOk/qw6JNTunhRFKEG\nDUQjMWc82ESNwsJE36L+/UV30jff5FSa2nBOn5zOuXOi6Nx9tzjGkAXfsXx9RfuGrCxgyhTg6lXZ\niciWWPTJqZw+Lc6A/ctfgBUrAE++F5XinnuA778XF3bHjhWHtpM6sOiT0zAagT59xJzy+++Ls2BJ\nnubNgW++AX76CXj6aR7OohZcvUNOobxcjPDHjgWmTZOdhq516ZLoWNqggTic3ctLdiK6Ge7IJZdR\nUQE8/LA4v5YF3/k0aCCurVRVieZ2nOpxbSz6JNXp06LgjxolGqeRc2rQQKzjv3AB+OtfWfhdmc2L\nfmJiIrRaLUJDQxEaGorN1xzWmZKSAp1Oh4CAAGzZssXWT00u5swZ4JFHxG7Q11+XnYZup2FD0eDu\n3DnxroyrelyTzef0k5KScOedd+Kll1667v6CggLExcVh3759MBqN6NevH44ePYp6f7paxzl993Dh\nAtC3ryj6KSmy01BtVFUBAwYAISHAwoVcx+8spM7p3+yJs7OzERsbCy8vL/j6+sLf3x979+61x9OT\nk7t0CRg2TBx8kpwsOw3VVqNGwMaNYi3/22/LTkO1ZZdV0IsWLcKqVavQvXt3zJs3D82bN0dZWRki\nIiLM36PVamE0Gm/6+MTERPPHkZGRiIyMtEdMkqC6WrTybdIEWLKEo0RX1bw5kJMD9O4NtGolevKT\nY+n1euj1+lo/zqrpnaioKFRUVNxw/8yZMxEREYG77roLAPDGG2+gvLwcaWlpmDRpEiIiIjBy5EgA\nwPjx4zF48GAMGzbs+kCc3lEtRRGHnxQWAps2iTlicm3FxeLQ9blzgdhY2Wncm10brm3dutWi7xs/\nfjyif2/Xp9FoUFJSYv5aaWkpNBqNNU9PLiolRRzXp9ez4KtF+/bA5s3i+oxGIzbXkXOz+Zx+eXm5\n+ePMzEwEBwcDAIYOHYo1a9bAZDKhuLgYBoMBYWFhtn56clJr1wJLl4r2yE2byk5DthQcLM45GD4c\nOHpUdhq6HZvP6U+bNg0HDx6Eh4cH2rdvj2XLlgEAgoKCMHz4cAQFBcHT0xOpqanw4ISuW9i9W7Tp\n3boVaNtWdhqyh6gocVH3j2MYW7eWnYhuhW0YyK6Ki8URh8uXi4JA6jZtmjiC8dtvxYYuchxLayeL\nPtnNuXOi4CckAJMmyU5DjnD1qpjmadgQyMjg6ixHYtEnqaqrgUcfBfz9gUWLZKchR6qqEit6nn4a\nmDpVdhr3weMSSao33wR++w147z3ZScjRGjUCMjPFKVwhIeJAHHIebLhGNrdhgzjI/PPP2YbXXfn4\niBVbo0YBx4/LTkPX4vQO2dSRI6Ivfk6OOP2K3NvixcCHH4qLu02ayE6jbpzTJ4c7dw7o0UNM7Ywa\nJTsNOQNFAcaNAyorxTs/Xti1HxZ9cqirV4GhQwE/P2DBAtlpyJn89pvYqTtiBPCn5rtkQ7yQSw71\n7rvAL7+IC3hE12rYUIzyw8LEEt5r+i6SBBzpU53t2iVaJe/bB9x3n+w05Kyys4EXXgDy80VnTrIt\nTu+QQ5w9C4SGAqmpYl0+UU1efll0Wd24EajHtYM2xaJPdvfHPH5goGitS3Q7ly+L+f1hw7hxy9ZY\n9Mnu3n1XrMnfsYPr8clyp06JVV5ffAE88IDsNOrBok92lZsrRvmcxydrfPml6Md06BDQrJnsNOrA\nok92c+EC0LUrMHs28MQTstOQq0pIEOv3V6+WnUQdWPTJbp59VjRUS0+XnYRc2cWLQLduwIwZPGrR\nFrhOn+wiMxP47jvg4EHZScjVNW4MfPopMHCgWL/frp3sRO6BI32yWHm5WJ75xRfiHymRLcyeDXz9\nNfD990D9+rLTuC5LaydXypJFFAUYMwb4n/9hwSfbeuUVUeznzJGdxD1wpE8WWbwYWLVK7L7l8kyy\ntZIS0ZV1yxaxSIBqjxdyyWYMBqBnT9Eet2NH2WlIrVauBN5/H9i7F/D2lp3G9XB6h2zi6lXRGvf1\n11nwyb6eeQbQaoHkZNlJ1I0j
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 12
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"mean(D)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 13,
|
||
|
|
"text": [
|
||
|
|
"0.39899698828239921"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 13
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"ML estimator of Mean and Variance\n",
|
||
|
|
"=================================\n",
|
||
|
|
"\n",
|
||
|
|
"Consider the 1D Gaussian, but assume that we want to estimate both $\\mu$ and $\\sigma^2$\n",
|
||
|
|
"\n",
|
||
|
|
"Let's use the variance $v=\\sigma^2$\n",
|
||
|
|
"\n",
|
||
|
|
"The log likelihood is\n",
|
||
|
|
"$$\\def\\pd#1{\\frac{\\partial}{\\partial#1}}$$\n",
|
||
|
|
"\n",
|
||
|
|
"$$l(\\mu,\\sigma^2) = \\sum_i^n \\ln p(x_i|\\mu,v) = -{1\\over2} \\ln 2\\pi v - {1 \\over {2v}} (x_i - \\mu)^2$$\n",
|
||
|
|
"\n",
|
||
|
|
"As before, to find a potential maximum, \n",
|
||
|
|
"we calculate the derivatives of the log likelihood function and set them equal to zero\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\pd{\\mu} l = \\sum_i^n {1 \\over {\\hat{v}}} (x_i - \\hat{\\mu}) = 0$$\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\pd{v} l = \\sum_i^n - {1\\over {2\\hat{v}}} + \\frac{(x_i-\\hat{\\mu})^2}{2\\hat{v}^2} = 0$$\n",
|
||
|
|
"\n",
|
||
|
|
"So, from the first, we obtain again that\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\hat{\\mu} = {1 \\over n} \\sum_i^n x_i$$\n",
|
||
|
|
"\n",
|
||
|
|
"For the second, we get\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\pd{\\hat{v}} l = \\sum_i^n - {1\\over {2\\hat{v}}} + \\frac{(x_i-\\hat{\\mu})^2}{2\\hat{v}^2} = 0$$\n",
|
||
|
|
"\n",
|
||
|
|
"Multiply by $v^2$, we get\n",
|
||
|
|
"\n",
|
||
|
|
"$$ n v = \\sum_i^n (x_i-\\hat{\\mu})^2 $$\n",
|
||
|
|
"\n",
|
||
|
|
"Or\n",
|
||
|
|
"\n",
|
||
|
|
"$$ \\sigma^2 = v = \\frac{1}{n} \\sum_i^n (x_i-\\hat{\\mu})^2 $$"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Bias\n",
|
||
|
|
"====\n",
|
||
|
|
"\n",
|
||
|
|
"Above, we calculated the maximum likelihood estimate of the variance.\n",
|
||
|
|
"\n",
|
||
|
|
"Wouldn't it be nice if the variance were actually, on average, the same a the true variance?\n",
|
||
|
|
"\n",
|
||
|
|
"Let's see whether that is true.\n",
|
||
|
|
"\n",
|
||
|
|
"The expected value of the variance is\n",
|
||
|
|
"\n",
|
||
|
|
"$$E[\\frac{1}{n}\\sum_i^n (x_i-\\bar{x})]$$\n",
|
||
|
|
"\n",
|
||
|
|
"Consider the case $n=1$.\n",
|
||
|
|
"\n",
|
||
|
|
"Then $\\bar{x} = \\hat{\\mu} = x_1$, so\n",
|
||
|
|
"\n",
|
||
|
|
"$$E[\\frac{1}{1}\\sum_{i=1}^1 (x_1-x_1)] = 0$$\n",
|
||
|
|
"\n",
|
||
|
|
"Well, $0 \\neq \\sigma^2$\n",
|
||
|
|
"\n",
|
||
|
|
"Tt turns out that an unbiased estimator for variance is\n",
|
||
|
|
"\n",
|
||
|
|
"$$E[\\frac{1}{n-1}\\sum_i^n (x_i-\\bar{x})]$$\n",
|
||
|
|
"\n",
|
||
|
|
"Note that this is undefined for $n=1$, so the above problem doesn't exist.\n",
|
||
|
|
"\n",
|
||
|
|
"Some terminology:\n",
|
||
|
|
"\n",
|
||
|
|
"- If an estimator is not completely unbiased but becomes approximately unbiased \n",
|
||
|
|
"for large $n$, we call it {\\em asymptotically unbiased}\n",
|
||
|
|
"- If an estimator is unbiased for all distributions, it is called {\\em absolutely unbiased}.\n",
|
||
|
|
"\n",
|
||
|
|
"So, we now have two different estimators for the variance:\n",
|
||
|
|
"\n",
|
||
|
|
"- the first one maximizes the likelihood\n",
|
||
|
|
"- the second one has an expected value that equals the true value\n",
|
||
|
|
"\n",
|
||
|
|
"Which one should we choose?\n",
|
||
|
|
"\n",
|
||
|
|
"There is no correct answer; the question itself is wrong.\n",
|
||
|
|
"\n",
|
||
|
|
"The reason is that our knowledge about the true variance is uncertain.\n",
|
||
|
|
"Any single value we estimate for the variance is incomplete \n",
|
||
|
|
"without a statement of the uncertainty associated with that value\n",
|
||
|
|
"\n",
|
||
|
|
"The point here is not that you should pick unbiased estimators (you may still choose\n",
|
||
|
|
"to do so if you have nothing better to do), rather it is to illustrate that\n",
|
||
|
|
"there are many choices in statistics that sound good on the surface but really \n",
|
||
|
|
"don't accomplish anything useful."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Bayesian Parameter Estimation\n",
|
||
|
|
"=============================\n",
|
||
|
|
"\n",
|
||
|
|
"In Bayesian parameter estimation, we need to decide what purpose we want \n",
|
||
|
|
"to use the estimate for.\n",
|
||
|
|
"\n",
|
||
|
|
"For example, we might want to estimate the parameters of a normal\n",
|
||
|
|
"density in order to perform classification.\n",
|
||
|
|
"\n",
|
||
|
|
"In a Bayesian approach, we use all available data:\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(\\omega_j|x) = p(\\omega_j|x,D)$$\n",
|
||
|
|
"\n",
|
||
|
|
"In order to perform estimation, we consider the case where...\n",
|
||
|
|
"\n",
|
||
|
|
"- the form of the density $p(x|\\theta)$ is known (parametric estimation)\n",
|
||
|
|
"- we assume that samples for each class can only give us information about that class\n",
|
||
|
|
"- assume we have a prior density $p(\\theta)$ for the parameters\n",
|
||
|
|
"\n",
|
||
|
|
"Then\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(x|D) = \\int p(x|\\theta) p(\\theta|D) d\\theta$$\n",
|
||
|
|
"\n",
|
||
|
|
"Now by Bayes formula,\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(\\theta|D) = \\frac{p(D|\\theta)p(\\theta)}{p(D)}$$\n",
|
||
|
|
"\n",
|
||
|
|
"and by independence\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(D|\\theta) = \\prod_{k=1}^n p(x_k|\\theta)$$\n",
|
||
|
|
"\n",
|
||
|
|
"Note that the maximum likelihood case is equivalent to this if we choose \n",
|
||
|
|
"\n",
|
||
|
|
"$p(\\theta|D)=\\delta(\\theta,\\hat{\\theta})$\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(x|\\omega_j,D) = \\int p(x|\\theta) \\delta(\\theta,\\hat{\\theta}) d\\theta$$\n",
|
||
|
|
"\n",
|
||
|
|
"Whether the maximum likelihood solution works well therefore depends on how\n",
|
||
|
|
"closely $p(\\theta|D)$ approximates a $\\delta(\\theta,\\hat{\\theta})$ \n",
|
||
|
|
"(note that \"how closely\" depends also on the smoothness of $p(x|\\theta)$)\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Parameter Updates\n",
|
||
|
|
"=================\n",
|
||
|
|
"\n",
|
||
|
|
"Above, we have seen how to compute distributions of parameters given\n",
|
||
|
|
"some samples.\n",
|
||
|
|
"\n",
|
||
|
|
"In many cases, it turns out that the distribution of parameters itself\n",
|
||
|
|
"has a parameteric form closely related to the parametric form of the\n",
|
||
|
|
"distribution we are trying to estimate.\n",
|
||
|
|
"\n",
|
||
|
|
"For example, the distribution of the estimates of a mean of a Gaussian\n",
|
||
|
|
"is itself a Gaussian."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"For now, let's look at a particularly simple case.\n",
|
||
|
|
"Assume we are given $p(\\theta)$ and a measurement $x$\n",
|
||
|
|
"we want to compute $p(\\theta|x)$, the distribution of the parameter values\n",
|
||
|
|
"assuming we are given a measurement.\n",
|
||
|
|
"\n",
|
||
|
|
"So\n",
|
||
|
|
"$$p(\\theta|x) = \\frac{p(x|\\theta) p(\\theta)}{p(x)}$$\n",
|
||
|
|
"\n",
|
||
|
|
"Here,\n",
|
||
|
|
"$$p(x) = \\int p(x|\\theta) p(\\theta) dx$$\n",
|
||
|
|
"\n",
|
||
|
|
"Let's look at this in the case of estimating the probability parameter \n",
|
||
|
|
"of a binomial distribution.\n",
|
||
|
|
"\n",
|
||
|
|
"For a given $q$, we have $p(k) = {n \\choose k} \\theta^k (1-\\theta)^{n-k}$\n",
|
||
|
|
"(the parameter $\\theta$ is just the probabilit; I'm avoiding writing $p$ because we have too many $p$'s already)\n",
|
||
|
|
"since, initially, we know nothing about $\\theta$, let's assume a uniform prior\n",
|
||
|
|
"\n",
|
||
|
|
"$p(\\theta) = 1$ \n",
|
||
|
|
"\n",
|
||
|
|
"this is conveniently normalized, since $\\theta\\in[0,1]$\n",
|
||
|
|
"\n",
|
||
|
|
"So, what do we get?\n",
|
||
|
|
"\n",
|
||
|
|
"Well, our measurement is $x=k$, so let's replace that:\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(\\theta|k) = \\frac{p(k|\\theta) p(\\theta)}{p(k)}$$\n",
|
||
|
|
"$$ = \\frac{{n\\choose k} \\theta^k (1-\\theta)^{n-k}}{\\int {n\\choose k} \\theta^k (1-\\theta)^{n-k}}$$\n",
|
||
|
|
"$$ = \\frac{ \\theta^k (1-\\theta)^{n-k}}{\\int \\theta^k (1-\\theta)^{n-k}}$$\n",
|
||
|
|
"\n",
|
||
|
|
"The denominator is called a beta function. It serves as a normalizing factor.\n",
|
||
|
|
"\n",
|
||
|
|
"We can write\n",
|
||
|
|
"$$ p(\\theta|k) = \\frac{ \\theta^k (1-\\theta)^{n-k}}{B(k,n-k)}$$\n",
|
||
|
|
"\n",
|
||
|
|
"This is the density function for the beta distribution\n",
|
||
|
|
"\n",
|
||
|
|
"Note that the beta FUNCTION and the beta DISTRIBUTION are different things\n",
|
||
|
|
"we will be writing $B(.)$ for the beta function and $\\beta(.)$ for the beta distribution\n",
|
||
|
|
"the beta function is the normalizing factor for the beta distribution.\n",
|
||
|
|
"\n",
|
||
|
|
"The formula for the beta distribution looks like the binomial distribution, with some\n",
|
||
|
|
"of the parameters taking on different meanings."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Incremental Updates and Conjugate Priors\n",
|
||
|
|
"========================================\n",
|
||
|
|
"\n",
|
||
|
|
"Above, we looked at $p(\\theta|D)$ for a single data set $D$ (observation of $k,n$ in this case)\n",
|
||
|
|
"\n",
|
||
|
|
"What if we make a second observation $k_2,n_2$?\n",
|
||
|
|
"\n",
|
||
|
|
"You can observe that having $n_1$ Bernoulli trials with $k_1$, giving\n",
|
||
|
|
"parameter estimate $\\theta_1$, followed by $n_2$ Bernoulli trials with $k_2$ successes, \n",
|
||
|
|
"giving parameter estimate $\\theta_2$ is the same as if we had just performed a single\n",
|
||
|
|
"experiment with $n_1+n_2$ trials and $k_1+k_2$ successes\n",
|
||
|
|
"\n",
|
||
|
|
"That means that the parameter distribution after the first update should give\n",
|
||
|
|
"$p(\\theta|k_1,n_1) = \\beta(\\theta;k_1,n_1)$\n",
|
||
|
|
"\n",
|
||
|
|
"And after the second update\n",
|
||
|
|
"$p(\\theta|k_1,n_1;k_2,n_2) = p(\\theta|k_1+k_2,n_1+n_2) = \\beta(\\theta;k_1+k_2,(n_1+n_2)-(k_1+k_2))$\n",
|
||
|
|
"\n",
|
||
|
|
"How does this look when we update the parameters directly?\n",
|
||
|
|
"\n",
|
||
|
|
"We have again\n",
|
||
|
|
"$$p(\\theta|x) = \\frac{p(x|\\theta) p(\\theta)}{p(x)}$$\n",
|
||
|
|
"\n",
|
||
|
|
"Here,\n",
|
||
|
|
"$$p(x) = \\int p(x|\\theta) p(\\theta) dx$$\n",
|
||
|
|
"\n",
|
||
|
|
"and our prior is not uniform anymore, but the Beta distribution that we obtained\n",
|
||
|
|
"from the first experiment\n",
|
||
|
|
"$p(\\theta) = \\beta(\\theta;k_1,n_1-k_1)$\n",
|
||
|
|
"\n",
|
||
|
|
"So, writing $b$ for the binomial probability,\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(k_2,n_2) = \\frac{b(k_2;n_2,\\theta) \\beta(\\theta;k_1,n_1)}{\\int b(k_2;n_2,\\theta) \\beta(\\theta;k_1,n_1) d\\theta}$$\n",
|
||
|
|
"\n",
|
||
|
|
"Now, recall that the binomial probabilities are\n",
|
||
|
|
"$$b(k;n,\\theta) = {n \\choose k} \\theta^k (1-p)^{n-k}$$\n",
|
||
|
|
"\n",
|
||
|
|
"And the Beta distribution is\n",
|
||
|
|
"$$\\beta(\\theta;a,b) = \\frac{1}{B(a,b)} \\theta^{a-1} (1-\\theta)^{b-1}$$\n",
|
||
|
|
"\n",
|
||
|
|
"So, let $k=a-1$ and $n-k=b-1$, or $a = k+1$ and $b=n-k+1$:\n",
|
||
|
|
"$$b(k;n\\theta) = \\beta(\\theta;k+1,n-k+1) B(k+1,n-k+1)$$\n",
|
||
|
|
"\n",
|
||
|
|
"We can plug that back in:\n",
|
||
|
|
"$$p(k_2,n_2) = \\frac{\\beta(\\theta;k_2+1,n_2-k_2+1) \\beta(\\theta;k_1,n_1)} {\\int \\beta(\\theta;k_2+1,n_2-k_2+1) \\beta(\\theta;k_1,n_1) d\\theta}$$\n",
|
||
|
|
"\n",
|
||
|
|
"What about the products of beta distributions?\n",
|
||
|
|
"$$\\beta(\\theta;a,b)\\beta(\\theta;a',b') = \\frac{1}{B(a,b)B(a',b')} \\theta^{a+a'-2} (1-\\theta)^{b+b'-2} = \\frac{B(a+a',b+b')}{B(a,b)B(a',b')} \\beta(a+a'-1,b+b'-1)$$\n",
|
||
|
|
"\n",
|
||
|
|
"So\n",
|
||
|
|
"$$p(k_2,n_2) = \\frac{C\\cdot\\beta(\\theta;k_1+k_2,(n_1+n_2)-(k_1+k_2))} {\\int C\\cdot\\beta(\\theta;k_1+k_2,(n_1+n_2)-(k_1+k_2)) d\\theta}$$\n",
|
||
|
|
"\n",
|
||
|
|
"Pull out the $C$ from the integral and cancel it and we're left with an integral \n",
|
||
|
|
"over a $\\beta$ distribution,\n",
|
||
|
|
"which is 1, so\n",
|
||
|
|
"$$p(k_2,n_2) = \\beta(\\theta;k_1+k_2,(n_1+n_2)-(k_1+k_2))$$\n",
|
||
|
|
"\n",
|
||
|
|
"So, what we have in this case is that starting with a uniform density as a prior,\n",
|
||
|
|
"every time we take another binomial sample, we get another beta distribution as the\n",
|
||
|
|
"density for the parameter we are trying to estimate.\n",
|
||
|
|
"\n",
|
||
|
|
"We say that the beta distribution is a _conjugate prior_ for the binomial distribution,\n",
|
||
|
|
"and we refer to the density as a _reproducing density_"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Updates to the Normal Density\n",
|
||
|
|
"===============================\n",
|
||
|
|
"\n",
|
||
|
|
"Let's repeat this exercise for the normal density\n",
|
||
|
|
"the parameter we are trying to estimate is $\\mu$\n",
|
||
|
|
"assume our prior is itself normal \n",
|
||
|
|
"\n",
|
||
|
|
"$p(\\mu) = N(\\mu;\\mu_0,\\sigma_0)$\n",
|
||
|
|
"\n",
|
||
|
|
"Furthemore, we assume that\n",
|
||
|
|
"\n",
|
||
|
|
"$p(x) = N(x;\\mu,\\sigma)$\n",
|
||
|
|
"\n",
|
||
|
|
"Where $\\mu$ is unknown, but $\\sigma$ is known and given\n",
|
||
|
|
"\n",
|
||
|
|
"Now, given a dataset $D = \\{x_1,...,x_n\\}$,\n",
|
||
|
|
"$$p(\\mu|D) = \\frac{p(D|\\mu)p(\\mu)}{\\int p(D|\\mu)p(\\mu)d\\mu} = C \\prod_i^n N(x_i;\\mu,\\sigma) N(\\mu;\\mu_0,\\sigma_0)$$\n",
|
||
|
|
"\n",
|
||
|
|
"Dropping the constant factors, we get\n",
|
||
|
|
"$$p(\\mu|D) \\propto\\prod_i^n \\exp(\\frac{-1}{2\\sigma^2}(x_i-\\mu)^2) \\exp(\\frac{-1}{2\\sigma_0^2}(\\mu-\\mu_0)^2)$$\n",
|
||
|
|
"$$= \\prod_i^n \\exp(\\frac{-1}{2\\sigma^2}(x_i-\\mu)^2 + \\frac{-1}{2\\sigma_0^2}(\\mu-\\mu_0)^2)$$\n",
|
||
|
|
"$$= \\exp(\\sum_i (\\frac{-1}{2\\sigma^2}(x_i-\\mu)^2 + \\frac{-1}{2\\sigma_0^2}(\\mu-\\mu_0)^2))$$\n",
|
||
|
|
"\n",
|
||
|
|
"You can work out that this is equal to a quadratic function for some parameters $a,b,c$:\n",
|
||
|
|
"$$= \\exp(\\sum_i a\\mu^2 + b\\mu + c)$$\n",
|
||
|
|
"\n",
|
||
|
|
"Furthermore, we know that the output must be a density, so the quadratic must be $\\geq 0$\n",
|
||
|
|
"what all of that means is that, for some $\\mu_1,\\sigma_1$,\n",
|
||
|
|
"$p(\\mu|D) = N(\\mu;\\mu_1,\\sigma_1)$\n",
|
||
|
|
"\n",
|
||
|
|
"We can compute $\\mu_1,\\sigma_1$, and eventually get\n",
|
||
|
|
"$$\\mu_1 = \\frac{n\\sigma_0^2}{n\\sigma_0^2+\\sigma^2}\\hat{\\mu}_1 + \\frac{\\sigma^2}{n\\sigma_0^2+\\sigma^2}\\mu_0$$\n",
|
||
|
|
"$$\\sigma_1^2 = \\frac{\\sigma_0^2\\sigma^2}{n\\sigma_0^2+\\sigma^2}$$"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Incremental Updates in General\n",
|
||
|
|
"===============================\n",
|
||
|
|
"\n",
|
||
|
|
"Recall\n",
|
||
|
|
"$$p(x|D) = \\int p(x|\\theta) p(\\theta|D) d\\theta$$\n",
|
||
|
|
"\n",
|
||
|
|
"Recall now by Bayes formula,\n",
|
||
|
|
"$$p(\\theta|D) = \\frac{p(D|\\theta)p(\\theta)}{p(D)}$$\n",
|
||
|
|
"\n",
|
||
|
|
"where $p(D) = \\int p(D|\\theta)p(\\theta)d\\theta$\n",
|
||
|
|
"and by independence\n",
|
||
|
|
"$$p(D|\\theta) = \\prod_{k=1}^n p(x_k|\\theta)$$\n",
|
||
|
|
"\n",
|
||
|
|
"Now write $D^n$ for the first $n$ samples\n",
|
||
|
|
"\n",
|
||
|
|
"Then, trivially\n",
|
||
|
|
"$$p(D^n|\\theta) = p(x_n|\\theta) p(D^{n-1}|\\theta)$$\n",
|
||
|
|
"\n",
|
||
|
|
"Substituting that above, we get\n",
|
||
|
|
"$$p(\\theta|D^n) = \\frac{p(x_n|\\theta) p(D^{n-1}|\\theta) p(\\theta)}{p(D^n)}$$\n",
|
||
|
|
"where $p(D) = \\int {p(x_n|\\theta) p(D^{n-1}|\\theta) p(\\theta)}d\\theta$\n",
|
||
|
|
"\n",
|
||
|
|
"Applying Bayes formula and canceling the $p(D^{n-1})$ factor, we get\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(\\theta|D^n) = \\frac{p(x_n|\\theta) p(\\theta|D^{n-1})} {\\int p(x_n|\\theta) p(\\theta|D^{n-1}|\\theta)d\\theta}$$\n",
|
||
|
|
"\n",
|
||
|
|
"This formula tells us how to update our density for the parameter $\\theta$ when a new sample $x_n$ arrives\n",
|
||
|
|
"\n",
|
||
|
|
"Note that, up to normaliation:\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(\\theta|D^n) \\propto p(x_n|\\theta) p(\\theta|D^{n-1})$$"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Incremental Update Example\n",
|
||
|
|
"===========================\n",
|
||
|
|
"\n",
|
||
|
|
"Assume we know that $x \\sim {\\cal N}(\\mu)$, where $p(\\mu) = {\\cal N}(1,1)$."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"xs = linspace(-4,6,1000)\n",
|
||
|
|
"C = (amax(xs)-amin(xs))/xs.size\n",
|
||
|
|
"pmu0 = norm.pdf(xs,loc=1)\n",
|
||
|
|
"plot(xs,pmu0)\n",
|
||
|
|
"C*sum(pmu0)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 184,
|
||
|
|
"text": [
|
||
|
|
"0.99899944201334256"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD9CAYAAAC2l2x5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X1c1FW+B/DPCJOluD5FPsyQk4DOaIAkSFTUWAJmNfnA\nvbGatUYst7batrZ9bfe2N+3eVtnd7uaKFbuVXdM1blkOFpCSjhoBY2ppUoYGOZK0PkQ+sDrO+Lt/\nHBkFZRhgZs48fN6v17xknN+Z+TDKlzPnd37nqBRFUUBERCGnj+wARETkGyzwREQhigWeiChEscAT\nEYUoFngiohDFAk9EFKK6LPAVFRXQ6/WIj49HYWFhp8dt3boVkZGRWL16dbfbEhGR96nczYN3Op0Y\nO3YsKisrodFokJqailWrVsFgMFx0XGZmJvr164d58+Zh1qxZHrclIiLfcNuDt1qtiIuLg06ng1qt\nRm5uLsxm80XHLVmyBDk5OYiOju52WyIi8o1Idw82NTUhJibGdV+r1aK2tvaiY8xmMzZs2ICtW7dC\npVJ53LbtWCIi6h5PFiFw24P3pAA//vjjWLRoEVQqFRRFcb2op8W7rU2435599lnpGQLlJuu9cDgU\nLF2qICZGQXy8guxsBRMmKBg8WMFvf6vg+PHweS8C8cb34vzNU2578BqNBjabzXXfZrNBq9W2O2bb\ntm3Izc0FABw+fBjl5eVQq9UetSUKFIcOAbNnA6dPA++9B0yceP6xxkbgd78DJkwA3n8f0OulxSTq\nFrcFPiUlBfX19WhsbMTIkSNRUlKCVatWtTvmm2++cX09b9483HXXXTCZTHA4HF22JQoEhw4BRiNw\nxx3A738PRHb4qdDpgDffBJYtE8eVlQHXXSchKFE3uS3wkZGRKCoqQnZ2NpxOJ/Ly8mAwGFBcXAwA\nKCgo6HZbujSj0Sg7QsDw53vxz38CU6cCM2cC//Vf7o+dNw/4yU+Au+4CPvkEGDXK9/n4/+I8vhfd\n53aapM9f/Ny4PZEMigL87GeAwwGsWAF4es7/f/5HHF9dDfTt69OIRJfkae3klawUtkpKgK1bgb/+\n1fPiDgC/+hVw9dXAM8/4LhuRN7AHT2HpyBHg2mvFCdXrr+9++8OHgYQEoLQUSE31fj4idzytnSzw\nFJbmzQMGDgRefLHnz/G//wssXQrU1AB9+FmY/IgFnqgTO3YA06YBX38NDBjQ8+c5exbIyBDj+Pn5\nXotH1CUWeKJOZGcDd98NPPxw759r2zbAZAL27gWuuKL3z0fkCZ5kJbqEjz4CvvnGez3uiROBSZOA\nV17xzvMReRN78BRWjEZR3OfM8d5zfv65+FSwbx/Qv7/3npeoM+zBE3VQXQ18+y1wzz3efd6kJOCm\nm4C//c27z0vUW+zBU9iYPh3IyvLO2HtHtbVAbq4Yi4+I8P7zE12IPXiiC+zZI3rw8+b55vnT0oCR\nI4E1a3zz/EQ9wQJPYeHll4G8PN/OdHniCeCFF3z3/ETdxSEaCnmtrWJpgW3bfLtAmNMJjB4tevHJ\nyb57HSIO0RCd89ZbQHq671d/jIgQnxJ4spUCBXvwFPJSUoDnnhNXr/ragQNAYiJgs3HKJPkOe/BE\nALZvFwuDZWf75/W0WuCGG4C33/bP6xG5wwJPIe3NN4H77vPv1MX8fA7TUGDgEA2FLIdD9Ki3bAHi\n4/37ujExwKZNwJgx/ntdCh8coqGwt3692E/Vn8UdEHu63nMPwC2ISTYWeApZb74JzJ0r57VnzwZW\nrhTbAhLJ0mWBr6iogF6vR3x8PAoLCy963Gw2IykpCcnJyZg4cSI2bNjgekyn0yExMRHJycmYNGmS\nd5MTuXH8OFBW5v11ZzyVmiqK+6efynl9IqCLMXin04mxY8eisrISGo0GqampWLVqFQwGg+uYkydP\nov+5+WC7du3CjBkzsHfvXgDANddcg23btmHIkCGXfnGOwZOPvPGG2I7PbJaXYf584McfgT//WV4G\nCk1eGYO3Wq2Ii4uDTqeDWq1Gbm4uzB1+YvpfMNn3xIkTuPLKK9s9zgJOMvzf/wE//ancDLNni4us\nnE65OSh8Rbp7sKmpCTExMa77Wq0WtbW1Fx23Zs0aPP300zh48CDWrVvn+nuVSoUpU6YgIiICBQUF\nyL/ELgvz5893fW00GmE0GnvwbRCd19ICfPwxUFIiN8eYMcDw4UBVFXDzzXKzUHCzWCywWCzdbue2\nwKtUKo+eZPr06Zg+fTq2bNmCuXPnYs+ePQCAqqoqjBgxAocOHUJmZib0ej0yMjLatb2wwBN5w9q1\nwK239m6/VW+ZNQtYvZoFnnqnY+d3wYIFHrVzO0Sj0Whgs9lc9202G7RabafHZ2RkwOFw4MiRIwCA\nESNGAACio6MxY8YMWK1Wj0IR9cY774jCGghmzgTefZezaUgOtwU+JSUF9fX1aGxshN1uR0lJCUwm\nU7tj9u3b5xpn3759OwBg6NChaG1txfHjxwGIE7Hr1q1DQkKCL74HIpfjx4GNG4G77pKdRBg3DoiK\nArZulZ2EwpHbIZrIyEgUFRUhOzsbTqcTeXl5MBgMKC4uBgAUFBRg9erVWL58OdRqNaKiovDWW28B\nAJqbmzFz5kwAgMPhwJw5c5CVleXjb4fC3QcfABkZwKBBspOc19aL50xh8jcuVUAhJSdHrBr5wAOy\nk5y3bZvYzu/rrwEPT2sRueVp7WSBp5Bx6hQwbBiwbx/QYbauVIoCXHMNUFoqlhIm6i2uRUNhx2IB\nEhICq7gDotc+Y4bci64oPLHAU8h4//3AObna0Z13ivMDRP7EIRoKCYoiVo4sKwPGj5ed5mJ2O3DV\nVcCePWIYiag3OERDYeWLL8SmHuPGyU5yaZddBkyZApSXy05C4YQFnkLC+++LYZBAnqXCYRryNxZ4\nCglr14oCGshuv11sQmK3y05C4YIFnoLeoUPA7t3ALbfITuLesGHA2LFiITQif2CBp6BXVibGt/v2\nlZ2ka3feKYaTiPyBBZ6CXnm5uHo1GLDAkz9xmiQFNadTDH189hngZqHTgKEogEYDbN4MxMXJTkPB\nitMkKSxs3y4KfDAUd0DM8snKEidbiXyNBZ6C2ocfioIZTLKyRG4iX2OBp6C2bh2QnS07RfdkZop1\nc86ckZ2EQh0LPAWtY8eAHTuCbzu86GggNha4xPbGRF7FAk9Ba+NG4PrrgX79ZCfpPg7TkD+wwFPQ\nCsbx9zbZ2WJ4iciXOE2SglZsLPDee8G5icbp02KoprERGDJEdhoKNpwmSSFt3z6gtVVs8BGM+vYV\n5w4qK2UnoVDWZYGvqKiAXq9HfHw8CgsLL3rcbDYjKSkJycnJmDhxIjZs2OBxW6KeahueCeTVI7vC\nYRryOcUNh8OhxMbGKg0NDYrdbleSkpKUurq6dsecOHHC9fXOnTuV2NhYj9t28fJEnbr7bkVZuVJ2\nit756itFiYlRlLNnZSehYONp7Yx0V/ytVivi4uKg0+kAALm5uTCbzTAYDK5j+vfv7/r6xIkTuPLc\nhpietAWA+fPnu742Go0wGo09/mVF4cHhEPPIi4tlJ+mdMWPEn3v2AHq93CwU2CwWCywWS7fbuS3w\nTU1NiImJcd3XarWovcTk3TVr1uDpp5/GwYMHse7cZ05P215Y4Ik8sWOHWJog2Le+U6mA224T0z1Z\n4Mmdjp3fBQsWeNTO7Ri8ysMBzunTp+PLL7/E2rVrMXfuXM6MIZ/auBEIlQ96t94KXHDaisir3BZ4\njUYDm83mum+z2aB1s6pTRkYGHA4Hjh49Cq1W2622RJ6yWIDJk2Wn8I7Jk8UvrLNnZSehUOS2wKek\npKC+vh6NjY2w2+0oKSmByWRqd8y+fftcPfbt27cDAIYOHepRW6LuOnMGqKoK/N2bPKXVAkOHAjt3\nyk5CocjtGHxkZCSKioqQnZ0Np9OJvLw8GAwGFJ87u1VQUIDVq1dj+fLlUKvViIqKwltvveW2LVFv\nbN8OjBoFnDuXHxJuu00M00yY
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 184
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Now we obtain a training sample $x=2$ and update our estimate."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"pmu1 = norm.pdf(xs,loc=2.0)*pmu0\n",
|
||
|
|
"pmu1 /= C*sum(pmu1)\n",
|
||
|
|
"plot(xs,pmu1)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 198,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x14e27990>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD9CAYAAAChtfywAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X9YVGWiB/DvGGOKkJv5q2ZQ1kAYVH4JGrXqeM0wV8nS\n7WEr67bkcutS2Y+ntu3ZG25tym7dbo9ml7vb71qirBU0mJRs8ieggqiARSk5ommalWYKDOf+8ToI\nCMMAM/POOef7eR6eh4nDzLfJvr7znvc9x6AoigIiIlKtfrIDEBFR37DIiYhUjkVORKRyLHIiIpVj\nkRMRqRyLnIhI5botcpvNhujoaERGRiInJ6fTY+x2OxISEjB+/HhYrVZvZyQiIjcM7taRO51OREVF\noaSkBCaTCcnJycjLy4PFYmk95vvvv8d1112Hjz/+GGazGcePH8fQoUP9Ep6IiLoZkZeXlyMiIgLh\n4eEwGo1IT09HQUFBu2P++c9/Yv78+TCbzQDAEici8rMgdz9saGhAWFhY62Oz2YyysrJ2x9TV1aGp\nqQnTp0/HqVOn8OCDD2LhwoXtjjEYDF6MTESkH55svnc7IvekgJuamlBRUYGioiJ8/PHHePrpp1FX\nV9dpGH4peOqpp6RnCJQvvhd8L/heuP/ylNsRuclkgsPhaH3scDhap1BcwsLCMHToUAwcOBADBw7E\n1KlTUVVVhcjISI9DEBFR77kdkSclJaGurg719fVobGxEfn4+0tLS2h1z0003YfPmzXA6nThz5gzK\nysoQExPj09BERHSB2xF5UFAQVqxYgdTUVDidTmRkZMBisSA3NxcAkJmZiejoaMyaNQuxsbHo168f\nFi1axCJ3g8szL+B7cQHfiwv4XvSc2+WHXnsRg6FH8z1EROR5d3JnJxGRyrHIiYhUjkVORKRyLHIi\nIpVjkRMRqRyLnIhI5VjkREQqxyIn6iVFAd5+G5g9G5gxA3j+eeDsWdmpSI+4IYioFxobgdtvB776\nCvjjH4FBg4CVK4Fjx4C1a4Fhw2QnJC3wtDtZ5EQ9pCjAv/87cPIk8P77wKWXXvjnjz8ObNoE2O0X\n/jlRb3FnJ5GPvPMOsHMn8O677cvaYACWLQNGjgSefFJePtIfjsiJeuDkSSAqCrDZgMTEzo85dgwY\nPx5Ytw6Ij/dvPtIWTq0Q+cAf/wgcPQq88or74156CfjoI6CoyD+5SJtY5ERe9s03wLhxQGUlMGqU\n+2PPnQMiIoAPPwSSk/2Tj7SHc+REXrZyJXDrrd2XOCDmzv/wB+Dpp32fi4gjciIPnDsHjB4NbNgA\neHrflJ9/BsLCgB07gPBwn8YjjeKInMiLVq0SJzB7cvOrgQOBhQuBv//dd7mIAI7IiTwyfTqQlQXM\nn9+z36utFb978CDQv79vspF2cURO5CUHDwK7dwNz5vT8dy0WcdJz3Trv5yJyYZETdSMvD1iwoPc7\nNX/7W7F5iMhXOLVC1I0JE8SKlSlTevf7R4+KTUSHDwPBwd7NRtrGqRUiL/jiC+DECeC663r/HCNG\niLXkH33kvVxEbbHIidwoLATS0oB+ffw/5Te/AVav9k4moo5Y5ERuFBQAN93U9+f59a/F9Vmam/v+\nXEQdsciJuvDtt2K1yvTpfX8uk0lsCtq6te/PRdQRi5yoC2vXAjNnAgMGeOf55swB1qzxznMRtcUi\nJ+rCRx8Bc+d67/nmzhV/ORB5G5cfEnXC6RS3a9u7F7jqKu88Z0uLmGLZvBm4+mrvPCdpm9eWH9ps\nNkRHRyMyMhI5OTkX/dxut2Pw4MFISEhAQkICnnnmmd4lJgogFRWiwL1V4oBY+TJzJlBS4r3nJAKA\nIHc/dDqdyMrKQklJCUwmE5KTk5GWlgaLxdLuuGnTpqGwsNCnQYn8qaQEuP567z/vjBliyiYz0/vP\nTfrldkReXl6OiIgIhIeHw2g0Ij09HQUFBRcdx2kT0hpfFvmGDWKahchb3I7IGxoaEBYW1vrYbDaj\nrKys3TEGgwFbt25FXFwcTCYTnnvuOcR0cq3P7Ozs1u+tViusVmvfkhP5yJkzQHk5MG2a95/bbAaG\nDgWqqoCEBO8/P6mb3W6H3W7v8e+5LXKDwdDtEyQmJsLhcCA4OBjFxcWYN28evvjii4uOa1vkRIFs\nyxZx0+TQUN88/4wZwCefsMjpYh0HuUuWLPHo99xOrZhMJjgcjtbHDocDZrO53TGhoaEIPn8loBtv\nvBFNTU347rvvPM1NFHB8Na3iMmMGT3iSd7kt8qSkJNTV1aG+vh6NjY3Iz89HWlpau2OOHj3aOkde\nXl4ORVEwZMgQ3yUm8rHPPgN8OfNntYodntyuT97idmolKCgIK1asQGpqKpxOJzIyMmCxWJCbmwsA\nyMzMxKpVq/Dyyy8jKCgIwcHBeJcXXiYV++knsXZ80iTfvcaQIeIGzrt3A4mJvnsd0g9uCCJq49NP\ngSef9P01Uf7jP8Tdgx580LevQ+rG65ET9cKmTb2/gURP/OpXYocnkTewyIna2LxZlKyvTZkiXosf\nVMkbWORE5zU3A6WlwLXX+v61Ro0CgoKA/ft9/1qkfSxyovOqqkTBXnGF71/LYBAj/02bfP9apH0s\ncqLzNm3yz7SKC+fJyVtY5ETnbdnSt5ss9xSLnLyFRU50XlkZcM01/nu98eOBw4eBkyf995qkTSxy\nIgBHjgCnTwMREf57zUsuERuCtm/332uSNrHIiSCudjhpkjgJ6U+TJ4tPAkR9wSIngijyyZP9/7os\ncvIGFjkRRJnKKPJJk8Rrc2MQ9QWLnHSvpQXYsQNITvb/a5vNQP/+QH29/1+btINFTrr3+ediE9Cw\nYXJen9Mr1FcsctI914lOWVjk1FcsctI9WfPjLq55cqLeYpGT7pWVyR2RJyWJ67w0NsrLQOrGIidd\n+/lnoLZW7o2QQ0OBMWOAPXvkZSB1Y5GTrlVWijv1DBwoN8fEicDOnXIzkHqxyEnXdu4UUxuyscip\nL1jkpGu7dsmdVnGZOBGoqJCdgtSKRU66VlkZGEUeFwdUV/OEJ/UOi5x0q7ER2LcPmDBBdhJg0CDg\nl78EampkJyE1YpGTbtXWAuHhQHCw7CRCYiLnyal3WOSkW4EyreLCeXLqLRY56dauXUB8vOwUF3Dl\nCvUWi5x0K9BG5PHxYlNQc7PsJKQ2LHLSpZaWwBuRh4YCYWFi7p6oJ1jkpEv19cBllwFDh8pO0h5P\neFJvdFvkNpsN0dHRiIyMRE5OTpfHbd++HUFBQfjwww+9GpDIFwJtWsWFJzypN9wWudPpRFZWFmw2\nG2pqapCXl4faTj73OZ1OPP7445g1axYU3rOKVCDQplVceMKTesNtkZeXlyMiIgLh4eEwGo1IT09H\nQUHBRcctX74cCxYswDBZt1gh6qFAHZEnJIhL2jqdspOQmgS5+2FDQwPCwsJaH5vNZpR1uAJ+Q0MD\nCgoKsGHDBmzfvh0Gg6HT58rOzm793mq1wmq19j41UR8FapEPHgxceaXYcTpunOw05G92ux12u73H\nv+e2yLsq5bYWL16MZcuWwWAwQFGULqdW2hY5kUzHjgFnzgCjR8tO0jnXqJxFrj8dB7lLlizx6Pfc\nFrnJZILD4Wh97HA4YDab2x2zc+dOpKenAwCOHz+O4uJiGI1GpKWleZqdyK9c8+MejFOkiI8XGW+7\nTXYSUgu3RZ6UlIS6ujrU19fjqquuQn5+PvLy8tods3///tbv7777bsydO5clTgEtUKdVXOLjgf/5\nH9kpSE3cFnlQUBBWrFiB1NRUOJ1OZGRkwGKxIDc3FwCQmZnpl5BE3rRrF3DjjbJTdM01IleUwP3U\nQIHFoPhhvaBr/pwoEERHA++/HxiXr+2MogAjRogyv+oq2WlIJk+7kzs7SVd++gk4eFCUeaAyGC6M\nyok8wSInXdm9G4iJAYxG2UncY5FTT7DISVcC/USnS1wci5w8xyInXQnUrfkdcUROPcEiJ11Ry4g8\nKgo4dAg4dUp2ElIDFjnpRnOz
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 198
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"We can also compute the map estimate $\\hat{\\mu}$ for the parameter."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"mumap = xs[argmax(pmu1)]\n",
|
||
|
|
"print mumap"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "stream",
|
||
|
|
"stream": "stdout",
|
||
|
|
"text": [
|
||
|
|
"1.4954954955"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 199
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Let's now compute $p(x|D)$ and compare it with $p(x|\\hat{\\mu})$."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"result = zeros(xs.shape)\n",
|
||
|
|
"for i,mu in enumerate(xs):\n",
|
||
|
|
" weight = pmu1[i]\n",
|
||
|
|
" result += weight * norm.pdf(xs,loc=mu)\n",
|
||
|
|
"result /= C*sum(result)\n",
|
||
|
|
"plot(xs,result)\n",
|
||
|
|
"plot(xs,norm.pdf(xs,loc=muml))"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 196,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x149c7d10>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD9CAYAAAC2l2x5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlclNX+wPHPAOOuuOSCDDoKKLhAJMp1y8FdU1xLvGpm\nLlwrrduvui23m9ZtsbIybcFyTTNumqKpuBRTrmAuuYCKCooohgtuqGzP748nUQKGQWEemPm+e81L\nZp5zZr5M8OXM9znPOTpFURSEEELYHSetAxBCCFE2JMELIYSdkgQvhBB2ShK8EELYKUnwQghhpyTB\nCyGEnSo2wUdFReHj44O3tzczZswost2uXbtwcXFhxYoVJe4rhBCi9OkszYPPycmhZcuWbN68GXd3\nd9q3b8+yZcvw9fUt0K5Xr15Uq1aNcePGMWzYMKv7CiGEKBsWR/CxsbF4eXlhNBrR6/WEhoYSGRlZ\noN3s2bMZPnw49evXL3FfIYQQZcPF0sGUlBQ8PDzy7hsMBmJiYgq0iYyM5Oeff2bXrl3odDqr+95u\nK4QQomSsWYTA4gjemgT83HPP8d5776HT6VAUJe9FrU3et/s4+u2NN97QPIbycqvI78XJ9JOM+WEM\ntd6tRad5nei5uCeNPmzEQ+EPsfboWod6L+Tnouxu1rI4gnd3dyc5OTnvfnJyMgaDIV+b3bt3Exoa\nCsD58+dZv349er3eqr5C2JN1Cet4YtUTTG4/mZPPnaR2ldoA5Cq5rD26lqnrp7Lm6Bpm95uNi5PF\nXz0hSoXFn7LAwEASEhJISkqicePGREREsGzZsnxtTpw4kff1uHHjGDhwICEhIWRnZxfbVwh7EXEw\nguc2PMfKESvp3KRzvmNOOicGthxIN2M3RiwfwdCIoax4bAV6Z71G0QpHYbFE4+Liwpw5c+jTpw+t\nWrVixIgR+Pr6Eh4eTnh4uMUnLqqvKJzJZNI6hHKjor0Xm45vYmrUVDaM3lAgud+tVuVarA5dTXZu\nNmE/hln1UbuivRdlSd6LkrM4TbLMX/zPur0QFVXy5WTaf9We74Z/h8losqrP9czrdFnQhUkPTWJy\n+8llG6CwS9bmTknwQtyjXCWX4EXB9PPqx8tdXi5R36MXjtJ5fmeix0bTpkGbMopQ2Ctrc6csVSDE\nPZq3Zx63sm/xYqcXS9y3Rb0WvNP9HSasnkCuklsG0QkhCV6Ie5J6LZXXfn6NuQPn4uzkfE/PMf6h\n8bg4uTB399xSjk4IlZRohLgHk9ZMwrWKKx/0+uC+nufAuQP0/KYnR585imsV11KKTtg7KdEIUUaO\nnD/CysMrebXLq/f9XG0btqWfVz9m7phZCpEJkZ+M4IUooUe/f5R2bu1KfGK1KEnpSbSb2464p+Jo\nWKNhqTynsG8ygheiDBw4d4Ctp7YyNWhqqT2nsbaRUW1H8cH2+yv3CPFXkuCFKIEPtn/A1A5Tqaav\nVqrP+0KnF5i/dz7pN9NL9XmFY5MEL4SVTl0+xdqEtWVycVIT1yYMaDGAL3/7stSfWzguSfBCWOmT\nnZ/wZMCTeYuIlbYXO73IrJhZ3Mq+VSbPLxyPJHghrHA98zqLfl/ElA5Tyuw12jZsS5sGbVgRv6L4\nxkJYQRK8EFaIOBRBZ4/ONHFtUqavMzlwspRpRKmRBC+EFb747QsmB5b9wmADWwzk2MVjHPzjYJm/\nlrB/kuCFKMZvZ37jfMZ5env2LvPX0jvrmfDQBMJ3W16OWwhrSIIXohjhu8OZ9NCke15zpqQmPjSR\npfuXcj3zuk1eT9gvSfBCWHAj6wbL45Yz9sGxNntND1cPggxBrDq8ymavKeyTJHghLFhzdA3tG7en\ncc3GNn3dMX5j+Gb/NzZ9TWF/JMELYcGS/UsY7Tfa5q872GcwMSkxnL161uavLeyHJHghinA+4zy/\nnvyVIT5DbP7a1fTVGNRyEMsOykb14t4Vm+CjoqLw8fHB29ubGTNmFDgeGRmJv78/AQEBtGvXjp9/\n/jnvmNFoxM/Pj4CAADp06FC6kQtRxiIORtDfuz81K9fU5PWlTCPul8XlgnNycmjZsiWbN2/G3d2d\n9u3bs2zZMnx9ffPaXL9+nerVqwNw4MABhgwZwrFjxwBo1qwZu3fvpm7duoW/uCwXLMqxjvM68p+H\n/0M/736avH5Obg5NP2nKpjGb8K3vW3wH4TBKZbng2NhYvLy8MBqN6PV6QkNDiYyMzNfmdnIHuHbt\nGg888EC+45LARUWUeCmR4xeP08uzl2YxODs5M7zVcJbHLdcsBlGxuVg6mJKSgoeHR959g8FATExM\ngXarVq3ilVde4ezZs2zcuDHvcZ1OR8+ePXF2diYsLIyJEycW6Dtt2rS8r00mEyaT6R6+DSFK1w/x\nPzDYZzAuThZ/Rcrc8FbDeXrd07ze7XVN4xDaMpvNmM3mEvez+NOr0+msepLBgwczePBgtmzZwpgx\nYzhy5AgA27Ztw83NjbS0NHr16oWPjw9du3bN1/fuBC9EebEifgX/6fYfrcOgk0cn0q6ncfTCUVrU\na6F1OEIjfx38Tp8+3ap+Fks07u7uJCcn591PTk7GYDAU2b5r165kZ2dz4cIFANzc3ACoX78+Q4YM\nITY21qqghNDSmatnOHz+MN2bddc6FJx0Tgz1HSplGnFPLCb4wMBAEhISSEpKIjMzk4iICEJCQvK1\nOX78eF6dfc+ePQDUq1ePjIwMrl69CqgnYjdu3Ejbtm3L4nsQolStjF/JIy0eoZJzJa1DAeDRVo9K\nghf3xGKJxsXFhTlz5tCnTx9ycnIYP348vr6+hIerCyGFhYWxYsUKFi9ejF6vp0aNGnz33XcApKam\nMnToUACys7MZNWoUvXuX/WJNQtyvHw7/UKbrvpdUlyZdOHP1DMcvHsezrqfW4YgKxOI0yTJ/cZkm\nKcqZ8xnn8fzUk7P/d7bU9129H5PXTqZZ7Wa81PklrUMR5UCpTJMUwtGsPrKa3p69y1VyBwhpEcKa\no2u0DkNUMJLghbjLD/E/MNRnqNZhFBDcLJj95/ZzIeOC1qGICkQSvBB/ysjK4NeTv2p25aolVVyq\n0L1Zd9YlrNM6FFGBSIIX4k/RidE85PYQtavU1jqUQg3wHsCPCT9qHYaoQCTBC/GntQlrecT7Ea3D\nKNIjLR5h4/GNZOZkah2KqCAkwQuBumbS2oS1PNKi/Cb4RjUa0aJeC7ac3KJ1KKKCkAQvBHAo7RBO\nOid8HyjfqzYObDFQZtMIq2m7kpIQ5cTao2p5xtr1l4qSmwvbt8PWrXDsGGRkQL160KYN9OgBXl73\nF+eAFgMY9r9hfNzn4/uOVdg/GcELwf3X32/ehE8/haZN4amnIC0NOnSA/v3VpL5jB3TpAg8/DJs3\n33uc/g39yczJ5PD5w/f+JMJhyAheOLxLNy6xL3UfJqPpnvr/8guMG6eO0letgnbtCm+XlQUrVsDk\nyeDjA19/DQ0bluy1dDodfb36suH4BtkERBRLRvDC4W04voFuxm5U1VctUT9FgTffhL//HWbNgtWr\ni07uAHo9hIZCXBw8+KB6u2uHS6v18ezDhuMbSt5ROBxJ8MLh3Ut5JisLHn8c1q6F3bth4EDr++r1\n8NZbsHSpmvC//bZk8fZo1oOtp7ZyM/tmyToKhyMJXji0XCWXjcc30terr9V9cnJgzBi4eBGio6FR\no3t77e7d1RH8v/4FixZZ369O1Tq0bdBWpkuKYkmCFw7twLkDuFZ2xVjbaFX73FwYPx4uXFDr6dXu\nc02yNm1g0yZ4+WX1+ax1uw4vhCWS4IVD23h8Y4k21n77bTh8WD2ZWqVK6cTg4wPr1sE//qGWe6wh\ndXhhDUnwwqFtOrGJ3s2t24gmMhLCw2HlSqhevXTjCAhQn3voUDh3rvj2gY0DOXP1DClXUko3EGFX\nJMELh3Uj6wY7Tu+wanrksWMwYYJaRvlzq+FSN3QojB0LI0aodX5LnJ2c6dm8JxuPbyybYIRdkAQv\nHNbWU1vxa+iHaxVXi+2ys2H0
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 196
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Finally, let's compute posteriors using the MAP and Bayesian estimates."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"p1 = norm.pdf(xs,loc=0)\n",
|
||
|
|
"p2map = norm.pdf(xs,loc=mumap)\n",
|
||
|
|
"p2b = result\n",
|
||
|
|
"plot(xs,p2map/(p1+p2map))\n",
|
||
|
|
"plot(xs,p2b/(p1+p2b))"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 200,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x14bfe4d0>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD9CAYAAAChtfywAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlYVPXix/E3mwsuuJsCuYECmktumVZYV1EyzbQuditz\nyywz7Va23ovWzTRbTFOxq9liZppdrBR/WlEqCq64Gy4o4L7ghmzD+f1xFFwRFTgzzOf1POeZOcMZ\n5vPMY5++fM/mYhiGgYiIOCxXqwOIiMitUZGLiDg4FbmIiINTkYuIODgVuYiIg1ORi4g4uHyLvH//\n/tSsWZM77rjjmtsMGzYMf39/mjVrxvr16ws9oIiI5C/fIu/Xrx9RUVHX/PnChQvZuXMnCQkJTJs2\njSFDhhR6QBERyV++RX7PPfdQuXLla/58wYIF9O3bF4C2bduSmprKoUOHCjehiIjky/1W3pySkoKv\nr2/uuo+PD8nJydSsWfOS7VxcXG7lY0REnFZBTr6/pSK/2odcq7R1JQBTeHg44eHhVsewC/ou8pTE\n78IwDI6kHWFv6l72ndzHwTMHOZx2mCNnj3D47OHc5fi545zKOEV2TjYVS3mR/VsONUJup7RREfec\nCrhke2JklSEnswzZGaXJPleGrPQy2NLLkJVemuz0MmSeK03WuVJkpLvhihulS7lTppQbpT3cKe3h\nhoe7Gx5u7ni4u1HK3R0PNzdKe5x/7uFKKQ8XPNxd8PBwoVQp87m7O7i5uuDm5pL36OaCmytXec0F\nNzfzdfdr/MzV1QVXFxdcz8+DuLiAq6v56ObqAi4XvXb+5107+BTou76lIvf29iYpKSl3PTk5GW9v\n71v5lSLiYI6cPcLWI1vZemQr245uY8exHbnl7enhSZ1KdajleTuetttwTa+B7VRjso53pMyx6lQ9\nXAO3A1VwTfHi6KEyuFR0obRrOLelhVOpElSuDJUqnV+q5D338oJy5czF0zPv0dMTPDys/kaK3y0V\neffu3Zk0aRJhYWGsWrWKSpUqXTGtIiIlx9G0o6xOWU1cShxx++NYnbKarJwsgqoH4ecVRJXsIJqc\n6cLt++tycMft7E0oz869kGBAnTp5i68v3NYIatbMW6pXh1KlIDzcXKTg8i3yPn368Mcff3D06FF8\nfX0ZNWoUWVlZAAwePJjQ0FAWLlyIn58f5cqV44svviiW0I4sODjY6gh2Q99FHnv9Lk5nnOaPvX+w\ndPdSlu5eStKpJFrXbk1AhTbckTmIBicjSFjnTfwGF+JPQaNGEBgIAQEQ0gf8/MzirlTJnCooCHv9\nLuyZS3FcxtbFxUVz5CIO4uCZg/y47Ud+2PYDsSmxtKndlsDSf8N939/YvaIFK2PccHODli3zlubN\nzVG2q04xLFQF7U4VuYhwNvMsc7fO5YsNXxB/MJ77aj9I9SO92PdbF2KXe3L77XDvvebSoQNoV1jx\nUJGLyHVtPryZSXGT+H7L9wRVuJtq+waw7aeunDxWhtBQCA2Fjh2halWrkzqngnbnLR9+KCKOxTAM\n/tz7J+NixhGXtI6gs0MoP38jR7N9uP8xeHOmOV2iaRLHoSIXcSLL9i7j1SWvsefQEcqufwWPuB9o\n/VgZPv4KWrQo+A5JsS8qchEnsOXwFl78+TVW791EztJ36OD1OM8NcSN0Dri5WZ1ObpWKXKQEO5t5\nluH/G8U3m2fisvx1BrWYx4tfl6Z+fauTSWFSkYuUUNOX/cKLi58ne3cHXmi0idfm1tROyxJKRS5S\nwuw7eJZun/6TLemLedJrOhMjHqBCBatTSVFSkYuUEDk58NbktYzb+TgNyrRl+/AN+N/uZXUsKQYq\ncpESYPNm6DnqCxIbvsp7nSfyamiY1ZGkGKnIRRxYdjaM/k8WH2wcgVeLJWwY9CeNawZaHUuKmYpc\nxEHt3g1hT59kZ8uetH/Qkx8ej8OrjKZSnJHO3RJxQLNmQeuOB0jpdB99/hbE4qcjVeJOTNdaEXEg\nWVnw8svwv2UJ2PqE8Gzb/rx5z5u6nWIJpWutiJQwhw/DY4+BUXUHWY/fT3jHf/NMy2esjiV2QFMr\nIg5g+3Zo2xaCOiSwq/0D/OeBd1XikktFLmLnYmIgOBiee3M3P1d9gH/f92/6tehndSyxIypyETu2\nYAE8/DB8NO0gU87+jdc7vM6gloOsjiV2RkUuYqfmzoVnnoF5C87w4cEHebr50wxpPcTqWGKHdNSK\niB36/nt48UX4eWE2b2/tTu0Ktfn8oc91dIqTKWh3akQuYmfmzDFLPCrKYFrK8xgYTHlwikpcrkmH\nH4rYkUWLzBJfsgRWZESwfN9yVg1YhYebh9XRxI5pakXETqxaBd27Q2QkGD4xPPzdw6zovwL/qv5W\nRxOL6IQgEQeybZt5dMrMmVCnyX7afP4YX/T4QiUuBaIiF7HY4cPQtSuMGwedu2QTPPMxBrcczIMN\nH7Q6mjgI7ewUsVBGBjzyCDz5JDz1FIz+YzTlSpXjzXvftDqaOBDNkYtYxDBg4EA4cQLmzYNl+/6g\nzw99WDd4HbeVv83qeGIHNEcuYuc+/RRWrzZPwT+Rfownf3ySGT1mqMTlhmlELmKBmBjo2dM8UqVu\nXYNe3/eiXuV6fNj5Q6ujiR3RiFzETh07BmFh8N//Qr168O2m2SQcT2B2r9lWRxMHpRG5SDHKyTGP\nFQ8MhA8+gINnDtJsajN+efwXWtVuZXU8sTM6RV/EDn34oTkif+89MAyDIb8MYeCdA1Xicks0tSJS\nTNavN0fhq1eDhwfM3vQdfx37i+96fWd1NHFwKnKRYpCRYR4n/uGHUKcOHEs7xojFI1jQZwGl3Utb\nHU8cnObIRYrByJGwc6d5vLiLCwz6aRBl3cvyaddPrY4mdqzQ5sijoqIICAjA39+fsWPHXvHzo0eP\n0qVLF5o3b06TJk2YOXPmTQUWKamWL4evvoKpU80Sj0mKYWHCQt7p+I7V0aSEyHdEbrPZaNSoEUuX\nLsXb25vWrVsze/ZsAgMDc7cJDw8nIyODMWPGcPToURo1asShQ4dwd8+btdGIXJxVWho0bWpOqfTo\nAdk52bSc1pLX2r9Gnzv6WB1P7FyhjMjj4uLw8/Ojbt26eHh4EBYWRmRk5CXb1KpVi1OnTgFw6tQp\nqlatekmJizizd96BVq3MEgeYGDuR6p7VCWsSZm0wKVHybdyUlBR8fX1z1318fIiNjb1km0GDBnH/\n/fdTu3ZtTp8+zffff3/V3xUeHp77PDg4mODg4JtPLeIANm6E6dPNR4D9p/fzn2X/IWZAjO72I1cV\nHR1NdHT0Db8v3yIvyD+29957j+bNmxMdHc2uXbvo1KkT8fHxVKhQ4ZLtLi5ykZLOZjNvnPzuu3Db\n+UunvPHrGwxqOYiGVRtaG07s1uWD3FGjRhXofflOrXh7e5OUlJS7npSUhI+PzyXbxMTE8OijjwLQ\noEED6tWrx44dOwqaW6REmjrVPFZ84EBzfe3+tSzetZjXO7xubTApkfIt8latWpGQkEBiYiKZmZnM\nmTOH7t27X7JNQEAAS5cuBeDQoUPs2LGD+vXrF11iETt36BCEh0NEBLi6mmdwjlg8gtHBo6lYuqLV\n8aQEyndqxd3dnUmTJhESEoLNZmPAgAEEBgYSEREBwODBg3njjTfo168fzZo1Iycnh3HjxlGlSpVi\nCS9ij9580zz5JyjIXJ+/bT4nM07Sv0V/a4NJiaUTgkQK0dq10K0bbN8OXl6QkZ1B0OQgpnWbxgP1\nH7A6njgYXTRLpJgZBrz4ormD08vLfO3T2E9pXL2xSlyKlA74Fikks2fDuXPw9NPm+vFzxxm7Yiwr\n+q+wNJeUfJpaESkEZ89CQAB89x20b2++NnLpSE6cO8G0h6ZZG04clu4QJFKMPv7YLPALJb7/9H4+\nX/s5G4dstDaYOAWNyEVu0dGj5mg8NhYaNDBfe+6X5/D08GR85/HWhhOHphG5SDF57z3zHpwXSnzX\n8V3M2TKHHUN1YpwUDxW5yC3Y
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 200
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"As you can see, the MAP-based estimate underestimates the error rate for smaller values of $x$.\n",
|
||
|
|
"\n",
|
||
|
|
"In addition, it results in different decision boundaries."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"dbmap = xs[find(p2map/(p1+p2map)>0.5)[0]]\n",
|
||
|
|
"dbb = xs[find(p2b/(p1+p2b)>0.5)[0]]\n",
|
||
|
|
"print dbmap,dbb"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "stream",
|
||
|
|
"stream": "stdout",
|
||
|
|
"text": [
|
||
|
|
"0.754754754755 0.844844844845"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 203
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Bayesian vs Maximum Likelihood Example\n",
|
||
|
|
"==============================\n",
|
||
|
|
"\n",
|
||
|
|
"Let's apply this to another, simple example and compare maximum likelihood and bayesian approaches\n",
|
||
|
|
"~\n",
|
||
|
|
"Assume that the samples $x$ come from a uniform density over the interval $[0,\\theta]$\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(x|\\theta) = U(x;0,\\theta) = 1/\\theta \\cdot \\lfloor x\\in[0,\\theta]\\rfloor$$\n",
|
||
|
|
"\n",
|
||
|
|
"We also assume a prior\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(\\theta) = U(\\theta;0,10)$$\n",
|
||
|
|
"\n",
|
||
|
|
"That is, $\\theta$ is distributed uniformly over the interval $[0,10]$\n",
|
||
|
|
"\n",
|
||
|
|
"Let's assume we see a sequence of training examples $D = \\\\{4,7,2\\\\}$"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"xs = linspace(-1,11,1000)\n",
|
||
|
|
"mus = xs\n",
|
||
|
|
"C = (amax(xs)-amin(xs))/len(xs)\n",
|
||
|
|
"def pxt(x,mu): return (xs>=0)*(x<=mu)*1.0/maximum(mu,1e-6)\n",
|
||
|
|
"plot(xs,pxt(xs,5.5))\n",
|
||
|
|
"pmu = (xs>=0)*(xs<=10)*0.1\n",
|
||
|
|
"plot(xs,pt)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 225,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x178d9490>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD9CAYAAABdoNd6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFcFJREFUeJzt3X9sU+e9x/GPabyLLmhQWAhgp4o2p9hp+TUljeiEFK2E\niGizBu0f6ZjgshRZMNRtKrv/3D8WJo0RoWqLmukqm0pXtIrm/rOGbqnVBtW6dDSJJuiYGrgNUzOc\n0DBUQYFmLMQ99w/mkITU8clJOOfJeb8kBI7PMd+g5/nkeb4+xwQsy7IEAJjT5rldAABg9hH2AOAD\nhD0A+ABhDwA+QNgDgA8Q9gDgA1OGfTKZVDQaVWlpqRobG+95/pVXXtHatWu1Zs0afe1rX9PZs2fz\nPhcAcH8Ecl1nn8lktGrVKnV0dCgUCqmiokLHjh1TLBYbPebdd99VWVmZFi1apGQyqYaGBnV2duZ1\nLgDg/si5su/u7lYkElFJSYmCwaDq6urU1tY27pgNGzZo0aJFkqTKykr19/fnfS4A4P4oyPXkwMCA\niouLRx+Hw2F1dXV97vEvvviiamtr8z43EAhMq2gA8Du7H36Qc2VvJ4zffvttHTlyZLQ3n++5lmUZ\n++vHP/6x6zVQv/t1+LF+k2ufC/VPR86VfSgUUjqdHn2cTqcVDofvOe7s2bPavXu3ksmkHnzwQVvn\nAgBmX86VfXl5uXp7e9XX16fh4WG1trYqHo+PO+bixYvatm2bfvvb3yoSidg6FwBwf+Rc2RcUFKi5\nuVk1NTXKZDKqr69XLBZTS0uLJCmRSOgnP/mJrl69qj179kiSgsGguru7P/fcuaSqqsrtEhyhfneZ\nXL/JtUvm1z8dOS+9nPW/PBCYdv8JAPxqOtnJHbQA4AOEPQD4AGEPAD5A2AOADxD2AOADhD0A+ABh\nDwA+QNgDgA8Q9gDgA4Q9APgAYQ8APkDYA4APEPYA4AOEPQD4QM7Ps0duH30kNTRImYzblcALFi+W\nDh+W+K+V4UV8nr0DJ05I3/uetH+/25XACxIJ6dYtKRh0uxLMddPJTlb2DoVC0jPPuF0FvGDvXrcr\nAD4fPXsHDN6UYJYwJuBVhD0A+ABh7xBvxiGLsQAvI+wdYMuOiRgT8CrC3iFWc8hiLMDLCHsA8AHC\n3gG27JiIMQGvIuwdYuuOLMYCvIywd4BVHCZiTMCrCHsA8AHC3iG27shiLMDLCHsH2LJjIsYEvIqw\nBwAfIOwdYuuOLMYCvIywd4AtOyZiTMCrCHuHWM0hi7EALyPsAcAHCHsH2LJjIsYEvIqwd4itO7IY\nC/Aywh6YQazs4VWEvQNMbACmIOwdYuuOLMYCvIywd4CVPSZiTMCrCHsA8AHC3iG27shiLMDLCHsH\n2LJjIsYEvIqwBwAfIOwdYuuOrECAlT28i7B3gIkNwBSEvUOs7JHFWICXTRn2yWRS0WhUpaWlamxs\nvOf58+fPa8OGDZo/f76ef/75cc+VlJRozZo1Wr9+vR577LGZqxrwKHZ78KqCXE9mMhnt27dPHR0d\nCoVCqqioUDweVywWGz1m6dKleuGFF/Taa6/dc34gEFAqldKSJUtmvnIPYGIDMEXOsO/u7lYkElFJ\nSYkkqa6uTm1tbePCvrCwUIWFhfrDH/4w6WtYUyRiQ0PD6J+rqqpUVVWVX+UewdYdWYwFzJZUKqVU\nKuXoNXKG/cDAgIqLi0cfh8NhdXV15f3igUBAmzZt0gMPPKBEIqHdu3ffc8zYsDcNK3tMxJjAbJi4\nED5w4IDt18gZ9gGHS5U//vGPWrFiha5cuaLq6mpFo1Ft3LjR0WsCAOzL+QZtKBRSOp0efZxOpxUO\nh/N+8RUrVki60+rZunWruru7p1mmd7F1RxZjAV6WM+zLy8vV29urvr4+DQ8Pq7W1VfF4fNJjJ/bm\nh4aGdOPGDUnSp59+qjfffFOrV6+eobK9gS07JmJMwKtytnEKCgrU3NysmpoaZTIZ1dfXKxaLqaWl\nRZKUSCQ0ODioiooKXb9+XfPmzVNTU5N6enr097//Xdu2bZMkjYyMaPv27dq8efPsf0cAgHsErKku\nl5nNvzwQmPJqHS/73e+ko0fv/A586UvS+fN3fgdm03SykztoHTD45xQAnyHsHeJNOWQxFuBlhD0w\ng9jtwasIeweY2ABMQdg7xNYdWYwFeBlh7wAre0zEmIBXEfYA4AOEvUNs3ZHFWICXEfYOsGXHRIwJ\neBVhDwA+QNg7xNYdWYEAK3t4F2HvABMbgCkIe4dY2SOLsQAvI+yBGcRuD15F2DvAxAZgCsLeIbbu\nyGIswMsIe2AGsduDVxH2DjCxAZiCsHeIrTuyuM4eXkbYO8DEBmAKwh4AfICwd4g2DrJo48DLCHsH\nmNgATEHYO8TKHlmMBXgZYQ/MIHZ78CrC3gEmNgBTEPYOsXVHFmMBXkbYAzOI3R68irB3gIkNwBSE\nvUNs3ZHFdfbwMsLeASY2AFMQ9gDgA4S9Q7RxkEUbB15G2DvAxAZgCsLeIVb2yGIswMsIe2AGsduD\nVxH2DjCxAZiCsHeIrTuyGAvwMsIemEHs9uBVhL0DTGwApiDsHWLrjiyus4eXEfYOMLEBmIKwBwAf\nIOwdoo2DLNo48DLC3gEmNgBTEPYA4ANThn0ymVQ0GlVpaakaGxvvef78+fPasGGD5s+fr+eff97W\nuXMBbRxk0caBl+UM+0wmo3379imZTKqnp0fHjh3TuXPnxh2zdOlSvfDCC9q/f7/tc03HxAZgipxh\n393drUgkopKSEgWDQdXV1amtrW3cMYWFhSovL1cwGLR97lzAyh5ZjAV4WUGuJwcGBlRcXDz6OBwO\nq6urK68XzvfchoaG0T9XVVWpqqoqr9f3gtvWP5Wef0Kv/1/G7VLgATdD0ol+6fxnbleCfDxe/LiW\n/vtSt8vISyqVUiqVcvQaOcM+4GCpku+5Y8PeNBf+eUr/u3SH/u30BrdLgQdcj0j/c0FaMOh2JZhK\nz5Ue7S3fq+cef87tUvIycSF84MAB26+RM+xDoZDS6fTo43Q6rXA4nNcLOznXFJ9ZGS25vVavP/26\n26XAA1Y1SP+9X1q1yu1KMJUfvfUjZSx/7chz9uzLy8vV29urvr4+DQ8Pq7W1VfF4fNJjrQnvVto5\nFwAwu3Ku7AsKCtTc3KyamhplMhnV19crFouppaVFkpRIJDQ4OKiKigpdv35d8+bNU1NTk3p6erRw\n4cJJz51LLMtSQLwrB5gmoMA9C9S5LmfYS9KWLVu0ZcuWcV9LJBKjf16+fPm4ds1U584lliyJsMe/\ncJ29OQKBwL/mr39wBy0A+ABh74Al2jiAifzYxiHsHaBnj7Fo45iDNg4AYE4i7B3gDVqMxcreHLRx\nYBNtHMBEtHFgi88WBgAMRtg7QBsHY9HGMQdtHNhEGwcwEW0c2OKzhQEAgxH2DtDGwVi0ccxBGwe2\ncFMVYCbaOACAOYmwd4Q2Du6ijWOOgFjZw4bPaOMARqJnD/vIevwLK3t4GWHvCCt7wES8QQtbPrPo\n2QMmoo0DYNpo48DLCHtHaOMAJqKNA1vuDBXCHjCNHxdphL0jlg+HDD4PbRyz0LNH3niDFjATbRxM\nA2EPmIY2DmyijYO7aOOYhTYO8mbRxgGMRBsHtvlxO4jJsbI3hx/nLWHvgN9WBsBcQhsHeaONA5iJ\nNg5sCwQIe9xBG8cctHFgi99WBsBcQhsHebP4bBzASLRxMA2EPe6gjWMOPy7SCHsH/LYNBOYSv81f\nwt4B2jiYyGf5YSzaOLCNsEcWF2aZw4/zlrB3wG/bQGAu8dv8Jewd4aYq3MUbtOagjQMAmJMIewc+\ns3iDFjBRQAHaOLDD4uMSMIo2jjlo4wAA5iTC3gGLN2gBI9HGgS3cVIWxaOOYgzYOAGBOIuwd4D8v\nwUSs7M1AGwe20MbBWFyYZQ7aOJNIJpOKRqMqLS1VY2PjpMc8++yzKi0t1dq1a3XmzJnRr5eUlGjN\nmjVav369HnvssZmr2iv8NVYA
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 225
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Now assume we draw the sample $x_1=4$. How should we update our estimate?\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(\\mu|x) = \\frac{p(x|\\mu) p(\\mu)}{p(x)}$$"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"pmu1 = pxt(4,mus)*pmu\n",
|
||
|
|
"pmu1 /= C*sum(pmu1)\n",
|
||
|
|
"plot(xs,pmu1)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 228,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x18290f90>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD9CAYAAABdoNd6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHqRJREFUeJzt3X9YVHW+B/D3EaZs8fpzlXAGY3PQGVIQG+Saa80qyLNU\ns4JtkpY+LhGXrrm1eZ+7PrubYGWybru5UrvUqunqKk91b2iLc9FytvwBc7taeoMSu87jgGLe0vxB\nLjGd+8dJLqAOM8PAd76c9+t5emSYc4YPNOfNl8/5nu9RVFVVQURE/doA0QUQEVHvY9gTEekAw56I\nSAcY9kREOsCwJyLSAYY9EZEOdBv2TqcTFosFiYmJKC0tver5yspKpKSkIDU1FbfffjveeeedgPcl\nIqK+ofibZ+/z+TB+/Hjs3r0bRqMRaWlp2Lp1K6xWa/s2ly5dQkxMDADgyJEjyMnJwbFjxwLal4iI\n+obfkb3b7YbZbEZCQgIMBgPy8vJQWVnZaZsrQQ8AFy9exHe/+92A9yUior4R7e/JpqYmxMfHtz82\nmUyora29ars333wTy5Ytw6lTp1BdXR3wvoqi9Kh4IiK9CnbxA78j+0DDePbs2aivr8eOHTvw0EMP\nBVWEqqrS/rd8+XLhNbB+8XXosX6Za+8P9YfCb9gbjUZ4vd72x16vFyaT6brbT58+HW1tbfjiiy9g\nMpmC2peIiHqP37C32WxoaGiAx+NBa2srKioq4HA4Om3z6aeftv+mOXjwIABgxIgRAe1LRER9w2/P\nPjo6GmVlZcjKyoLP50N+fj6sVivKy8sBAIWFhXjjjTewadMmGAwGDBo0CNu2bfO7b39it9tFl9Aj\nrF8smeuXuXZA/vpD4XfqZa9/cUUJuf9ERKRXoWQnr6AlItIBhj0RkQ4w7ImIdIBhT0SkAwx7IiId\nYNgTEekAw56ISAcY9kREOsCwJyLSAYY9EZEOMOyJiHSAYU9EpAMMeyIiHWDYExHpAMOepPVv/wbs\n3y+6CiI5+L15CVEkczqBDz8EamoA3rueyD+O7ElqbjdQXS26CqLIx7AnaakqYLcDTz+tfUxE18ew\nJ6nNnQt89hngcomuhCiyMexJalFRwC9+oY3uiej6GPYkLVXVTszOmwd4PMC+faIrIopcDHuSnsEA\nLFsGrFghuhKiyMWwp35h4UKgoQF4913RlRBFJoY9SetKGwcAbrgBKCnR+vecmUN0NYY99Rvz5gFf\nfAHs3Cm6EqLIw7AnaXUdwUdFAc88o43uv/lGTE1EkYphT1LrukzC7NnaCdvXXxdTD1GkYthTv6Io\nwMqVwK9+BbS1ia6GKHJ0G/ZOpxMWiwWJiYkoLS296vktW7YgJSUFycnJmDZtGg4fPtz+XEJCApKT\nk5GamoopU6aEt3LSvY4naDuaORMwGoF16/q+JqJI5XfVS5/Ph8WLF2P37t0wGo1IS0uDw+GA1Wpt\n3+bWW2/Fu+++iyFDhsDpdOKRRx5BTU0NAEBRFLhcLgwfPrx3vwuiDhQF+M1vgLvvBh54ABg8WHRF\nROL5Hdm73W6YzWYkJCTAYDAgLy8PlZWVnbaZOnUqhgwZAgBIT09HY2Njp+dVzoMjASZPBrKygFWr\nRFdCFBn8juybmpoQHx/f/thkMqG2tva6269btw7Z2dntjxVFQUZGBqKiolBYWIiCgoKr9ikuLm7/\n2G63w263B1E+6dn12jhXPPsskJwMFBYCt9zSd3URhZvL5YKrh6v9+Q17JYg7QuzZswfr16/Hvg4L\nlOzbtw9xcXE4c+YMMjMzYbFYMH369E77dQx7onAyGoHHHtOWUvjLX0RXQxS6rgPhkpKSoF/DbxvH\naDTC6/W2P/Z6vTCZTFdtd/jwYRQUFGD79u0YNmxY++fj4uIAACNHjkROTg7cbnfQBRJdTyAdwn/5\nF20JhW9PIxHplt+wt9lsaGhogMfjQWtrKyoqKuBwODptc+LECeTm5mLz5s0wm83tn29pacGFCxcA\nAJcuXUJ1dTUmTpzYC98C6Vl3f3zGxGgXWv3sZ1xGgfTNbxsnOjoaZWVlyMrKgs/nQ35+PqxWK8rL\nywEAhYWFWLFiBc6ePYuioiIAgMFggNvtRnNzM3JzcwEAbW1tmD9/PmbNmtXL3w7R1RYsAMrKgM2b\ngYceEl0NkRiKKnC6jKIonK1DIVuwQJtTv3Bh99u63cCPfgTU1wNDh/Z+bUS9KZTs5BW0JLVA5xBM\nmQLcey/w1FO9Ww9RpGLYk2489xxQUQF88IHoSoj6HsOepNXdPPuuRozQTtY++ihXxST9YdiTruTn\nAz4fsHGj6EqI+hbDnnRlwADgD38Afv5z4LPPRFdD1HcY9iStYNs4V0yerM3kefzx8NdEFKkY9qRL\nJSXadMy33hJdCVHfYNiTtHpyicZ3vgO8/LJ2svb8+fDVRBSpGPYktVDaOFfMmKEtg/zzn4evHqJI\nxbAnXVu9Gti+XVssjag/Y9iTtMKx0sbQocBLLwGLFgHfrttH1C8x7ElqPWnjXOFwAHfdBTz5ZM9f\niyhSMeyJALzwArBrF2fnUP/FsCdphTrP/loGD9auqn3kEeDMmfC8JlEkYdgTfevOO4H584F/+ife\n6IT6H4Y9UQdPPw0cPQps2iS6EqLwYtiTtMLZxrli4EBgyxZg6VLgk0/C+9pEIjHsibpITtZG+HPn\nApcvi66GKDwY9iSt3uyrFxYC48ZxOib1Hwx7klq42zgdX/eVV4CdO4E33uidr0HUlxj2RNcxZIh2\nG8OiIuD4cdHVEPUMw56k1RfTI9PSgGXLgB//GPjqq97/ekS9hWFPUuutNk5Hjz8OmM3acsicf0+y\nYtgTdUNRgHXrgPffB/74R9HVEIUmWnQBRKHqjXn21xMTA/z7vwPTpgEpKcAdd/TN1yUKF47siQJk\nNgMbNgD33w+cOiW6GqLgMOyJgpCdrS2Wdt99wN//LroaosAx7ElafdnG6eiXvwRGjwYefpgnbEke\n3Ya90+mExWJBYmIiSktLr3p+y5YtSElJQXJyMqZNm4bDhw8HvC+RjAYM0JZD/uQT4JlnRFdDFCDV\nj7a2NnXs2LHq8ePH1dbWVjUlJUWtq6vrtM3+/fvVc+fOqaqqqjt37lTT09MD3rebL0/kV26uqr72\nmrivf/Kkqo4Zo6rbtomrgfQplOz0O7J3u90wm81ISEiAwWBAXl4eKisrO20zdepUDBkyBACQnp6O\nxsbGgPcl6ikRbZwr4uKAHTuAxx4DamrE1UEUCL9TL5uamhAfH9/+2GQyoba29rrbr1u3DtnZ2UHt\nW1xc3P6x3W6H3W4PtHYi4ZKTtRk6ubnA3r3ArbeKroj6I5fLBZfL1aPX8Bv2ShDDpj179mD9+vXY\nt29fUPt2DHuiYETKydG77wZ+9SsgKwvYtw8YNUp0RdTfdB0Il5SUBP0afsPeaDTC6/W2P/Z6vTCZ\nTFdtd/jwYRQUFMDpdGLYsGFB7UvUEyLbOB0VFQHNzdrUzD17gH/4B9EVEXXmt2dvs9nQ0NAAj8eD\n1tZWVFRUwOFwdNrmxIkTyM3NxebNm2E2m4Pal6g/KS4Gbr8dyMnhHHyKPH7DPjo6GmVlZcjKykJS\nUhLmzp0Lq9WK8vJylJeXAwBWrFiBs2fPoqioCKmpqZgyZYrffYnCRdQ8++tRFOCll4DBg4GFC4Fv\nvhFdEdH/U76dxiPmiysKBH55klxODrBggfZvJLl8WevfT5wIrF0bWb+QqH8IJTt5BS1JK1LHCQMH\nApWVwIEDwL/+a+TWSfrCsCepReqoeehQoLoa+I//AJYvF10NEZc4Juo1I0YAu3YBP/gBcOONwC9+\nIboi0jOGPUlLhvbIqFHA7t3AXXdpgb90qeiKSK8Y9iS1SG3jdBQXB7zzjhb4AwYAP/uZ6IpIjxj2\nRH3AZAJcLmDmTO3G5WzpUF9j2JO0ZGjjdBQfD/ztb0BGBtDSoi2PLMNfJtQ/cDYOSU22sIyL00b4\nVVVaO0e2X1gkL4Y9UR8bOVLr
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 228
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"The maximum likelihood estimate is clearly at $\\mu=4$. If we now plug this\n",
|
||
|
|
"into $p(x|\\mu)$ we get..."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"ylim(0,1)\n",
|
||
|
|
"plot(xs,pxt(xs,4))"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 229,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x1829cd10>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD9CAYAAABDaefJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEzBJREFUeJzt3H9sXXX9x/HXXe41kqHb+LHC7r1LhXvpvWVbi+msSDDX\njdKlgYtuIymKLKXMpmQBjCYmJmpr4ljlD12sfxTD+KFQBgnaabpr7OQiMtuCm5uBBQrScFu0Scm6\nQSp2uzvfP9D7pWt7dk5/fXY/ez6SG3p3z+55p2FPzt49l4DjOI4AAFZYYnoAAMD8IeoAYBGiDgAW\nIeoAYBGiDgAWIeoAYBHXqN99990qKSnR2rVrZzzmvvvuUzweV0VFhQ4fPjzvAwIAvHONekNDgzKZ\nzIyvd3d3680339TAwIAefvhhNTc3z/uAAADvXKN+4403asWKFTO+vm/fPm3btk2SVF1drbGxMY2M\njMzvhAAAz4Jz+c3Dw8OKRqOF55FIRENDQyopKZl0XCAQmMtpAOCC5fdD/3P+QenZJ5wp4I7jFO3j\nBz/4gfEZLtT5i3l25jf/KPb5Z2NOUQ+Hw8rlcoXnQ0NDCofDc3lLAMAczCnq6XRaTzzxhCSpt7dX\ny5cvn7J6AQAsHted+h133KEXXnhBo6Ojikajam1t1alTpyRJTU1NqqurU3d3t2KxmJYuXapHH310\nUYZebKlUyvQIc1LM8xfz7BLzm1bs889GwJnt4sbPSQKBWe+HAOBCNZt28olSALAIUQcAixB1ALAI\nUQcAixB1ALAIUQcAixB1ALAIUQcAixB1ALAIUQcAixB1ALAIUQcAixB1ALAIUQcAixB1ALAIUQcA\nixB1ALAIUQcAixB1ALAIUQcAixB1ALAIUQcAixB1ALAIUQcAixB1ALAIUQcAixB1ALAIUQcAixB1\nALAIUQcAixB1ALAIUQcAixB1ALAIUQcAixB1ALDIOaOeyWSUSCQUj8fV1tY25fXR0VFt2rRJlZWV\nWrNmjR577LGFmBMA4EHAcRxnphfz+bzKysrU09OjcDis9evXq7OzU8lksnBMS0uL/vOf/+jBBx/U\n6OioysrKNDIyomAw+P8nCQTkchoAwDRm007XK/X+/n7FYjGVlpYqFAqpvr5eXV1dk4658sordfLk\nSUnSyZMndemll04KOgBg8bjWd3h4WNFotPA8Eomor69v0jHbt2/Xhg0btGrVKr3//vt65plnpn2v\nlpaWwtepVEqpVGr2UwOAhbLZrLLZ7JzewzXqgUDgnG+wc+dOVVZWKpvN6q233lJNTY2OHDmiT33q\nU5OO+3jUAQBTnX3B29ra6vs9XNcv4XBYuVyu8DyXyykSiUw65uDBg7r99tslSVdffbU+85nP6PXX\nX/c9CABg7lyjXlVVpYGBAQ0ODmpiYkJ79+5VOp2edEwikVBPT48kaWRkRK+//rquuuqqhZsYADAj\n1/VLMBhUe3u7amtrlc/n1djYqGQyqY6ODklSU1OTvvvd76qhoUEVFRU6c+aMfvzjH+uSSy5ZlOEB\nAJO53tI4byfhlkYA8G3eb2kEABQXog4AFiHqAGARog4AFiHqAGARog4AFiHqAGARog4AFiHqAGAR\nog4AFiHqAGARog4AFiHqAGARog4AFiHqAGARog4AFiHqAGARog4AFiHqAGARog4AFiHqAGARog4A\nFiHqAGARog4AFiHqAGARog4AFiHqAGARog4AFiHqAGARog4AFiHqAGARog4AFiHqAGARog4AFiHq\nAGARog4AFjln1DOZjBKJhOLxuNra2qY9JpvN6rrrrtOaNWuUSqXme0YAgEcBx3GcmV7M5/MqKytT\nT0+PwuGw1q9fr87OTiWTycIxY2NjuuGGG/T73/9ekUhEo6OjuuyyyyafJBCQy2kAANOYTTtdr9T7\n+/sVi8VUWlqqUCik+vp6dXV1TTrmqaee0pYtWxSJRCRpStABAIsn6Pbi8PCwotFo4XkkElFfX9+k\nYwYGBnTq1Cl96Utf0vvvv6/7779fX//616e8V0tLS+HrVCrFmgYAzpLNZpXNZuf0Hq5RDwQC53yD\nU6dO6dChQzpw4IDGx8d1/fXX6/Of/7zi8fik4z4edQDAVGdf8La2tvp+D9eoh8Nh5XK5wvNcLldY\ns/xPNBrVZZddposuukgXXXSRvvjFL+rIkSNTog4AWHiuO/WqqioNDAxocHBQExMT2rt3r9Lp9KRj\nbrvtNv35z39WPp/X+Pi4+vr6VF5evqBDAwCm53qlHgwG1d7ertraWuXzeTU2NiqZTKqjo0OS1NTU\npEQioU2bNmndunVasmSJtm/fTtQBwBDXWxrn7STc0ggAvs37LY0AgOJC1AHAIkQdACxC1AHAIkQd\nACxC1AHAIkQdACxC1AHAIkQdACxC1AHAIkQdACxC1AHAIkQdACxC1AHAIkQdACxC1AHAIkQdACxC\n1AHAIkQdACxC1AHAIkQdACxC1AHAIkQdACxC1AHAIkQdACxC1AHAIkQdACxC1AHAIkQdACxC1AHA\nIkQdACxC1AHAIkQdACxC1AHAIkQdACxC1AHAIueMeiaTUSKRUDweV1tb24zHvfzyywoGg3ruuefm\ndUAAgHeuUc/n89qxY4cymYxee+01dXZ26tixY9Me953vfEebNm2S4zgLNiwAwJ1r1Pv7+xWLxVRa\nWqpQKKT6+np1dXVNOe5nP/uZtm7dqssvv3zBBgUAnFvQ7cXh4WFFo9HC80gkor6+vinHdHV16Y9/\n/KNefvllBQKBad+rpaWl8HUqlVIqlZr91ABgoWw2q2w2O6f3cI36TIH+uAceeEC7du1SIBCQ4zgz\nrl8+HnUAwFRnX/C2trb6fg/XqIfDYeVyucLzXC6nSCQy6Zi//vWvqq+vlySNjo5q//79CoVCSqfT\nvocBAMxNwHH5yebp06dVVlamAwcOaNWqVfrc5z6nzs5OJZPJaY9vaGjQrbfeqs2bN08+yX+v4gEA\n3s2mna5X6sFgUO3t7aqtrVU+n1djY6OSyaQ6OjokSU1NTbOfFgAw71yv1OftJFypA4Bvs2knnygF\nAIsQdQCwCFEHAIsQdQCwCFEHAIsQdQCwCFEHAIsQdQCwCFEHAIsQdQCwCFEHAIsQdQCwCFEHAIsQ\ndQCwCFEHAIsQdQCwCFEHAIsQdQCwCFEHAIsQdQCwCFEHAIsQdQCwCFEHAIsQdQCwCFEHAIsQdQCw\nCFEHAIsETQ9QDI4ckX79a9NTXLhiMenOO01PARQHrtQ9ePZZ6YUXTE9xYTp+XPr+901PARQPrtQ9\ncBxpwwbpe98zPcmF5x//kPbtMz0FUDy4UvfAcaQlfKeMWLLko+8/AG9IlQeOIwUCpqe4MAUCRB3w\ng6h7cOYMUTclEPjo+w/AG6LuAesXc1i/AP6QKg9Yv5jD+gXw55xRz2QySiQSisfjamtrm/L6k08+\nqYqKCq1bt0433HCDjh49uiCDmsT6xRzWL4A/rrc05vN57dixQz09PQqHw1q/fr3S6bSSyWThmKuu\nukp/+tOftGzZMmUyGX3jG99Qb2/vgg++mFi/mMP6BfDHNVX9/f2KxWIqLS1VKBRSfX29urq6Jh1z\n/fXXa9myZZKk6upqDQ0NLdy0hrB+MYf1C+CP65X68PCwotFo4XkkElFfX9+Mxz/yyCOqq6ub9rWW\nlpbC16lUSqlUyt+kBrF+MYf1Cy4k2WxW2Wx2Tu/hGvWAj5I9//zz2rNnj1566aVpX/941IsN6xdz\nWL/gQnL2BW9ra6vv93CNejgcVi6XKzzP5XKKRCJTjjt69Ki2b9+uTCajFStW+B7ifMf6xRzWL4A/\nrtefVVVVGhgY0ODgoCYmJrR3716l0+lJx7zzzjvavHmzfvWrXykWiy3osKawfjGH9Qvgj+uVejAY\nVHt7u2pra5XP59XY2KhkMqmOjg5JUlNTk374wx/q+PHjam5uliSFQiH19/cv/OSLiPWLOaxfAH8C\njrPwf2QCgYAW4TQLprlZWrtW
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 229
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"This is odd because it predicts that only values between 0 and 4 can occur.\n",
|
||
|
|
"But the training sample $x_1=4$ only excludes that $\\mu\\lt4$; it doesn't \n",
|
||
|
|
"exclude any values greater than $4$.\n",
|
||
|
|
"\n",
|
||
|
|
"What's the Bayesian estimate?\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(x|D) = \\int p(x|\\theta) p(\\theta|D) d\\theta \\propto \\int_0^{10} 1/\\theta \\cdot \\lfloor x\\in[0,\\theta]\\rfloor \\cdot 1/\\theta \\cdot \\lfloor \\theta\\in[4,10]\\rfloor d\\theta\n",
|
||
|
|
"= \\int_{x_1}^{10} 1/\\theta^2 \\cdot \\lfloor x\\in[0,\\theta]\\rfloor d\\theta$$\n",
|
||
|
|
"\n",
|
||
|
|
"You can either think about it, or we can simply perform this integral numerically."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"result = zeros(xs.shape)\n",
|
||
|
|
"for i,mu in enumerate(mus):\n",
|
||
|
|
" weight = pmu1[i]\n",
|
||
|
|
" result += weight * pxt(xs,mu)\n",
|
||
|
|
"result /= C*sum(result)\n",
|
||
|
|
"plot(xs,result)\n",
|
||
|
|
"plot(xs,pxt(xs,4))"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 230,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x183d6c10>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD9CAYAAABdoNd6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X9YVXWeB/D3Ee5oafkrReTiQ3XJC8oPW5Axx4lJkGSN\ngmyGsnQNifBh1SanZpqpAbdUntbKjWYGS83KJWp2RqzFu0YrY5J6t7RoxQrb7nohMadH0yRDrmf/\nOBMripdz+cL9nnPP+/U8PA8Xzrl8OCfeff2ez/keRVVVFUREFNIGyS6AiIgGHsOeiMgCGPZERBbA\nsCcisgCGPRGRBTDsiYgsoNewd7lccDqdiI2NRXl5+UXf37x5M5KSkpCYmIjp06ejsbGx63sxMTFI\nTEzElClTMHXq1P6tnIiIdFP89dn7fD5MnDgRdXV1iIqKQmpqKqqqqhAXF9e1ze7duxEfH4/hw4fD\n5XKhtLQUe/bsAQBcffXVeP/99zFq1KiB/02IiOiS/I7s3W43HA4HYmJiYLPZkJ+fj5qamm7bTJs2\nDcOHDwcApKWloaWlpdv3ec8WEZF84f6+2draiujo6K7Xdrsde/fuveT269evR3Z2dtdrRVGQkZGB\nsLAwFBUVobCwsNv2iqL0tW4iIksLdCDtN+wDCeMdO3Zgw4YNaGho6PpaQ0MDIiMjcezYMWRmZsLp\ndGLGjBlCBRvJgqULcCjpEF6/43XZpfTJmlVr8OCvHpRdRp9NnDsRTa81IXp4dO8bG1BpaSlKS0tl\nl9EnZq4dMH/9fRko+w37qKgoeL3ertderxd2u/2i7RobG1FYWAiXy4WRI0d2fT0yMhIAMGbMGOTm\n5sLtdl8U9mY3JHwIxl8xXnYZfXLFD64wbe0AMEhhMxmRXn7/WlJSUtDc3AyPx4OOjg5UV1cjJyen\n2zaHDx9GXl4eXnnlFTgcjq6vt7e349SpUwCA06dPY/v27UhISBiAX0EeFeb9V0mo4Dkg0sfvyD48\nPBwVFRXIysqCz+dDQUEB4uLiUFlZCQAoKirCihUrcPz4cRQXFwMAbDYb3G432trakJeXBwDo7OzE\nvHnzMGvWrAH+dYIrKS0JrZ2tssvos/T0dNklCBl87WDZJQgx8/E3c+2A+evvC7+tlwP+wxXF1HP2\ndf9Th9W7VqNufp3sUixpwtMTsOveXZgwfILsUoiCqi/ZyUlPAWb+H1Wo4Dkg0odhT0RkAQx7QbxX\nQB4eeyL9GPYC2AkiH88BkT4MeyIiC2DYC1LAqQRZeOyJ9GPYC2AniHw8B0T6MOyJiCyAYS+IHSHy\n8NgT6cewF8BOEPl4Doj0YdgTEVkAw14QO0Lk4bEn0o9hL4CdIPLxHBDpw7AnIrIAhr0gdoTIw2NP\npB/DXgA7QeTjOSDSh2FPRGQBDHtB7AiRh8eeSD+GvQB2gsjHc0CkD8OeiMgCGPaC2BEiD489kX4M\neyIiC2DYC2Dbn3w8B0T6MOwFsSNEHh57Iv0Y9kREFsCwF8C2P/l4Doj0YdgLYkeIPDz2RPox7ImI\nLIBhL4CdIPLxHBDpw7AXxI4QeXjsifRj2BMRWQDDXgA7QeTjOSDSh2EviB0h8vDYE+nHsCcisgCG\nvQB2gsjHc0CkT69h73K54HQ6ERsbi/Ly8ou+v3nzZiQlJSExMRHTp09HY2Oj7n1DATtC5OGxJ9LP\nb9j7fD6UlJTA5XKhqakJVVVVOHjwYLdtrrnmGuzcuRONjY149NFHcd999+nel4iIgsNv2Lvdbjgc\nDsTExMBmsyE/Px81NTXdtpk2bRqGDx8OAEhLS0NLS4vufc2OnSDy8RwQ6RPu75utra2Ijo7uem23\n27F3795Lbr9+/XpkZ2cHtG9paWnX5+np6UhPT9dbuyGwI0QeHnuyivr6etTX1wu9h9+wD+SPaceO\nHdiwYQMaGhoC2vf8sCciootdOBAuKysL+D38hn1UVBS8Xm/Xa6/XC7vdftF2jY2NKCwshMvlwsiR\nIwPa18zYCSIfzwGRPn7n7FNSUtDc3AyPx4OOjg5UV1cjJyen2zaHDx9GXl4eXnnlFTgcjoD2DQXs\nCJGHx55IP78j+/DwcFRUVCArKws+nw8FBQWIi4tDZWUlAKCoqAgrVqzA8ePHUVxcDACw2Wxwu92X\n3JeIiIJPUSW2MyiKYupuin9r+jds/mgz/vSzP8kuxZIm/24yqm6vQkJEguxSiIKqL9nJO2gFsSNE\nHh57Iv0Y9kREFsCwF8BOEPl4Doj0YdgLYkeIPDz2RPox7ImILIBhL8DMnUShgueASB+GvSB2hMjD\nY0+kH8OeiMgCGPYC2AkiH88BkT4Me0HsCJGHx55IP4Y9EZEFMOwFsBNEPp4DIn0Y9oLYESIPjz2R\nfgx7IiILYNgLYCeIfDwHRPow7AWxI0QeHnsi/Rj2REQWwLAXwE4Q+XgOiPRh2AtiR4g8PPZE+jHs\niYgsgGEvgJ0g8vEcEOnDsCcisgCGvSC2/8nDY0+kH8NeADtB5OM5INKHYU9EZAEMe0Fs/5OHx55I\nP4a9AHaCyMdzQKRPuOwCzKylBajZAkxYJrsSazqaA+y4DJh6t+xKiIyPYS/g66+Byy5TsGuX7Eqs\n6aYqBS++CDzMsCfqFcNegKqqCAsDJkyQXYk1jRoFtHyt4r/+C0hNlV0NkbFxzp5M7bZc4A9/kF0F\nkfEx7AWoKm/skUlRFNxyC/CnPwEnTsiuhsjYGPYC2Aki38iRKm6+GXj5ZdmVEBkbw14UB/bS3X+/\nNpXDm2mJLo1hL0pl2svy/RTaj38MnDsHdkUR+cGwF3BOVcGbOOVS/3YO7r8f+P3vZVdDZFy9hr3L\n5YLT6URsbCzKy8sv+v7HH3+MadOmYciQIVizZk2378XExCAxMRFTpkzB1KlT+69qogvMnw9s2wYc\nOSK7EiJj8ttn7/P5UFJSgrq6OkRFRSE1NRU5OTmIi4vr2mb06NF49tlnsWXLlov2VxQF9fX1GDVq\nVP9XbgQqwEl7ec5fG2fkSODOO7XR/YoVEosiMii/I3u32w2Hw4GYmBjYbDbk5+ejpqam2zZjxoxB\nSkoKbDZbj+8RykvQshtHvvPPwZIlQGUlcOaMxIKIDMrvyL61tRXR0dFdr+12O/bu3av7zRVFQUZG\nBsLCwlBUVITCwsKLtiktLe36PD09Henp6brfXzatz56MwukErr8eqKoCFi6UXQ1R/6mvr0d9fb3Q\ne/gNe9ElZBsaGhAZGYljx44hMzMTTqcTM2bM6LbN+WFvTox7WXq6oW3ZMuChh4B/+Afw4jmFjAsH\nwmVlZQG/h99pnKioKHi93q7XXq8Xdrtd95tHRkYC0KZ6cnNz4Xa7Ay7QyFR240h34TThrFnA2bPA\nX/4iqSAig/Ib9ikpKWhubobH40FHRweqq6uRk5PT47YX/tG1t7fj1KlTAIDTp09j+/btSEhI6Key\niXqmKMDSpcAzz8iuhMhY/E7jhIeHo6KiAllZWfD5fCgoKEBcXBwqKysBAEVFRWhra0NqaipOnjyJ\nQYMGYe3atWhqasKXX36JvLw8AEBnZyfmzZuHWbNmDfxvFETa/944tJflUtOM99wD/OY3wGefAdde\nG+SiiAyq1yWOZ8+ejdmzZ3f7WlFRUdfn48aN6zbV871hw4bhgw8+6IcSjY1RbzyXXw4UFgJPPQU8\n95zsaoiMgXfQCgjltlKzuFT769KlWlfOl18GuSAig2LYC+PYXhZ/y0tHRAD5+cC//EsQCyIyMIa9\nAA7sjW35cm01zJMnZVdCJB/DXghbL2XzN5V2zTVAZiawbl0QCyIyKIa9ABXgEscS6bnp76GHgKef\nBr77LggFERkYw14Us97QpkwBEhKAV16RXQmRXAx7AezGkU/PYnS//CVQXg74fEEoiMigGPaC+MBx\nefQe+xtvBK66CnjttQEuiMjAGPYCOLA3B0UBSku1de45uierYtgL4Hr28umdSsvMBEaNAqqrB7gg\nIoNi2ItQOY0jUyBLcHN0T1bH
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 230
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Now assume we get another sample, $x_1=7$"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"pmu2 = pxt(7,mus)*pmu1\n",
|
||
|
|
"pmu2 /= C*sum(pmu2)\n",
|
||
|
|
"plot(xs,pmu2)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 236,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x19319d50>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD9CAYAAABDaefJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGdBJREFUeJzt3X10U/X9B/B3IKlou5WH8SBJsUdSSCpSqu1B3NAcATsQ\nIlB2rE8wVrFDq4DjMB/2Y2VTpMOnjvpQH1CYnNLjdAS3krmKUQbSwlBxwrQqkVCQoxVUZKw03t8f\n9zSnj2lumuSb773v1zk5NPSa+zanvPPtJzf3mhRFUUBERLrQT3QAIiKKHZY6EZGOsNSJiHSEpU5E\npCMsdSIiHWGpExHpSK+l7vV64XA4kJWVhfLy8i7f9/l8SE9PR25uLnJzc3H//ffHJSgREfXOHO6b\nwWAQpaWlqKurg9VqRX5+PtxuN5xOZ4ftrrzySmzdujWuQYmIqHdhV+oNDQ2w2+3IzMyExWJBUVER\nPB5Pl+34+SUiouQQdqXe1NSEjIyM0H2bzYb6+voO25hMJuzatQs5OTmwWq146KGHkJ2d3WUbIiLS\nTuuiOexKPZIyvuSSSxAIBPDee+/hjjvuwOzZs3sMJuvtt7/9rfAMRs0vc3bmF3+TPX80wpa61WpF\nIBAI3Q8EArDZbB22+cEPfoDzzjsPADB9+nScPXsWX331VVRhiIiob8KWel5eHhobG+H3+9HS0oKa\nmhq43e4O2xw/fjz0itLQ0ABFUTB48OD4JSYioh6FnambzWZUVlaioKAAwWAQxcXFcDqdqKqqAgCU\nlJTgz3/+M5588kmYzWacd9552Lx5c0KCJ5LL5RIdoU9kzi9zdoD5RZM9fzRMSrSDGy07MZming8R\nERlVNN3JT5QSEekIS52ISEdY6kREOsJSJyLSEZY6keR4DAK1x1Inktjf/w4UFopOQcmEpU4ksZMn\ngS1bgEOHRCehZMFSJ5KcogBPPy06BSULljqRxBQFmDABWL8eaGkRnYaSAUudSHJjxwLjxgGvvCI6\nCSUDljqRDvzyl8BTT4lOQcmApU4kMUUBTCbg2muBDz8EDh4UnYhEY6kT6UBKClBczNU6sdSJdGPR\nIuDFF4HTp0UnIZFY6kQSaxu/AMAFFwCTJgHV1WIzkVgsdSIdueMO4I9/5KkDjIylTiSxzuV99dXq\n8epvvikmD4nHUieSXNv4pe3rJUuAxx4Tl4fEYqkT6czNNwP//Cfw6aeik5AILHUiiXU3O09NVQ9v\nrKxMfB4Sj6VOJLn245c2t98ObNgAfPtt4vOQWCx1Ih0aNQq46irghRdEJ6FEY6kTSSzcoYtLlwLr\n1gHff5+4PCQeS51Ict2NXwDg8suB9HTg1VcTm4fEYqkT6ZTJBKxYAZSX88NIRsJSJ5JYb2U9dy7w\nxRfqIY5kDCx1Isn1NH4BgP79geXLgT/8IXF5SCyWOpHOzZ8P7N0L/PvfopNQIrDUiSQWyaz83HPV\nE32tXRv/PCSeWXQAIuqbcOOXNosXA6NHA4cPq8ewk35xpU5kAIMGAQsXAo8+KjoJxRtLnUhi7S+S\n0Ztly9RTB3zxRXwzkVgsdSKDsNmA664DHnlEdBKKJ5Y6kYHccw/w9NNAc7PoJBQvLHUiiWkZvwDq\nm6SFhVyt61mvpe71euFwOJCVlYXy8vIet9uzZw/MZjNeeeWVmAYkoti6917gqaeAr74SnYTiIWyp\nB4NBlJaWwuv14sCBA6iursbBgwe73e7Xv/41fvrTn0LhSSaIEiaaf26ZmcCcObzknV6FLfWGhgbY\n7XZkZmbCYrGgqKgIHo+ny3br1q3DvHnzMHTo0LgFJaLuaRm/tLn3XuCJJ4ATJ2Kfh8QK++GjpqYm\nZGRkhO7bbDbU19d32cbj8WD79u3Ys2cPTD38hJWVlYW+drlccLlc0acmoj658EJg1iygogJo90+T\nBPP5fPD5fH16jLCl3lNBt7d06VKsWbMGJpMJiqL0OH4p408OUcz1Zdp5333AZZcBd94JDB4cu0wU\nvc4L3lWrVml+jLClbrVaEQgEQvcDgQBsNluHbf71r3+hqKgIAPDll19i27ZtsFgscLvdmsMQkXbR\njF8AwG5XT81bXq7eSB/ClnpeXh4aGxvh9/sxcuRI1NTUoLq6usM2n376aejrhQsXYtasWSx0Ikms\nXAmMHw8sWQKMHCk6DcVC2DdKzWYzKisrUVBQgOzsbFx33XVwOp2oqqpCVVVVojISUQ/6erCZzQYU\nFwO//31s8pB4JiUBxyC2zduJKLaefRbYvVv9M1rNzcDYserj2O2xy0Z9F0138hOlRAY3ZIg6flm5\nUnQSigWWOpHEYvUL8LJlwPbtwLvvxubxSByWOpHkoj36pb20NPUDSffe2/fHIrFY6kQEACgpAT78\nEHj9ddFJqC9Y6kQSi+XxB+ecox6vftddQDAYu8elxGKpE0kuFuOXNoWFQHo68PzzsXtMSiyWOhGF\nmEzqudZXrgS+/VZ0GooGS51IYlovkhGJvDxg6lRgzZrYPi4lBkudiLpYvVq9kMZnn4lOQlqx1Imo\nC5sNKC1Vr2lKcmGpE0ksHuOXNitWADt2qDeSB0udiLqVmqq+aXr77UBrq+g0FCmWOpHE4n2evHnz\ngGHDgMcfj+9+KHZY6kSSi9f4pe2xKyuB++8Hjh2L334odljqRBSWw6Gec33FCtFJKBIsdSKJJeoy\nBb/5DfDmm8BbbyVmfxQ9ljqR5OI5fmmTlqa+aXrbbcDZs/HfH0WPpU5EESksBEaNAtauFZ2EwmGp\nE0kskVeJNJmAJ59UV+wffpi4/ZI2LHUiySVi/NLmggvUk33deivw/feJ2y9FjqVORJrcfjvwv/8B\nzzwjOgl1h6VOJLFEjl/a9O8PPPusekRMU1Pi90/hsdSJJJfI8UubceOAxYvVVbuIFxbqGUudiKJy\n333qG6YvvSQ6CbXHUieSmMhV8jnnAC+8ANxxB08hkExY6kSSEzF+aTNxIrBokXrjGCY5sNSJqE9W\nrlTfMF2/XnQSAljqRFKL50UyIpWSAmzcCNx9N+D3i81CLHUiioGLLwaWLwd+/nN+KEk0ljoRxcTy\n5erJvioqRCcxNrPoAEQUvWQYv7Tp318dw1x2GeByAbm5ohMZE1fqRBQzo0erK/WiIuDUKdFpjIml\nTiSxZDyM8IYbgMsvB+68U3QSY2KpE0kuWcYv7a1bB+zcCVRXi05iPCx1Ioq5tDRg82Z1tf7pp6LT\nGAtLnUhiyTh+aZObq54fpqhIPVUvJUavpe71euFwOJCVlYXy8vIu3/d4PMjJyUFubi4uvfRSbN++\nPS5Biah7yTh+abNkCWC1AnfdJTqJcZgUpefX+mAwiLFjx6Kurg5WqxX5+fmorq6G0+kMbfPdd98h\nNTUVAPD+++9jzpw5+PjjjzvuxGRCmN0QUZQefRQ4fFj9M1l9/TWQn6+eTuCmm0SnkUs03Rl2pd7Q\n0AC73Y7MzExYLBYUFRXB4/F02Kat0AHg1KlT+NGPfqQpABFFT4a1Uno68PLLwLJlwP79otPoX9gP\nHzU1NSEjIyN032azob6+vst2W7ZswT333INjx47htdde6/axysrKQl+7XC64XK7oEhNRB8k8fmlz\n8cXAY48BhYXAnj3AwIGiEyUnn88Hn8/Xp8cIW+qmCH9aZs+ejdmzZ2PHjh24+eab8WE3lxpvX+pE\nZDw33gi8/bZ6fphXXgH68TCNLjoveFetWqX5McI+rVarFYFAIHQ/EAjAZrP1uP3kyZPR2tqK5uZm\nzUGISDsZxi/tPfIIcPw48OCDopPoV9hSz8vLQ2NjI/x+P1paWlBTUwO3291hm08++SQ0yN+3bx8A\nYMiQIXGKS0SdyTB+aZOSos7Xn3wS+MtfRKfRp7DjF7PZjMrKShQUFCAYDKK4uBhOpxNVVVUAgJKS\nErz88svYuHEjLBYL0tLSsHnz5oQEJyI5jRypFvqMGcCFFwI5OaIT6UvYQxpjthMe0kgUFw89pF4f\n9OGHRSfRrqYGWLECaGgAhg8X
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 236
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"result = zeros(xs.shape)\n",
|
||
|
|
"for i,mu in enumerate(mus):\n",
|
||
|
|
" weight = pmu2[i]\n",
|
||
|
|
" result += weight * pxt(xs,mu)\n",
|
||
|
|
"result /= C*sum(result)\n",
|
||
|
|
"plot(xs,result)\n",
|
||
|
|
"plot(xs,pxt(xs,7))\n",
|
||
|
|
"plot(xs,pxt(xs,4))"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 237,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x194731d0>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD9CAYAAABdoNd6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH1NJREFUeJzt3X94FPWdB/D3hixyB5VfBxF2wxNlo7tBEvBJSCPN3Z4Q\ncqa6lUjbaHxEDOmaPjlR8a7Xx7MXsFBSj2pKuLtIAVF5IHetNegT9jAe+6jRZK8FGh/BI1hTNoEg\nIggSacgy98eWNFmyk9l8s/ud2X2/nocnbDKz+WRm8853PvudGZOiKAqIiCiuJckugIiIoo9hT0SU\nABj2REQJgGFPRJQAGPZERAmAYU9ElACGDXuPxwO73Y709HRUV1df8/WdO3ciKysLmZmZWLhwIdra\n2vq/lpaWhszMTMyfPx8LFiwY3cqJiEgzk9o8+0AggFtuuQVNTU2wWCzIycnBrl274HA4+pd5//33\nkZGRgYkTJ8Lj8aCqqgotLS0AgBtvvBG//e1vMWXKlOj/JEREFJbqyN7n88FmsyEtLQ1msxklJSVo\naGgYtExeXh4mTpwIAMjNzUVnZ+egr/OcLSIi+ZLVvtjV1YXU1NT+x1arFa2trWGX37p1K4qKivof\nm0wmLF68GGPGjIHb7UZ5efmg5U0m00jrJiJKaJEOpFXDPpIw3r9/P7Zt24bm5ub+zzU3N2PGjBk4\nffo0CgoKYLfbkZ+fL1SwnlQtX46qY8eA//ov2aWMSNXGjahavVp2GSNWdcstqDp8GBgwIDGSqqoq\nVFVVyS5jRIxcO2D8+kcyUFYNe4vFAr/f3//Y7/fDarVes1xbWxvKy8vh8XgwefLk/s/PmDEDADBt\n2jQsXboUPp/vmrA3vHHjgJkzZVcxMl/7mnFrB4AkTiYj0kr1tyU7Oxvt7e3o6OhAb28v6uvr4XK5\nBi1z/PhxFBcX45VXXoHNZuv/fE9PDy5cuAAAuHjxIvbt24e5c+dG4UeQyMBHJXGD+4BIE9WRfXJy\nMmpra1FYWIhAIICysjI4HA7U1dUBANxuN9auXYuzZ8+ioqICAGA2m+Hz+dDd3Y3i4mIAQF9fH0pL\nS7FkyZIo/zix5czKArq6ZJcxYk6nU3YJQpzXXSe7BCFG3v5Grh0wfv0joTr1Murf3GQydM8eTU3A\nhg3BjxR7s2YB774b/EiUQEaSnWx6ijDyH6p4wX1ApAnDnogoATDsRfFcAXm47Yk0Y9iLYAtBPu4D\nIk0Y9kRECYBhL4qtBHm47Yk0Y9iLYAtBPu4DIk0Y9kRECYBhL4qtBHm47Yk0Y9iLYAtBPu4DIk0Y\n9kRECYBhL4qtBHm47Yk0Y9iLYAtBPu4DIk0Y9kRECYBhL4qtBHm47Yk0Y9iLYAtBPu4DIk0Y9kRE\nCYBhL4qtBHm47Yk0Y9iLYAtBPu4DIk0Y9kRECYBhL4qtBHm47Yk0Y9gTESUAhr0I9ovl4z4g0oRh\nL4qtBHm47Yk0Y9gTESUAhr0IthDk4z4g0oRhL4qtBHm47Yk0Y9gTESUAhr0IthDk4z4g0oRhL4qt\nBHm47Yk0Y9gTESUAhr0IthDk4z4g0oRhL4qtBHm47Yk0Y9gTESUAhr0IthDk4z4g0mTYsPd4PLDb\n7UhPT0d1dfU1X9+5cyeysrKQmZmJhQsXoq2tTfO6cYGtBHm47Yk0Uw37QCCAyspKeDweHD58GLt2\n7cKRI0cGLXPTTTfh7bffRltbG55++ml873vf07wuERHFhmrY+3w+2Gw2pKWlwWw2o6SkBA0NDYOW\nycvLw8SJEwEAubm56Ozs1Lyu4bGFIB/3AZEmyWpf7OrqQmpqav9jq9WK1tbWsMtv3boVRUVFEa1b\nVVXV/3+n0wmn06m1dn1gK0EebntKEF6vF16vV+g5VMPeFMEv0/79+7Ft2zY0NzdHtO7AsCciomuF\nDoTXrFkT8XOohr3FYoHf7+9/7Pf7YbVar1mura0N5eXl8Hg8mDx5ckTrGhpbCPJxHxBpotqzz87O\nRnt7Ozo6OtDb24v6+nq4XK5Byxw/fhzFxcV45ZVXYLPZIlo3LrCVIA+3PZFmqiP75ORk1NbWorCw\nEIFAAGVlZXA4HKirqwMAuN1urF27FmfPnkVFRQUAwGw2w+fzhV2XiIhiz6Qo8o6DTSYTJH57cb/6\nFbBzJ/Dqq7IrSUy33grs2gXMnSu7EqKYGkl28gxaUWwlyMNtT6QZw56IKAEw7EUYuQUVL7gPiDRh\n2ItiK0EebnsizRj2REQJgGEvgi0E+bgPiDRh2ItiK0EebnsizRj2REQJgGEvgi0E+bgPiDRh2Iti\nK0EebnsizRj2REQJgGEvgi0E+bgPiDRh2ItiK0EebnsizRj2REQJgGEvgi0E+bgPiDRh2ItiK0Ee\nbnsizRj2REQJgGEvgi0E+bgPiDRh2ItiK0EebnsizRj2REQJgGEvgi0E+bgPiDRh2BMRJQCGvSj2\njeXhtifSjGEvgi0E+bgPiDRh2BMRJQCGvSi2EuThtifSjGEvgi0E+bgPiDRh2BMRJQCGvSi2EuTh\ntifSjGEvgi0E+bgPiDRh2BMRJQCGvSi2EuThtifSjGEvgi0E+bgPiDRh2BMRJQCGvSi2EuThtifS\njGEvgi0E+bgPiDQZNuw9Hg/sdjvS09NRXV19zdc/+ugj5OXlYdy4cdi4ceOgr6WlpSEzMxPz58/H\nggULRq9qIiKKSLLaFwOBACorK9HU1ASLxYKcnBy4XC44HI7+ZaZOnYpNmzbhtddeu2Z9k8kEr9eL\nKVOmjH7lesFWgjzc9kSaqY7sfT4fbDYb0tLSYDabUVJSgoaGhkHLTJs2DdnZ2TCbzUM+hxLPh9nx\n/LMZBfcBkSaqI/uuri6kpqb2P7ZarWhtbdX85CaTCYsXL8aYMWPgdrtRXl5+zTJVVVX9/3c6nXA6\nnZqfn4goEXi9Xni9XqHnUA17k+BhcnNzM2bMmIHTp0+joKAAdrsd+fn5g5YZGPaGxFaCPNz2lCBC\nB8Jr1qyJ+DlU2zgWiwV+v7//sd/vh9Vq1fzkM2bMABBs9SxduhQ+ny/iAnWNLQT5uA+INFEN++zs\nbLS3t6OjowO9vb2or6+Hy+UactnQ3nxPTw8uXLgAALh48SL27duHuXPnjlLZREQUCdU2TnJyMmpr\na1FYWIhAIICysjI4HA7U1dUBANxuN7q7u5GTk4Pz588jKSkJNTU1OHz4MD799FMUFxcDAPr6+lBa\nWoolS5ZE/yeKNbYS5OG2J9LMpEicLmMymYw9W2f7duDtt4MfKfYWLgR++tPgR6IEMpLsVB3Zk7rA\nlT58dvFT/N8f3pZdSkLK+uN5fHLyIM7/ISC7FF2a+hdTMWf6HNllkE4w7AUcO3MM//v7N/HC/1yQ\nXUpCqj1/HFt+uwUfXJgouxTduXzlMo59fgyn/+G07FJIJxj2AgJKAFP/8q/w9gqO7KXY+g1s/uYG\n4BvfkF2J7nz+1eeY/fPZsssgHeGF0EQY+f2GeMF9QKQJw16AogDghBB5OBsnLBMMPvmBRh3Dnogo\nATDshVwBh/aScfQ6JJPJBAXcNvRnDHtRzHp52MYJy8QXJoVg2AvgoJL0jD17GohhL0JhG0c6BtqQ\n2MahUAx7Ucx6edjGIdKMYS8gOKhk4JD+cOolhWLYCzDxMFk+BhqRJgx7AQoUDuxlYhsnLPbsKRTD\nXhgDh/SHUy8pFMNexBWOnKRjGycs9uxpIIa9ALZxJGMbJyy2cSgUw56IKAEw7EXwspfysVUxJE69\npFC8eYmAwBXg0h8Br1d2JYkp6wsT/nAIOMchyzX+GODfQRqMYS+guxs4fRqoqZJdSWL62SfAyzuA\ng3tkV6I/V5JM6F2ooLsbuOEG2dWQHjDsBSgBBWPHmjiyl+UO4LanFGCR7EL05499Jvzlj4Ht24Ef\n/lB2NaQHPAAWoHC+g1ycjaMqKUnBCy8AgYDsSkgPGPYC2BMlvTKZTIBJwbRpwH//t+xqSA8Y9gJM\nTHv5uA9UPfII8B//IbsK0gOG
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 237
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"This is even weirder. After seeing the first sample, the maximum likelihood\n",
|
||
|
|
"estimator predicts only values between 0 and 4 occurring, but after seeing\n",
|
||
|
|
"another training sample, it is changing its mind and now predicts that values\n",
|
||
|
|
"between 0 and 7 can occur.\n",
|
||
|
|
"\n",
|
||
|
|
"The Bayesian estimator, in contrast, \"knows\" that the parameter must be greater than 7,\n",
|
||
|
|
"so it predicts a uniform distribution for the interval [0...7] and then a tradeoff\n",
|
||
|
|
"between the parameter distribution and the uniform distribution of the parameters."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"The last sample illustrates this further.\n",
|
||
|
|
"\n",
|
||
|
|
"A sample of $x_3=2$ doesn't cause any update to the maximum likelihood estimator,\n",
|
||
|
|
"but it does cause an update to posterior distribution."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"pmu3 = pxt(2,mus)*pmu2\n",
|
||
|
|
"pmu3 /= C*sum(pmu3)\n",
|
||
|
|
"plot(xs,pmu3)\n",
|
||
|
|
"plot(xs,pmu2)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 238,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x19300650>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD9CAYAAABDaefJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XtcVGX+B/DPwRlTwUy8tjP4IxuUQVMxWLbMnPKCmpIi\nJVpphka2mHZZt9qtdC2FrbwEleSlzCuWJdjiWKSzmhfIrLS0pJIcSS1NMzVDxvP7Y5bJgZkDAzNz\neGY+79eL14uZ88ycr7z045dnnvMcSZZlGUREFBBC1C6AiIi8h6FORBRAGOpERAGEoU5EFEAY6kRE\nAYShTkQUQGoNdbPZjOjoaERFRSErK8vlGIvFgtjYWHTv3h0mk8nbNRIRUR1JSuvUbTYbunbtiqKi\nIuh0OsTHx2P16tUwGo2OMadPn0afPn2wadMm6PV6nDhxAm3btvVL8URE5EyxUy8pKYHBYEBkZCS0\nWi1SU1ORn5/vNGbVqlUYNWoU9Ho9ADDQiYhUpFE6WF5ejoiICMdjvV6P4uJipzGlpaW4ePEibrnl\nFvz666+YOnUq7rnnHqcxkiR5sWQiouDh6UX/ip16XcL44sWL2LNnDwoLC7Fp0ybMmjULpaWlLgsT\n9euZZ55RvYZgrV/k2lm/+l+i118fip26TqeD1Wp1PLZarY5plioRERFo27YtmjdvjubNm+Pmm2/G\n559/jqioqHoVRERE9afYqcfFxaG0tBRlZWWoqKhAXl4ekpKSnMbcfvvt+Oijj2Cz2XD+/HkUFxcj\nJibGp0UTEZFrip26RqNBTk4OEhMTYbPZkJaWBqPRiNzcXABAeno6oqOjMXjwYPTo0QMhISGYNGlS\nwIW66Ms0Ra5f5NoB1q820euvD8UljV47iSTVe36IiChY1Sc7eUUpEVEAYagTEQUQhjoRUQBhqBMR\nBRCGOhFRAGGoExEFEIY6EVEAYagTEQUQhjoRUQBhqBMJbP9+IDtb7SqoMWGoEwnswAFgzhyAu3BQ\nFYY6keCOHgX27VO7CmosGOpEAqvq0DdtUrcOajwY6kSCCw0FzGa1q6DGgqFOJLibbgJKSoBz59Su\nhBoDhjqRwGQZaNkSuP56wGJRuxpqDBjqRAFg8GDOq5MdQ51IcJIEJCZyXp3sGOpEAqta/dKzJ/DL\nL8ChQ+rWQ+pjqBMFgJAQe7fOKRhiqBMJTJbt0y8Ap2DIjqFOFCAGDQK2bAF+/13tSkhNDHUiwVV1\n6u3aAd26cWljsGOoEwms+kZew4cD772nTi3UODDUiQLI8OHAhg3ctTGYMdSJBFc1/QLYp18kCfji\nC/XqIXUx1IkEVr0jlyRg2DB7t07BiaFOFGA4rx7cag11s9mM6OhoREVFISsrq8Zxi8WCVq1aITY2\nFrGxsXj22Wd9UigRuXb59AsA9OsHfPkl8OOP6tRD6tIoHbTZbMjIyEBRURF0Oh3i4+ORlJQEo9Ho\nNK5fv34oKCjwaaFEVJOrD0SvuAIYMAAoLATuvdfvJZHKFDv1kpISGAwGREZGQqvVIjU1Ffn5+TXG\nyfyonahRqVoFQ8FHsVMvLy9HRESE47Fer0dxcbHTGEmSsGPHDvTs2RM6nQ4vvPACYmJiarzXjBkz\nHN+bTCaYTKaGVU5EAGpOvwDA0KHA1KnAhQtAs2b+r4nqx2KxwNLAq8cUQ11y9belmt69e8NqtaJF\nixbYuHEjRowYgYMHD9YYd3moE5F3uPsluX17+86NRUX21TAkhuoN78yZMz1+D8XpF51OB6vV6nhs\ntVqh1+udxrRs2RItWrQAAAwZMgQXL17Ezz//7HEhRORdycnAO++oXQX5m2Kox8XFobS0FGVlZaio\nqEBeXh6SkpKcxhw/ftwxp15SUgJZlhEeHu67ionI4fJdGqtLTgYKCoCLF/1bE6lLcfpFo9EgJycH\niYmJsNlsSEtLg9FoRG5uLgAgPT0db7/9Nl599VVoNBq0aNECa9as8UvhRKSsUyfgmmuArVuB/v3V\nrob8RZL9sHRFkiSukCHygRUr7Huor1jh+nhmJnD4MPDKK/6ti7yjPtnJK0qJBKY0/QLYp2DefRe4\ndMl/NZG6GOpEAaxLF6BtW2DnTrUrIX9hqBMJrraVx6NGcRVMMGGoEwnsWMW3+DZsmeKY5GRg3Tru\nsR4sGOpEAjtc8RmK203G8bPH3Y657jqgaVNg924/FkaqYagTCc4W8hte3Pmi2+OSBIweDXC1cXBg\nqBMJTJZltLkQj8V7FuPE+RNux6WmAnl5XAUTDBjqRIILreyElJgUzN813+2Ybt2A8HDgo4/8WBip\ngqFOJDAZMiQJePymx/Hq7ldx+sJpt2PHjOEUTDBgqBOJTAYACZ1bd8bwLsPxUvFLboempgJvv829\nYAIdQ50oQDzZ90lkl2TjzO9nXB6/5hqgc2fgww/9XBj5FUOdSGCXZBkS7FcfdWnTBQM7D0R2cbbb\n8ZyCCXwMdaIAMsM0A/N2zcOp3065PH7nnUB+vv2OSBSYGOpEQnO+TLRLmy4YET0CL+x8weXoq68G\nevcG/vMff9RGamCoEwnMHunOm7883e9pLNy90O1VpvfcAyxT3lmABMZQJxJazQ1dOrXqhLt73I05\nH81x+YpRo+w3zjjufmcBEhhDnUhwEmpu0/jkTU9i+d7lOPzL4RrHWrYEbr8dWLXKH9WRvzHUiQTm\nbufFDmEdkH59OmZtneXy+PjxnIIJVAx1IoHJLqZfqvztxr9h/Vfr8fWJr2scM5mAU6eAzz7zYXGk\nCoY6keBcTb8AQOvmrTH9xumYXjS9xrGQEGDcOHbrgYihTiSw2m5KPCVhCvYe3wtLmaXGsXHj7PPq\n3DYgsDDUiYTn/n52zTTNkNk/E4++/yguyc777kZFAQYDsHGjr+sjf2KoEwmutnuU3tntTmhDtFi5\nd2WNY/feC7zxhk/KIpUw1IkEZp9+UU51SZIwN3Eu/rH5Hzh/8bzTsdGjgS1bgGPHfFgk+RVDnSgI\n3BhxIxL0CZi3c57T81deCaSkAEuXqlQYeR1DnUhgSksaq8vsn4m5u+bi2Fnntjw9HVi0iLe6CxQM\ndSLBuVvSWN214dciLTYN0z9wXuJ4/fVA69bABx/4ojryN4Y6kcBqW9JY3VM3P4XNhzbjo8N/3KxU\nkuzdem6ut6sjNdQa6mazGdHR0YiKikJWVpbbcR9//DE0Gg3eeecdrxZIRO7ZPyatW6cOAC2vaIkX\nBr2AjMIMVF6qdDw/dqz9A9MffvBBkeRXiqFus9mQkZEBs9mM/fv3Y/Xq1Thw4IDLcX//+98xePBg\njzsHIvKv0d1GI7x5OBbuXuh4rmVL+w00+IGp+BRDvaSkBAaDAZGRkdBqtUhNTUV+fn6NcdnZ2UhJ\nSUG7du18VigRueJ5EyVJErKHZGPmf2fix3M/Op5PTwcWLwZsNm/WR/6mUTpYXl6OiIgIx2O9Xo/i\n4uIaY/Lz87F582Z8/PHHkNxcCTFjxgzH9yaTCSaTqf5VExGAql0a6z79UqVb+24Y13McHi96HEtv\nt7fnvXsD7dvbrzAdNsy7dVLdWCwWWCyWBr2HYqi7C+jLTZs2DZmZmZAkCbIsu51+uTzUicg7PFnS\nWN0z/Z6B8WUjth/ejj6d+gAApkwBXnqJoa6W6g3vzJkzPX4PxekXnU4Hq9XqeGy1WqHX653GfPLJ\nJ0hNTcU111yDdevW4cEHH0RBQYHHhRBR/dSl+XLlyiuuxPzE+Zi0YRJ+r/wdgH1efd8+YP9+b1ZI\n/qQY6nFxcSgtLUVZWRkqKiqQl5eHpKQkpzHfffcdDh06hEOHDiElJQWvvvpqjTFE5CMNXJeQEpMC\nQ7gBmR9lAgCuuAJ44AF7t05iUgx1jUaDnJwcJCYmIiYmBqNHj4bRaERubi5yuaiVSHUNmX4B7F3+\ny0NfRnZJNg78ZF/Zlp4O5OUBP//sjQrJ3yTZD2sQq+bbici7xs19AyU/WvBV5hsNep+ckhys+WIN\ntk7YihApBOPGAd27A9Nr3l+D
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 238
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"result = zeros(xs.shape)\n",
|
||
|
|
"total = 0\n",
|
||
|
|
"for i,mu in enumerate(mus):\n",
|
||
|
|
" weight = pmu3[i]\n",
|
||
|
|
" result += weight * pxt(xs,mu)\n",
|
||
|
|
" total += weight\n",
|
||
|
|
"result /= total\n",
|
||
|
|
"plot(xs,result)\n",
|
||
|
|
"plot(xs,pxt(xs,7))"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 239,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x194884d0>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD9CAYAAABdoNd6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X10VOWdB/DvhBmLJgdQNi/dmXCizmgmoElgsimsriMS\nUyPOQYhrACXGULO4EXFx157uS5O2vKSUxTTRNngqwtGGbHVLIh2nNcU5UjFJbeJLTTgMHqZOgqCr\nhhRRh4x3/xiTk4Rwk8mdyXPv3O/nHM5hMvfe+Q3n5stzn+e5zzVIkiSBiIjiWoLoAoiIKPYY9kRE\nOsCwJyLSAYY9EZEOMOyJiHSAYU9EpAMThr3H40FmZiZsNhtqamoueP/o0aNYvHgxZs6ciZ07d456\nr7+/H8XFxbDb7cjKykJbW1v0Kiciokkzyr0ZCoVQWVmJ1tZWmM1m5OXlweVywW63D28zd+5c1NXV\n4cCBAxfs//DDD6OoqAjPP/88BgcH8dlnn0X/GxAR0YRkW/YdHR2wWq3IyMiAyWRCSUkJmpubR22T\nnJwMh8MBk8k06udnzpzB4cOHcf/99wMAjEYjZs+eHeXyiYhoMmRb9n19fUhPTx9+bbFY0N7ePqkD\nnzhxAsnJySgrK8Nbb72FRYsWoba2FpdddtnwNgaDYYplExHpW6SLH8i27JWE8eDgIDo7O/Hggw+i\ns7MTiYmJ2L59+wXbSZKk2T/f//73hdfA+sXXocf6tVx7PNQ/FbJhbzabEQgEhl8HAgFYLJZJHdhi\nscBisSAvLw8AUFxcjM7OzikVSUREysiGvcPhgM/ng9/vRzAYRFNTE1wu17jbjv3fJi0tDenp6Th2\n7BgAoLW1FfPnz49S2UREFAnZPnuj0Yj6+noUFhYiFAqhvLwcdrsdDQ0NAICKigqcOnUKeXl5GBgY\nQEJCAmpra9Hd3Y2kpCTU1dVh7dq1CAaDuPrqq7Fnz55p+VLTxel0ii5BEdYvlpbr13LtgPbrnwqD\nNNUOoGh8uMEw5f4nIiK9mkp28g5aIiIdYNgTEekAw56ISAcY9kREOsCwJyLSAYY9EZEOMOyJiHSA\nYU9EpAMMeyIiHWDYExHpAMOeiEgHGPZERDrAsCci0gGGPRGRDjDsiYh0gGFPRKQDDHsiIh1g2BMR\n6YDsM2hJ3umzp3HVT6/Cl4Nfii6F6AJJlyTh1KOnMNM4U3QppAIThr3H48GmTZsQCoWwfv16PPbY\nY6PeP3r0KMrKytDV1YUtW7Zg8+bNo94PhUJwOBywWCx48cUXo1u9YGeDZ5GSmALfQz7RpRBd4PKa\nyxEMBRn2BGCCsA+FQqisrERrayvMZjPy8vLgcrlgt9uHt5k7dy7q6upw4MCBcY9RW1uLrKws/PWv\nf41u5SqRYEiAMYEXSKQ+BhhEl0AqIttn39HRAavVioyMDJhMJpSUlKC5uXnUNsnJyXA4HDCZTBfs\n39vbC7fbjfXr10f8JHQtkBB/34niSzz+3tHUyDZJ+/r6kJ6ePvzaYrGgvb190gd/5JFHsGPHDgwM\nDFx0m6qqquG/O51OOJ3OSR9fDdh6IrUyGHhuxguv1wuv16voGLJhr+RkOXjwIFJSUpCbmytb5Miw\n1xq2mkjtePUZH8Y2hKurqyM+hmw3jtlsRiAQGH4dCARgsVgmdeAjR46gpaUFV155JVavXo1Dhw5h\n3bp1ERdIRETKyYa9w+GAz+eD3+9HMBhEU1MTXC7XuNuObeVu3boVgUAAJ06cwP79+7F06VLs27cv\nepWrBC+VSa3YxUgjyXbjGI1G1NfXo7CwEKFQCOXl5bDb7WhoaAAAVFRU4NSpU8jLy8PAwAASEhJQ\nW1uL7u5uJCUljTpWPIYiL5FJ7djVSEMMksCzwWAwaPpkPPbxMdz+y9s5z55Uac72OTjx8Alcfunl\nokuhKJtKdnK5BIV4qUxqFY9X0zR1DHsFtHxVQvrArkYawrBXiK0nUiteddJIDHsiIh1g2CvAS2RS\nO3Y10hCGvUK8VCa1YhcjjcSwV4CtJlI7Xn3SEIY9EZEOMOwV4qUyqRW7GGkkhr0CvEQmtWNXIw1h\n2BMR6QDDXiFeKpNasYuRRmLYK8BLZFI7djXSEIa9Qmw9kVrxqpNGYtgrwFYTqR2vPmkIw56ISAcY\n9grxUpnUil2MNBLDXgFeIpPasauRhsg+g5bkffIJ0NMDXHqp6EqILvTlRuDMGSAtaeJtKf5NGPYe\njwebNm1CKBTC+vXr8dhjj416/+jRoygrK0NXVxe2bNmCzZs3AwACgQDWrVuHDz/8EAaDAQ888AA2\nbtwYm28hyGefATNmGPDJJ6IrIbrQ7B8a8Kc/AdeaRVdCaiAb9qFQCJWVlWhtbYXZbEZeXh5cLhfs\ndvvwNnPnzkVdXR0OHDgwal+TyYRdu3YhJycHZ8+exaJFi1BQUDBqX6376utuHLbsSY0uuQR4408S\n1rhEV0JqIBv2HR0dsFqtyMjIAACUlJSgubl5VGAnJycjOTkZv/nNb0btm5aWhrS0NABAUlIS7HY7\nTp48eUHYV1VVDf/d6XTC6XQq+DrTjwO0pFaXmAz40xHRVVA0eL1eeL1eRceQDfu+vj6kp6cPv7ZY\nLGhvb4/4Q/x+P7q6upCfn3/BeyPDnoiix3QJ0PUm8NVXQAKnYmja2IZwdXV1xMeQPQWiMXXr7Nmz\nKC4uRm1tLZKS4muk6CvOxiEVS0gArrhCQk+P6EpIDWTD3mw2IxAIDL8OBAKwWCyTPvj58+exatUq\n3HPPPVixYsXUq1Q1duOQOhlgwKJFwOuvi66E1EA27B0OB3w+H/x+P4LBIJqamuByjT/aM3bOuSRJ\nKC8vR1ZWFjZt2hS9ilWE8+xJ7RYtkhj2BGCCPnuj0Yj6+noUFhYiFAqhvLwcdrsdDQ0NAICKigqc\nOnUKeXl5GBgYQEJCAmpra9Hd3Y0333wTzz77LK6//nrk5uYCALZt24Zvf/vbsf9WRAQAWLQIePZn\noqsgNTBIApunBoNB063jA21v4u7G+/Bl7ZuiSyG6gPm/zThS1oEF88x4/33g8stFV0TRMpXs5Bi9\nArwVndRuxgwJDgfQ0SG6EhKNYa8Q59mTWg2dm9/6FgdpiWGviIZ7oEhHFi9m2BPDXiGmPambJEn4\n1reA9vbwzVWkXwx7BcIte3bjkDoN3RSZkgL8zd+AN1fpHMOeSAduuAH4wx9EV0EiMewV4HIJpHZD\nM8ZuvBF49VXBxZBQDHvF2I1D6jRyptiNNwKHDwsshoRj2CsgSRKjnlRt6MYbmw0IBoG//EVwQSQM\nw14B9uKQVhgM7MrRO4a9YmzbkzqNXaKcXTn6xrBXQMvr+pA+jFzSg2Gvbwx7xdiyJ3Uau5TH9dcD\nH3wAfPihoIJIKIa9AmzXk5bMmAEsWcL59nrFsFeA3TikdmPPUXbl6BfDXiGueklqNd4zpBn2+sWw\nV4Ate1K7sc9cyMsDjh4FBgYEFUTCMOyJdOQb3wAWLuSSx3o0Ydh7PB5kZmbCZrOhpqbmgvePHj2K\nxYsXY+bMmdi5c2dE+2odV70kNbtYF6PTCXi901oKqYBs2IdCIVRWVsLj8aC7uxuNjY3oGbNO6ty5\nc1FXV4dHH3004n21jo8lJLUbr6tx6VLg0CEBxZBQsmHf0dEBq9WKjIwMmEwmlJSUoLm5edQ2ycnJ\ncDgcMJlMEe+rdZLEAVpSr/EGaIHwYwrffRc4c2aaCyKhjHJv9vX1IT09ffi1xWJBe3v7pA482X2r\nqqqG/+50OuF0Oid1fCKampkzw4H/6qvAHXeIroYmw+v1wquw70027C/WMpiMye47Muy1ht04pHYX\nO0eXLgVeeYVhrxVjG8LV1dURH0O2G8dsNiMQCAy/DgQCsFgskzqwkn01QwIgsRuH1Emui5H99voj\nG/YOhwM+nw9+vx/BYBBNTU1wuVzjbjt2ICiSfTWNWU8a5HAAJ04A//d/oiuh6SLbjWM0GlFfX4/C\nwkKEQiGUl5fDbrejoaEBAFBR
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 239
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"In fact, if we repeat the same process with a lot of samples (in this case\n",
|
||
|
|
"the true parameter is 7), we see that the Bayesian parameter estimate\n",
|
||
|
|
"becomes an increasingly peaked distribution close to the true value.\n",
|
||
|
|
"\n",
|
||
|
|
"I.e., if, out of 100 samples, we haven't seen a value greater than 7,\n",
|
||
|
|
"then the probability that the mean is significantly greater than 7 must\n",
|
||
|
|
"be very small."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"p = pmu.copy()\n",
|
||
|
|
"for i in range(100):\n",
|
||
|
|
" p = pxt(rand()*7,mus)*p\n",
|
||
|
|
" p /= C*sum(p)\n",
|
||
|
|
"plot(mus,p)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 240,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x19c3f0d0>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXEAAAD9CAYAAABKgkezAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFLtJREFUeJzt3X9M1Pcdx/HXCbjqrFatHqtAS1uJd4hAxZiSsV5nz2Ym\nGGvZUtRoQLdkzZbaNWZdsmy4xB+dWzrbbf+YaWm61K1tUmnHbhG7c2aErpbWGddIa7wNa8uWWnAU\n6w/87g/2xeMrHNz3Dr585PlISPCA771i8OX73t/v3fksy7IEADDSJK8DAADco8QBwGCUOAAYjBIH\nAINR4gBgMEocAAyWsMRra2vl9/tVVFR03dd+/vOfa9KkSTp37tyohQMAJJawxGtqahSJRK67vb29\nXQcPHtTtt98+asEAAMNLWOIVFRWaOXPmdbd/73vf009/+tNRCwUAGJnMZH/gwIEDysnJ0aJFi4b8\nHp/Pl1IoAJiokn0SfVInNnt6erR9+3Zt3bp12Du0LMvYjx//+MeeZyC/9zkmWnbye//hRlIlfurU\nKcViMRUXFys/P19nzpzR4sWL9e9//9vVnQMAUpPUOqWoqEgdHR39f87Pz9fbb7+tWbNmpT0YAGB4\nCSfx6upqlZeXq62tTbm5udq3b9+Ar9+ou+9QKOR1hJSQ3zsmZ5fIbyKf5XYRk+igPp/r/Q4ATFRu\nupNnbAKAwShxADAYJQ4ABqPEAcBglDgAGIwSBwCDUeIAYDBKHAAMRokDgMEoccAwsZjXCTCeUOKA\nQT74QMrP9zoFxhNKHDDIxYteJ8B4Q4kDBsnI8DoBxhtKHDDIJP7FwoFfCcAglDic+JUADEKJw4lf\nCcAglDic+JUADEKJw4lfCcAglDic+JUADGK/NzlvYQsbJQ4YxC7xq1e9zYHxgxIHDGJP4JQ4bJQ4\nYKDeXq8TYLxIWOK1tbXy+/0qKirqv23Lli0KBAIqLi7W6tWr1dXVNeohAQzEJA5bwhKvqalRJBIZ\ncNvy5ct14sQJHTt2TAUFBdqxY8eoBgRwjb1OYRKHLTPRFysqKhRzvHhxOBzu/3zp0qV65ZVXBv3Z\nurq6/s9DoZBCoZDrkAAGYhK/MUSjUUWj0ZSO4bOsxBcrxWIxVVZW6vjx49d9rbKyUtXV1VqzZs3A\ng/p8GuawAFxob5fy8qRz56SZM71Og3Rz052uT2xu27ZNkydPvq7AAYw+1imwJVynDOW5555TY2Oj\nDh06lO48AEaAdQpsSZd4JBLRrl27dPjwYd10002jkQnAEDixCaeE65Tq6mqVl5fr5MmTys3N1d69\ne/Xd735X3d3dCofDKi0t1aOPPjpWWQH8H5M4bMOe2HR1UE5sAqPiX/+Sbr9d+uc/+05w4sYypic2\nAYw9nnYPJ0ocMBA7cdgoccBATOKwUeKAQVinwIkSBwzEOgU2ShwwEJM4bJQ4YBCe7AMnShwwEJM4\nbJQ4YBAmcThR4oCBmMRho8QBAzGJw0aJAwbhOnE4UeKAgShx2ChxwECsU2CjxAGDsE6BEyUOGIhJ\nHDZKHDAQkzhslDhgEJ7sAydKHDAQkzhslDhgIEocNkocMAjrFDhR4oCBmMRhS1jitbW18vv9Kioq\n6r/t3LlzCofDKigo0PLly9XZ2TnqIQH0YRKHU8ISr6mpUSQSGXDbzp07FQ6H1dbWpmXLlmnnzp2j\nGhDA9ZjEYUtY4hUVFZo5c+aA2xoaGrRhwwZJ0oYNG/Tqq6+OXjoAg2IShy0z2R/o6OiQ3++XJPn9\nfnV0dAz6fXV1df2fh0IhhUIhVwEBXMPT7m8s0WhU0Wg0pWMkXeLxfD6ffD7foF+LL3EA6cUkfmNw\nDrhbt25N+hhJX53i9/v18ccfS5I++ugjzZ07N+k7BZAaJnHYki7xlStXqr6+XpJUX1+vVatWpT0U\ngMGxToFTwhKvrq5WeXm5Tp48qdzcXO3bt09PPvmkDh48qIKCAr3xxht68sknxyorgP9jnQJbwp34\niy++OOjtTU1NoxIGwMgwicPGMzYBg/BkHzhR4oCBmMRho8QBAzGJw0aJAwbh6hQ4UeKAgZjEYaPE\nAYMwicOJEgcMRInDRokDBmKdAhslDhiE68ThRIkDBrpyxesEGC8occBATOKwUeKAQex1CpM4bJQ4\nYCAmcdgoccBATOKwUeKAQbg6BU6UOGAgJnHYKHHAQEzisFHigEFYp8CJEgcMxDoFNkocMAiTOJwo\nccBATOKwUeKAgZjEYaPEAYPwtHs4uS7xHTt2qLCwUEVFRVqzZo0uXryYzlwAEmASh81VicdiMe3Z\ns0etra06fvy4ent7tX///nRnAzAEJnHYMt380PTp05WVlaWenh5lZGSop6dH8+bNS3c2AA5cnQIn\nVyU+a9YsPfHEE8rLy9OUKVP04IMP6oEHHhjwPXV1df2fh0IhhUKhVHICiMMkfmOIRqOKRqMpHcNn\nWfb/7SN36tQpVVZW6siRI5oxY4a+/vWvq6qqSmvXru07qM8nF4cFMIy335bKyqSKCukvf/E6DdLN\nTXe62okfPXpU5eXlmj17tjIzM7V69Wo1Nze7ORSAJLBOgZOrEl+wYIFaWlp04cIFWZalpqYmBYPB\ndGcDMATWKbC5KvHi4mKtX79eZWVlWrRokSTpW9/6VlqDARgakzhsrnbiwx6UnTgwKt56S7r3Xmnh\nQundd71Og3Qbs504AO9kZjKJ4xpKHDCIZfWVODtx2ChxwDBM4ohHiQOGychgEsc1lDhgEHudwiQO\nGyUOGIYSRzxKHDAM6xTEo8QBg7BOgRMlDhiGSwwRjxIHDMMkjniUOGAQnuwDJ0ocMAyTOOJR4oBh\nuDoF8ShxwCCW1VfiV69ee4MITGyUOGCYSZP6PlipQKLEAaPY03dGBiWOPpQ4YBifj5ObuIYSBwzE\nyU3YKHHAIPY6hWvFYaPEAcP4fFJWlnT5stdJMB5Q4oCBKHHYKHHAIPY6hRKHjRIHDMM6BfFcl3hn\nZ6eqqqoUCAQUDAbV0tKSzlwAEuDEJmyZbn/wscce04oVK/Tyyy/rypUr+uyzz9KZC8AgWKfAyVWJ\nd3V16ciRI6qvr+87SGamZsyYkdZgAAbHOgXxXJX46dOnNWfOHNXU1OjYsWNavHixdu/eralTp/Z/\nT11dXf/noVBIoVAo1azAhMckfmOJRqOKRqMpHcNnWcm/FtrRo0d17733qrm5WUuWLNHmzZs1ffp0\n/eQnP+k7qM8nF4cFMIzDh6Uf/ajvVQy3bZO+8hWvEyGd3HSnqxObOTk5ysnJ0ZIlSyRJVVVVam1t\ndXMoAC4wicPmqsSzs7OVm5urtrY2SVJTU5MKCwvTGgzA9VinwMn11SnPPvus1q5dq0uXLumuu+7S\nvn370pkLwBDsE5tcYggphRIvLi7WW2+9lc4sAEYoM5NJHH14xiZgENYpcKLEAcNwnTjiUeKAgShx\n2ChxwCCsU+BEiQOGYZ2CeJQ4YCBKHDZKHDAI77EJJ0ocMAzrFMSjxAGDcGITTpQ4YBgmccSjxAED\nUeKwUeKAQVinwIkSBwzDOgXxKHHAQFxiCBslDhiEdQqcKHHAMKxTEI8SBww0ebJ06ZLXKTAeUOKA\nQex1CiUOGyUOGMbnk77wBeniRa+TYDygxAEDUeKwUeKAQex1CiUOGyUOGIZ1CuJR4oBBmMTh5LrE\ne3t7VVpaqsrKynTmATAMJnHEc13iu3fvVjAYlM/nS2ceACPAJYawuSrxM2fOqLGxUZs2bZJlP74D\nMOpYp8Ap080PPf7449q1a5fOnz8/5PfU1dX1fx4KhRQKhdzcFQAH1ik3jmg0qmg0mtIxki7x119/\nXXPnzlVpaWnCO48vcQDpRYnfGJwD7tatW5M+RtLrlObmZjU0NCg/P1/V1dV64403tH79+qTvGEDy\nWKfAyWelsNQ+fPiwfvazn+m1114beFCfj105MAr+8Afp17+WGhr6Xsmwt7dvvYIbg5vuTPk6ca5O\nAcZeRoY0aRJvDAGXJzZt9913
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 240
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Loss Functions for Parameter Estimation\n",
|
||
|
|
"========================================\n",
|
||
|
|
"\n",
|
||
|
|
"Consider $p(\\theta|x)$ from the previous example again."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"plot(mus,pmu2)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 241,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x19e5a7d0>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD9CAYAAABDaefJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGdBJREFUeJzt3X10U/X9B/B3IKlou5WH8SBJsUdSSCpSqu1B3NAcATsQ\nIlB2rE8wVrFDq4DjMB/2Y2VTpMOnjvpQH1CYnNLjdAS3krmKUQbSwlBxwrQqkVCQoxVUZKw03t8f\n9zSnj2lumuSb773v1zk5NPSa+zanvPPtJzf3mhRFUUBERLrQT3QAIiKKHZY6EZGOsNSJiHSEpU5E\npCMsdSIiHWGpExHpSK+l7vV64XA4kJWVhfLy8i7f9/l8SE9PR25uLnJzc3H//ffHJSgREfXOHO6b\nwWAQpaWlqKurg9VqRX5+PtxuN5xOZ4ftrrzySmzdujWuQYmIqHdhV+oNDQ2w2+3IzMyExWJBUVER\nPB5Pl+34+SUiouQQdqXe1NSEjIyM0H2bzYb6+voO25hMJuzatQs5OTmwWq146KGHkJ2d3WUbIiLS\nTuuiOexKPZIyvuSSSxAIBPDee+/hjjvuwOzZs3sMJuvtt7/9rfAMRs0vc3bmF3+TPX80wpa61WpF\nIBAI3Q8EArDZbB22+cEPfoDzzjsPADB9+nScPXsWX331VVRhiIiob8KWel5eHhobG+H3+9HS0oKa\nmhq43e4O2xw/fjz0itLQ0ABFUTB48OD4JSYioh6FnambzWZUVlaioKAAwWAQxcXFcDqdqKqqAgCU\nlJTgz3/+M5588kmYzWacd9552Lx5c0KCJ5LL5RIdoU9kzi9zdoD5RZM9fzRMSrSDGy07MZming8R\nERlVNN3JT5QSEekIS52ISEdY6kREOsJSJyLSEZY6keR4DAK1x1Inktjf/w4UFopOQcmEpU4ksZMn\ngS1bgEOHRCehZMFSJ5KcogBPPy06BSULljqRxBQFmDABWL8eaGkRnYaSAUudSHJjxwLjxgGvvCI6\nCSUDljqRDvzyl8BTT4lOQcmApU4kMUUBTCbg2muBDz8EDh4UnYhEY6kT6UBKClBczNU6sdSJdGPR\nIuDFF4HTp0UnIZFY6kQSaxu/AMAFFwCTJgHV1WIzkVgsdSIdueMO4I9/5KkDjIylTiSxzuV99dXq\n8epvvikmD4nHUieSXNv4pe3rJUuAxx4Tl4fEYqkT6czNNwP//Cfw6aeik5AILHUiiXU3O09NVQ9v\nrKxMfB4Sj6VOJLn245c2t98ObNgAfPtt4vOQWCx1Ih0aNQq46irghRdEJ6FEY6kTSSzcoYtLlwLr\n1gHff5+4PCQeS51Ict2NXwDg8suB9HTg1VcTm4fEYqkT6ZTJBKxYAZSX88NIRsJSJ5JYb2U9dy7w\nxRfqIY5kDCx1Isn1NH4BgP79geXLgT/8IXF5SCyWOpHOzZ8P7N0L/PvfopNQIrDUiSQWyaz83HPV\nE32tXRv/PCSeWXQAIuqbcOOXNosXA6NHA4cPq8ewk35xpU5kAIMGAQsXAo8+KjoJxRtLnUhi7S+S\n0Ztly9RTB3zxRXwzkVgsdSKDsNmA664DHnlEdBKKJ5Y6kYHccw/w9NNAc7PoJBQvLHUiiWkZvwDq\nm6SFhVyt61mvpe71euFwOJCVlYXy8vIet9uzZw/MZjNeeeWVmAYkoti6917gqaeAr74SnYTiIWyp\nB4NBlJaWwuv14sCBA6iursbBgwe73e7Xv/41fvrTn0LhSSaIEiaaf26ZmcCcObzknV6FLfWGhgbY\n7XZkZmbCYrGgqKgIHo+ny3br1q3DvHnzMHTo0LgFJaLuaRm/tLn3XuCJJ4ATJ2Kfh8QK++GjpqYm\nZGRkhO7bbDbU19d32cbj8WD79u3Ys2cPTD38hJWVlYW+drlccLlc0acmoj658EJg1iygogJo90+T\nBPP5fPD5fH16jLCl3lNBt7d06VKsWbMGJpMJiqL0OH4p408OUcz1Zdp5333AZZcBd94JDB4cu0wU\nvc4L3lWrVml+jLClbrVaEQgEQvcDgQBsNluHbf71r3+hqKgIAPDll19i27ZtsFgscLvdmsMQkXbR\njF8AwG5XT81bXq7eSB/ClnpeXh4aGxvh9/sxcuRI1NTUoLq6usM2n376aejrhQsXYtasWSx0Ikms\nXAmMHw8sWQKMHCk6DcVC2DdKzWYzKisrUVBQgOzsbFx33XVwOp2oqqpCVVVVojISUQ/6erCZzQYU\nFwO//31s8pB4JiUBxyC2zduJKLaefRbYvVv9M1rNzcDYserj2O2xy0Z9F0138hOlRAY3ZIg6flm5\nUnQSigWWOpHEYvUL8LJlwPbtwLvvxubxSByWOpHkoj36pb20NPUDSffe2/fHIrFY6kQEACgpAT78\nEHj9ddFJqC9Y6kQSi+XxB+ecox6vftddQDAYu8elxGKpE0kuFuOXNoWFQHo68PzzsXtMSiyWOhGF\nmEzqudZXrgS+/VZ0GooGS51IYlovkhGJvDxg6lRgzZrYPi4lBkudiLpYvVq9kMZnn4lOQlqx1Imo\nC5sNKC1Vr2lKcmGpE0ksHuOXNitWADt2qDeSB0udiLqVmqq+aXr77UBrq+g0FCmWOpHE4n2evHnz\ngGHDgMcfj+9+KHZY6kSSi9f4pe2xKyuB++8Hjh2L334odljqRBSWw6Gec33FCtFJKBIsdSKJJeoy\nBb/5DfDmm8BbbyVmfxQ9ljqR5OI5fmmTlqa+aXrbbcDZs/HfH0WPpU5EESksBEaNAtauFZ2EwmGp\nE0kskVeJNJmAJ59UV+wffpi4/ZI2LHUiySVi/NLmggvUk33deivw/feJ2y9FjqVORJrcfjvwv/8B\nzzwjOgl1h6VOJLFEjl/a9O8PPPusekRMU1Pi90/hsdSJJJfI8UubceOAxYvVVbuIFxbqGUudiKJy\n333qG6YvvSQ6CbXHUieSmMhV8jnnAC+8ANxxB08hkExY6kSSEzF+aTNxIrBokXrjGCY5sNSJqE9W\nrlTfMF2/XnQSAljqRFKL50UyIpWSAmzcCNx9N+D3i81CLHUiioGLLwaWLwd+/nN+KEk0ljoRxcTy\n5erJvioqRCcxNrPoAEQUvWQYv7Tp318dw1x2GeByAbm5ohMZE1fqRBQzo0erK/WiIuDUKdFpjIml\nTiSxZDyM8IYbgMsvB+68U3QSY2KpE0kuWcYv7a1bB+zcCVRXi05iPCx1Ioq5tDRg82Z1tf7pp6LT\nGAtLnUhiyTh+aZObq54fpqhIPVUvJUavpe71euFwOJCVlYXy8vIu3/d4PMjJyUFubi4uvfRSbN++\nPS5Biah7yTh+abNkCWC1AnfdJTqJcZgUpefX+mAwiLFjx6Kurg5WqxX5+fmorq6G0+kMbfPdd98h\nNTUVAPD+++9jzpw5+PjjjzvuxGRCmN0QUZQefRQ4fFj9M1l9/TWQn6+eTuCmm0SnkUs03Rl2pd7Q\n0AC73Y7MzExYLBYUFRXB4/F02Kat0AHg1KlT+NGPfqQpABFFT4a1Uno68PLLwLJlwP79otPoX9gP\nHzU1NSEjIyN032azob6+vst2W7ZswT333INjx47htdde6/axysrKQl+7XC64XK7oEhNRB8k8fmlz\n8cXAY48BhYXAnj3AwIGiEyUnn88Hn8/Xp8cIW+qmCH9aZs+ejdmzZ2PHjh24+eab8WE3lxpvX+pE\nZDw33gi8/bZ6fphXXgH68TCNLjoveFetWqX5McI+rVarFYFAIHQ/EAjAZrP1uP3kyZPR2tqK5uZm\nzUGISDsZxi/tPfIIcPw48OCDopPoV9hSz8vLQ2NjI/x+P1paWlBTUwO3291hm08++SQ0yN+3bx8A\nYMiQIXGKS0SdyTB+aZOSos7Xn3wS+MtfRKfRp7DjF7PZjMrKShQUFCAYDKK4uBhOpxNVVVUAgJKS\nErz88svYuHEjLBYL0tLSsHnz5oQEJyI5jRypFvqMGcCFFwI5OaIT6UvYQxpjthMe0kgUFw89pF4f\n9OGHRSfRrqYGWLECaGgAhg8X
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 241
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Assume now that we are supposed to return a \"best estimate\" of the parameter.\n",
|
||
|
|
"\n",
|
||
|
|
"By itself, that isn't sufficient.\n",
|
||
|
|
"\n",
|
||
|
|
"But now assume that we are given a loss function: if our estimate is with in $\\pm 0.5$ of the true value,\n",
|
||
|
|
"we don't need to pay a penalty, otherwise, we need to pay a penalty of 1. What value should we return?\n",
|
||
|
|
"\n",
|
||
|
|
"The most likely value is 7, but no values less than 7 can occur.\n",
|
||
|
|
"\n",
|
||
|
|
"Therefore, it is better to return 7.5. That way, not only do we have the most likely value, but we also\n",
|
||
|
|
"get all the probability mass between 7 and 8 as well and our expected loss is about half of what it would be\n",
|
||
|
|
"if we had returned 7.\n",
|
||
|
|
"\n",
|
||
|
|
"Now assume we are penalized if we are outside the range $\\pm 1$ from the true value.\n",
|
||
|
|
"By the same reasoning, our parameter estimate should now be 8.\n",
|
||
|
|
"\n",
|
||
|
|
"As you can see from this simple example, there is not \"best\" answer to the parameter estimation\n",
|
||
|
|
"problem; our answer depends on the loss function.\n",
|
||
|
|
"\n",
|
||
|
|
"But we can see that for any symmetric loss function, the value 7 (the maximum likelihood estimate)\n",
|
||
|
|
"is never the optimal answer."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": []
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"metadata": {}
|
||
|
|
}
|
||
|
|
]
|
||
|
|
}
|