mirror of
https://github.com/saymrwulf/nvidia-AI.summerschool17-notebooks.git
synced 2026-11-21 09:54:43 +00:00
890 lines
197 KiB
Text
890 lines
197 KiB
Text
|
|
{
|
||
|
|
"metadata": {
|
||
|
|
"name": "25-mlp-posterior"
|
||
|
|
},
|
||
|
|
"nbformat": 3,
|
||
|
|
"nbformat_minor": 0,
|
||
|
|
"worksheets": [
|
||
|
|
{
|
||
|
|
"cells": [
|
||
|
|
{
|
||
|
|
"cell_type": "heading",
|
||
|
|
"level": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Least Square Approximations Yield Posteriors"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Significance)\n",
|
||
|
|
"\n",
|
||
|
|
"Our justification of MLPs has so far been mostly heuristic.\n",
|
||
|
|
"\n",
|
||
|
|
"However, there is a close connection with Bayesian classification.\n",
|
||
|
|
"\n",
|
||
|
|
"In particular, the MLP outputs approximate the posterior probability.\n",
|
||
|
|
"\n",
|
||
|
|
"Recall that classification with $\\arg\\max_\\omega P(\\omega|x)$ is optimal under 0-1 loss."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Unary Encoding of Classes)\n",
|
||
|
|
"\n",
|
||
|
|
"For classification problems, we encode classes using a *unary code*.\n",
|
||
|
|
"\n",
|
||
|
|
"That is, if the class is $i$, then the target vector is \n",
|
||
|
|
"\n",
|
||
|
|
"$$z_j = \\delta(i,j) = (0, ... , 0, 1, 0, ... , 0)$$\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Criterion Function)\n",
|
||
|
|
"\n",
|
||
|
|
"Let $\\tilde{z} = g(x;\\theta)$ be the predicted value of the MLP (or\n",
|
||
|
|
"classifier in general).\n",
|
||
|
|
"\n",
|
||
|
|
"The criterion function that we optimize is ($t$ is the target vector)\n",
|
||
|
|
"\n",
|
||
|
|
"$$ e = J(\\theta) = \\sum (g(x;\\theta) - t)^2$$\n",
|
||
|
|
"\n",
|
||
|
|
"minimizing this is equivalent to minimizing each term and each output unit."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(One Unit)\n",
|
||
|
|
"\n",
|
||
|
|
"Consider output unit $k$; we're minimizing\n",
|
||
|
|
"\n",
|
||
|
|
"$$ J_k(\\theta) = \\sum_x (g_k(x;\\theta) - t_k)^2 $$\n",
|
||
|
|
"\n",
|
||
|
|
"$$ ~~~~~~~~ = \\sum_{x\\in\\omega_k} (g_k(x;\\theta)-1)^2 + \\sum_{x\\not\\in\\omega_k} (g(x;w)-0)^2 $$"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Pulling Out Factors)\n",
|
||
|
|
"\n",
|
||
|
|
"$$ ~~~~~~~~ = \\sum_{x\\in\\omega_k} (g_k(x;\\theta)-1)^2 + \\sum_{x\\not\\in\\omega_k} (g(x;w)-0)^2 $$\n",
|
||
|
|
"\n",
|
||
|
|
"let's pull out some factors\n",
|
||
|
|
"\n",
|
||
|
|
"$$ ~~~~~~~~ = n \\left( \\frac{n_k}{n}\\frac{1}{n_k} \\sum_{x\\in\\omega_k} (g_k(x;\\theta)-1)^2 + \\frac{n-n_k}{n}\\frac{1}{n-n_k} \\sum_{x\\not\\in\\omega_k} (g_k(x;\\theta) - 0)^2 \\right)$$"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Limit)\n",
|
||
|
|
"\n",
|
||
|
|
"Let us now consider the limit as $n\\rightarrow\\infty$; then\n",
|
||
|
|
"\n",
|
||
|
|
"$$ \\frac{n_k}{n} \\rightarrow P(\\omega_k) $$\n",
|
||
|
|
"\n",
|
||
|
|
"$$ \\frac{n-n_k}{n} \\rightarrow P(\\omega^c_k) $$"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Sums to Integrals)\n",
|
||
|
|
"\n",
|
||
|
|
"In the limit, sums over random samples turn into integrals:\n",
|
||
|
|
"\n",
|
||
|
|
"$${\\frac{1}{N}\\sum_{x_i\\tilde\\; p(x_i)} f(x_i)} \\rightarrow {\\int f(x) p(x) dx} $$"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Sampled Sums to Integrals)\n",
|
||
|
|
"\n",
|
||
|
|
"The remaining sums converge to the corresponding integrals in the following equation:\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\lim_{n\\rightarrow\\infty} \\frac{1}{n} J_k(\\theta) $$\n",
|
||
|
|
"$$ ~~~~~~~~ = P(\\omega_k) \\int (g_k(x;\\theta)-1)^2 p(x|\\omega_k) dx + P(\\omega_k^c) \\int g_k^2(x;\\theta) p(x|\\omega_k^c) dx$$\n",
|
||
|
|
"...\n",
|
||
|
|
"$$ ~~~~~~~~ = \\int g_k^2(x;\\theta) p(x) dx - 2 \\int g_k(x;\\theta) p(x,\\omega_k) dx + \\int p(x,\\omega_k) dx $$\n",
|
||
|
|
"$$ ~~~~~~~~ = \\int [ g_k(x;\\theta)-P(\\omega_k|x) ]^2 p(x) dx + \\int P(\\omega_k|x) P(\\omega_k^c|x) p(x) dx $$\n",
|
||
|
|
"\n",
|
||
|
|
"the second term is independent of $\\theta$."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Term-wise minimization)\n",
|
||
|
|
"\n",
|
||
|
|
"Therefore, least square minimization minimizes\n",
|
||
|
|
"\n",
|
||
|
|
"$$ \\int [g_k(x;\\theta) - P(\\omega_k|x)]^2 p(x) dx $$\n",
|
||
|
|
"\n",
|
||
|
|
"This is minimized for each term,\n",
|
||
|
|
"therefore $g_k(x;\\theta) \\approx P(\\omega_k|x)$ asymptotically"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Notes)\n",
|
||
|
|
"\n",
|
||
|
|
"- this relationship is true only asymptotically\n",
|
||
|
|
"- the sum of the network outputs is not necessarily $1$\n",
|
||
|
|
"- if the sum of the network outputs differs substantially from $1$, that's an indication that the pattern is outside the domain over which the network predicts probabilities well\n",
|
||
|
|
"\n",
|
||
|
|
"An alternative approach is to use *softmax*. In softmax, the output transfer function\n",
|
||
|
|
"is replaced by $e^{By+b}$, and the actual output of the network is computed as the ratio of the output divided by the sum of all the outputs"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Simple Demonstration with Gaussians\n",
|
||
|
|
"===================================="
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"from pylab import *\n",
|
||
|
|
"from scipy.stats import norm"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 1
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Let's consider a very simple, 1D classification problem.\n",
|
||
|
|
"Our class-conditional densities $p(x|c)$ are normally\n",
|
||
|
|
"distributed with means 0 and 1.\n",
|
||
|
|
"We can calculate and plot the exact posterior directly."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"lxs = linspace(-4,5,1000)\n",
|
||
|
|
"ys0 = norm.pdf(lxs); ys0 /= amax(ys0)\n",
|
||
|
|
"plot(lxs,ys0)\n",
|
||
|
|
"ys1 = norm.pdf(lxs,loc=2); ys1 /= amax(ys1)\n",
|
||
|
|
"plot(lxs,ys1)\n",
|
||
|
|
"posterior = ys1/(ys0+ys1)\n",
|
||
|
|
"plot(lxs,ys1/(ys0+ys1))"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 2,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x2d24ed0>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD9CAYAAAChtfywAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XdcU+f+B/BPEBARxb0ARQEZKkNRFBcuwAHugtpqXfVa\nR/XaW9ve21b769L2duq1VqtWrYpaLYoMJypYReugisoQBVFRnMgQSJ7fH09BERICJHlOku/79cpL\nOJyc82kK35w85xkyxhgDIYQQvWUiOgAhhJDaoUJOCCF6jgo5IYToOSrkhBCi56iQE0KInqNCTggh\neq7KQj5t2jS0bNkSXbp0UbrP/Pnz4eTkBA8PD5w7d06jAQkhhKhWZSGfOnUqoqOjlf48MjISqamp\nSElJwU8//YTZs2drNCAhhBDVqizkffv2RePGjZX+fM+ePZgyZQoAwMfHB48ePUJ2drbmEhJCCFGp\n1m3kWVlZsLOzK/ve1tYWN2/erO1hCSGEqMlUEwd5eZS/TCarsE9l2wghhFStqplUan1FbmNjg8zM\nzLLvb968CRsbG6VhpPT46KOPhGfQh0wic926xdCzJ0NwMMPNm5VnUigYfv2VoXlzhjVrjPN1UigU\n+PTYp7D5rw0irkYozZR6PxX91/dH4OZAPC58LL3XqbgYLDUVLCYGbOVKsIULwcaMAevZE8zODszM\nDKxpUzB3dzB/f7CJE8HmzgX76COw774D27QJLDIS7I8/wJKSwDIywB48AHv2TNL//1Q91FHrK/Lg\n4GCsWLECoaGhOHnyJBo1aoSWLVvW9rCE4NYtwM8PmDgR+PBDwETJZYdMxvfp3h0ICAAePgT+9S+d\nRhWKMYZ/HfgXDqcfRsLMBLRp0Ebpvg5NHHBw8kHMjZyLgb8MxKHJh2BtYa3DtC+4fh1ITAQuXOD/\nJibyba1bAw4OgKMjf/TsCdjY8EerVoCFhZi8ElZlIZ8wYQKOHj2KnJwc2NnZYenSpSguLgYAzJo1\nC8OGDUNkZCQcHR1Rv359rF+/XuuhieHLzQX8/YGpU4H33lPvOU5OQFwc0Ls30KwZf64xWBa/DDFp\nMTj2+jE0rqe8Y0IpUxNTrBq+CvOi5iFoaxAOvHYAdU3rajdkYSFw5gwQH88fBw8CP/0EeHjwx+jR\nwEcf8f+JdbWcxRAxHdHhqdR25MgR0REqkGImxnSbSy5nbORIxmbNUr2fskxXrjDWogVjJ05oPltV\ndP3/7/fLv7O237RlWU+ylO6jLJNcIWejt41msyNmaz6YQsHY1auMffstY4GBjFlZMdatG2Pz5zMW\nFsaObN+u+XNqgBT//tSpnbK/d9Q6mUymdnsPMW5ffAHs3QscOQKYm9fsGOHhwFtvAWfPAk2aaDaf\nVNx4dAM91vbA7yG/o5ddrxod43HhY3Rf0x0f9v8Qr7q/WvtQFy8C27YB27cDeXnA0KFAYCAweDDQ\nqFHtj2+E1KmdVMiJpPz1FzBwIP8U3q5d7Y61cCGQlcVriqFRMAX6b+iP4I7B+Ffv2t0QSMxOxKCN\ng3Bu1jnYNrSt/gFycoD164GNG4FHj4DQUCAkBOjWjd/AILWiTu2kuVaIZBQXA6+/Dnz+ee2LOAB8\n9hlw7hy/ujc0a/5cA7lCjkW+i2p9LPeW7pjbfS5m75tdvYutU6eAyZN5u/alS8D//gfcuAF8+SXg\n7U1FXIeokBPJWLUKaNwYmD5dM8erVw/48Udgzhx+89RQ3Hl6Bx8c+QA/Bf0EE5lm/oTf6/serj+6\njh1JO1TvyBhv8xo4kF91e3gAqanAhg1A377KuxYRraKmFSIJOTmAqysQGwt06qTZY7/2Gr/C/+QT\nzR5XlGnh09C8fnMsG7xMo8c9ev0opvw+BVfmXoGFaSVd/I4f512I7t4F3n8fmDQJMDPTaAZSEbWR\nE73x5puAqSnw/feaP3ZmJuDpybspKxmrpjcu3r2Igb8MRMq8FK30/x4TNgY9bHrg3T7vPt+Yng4s\nXgycPAl8+invtF+njsbPTSpHhZzohWvX+GCe5GSgaVPtnKP0QvLnn7VzfF0ZsWUEBrUfhIW9Fmrl\n+KkPUtFzbU9cmXsFzcysgeXLga+/BhYsABYtAiwttXJeohwVcqIXpk0D7OyApUu1d47Hj/kgwfh4\noGNH7Z1Hm47dOMabPuZc0eoAnn9E/AOdbpVg3qo/+UjK1auBtm21dj6iGhVyInmpqXwEdmqq9rsZ\nL13KO1WsW6fd82jL4I2DMbHLREzzmqa9kygUePDx+5B/tRz1vlkBqxmzqfeJYOrUTo3MfkhITX3y\nCTBvnm7Gisybx3vK3bihme6NunQ66zSS7ydrZtCOMjk5wOTJaPL4Md75PgQW7e/gYyrieoH6ChFh\nMjOBPXv4CExdaNIEeOMN3uyrbz6P+xyLei2CeZ0aDnWtyvnzfABP585AbCz+MfpT/O/0//C48LF2\nzkc0igo5EWbFCmDKFN2O3F64ENi6ld/41BeX711GXEYcZnSdoZ0TREQAQ4YAX33F3+XMzNChcQf4\nO/hj3Tk9bYcyMlTIiRC5ubwHyfz5uj1vixbAuHF84j198e2pbzGn+xzUN6+v+YOvXMk/pkREAOPH\nl/vRgp4L8H3C95Ar5Jo/L9EoKuREiPXrgQEDgPbtdX/uefP4KNK/Z2OWtEeFj7D90nbM8p6l+YN/\n/jnwzTfAiROAj0+FH/ew6YHWVq0RfjVc8+cmGkWFnOicQsEH/izUTlfoKnXpAjg7A7/9Jub81bHh\n/AYMdRyKVlatNHdQxoAPPgA2bQKOHQPs7ZXuuqDnAnx78lvNnZtoBRVyonOHDgFWVkCvms28qhHz\n5wPffSfu/OpQMAVWnl6JOd3naPbAH37IZxI7ehRoo3w1IQAY4zoG1x5ew4U7FzSbgWgUFXKicz/9\nBMyaJbZ7clAQn+L2goTr04G0A7Ayt4Kvna/mDvrtt8COHcCBA0Dz5lXubmpiimle07D23FrNZSAa\nR4Wc6FR2Nl/la+JEsTnq1OFLwUl5yH7p1bhMU+94v/zC28T371eriJea7jUdW/7agoLiAs3kIBpH\nhZzo1Pr1wNixgLWg9X5fNHUqsGULX05Sam7n3sbxjOOY0HmCZg548CCf+ComptrD7ds1aofubbpj\nZ9JOzWQhGkeFnOiMQgGsWcN7u0mBvT3QtSuwe7foJBVtStyEMa5jNNPlMCWFTzkbFga4uNToEDO7\nzsSas2tqn4VoBRVyojOxsfwmZ/fuopM8N2MGsFZizb+MMWw4vwFTPafW/mCPHgHBwcD//R/Qv3+N\nDxPkHITk+8m4knOl9pmIxlEhJzqzeTMfySml6TtGjuTzlF+7JjrJcwlZCShWFKO3Xe/aHUihAF59\nlS98XMuPQeZ1zDHZYzI2nN9Qu0xEK6iQE53Iz+dNGBM01OSrKXXr8huvmzaJTvLc+vPr8brH67W/\nyfn118CDB/xfDXjV/VVs+WsLFEyhkeMRzaFCTnRizx6gRw+gdWvRSSqaNAn49Vc+Tka0guIC7Eja\ngckek2t3oFOn+CLIW7dqbDk295buaGTRCHEZcRo5HtEcKuREJzZv5p/ypah7d94K8eefopMAkSmR\n8GzlCTtru5of5NEjIDSULwih4fl6J3WZhM2JmzV6TFJ7VMiJ1t29C8TFAaNHi05SOZmMN69s2SI6\nCbD14tbadzmcNw8YPhwYNUozoV4wsctE/Hb5NzwreabxY5Oao0JOtC4sDBgxgvdYkaqJE4Ft2wC5\nwIn+njx7ggPXDmCM65iaHyQ8HPjjD2DZMs0Fe4GdtR26tOiCqNQorRyf1AwVcqJ1Um5WKeXiwtvv\nY2PFZdhzdQ/6teuHJvWa1OwA9+8Ds2fzUVf1tTDl7d+oeUV6qJATrbp+nXftGzxYdJKqiW5e2Xpx\nK0I7hdb8APPnA6+8AvTtq7lQlRjnNg4Hrh2g1YMkhAo50aqdO3nbuKkerA4bGsq7SIoYsn8//z7i\nMuIw0mVkzQ4QHQ2cPAl89plm
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 2
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Now we take 100 samples from each class conditional density.\n",
|
||
|
|
"These are the training samples for our classifier."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"samples0 = norm.rvs(size=100)\n",
|
||
|
|
"samples1 = norm.rvs(size=100,loc=2)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 3
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"_=hist([samples0,samples1])"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD9CAYAAACcJ53WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFHlJREFUeJzt3WtsU+fhx/Hf4VIxBs5FSwxaqILaReSG7Y011UQ2Z6mh\nHQIFmKqWlWYlbFOlvaCRpoK2aObFaLYOVYP1BZroGlSJrdPGZRONSLt6pVRr/oVkrVoE7daogZl0\nLCQ4XJSLz/8Fq7ssJr7E9skTvh/JknHO8fk1rn968vjxOZZt27YAAMaY5XQAAEBqKG4AMAzFDQCG\nobgBwDAUNwAYhuIGAMNMWtw3btxQTU2NvF6vKioqtGPHDklSMBhUSUmJfD6ffD6f2tvbcxIWACBZ\nidZxX7t2TfPnz9fo6KhWrlypn//853rllVe0cOFCNTc35yonAOA/Ek6VzJ8/X5I0PDyssbExFRQU\nSJL43g4AOCNhcUejUXm9XrndbtXV1amyslKStHfvXnk8HjU1NWlgYCDrQQEANyWcKvnE4OCgVq9e\nrdbWVlVUVKioqEiS1NLSonA4rP37949/YsvKfFoAuA0kquWkV5Xk5eVpzZo1euutt1RcXCzLsmRZ\nlrZu3arOzs5bHny633784x87noGc5CQnGT+5JWPS4r506VJsGuT69evq6OiQz+fTxYsXY9scOnRI\n1dXVyfY/AGCK5kz2w3A4rMbGRkWjUUWjUW3evFn19fV69NFH1d3dLcuytHTpUu3bty9XeQHgtjdp\ncVdXV+v06dMTHj9w4EDWAuWa3+93OkJSyJlZ5MwsE3KakDFZSX84mfITW1bS8zUAgJuS6U6+8g4A\nhqG4AcAwFDcAGIbiBgDDUNwAYBiKGwAMQ3EDhnPlu2KnoEj25sp3OR0bU8A6bsBwlmVJwRR3CnJq\n5umKddwAMANR3ABgGIobAAxDcQOAYShuADAMxQ0AhqG4AcAwFDcAGIbiBgDDUNwAYBiKGwAMQ3ED\ngGEobgAwDMUNAIaZtLhv3Lihmpoaeb1eVVRUaMeOHZKk/v5+BQIBlZWVadWqVRoYGMhJWABAguKe\nN2+eXn31VXV3d+vtt9/Wq6++qtdff12tra0KBAI6d+6c6uvr1dramqu8AHDbSzhVMn/+fEnS8PCw\nxsbGVFBQoKNHj6qxsVGS1NjYqMOHD2c3JQAgZk6iDaLRqL74xS/q73//ux5//HFVVlaqr69Pbrdb\nkuR2u9XX1xd332AwGLvv9/vl9/szEhq543IVKhK5nPT2CxcW6MqV/iwmAmaWUCikUCiU0j5JX7ps\ncHBQq1ev1lNPPaUNGzbo8uVP38yFhYXq7x//ZuXSZTODZVmSUnkded1zjUuXzSwZvXRZXl6e1qxZ\no1OnTsntduvixYuSpHA4rOLi4qklBQAkbdLivnTpUmzFyPXr19XR0SGfz6d169apra1NktTW1qaG\nhobsJwUASEowxx0Oh9XY2KhoNKpoNKrNmzervr5ePp9PDz74oPbv36/S0lK9+OKLucoLALe9pOe4\nU35i5rhnBOa4pz/muGeWjM5xAwCmB4obAAxDcQOAYShuADAMxQ0AhqG4AcAwFDcAGIbiBgDDUNwA\nYBiKGwAMQ3EDgGEobgAwDMUNAIahuAHAMBQ3ABiG4gYAw1DcAGAYihsADENxA4BhKG4AMAzFDWSA\nK98ly7JSurnyXU7HhqHmOB0AmAkig5GUr7QeCUaykgUzHyNuADDMpMXd29ururo6VVZWqqqqSnv2\n7JEkBYNBlZSUyOfzyefzqb29PSdhAQAJpkrmzp2rZ555Rl6vV0NDQ/rSl76kQCAgy7LU3Nys5ubm\nXOUEAPzHpMW9aNEiLVq0SJK0YMEClZeX68KFC5Ik27aznw4AMEHSH0729PSoq6tL9957r06ePKm9\ne/fqwIEDWrFihXbv3q38/PwJ+wSDwdh9v98vv9+ficwAMGOEQiGFQqGU9rHsJIbOQ0ND8vv9+tGP\nfqSGhgZ9/PHHKioqkiS1tLQoHA5r//7945/YshiVzwCWZUlK5XW8PV93y7JSXlWiYGb+cnXy2Mi8\nZLoz4aqSkZERbdy4UY888ogaGhokScXFxbG1qFu3blVnZ2dmEgMAEpq0uG3bVlNTkyoqKrRt27bY\n4+FwOHb/0KFDqq6uzl5CAMA4k85xnzx5Ui+88IKWL18un88nSdq1a5cOHjyo7u5uWZalpUuXat++\nfTkJCwBIUNwrV65UNBqd8PgDDzyQtUAAgMnxzUkAMAzFDQCGobgBwDAUNwAYhuIGAMNQ3ABgGIob\nAAxDcQOAYShuADAMxQ0AhqG4AcAwFDcAGIbiBgDDUNwAYBiKG9OWy1UYu9JSMjeXq9DpyEBOJH2x\nYCDXIpHLSuV6l5GIlb0wwDTCiBsADENxA4BhKG4AMAzFDQCGobgBwDAUNwAYhuIGAMNMWty9vb2q\nq6tTZWWlqqqqtGfPHklSf3+/AoGAysrKtGrVKg0MDOQkLAAgQXHPnTtXzzzzjN5991399a9/1bPP\nPqszZ86otbVVgUBA586dU319vVpbW3OVFwBue5MW96JFi+T1eiVJCxYsUHl5uS5cuKCjR4+qsbFR\nktTY2KjDhw9nPykAQFIKX3nv6elRV1eXampq1NfXJ7fbLUlyu93q6+uLu08wGIzd9/v98vv9UwoL\nADNNKBRSKBRKaR/Ltu2EJ4MYGhrS1772NbW0tKihoUEFBQW6fPly7OeFhYXq7+8f/8SWpSSeGtOc\nZVlK5XwhUuZedyePnSrLsqRgijsFlZG8Th4bmZdMdyZcVTIyMqKNGzdq8+bNamhokHRzlH3x4kVJ\nUjgcVnFxcQbiAgCSMWlx27atpqYmVVRUaNu2bbHH161bp7a2NklSW1tbrNABANk36VTJ66+/rq9+\n9atavnz5f/5slZ566indc889evDBB/XRRx+ptLRUL774ovLz88c/MVMlMwJTJUkemakSZEgy3Tnp\nh5MrV65UNBqN+7OXX345/WQAgLTxzUkAMAzFbYBUL+HFZbyAmY1Llxkg1Ut43dyHy3gBMxUjbgAw\nDMUNAIahuAHAMBQ3ABiG4gYAw1DcAGAYihsADENxA4BhKG4AMAzFDQCGobgBwDAUNwAYhuIGAMNQ\n3ABgGIobAAxDcQOAYShuADAMxQ0gba58V2qX1Mt3OR15RuDSZQDSFhmMSMEUtg9GspbldsKIGwAM\nk7C4t2zZIrfbrerq6thjwWBQJSUl8vl88vl8am9vz2pIAMCnEhb3Y489NqGYLctSc3Ozurq61NXV\npfvvvz9rAQEA4yUs7traWhUUFEx43LbtrAQCAEwu7Q8n9+7dqwMHDmjFihXavXu38vPzJ2wTDAZj\n9/1+v/x+f7qHA4AZKRQKKRQKpbSPZScxdO7p6dHatWv1zjvvSJI+/vhjFRUVSZJaWloUDoe1f//+\n8U9sWYzKM8SyLEmp/i4z8/tP/diZe92dPHaqLMtKaXWFJCmYmb9cjTp2ho47kyXTnWmtKikuLo6t\ny9y6das6OzvTCggASF1axR0Oh2P3Dx06NG7FCQAguxLOcT/88MP6y1/+okuXLmnJkiXauXOnQqGQ\nuru7ZVmWli5dqn379uUiKwBASRT3wYMHJzy2ZcuWrIQBACTGNycBwDAUNwAYhuIGAMNQ3ABgGIob\nAAxDcQOAYShuADAMxQ0AhqG4AcAwFDcAGIbiBgDDUNwAYBiKGwAMQ3EDgGEobgAwDMUNAIahuAHA\nMBQ3ABiG4gYAw1DcAGAYihszxyzJsqyUbq58l9OpgZQlvMo7YIyopGBqu0SCkWwkAbKKETcAGCZh\ncW/ZskVut1vV1dWxx/r7+xUIBFRWVqZVq1ZpYGAgqyEBAJ9KWNyPPfaY2tvbxz3W2tqqQCCgc+fO\nqb6+Xq2trVkLCAAYL2Fx19bWqqCgYNxjR48eVWNjoySpsbFRhw8fzk46AMAEaX042dfXJ7fbLUly\nu93q6+uLu10wGIzd9/v98vv9
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 4
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Posteriors by Histogramming\n",
|
||
|
|
"----------------------------\n",
|
||
|
|
"\n",
|
||
|
|
"We can compute the posteriors by binning and histogramming\n",
|
||
|
|
"(this has nothing to do with the proof above, but it\n",
|
||
|
|
"just is an illustration of the computation of posteriors).\n",
|
||
|
|
"In the 1D case, this actually often works quite well."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"xs = []\n",
|
||
|
|
"ys = []\n",
|
||
|
|
"for x in range(-5,5):\n",
|
||
|
|
" y0 = sum(floor(samples0)==x)\n",
|
||
|
|
" y1 = sum(floor(samples1)==x)\n",
|
||
|
|
" xs.append(x+0.5)\n",
|
||
|
|
" ys.append(y1*1.0/max(1.0,y0+y1))\n",
|
||
|
|
"plot(xs,ys)\n",
|
||
|
|
"plot(lxs,posterior,color='r')"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 5,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x2c2dfd0>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD9CAYAAAChtfywAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XtcVWW+x/EPCmZpec8SKFRQsLwVpo5ToakwljR5i2oa\nR83IM9VxOk2ONTOCk9eZOuOJmbIzaSc1pEwlDckbu8xR0TS1NAKVAnQsvOQtBTbr/LGMAYQNyt57\n7cv3/Xrxgr3Xs+G3X8G3x2c9lwDDMAxERMRrNbK6ABERaRgFuYiIl1OQi4h4OQW5iIiXU5CLiHg5\nBbmIiJerM8jHjx9P+/bt6d69e61tnn76aSIiIujZsye7du1yaoEiIuJYnUE+btw4MjMza72ekZFB\nXl4eubm5vP7660yaNMmpBYqIiGN1Bvmdd95Jq1atar3+/vvvM3bsWAD69u3LyZMnOXr0qPMqFBER\nhxo8Rl5UVERoaGjF45CQEAoLCxv6bUVEpJ4CnfFNqq/yDwgIuKRNTc+JiEjd6tpJpcFBHhwcTEFB\nQcXjwsJCgoODr6gYb5aUlERSUpLVZbiML78/X35v4AHvr7wccnPh009hzx746ivO782Fgwc407gl\n13S7mWvCg6FDB/MjOBhuuAFat4ZWrcyPFi2gUc0DCJa/PxerTye4wUEeHx9PSkoKCQkJbN26lZYt\nW9K+ffuGflsR8VbnzsE//wkbN8Inn8Bnn0HbtnDbbZR268kyI4HXi7vwi5fDGfdU89ryWS5DnUH+\n0EMP8dFHH1FcXExoaCjJycmUlpYCkJiYyLBhw8jIyCA8PJxmzZqxcOFClxctIh6mqAhWrIDlyyE7\nG3r2hEGD4A9/gNtvh9at2bABEhPhttvg7X1w441WF+076gzy1NTUOr9JSkqKU4rxZjExMVaX4FK+\n/P58+b2BC9/fmTOQlgZvvAE5OXDfffCf/wn33APNm1c0Ky6G/xoLNhv87W9mM2fy9f9+9RHgrv3I\nAwICfHqMXMRvHDgAL70ES5fCXXfBY4/B0KHQpEmVZoYBixfDb38LDz0Ef/pTlXyXeqpPdjpl1oqI\n+IGcHJgxAzIyYNIk2LvXvDFZg7w8eOIJOHYMVq+G6Gg31+pndJtBRBw7fhyefhp++lPo2tXskf/p\nTzWGeGkpzJ4N/fpBXBxs364Qdwf1yEWkZj+OjTz7LIwYAfv3m7NParF1Kzz+uJnv27dDx45urNXP\nKchF5FLffmuOjeTmwpo15lSTWpw6BS+8AMuWwcsvQ0ICaP2fe2loRUSq2rwZeveGiAjYscNhiK9c\nCbfcAj/8AF98Yd7UVIi7n3rkIvJvr70G06bBm2/Cz35Wa7OiInjqKTO8Fy0CzQC0lnrkImKOh//u\ndzBvntkjryXE7XZzLnjPntC9O+zerRD3BOqRi/i7sjJzPPzzz80l9W3a1Nhs717zZmbjxvDxx9Ct\nm5vrlFqpRy7iz8rLYcIEyM+H9etrDPEffoDnnzdX3P/qVwpxT6QeuYi/MgxzfvihQ5CZCddcc0mT\nyvuj7Nmj/VE8lYJcxF8lJ5uTvzdsuCTET540t01x1f4o4lwKchF/lJZmzkzJzjb3+q7mt781V2l+\n8YX2R/EG2jRLxN98+qm5fn79enP6STWHDpnL6nNzzbMdxFr1yU7d7BTxJydOmMvtX3utxhAHePFF\n+I//UIh7E/XIRfyFYcDo0RASAn/9a41NDhyAvn3N3nirVm6uT2qkbWxF5N/eeMNM6MWLa23y4ovw\n618rxL2NeuQi/uDHrraDSeB5eeb2s3l50LKlm+uTWmmMXETMIZUnnoApUxyu5PnTn8z9UxTi3kdD\nKyK+btEi8+DM3/ym1iZffWUe/JOX58a6xGk0tCLiy777Dm691Uzp22+vtdmjj0KXLuah9+JZ6pOd\nCnIRX/bEE3DVVeauhrXIyTFPcTtwAK67zo21Sb1o1oqIP/viC1i+HL780mGz6dNh8mSFuDdTj1zE\nVw0bBkOHmildi/374e67zbFxBblnUo9cxF+tXWvOGV+50mGz6dPNe6AKce+mHrmIrzEM6NMHpk6F\nkSNrbfbFFzBwoDk2fu21bqxPLovmkYv4ow8+MLcufOABh82Sk+HZZxXivkA9chFfUs/e+N69MGSI\n2Rtv1syN9cllU49cxN9kZEBJSb174wpx36AeuYivMAxzP5XnnoNRo2pttnu3uR15Xp6C3BuoRy7i\nTzZtMs9oGzHCYbPkZPMEIIW479D0QxFf8fLL5lzCRrX3zz77zDym08FOtuKFNLQi4gtyc+EnP4H8\nfIdd7Z//HGJiHK4REg+jBUEi/mLePHj8cYchvnMnbN8OqalurEvcQj1yEW934gR07gyffw4dOtTa\nLD4eBg+Gp592Y23SYOqRi/iDt94yp6E4CPEdO8we+TvvuLEucZs6Z61kZmYSGRlJREQEc+bMueR6\ncXExcXFx9OrVi1tvvZU333zTFXWKSE0MA+bPh8REh82SkuB3v4OmTd1TlriXw6EVu91O165dWb9+\nPcHBwfTp04fU1FSioqIq2iQlJXHhwgVmzZpFcXExXbt25ejRowQGVu3sa2hFxAU++QQee8zcxjAg\noMYm2dnmIs/cXAW5N2rwPPLs7GzCw8MJCwsjKCiIhIQE0tPTq7S58cYbOXXqFACnTp2iTZs2l4S4\niLjI/PnmTc5aQhzM3vjUqQpxX+YwcYuKiggNDa14HBISwrZt26q0mThxIoMGDaJDhw6cPn2adxwM\nwiUlJVV8HRMTQ0xMzJVVLSJw7BisWgV//WutTbZuNe+BrljhxrqkQWw2Gzab7bJe4zDIAxz8X/5H\nM2fOpFevXthsNg4cOMCQIUPYvXs319awpVrlIBeRBlqyBO69F9q0qbVJUhI8/7x52pt4h+qd3OTk\n5Dpf43BoJTg4mIKCgorHBQUFhISEVGnzz3/+k9GjRwPQuXNnOnbsSE5OzuXULSJX4q23YOzYWi9v\n2WIOnY8f78aaxBIOgzw6Oprc3Fzy8/MpKSkhLS2N+Pj4Km0iIyNZv349AEePHiUnJ4dOnTq5rmIR\nMRP6yBG4555am0ybBi+8AE2auLEusYTDoZXAwEBSUlKIjY3FbrczYcIEoqKimD9/PgCJiYk8//zz\njBs3jp49e1JeXs7cuXNp3bq1W4oX8VuLFsHDD0PjxjVe3rzZnKXyq1+5tyyxhlZ2inib8nLo2NG8\n0dmjR41NBg+GhARzZqJ4N21jK+KLPv4YWrasNcQ3bYKDBx0On4uPUZCLeJtFi+DRR2u9PG0a/P73\nEBTkxprEUhpaEfEmFy7AjTfWukHWRx+Zs1S+/FJB7is0tCLia9atg+7da90ga9o0+MMfFOL+RkEu\n4k2WLav1PM6sLCgqgl/8ws01ieU0tCLiLUpKzGGVPXsgOLjKJcOAu++GiRMdDp+LF9LQiogv2bgR\nIiMvCXGADRvg6FF46CEL6hLLKchFvEUtwyqGYY6N//GPoI1H/ZOCXMQblJbCypXmxuLVrFsHx4+b\nC4DEPynIRbyBzQbh4XDTTVWertwbr2W1vvgBBbmIN1i2rMbe+Icfwvffw5gxFtQkHkOzVkQ8XXk5\nhISYS/PDwyueNgzo1w+eeQYefNDC+sSlNGtFxBfs3AnXXVclxAHWrIGzZ+HicQDixxTkIp5u1SoY\nPrzKUz+OjU+bBo30V+z39Csg4ulqCPIPPjC3Xalh2Fz8kMbIRTxZUZG5Xe3RoxWTxA0D+vSBqVMV\n5P5AY+Qi3m71aoiLq7LSZ9Uqc1r5Aw9YWJd4FAW5iCerNqxiGJCUZH5obFx+pF8FEU917pw55TAu\nruKp9HQzzH/+cwvrEo+jnRlEPNX69RAdbR7rhjmdPCkJkpMhIMDa0sSzqEcu4qlWr4b77qt4uHKl\nuQw/Pt7CmsQjadaKiCcyDHNflXXrIDKS8nLo1QtmzLhkJqL4OM1aEfFW+/ebdzO7dgUgI8M8vq1S\nB12kgoJcxBOtXQuxsRWD4W+/
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 5
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Least Square Fit with a Polynomial\n",
|
||
|
|
"----------------------------------\n",
|
||
|
|
"\n",
|
||
|
|
"Let's try a simple least square fit, with a polynomial.\n",
|
||
|
|
"Before we do that, let's look at the function we're actually\n",
|
||
|
|
"trying to approximate.\n",
|
||
|
|
"From a least square fitting point of view, this function\n",
|
||
|
|
"looks rather odd."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"data = c_[r_[samples0,samples1],r_[zeros(len(samples0)),ones(len(samples1))]]\n",
|
||
|
|
"order = argsort(data[:,0])\n",
|
||
|
|
"s = data[order,:]\n",
|
||
|
|
"ylim(-0.1,1.1)\n",
|
||
|
|
"plot(s[:,0],s[:,1])"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 9,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x3269210>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD5CAYAAAA6JL6mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnX9wXWWdxp/bJqUIWMBCLLlhIiTdpLQk1WBksG6KdNpU\nG4HiGrsOWGLJdOywsOKAOq6tuzIWxj+EOmMBKSJaqyNOwEmvUMYrFaYNQqEz2y4bhMhNdLtTsba0\ndtPcnv3jzpu+OX3POe/5fd/T5zPT6f1x7jnfJO1znjzv933fnGVZFgghhBjLtLQLIIQQEg4KOSGE\nGA6FnBBCDIdCTgghhkMhJ4QQw6GQE0KI4dQkdaFcLpfUpQghJFN4dYkn6sgty6r6P9/4xjdSr4F1\nskbWyTrFHx0YrRBCiOFQyAkhxHAo5Da6urrSLkEL1hkdJtQIsM6oMaVOHXKWbggT9kK5nHbeQwgh\npIKOdtKRE0KI4VDICSHEcCjkhBBiOBRyQggxHAo5IYQYDoWcEEIMh0JOCCGGQyEnhBDDoZATQojh\neAr5rbfeirq6OixYsMDxmNtvvx3Nzc1oa2vDnj17Ii2QEEKIO55Cvnr1ahQKBcf3BwcH8cYbb2B4\neBgPPfQQ1q5dG2mBhBBC3PHcWGLRokUYGRlxfP+pp57CLbfcAgDo7OzEoUOHcODAAdTV1UVWJAnO\nH/8I1NUBM2fG+5mwvP02cNFFwNlnA++8A5TLledBePllYM4c4P3vB559Fhgfr7w+cybQ3g7s2jX1\n+FwOWLwYOOcc4PBh4He/qzwvlYCmJmDHDuDqq4F33wVOnAD+67+As86qHGs/z7XXAnv2VL4GFVdf\nDcyYARw9CsyeXamvXK68N3cucMklQLFYef6hD1XOVVMDXHZZ5bq6nHMO8H//B0xM6B0/Zw5w6aVA\nbS1wwQWV1156Cfif/9G/JgnGhz9c+f8WhtA7BI2NjaGhoWHyeT6fx+joqFLI169fP/m4q6srU6uP\nVSv/8i9AXx+wYoW/z3zhC8AnPxlfXXb+9V+Bf/5n4IYbgM2bgWPHgH//92Dn6uioCPH3vw+sXFl5\nDFRE87bbgO3bgZaWU8f//vfAgw8CN90EPPkksHo1MDhY+T5873vA0qXAE08An/vc1Ov84z8C5513\n6vnQEPDII8CnPw18/OPANNvvu/v3A2vWVIRyzx7g5psrX/NHPwr87/9WhP0znwG+9rWKeP/DP1SO\nO3q0IuS1tRWh1+FXv6qIufja3fj734H//m9g1apKDXfdVXn9+uuBefOSvaGfidTVTRXyYrGIorib\naxLJVm/2lbmctnWThZwkw/j4Kceny4kT/j8TlvHxU+6xXD7looMi3OillwJPP115rb6+ct7PfAb4\nj/84dew//RNw8mTlsajh5MnK9+H48VM12fn+96feEK6/vnJcuQz88pcV5y3zla+cOu/EROW4efMq\n9W3fDjzwQOW9JUuA974X2LsX6OwE3nij8vq//VvlZqNDLle5EYiv3Y1SCfjIRyo1yQ7+xAngxz8G\nLr5Y75okGuwmd8OGDZ6fCd21Ul9fj1KpNPl8dHQU9fX1YU9LImJiAvC7evDJk/4/E5Zy+ZSYWlb4\nG4llVc4ne4pp07y/LvG++LyoSaeeXK7yOcuael074mtVHXfyJDB9+qkacrlT5/WL7ja58vdFvs7E\nROU3A1L9hBbynp4ePP744wCAXbt24fzzz2c+XkWUy+YIuXxNIaBBEV+DHG/kcpXruAmck5Dr1DNt\nmt5xbkJeLp+q+eTJU4/jFnJRj1x/uXzqpkKqG8/77Wc/+1n89re/xcGDB9HQ0IANGzbgxIkTAID+\n/n4sX74cg4ODaGpqwjnnnIMtW7bEXjTR50x15EKEZSHXEVr5fb9Cnss5C7RMtTly+TcJOnIz8fwx\nbd261fMkmzZtiqQYEj1BBDGIiw+LLOTieRiECMtiJoTWjyMXdeg6cq9oRXb6uo5cCK2uMAv8OnJx\nTcHEBB25KXBmZ8YxJVqRf3OIypGrohUnQVZlxEGjlaCOXIj89OlTbzppOfJymY7cFCjkGcfEaEU8\nD4NTtKISWfm5HO8EiVbcvm/iOiohl9+LKiPXRb4B2W+mdORmQCHPOKY48rgy8qDRijiHqEOnHjmi\niCIjtzvyuKIV+QYk37hErEOqHwp5xjHJkdt/rQ+DKlrRGewMk5HL0Y2bAIqbjOqmIjtyUX8S7Yd2\nR86BTrOgkGccUx15FO2H9mhFx5H76Vqxn2faNL0bUFBHHhcqR85YxSwo5BkniLNNW8jF8zC4Cbkb\ncfeRi9jIq2tF1Crn5X6hIz9zoJBnHBOjlahmdtqFUjhmlcBF0bWi4/gB764VWbyFIxfn90OQrhU6\ncjOhkGccE6MV8TwMfhx5VF0rbjcK+TpejlyeEJRkRi6uCdCRmwaFPOOY5Mij6FqRs17d9kPV58U5\n/HSt6Dpy1YQg8bfdkcsxi1/COHJOBjILCnnGMcWR2284QYVcdtBO7YduhM3IvdZyAfQdeVLth6qM\nnJOBzIJCnnFMdOTiedDzAO4zO/10rfidoq9znFfXimqwM4mZneL6AB25aVDIM46JXSth2g9l4VVF\nK3E6ct0JR34y8qSWsaUjNxsKecYJEq2ktWhWFNGKXch1Z3bau1ZUMx3tOPWRB8nI5feSXjRLtdYK\nBzvNgkKecUyMVsIMdrpFK8J56qy1Yh/kjMKRe3Wt2Nc3CevIdZFXbWT7oZlQyDNO0MHOpIkqI7cP\ndgadEGR34rprrQSd2anqWrFP1/eLH0curkFHbiYU8oxjkiOPYkJQ0GhFYBfuIDM7dbpWVBOWxHWS\nXsZWHCvfTDnYaRYU8oxjSvth1F0rQiiDDnbKG0ED0c7sdMvI3RbNiisjB6YOeIo66MjNgUKeYYIK\nctpCHqUj99t+6BStRNVH7tW1Ihy5eJykI5e7dOjIzYJCnmFkd+qHtCYE2Xu4g+A2IcjNkatW/5P/\n9uPIvfC7+XISQk5HbjYU8gxjkpDHkZHrTgiyd46Iz6v+dsMrI9fpWrE7cjlmiTNaUTlyCrk5UMgz\njMh5q13I7ZNvgHijFa9a7OdxqkfVRx40I5dF3p6Lp+XIGa2YA4U8w5jiyO2uN66uFR2hdWo71I1W\nwq61onLkcfeRA3TkpkMhzzCmOHLVDSettVacHHlUg53i3LpdK8KRBxkzoCM/c6CQZ5gwk2rSEPIo\nHLnbhCAxg9GNMBm5bnQjC/k02//AKDNyP6j6yOnIzYFCnmFMc+RxZOR+11oJ07USJiOX65dXPxSO\nPAhhHDnbD82CQp5hgmbkSS+aZa8zrtUPnRxzlF0rYXYIEl93Gu2H4nvD9kMzoZBnGFMGO+Nw5E4z\nO3Um7MjniWOHILe1VpwWzdI5r6oeXYQj54QgM6GQZxhTohVRZ5RdK+J8ftsPnZy4H0euU6Nb1wod\nOfGLp5AXCgW0tLSgubkZGzduPO39gwcPYtmyZWhvb8f8+fPx2GOPxVEnCYBpjjyKrhX77FC/7Ydh\nulZ0u2K8MnL7FH25Lj+EdeQUcnNwFfJyuYx169ahUChg37592Lp1K/bv3z/lmE2bNmHhwoV49dVX\nUSwW8aUvfQkTwmKRVAn6Y8hCtAI4tx+64TTo6ZWtA/5WP/Ry5KLWJB25PDbC9kOzcBXyoaEhNDU1\nobGxEbW1tejt7cXAwMCUY+bMmYPDhw8DAA4fPoz3ve99qOGtvCowzZEnEa04tfF5CXiSfeRJb74s\naqcjNxfXH9XY2BgaGhomn+fzeezevXvKMWvWrMG1116LSy65BEeOHMHPfvYzx/OtX79+8nFXVxe6\nurqCVU20CCLkYpDQ1GjFLuQ6i2ZF1bXi5fi9ulbEdZyWsfVL2IycjjwdisUiisWir8+4CnlO41/C\nvffei/b2dhSLRfzhD3/AkiVL8Nprr+G888477VhZyEn8BBnslFsAk0LlyIO2H8qfS2NmZ5i1VrwW\nzYoTOvLqwW5yN2zY4PkZ12il
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 9
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"We can use the function `polyfit` to perform polynomial fitting."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"fit = polyfit(data[:,0],data[:,1],5)\n",
|
||
|
|
"plot(lxs,polyval(fit,lxs))\n",
|
||
|
|
"plot(lxs,posterior,color='r')"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 10,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x2e35c50>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD9CAYAAACyYrxEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VdW99/HPyQSGhEmmNAlEBUyiIQmC6VWDByTkYiCC\nbQWsVwtKUxla1Fa0thWsTA6veyv4+NRWEGsfQPFiqIYw6RFEQ5RZQIiVSEaMQIAQIMnJfv7YEsAM\nnJyTcIZ836/XfuUM6+z1A/R7Vtbee22LYRgGIiLis/zcXYCIiLQuBb2IiI9T0IuI+DgFvYiIj1PQ\ni4j4OAW9iIiPcznoJ02aRM+ePYmLi2vwfZvNRqdOnUhMTCQxMZFnn33W1S5FRKQZAlzdwcSJE5k+\nfTr3339/o21uv/12Vq9e7WpXIiLiBJdH9MnJyXTp0qXJNromS0TEfVp9jt5isfDJJ58QHx/PnXfe\nyb59+1q7SxERuYjLUzeXM3DgQAoKCggODmbNmjWMGTOGgwcP1mtnsVhauxQREZ90uVmTVh/Rh4aG\nEhwcDMDIkSOprq7m2LFjDbY1DMOjtqefftrtNagm36pLNammlt4c0epBf+TIkbpicnNzMQyDrl27\ntna3IiLyPZenbiZMmMBHH33Ed999R2RkJLNnz6a6uhqAjIwMVq5cySuvvEJAQADBwcEsX77c5aJF\nRMRxLgf9smXLmnx/6tSpTJ061dVu3MJqtbq7hHpUk+M8sS7V5BjV1LIshqOTPK3MYrE4PN8kIiIm\nR7JTSyCIiPg4Bb2IiI9T0IuI+DgFvYiIl3J0oQEFvYiIF8rKghEjHGuroBcR8TJffQW/+AWsWOFY\newW9iIgXOX0axo6FWbPg1lsd+4zOoxcR8RI1NTBmDPTqBX/7G1gsOo9eRMRnGAZkZEBtLbzyihny\njmr1ZYpFRMQ1hgFPPQV79sAHH0BgYPM+r6AXEfFghgG//z28/z5s3AghIc3fh4JeRMRDGQY8/jhs\n2GCO5Lt1c24/CnoREQ907hw8+CDk5ZkjeVdu46GDsSIiHqasDIYPh7Nn4cMPXQt5UNCLiHiUjz6C\ngQMhORneegu+vxOrS1wO+kmTJtGzZ0/i4uIabfPrX/+afv36ER8fz44dO1ztUkTE51RVwdNPw7hx\n8OqrMHcu+LXQUNzl3UycOJHs7OxG38/KyuKrr74iLy+PV199lYcfftjVLkVEfMqWLZCYCNu3w7Zt\nMHJky+7f5YOxycnJ5OfnN/r+6tWreeCBBwBISkqivLycI0eO0LNnT1e7FhFxH8MwN2cfAwcPGDz7\nZ4OcHHhugcFd6YZ5IVS54/twRKufdVNUVERkZGTd84iICAoLCxsM+lmzZtU9tlqtXn2PRhFpWFUV\nVFRAZaW5bsvp01B91k7A6RMEVp7A/+xp/M9V0s5eSbClkuDa01xlVBJQVXnhQ2fOmDs6v1VXO/bY\nbjcvLf3hz4Zea+q92toLf6Dzl6ia6xE49LjWgOoaC+E18PcgC4FBFixTgWmX34etpgZbTU2z/s6v\nyOmVP1yHwdLItbsXB72IeJ9z5+DQ1waHdx7jyK5Sjh84gr2olICyUtqVH6FDxRFC7OVc7V9OF47T\nmXJ+VFtOcG0Fp/07ctq/I5V+IZy1BFNm6UBFbTCn7MGcrAmm0tIBe1AwlpBgAkNDCerYjuBOQQR3\nCiSkaxCdfxRE116BXB0WxFUdg8zLR4OCzC0w0NwCAsyJbz8/8Pe/9GdDrzX1XnPWIABOnoTMTPj7\n3+Hf/4Zf/QqmTIEOzTyjxvr9dt5sB+po9aAPDw+noKCg7nlhYSHh4eGt3a2ItKYTJziR+yXffPA1\nx3fmU5OXT/vSQ/SozOcaDhPuH0xFaC9qru6J0aMXfoN6EdS7JyHXXM9V4V2xdOkMnS/aOnako58f\nHRvpzjDML5GTJ+Hbb83tyBEo+v5naSkU7IDDh6GgAK66Cnr3vnSLjLywhYU1fxmB5jIMOHDAvNBp\n7Vqw2eD222HqVHP1ydbu/2KtHvTp6eksWrSI8ePHk5OTQ+fOnTU/L+ItqqrMBVa2b+fU1n2cyt3H\nVV/vpd2Zcv5NNCeuvo7aPlF0uC2RHkljibgtiqC+fWh31VWEtmAZFgu0b29uPXo03dYw4OhRM/Qv\n3nJzobDQ/CL49lvo3v1C8EdEXPjZrZt53nqXLubPkJDGB++1teaXT3Gxue/CQti7F3bvhl27zC+c\nYcPgZz+DpUvN7zR3cHmZ4gkTJvDRRx/x3Xff0bNnT2bPnk11dTUAGRkZAEybNo3s7Gw6dOjAkiVL\nGDhwYP1CtEyxiPt98w1s2gSffUbt1lyM3Xs4EnIdOVU3savmBgLjYwm7I5b40b1JGOhHgJdeW19T\nAyUlZugXFFz4AigsNL8kjh2D48fNn2fPQrt2F2Z/AgPN3y4qK82fISEQHm5uEREQEwMDBkBcnPla\nM2d4ms2R7NR69CJtWWmpeenlBx/ABx9Qe6qCwmtv54PTSSz/+mbOxSYydHQId95pXsTTUud1e5OL\nj+VWV5tfEu3amRcytW/f+kF+OQp6EbmUYcD+/eZRwXffhYMHqUm2sqf7MN4oGMbinFhuvc3CPfdA\nWpo5vSGeTUEvIqaDB+Ef/zBvMnr2LMZdd7G//xhe2jmEFf8byODBcM895kHCq692d7HSHI5kp5fO\nsInIZR09CsuXwxtvmEcj772X4/9nGUt2DeS1xRaq18KkSbBvn3kWivgujehFfM327bBoEaxaZV5L\nf//9bOsynL+8HMDq1XDXXebyt8nJ7p9fFtdpRC/SVtTUwDvvwEsvmaePTJlCzf48/ndTN1561nxp\n6lT4n/9xfclb8T4KehFvVlVlzr3Pm2fOvzz2GBXD0nl1cQD/PRiuuQYeecQcxXvrqZDiOv3Ti3ij\ns2fNa+mfe848cXvxYo7dOISFC2FRBgwdap5Uc9NN7i5UPEEbPCtWxIvZ7eYllv37m9fVr1xJ8ZK1\n/Hb1EPr2NY+5fvyxecMKhbycpxG9iDcwDMjOhpkzzUsxly2j+JpbmTsX/vlPuP9+2LnTXNNF5IcU\n9CKebs8emDEDiopg3jzKbh3DgucsLF4MEyeaC2ddbv0Xads0dSPiqU6ehEcfhTvugLvv5vjmL/jD\ntrFEx1g4c8bM/xdfVMjL5SnoRTyNYZjzMTExcOIEp3L28udjU+kXE0BpqXmruZdfNhfMEnGEpm5E\nPMm+ffDww3DqFDXLV/LX3f/Bn28xB/Wffgr9+rm7QPFGCnoRT1BVBQsWwEsvYTw9i3d7/YonHvKn\nTx/zGGxCgrsLFG+moBdxt88/N9ckiIhg+9+385sXIjlxwrzINTXV3cWJL9AcvYi7nDljni6ZlsaR\n+3/HPcHvkT41kokTYccOhby0HJeDPjs7m+joaPr168eCBQvqvW+z2ejUqROJiYkkJiby7LPPutql\niPfbtAni4zl3MJ8/3LWH2Ln3MSDewoED5oqS/v7uLlB8iUtTN3a7nWnTprFhwwbCw8MZPHgw6enp\nxMTEXNLu9ttvZ/Xq1S4VKuITTp2CJ57AePdd3h3+Mr/MGsPPfmYeg9WtlKW1uBT0ubm59O3bl6io\nKADGjx9PZmZmvaDX8sMiwLp1GJMn83XUHYz1+4JrT3Rh82aIjnZ3YeLrXAr6oqIiIiMj655HRESw\ndevWS9pYLBY++eQT4uPjCQ8P54UXXiA2NrbB/c2aNavusdVqxWq1ulKeiGcoL4fHHuPs+xv4beir\n5J5JZdE/YcgQdxcm3shms2Gz2Zr1GZeC3uLAXQsGDhxIQUEBwcHBrFmzhjFjxnDw4MEG214c9CI+\n4f33qX7oV2xol8bM9nv4/Z878tI9bfMm29IyfjgInj179mU/49J/buHh4RQUFNQ9LygoICIi4pI2\noaGhBAcHAzBy5Eiqq6s5duyYK92KeL5jxzhzz/2UTZjOPZVL2f/r/8tnBzoyfrxCXq48l/6TGzRo\nEHl5eeTn51NVVcWKFStIT0+/
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 10
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"The fit actually approximates the data fairly well over the range where we have data points.\n",
|
||
|
|
"\n",
|
||
|
|
"This looks even better if we clip the output values."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"plot(lxs,clip(polyval(fit,lxs),0,1))\n",
|
||
|
|
"plot(lxs,posterior,color='r')"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 11,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x2e4c3d0>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD9CAYAAAChtfywAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X98zXX/x/HHZqMiUn7ENi02NmGT38nVKJFY6Qcj6pLk\n0iVXqm/SL3SFJkqZauS3YvrhmkszzGUlYkJJKnOh/YhFtfze5uzz/eOtc5kf29jZPudsz/vtdm7b\nOfvsnJe1nue91+f9fn+8LMuyEBERj+VtdwEiIlIyCnIREQ+nIBcR8XAKchERD6cgFxHxcApyEREP\nV2SQP/zww9StW5fmzZtf8JgRI0YQHBxMWFgY27Ztc2mBIiJSuCKDfNCgQSQmJl7w6wkJCezevZvU\n1FRmzJjBsGHDXFqgiIgUrsgg79SpEzVr1rzg15ctW8ZDDz0EQLt27cjOziYrK8t1FYqISKFK3CPP\nzMwkICDAed/f35+MjIySPq2IiBSTjyue5OxV/l5eXuccc77HRESkaEXtpFLiEbmfnx/p6enO+xkZ\nGfj5+V2wGHe6jRkzxvYaPKEmd61LNZXDmvLysHbvxlq5Emv6dKyRI7HuuQerfXusgAAsX1+sa67B\natEC6/bbsfr3xxo+HGvMGKw338RasAArIQHryy+xdu7ESkvD+u03rJycMv9ZPf64xVNPlfx5iqPE\nI/LIyEhiYmKIiopi48aNXHXVVdStW7ekTysi5d2+fbB9O3zzjfm4fbt5rF49aNQIgoLMrX178PMz\nt2uvhcsus7vyIh04AAsXws6dZfN6RQZ5v379+Oyzzzh06BABAQGMGzeOvLw8AIYOHUqPHj1ISEgg\nKCiIqlWrMmfOnFIvWkTc1/HjsGEDfP65CbJduyD7wEmaHPmKNrnr6VRpPSdOJXFo4gz21wkjJySM\nWl16E/DCGCqFBEOVKnb/E0rs9ddhwADzvlMWigzyRYsWFfkkMTExLimmrEVERNhdwjncsSZwz7pU\nU/GURU25ufDppzB/PqxeDWEtLO5tkcpz1VYQdGUi1fZ8gRXaBOumjpxoOYA/jg/km6b3s2sXbN0K\n6+bBgWjo1g3uvRd69IBq1Uq97HO44meVmQmzZsHXX5e8nuLysorbhCnpC3l5FbvfIyKe4dgxmDED\npkwx3ZB/3LaDHocXc1n8EvPFO+6A7t3httvgqqsKfa6sLFi2DD7+GDZuNIE+dCi0aQOeNFdi0CAz\nEp840TXPV5zsVJCLyEXLz4e5c+H556F760P8s9Ec/NfMh+xsiIqCvn2hVatLTuCsLJgzx7xJ1KwJ\nzzwD990HlSq59t/hatu2mfeuXbugenXXPKeCXERcbscOeOQRaHp0E69dN51rNvwb7roLBg+Gjh3B\n23VbOOXnQ0KCGd3+8osJ9IcegsqVXfYSLuNwmH/+oEHmLwlXUZCLiMtYFkyPsUh6IZnp9f5J/ZN7\n8Hr8cfjrX+Gaa0r9tdetg1degd274Z//hH79XPqeUWJTp8LSpbB2rWvrUpCLiEscOQITe6yjz9ej\naVrrFyqPeQ4eeAB8fcu8luRkGD3azI6ZMMGcGLW7h56aCh06wJdfQnCwa59bQS4iJZaxbi/f9RpF\n61MbqTFtPD4P9re9WW1Z5sToc8/B1VdDdDTcdJM9tZw4Yaa6Dx0Kjz3m+ucvTna60R8mIuJW8vLY\nP3w8V9zSmuo3NefqrB/wGTTQ9hAHMwK/6y6zhmjwYHN+9e67y24Bzp8sy4R3s2Zg58avCnIROdfX\nX3PshrbsnPkFX07fRoeEF/GqeoXdVZ2jUiXTot+1Czp1gogIcyK2rPbtGzPGvJnExtrb3lGQi8j/\n5OdDdDS5nW9n1IEnyF2awJ3DGthdVZEuuwyeesoEeu3aEBYGo0bB77+XzutZFkyaBHFxsGKFPYuX\nzqQgFxHj0CHo2ZOji5bR1nsLd338EHf08KCVOJg1RxMnmlHyb79B48amf37kiOtew+GAkSNh3jxI\nSoI6dVz33JdKQS4iZj15q1b8Vr8ZoQeSeWlmAF272l3UpfPzg5kzzX4vW7fC9dfDs8+a5fMlsW8f\ndO4M334LX3wBZ1yKwVYKcpGKbvly6NqV35+fTMvVkxg3wZd77rG7KNcIDTXtj82bzeyS5s3NCtF/\n/Qtycor/PNnZ8OKLZrFqr16wapVZceouNP1QpCKbPh3Gjydn8VJueaYdd9xhTuCVV9nZ8NFHsGCB\nGVV37mxurVqZHXOvucYs5jlxAtLTISXFrCz99FPo3RvGjoXAwLKtWfPIReTCJk6EWbOwVicx8MVA\n8vJg8WL7F9eUlZ9/Nqsw1641PfXUVBP0vr4mzOvXNwHfuTP06QO1atlTp4JcRM5lWfDSS2abwaQk\nJi2sz5Ilpp98hfvNMCxTlmW25K1c2X3e0BTkInKuF1+Ef/8bVq/m8+9r06eP6SG7y4k7Kag42emS\niy+LiIeYOhU+/BDWreMXqzb9+5vtYhXink1BLlJRzJsHb7wB69aRf01tBt4BDz5o9s8Wz6YgF6kI\nkpLMUsfkZGjQgNcnm90DX37Z7sLEFdQjFynvUlPh5pthyRK45Ra++87sSZKSYhbKiHvT7ociFV12\nNkRGmisx3HILeXmmnTJhgkK8PNGIXKS8ys83IX799TBtGmAW+2zebBa4uMv0OimcZq2IVGSvv252\njlq6FIAtW+Ddd80FghXi5YuCXKQ82rQJXnvNNMJ9fTl1Ch591Gy9Wr++3cWJq6lHLlLeZGebS+bE\nxsJ11wEQEwM1apj+uJQ/6pGLlDcDB5rUjokBzOZPLVvChg1mf27xLOqRi1Q08fHmUu7ffON8aMQI\nc1OIl18KcpHy4tdfzRWA4+KgalXAbKny/fdmV0Mpv9RaESkvHnjAXLBy6lTAXDjhhhvgnXfw6Kv9\nVHRqrYhUFImJsHGjuVrCaW++CU2bKsQrAo3IRTzdyZPmGmZvveXcAevAAWjWzLTLg4Ntrk9KREv0\nRSqC114zqX3GNobPPw+DBinEKwqNyEU82d690KaNWbZ5es74li3Qsyf88IOZhSieTSNykfJu5Eh4\n6ilniFsWPPmk2Z5WIV5x6GSniKf67DMzXzwuzvlQYiIcPGjaKlJxaEQu4ony8+Hpp81+tFWqOB8a\nPRrGjwcfDdEqlCKDPDExkZCQEIKDg4mOjj7n64cOHaJ79+6Eh4fTrFkz5s6dWxp1isiZ4uLMFoZ9\n+zofWrwYLrsM7r7bxrrEFoWe7HQ4HDRp0oSkpCT8/Pxo06YNixYtIjQ01HnM2LFjycnJYeLEiRw6\ndIgmTZqQlZWFz1lDAp3sFHGRnBwICTHX4PzLXwDIzYXQUJg1y1z9R8qPEp/sTElJISgoiMDAQHx9\nfYmKiiI+Pr7AMfXq1ePw4cMAHD58mGuuueacEBcRF3rnHTNv/HSIA7z3nplqqBCvmApN3MzMTAIC\nApz3/f392bRpU4FjhgwZQpcuXahfvz5HjhxhyZIlF3y+sWPHOj+PiIggQr91Ihfn+HGIjoaVK50P\nHTsGr7wCy5fbWJe4THJyMsnJyRf1PYUGuVcxLiMyYcIEwsPDSU5O5r///S9du3blm2++4corrzzn\n2DODXEQuwbvvQseO0KKF86Fp06BTJ7jxRhvrEpc5e5A7bty4Ir+n0NaKn58f6enpzvvp6en4+/sX\nOGbDhg3cf//9ADRq1Ijrr7+eH3/88WLqFpHiOH7crOJ86SXnQ0ePwhtvmGtxSsVVaJC3bt2a1NRU\n9u3bR25uLnFxcURGRhY4JiQkhKSkJACysrL48ccfadiwYelVLFJRvfMO3HxzgdH422+bvnjTpvaV\nJfYrcon+ihUreOKJJ3A4HAwePJjRo0cTGxsLwNChQzl06BCDBg0iLS2N/Px8Ro8eTf/+/c99Ic1a\nEbl0x45BUBCsWmVOdJ5+qFEjSEoyW61I+VSc7NReKyKe4PXXzVaGH37ofGjKFPPQRx/ZWJeUOgW5\nSHmQlwcNG5rLuJ0+o3n8uBmN
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 11
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Linear Least Square\n",
|
||
|
|
"-------------------\n",
|
||
|
|
"\n",
|
||
|
|
"For comparison, let's also look at a simple linear fit."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"data = r_[c_[samples0,zeros(100)],c_[samples1,ones(100)]]\n",
|
||
|
|
"fit = polyfit(data[:,0],data[:,1],1)\n",
|
||
|
|
"plot(lxs,polyval(fit,lxs))\n",
|
||
|
|
"plot(lxs,posterior,color='r')"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 12,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x38c1d50>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD9CAYAAACyYrxEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XtYlGXeB/DvKFai5mFTJMBVFxFQTp7Y3MhJQ15RUTuq\nWa7prnnIVbNV21x1S9NOpvnWWruZvrUe0kwqJI/jMUQFTQUSW0xAIRRREBEY7vePX5KK6DAHnmeG\n7+e65uL0zMzv0vp6cz/3/bsNSikFIiJyWfW0LoCIiByLQU9E5OIY9ERELo5BT0Tk4hj0REQujkFP\nROTibA765557Dh4eHggKCrrlz00mE5o2bYqwsDCEhYXhtddes/UtiYioBtxsfYFRo0bhhRdewLPP\nPlvtNb169UJsbKytb0VERFaweUQfERGB5s2b3/Ya7skiItKOw+foDQYD9u3bh5CQEERHRyMlJcXR\nb0lERNexeermTrp06YLMzEy4u7tj06ZNGDx4ME6cOFHlOoPB4OhSiIhc0p1mTRw+om/SpAnc3d0B\nAP369UNZWRny8/Nvea1SSleP2bNna14Da3KtulgTa7LXIydHYdYsy6bFHR70ubm5UEqKSUxMhFIK\nLVq0cPTbEhG5pPJyYPFioHNn4PJly55j89TNsGHDsHPnTpw7dw4+Pj6YO3cuysrKAABjx47FunXr\n8MEHH8DNzQ3u7u5YvXq1rW9JRFQnmUzACy8ArVsDu3YBAQHAO+/c+Xk2B/2qVatu+/MJEyZgwoQJ\ntr6NJoxGo9YlVMGaLKfHuliTZVjTjbKygGnTgO++k2B/9FGgJrc1DeravIrGDAYDdFIKEZEuXL0q\nwf7WW8CECcCMGcAvtzwrWZKdDl91Q0RENRcXB/zlL0BgIHDgANC+vfWvxaAnItKRH38EpkwB0tKA\nJUuAfv1sf002NSMi0oHiYmDWLCA8HOjZEzh61D4hD3BET0SkKaWA9euBF1+UgD98GPD2tu97MOiJ\niDSSkiLLJX/+GVixAnDUwh5O3RAR1bKLF4GpU4FevYDBg4HkZMeFPMCgJyKqNRUVwMqVstHp4kXg\n+HEZ0bs5eG6FUzdERLUgKQmYOFFaGGzYIDddawtH9EREDnT+PPD880B0NDB6NJCQULshDzDoiYgc\nwmwGPvhApmnuugtITZWgr6dB6nLqhojIzvbulWmae+8Ftm4FgoO1rYdBT0RkJ2fPAtOnA9u3A2++\nCQwdWrPmY47CqRsiIhuVlQFvvw0EBQH33y/tC4YN00fIAxzRExHZZOtWYNIkoE0bmbLp2FHriqpi\n0BMRWeGnn6RtQVISsGgREBOjnxH8zTh1Q0RUAyUlwKuvAl26yE3W48eBQYP0G/IAR/RERBZRCvjq\nK2DyZCAsDDh0CGjbVuuqLMOgJyK6gxMn5BCQU6eAZcuAyEitK6oZTt0QEVWjqAiYOVPaB/fpAxw5\n4nwhDzDoiYiqUApYvVp2tWZlySEg06bJDldnxKkbIqLrHD0qHSULCoBVq4AHH9S6IttxRE9EBAn2\nSZNkiubJJ+VmqyuEPMCgJ6I6rqIC+PhjwN8fuHpVTn0aPx6oX1/ryuyHUzdEVGclJso0Tb16wDff\nAF27al2RY3BET0R1Tl4eMGaMbHQaP15aF7hqyAMMeiKqQ8rLgaVLgcBAaSGclgaMHKlNj/jaxKkb\nIqoTdu2SHvEtWwImE9Cpk9YV1R4GPRG5tOxs4KWXgD17pJXw44/ruy+NI7j4LyxEVFddvQosXAiE\nhADt28tRfk88UfdCHuCInohcUHy89Kbx85PDuH19ta5IWwx6InIZ//0vMHUqcOwYsHgx0L+/1hXp\nA6duiMjpFRcDs2cD3bsDPXpI0DPkf8URPRE5LaWADRtkFB8eDhw+DPj4aF2V/jDoicgppaVJb5oz\nZ4Dly4GHH9a6Iv3i1A0ROZVLl2S5ZEQEEB0NJCcz5O/E5qB/7rnn4OHhgaCgoGqvmTRpEjp06ICQ\nkBAkJyfb+pZEVAcpBXz6qfSIP39e5uEnTwYaNNC6Mv2zOehHjRqF+Pj4an8eFxeHkydPIj09HR9+\n+CHGjRtn61sSUR1z+LCM4N99F1i/XrpNenhoXZXzsHmOPiIiAqdOnar257GxsRg5ciQAIDw8HAUF\nBcjNzYUH/5aI6A7y84FXXpFwf/VVYPRoDdoHKyW9jCsqALNZHpZ+XlEhz7/5ce117fF9Czj8Zmx2\ndjZ8rrsN7u3tjaysrFsG/Zw5cyo/NxqNMBqNji6PiHTIbAb+9S/g73+X3aypqUAL9xLg5wvAhV8e\nRUXA5cs3PoqLq35dWlqzR1lZ1bA2GORfmHr15KOlnxsM1T8Aq75vKiyEqbDw1+ssUCurbtRN//IY\nqinu+qAnIhdXUQHk5MiymZwc4OxZICcHOclnkbIjBw9U/IxT911Awy8uAP++INc3b/7ro0kToFEj\nwN1dPl57NG0K3H+/fN6wIXDPPXLYq6UPN7dfg/pacOuob4Lxl8c1cy2ozeFB7+XlhczMzMqvs7Ky\n4OXl5ei3JSI9MJuBjAy5c3rypHz+3//K46efgGbNJJQ9PVHctDW2HvfE/p8CETWyD4IeawVDi+uC\nvWFDXQWuM3F40MfExGDp0qUYOnQoEhIS0KxZM87PE7mikhIgKQnYvx84ckTCPTVV+gJ36iSNZ/z9\nZU1k+/ZA27ZAw4YoK5Me8fPnA6NGAbNmyWCd7MfmoB82bBh27tyJc+fOwcfHB3PnzkVZWRkAYOzY\nsYiOjkZcXBx8fX3RqFEjLF++3OaiiUgH8vOBbduA3bulc9jx47L2MTxcTtV+/vlfT/ioxvbtcpSf\nl5e8jL9/LdZfhxjUzRPoGjEYDFXm8olIR8xm4LvvgG+/BTZvltF6RARgNAIPPAB06SLz5RY4fRqY\nNk3ObF20CBg8mLMy1rIkOxn0RFQ9s1mG2p9/DnzxBdCqlXQL69tXwv3uu2v0ciUlcvjHokVy2tNf\n/2rxvw1UDUuyk71uiKiq9HTg3/8GVqwAPD1ljePOnTLPbqWvv5adrEFBwIEDQLt2dqyXbotBT0Si\nvFxG7R98IPPtzz4L7Nhh88T5yZMS8OnpwP/+LxAVZad6yWIMeqK6rrgY+OQTmVPx9JSjmQYNkjXl\nNrh8WVbSLFsmUzRffGHzS5KVGPREddWVK7Ku8a23ZL595UrgD3+w+WWVkin9adPkXu2RI7KqhrTD\noCeqa8rLZQQ/d64cybR9u6xzt4Njx6RH/Pnz0mnyoYfs8rJkI/ajJ6pLdu0CQkOBzz4D1q2T+RQ7\nhPzFi8CUKdIX/tFHgUOHGPJ6whE9UV2QmyundezYIb1+H33ULgvXKypkxmfmTGDAACAlRTbCkr4w\n6IlcmVIyep86FfjjH2WTU+PGdnnpgwdlV2tFBRAbK7NApE8MeiJXlZcHjBsnh6vGx8vOVTs4dw54\n+WXgq69kVc3IkdLgkfSLfz1Erujbb4GQENmVdPCgXUK+vFzWwQcGym7W1FRpQsaQ1z+O6IlcidkM\n/OMfcmrHf/4jfWjsYM8eaVnQvLn0MbvNEdGkQwx6IleRlwc8/bSckHToENC6tc0veeaMbHbauVOW\n2z/5JJuPOSP+0kXkCo4dA3r0kCmaLVtsDvnSUuDNN4HgYKBNG5mmeeophryz4oieyNnFx0tfmkWL\nZERvo82bZdNT+/bSlbhDBzvUSJpi0BM5s/ffB159Fdiwweb2BadOySrMI0dkqf2AARzBuwpO3RA5\nI6WAv/8dWLxY7pTaEPJXrkg3hK5dZebn+HFg4ECGvCvhiJ7I2VRUSN/f3bvl0aqVVS+jFLBxo7Qu\n6NYNSE6W+XhyPQx6ImdSXg489xyQkSHtDJo1s+plfvhBuhGfPi0rMfv0sXOdpCucuiFyFmVlsvQl\nL082RFkR8oWFwPTpMtMTFSXz8Qx518egJ3IG5eWyoubqVZlvqeFBq0rJ/qmAACAnR1ZjTpkCNGjg\noHpJVzh1Q6R3ZrM0lLl0Cfjy
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 12
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Note that these kinds of linear fits are very sensitive\n",
|
||
|
|
"to outliers. \n",
|
||
|
|
"We add two data points that ought to be really easy\n",
|
||
|
|
"to classify, and they completely change the approximation\n",
|
||
|
|
"to the posterior."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"data1 = data.copy()\n",
|
||
|
|
"data1[0,:] = array([-100,0])\n",
|
||
|
|
"data1[-1,:] = array([100,1])\n",
|
||
|
|
"fit = polyfit(data1[:,0],data1[:,1],1)\n",
|
||
|
|
"plot(lxs,polyval(fit,lxs))\n",
|
||
|
|
"plot(lxs,posterior,color='r')"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 13,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x3c3bd10>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD9CAYAAAChtfywAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH6FJREFUeJzt3Xtc1vXdx/EXCJmapzwLKBnEYSo4NTpZaFPUlGaHhaY1\nM2/nbmu67b6ttru03WZ2uq2oZrW0dENsqxtXiE7X1UFFKk1reicuSaCiqDyn4MXv/uM3SQSui8N1\nXd/r8H4+Hr8HXPADPmJ77+vne/iFWZZlISIiASvcdAEiItI6CnIRkQCnIBcRCXAKchGRAKcgFxEJ\ncApyEZEA5zbIb7vtNnr16sWgQYMavefOO+8kPj6elJQUduzY4dECRUTENbdBPn36dAoKChr9fH5+\nPvv27aO4uJhnn32W2bNne7RAERFxzW2Qjxgxgq5duzb6+bVr13LrrbcCkJaWxsGDB6moqPBchSIi\n4lKre+Tl5eXExMTUvo6OjqasrKy131ZERJoowhPf5Oxd/mFhYfXuaehjIiLinruTVFo9Io+KiqK0\ntLT2dVlZGVFRUY0W40/XfffdZ7yGQKjJX+tSTUFYU3U11r59WOvXYz31FNa8eVjXXYd1ySVYMTFY\nkZFY3bphDR6MNWYM1pQpWHPmYN13H9bjj2OtXImVn4+1dSvW7t1YBw5gffMN1smTAfu7aopWj8gz\nMzPJzs4mKyuLwsJCunTpQq9evVr7bUUk2JWUwK5dsHOn/XbXLvtjffrAhRdCXJx9XXIJREXZV+/e\ncO65piv3O26DfPLkybz55ptUVlYSExPDwoULqa6uBmDWrFmMHz+e/Px84uLi6NChA8uXL/d60SIS\nYE6cgPfeg82b7WvjRnj2WUhJsa9Jk+C++yA+Htq2NV1twHEb5Dk5OW6/SXZ2tkeK8bX09HTTJdTj\njzWBf9almprGSE2WBcXFsG4dFBTAO+9AQgJcfjlMnUr6tGlw442+r8sNf/z7a4owq6lNmNb+oLCw\nJvd7RCRAffQRrF4Na9bAsWMwbhyMHQs/+hF06WK6uoDUlOz0yKoVEQlhlZWwfDm89BIcPAhZWfCn\nP8HQoaDVaj6hIBeRltm2DZ56Cv76V7j2Wnj6abt1Eq4jnHxNrRURaTrLAocDfvc7+OQTuOMO+OlP\noVs305UFLbVWRMRz3n4b7r4bvvwS7rkHbr4ZIiNNVyUoyEXEnf37Yf58KCyERYtgyhRo08Z0VXIG\nNbNEpGHV1XZwDxsGgwbB//0fTJumEPdDGpGLSH0ffADTp9s7KXfsgH79TFckLmhELiLfq6mBJUtg\nzBiYOxfy8xXiAUAjchGxVVbCLbfAoUPw/vtwxvHU4t80IhcRu5UydCgMHGgvL1SIBxSNyEVC3Wuv\n2f3wp5/2y/NPxD0FuUgoe+ope2XKa69BWprpaqSFFOQioWrxYvjDH2DLFoiNNV2NtIKCXCTUWBbc\ney/85S/w1lvQt6/piqSVFOQioebee+2Drt58E3r0MF2NeICCXCSULF0KL79sn5uiEA8aCnKRUPHi\ni/A//6MQD0I6xlYkFGzcCFOn2mvEExNNVyPN0JTsVJCLBLviYrjiCvvxa1ddZboaaaamZKd2dooE\ns4MHITPTfhCEQjxoaUQuEqxqauwQv+ACePJJ09VIC+kJQSKh7LHH4Jtv4NVXTVciXqYRuUgw2rbN\nHo0XFUH//qarkVZQj1wkFB08CFlZsGyZQjxEaEQuEmymTYPOnSE723Ql4gHqkYuEmrw82LoVdu40\nXYn4kEbkIsHi66/thyTn5sKIEaarEQ/RhiCRUHLzzfbW+6VLTVciHqTWikioKCiAwkL48EPTlYgB\nWrUiEuhOnIA77rAnN9u3N12NGKAgFwl0Dz9sPzR53DjTlYgh6pGLBLL9+2H4cHj/fa0ZD1LaECQS\n7ObNg1/9SiEe4jTZKRKo3nzTXi+em2u6EjFMI3KRQFRTA7/+NTzwALRta7oaMcxtkBcUFJCYmEh8\nfDxLliyp9/nKykrGjh1LamoqAwcOZMWKFd6oU0TOlJsLYWFw002mKxE/4HKy0+l0kpCQwMaNG4mK\nimL48OHk5OSQlJRUe8+CBQs4efIkixcvprKykoSEBCoqKoiIqNu10WSniIecPGk/ru3FF+HKK01X\nI17W6snOoqIi4uLiiI2NJTIykqysLPLy8urc06dPHw4fPgzA4cOH6datW70QFxEPeuYZeyu+Qlz+\nxWXilpeXExMTU/s6Ojqabdu21bln5syZjBo1ir59+3LkyBHWrFnT6PdbsGBB7fvp6emkp6e3rGqR\nUHX8OCxZAuvXm65EvMThcOBwOJr1NS6DPCwszO03eOCBB0hNTcXhcPDPf/6T0aNHs3PnTjp27Fjv\n3jODXERa4Pe/h8svh8GDTVciXnL2IHfhwoVuv8ZlayUqKorS0tLa16WlpURHR9e5Z8uWLdx4440A\nXHjhhVxwwQV8/PHHzalbRJri+HF7F+e995quRPyMyyAfNmwYxcXFlJSUUFVVRW5uLpmZmXXuSUxM\nZOPGjQBUVFTw8ccfM2DAAO9VLBKqnnkGrrhCo3Gpx2VrJSIiguzsbDIyMnA6ncyYMYOkpCSWLVsG\nwKxZs7jnnnuYPn06KSkp1NTU8NBDD3H++ef7pHiRkHHsGDzyCGzYYLoS8UM6a0UkEDz2mP3kn5df\nNl2J+JgeLCESDKqrYcAA+zFuP/yh6WrEx3RolkgwWL0aEhIU4tIo7dwR8WeWBQ89ZPfHRRqhEbmI\nPysogPBwGDPGdCXixxTkIv7s4YfhP/7DPiBLpBEKchF/9d57sG+fTjgUtxTkIv5q6VL4xS8gMtJ0\nJeLntPxQxB9VVNhH1X7yCXTtaroaMUjLD0UC1XPPwY03KsSlSTQiF/E31dVwwQXw+uuQkmK6GjFM\nI3KRQJSXZ+/kVIhLEynIRfxNdjbMmWO6Cgkgaq2I+JNdu2DcOCgp0WoVAdRaEQk8Tz0Fs2YpxKVZ\nNCIX8RdHjkC/frB7N/TpY7oa8RMakYsEktxcuOoqhbg0m4JcxF88/zzMnGm6CglACnIRf/Dhh1BW\nBhkZpiuRAKQgF/EHzz8P06dDhB4RIM2nyU4R006cgOhoePdde0enyBk02SkSCF59FYYMUYhLiynI\nRUzTJKe0klorIiZ98gmkpdkTnW3bmq5G/JBaKyL+buVKyMpSiEuraIpcxBTLsoM8J8d0JRLgNCIX\nMWXrVvtMlWHDTFciAU5BLmLKSy/BLbdAWJjpSiTAabJTxISTJ6FvX9ixwz4oS6QRmuwU8VevvWY/\nAUghLh6gIBcx4XRbRcQD1FoR8bXKSoiLg9JS6NjRdDXi59RaEfFHq1fDNdcoxMVjFOQivrZypdoq\n4lEKchFf2rsXDhyAq682XYkEEQW5iC+tXg0/+YnOHRePchvkBQUFJCYmEh8fz5IlSxq8x+FwMGTI\nEAYOHEh6erqnaxQJHmvWwE03ma5CgozLVStOp5OEhAQ2btxIVFQUw4cPJycnh6SkpNp7Dh48yOWX\nX8769euJjo6msrKS7t271/9BWrUioe4f/4Bx46CkBML1j2FpmlavWikqKiIuLo7Y2FgiIyPJysoi\nLy+vzj1/+tOfuP7664mOjgZoMMRFBMjNhRtvVIiLx7n8L6q8vJyYmJja19HR0ZSXl9e5p7i4mG++\n+YaRI0cybNgwVq5c6Z1KRQKZZdlBrraKeIHLGZewJhzmU11dzfbt29m0aRPHjx/n0ksv5ZJLLiE+\nPr7evQsWLKh9Pz09Xf10CR27dkFVFQwfbroS8XMOhwOHw9Gsr3EZ5FFRUZSWlta+Li0trW2hnBYT\nE0P37t1p164d7dq148orr2Tnzp1ug1wkpOTm2qtVdNKhuHH2IHfhwoVuv8Zla2XYsGEUFxdTUlJC\nVVUVubm5ZGZm1rnn2muv5Z133sHpdHL8+HG2bdtGcnJyy/4EIsFIbRXxMpcj8oiICLKzs8nIyMDp\ndDJjxgySkpJYtmwZALNmzSIx
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 13
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Logistic Regression\n",
|
||
|
|
"--------------------"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Logistic regression is a much better fit."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"from sklearn import linear_model\n",
|
||
|
|
"lr = linear_model.LogisticRegression()\n",
|
||
|
|
"lr.fit(data[:,0].reshape(200,1),data[:,1])\n",
|
||
|
|
"plot(lxs,lr.predict_proba(lxs.reshape(1000,1))[:,1])\n",
|
||
|
|
"plot(lxs,posterior,color='r')"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 14,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x3f4edd0>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD9CAYAAAChtfywAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlcVXX+x/EXiOaW+xpgZKCAG44aljmhZi65trhlNWbm\nWNZo04zjTDOD/bKyrExtSi211BRrMqwUt7yWGpLmkpmKJbKkphbhhsDl/P740h0QBFTg3Avv5+Nx\nHvfecw74Qe3d1+/5Ll6WZVmIiIjH8ra7ABERuToKchERD6cgFxHxcApyEREPpyAXEfFwCnIREQ9X\nZJA/9NBDNG7cmDZt2lzynieeeIKgoCDatWvHzp07S7RAEREpXJFBPmrUKGJiYi55fdWqVRw6dIj4\n+Hjmzp3LuHHjSrRAEREpXJFB3rVrV+rWrXvJ6ytXruTBBx8EIDw8nNTUVI4fP15yFYqISKGuuo88\nJSUFf39/12c/Pz+Sk5Ov9tuKiEgx+ZTEN7l4lr+Xl1e+ewo6JyIiRStqJZWrDnJfX1+SkpJcn5OT\nk/H19b2iYspaZGQkkZGRdpeRhzvWBO5Zl2oqHo+qKSsLjhyB77+HQ4fMceQI/PgjpKTAsWNQqxb4\n+kKTJtCgAdSrB/Xr46xTj1NWPX68UJ/EtLokn65NcmpNjpyqScKJGiQdr8Lx41CtGjRsaL6sbt28\nr3FxkQwfHknt2lCzpjlq1Mj7Wq0aeJfheL/iNIKvOsgHDBjA7NmzGTZsGLGxsdSpU4fGjRtf7bcV\nkfIuIQH27IHdu83rnj3mXNOmcOONEBhojs6dTXD7+mI1bsKPP1dl3z749lvYtw8OfQuHD5usb9IE\nbrjBHH5+4BsMHZuab9mkiTmqVbt0SZGR8Ic/lM2PX5KKDPLhw4ezadMmTp48ib+/P1OmTCEzMxOA\nsWPH0rdvX1atWkVgYCA1atRgwYIFpV60iHiY9HTYvh22bDHH+vUwdy60a2eOwYPh3/+GoCC45hoA\nsrNNgzwuDr5aDjt2mPCuUgVatYLQUAgLg3vvNcHdrJm5VhEVGeRLly4t8pvMnj27RIopaxEREXaX\nkI871gTuWZdqKh5barIsiI+H1ashJgY2b4aWLaFLFxg5koj77zcJnEtmJnz1FWzcCJs2mfe1a8NN\nN5lj8GBo3dr0ppQWd/zzKw6vslqP3MvLy+36yEWkhO3dC8uWwfLlcPYs9OkDvXvD7bdDnTr5bk9I\ngOhoWLPGZP2NN0L37hARAeHh0KhRmf8Ebqc42akgF5Grc/IkLFgA774LqakwbBgMHQodOkABD+q+\n+Qbef98E+NGj0K8f9O1rwrs0W9ueSkEuIqVn2zZ4/XX4+GMYOBBGjzZdJwUM6Th2DN57DxYtglOn\nTM4PGmSeY1aqZEPtHkRBLiIly7LA4YD/+z/44Qd4/HEzzKN+/QJv3bgRZs40fd6DBsEDD8Btt5Xt\n8D1PV5zsLJEJQSJSAXzxBUyeDD/9BH//O9x3H1SunO+29HTT8p4504w8eeIJWLLEjMOW0qEgF5HC\nHT4MkyZBbCxMnQojRhTYH5KeDvPmwbRpZkThjBnmwaUmdZc+/QNHRAqWmWmCu2NHaNMG9u+H++/P\nF+JZWfDGG2bEyfr15iHmp59Cjx4K8bKiFrmI5LdrF4waZaZC7txpZtsUYN06mDjRTHlfudIMVJGy\npyAXkf/JzoaXXoKXXzavDzxQYLM6KQnGjzfDxqdPNw8y1fq2j4JcRIyTJ01w//qrmQ+fa3nq31iW\n6Qf/xz9MkC9f7ppRLzZSkIuI6UoZONAM8J46tcDRKImJprfl9GkzrLB1axvqlALpYadIRffJJ9Cz\np+kjefHFAkM8Oho6dTK3bd2qEHc3apGLVGSvv25a4J98YhY3uUhGBvz1r/DRR+a4+WYbapQiKchF\nKqrnn4e33zZN7ICAfJdPnIC77zZrXe3caTZfEPekrhWRisay4J//NNMvP/+8wBDfu9c00G+91bTE\nFeLuTS1ykYrmX/8yC11t2mQGgF9k3Toz+/7VV82ruD8FuUhFMmOGWUP2iy8KDPEPPoBHH4UPPzSt\ncfEMCnKRiuKdd0wz+xIh/vbbpsdl7VqzhZp4Di1jK1IRrF8PI0eaJWiDg/Ndfv11M5Fz7Vpo0aLs\ny5NL0zK2ImL2zrzvPjMNs4AQf/tts2Lhpk1mE2PxPApykfIsNRUGDDAbQdx2W77L771nnn1u3KgQ\n92TqWhEpr7KzTYjfcAPMmpXv8scfwyOPmF6XVq1sqE+KRV0rIhXZK6/Azz/DihX5Lm3fDg89ZNYN\nV4h7PgW5SHm0bZt5ehkXl2/tlCNHzPpY8+bBTTfZVJ+UKM3sFClvUlNh2DCYMweuvz7fpb59zfop\ngwbZVJ+UOPWRi5Q3998PtWvD7Nl5Tv/WZR4QkO+SuDH1kYtUNNHR8OWXsHt3vkvPPmta5K++akNd\nUqoU5CLlxalTMG4cREVBjRp5Lq1ebXpatm8vcLlx8XDqWhEpL+67z0y9nzEjz+nDh6FzZ/jvf7V+\niidS14pIRRETA7Gx8M03eU5nZcGIETBpkkK8PNOoFRFPl54Ojz9unmBWr57n0rPPQq1aMGGCTbVJ\nmVCLXMTTvfSS2USzT588p7dsgTffNLv7eKvJVq6pj1zEkx0+bHZF3rEjz5jxX381S9G+9poZciie\nqzjZqSAX8WSDBpk92SZPznN6zBjTCp8zx6a6pMToYadIebZpkxkvHhWV5/SGDWZd8Yuee0o5pp4z\nEU+UnQ1PPQXPPQfXXOM6ffasaY2/8YZ5yCkVQ5FBHhMTQ3BwMEFBQUybNi3f9ZMnT9K7d2/CwsJo\n3bo1CxcuLI06RSS3qCjw8oKhQ/Ocfvpp6NLFrKciFUehfeROp5OWLVuyfv16fH196dSpE0uXLiUk\nJMR1T2RkJBcuXOD555/n5MmTtGzZkuPHj+Pjk7fXRn3kIiXkwgWz088778Dvf+86HRsLgwfD3r1Q\nv76N9UmJKk52Ftoij4uLIzAwkICAACpXrsywYcOIjo7Oc0/Tpk1JS0sDIC0tjfr16+cLcREpQW+8\nAW3a5AlxpxMefRSmT1eIV0SFJm5KSgr+/v6uz35+fmzbti3PPWPGjKF79+5cd911nD59muXLl1/y\n+0VGRrreR0REEBERcWVVi1RU586ZDTbXrMlzes4c0yc+YoRNdUmJcTgcOByOy/qaQoPcy8uryG/w\n3HPPERYWhsPh4Pvvv6dnz57s3r2ba6+9Nt+9uYNcRK7Am2+aTvC2bV2nTpyAyEj47DPTbS6e7eJG\n7pQpU4r8mkK7Vnx9fUlKSnJ9TkpKws/PL889W7du5d577wXgxhtv5IYbbuDAgQOXU7eIFMe5c2YW\n57/+lef05MlmvazWrW2qS2xXaJB37NiR+Ph4EhISyMjIICoqigEXTRMLDg5m/fr1ABw/fpwDBw7Q\nvHnz0qtYpKJ64w2z8lWu1nhcHKxaZVrkUnEV2rXi4+PD7Nmz6dWrF06nk9GjRxMSEsKcnOliY8eO\n5e9//zujRo2iXbt2ZGdn8+KLL1KvXr0yKV6kwjh71jzJXLvWdcqyYOJEmDrVbAgkFZem6It4glde\nMTv/vP++69SKFfDvf5tFsSpVsrE2KVVaa0WkPMjMhObNzTZuv/ud61SrVjBrFvTqZXN9Uqquehy5\niLiBZcugZUtXiAPMnWs2UVaIC6hFLuLeLMs83Jw+3ZXaaWnQooXZFCgszOb6pNSpRS7i6WJizHq0\nd9zhOjVtmsl0hbj8Ri1yEXfWvTs89BCMHAnATz+ZZVZ27YJmzWyuTcqEWuQinmz7djh0KM8Khy++\naKbhK8QlN61uJeKuZsyAP/0JKlcG4OhRmD/frG4okpu6VkTc0fHjpg/lhx+gbl0AJkwwa6m8+qrN\ntUmZ0lZvIp5q3jy4915XiCcnw6JF8O23NtclbkktchF3k5kJN9wAn34K7doB8NhjUKOG6SOXikUt\nchFPFB1tZnLmhHhyspkTtH+/
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 14
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Note that logistic regression is not influenced by outliers\n",
|
||
|
|
"in the same way as linear least squares.\n",
|
||
|
|
"That's because the derivative of the sigmoid is nearly zero\n",
|
||
|
|
"for points that are correctly classified but far from the\n",
|
||
|
|
"decision boundary."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"lr = linear_model.LogisticRegression()\n",
|
||
|
|
"lr.fit(data1[:,0].reshape(200,1),data1[:,1])\n",
|
||
|
|
"plot(lxs,lr.predict_proba(lxs.reshape(1000,1))[:,1])\n",
|
||
|
|
"plot(lxs,posterior,color='r')"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 15,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x3c34890>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD9CAYAAAChtfywAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlYVeX6xvEvCA6pOeQYoGigQCqYGnbKwspMU8wmycqO\noccsK+vUz6ZzDlZWNplmg1mpaSnahJmiaWHmADmPKRYKkjmkhIpMm/X74zUCQUAF1t5wf65rX+xh\ngY/mdff6rHdwsyzLQkREXJa73QWIiMj5UZCLiLg4BbmIiItTkIuIuDgFuYiIi1OQi4i4uFKD/L77\n7qN58+Z07NjxjNc8/PDD+Pv7ExwczIYNG8q1QBERKVmpQT506FBiY2PP+PnChQvZvXs3iYmJvP/+\n+4wcObJcCxQRkZKVGuQ9evSgUaNGZ/x8/vz53HvvvQCEhoaSlpbGgQMHyq9CEREp0Xn3yFNTU/Hx\n8cl/7e3tzb59+873x4qISBl5lMcPOX2Vv5ubW5FrintPRERKV9pOKucd5F5eXqSkpOS/3rdvH15e\nXudUTGWLiooiKirK7jIKccaawDnrUk1l41I15ebC3r3wyy+we7d57N0Lv/0Gqanw++9w4YXg5QUt\nWkCTJtC4MVx0ETRuzMk6jdmffRHJxxqR/GcDUv+sR/KReuw9XJeUAzXZvx/+/NNcfupbaNTIfG3c\nGDZujOLWW6No1Ajq14d69aBuXfO14POaNaGyxqZlGQSfd5CHh4czefJkIiIiWLNmDQ0bNqR58+bn\n+2NFpKrbswc2b4ZNm8zXzZvNey1bwiWXgJ+feXTvboL7VHhnudVm1y7Yts08du2CX9dAUhKcPAlt\n2kDbttCqFbT0gm5dIbyl+bEtW5rsr1Gj+JKiouDBByvxz6CclBrkd955J8uXL+fw4cP4+PgwduxY\ncnJyABgxYgR9+/Zl4cKF+Pn5UbduXaZNm1bhRYuIi8nMhLVrYeVK81i6FN5/H4KDzWPgQPjf/8Df\nH2rVyv+248dh/XpIWA0//WSyPikJfH3h0kvNY8AAE9xt2kCzZpU3UnYmpQb57NmzS/0hkydPLpdi\nKltYWJjdJRThjDWBc9almsrGlposCxITYdEiiI2FH3+E9u3hyivh7rsJu+ceuP32It928CB89x18\n/z2sXm06Kx07Qmgo9OsHzzxjfkyBrC9XzvjfryzcKms/cjc3N6frkYtIOdu6FebMgblz4cQJ6NMH\nbrwRrr8eGjYscnlODixfDgsWwLJlkJICV18N115rMr9Tp4oLbVdRluxUkIvI+Tl8GKZNg48/hrQ0\niIiAQYOgS5di+xwZGfDNN/DVV2bA3q4d9O8PvXrBZZeBR7nMpas6FOQiUnHi4+Htt+Hrr02jOjLS\nDKPdiy5PycuDFStgxgz48ku4/HK45RYT4BdfbEPtLkRBLiLly7IgLg6efx5+/RUeegj++U8zl68Y\nf/wBU6fClClm2t6998Jdd5nZI1I2ZclO/SNGRMpmxQp46ilzR/Lpp00ie3oWe+m2bTBxIsybZwbr\n8+adsdMi5UBBLiIlS0qCMWNgzRoYNw4GDz7jROwtW+C550zmP/AA/PwzaFlJxdN+5CJSvJwcE9xd\nu5o5gD//DPfcU2yIJybCHXeYG5ahoWZh5n//qxCvLBqRi0hRGzfC0KFmGfyGDWaZZDHS0uCFF2D6\ndHj8cTN5pW7dyi1VNCIXkYLy8mD8eLjhBhg9GhYuLDbELcuEdkCACfOtW+HJJxXidtGIXESMw4dh\nyBCzq9S6dVBge+qC9u6Ff/0LDh0y88G7dKnkOqUIjchFxLRSunSBDh3M9MJiQtyy4L33zGXXXGOm\nkSvEnYNG5CLV3YIFph/+zjvF7n8CcPSoWe+zdy/88AMEBVVyjVIijchFqrO33zZ9kgULzhji8fFm\n6by3N6xapRB3RhqRi1RXL70EH35o0tnXt9hLPvrI3MScMsXsNCvOSUEuUt1Ylpnk/fnnpk9SzGYn\nDocJ8K++MpcEBNhQp5SZglykuvnvf81GV8uXQ9OmRT7OyDAbGB47ZhZznmEbFXEi6pGLVCdvvmk2\nPvn222JDPC3NTCFv0ACWLFGIuwoFuUh1MWMGTJhgErqYED94EHr2hM6dzaVn2A9LnJCCXKQ6WLrU\nbHy1eHGxKzX37zcn8/TvD5MmFbuluDgx7UcuUtUlJsJVV5nj1665psjHhw5BWJjpi//nP5VfnpSs\nLNmp/++KVGVpaRAebg6CKCbEjx41PfEBA+DZZ22oT8qFRuQiVVVengnxNm3grbeKfHzihDkTOTTU\ntM516INz0glBItXZG2/AkSPmkMzTOBzmgB9/f4V4VaAgF6mK4uPh1VchIaHY6SePPw7p6aZtrhB3\nfQpykaomLc3cuZwyBVq3LvLxW29BbKxZmV+zpg31SblTj1ykqrnnHrOiZ/LkIh8tWwZ3321CvE0b\nG2qTs6YeuUh1ExMDq1fDpk1FPkpONn3x2bMV4lWNRuQiVcUff5hDkqOjoUePQh9lZpoFP7ffDk88\nYVN9ck7Kkp0KcpGq4q67zNL7N98s8tGIESbn583TzU1Xo9aKSHURG2u2KtyypchH0dHm9La1axXi\nVZVG5CKuLjPTtFQmTYI+fQp9lJwMXbvCokU6X9NVaYm+SHXw6qvm0OTTQtzhgCFD4N//VohXdWqt\niLiypCSYOBHWrSvy0auvmlbK44/bUJdUKrVWRFzZzTebzVKeeqrQ25s2Qa9eJt99fGyqTcqFbnaK\nVGXLl5vEjo4u9HZuLkRGwvjxCvHqQj1yEVeUl2d6Ji++CLVqFfrojTegUSP45z/tKU0qX6lBHhsb\nS0BAAP7+/owfP77I54cPH+bGG28kJCSEDh06MH369IqoU0QKio42DfBBgwq9vWsXvPIKvP++phpW\nJyX2yB0OB+3bt2fp0qV4eXnRrVs3Zs+eTWBgYP41UVFRZGVl8dJLL3H48GHat2/PgQMH8PAo3LVR\nj1yknGRlQUCAOVjz6qvz387LMyf93HYbPPywfeVJ+Trv6YcJCQn4+fnh6+uLp6cnERERxMTEFLqm\nZcuWpKenA5Cens5FF11UJMRFpBy9+66ZN14gxMHkelYWPPigTXWJbUpM3NTUVHwK3C3x9vYmPj6+\n0DXDhw/n2muv5eKLL+bYsWPMnTv3jD8vKioq/3lYWBhhYWHnVrVIdZWRYe5iLl5c6O2jR83ElQUL\noEYNm2qTchEXF0dcXNxZfU+JQe5Whibbiy++SEhICHFxcfzyyy/06tWLTZs2Ub9+/SLXFgxyETkH\n770HV14JnToVevu//zUzEbt2takuKTenD3LHjh1b6veUGOReXl6kpKTkv05JScHb27vQNatWreKZ\nZ54B4JJLLqFNmzbs3LmTrvobJVK+MjLMKp/TRuMbN5qTfrZvt6kusV2JPfKuXbuSmJjInj17yM7O\nJjo6mvDw8ELXBAQEsHTpUgAOHDjAzp07adu2bcVVLFJdvfsuXHVVodG4ZcGoUfD883DRRTbWJrYq\ncUTu4eHB5MmT6d27Nw6Hg8jISAIDA5kyZQoAI0aM4Omnn2bo0KEEBweTl5fHK6+8QuPGjSuleJFq\n48QJeO01WLKk0Ntz58LJk2YBkFRfWqIv4greeMOc/DNvXv5bWVkQFAQffAA9e9pYm1QoLdEXqQpy\ncmDCBHOMWwHvvguBgQpxUZCLOL85c6B9e7jssvy30tLM6vzvv7exLnEaaq2IODPLMjc3X3sNevfO\nf/v//s/MHZ861cbapFKotSLi6mJjwd0dbrgh/629e+HDD2HrVhvrEqei3Q9FnNmrr5pj7wsszhs7\nFh54AFq2tLEucSoakYs4q7VrYffuQjscJibC11+bryJ/0YhcxFm9+SY88gh4eua/9dxz5q2GDW2s\nS5yObnaKOKMDB8xWtb/+ak6JAHbsgGuuMYP0Cy+0uT6pNOe9ja2I2GTqVLj99vwQB9Mbf+wxhbgU\npRG5iLPJyYE2beCbbyA4GIAt
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 15
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"**Homework** Given class conditional densities $p(x|c)$ for $c\\in\\\\{0,1\\\\}$\n",
|
||
|
|
"that are normal densities\n",
|
||
|
|
"with means 0 and $\\mu$ and\n",
|
||
|
|
"a standard deviation $\\sigma=1$, compute the posterior probability\n",
|
||
|
|
"$P(c=1|x)$."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Least Squares with Kernel Functions\n",
|
||
|
|
"-----------------------------------"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Nonlinear Approximations to the Posterior)\n",
|
||
|
|
"\n",
|
||
|
|
"We are trying to predict a non-linear function $y = f(x)$.\n",
|
||
|
|
"\n",
|
||
|
|
"We have tried different approximations:\n",
|
||
|
|
"\n",
|
||
|
|
"- A linear fit: $y = mx+b$\n",
|
||
|
|
"- A polynomial fit: $y = ax^5 + bx^4 + cx^3 + dx^2 + ex + f$\n",
|
||
|
|
"- Logistic regression: $y = \\sigma(mx + b)$"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Generalization)\n",
|
||
|
|
"\n",
|
||
|
|
"Note that the polynomial fit is really just a multidimensional\n",
|
||
|
|
"linear fit to a bunch of non-linear functions of the input.\n",
|
||
|
|
"We can generalize this approach to a generalized linear fit\n",
|
||
|
|
"(note that we can choose $\\phi_r(x) = 1$ to handle the constant term).\n",
|
||
|
|
"\n",
|
||
|
|
"$y = w_1 \\phi_1(x) + w_2 \\phi_2(x) + ... + w_r \\phi_r(x)$\n",
|
||
|
|
"\n",
|
||
|
|
"A common choice for the $\\phi_i$ is the use of a kernel, \n",
|
||
|
|
"a single non-linear function that is shifted around:\n",
|
||
|
|
"\n",
|
||
|
|
"$y = w_1 \\phi(x-\\mu_1) + ... + w_r \\phi(x-\\mu_r) $"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Gaussian Kernels)\n",
|
||
|
|
"\n",
|
||
|
|
"And a particularly common choice for the $\\phi$ is Gaussian functions,\n",
|
||
|
|
"$\\phi(x) = e^{-\\frac{x^2}{2\\sigma^2}}$\n",
|
||
|
|
"\n",
|
||
|
|
"This kind of representation forms the basis of *support\n",
|
||
|
|
"vector* machines. Support vector machines (or kernel machines)\n",
|
||
|
|
"effectively also perform a fit to the data points, just like\n",
|
||
|
|
"least square methods, but using a different loss function (hinge loss).\n",
|
||
|
|
"\n",
|
||
|
|
"To perform a least square fit, we first take the original data\n",
|
||
|
|
"$(x_i,y_i)$, transform it by computing the $\\phi_r$ functions,\n",
|
||
|
|
"and then performing a linear fit on the outputs."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Let's start by sorting the data by its $x$ coordinate;\n",
|
||
|
|
"that helps with plotting later (it has no influence on the fit).\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"data = data[argsort(data[:,0]),:]"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 16
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"A Gaussian kernel with a smallish $\\sigma$ is given by this function.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"def K(x,mu): return exp(-(x-mu)**2/0.2)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 17
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Now we define a function that computes all the $\\phi_r$ for a given\n",
|
||
|
|
"input $x$.\n",
|
||
|
|
"We can view this as a kind of ``feature extraction'',\n",
|
||
|
|
"a preprocessing of the data that helps with classification.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"def Kv(x): return array([K(x,mu) for mu in linspace(-2,3,11)])"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 18
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"To perform the linear fit, we first need to transform all the data points\n",
|
||
|
|
"in the training set.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"zs = array([Kv(x) for x in data[:,0]])"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 19
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"To make sure that this worked as expected, let's plot\n",
|
||
|
|
"the output of $\\phi_5(x_i)$ against the $x_i$.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"plot(data[:,0],zs[:,5])"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 20,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x43d2d10>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD9CAYAAAChtfywAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHcxJREFUeJzt3XtwFHWCB/DvhAQSEXkTkplgIA8ygTDJXTB6HjqiHCxi\ndhXcjSy7ihFTOVOerluL5a4l7IMSrb0q1+xVxb1F1wfZWJ63QSrMKq4jCAURDOR2E3CAhMwMEAwQ\nw0NMmPT90U4kJJmZZLr7193z/VRNOcP0dH+J5Jtffv2ySJIkgYiIDCtOdAAiIooOi5yIyOBY5ERE\nBsciJyIyOBY5EZHBsciJiAwubJE//PDDSE5ORl5e3pDLPP7448jKyoLD4UBDQ4OiAYmIKLSwRb56\n9Wq4XK4h36+rq8ORI0fg8XjwyiuvoLy8XNGAREQUWtgiX7BgASZOnDjk+1u2bMGDDz4IACgqKkJn\nZyfa29uVS0hERCFFPUfu9/uRlpbW99pms8Hn80W7WiIiilC8Eiu59ix/i8UyYJnB/oyIiMILdyWV\nqEfkVqsVXq+377XP54PVah0yjN4fzz33nPAMzDnwsWyZhE2b5Ocvvihh9Wr9ZTTK15I5jZUzElEX\neXFxMV5//XUAwJ49ezBhwgQkJydHu1qiPj4fsGsX8P3vy68feAD4y1+Ay5fF5iLSi7BTKw888AA+\n/vhjdHR0IC0tDevXr0dPTw8AoKysDEuXLkVdXR0yMzMxduxYvPrqq6qHptiyaRNQUgKMHSu/tlqB\nggJg61ZgxQqx2Yj0IGyRV1dXh11JZWWlImH0wOl0io4QkVjK+dprwDvv9P+zH/4Q2LxZmSKPpa+l\nFphTexYp0kmYaDdksUQ830MU5PUChYXAqVPA1fvLOzvlkfm5c8Do0eLyEaktku7kKfqka7t2Af/y\nL/1LHAAmTABmzQIaG8XkItITFjnp2u7dcpEPpqgI2LtX2zxEesQiJ11jkROFxyIn3bpwAWhuBv75\nnwd/n0VOJGORk259+ingcACJiYO/P2cOcOIEcPastrmI9IZFTrq1bx9w001Dvz9qlDxar6/XLhOR\nHrHISbeam4Hc3NDLFBWxyIlY5KRbzc1ATk7oZRwO4O9/1yYPkV6xyEmXJEkucrs99HJ2u7wcUSxj\nkZMutbcD8fHA1Kmhl5s9GzhyBLhyRZtcRHrEIiddimQ0DgDXXQdMnw60tKifiUivWOSkS5EWOSAv\n19Skbh4iPWORky5FsqMzKDeX8+QU21jkpEuHDg1vRM4ip1jGIiddGu7UCoucYhmvR06609UFpKQA\n588DcREMNc6cATIy5GuT8x7fZDa8HjkZ0qFDQHZ2ZCUOAJMmyf/lNVcoVrHISXeGM60CyKPwWbOA\nY8fUy0SkZyxy0h2PRz7RZzgyMoCjR9XJQ6R3LHLSndZWYObM4X0mI4MjcopdLHLSndZW4MYbh/eZ\nWbM4IqfYxSIn3WltBdLTh/cZTq1QLGORk650dwOnTwNW6/A+x52dFMtY5KQrXq9c4vHxw/tcWpp8\nxcSvv1YnF5GeschJV0YyrQLIxZ+WJn+eKNawyElXRlrkgFzkXq+SaYiMgUVOujKSI1aCZsxgkVNs\nYpGTrnBETjR8LHLSlePHWeREw8UiJ12JdkTe1qZkGiJjYJGTbnR3y4cQ2mwj+zznyClWschJN3w+\n+Trkwz2GPCg4tcLL3lOsYZGTbkQzrQIA48fLl7Tt7FQqEZExsMhJN44fH/mhh0GcXqFYxCIn3Th+\nXC7iaPDIFYpFYYvc5XIhJycHWVlZ2Lhx44D3Ozo6sGTJEuTn52Pu3Ll47bXX1MhJMcDnk4s4Gixy\nikUhizwQCKCiogIulwtNTU2orq5G8zW3K6+srERBQQEOHDgAt9uNp556CleuXFE1NJmTzzfyI1aC\neAgixaKQRV5fX4/MzEykp6cjISEBJSUlqK2t7bdMSkoKurq6AABdXV2YPHky4kd62AHFNCVG5Jwj\np1gUsnH9fj/SrvrOstls2Lt3b79l1qxZg4ULFyI1NRXnz5/H22+/PeT61q1b1/fc6XTC6XSOLDWZ\nklIjchY5GZnb7Ybb7R7WZ0IWucViCbuCDRs2ID8/H263G0ePHsWiRYtw8OBBjBs3bsCyVxc50dXO\nnwd6eoAJE6JbD4ucjO7aQe769evDfibk1IrVaoX3qu8Kr9cL2zVDpt27d+P+++8HAGRkZGDmzJk4\nfPjwcHITwe+XR+MRjB1CstnkkX1vrzK5iIwgZJEXFhbC4/GgtbUV3d3dqKmpQXFxcb9lcnJysH37\ndgBAe3s7Dh8+jFmzZqmXmEzJ641+WgUAkpLkE4NOn45+XURGEXJqJT4+HpWVlVi8eDECgQBKS0th\nt9tRVVUFACgrK8MzzzyD1atXw+FwoLe3Fy+88AImTZqkSXgyDyV2dAYFp1emT1dmfUR6Z5Ekba5M\nYbFYoNGmyIB+9Sv5fpu//nX067r3XmDVKmD58ujXRSRaJN3JMztJF5Q4YiWIOzwp1rDISRf8fsBq\nVWZdLHKKNSxy0oUTJ4DUVGXWZbPJPxiIYgWLnHRBySJPTZXXRxQrWOQkXE8PcPYsMG2aMutjkVOs\nYZGTcO3twNSpwKhRyqwvJUUuch4kRbGCRU7CnTyp3LQKAFx/PZCQAHz5pXLrJNIzFjkJd+KEPIpW\nEqdXKJawyEk4JXd0BrHIKZawyEk4FjlRdFjkJJzSc+QAi5xiC4uchOMcOVF0WOQknFpTKzy7k2IF\ni5yE4xw5UXRY5CRUdzfQ2SmfEKQkFjnFEhY5CdXeDiQnA3EK/0tMSZF3ovKWbxQLWOQklBo7OgEg\nMREYNw44c0b5dRPpDYuchFJjfjyI0ysUK1jkJBSLnCh6LHISikVOFD0WOQl18qQ6c+QAi5xiB4uc\nhOKInCh6LHISikVOFD0WOQmldpHzNH2KBSxyEubrr4Hz54HJk9VZf2qqPAdPZHYschLm1Cl1zuoM\nSk4GTp8GAgF11k+kFyxyEkbNaRUAGD0amDgR+OIL9bZBpAcschJG7SIHvr3mCpGZschJGC2KnEeu\nUCxgkZMwap4MFMQROcUCFjkJw6kVImWwyEkYTq0QKYNFTsJwRE6kDBY5CXPyJEfkREpgkZMQX30F\nXLwITJqk7nY4IqdYELbIXS4XcnJykJWVhY0bNw66jNvtRkFBAebOnQun06l0RjKh4LSKxaLudqZP\nl+8Lynt3kpnFh3ozEAigoqIC27dvh9Vqxfz581FcXAy73d63TGdnJx577DH89a9/hc1mQ0dHh+qh\nyfh8PsBqVX87Y8YAN9wAdHQA06apvz0iEUKOyOvr65GZmYn09HQkJCSgpKQEtbW1/ZbZvHkzli9f\nDpvNBgCYMmWKemnJNPx+bYoc4PQKmV/IIvf7/UhLS+t7bbPZ4L/muqAejwdnz57FHXfcgcLCQrzx\nxhvqJCVT0bLIucOTzC7k1IolggnMnp4efPbZZ/jwww9x6dIl3HLLLbj55puRlZU1YNl169b1PXc6\nnZxPj2F+PzBjhjbb4oicjMTtdsPtdg/rMyGL3Gq1wuv19r32er19UyhBaWlpmDJlCpKSkpCUlITb\nbrsNBw8eDFvkFNv8fuCWW7TZFq9LTkZy7SB3/fr1YT8TcmqlsLAQHo8Hra2t6O7uRk1NDYqLi/st\n893vfheffPIJAoEALl26hL179yI3N3dkfwOKGVrPkXNqhcws5Ig8Pj4elZWVWLx4MQKBAEpLS2G3\n21FVVQUAKCsrQ05ODpYsWYJ58+YhLi4Oa9asYZFTWH4/cM0vd6pJTQU++kibbRGJYJEkSdJkQxYL\nNNoU6VxvL5CUJN/mbfRo9be3ezfwk58Ae/aovy0ipUXSnTyzkzR3+jQwfrw2JQ5wZyeZH4ucNKfl\n/DggF/mpUwB/ISSzYpGT5rQu8sREYOxY4MwZ7bZJpCUWOWlO6yIHeAgimRuLnDQnosh5CCKZGYuc\nNCeqyDkiJ7NikZPmtDyGPIjX
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 20
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"We can perform the least square fit with a pseudo-inverse:\n",
|
||
|
|
"\n",
|
||
|
|
"$$w = Z^\\dagger \\cdot y$$\n",
|
||
|
|
"\n",
|
||
|
|
"Here, $Z$ is the transformed data matrix,\n",
|
||
|
|
"$y$ is the desired outputs,\n",
|
||
|
|
"and $M^\\dagger$ is the pseudo-inverse of the matrix.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"w = dot(pinv(zs),data[:,1].reshape(200,1)).ravel()\n",
|
||
|
|
"w"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 21,
|
||
|
|
"text": [
|
||
|
|
"array([ 0.04377891, -0.06688098, 0.08231902, 0.03663106, 0.01220642,\n",
|
||
|
|
" 0.05338284, 0.46667431, 0.34844818, 0.70233284, 0.31302812,\n",
|
||
|
|
" 1.06082602])"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 21
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"We can now compute the prediction of this least square approximation\n",
|
||
|
|
"and compare it with the true posterior.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"plot(lxs,[dot(w,Kv(x)) for x in lxs])\n",
|
||
|
|
"plot(lxs,posterior,color='r')\n",
|
||
|
|
"\n"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 22,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x43c39d0>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD9CAYAAABZVQdHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlcVPX6wPHPIKik5r6ySIoKqCylkZqFEmGmdMt+N/Va\nXjM1vS3W73a17Qr9rqnZptFimWalprebYoXYFSV3cN9TXAgk9x1BgeH8/vgKsjPMDHOGmef9ep0X\nM+ecmfOA+PCd53wXg6ZpGkIIIRySi94BCCGEqDmS5IUQwoFJkhdCCAcmSV4IIRyYJHkhhHBgkuSF\nEMKBWZzkn376aVq3bk337t3LPb5w4UKCgoIIDAykT58+7Nmzx9JLCiGEMJHFSX7UqFEkJCRUeLxD\nhw6sW7eOPXv28OabbzJ27FhLLymEEMJEFif5vn370rRp0wqP9+rVi8aNGwMQGhrKiRMnLL2kEEII\nE9m0Jv/ll18ycOBAW15SCCGcmqutLrR27VrmzZvHxo0byxwzGAy2CkMIIRxKVTPT2KQlv2fPHsaM\nGcOKFSsqLO1ommZ325QpU3SPQWKSmJwxLonJtM0UNZ7k09PTeeyxx/j222/x9fWt6csJIYQoxuJy\nzbBhw/j11185d+4cXl5exMTEkJeXB8C4ceN46623uHjxIuPHjwfAzc2NlJQUSy8rhBDCBBYn+cWL\nF1d6fO7cucydO9fSy+giLCxM7xDKkJhMIzGZzh7ievddmDoVQkLg3/+2j5hKs8eYTGHQTC3s1GQQ\nBoPJ9SUhhGNZsgTeeANWroRPP4V9+yAhAaQ/RtVMyZ2S5IUQusnJgU6d4Pvv4Z57IC8PAgNh1ix4\n8EG9o7N/puROmbtGCKGbb75RJZp77lHP3dzglVfgo4/0jcuRSEteCKGbu++GmBh46KFb+65dAw8P\nOHwYWrXSL7baQFryQgi7tXcvnDxZtizToAE88AD8/LM+cTkaSfJCCF0sXQrDhkGdOmWPRUXBjz/a\nPiZHJOUaIYQuAgPhs8+gd++yx86dg44d4exZqFvX9rHVFlKuEULYpePH4fRpCA0t/3iLFtChA+zY\nYdu4HJEkeSGEzf38MwwcWH6pptC990I58xmKapIkL4SwuTVr1M3VyvTpAxs22CYeRyY1eSGETRmN\n0LIl7N8PbdtWfF5GBtx1lyrryOjX8klNXghhd3btgjZtKk/wAF5eanDU77/bJi5HJUleCGFTiYkQ\nHm7auSEhsHNnzcbj6CTJCyFsKikJ+vUz7dzgYNXyF+aTJC+EsJmCAti8ufy+8eWRlrzlJMkLIWzm\n8GFo0kTV5E0hSd5ykuSFEDazeTP06mX6+XfcAZcvw4ULNReTo5MkL4SwmeomeYMB/Pzgt99qLiZH\nJ0leCGEzW7bcmjveVP7+cPBgzcTjDCTJCyFs4soVOHYMgoKq9zpJ8paRJC+EsImUFHUjtbqzSkqS\nt4wkeSGETZhTqgFJ8paSJC+EsInq3nQt1KGDWkEqJ8f6MTkDi5L8008/TevWrenevXuF57zwwgt0\n6tSJoKAgdkqHVyGckqaZ35J3dVULiBw6ZP24nIFFSX7UqFEkJCRUeDw+Pp4jR46QmprK559/zvjx\n4y25nBCilkpNhYYNoV07814vJRvzWZTk+/btS9OmTSs8vmLFCkaOHAlAaGgoly5d4vTp05ZcUghR\nC5lbqink6wtHj1ovHmfiWpNvnpmZiZeXV9FzT09PTpw4QevWrcucGx0dXfQ4LCyMsLCwmgxNCGFD\nlib5jh0tWCWqoACuXSt/y8pSX7OzITdXbTdu3Hpc+nnxx/n5anL8wq2goOTzivaV3l9QoOpZxbcK\n9iXl55OUn6++LxPX4KjRJA+UmdDeUMHs/8WTvBDCsWzZAqNGmf/6jh3hm29Qye/8eXUn9o8/1NfT\np9W8Bxcvlvxa+PjaNXB3hwYNVM2oQYOSW8OGcNttUK+e6t9ZuNWvD7ffrh4XP1avnpro3tVVrV9Y\nuLm4lHxu6n4XFzW0t3Ar/bzYvrCbW+H+mIYNq/zZ1WiS9/DwICMjo+j5iRMn8PDwqMlLCiHsTFaW\nqsmHhJhwsqbBqVNq1NTRo0Vfex88xqJdGeB++lZxv21btbVuDc2bq5pOs2bQtOmtr02bQqNGKkk6\nqRpN8lFRUcTGxjJ06FC2bNlCkyZNyi3VCCEcV0qKmhe+zCConBzYs6fsVtidpnALD8d19Fj6RXqz\n+3Qb3JvU0+X7qK0sSvLDhg3j119/5dy5c3h5eRETE0NeXh4A48aNY+DAgcTHx+Pr60uDBg2YP3++\nVYIWQtQeRfX4U6fUytybNqlt717o0kXNcxAYCH/6E3TvDq1alXmPOoDBB47/AQFNbBt/bScLeQsh\nakZuLmzaxL+fSSDCmECTS7/DvfeqFUP69IEePVQt3EQDB8KECTBoUA3GXMuYkjtr/MarEMKJ5OWp\nRVy/+w5WrEDz9SXtjwHkLf4EHr5blWLM1LGjdKM0h/PejRBCWM++ffD88+qGaEyMKsLv3Uvqtyl8\n1OItWj7S26IED2p6A0ny1ScteSGEefLyYOlS+OQTSEuDZ56BrVvBx6folM0LLOsfX1zHjupDgqge\nSfJCiOrJyYH582HmTJXQX3lFFcrLaalbOgiqOCnXmEfKNUII0+Tlwccfq7pJQgIsWgRr16peMRWU\nYqyZ5O+4Q31gKCiwzvs5C2nJCyEqp2mwfDlMngze3hAfb9LIpqtX4cgREwdBmeC226BJEzXIVcZU\nmk6SvBCiYsePw/jxcOIEzJoFkZFqSL0Jtm5VXeCruxJUZby9IT1dknx1SLlGCFGW0QjvvQc9e0JY\nGOzcCQMGmJzgQY176tPHumG1b6+SvDCdtOSFECWlp8OIEarOnpys7niaYe1adU/Wmry94fffrfue\njk5a8kI4IU1TN0XnzoV164rNWvuf/6jW+8MPw+rVZif469dVuebee60XM6iWvCT56pGWvBBO5vx5\n1VA/dkzNMPDBB9CymZEfAybRaPUy+PFHuPtui66RnAwBAWqmXmvy9oZffrHuezo6SfJCOJErVyA8\nHPr3V7nc1RWM5y+R1nsYu7fl4ZO8Fc/AZhZfJykJ+vWzPN7SpCVffVKuEcJJaBo8/TSEhqp7qq6u\nwPHj1OlzDx0jO5EcncCgp5qRnW35tdauVfdrra2wd40wncxCKYSTWLQIpk1TtfL69VHzzQwYAK+9\nBhMmoGkwfDi0aaNKOObKyYGWLVV/9kaNrBY+oP5Q3X676tHZuLF137s2MiV3SkteCCdw7Rr84x/q\nRmv9+qii+QMPqKkJJkwAVO/I2FhYsgTWrzf/WmvXwp13Wj/Bg4pRWvPVI0leCCfw/vuqp0toKCqD\nDx4MX34Jw4aVOK95c5g9W+X9wvWiq+unn9Tb1xSpy1ePJHkhHNylS/Dhh/D226hazZAhsHCh6iZZ\njiFDVLnl88+rfy1NU0m+Jhf2kJZ89UiSF8LBffaZyucdru291YKPiKjwfINB/VGIjoYLF6p3rT17\nwM0N/Pwsi7ky0pKvHknyQjiw69fVlDOvDz2qbrLOmmVSLSUwULXoo6Ord70fflBvX43ZD6pNWvLV\nI0leCAf2zTdwX/eLdHn5YXjjDXjiCZNf+9ZbsHgxHDhg2vmapqpAw4ebGayJpCVfPZLkhXBQmgaf\nzsrl0zND1CrY48dX6/UtW6q/CxMnFpv2oBIpKeDiomZFqEkyf031SJIXwkFt2azxavp4mrZvpLpK\nmmHCBNUn/ccfqz73889h5MiaLdWAWkb23DnIza3Z6zgKGQwlhIP69u7ZRGTOp/Wh9dCwodnvs2oV\n/O1vsH8/1KtX/jlnzkCXLpCaCi1amH0pk7Vvr6ZOuOOOmr+WPbPJYKiEhAT8/Pzo1KkTM2bMKHP8\n3LlzDBgwgODgYLp168ZXX31l6SWFEFW4smozD26bituKHyxK8KDWCQkKgn/+s+Jz3n8f/vxn2yR4\nAC8vyMiwzbVqPc0C+fn5WseO
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 22
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"As you can see, the fit works OK. The fact that it is near zero \n",
|
||
|
|
"again above $x=4$ is just a consequence of the fact that we have no data there."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"These kinds of fits can be tricky, though. For example, if\n",
|
||
|
|
"we add another $\\phi_r$ centered in a region where there is no data,\n",
|
||
|
|
"then the weight $w_r$ is adjusted based on the tail of that\n",
|
||
|
|
"Gaussian, leading to an unreasonable posterior.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"def Kv(x): return array([K(x,mu) for mu in linspace(-3.5,3,11)])\n",
|
||
|
|
"zs = array([Kv(x) for x in data[:,0]])\n",
|
||
|
|
"w = dot(pinv(zs),data[:,1].reshape(200,1)).ravel()\n",
|
||
|
|
"plot(lxs,[dot(w,Kv(x)) for x in lxs])\n",
|
||
|
|
"plot(lxs,posterior,color='r')"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 25,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x4675750>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD9CAYAAAClQCyNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH8BJREFUeJzt3XtwVFWeB/Dv7c6TBEIC5J0QXiEvSFqCGZGMjRgoFSh8\nrQMitYJrbVmlorMz4I6u0VkGRsZRUNyqxR1hxillR2dGdDEjDxsiEB4mREBCQJIQkhACIYE8O+m+\n+8chCQGSdLo7fe/t+/1U3Uqn+/Y9P2Pzzcm5554rybIsg4iINMegdAFEROQcBjgRkUYxwImINIoB\nTkSkUQxwIiKNYoATEWmUQwG+bNkyREREYMqUKd3P1dfXIycnB4mJiZgzZw4aGhqGrEgiIrqVQwH+\n1FNPIS8vr9dza9euRU5ODkpLSzF79mysXbt2SAokIqLbkxy9kKe8vBzz58/HsWPHAABJSUnYs2cP\nIiIicOHCBZjNZpSUlAxpsURE1MPpMfDa2lpEREQAACIiIlBbW+u2ooiIaGA+7jiIJEmQJKnP14iI\naPAGGiBxugfeNXQCADU1NQgPD++3CDVtr732muI1sCbvqos1sSZ3b45wOsAXLFiALVu2AAC2bNmC\nhQsXOnsoIiJygkMBvmjRIsyYMQOnTp1CXFwcPvzwQ6xatQo7duxAYmIidu/ejVWrVg11rUREdAOH\nxsA//vjj2z6/c+dOtxbjKWazWekSbsGaHKfGuliTY1iTezk8jdDpBiTJ4fEcIiISHMlOXkpPRKRR\nDHAiIo1igBMRaRQDnIhIoxjgREQaxQAnItIoBjgRkUYxwImINIoBTkSkUQxwIiKN0kSAHzsGbNwI\nXLmidCVEROqh+gD/4Qfg3nuBvDxgzhygvV3pioiI1EH1Af7CC8DrrwPbtgGjRwObNildERGROqh6\nNcLiYmDePODHHwE/P+DwYeCxx4CzZwGD6n/1EBE5T/OrEW7dCixZIsIbADIzgeHDgfx8ZesiIlID\n1Qa4LAOffQY8/HDPc5IEPP448Pe/K1cXEZFaqDbAT54EWlpEr/tGc+cCO3YoUxMRkZqoNsB37xZh\nLUm9n7/jDqCmBqiqUqYuIiK1UG2Af/stkJ196/NGo5hWqNHbcRIRuY0qA1yWxYnKmTNv/3p2NnDg\ngGdrIiJSG1UGeHk5YLcD48ff/vU77wQOHfJoSUREqqPKAD94EPjJT24d/+6SkQGUlACtrZ6ti4hI\nTVQZ4MXFgMnU9+sBAUByMnD0qOdqIiJSG1UG+NGjQHp6//tMny6uzCQi0itVBnhxsRgm6U96OvD9\n956ph4hIjVQX4BcvirHt+Pj+90tLA44f90xNRERqpLoALy4Wveu+TmB2SUsDTpwQs1WIiPTI5QBf\ns2YNUlNTMWXKFCxevBjtLi7Y3RXgAwkNBUJCgIoKl5ojItIslwK8vLwcmzZtQmFhIY4dOwabzYZP\nPvnEpYKOHQOmTnVs3ylTxP5ERHrkUoCPGDECvr6+aGlpQWdnJ1paWhATE+NSQSUlQFKSY/t2DaMQ\nEemRSwEeFhaGn//854iPj0d0dDRGjhyJ++67z+njyTJw6hQwebJj+0+eDJSWOt0cEZGm+bjy5h9/\n/BHvvPMOysvLERISgsceewx//vOf8cQTT/TaLzc3t/ux2WyG2Wy+7fEuXhSLVY0e7Vj7kyYBmzc7\nVzsRkZpYLBZYLJZBvcelW6pt3boVO3bswAcffAAA+NOf/oSCggJs3Lixp4FB3FJt715g1Spg/37H\n2q+pESc8L14cdOlERKo25LdUS0pKQkFBAVpbWyHLMnbu3ImUlBSnjzeY4RMAiIwUc8avXHG6SSIi\nzXIpwNPT07F06VJkZmZi6vWpI88884zTxyspGVyASxKQmAicPu10k0REmqWqu9LPmwcsXw489JDj\nx1+0CHjwQXHzYyIib6G5u9KfOuX4FMIuiYmciUJE+qSaAO/sBM6d6/smDn3hEAoR6ZVqAryyEoiI\nAPz9B/e+SZPYAycifVJNgJ89O/jeNyDeU1bm/nqIiNRO8wE+ahTQ0QE0Nrq/JiIiNdN8gEsSMG4c\ne+FEpD+aD3CAAU5E+sQAJyLSKNUEeFmZ8wGekMAAJyL9UUWANzYCbW3AmDHOvZ89cCLSI1UEeFfv\ne6D7YPaFAU5EeqSKAHdl/BsQAV5eLm4IQUSkF6oJ8HHjnH//8OFAYCDXBSciffGKAAc4jEJE+qOK\nAK+oYIATEQ2WKgL83DkgPt61YzDAiUhvvCbA4+PFioZERHqheIA3NgJ2OzBypGvHiY8XvwiIiPRC\n8QCvqADGjnV+DngXBjgR6Y3iAe6O4RMAiIvjEAoR6YvXBHhoqLgtG9cFJyK98JoAlySeyCQiffGa\nAAcY4ESkL6oI8LFj3XOsuDieyCQi/VA8wCsq3NsDZ4ATkV4oGuAdHUBtLRAd7Z7jcQiFiPRE0QCv\nrgYiIgBfX/ccj0MoRKQniga4O09gAhxCISJ9UTzA3XUCEwBiY4GqKnFpPhGRt3M5wBsaGvDoo48i\nOTkZKSkpKCgocPi97jyBCQABAWJNldpa9x2TiEitfFw9wAsvvIAHHngAn376KTo7O9Hc3Ozwe8+d\nA6ZOdbWC3rqGUaKi3HtcIiK1cakH3tjYiPz8fCxbtgwA4OPjg5CQEIff7+4xcIDj4ESkHy71wMvK\nyjBmzBg89dRTKC4uxrRp07B+/XoMGzas1365ubndj81mM8xmM4ChCXAuakVEWmSxWGCxWAb1HkmW\nnb+X+5EjR3DXXXdh//79mD59OlasWIERI0bgjTfe6GlAknC7JmQZGDFChK2ra4Hf6Pe/F78Y3nnH\nfcckIvK0vrLzRi4NocTGxiI2NhbTp08HADz66KMoLCx06L2NjYDB4N7wBjiEQkT64VKAR0ZGIi4u\nDqWlpQCAnTt3IjU11aH3unsGShdezENEeuHyLJR3330XTzzxBKxWKyZMmIAPP/zQofcNxfg3wMvp\niUg/XA7w9PR0HD58eNDvG6oAj4gAGhqAtjYxL5yIyFspdiXmUAW4wQDExADnz7v/2EREaqJogLvz\nMvobcSohEemBYgE+VCcxAZ7IJCJ98LohFIAnMolIHxQJ8I4O4OJF993I4WYcQiEiPVAkwKuqgMhI\nwMflOTC3x4t5iEgPFAnwoRw+AdgDJyJ98MoAZw+ciPRAsQAfqimEABASIhbLamwcujaIiJTmlT1w\nSeIwChF5P68McIDDKETk/bw2wNkDJyJv5/EAl+WhvQqzC3vgROTtPB7gDQ1iwalB3DrTKeyBE5G3\n83iAe2L4BGAPnIi8n9cGOHvgROTtvDbAY2PFJft2+9C3RUSkBK8N8MBAcdf7ixeHvi0iIiV4bYAD\nHEYhIu/m1QHOE5lE5M28OsDZAycib+bRAO/oAGprh+5GDjfjrdWIyJt5NMCrq8WNHHx9PdMeb61G\nRN7MowHuyeETgEMoROTdvDrAeRKTiLyZRwPcE4tY3SgqCrh0CbBaPdcmEZGneHUP3GgUIV5V5bk2\niYg8xasDHOA4OBF5L7cEuM1mg8lkwvz58/vdjwFOROQ+bgnw9evXIyUlBZIk9bmPLIsAj4tzR4uO\n44lMIvJWLgf4+fPnsX37djz99NOQZbnP/a5cETcbHjnS1RYHhz1wIvJWPq4e4MUXX8S6detw9erV\nPvfJzc1FTY1YIXDPHjPMZrOrzTosPh7Iy/NYc0RETrFYLLBYLIN6j0sB/uWXXyI8PBwmk6nfhnNz\nc/Hpp2JpVw9mNwD2wIlIG8zm3p3b119/fcD3uDSEsn//fmzbtg3jxo3DokWLsHv3bixduvS2+5aV\nAePGudKac7geChF5K0nub+B6EPbs2YPf/e53+OKLL3o3IEmQZRnPPgskJwPPPeeO1hwny0BQkOj9\nBwd7tm0iImd1ZWd/3DoPvL9ZKEr1wCWJwyhE5J3cFuD33HMPtm3b1ufrSgU4wKmEROSdPHIlpt0u\n1kFJSPBEa7diD5yIvJFHAvzC
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 25
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"This means that it is prudent to pick values for the $\\mu_i$ that are close\n",
|
||
|
|
"to the decision boundary, since using $\\mu_i$ centered on outliers may give\n",
|
||
|
|
"rise to bad generalization.\n",
|
||
|
|
"\n",
|
||
|
|
"This is, in effect, what support vector machines are trying to do (among\n",
|
||
|
|
"other things)."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 23
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": []
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"metadata": {}
|
||
|
|
}
|
||
|
|
]
|
||
|
|
}
|