nvidia-AI.summerschool17-no.../teaching-lw/118-parametric-gaussian-classifier.ipynb

575 lines
334 KiB
Text
Raw Permalink Normal View History

2017-09-22 17:26:46 +00:00
{
"metadata": {
"name": "110-parametric-gaussian-classifier"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Title: Parametric Classifiers\n",
"Author: Thomas Breuel\n",
"Institution: UniKL"
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"\n",
"from numpy import *\n",
"from scipy import *\n",
"from pylab import *"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 1
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"coords = array(meshgrid(linspace(-5,5,200),linspace(-5,5,200)))\n",
"coords = transpose(coords,[1,2,0])\n",
"grid = coords.reshape([prod(coords.shape[:-1]),coords.shape[-1]])\n",
"def G(a): return a.reshape(coords.shape[:2])"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 2
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Gaussian Densities\n",
"=================="
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's start by defining a function computing a Gaussian density:\n",
"\n",
"$$ p(x;\\mu,\\Sigma) = ((2\\pi)^{d/2} |\\Sigma_i|)^{-1} e^{-\\frac{1}{2}(x-\\mu_i)\\cdot \\Sigma_i^{-1} \\cdot (x-\\mu_i)} $$"
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"def normal(x,a,sigma):\n",
" invsigma = inv(sigma)\n",
" d = len(x[0])\n",
" delta = x-a[newaxis,:]\n",
" raw = exp(-0.5*sum(delta*dot(invsigma,delta.T).T,axis=1))\n",
" result = raw * (linalg.det(invsigma) / (2*pi)**d)**0.5\n",
" return result"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 3
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There is a second kind of function we need, namely a function that generates random samples\n",
"from a normal density. \n",
"\n",
"We don't define that here, we just use a library function."
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"from numpy.random import multivariate_normal as N"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 4
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A Two-Class Problem with Normal Densities\n",
"=========================================\n",
"\n",
"Consider a two class problem:\n",
"\n",
"$p(x|c=0) = {\\cal N}(x;\\mu_0,\\Sigma_0)$\n",
"\n",
"$p(x|c=1) = {\\cal N}(x;\\mu_1,\\Sigma_1)$"
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"mu0 = ones(2); sigma0= diag([3,1])\n",
"mu1 = -ones(2); sigma1 = diag([1,2])"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 5
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we generate two training sets, one for each class: $S_0$ and $S_1$."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"samples0 = N(mu0,sigma0,1000)\n",
"samples1 = N(mu1,sigma1,1000)\n",
"samples0[:5]"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 6,
"text": [
"array([[ 0.99086233, 0.65984265],\n",
" [ 0.02339039, 0.50630836],\n",
" [-1.15234377, 0.89216093],\n",
" [-0.77827059, -0.02713438],\n",
" [ 1.65490177, 0.15463286]])"
]
}
],
"prompt_number": 6
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's plot these samples in a scatterplot."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"figsize(6,6)\n",
"plot(samples0[:,0],samples0[:,1],'+')\n",
"plot(samples1[:,0],samples1[:,1],'+')"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 7,
"text": [
"[<matplotlib.lines.Line2D at 0x2b73910>]"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAFtCAYAAAAeUTF6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvW2MHNd97vlrm5RaIm0WFcXTsRn30aVyM9J6Pa3EirkX\nAvps3kRfJE7fjRdRAsHTgI1EHwRnAgSIgF2AZ4B7E+5NgIwDf1CwC2x1lnsjA/nQNgSDRLLJaUAx\nxo6x7jaEZAyL8GmL9u3x8po19lBpSqRnP5w+Vadqqnt6XjlD1gMMpru66pxT1V3P+dfzfzmljY2N\nDQoUKFCgwJHDu+72AAoUKFCgwM5QEHiBAgUKHFEUBF6gQIECRxQFgRcoUKDAEUVB4AUKFChwRFEQ\neIECBQocUewJgd+5c4ennnqKX//1X9+L5goUKFCgwBTYEwL/7Gc/y5NPPkmpVNqL5goUKFCgwBTY\nNYFfu3aNL33pS3z605+myAkqUKBAgYPDrgn8D/7gD/jTP/1T3vWuQk4vUKBAgYPEsd0c/Oqrr/K+\n972Pp556Cq117j6FrFKgQIECO8NWqsauzOYvf/nLfPGLX+Sxxx7jt3/7t/n7v/97PvnJT+YO4qj+\nXbhw4a6PoRj/1vv9wz/stp/i+t9LY78Xxj8NdkXgf/zHf8ybb77Jt7/9bV555RV+8Rd/kb/6q7/a\nTZMFCuwIYx4AC9wHuJ+/+11JKFkUckmBowStk5t/cTHZLqX9K3A0oPX9+33tGYHX63Xq9fpeNXdo\nII/4L+OwjX+7N9uk8e+WgLP7KTX9uKbFYbv+28FRHjsc/fFPg9LGtGLLTjsolabWcwrc+1Bqf4hy\nt+3u17gK7A+yk/eFC/b1vfT0NA137qmEUqDAUcW9ctMfBA6DZHEQT09HAQWBF9g3uBv9ILTm3bZz\ntwnpKOEwEHgBi4LAC+wb3I1+MFrz3rdZ4Gjgfv7uCwIvUKDAljjMETt3u/+7iYLAC+wptrrR7+eb\n7Sij0JwPJwoCL7Cn2OpGPyoEPknnLTTgAocFRQWqAltCG323h3DgmJTddz9n/kExeR0mFAReYEvs\nlMCLG/3eRPG9Hh4UEkqBfcNRu9En6ffu87zPjtp5Frh3UBB4gVxoo2PLe7GTMJYUEink9trSe0ty\n+6lB+5r9JP2+cOIVOAwoCLxALrJEraTacVtHhcAL5+TeoLiOB4dCAy9QIAeTCKggp8m43528B4nC\nAr9PsBuraLuSietvLzXj/Uok2Ur3zkNB4AUOCwoCv09w0I+1e534sVV7Oz2/vRhnIRkc7kzNexkF\ngRfYEtroHVnhB4m7SaJ+3/crmReZmncHBYHfwzgsVpHf114Q3H6NfS/avV8JvMDdQUHg9zCyRL0d\n4t7LMML9IvC9nqC2c8y4vo3Zfr/3GooJ7OBQEPh9hO2Q516GEe4X9uOx3ZHyVpKI37dP2q0WCJE/\nvvsF9+M53y0UBH6fwC2scDdwWKScaTAtgfsQYnwCUCGpFNhPFAR+j2MvyHO3Dsx7fUGHSX3vB4EX\nk0IBh4LA73HsBXke9ggU2H0seBhaKaTTsdvCEIIAer10H+PklHHj2A9NvJgUCjgUBF7gQHGQJDEt\nKfnE7CY4//92Jj0nVbmnnlbL/hdib5KO9uv6FQR+NFEQ+H2Ew3CD3g0Cn5ac9spH4BO11ps18nF9\nbyXFKGX3Oex+hAIHh4LA7yPcTze6DYGU9rWensCz++027d9JMo58x7W31RjdsdknhN3gKDmXC+Sj\nIPACe45pCHM/KwpqDRpNZ1Gmtm3VnzGbo092Gk/e7UIUwdycJXE/Cmg7JJ5HstOezzTjLLInjzYK\nAi+w57ibBO5I6fxFqNfttk4nTXrZxCJfr3Yx3D6Z7wRRlBBio5FPjlly9sfn+s4j2b26doXuffRR\nEPhdxP14A+1nXRU/e/TKrUXqEpAwj0Sp/D6zevVeShSu/XHaep6FP02/e/Wb8X9/99vv8F5BQeB3\nEUedwLNFnLbSU7UGpTUSuS+aq589qrXNHpUSlB5/TFavduNw2vW48eUVsBp3DYJg6/7b7a1DFvfz\nt3KUf4f3MwoCv8+xm0kka8FtpadKad2Kzhje64zFLBnr0bZJBOqP2xgb/+22TypZm/fa/bkxTGtN\nSwnLy+OlFn/f3aJwXN5bKAj8gHHYbqDdEOe0SSp5hbE0oE3aYt5pxEfeJGKU3LYM4vTvafraar/t\nYjicbr/dTnSF4/LeQkHgB4yjfgONc/plHW8+8gpjaUCKvRlPHqEJcjZugSBIvo9OJ3ntzs2YZJtz\nOrp9HbajKWefGFy8+G7CDQvcXygI/B7HtKFpMN1TwDinX3afrTBOMzYGms3tk1T2PHdCcgsLyWtH\n1nFYorYT1oUL9vP5+URucaTr9vPD/bbzZLXVE8Beo5gIjj4KAr+LOIgbaFxyynafAvIcdXlW6lbI\nSibZcfhkOMkKzYuNdse68ewGTh4ad62y12wn13SaydB9th+yW0HgRx8Fgd9FHMYbaBxx5sUo+5LC\ntJikeU+TPbkV8e8Gfp9bWcO7tfazmNTfUZfdCuwfCgK/B7Edi21SeNxWyJLObhyRxoAKx2dP+k8A\n/jZnueadp9t3aSktj+T1L6WVRCbp++NIey/C/ZrN7R9ToEBB4PcgtmOxTSKbrSaC7ZL/UlsTdWVu\ne80mhAYYkz05TgryLe9xYX9hOB2BT1qYwfU3DbZ7XbYz8e02UqfAvYWCwAuMT0Cp6VQGoyM0J3Vs\nlxSiYHN7LsSw3Ybe6fzsyWks7Yn9RsnrLJmNa3sv63hP8yRx2Am8mAQOJwoCvwcw6eaaNiIk11Gn\nNU6TTvVndErLdhKIq30NNiRvktUb9z0KMdQKLiiQKGtZ6/wsyXp983jzZJXlZbhyxVrf/T7UanZM\nb70FX/1qfgif3/ZOYrm3gv9EEYb7R6rjIo92099WSUwF7g4KAr8HsBMC3+6NN6l9J2VcuLA5SkMb\nTag1QqRXtw9qEq1lOpZab253GmelX3jKJ+Dh0L5eXLRZjmBT1v39fLhFFyad77QYFylTLtvX3W46\nptxhO5Elkwjc/z+uUNZOURD44UFB4EcIe1kIKtQamXPnBTU9srwzhFsO0DratD2bpJOFFBJtJKpp\n3/ur22etXOFZ+1tNSEtLiTQyzuHoJpaXX7afGWPrjYwLfdxLYvPbdnJMFNn+pbT/azU77vn5vY8s\nySPq7fYxTlrbj2XiCuwMBYEfEPaCfPOkC9iZBWfQ5MkjCw2Z2u4Trg8lFUttjQ4lejSGZ59NnIG+\nnDLuhvfHqjWEnj7uI0uMkC7XmhdD7R/zwQ/mj6Pb3Sz/bDf5Zlyf/jZj7DXp9eDUqeSznSTujPVX\nBMmE5idFuf9bFefKQ94kBPZaufc7abfA3qEg8APCbgk8a62O0623aiObhGPQNOX2HZIA3UinSHfc\nGGIyF5bs8/qaRGYXL8Lly+ltfgx6XkKRT+q/8ztWj59UKnaaZc+mQZbEpbTyTRRZjd1p+U72Gae3\nj5MmtvreXVvZQmHueuxU8hgXodNsFnHpdxMFgR9yuCgNraFTml66yIXQILW1nkqLUAdjNAhFnjU+\nrn23fdKjtH/DOwt0XE1umBwHvbJi/2dlE5fW/uyz48lJa6t7d7v2mLk5u2+ek3Un5Ja1iLNas9ZJ\nP0ol1yWr2+e1O+6zrcbjv85OEpPOcTtOUYfDJqfcb5p8QeD7iLwqfLA98nX7Kg2yPioEpUkVgtrq\nB+us/7gtBVU0SiqUVmOzI/PG6M4p1JpOaTGutR3UJP4k4LfjCHzieWa68omx308IOi/u2/33E3F8\neUEI+3fhQlobz/a/nciQSU5BN1Z/P9eH2z5N4s5WZDTOkehvd++nOa+8EEe33b9e/mcHXb9lKxQE\nXmDPkFeFbzvIWnd1bHhddrmvaQncwaAZlg1Kq9TEYrTMdWymYCRoidFACdhQANQyhzki0NpazdvV\n6Z0+HZ+Dtufd7W4+1r3PWv0O
}
],
"prompt_number": 7
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Parameter Estimation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(means and variances)\n",
"\n",
"If we knew the parameters $(\\mu_i,\\Sigma_i)$, we could perform optimal classification.\n",
"\n",
"However, in practice, we need to estimate these parameters. We use the usual formulas for this.\n",
"\n",
"$\\hat{\\mu} = \\frac{1}{N} \\sum_{r=1}^N x^{(r)}$\n",
"\n",
"Here, $x^{(r)}$ is training sample $r$. \n",
"Remember that $x\\in R^2$, so $x = (x_0,x_1)$.\n",
"\n",
"The general definition of the empirical covariance is:\n",
"\n",
"$\\hat{\\Sigma}_{ij} = \\frac{1}{N} \\sum (x_i^{(r)}-\\hat{\\mu}_i) (x_j^{(r)}-\\hat{\\mu}_j)$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(diagonal covariance matrices)\n",
"\n",
"However, of special importance are _diagonal covariance matrices_.\n",
"\n",
"For these, we assume that only the diagonal elements of the covariance matrix are non-zero.\n",
"\n",
"This is also the same as just computing the variances for each dimension separately:\n",
"\n",
"$\\hat{\\Sigma} = \\hbox{diag}(\\sigma_1^2,...,\\sigma_d^2)$\n",
"\n",
"where\n",
"\n",
"$\\sigma_i^2 = \\sum_{i=1}^N (x_i^{(r)} - \\hat{\\mu}_i)^2$\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are simple built-in functions for computing the means and covariances.\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"mean0 = mean(samples0,axis=0)\n",
"sigma0 = var(samples0,axis=0)\n",
"mean0,sigma0"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 8,
"text": [
"(array([ 0.96306184, 0.99597855]), array([ 2.92315936, 0.99367374]))"
]
}
],
"prompt_number": 8
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Remember that the true values are (1,1) and (3,1), so these are pretty good estimates.\n",
"\n",
"Let's repeat this for class 1."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"mean1 = mean(samples1,axis=0)\n",
"sigma1 = var(samples1,axis=0)\n",
"mean1,sigma1"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 9,
"text": [
"(array([-0.99250469, -0.99305026]), array([ 1.06366716, 1.92246796]))"
]
}
],
"prompt_number": 9
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now plot the corresponding densities."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"figsize(12,6); subplot(121)\n",
"imshow(-G(normal(grid,mu0,diag(sigma0))),origin='lower',extent=(-5,5,-5,5),cmap=cm.RdBu,vmin=-0.1,vmax=0.1)\n",
"subplot(122)\n",
"imshow(G(normal(grid,mu1,diag(sigma1))),origin='lower',extent=(-5,5,-5,5),cmap=cm.RdBu,vmin=-0.1,vmax=0.1)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 13,
"text": [
"<matplotlib.image.AxesImage at 0x3236d50>"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAr0AAAFKCAYAAAAQW+fkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvb2uPc1W3vuM6hXAlnWOdc6xxC1AYNlyQLS3CEmQkEgt\nEiOHlh2b0FsObAcOuAGuAAGCFDaYlFuwZMtChAhtW7yra5xgfNSo6uqePT/W11xj6P2/q2d1VXXP\n7tnVv376qSpiZkZGRkZGRkZGRkbGE0f56B3IyMjIyMjIyMjIeOtI6M3IyMjIyMjIyHj6SOjNyMjI\nyMjIyMh4+kjozcjIyMjIyMjIePpI6M3IyMjIyMjIyHj6eHnrDRDRW28iIyMj403jOw1yk212RkbG\nV4+9NvvNoRcA/vfPf/4em9nEf/jpT/G7//7ff8i2Pyq+23f+bt8XyO/83vGLP/rRh2z3I+PnH9Rm\n//SnP8W//2a/7fzOzx/f7fsCH/udf3TQZqe9ISMjIyMjIyMj4+kjoTcjIyMjIyMjI+Pp46mh9yc/\n/vFH78K7x3f7zt/t+wL5nTOeN378Dc9zfufnj+/2fYHP+53prachJqIP8/RmZGRk3Bu/+KMffbuO\nbB/l6c3IyMi4N3500GY/tdKbkZGRkZGRkZGRAST0ZmRkZGRkZGRkfINI6M3IyMjIyMjIyHj6SOjN\nyMjIyMjIyMh4+kjozcjIyMjIyMjIePpI6M3IyMjIyMjIyHj6SOjNyMjIyMjIyMh4+kjozcjIyMjI\nyMjIePpI6M3IyMjIyMjIyHj6SOjNyMjIyMjIyMh4+kjozcjIyMjIyMjIePpI6M3IyMjIyMjIyHj6\nSOjNyMjIyMjIyMh4+kjozcjIyMjIyMjIePpI6M3IyMjIyMjIyHj6SOjNyMjIyMjIyMh4+kjozcjI\nyMjIyMjIePpI6M3IyMjIyMjIyHj6SOjNyMjIyMjIyMh4+kjozcjIyMjIyMjIePpI6M3IyMjIyMjI\nyHj6SOjNyMjIyMjIyMh4+ngI9K7rin/+z/85fuM3fuMR1WVk3BkMcH2Cf/zRBzLjSSPb7IyMjO8Y\nD4He//pf/yt+5Vd+BUT0iOoyMobg6/49DSuygu8t/zIy9iPb7IyMjO8Yd0Pv//yf/xN/8id/gt/5\nnd8BpzKV8RbBEPg7+++poO+K7+3/PnqfMz5zZJudkZHxXePl3gr+3b/7d/hP/+k/4e/+7u928/yH\nn/7Ul3/y4x/jJz/5yb2bzXiWOHXTzRvzdXESfFPlm8bPfvYz/Owv/uKjd+PN4kyb/dPQZv842+yM\njIxPHD/72c/wFyfbbOI7HvX/+I//GH/6p3+K3/u938Of/dmf4b/8l/+CP/qjP+o3QIT//fOf37qJ\njKeIg59YKk0fGATscm8CscUv/uhHT6OInm2zf55tdkZGxheNHx202XcpvX/1V3+FP/zDP8Sf/Mmf\n4P/8n/+Dv/u7v8Nv//Zv4/d///fvqTbjmeLp7AbPFBcU4VSCny6yzc7IyPjOcZfSG+PP//zP8Z//\n839Opfc7xuFP6AsC71uoeuT/+0JxsL9f8vvcFs+k9MY4arNT6c3IyPiq8WZK7xjZE/jZY+/G/1mA\n4AH7wQ+qZ1MvAfSIet/zGjuypcy+T17/Xy2yzc7IyPhO8TCld3cDqfQ+SfAn999+9v17VNDntR3Q\nc85186xK716k0puRkfGV492U3oxnitHv+d43/WuH3vouUHLFcXlvC8IeGH5WSM/IyMjI+FaR0Juh\nMQDLuyhbn2lUh/fc3r0QeHJfT1kqHgmkV9F4RkZGRkbGu0ZCbwY+xhrwmewI77wv76bAfpLxemfH\nNtXfjIyMjIx3joTe7xgbCHkr4DuCrgdu89PA88l4xP4+DBp3ztGbgvlkmwnBGRkZGRlvHAm93yLe\nA3Indd4Nd2fLvyG03x17MPceoH4WJGfnbs8a8UDYHj92VScEZ2RkZGQ8NhJ6nz3eY3IIrm9Q58dO\nakEP3D6/hYp55oHiLrX2QKV/E1U21d+MjIyMjLeNhN5nize3Llw7qsJRVW8xqQU/Zjjcx33JB+3P\nDfD8JmD8XnaIsJ1vNBFGRkZGRsbbRULvU8V7qLr+vzvreYt9ZVVoP1s8cp+uhb8L22a6gSdndgi8\nDfi+Wd0ZGRkZGd8tEnq/fLzlyAOPqPv+Oh5pNfDgDx49gq6fZOL8caArVOG943DDJBjj/hHhYaD6\nlnVnZGRkZHyLSOj9qvEmwHaPdWGv7LkKjxXaS2rlZ1R3L8S9+3wIpMeWCj6lmj7AxjB+x0eqtbHu\nVIEzMjIyMk5EQu+XjLexBtwFYnfZHu74Ph+t2N4Tbwy+u8WYFHwvxd6oDmfKTup4uE3B7A+3WDQy\nMjIyMr5bJPR+hXgj/+vt0HV92ZssCvcC7QePAHF/HFgMzhybXQvFsfeZqRxUOtnuNVaDzTm5wUZx\ntE9pe8jIyMjI2ImE3k8b8fXyI8CNuz837wtdqiO8WidP2i90NdSe2BdPjysPvv/Feoa0WT2PSN/A\n3xXfdU9BPTq+O7DpQExxFw9A0o71TePsPniUhmHfE4AzvltsLvl727d8kZLxRJHQ+ynj0a/sH2hd\nuAi8w+vsi3Vfq/6e2JdZ+tF2rqrnAXUc1T0D3zP1zF7xnxqybK4Exz+ES3aIyT5eBbCPtikYSOfd\nOuN7xfTd1r3tmz5U56WU8QyR0PuZ4tGv42+q7xpAvmKIsDOv46/e39tg/jMOayajLVz7faI14ELZ\n2Wv/2TmZWiL2zvPBKBE3jbZwp3XisL5H2CgyMj4+mB96l7h7m4S8tDK+TiT0foqwFuXRNoZTUuvh\nx6OVUxDaA68u/Yz0cGLf7zpcnwt8iXHDzG1nvwOpWnOU37wM19ghWPd7Us8YPNofDvL2m8DWL3NL\n7L3Hzcj43MHDh8/Vcu07JzIyPmMk9H6GeCTwXm0XeKCqe7bj2ak8/r+LcbVy+xbTJt8bdIFJh3BA\nPm1huHC3jHaEG8B3um9j3DSE2QM9v53qe3s1GRnvFR+h6t4SMx9xXmIZnzESej8sHuXbvRV0j8uc\nGm3hEuSereMgzymgfSdbxNvFXm+3/fxnbigNjGeQP7zuP2NHGM/3KSvEGQvESetB3Me0PWQ8YXwV\nyL0U4/dIC0TGZ4mE3neN0Aw8Yqazocpz2z0qdMFeMO7z5jscbefMNvs4B94AcEK5Pb0/HxHmETgr\njzDA5UJeAjEfWCbYeXNWdm5HCOuB+W/4ogVip+Ncty9pe8j4PsFh4TO1So+MtEBkfJZI6H3P+Agb\nwxUq6KGqeq+qe2F/59s+8z3PWio+oaWhC6O+k7cCqqOZdlgv6/bOKR9ZHsxGsDs9cazoPPjKdn0D\nmzzX2RhGKekRtoc768nIuDIYn+yl0xvGqaYiI+ONI6H3veLukRniTf4MCF7KO3oxL6i4Z1XePXjV\n2AXrm6D3JBRfJaEw6A30Fj4FtJsXggdZef8VP0Ehf6f8jn941yfcQejkWB75gYe72nj+pxB89d3x\nWmA+qqreaZ3IyNiPUff4Jrw7je5YpAc4450iofdNI1oQbm3eztZx1joh62hWH+/VMeaLaWetEwd2\nhd10YNe64NvaK3fByrCzr7SX/86gXZQeldN+T6Z5PC/N17OquEflZqzMAXzH89rlHxI4UnRQiGmi\nFpvC3G2zy7Dd/nZPJ2lBSr7X9sB4kHUiI6PFd1J1z4ZfZaEdyCsu4y0joffN4goLwiPqOAHWFz2y\nuyrsTt079e3bJPa+z6PS235dVmsfcX6uCd5pzPdUxV71nZeOcDkh50365dgYE65Vf49UXy+7p/4e\nTIJx2oIwKNE3vT8dv09Gxn3xkcB7y2bf+2c/3mLS9pDxVpHQ+yZxD1DFm/YZcLuUN9oY9sD1mrq3\n6ectC9dALGMDMJpOO+lb1XnclxPn5NEThFwcHWDyXTav1yPyziCZ5/YJt0CgX9eLrSGdAS7j3vdV\nHg2VNlN2LT3u0yxdVp7w/mod
}
],
"prompt_number": 13
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can overlay the original samples on top of this.\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"imshow(-G(normal(grid,mu0,diag(sigma0)))+G(normal(grid,mu1,diag(sigma1))),\n",
" origin='lower',extent=(-5,5,-5,5),cmap=cm.RdBu,vmin=-0.1,vmax=0.1)\n",
"plot(samples0[:,0],samples0[:,1],'r+')\n",
"plot(samples1[:,0],samples1[:,1],'b+')"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 14,
"text": [
"[<matplotlib.lines.Line2D at 0x3422a50>]"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAFtCAYAAAAeUTF6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd8XEe5/r8zu2q2JMtyt9wdO+62bMvqJYZQwqUmPzrc\nC6GElgYECC0EUiCEELjk0rmEAAmEcOkhlNjqcu+xnbhJcrdlq0u7e2Z+f8zZs7vSrixbchIl83w+\nslfnzJkze/boOe8+7zPvCK21xsLCwsJixEG+0AOwsLCwsLg0WAK3sLCwGKGwBG5hYWExQmEJ3MLC\nwmKEwhK4hYWFxQiFJXALCwuLEYohE/j58+e57rrrWLhwIYsWLaK+vn44xmVhYWFhcQH4h9rBTTfd\nxDXXXMPjjz9OKBSis7NzOMZlYWFhYXEBiKFM5GltbSU3N5eDBw8O55gsLCwsLAaBIUkohw4dYsKE\nCbzvfe9j5cqVfPCDH6Srq2u4xmZhYWFhMQCGFIFv2rSJwsJCamtrycvL4+abbyYzM5M777wzcgIh\nhmWgFhYWFi83XIiehxSBT5s2jWnTppGXlwfAddddx5YtW+IOYqT+fPnLX37Bx2DH/8KP4+U4/pE8\n9pfC+AeDIRH45MmTmT59Ovv37wfgn//8J4sXLx5KlxYWFhYWg8SQXSjf/e53ede73kUgEGDu3Ln8\n7Gc/G45xWVhYWFhcAEMm8OXLl7Nx48bhGMuLEhUVFS/0EIYEO/4XFiN5/CN57DDyxz8YDCmJOagT\nCDFoPcfCwsLCwmAw3Gmn0ltYWFiMUFgCt7CwsBihsARuYWFhMUJhCdzCwsJihMISuIWFhcUIhSVw\nCwsLixEKS+AWFhYWIxSWwC0sLCxGKCyBW1hYWIxQWAK3sLCwGKGwBG5hYWExQmEJ3MLCwmKEwhK4\nhYWFxQiFJXALCwuLEQpL4BYWFhYjFJbALSwsLEYoLIFbWFhYjFAMeUm1Fxo93d0v9BAsLCwuEalp\naS/0EEY0bARuYWFhMUJhCdzCwsJihMISuIWFhcUIhSVwCwsLixEKS+AWFhYWIxSWwC0sLCxGKCyB\nW1hYWIxQWAK3sLCwGKGwBG5hYWExQmEJ3MLCwmKEwhK4hYWFxQiFJXALCwuLEQpL4BYWFhYjFJbA\nLSwsLEYoLIFbWFhYjFBYArewsLAYobAEbmFhYTFCYQncwsLCYoTCEriFhYXFCIUlcAsLC4sRimEh\ncMdxyM3N5fWvf/1wdGdhYWFhMQgMC4E/+OCDLFq0CCHEcHRnYWFhYTEIDJnAm5ub+etf/8oHPvAB\ntNbDMSYLi0uABq1e2B/s/W/x/MI/1A5uueUW7rvvPtra2oZjPBYW/aE1I4IcL2acwqafLIaOIRH4\nn//8ZyZOnEhubi7r1q1L2O6OO+7wXldUVFBRUTGU01q8VJHwG9yFSVFWVqHKSi/51EM9/qKhFZBA\ncrRS5MsS69atG5BH40HoIeget99+O7/4xS/w+/309PTQ1tbGtddey8MPPxw5gRCXVVrp6e6+bH1b\nXC7oBJx86feJ/667CX3+9hfs+OHFy4fYU9PSht7JunXwEgwKB8OdQ/oed/fdd9PU1MShQ4d49NFH\nWbt2bQx5W1hEoCM/Wsf+zgiRSJ43JLg+9rrFx0VGrS8lDFkDj4Z1oVjERZQ2PJxShaysQlZVAeC/\n+25vuyotHdQ5hnr8848E31yE94/FywzDRuDl5eWUl5cPV3cWIxZ6AC0bZNXwEbgqLUaVFnvnDd3+\n2ahhqMt+/AXxfCUq+yZPL/N5ZWUlqqzssp7jgli3LhJ5f+Urke0VFS9JOSURhjUCt3iZ4nK6RIaD\nSF8oXGjsl4to+51XDKt+bh7CLzCB9yXqKKPEywmWwC0uAX2/yscn77BcMiipIm7UPviHgiotGWTL\n+DKEKimOP4bLKU8M6EQZzvP2ec9W6nzJwBK4xeAQQ26DI9awXBL+CSP0+c9FuvD6vcgIvg/ZqpKS\nAaWbqAPjbjUPgDj7hvrF4oJkmeAEcbXuoRCvjvtysA8KWVkZeQjfdZe33TyEXwTR+MsUlsAtLoBL\nJNi4fbivB20rHajd8+m+uFznukhy1/EOuRRS70PmMV3E70+VlcUQdejzn7+E814mWAK3sIiDS9C2\n+8sl5nhVWmKitYuSOl44i5ysqk441oH29cNA7+GiZZI48s9wSC19x2gllhEDS+AWsYiTeKuslJSV\nDS6ZqMpK+jg7Phe7vzSBA+USk5XiMiVQZVUVuqToove5o0IPhgQTjV0IBk3K8foYUnI06sGZoJ+E\nn6HF8w5L4BZcKNqtrPINTOAXS6IX0V4MlwtFXWQ/Wsc/Rg6GHLX7YBmgxUAkm5DYB0nM0dfsYh4G\ng+znBde8LTxYAn+54hKSkon7GCgxCLKqElUSHbXFb9+f9BK0q6pGlxTH3TfwOAeG/J/vI1pNUTb/\nvd/wtiv3XLK6JrLP7VOVFKMHklMSROIigQNlwMg93jEXivSHSx6J7sdKLC8aWAJ/uWGQro/KSkll\nlQ+Au+5O9raXlTqUlToX7kNrzxkiK6vN69gGiH6H903aJbAnVlXjFMeRMIaomYvzrTifvc3ry/nM\np2P2e+fsuy8cqV8sscVp3v+auKf0+o6T2IzpL94YhuZA6XdwuI8E7/dFMdHnZQJL4C8LaPNHVVrG\nYKPtsjIVI5t84fOBQRDkhaL6yLZ+0Xa8vhNG5HEkBt13/6VAAyrO60TtRP/NfTlN92PMC4/hgg+C\n6P0X6yhJ1P4SiTzOQ+BFMdHnZQJL4C8HaG0m1VxS8sklywuRdx/dVlZVI6uqAfDfc6+3L67koC/c\nv6iqjkgYX7/PO5UqKeovp1xEJC6qa5E1tW6/34x0kZmZsB9VXBSV6OtDphc6dXT7AR0qsaTY94GX\nUGqJkTq8fxLDex+DaNv/4AtG4xaXF5bAX7IYgg2vT+JwzJgE0fQA/avSkiinhsb53Gf6HJ7g+AQJ\nPF1ShFNSRPiB4nzmkzHjFdW1cZwh8WSaPiguRBUXAuBojbot0m+iRGT0+0LrBOeOgpcATHTNRH9i\n79suJnnaP0mq4yUrL6ZGSkzbi516r5Hr3Yk+Qrz4Jvq8hGEJ/KWGPpX/LqraXgLHR2tr36gxESsm\ncF8kcnRc4LwDWQv7nsdXXYPjEnH0eC7uIaYHPKd37j5VmKPP3ZfMtRADX7MwUV6I2PtePxFLsn3t\nlHFdLoN2p1zYStgXMbNttXZrq9uo/HLDEvhLAQkcJX2nsMcn7vDX4MFIJPHbiUT7opwaXtt+DhId\n07bv9ngPBF1UOAjN/OK/gahE/fY7j9OfAN3jZE1tzINkIAqLkHsfRCcsdbztXOC4WJeLjiePDNZV\nEu7nYiUW61p5XmAJfMQi+o97cEQVU8r1Am6UyiqfcaFouOueFK99WWmIslKnD7FGJ8b6vtboKM1Y\nVtdEpJCYUyci7P66si4uBBSiug5ZUweA775vRR4YxQXo4oLBXpZIv0X5CFdn1/2i+ViImjpkTT0g\nYs5NU1Mk0hWAThDBChH3wWdIPWZUXvt+70cPkPAUgsg1jTqkr2YfffqERButdXv/JEQk1zLY/i0u\nFZbARxwSRawDQ5WWunLKwMQd3ldWGqKsJERYc/7C7b3EEsJApJ1onDryE2f8cWdVxpM0vAdDPk5x\nvte3c9vNEbniYiYAee9HIGvNAyHcr6ipNw+DvocU5eMUudsbG71x+x79LUyfBpgHCYgE+vilzJaM\nYdHEu6ORyH4Y008U0Q7oYtF92iSqmxLnW57XvyXx4YQl8BGFi5cFLlSbJP5p4p8j4ezCuFFzZJuo\nrolykHzT60sVF8WSWz/S7ft+E8k9LmkrlWDfwBDxHkDuWGRNHU7Rmr5HxPY9PQfnUze5mzXOp2/2\nWvrue8AQfd/IVau40Wxc418fqUVU10ScPIkm6iSQMAZ2swwg2cQcFP5WkGjEiaAtiQ8zLIGPCFw8\ncYehykoHrE1y4XNoykuC/ae0
}
],
"prompt_number": 14
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"How do we take advantage of knowledge of knowledge of the densities?\n",
"\n",
"$$ P(c|x) = \\frac{ p(x|c) P(c) } { p(x) } $$\n",
"\n",
"Recall that any given $x$, we decide that the class that maximizes the posterior is the class that we want. The decision is independent of $p(x)$. Also, we have equal priors, $P(c=0)=P(c=1)$.\n",
"\n",
"So, we just compare:\n",
"\n",
"$$ g_0(x) = p(x|c=0) $$\n",
"$$ g_1(x) = p(x|c=1) $$\n",
"\n",
"We call these functions _discriminant functions_.\n",
"\n",
"The _decision function_ is:\n",
"\n",
"$$ D(x) = \\arg\\max g_c(x) $$\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's plot these decision regions.\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"imshow((G(normal(grid,mu0,diag(sigma0)))<G(normal(grid,mu1,diag(sigma1))))-0.5,\n",
" origin='lower',extent=(-5,5,-5,5),cmap=cm.RdBu,vmin=-2,vmax=2)\n",
"plot(samples0[:,0],samples0[:,1],'r+')\n",
"plot(samples1[:,0],samples1[:,1],'b+')"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 19,
"text": [
"[<matplotlib.lines.Line2D at 0x419abd0>]"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAFtCAYAAAAeUTF6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvX1YVPed9/86Z4CogBFEEUSDTyimEM1MSowBBm/tQ1JS\n15ht9qZNY9qEPqbd3t0mu5uID722adpfNrHb7dLeW5N02aab6KahySaVOzAYNSQzjYE2ClFExUEM\nCPIgBpw5vz++55w5M8zwIPiAfl/XlSs4zJxzZoD393Pe38+DommahkQikUgmHOrlvgCJRCKRXBhS\nwCUSiWSCIgVcIpFIJihSwCUSiWSCIgVcIpFIJihSwCUSiWSCMmYB7+zsZP369WRmZrJ06VLefvvt\n8bguiUQikQxD1FgP8J3vfIc77riDl156ifPnz9Pb2zse1yWRSCSSYVDGUshz5swZli9fTmNj43he\nk0QikUhGwJgslCNHjjBjxgw2bNjAzTffzIMPPsjZs2fH69okEolEMgRjisDdbjcrVqxg79693HLL\nLXz3u99l6tSpbNmyJXACRRmXC5VIJJJrjeHkeUwReFpaGmlpadxyyy0ArF+/nj/96U9hL2Ki/ldS\nUnLZr0Fe/+W/jmvx+ifytV8N1z8SxiTgs2bNYs6cOTQ0NABQUVHBjTfeOJZDSiQSiWSEjDkL5Wc/\n+xlFRUX09/ezYMECtm/fPh7XJZFIJJJhGLOA33TTTbz77rvjcS1XJE6n83JfwpiQ1395mcjXP5Gv\nHSb+9Y+EMW1ijugEijJiP0cikUgkgpFopyyll0gkkgmKFHCJRCKZoEgBl0gkkgmKFHCJRCKZoEgB\nl0gkkgmKFHCJRCKZoEgBl0gkkgmKFHCJRCKZoEgBl0gkkgmKFHCJRCKZoEgBl0gkkgmKFHCJRCKZ\noEgBl0gkkgmKFHCJRCKZoEgBl0gkkgmKFHCJRCKZoEgBl0gkkgnKmEeqXW4G3vufy30JEonkAole\n/tnLfQkTGhmBSyQSyQRFCrhEIpFMUKSASyQSyQRFCrhEIpFMUKSASyQSyQRFCrhEIpFMUKSASyQS\nyQRFCrhEIpFMUKSASyQSyQRFCrhEIpFMUKSASyQSyQRFCrhEIpFMUKSASyQSyQRFCrhEIpFMUKSA\nSyQSyQRFCrhEIpFMUKSASyQSyQRFCrhEIpFMUKSASyQSyQRFCrhEIpFMUMZFwH0+H8uXL6ewsHA8\nDieRSCSSETAuAv7MM8+wdOlSFEUZj8NJJBKJZASMWcCbm5t57bXX+OpXv4qmaeNxTRKJRCIZAWMW\n8L/927/lJz/5Caoq7XSJRCK5lESN5cV/+MMfmDlzJsuXL6eqqiri8zZt2mR+7XQ6cTqdYzmtRDII\nxV2L5si+bK+XSMZKVVXVkDoaDkUbg+/xD//wD/zmN78hKiqKc+fO0dXVxd13383zzz8fOIGiXFRr\nZeC9/7lox5ZMHNTSMvzFRZft9ZILI3r5Z8d+kKoquAqDwpFo55h8j3/6p3/i+PHjHDlyhBdeeIFV\nq1YFibdEIpFcdEYZtV5NjMlCCUVmoUiGYzytCsVdi+KpA8BWWmY+rtmzRnSOsb5eIrncjJuA5+fn\nk5+fP16Hk1ylKJ66cRNHzZEddKzRWiBjff21yhWxX1BVFYi8N28OPO50XpV2SiTGNQKXSCRXP+O5\nCF8woUJtSZS4lpACLrloGJHapbAqNHvWZX29RHI5kAIuuWgYkdqlsCrGuhBc9ojyCueK3i+4hiyT\nUKSASySSYbmi9wukgEsk48NwkdpEsSqG2qi7IjbxJBKkgEtGgMudSL7j9IieO1ykNlGEb6iNuiti\nE+8yMlEW4WsB2cBEMizVnsTLfQmSK4hrefG60pARuOSiYURq4205XCwLQy17GXp6gcH2D3DlbuJJ\nrlmkgEvC4nInmpH31tJF5uN59tOjslNg/C2Hi2Zh9PQGWT5D2T9X1CbeFYbcI7h0SAG/RhjtH1W+\nI1ioHy8+dDEuS3IVcq3vEVxKpIBfI1zqP6rxzhu+WHnIkY5LXGzE18hNPMmVghRwybBcHzcw6teM\nd97wcMe70Nv2C7nO0PNIy+AKL/S5ipECfhUzXn9UZ3qix/3axpvLedtuPfe1KuZXdKHPVYwU8KuY\n0D+qyxUNWS2H8RC4i2VhjMdxpf8ruZRIAb+GGI24jEcWioH1nOMhcNZodzxv243GW9ZzRCLSuRVv\n66jPe7Uh9wguHVLArxE0e5YpOCNhImShXIzbduMzGs4SCTq3RbTV8gq01GTxHF3IrrWI/Fp7v5cT\nKeBXOVfC5tKVcA0XyojuGFKTI+aPq6Vl4/4er1WfXTIYKeBXOeMRpebZR2eZXIxrGPYcY7htV9y1\nKOUVKN5WVH2hUdy1kJrMSMZxD3nui2CpXAyfXS4KExMp4JJhGa3nfTkIJz7WgRJDiZN1gdFKy1C8\nraYNYistA4slEslOMb+2ZwXfcZRXiG+kJo9LzvrFElm5+ToxkQJ+DXElbC5dymswRGmk4mRuYA5h\niQyHuRBYPPRQiyXSuYe6RsVdK+wYT92Es6EkFw8p4NcQl9LzjnSuS5Uv7XInUjDK1yjlFWiFq8XX\nY7g+xV1rbmaqnjr8iLafQ4lt6OZpKJojG81TZy4E47JhO4H3JiQCKeCScWckEe/FumU3RGn3q3ms\nbi4TYuqpM22RIUXU24pf/56x+XghdwyaIxtNt060jPmonjp8+p2A8f2w5w/zmYQTWcVdizJOInuh\ndxqSKwMp4JeRa3HjaDTDIS4E08929+G7swh/cRFaaVlEcbIKpOqpQzNEUt98vOCfj9U2KVgR9vyh\n4uy3Zw2K1MNtAI+XeEvfe+IjBfwyciX8AY3VKhhNUY3irmV36SIKPDsvyi17UPGRRxQfaSzE6c0k\nd4THMHxw1VMXlM89VP8T62bpoM/A24oS4VzWdrv+tFkjjtQvxu/MlbA/Ihk9UsCvccayiFhfO5JU\nQeHjLsRfPHnQc8bjbsRJFQXo0TQLeTx1D5AMGbH4uTnsa6zXrbhr8f3qx+IbDz4yZMOsoP4n+tfG\nf4q7Fl9xkfl6Y1EY6vxRr+wKek3Y5+oZLhej+6IU8ImJFPBLzFW1cTTCHOdwZfmKex259oCdcqEL\niVXQgtIB3X34N+eM7mB6xB3xXCPNZgmpeB3R+4qPG/YpmiN7zIVBsunU1YUU8EvMpf4DChexjWUR\nGZTjHMZmCI3mwpXlK/YP0RyTR3S9Q16PkSoY8rr81AMQIeqOhJYxH1X/PKx+uPnevK3m922lZUF2\ni3kMo3x+BBGtWvYyStU+8T4aGlHR379zBf6itaO6dsm1iRTwq5xwUeOFLCKGQIZaDpGskhFdW4QM\nC/8ojwOD32deYdSIqiitWEXTeG/GNSqeOmzlFfiMNL7C1fg2f09cZ2mZ2WtG0fO0fcVF4pqGWBT9\nRWvBOOeDjwTsm9D3dpHu2qRtMvGRAn4ZuRL/gCJFwNZIN1zWxkjFxCjLV/ScZutrKu3fpoCfhbUK\nrBuFxushcmrdSCowR/LeIy12qkVIh3reeHCxjj3hLDvJIKSAX0YuWln0KCK20EVkOJ83KAL3to5q\nIzJSCqHirkWt3Mdu9zpWe8ILsrWq0hB+xV2LX79+1VOHpkfAxvs0XqM+8a/4H/1GxOsyF4fyihEv\nTtbPbawLsWGbSCSjRQr4VchoIrahBLe6rANnz2tA8EJQFXcHeUUJZpqdwXDiX10+QIEn4CGb12DP\nwj89AS1pJn7CC3Kka9f07w+VwaHufmdoATeue5gS+iDRtrzPYRfFCCmHQWmEI4yqL2SxuBbrDa4V\npIBfBVys1DKX92vkbR4saK7SheRxaNCknWGvKYxAGhkqStNUtu67HcW+CY0snIWZ5NmjhK9cXoGt\nvCJ8VWVIn/Ow76W7N2JjK8VdK47rbQ00niL8YIaRfsaDNo1D7Cfr3cFoCWcrhTLoPQ6TVz4Swi1C\nksuPFPCrgKHEIFLEFin9DiyR
}
],
"prompt_number": 19
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Full Covariances\n",
"================"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Above, we used diagonal covariance matrices:\n",
"\n",
"$$\\Sigma = \\hbox{diag}(\\sigma_1^2 ... \\sigma_d^2) = \n",
"\\begin{pmatrix} \n",
"\\sigma_1^2 & 0 & ... & 0 \\\\\\ \n",
"\\vdots & \\ddots & & \\vdots \\\\\\\n",
"0 & ... & \\sigma_{d-1}^2 & 0 \\\\\\ \n",
"0 & ... & 0 & \\sigma_d^2 \n",
"\\end{pmatrix}$$\n",
"\n",
"That matrix is just a diagonal matrix with the variance of each variable separately along the diagonal.\n",
"\n",
"A full covariance matrix has non-zero entries everywhere:\n",
"\n",
"$$\n",
"\\Sigma = \n",
"\\begin{pmatrix} \n",
"\\Sigma_{1,1} & \\Sigma_{1,2} & ... & \\Sigma_{1,d} \\\\\\ \n",
"\\vdots & \\ddots & & \\vdots \\\\\\ \n",
"\\Sigma_{2,1} & ... & \\Sigma_{2,d-1} & \\Sigma_{2,d} \\\\\\ \n",
"\\Sigma_{d,1} & ... & \\Sigma_{d,d-1} & \\Sigma_{d,d} \n",
"\\end{pmatrix}\n",
"$$\n",
"\n",
"We already have given the formula for estimating this from data:\n",
"\n",
"$\\hat{\\Sigma}_{ij} = \\frac{1}{N} \\sum (x_i^{(r)}-\\hat{\\mu}_i) (x_j^{(r)}-\\hat{\\mu}_j)$\n",
"\n",
"Python provides the `cov` function for estimating covariances.\n",
"Note that it computes the covariances of column vectors of an array,\n",
"but we want covariances of row vectors, so we need to transpose."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"# covariances\n",
"sigma0 = cov(samples0.T)\n",
"sigma1 = cov(samples1.T)\n",
"print sigma0\n",
"print sigma1"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"[[ 2.92608545 0.05090811]\n",
" [ 0.05090811 0.99466841]]\n",
"[[ 1.06473189 -0.04311753]\n",
" [-0.04311753 1.92439235]]\n"
]
}
],
"prompt_number": 23
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since the true covariance matrices are diagonal,\n",
"these estimates have small off-diagonal entries.\n",
"However, these entries are not quite zero.\n",
"\n",
"Note that the number of covariance parameters we need\n",
"to estimate from data grows quadratically in the number\n",
"of dimensions.\n",
"There is often simply not enough data to estimate a full\n",
"covariance matrix reliably."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"# full covariance matrices\n",
"imshow((G(normal(grid,mu0,sigma0))<G(normal(grid,mu1,sigma1)))-0.5,\n",
" origin='lower',extent=(-5,5,-5,5),cmap=cm.RdBu,vmin=-2,vmax=2)\n",
"plot(samples0[:,0],samples0[:,1],'r+')\n",
"plot(samples1[:,0],samples1[:,1],'b+')\n",
"\n"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 25,
"text": [
"[<matplotlib.lines.Line2D at 0x4b09590>]"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAFtCAYAAAAeUTF6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvX1cXOWd9/8+Z4AkPOSBkBAIieQ5xAYTGcUYgSF30rVa\n3GyiW/ujtcZW6aPb7d1Wd1dDHnpvre3P1XS7Xba/3ahdtnbVrCvVtQ13YYhJRGc0gjUBDSEJGUKE\nhPAYgZnz++M658yZYYaHAAGS6/16+ZLMnDnnmoH5XN/zfVQ0TdOQSCQSyaRDHe8FSCQSieTykAIu\nkUgkkxQp4BKJRDJJkQIukUgkkxQp4BKJRDJJkQIukUgkk5QRC3hrayt33303aWlprFq1irfeems0\n1iWRSCSSQYgY6Qn+6q/+ijvuuIOXXnqJvr4+Ojs7R2NdEolEIhkEZSSFPBcvXmTt2rXU1dWN5pok\nEolEMgRG5EI5ceIEc+bMYdu2bdx44408+OCDdHV1jdbaJBKJRDIAI7LAXS4X69at49ChQ9x00018\n97vfZfr06ezatct/AUUZlYVKJBLJtcZg8jwiCzwlJYWUlBRuuukmAO6++27efffdkIuYrP8VFhaO\n+xrk+sd/Hdfi+ifz2q+G9Q+FEQn4vHnzWLBgAbW1tQCUlpZy/fXXj+SUEolEIhkiI85C+fnPf05+\nfj49PT0sWbKEvXv3jsa6JBKJRDIIIxbwG264gXfeeWc01jIhcTgc472EESHXP75M5vVP5rXD5F//\nUBhREHNIF1CUIftzJBKJRCIYinbKUnqJRCKZpEgBl0gkkkmKFHCJRCKZpEgBl0gkkkmKFHCJRCKZ\npEgBl0gkkkmKFHCJRCKZpEgBl0gkkkmKFHCJRCKZpEgBl0gkkkmKFHCJRCKZpEgBl0gkkkmKFHCJ\nRCKZpEgBl0gkkkmKFHCJRCKZpEgBl0gkkkmKFHCJRCKZpIx4pNp40/ve/4z3Ekx6lUjei/0Mvcqk\n/1hZcukk83rOjfcyJFc5kWs/N95LmNRMfqWZQERofazs+oi6qdfRaYse7+UMmwjNy7LuE9joY5r3\n0/FejkQiGQQp4KOIgsZ0bwcLPvXQo0bSZoujOTJ+vJc1JKb5LjH/00Zm9l1ExTfey5FIJENACvgY\nMLvvAgDRtm4itD58ikpzZDy+CRhymO5tJ9rbzTTfJRJ7m8d7ORKJZBhIAR9DZnjbmeFtp0+x0adE\n4FVsdKtT6VEix3VdChqx3k5UNJI/bSJe33AkEsnkQgr4FSBC85LW9REA9VMXcC4yAQAN6LuCAU8V\nHzbNR4TWx6quj4jQ+q7YtSUSyegjBfwKs+BTD/M/PQtArxJBdczKKybic3rPc92lBkCT4i2RXAVI\nAb/C2DQvNryAyFpJ6/oITVHwRM3jfMTMUb2WNasEIMrXS6TWO6rXkEgk44cU8HHEyFoBQIOZfRf7\nHdMcGU+bLW5I50vuaWKq75L5b1XzyawSieQqRgr4BMEIeAYTqfURbese0jmSes4FCLhEIrm6kQI+\nwUnoPU9C7/nxXoZEIpmATLzEZIlEIpEMCSngEolEMkmRAi6RSCSTFCngEolEMkmRAi6RSCSTFCng\nEolEMkmRAi6RSCSTFCngEolEMkmRAi6RSCSTFCngEolEMkkZFQH3er2sXbuWvLy80TidRCKRSIbA\nqAj4M888w6pVq1AUZTROJ5FIJJIhMGIBb2ho4PXXX+drX/samqaNxpokEolEMgRGLOB//dd/zU9/\n+lNUVbrTJRKJ5Eoyonayv/vd75g7dy5r166lvLw87HE7duwwf3Y4HDgcjpFcViLph+KqQrOnj9vr\nJZKRUl5ePqCOhkLRRuD3+Nu//Vt+/etfExERwaVLl2hra2Pr1q08//zz/gsoypi6Vnrf+58xO7dk\n8qAWFeMryB+310suj8i1nxv5ScrL4So0CoeinSPye/z93/89p0+f5sSJE7zwwgts2LAhQLwlEolk\nzBmm1Xo1MaoTeWQWimQwRtNVobiqUNzVANiKis3HtYzVQ7rGSF8vkYw3oybgOTk55OTkjNbpJFcp\nirt61MRRs6cHnGu4LpCRvv5aZULEC8rL/Zb3zp3+xx2Oq9KdEg45E1MikQyL0dyEL5tgobYkSlxL\nSAGXjBmGpXYlXBVaxupxfb1EMh5IAZeMGYaldiVcFSPdCMbdopzgTOh4wTXkMglGCrhEIhmUCR0v\nkAIukYwOg1lqk8VVMVCgbkIE8SQSpIBLhoDTFU+O/fyQjh3MUpsswjdQoG5CBPHGkcmyCV8LyAYm\nkkGpcMeP9xIkE4hrefOaaEgLXDJmGJbaaLscxsqFoRa/Ah2dQH/3DzBxg3iSaxYp4JKQOF3xpuW9\nu2iZ+Xh2xvlhuVNg9F0OY+bC6OgMcPkM5P6ZUEG8CYaMEVw5pIBfIwz3S5VjDxTqxws+HotlSa5C\nrvUYwZVECvg1wpX+Uo123vBY5SGHOy+xMWFfI4N4komCFHDJoMyI7R32a0Y7b3iw813ubfvlrDP4\nOtJlMMELfa5ipIBfxYzWl+piR+Sor220Gc/bduu1r1Uxn9CFPlcxUsCvYoK/VONlDVldDqMhcGPl\nwhiN80r/r+RKIgX8GmI44jIaWSgG1muOhsBZrd3RvG03Gm9ZrxGOcNdWPE3Dvu7VhowRXDmkgF8j\naBmrTcEZCpMhC2UsbtuNz2gwl0jAtS2irZaUoiUnimN0IbvWLPJr7f2OJ1LAr3ImQnBpIqzhchnS\nHUNyYtj8cbWoeNTf47XqZ5f0Rwr4Vc5oWKnZGcNzmYzFGga9xghu2xVXFUpJKYqnCVXfaBRXFSQn\nMpRx3ANeewxcKmPhZ5ebwuRECrhkUIbr8x4PQomPdaDEQOJk3WC0omIUT5PpBrEVFYPFJRLOnWL+\nnLE68I6jpFQ8kZw4KjnrYyWyMvg6OZECfg0xEYJLV3INhigNVZzMAOYALpHBMDcCiw892MUS7toD\nrVFxVQl3jLt60rmhJGOHFPBriCvp8w53rSuVL+10xZM7zNcoJaVoeRvFzyNYn+KqMoOZqrsaH6Lt\n50BiGxw8DUazp6O5q82NYFQCtpM4NiERSAGXjDpDsXjH6pbdEKUDr2WzsaFYiKm72nSLDCiiniZ8\n+nNG8PFy7hg0ezqa7jrRli9GdVfj1e8EjOdDXj/EZxJKZBVXFcooiezl3mlIJgZSwMeRazFwNJzh\nEJeD6c92deO9Mx9fQT5aUXFYcbIKpOquRjNEUg8+Xvbvx+o2yV0X8vrB4uzLWN3PUg8VAB4t8ZZ+\n78mPFPBxZCJ8gUbqKhhOUY3iquJA0TJy3fvG5JY9oPjILYqPNJbi8KSRNcRzGH5w1V0dkM89UP8T\na7C032fgaUIJcy1ru11fyrwhW+pj8TczEeIjkuEjBfwaZySbiPW1Q0kVFH7cpfgKpvU7ZjTuRhyU\nk4tuTbOUx5MPAomwPAYfN4Z8jXXdiqsK769+Ip548JEBG2YF9D/Rfzb+U1xVeAvyzdcbm8JA1494\ndX/Aa0Ieq2e4jEX3RSngkxMp4FeYqypwNMQc51Bl+YprC1kZfnfK5W4kVkELSAd0dePbmTm8k+kW\nd9hrDTWbJajidUjvKy520EM0e/qIC4Nk06mrCyngV5gr/QUKZbGNZBPpl+Mcws0QbM2FKstXMj5C\ns08b0noHXI+RKhj0upzkoxDG6g6Htnwxqv55WP3h5nvzNJnP24qKA9wt5jmM8vkhWLRq8Sso5YfF\n+6itQ0V//451+PI3D2vtkmsTKeBXOaGsxsvZRAyBDHY5hHOVDGltYTIsfMM8D/R/n9l5EUOqorRi\nFU3jvRlrVNzV2EpK8RppfHkb8e78nlhnUbHZa0bR87S9BfliTQNsir78zWBc88FH/O6b4Pc2Rndt\n0m0y+ZECPo5MxC9QOAvYaumGytoYqpgYZfmKntNsfU1ZxnfI5echXQXWQKHxegifWjeUCsyhvPdw\nm51qEdKBjhsNxurck85lJ+mH
}
],
"prompt_number": 25
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That decision region is slightly different from what we got\n",
"with a purely diagonal covariance matrix.\n",
"In higher dimensions, those differences become much larger."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(prior knowledge)\n",
"\n",
"Note that we have two forms of _prior knowledge_:\n",
"\n",
"- parametric knowledge, e.g., the prior probability of each class\n",
"\n",
"- structural knowledge, e.g., that the ccd's are Gaussian and that the covariance matrices are diagonal\n",
"\n",
"If our prior knowledge is too limited or our assumptions are wrong, classification accuracy drops.\n",
"\n",
"Limited prior knowledge can be compensated with more data (eventually, we figure out that the covariance matrix is diagonal).\n",
"\n",
"Erroneous prior assumptions cannot be fixed with more data."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [],
"language": "python",
"metadata": {},
"outputs": []
}
],
"metadata": {}
}
]
}