nvidia-AI.summerschool17-no.../teaching-ncso/12-linear-threshold-classification.ipynb
2017-09-22 19:38:09 +02:00

177 lines
No EOL
20 KiB
Text

{
"metadata": {
"name": "12-linear-threshold-classification"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Title: Linear Threshold Units\n",
"Author: Thomas M. Breuel\n",
"Institution: UniKL"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"\n",
"from pylab import *\n",
"from IPython.core.display import Image \n",
"def fig(x): return Image(filename=\"Figures/\"+x+\".png\") \n",
"from pylab import *\n",
"def figs(*args):\n",
" for i,f in enumerate(args):\n",
" subplot(1,len(args),i+1)\n",
" axis(\"off\")\n",
" fig = imshow(imread(\"Figures/\"+f+\".png\"))\n",
" \n",
"from scipy import linalg"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 267
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Neurons as Linear Decision Makers"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Before embarking in more detail on learning algorithms, let's take a general look at neurons as classifiers or decision makers."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(McCulloch-Pitts Neurons)\n",
"\n",
"Recall that we arrived at the McCulloch-Pitts model as a simplified model of the computation performed by neurons.\n",
"\n",
"Component-wise notation:\n",
"\n",
"$$ y_k = \\phi(\\sum_{j=1}^m w_{kj}x_j+x_0) $$\n",
"\n",
"Matrix notation:\n",
"\n",
"$$ y = \\phi(W \\cdot x + x_0) $$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(Linear Decisions)\n",
"\n",
"For many kinds of problems, linear decision rules are common, and these map well onto McCulloch-Pitts neurons.\n",
"\n",
"That is, rules of the form: accept if $x_1 + 0.7 x_2 \\leq 1.1$ .\n",
"\n",
"The reason is that a lot of decision rules are really rules based on costs, and costs are of the form $\\hbox{quantity} \\times \\hbox{unit cost}$.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(Perceptrons)\n",
"\n",
"Here is an example of this.\n",
"\n",
"The _decision rule_ is \n",
"\n",
"$$D(x) = \\left\\lfloor {1 \\choose 0.7} \\cdot {x_1 \\choose x_2} \\leq 1.1 \\right\\rfloor$$\n",
"\n",
"This is obviously the same form as for a McCulloch-Pitts neuron with a Heaviside function as the nonlinearity.\n",
"\n",
"These kinds of linear decision units are called _perceptrons_.\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"\n",
"xs = rand(100,2)\n",
"ys = 1*(xs[:,0]+0.7*xs[:,1]>1.1)\n",
"plot(xs[ys==0,0],xs[ys==0,1],'b.')\n",
"plot(xs[ys==1,0],xs[ys==1,1],'r.')\n",
"plot([0,1.1/0.7],[1.1,0],color='green')"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 284,
"text": [
"[<matplotlib.lines.Line2D at 0x1021a190>]"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD9CAYAAABHnDf0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XtUU2e6P/BvEI7FVlGRQSUwyE3uYahA0VZivUfqcTml\nh9bTUWsxERxLO22dmdrjpbNQO61rvGFiq7aO4rKjU+0AUrUaq3jBNv6AAazUgwWp9YgVddRCAu/v\nD0YqF5MQdvYtz2ctVk2z2fsbdvaTd7/vu3cUjDEGQgghsuImdABCCCHco+JOCCEyRMWdEEJkiIo7\nIYTIEBV3QgiRISruhBAiQzaL+0svvQRfX1/ExMR0+/zOnTuhUqkQGxuLMWPGoKysjPOQhBBCesZm\ncZ87dy6Kiooe+nxQUBC+/PJLlJWV4e2338b8+fM5DUgIIaTnbBb3p556CoMGDXro88nJyfDy8gIA\nJCUl4fLly9ylI4QQ4hBO+9y3bNkCjUbD5SoJIYQ4wJ2rFR09ehRbt25FcXFxt88rFAquNkUIIS7F\nkbvEcNJyLysrQ0ZGBj777DOrXTiMMdH/LF26VPAMlJMyUk7Kef/HUb0u7rW1tZg5cyZ27NiBkJCQ\n3q6OEEIIB2x2yzz//PM4duwYGhoa4O/vj+XLl8NsNgMAtFotVqxYgRs3bmDBggUAAA8PD5SUlDg3\nNSGEEKtsFvddu3ZZff7DDz/Ehx9+yFkgoanVaqEj2IVyckcKGQHKyTWp5HSUgvWmU6cnG1IoetV/\nRAghrsjR2km3HyCEEBmi4k4IITJExZ0QQmSIijshhMgQFXdCCJEhKu6EECJDVNwJIUSGqLgTQogM\nUXEnhBAZouJOCCEyRMWdEEJkiIo7IYTIEK/Ffd2ZdWhpbeFzk4QQ4pJ4Le57q/biiS1PwHTFxOdm\nCSHE5fBa3I2zjchKyMLUnVORXZSN2023+dw8IYS4DF6Lu0KhwJy4OajIrMDNppuIzI3Ep1Wf0n3e\nCSGEY4J+WYfxkhG6fB3CvMOwQbMBAV4BfEQhhBDJkOSXdagD1SjVlSJheALiDfF4/+T7sLRahIxE\nCCGyIJqv2au+Xo3Mwkxcu3MNhlQDkpRJfMQihBBRc7TlLpriDgCMMeSV5+H1Q69jZsRM5DydA69H\nvPiIRwghoiTJbpnOFAoFZsXOQmVmJSytFkTmRmL3P3fTgCshhPSQqFrunRXXFkNXoINygBIbNRsR\nNCjISekIIUScZNFy72xMwBiY5pug/qUaiR8kYuXxlWhuaRY6FiGEiJ6oW+4PqrlRg6zCLNTerIU+\nVY8nA57kMB0hhIiTLAZUbWGMYU/lHmR/ng1NqAarJ6zGYM/BHCUkhBDxkWW3TGcKhQJpUWmozKzE\nI+6PIHJjJHaU7aABV0II6URSLffOSupLoM3XwtvTG7nTchHmHcbp+gkhRGhOa7m/9NJL8PX1RUxM\nzEOXWbRoEUJDQ6FSqXDu3Lkeh3BUol8izmachSZUg9FbRmO5cTmaLE28bZ8QQsTKZnGfO3cuioqK\nHvp8YWEhvv32W1RXV2Pz5s1YsGABpwFtcXdzx2vJr8GkNeHcD+eg0qtwtOYorxkIIURsbBb3p556\nCoMGDXro85999hlmz54NAEhKSkJjYyOuXr3KXUI7BXgFYF/6PqyasAqz983G7H2zce3ONd5zEEKI\nGPR6QLW+vh7+/v7tj5VKJS5fvtzb1TpsRvgMVGRWwNvTG9GborH13FYacCWEuBx3LlbSuXgqFIpu\nl1u2bBkA4B//AFpa1Bg+XI28PGDgQC5S/Kx/3/5YM3kNXox9Edp8LT76fx9Bn6pHpE8ktxsihBCO\nGY1GGI3GXq/Hrtkyly5dwjPPPIPy8vIuz+l0OqjVaqSnpwMAwsPDcezYMfj6+nbc0AMjvmo1cOxY\n2/9PSwM++aSXr8KKltYWbPpqE5YfWw7t41q89dRb8PTwdN4GCSGEQ4LNc58+fTq2b98OADh9+jQG\nDhzYpbB31q9f238TEoDNm3ubwLo+bn2wMHEhSnWluHD9AmI2xeDgxYPO3aiYzZ/f9umq0QCNjUKn\nIYQ4ic2W+/PPP49jx46hoaEBvr6+WL58OcxmMwBAq9UCABYuXIiioiI8+uij2LZtG+Lj47tu6IFP\nn8bGthqzeTP3XTK2FFYXIqswC8nKZKyZvAZDHxvKbwCh8XnaRAjpNZe4/QBX7prvYsWxFdhybgve\nGfcO5j8+H24KSV2s6ziNBjhwoO206eBB/j9dgbZP9gsX2k7hnDHoQoiMUHF3QPnVcmjztWBgMKQa\nEOsbK3Qk5xPytOm+B88ehg4FqqqowBPyEC5xbxmuxfjG4MRLJzA3bi4mbJ+ANw+9iTvNd4SO5VwD\nB7Z1xQhZTO8PugDADz+0fdgQQjjl0i33B13911W8dvA1FNcWY4NmA1LDUoWOJF+NjUBERFthF7J7\niBAJcOluGS67cA9dPITMwkyofFVYO2Ut/Ab4cRfUDi7THS2G7iFCJMClu2UuXGjrwj1woPdn+BOD\nJ6JMV4YInwio9CqsO7MOLa0t3AS1A5evRdTE0D1EiIzJorhzPW/e08MT74x7B8fnHsfeqr14YssT\nMF0x9X7FduDzGgBCiHzJolvGmWf4jDF8XPoxFh9ejOejn8c7495B/779ud3IA6i3ghDyIJfuc+dD\nw90GvHHoDRz+38NYN2UdZoTPeOg9dIhMucyACBETKu48MV4yQpevQ5h3GDZoNiDAK0DoSIQvdHUv\nEYBLD6jySR2oRqmuFAnDExBviMf7J9+HpdUidCzCBxoQIRJCLfdeqL5ejczCTFy7cw2GVAOSlElC\nRyLORAMiRADULSMQxhjyyvPw+qHXMTNiJnKezoHXI15CxyKEyAR1ywhEoVBgVuwsVGZWwtJqQWRu\nJHb/c7csP8gIIdJBLXeOFdcWQ1egg3KAEhs1GxE0KEjoSKQ3aIYMERi13EViTMAYmOaboP6lGokf\nJGLl8ZVobmkWOhZxlMtcMkzkhoq7E3j08cDiJxfjbMZZHK89jnhDPE7UnhA6FnGEPTNk6NutiAhR\nt4yTMcawp3IPsj/PhiZUg9UTVmOw52ChYxF72TNDhua/EyeibhmRUigUSItKQ2VmJR5xfwSRGyOx\no2yHS37QSZI9Nzij+e9EhKjlzrOS+hJo87Xw9vRG7rRchHmHdXiexu8kiOa/Eyeiee4SYmm1YN2Z\ndcg5noPfJv4Wv3/y9+jr3hcAneETQjqibhkJcXdzx2vJr8GkNeHcD+eg0qtgvGQEQGf4hBBuUMtd\nBPad34dFBxZh3IhxeDvxPfwx24fO8AkhAGTQcnfl2WQzwmegIrMC3p7eGJMXjSl/2AovL/ogBODa\nbwxCekE0LXfqa25jumKCNl8LT3dP6FP1iPSJFDqSsOiNQVyc5Fvu1NfcJn5YPE7PO43nop5Dykcp\nWHJkCe6Z7wkdSzj0xiDEIaJpudNssq6+v/09souyYbpiQu60XEwKniR0JP7RG4O4OJoKKWOF1YXI\nKsxCsjIZayavwdDHhgodiRDCE8l3y5CH04RqUJFZgQCvAMRsioH+Kz1aWavQsQghIkYtd4kpv1oO\nbb4WAKBP1SPWN1bgRIQQZ3Jay72oqAjh4eEIDQ3F6tWruzzf0NCAKVOmIC4uDtHR0fjoo496HILY\nL8Y3BideOoE5cXMwYfsEvHnoTdxpviN0LEKIyFhtube0tGDkyJE4fPgw/Pz8kJCQgF27diEiIqJ9\nmWXLlqGpqQkrV65EQ0MDRo4ciatXr8Ld3b3jhqjlzrmr/7qK1w6+huLaYmzQbEBqWKpdv0f3ryFE\nOpzSci8pKUFISAgCAwPh4eGB9PR07N+/v8Myw4YNw61btwAAt27dgre3d5fCTpzD9zFf7Jy5Ex88\n8wFe/fxVPPvJs6i/VW/z9wT5/gm6GIkQXlmtwvX19fD3929/rFQqcebMmQ7LZGRk4Omnn8bw4cNx\n+/ZtfGLlIpNly5a1/1utVkOtVjuWmnQwMXgiynRlyDmRA5Vehf9J+R9kJWShj1ufbpfnZep459OD\n+58o95+ji5EI6ZbRaITRaOz1eqx2y+zduxdFRUX44IMPAAA7duzAmTNnsH79+vZl/vSnP6GhoQF/\n+ctfcPHiRUycOBGlpaXo379/xw1Rtwwvqq5VQVegw13zXRhSDYgfFt9lGV6mjne+svRf/2o7VUhI\nAA4efPiGqc+IkA6c0i3j5+eHurq69sd1dXVQKpUdljl58iTS0tIAAMHBwRgxYgS++eabHgch3Ijw\niYBxthFZCVmYunMqXv38Vdxuut1hGXu+f6LXOp8e5OW1FXlrhR2g7ywlhCNWi/uoUaNQXV2NS5cu\nobm5Gbt378b06dM7LBMeHo7Dhw8DAK5evYpvvvkGQUFBzkvsJHLqElYoFJgTNwcVmRVo/KkRUblR\n+LTqU37PnDoXc3s/Ueh2A4Rwg9lQWFjIwsLCWHBwMMvJyWGMMabX65ler2eMMXbt2jWWmprKYmNj\nWXR0NNu5c2e367FjU4JKSWEMaPtJSxM6DbeO1hxlI9ePZM/kPcO+a/xO2DAZGW1/7KlTGbtxo+vz\nN2607YDuniPEBTlaO+kipn/TaOzrEpaqJksT3i1+F2vPrMUfnvwDXnniFbi7CTCrSep3eaQxAcIz\nuv1AL9nbJSxVfd374u2Ut3Fq3ikc+PYARm0ehTOXz9j+Ra5Jqdulu746GhMgEkEtdxfEGENeeR5e\nP/Q6ZkbMRM7TOfB6xIufjUvpLo/dnWXI/RSPiA613IndFAoFZsXOQmVmJSytFkTmRmL3P3fz8+HL\ny1QdjnR3liH3UzwiG7JquVN3qGOKa4uhK9BBOUCJjZqNCBokvdlOTiHkWQa9mcm/Ucsd1B3qqDEB\nY2Cab4L6l2okfpCIlcdXormlWehYwhPyLIPezKSXZFXcpTRWJzYefTyw+MnFOJtxFsdrjyPeEI8T\ntSeEjuW66M1MeklW3TJSGqsTM8YY9lTuQfbn2dCEarB6wmoM9hwsdCzXQm9m8m/0NXuEczd/uom3\njryFPZV78N6k9zArZhYUCoXQsQhxKVTcBcDlmJeYx89K6kugzdfC29MbudNyEeYdJnQk/ol5BxFZ\nc7kBVTHcC4bLMS8xj58l+iXibMZZaEI1GL1lNFYcW4EmS5PQsfgl5h1ESDckW9zFcKxxOeYl9vEz\ndzd3vJb8GkxaE0xXTFDpVTBeMgodiz9i30GEdCLZbhkxXCjY3ZiXo2fvUhs/23d+HxYdWIRxI8bh\nvYnvwedRH6EjOZfUdhCRDZfrcxfrscbnfbGE7ga+3XQbS41LsbN8J1aOX4m5cXN7PuAq9IsgRORc\nrriLFZ9nFGK5waLpignafC083T2hT9Uj0ifS/l8Wy4sgRKRcbkBVrPi89YhYuoHjh8Xj9LzTeC7q\nOaR8lIIlR5bgnvmefb8slhdBiMxQy13CxNg19f3t75FdlA3TFRNyp+ViUvAk678gxhdBiIhQtwwR\nlcLqQmQVZiFZmYw1k9dg6GNDhY5EiCRRtwwRFU2oBhWZFQjwCkDMphjov9KjlbUKHYsQl0Etd+J0\n5VfLoc3XAgD0qXrE+sYKnIgQ6aCWOxGtGN8YnHjpBObEzcGE7RPw5qE3caf5jtCxCJE1Ku6EF24K\nN8x/fD7KF5Sj/nY9onKjkH8hX+hY9hPD/S4I6QHeu2XomhUCAIcuHkJmYSZUviqsnbIWfgP8hI5k\nHc3HJwKRTLeMPfeEoUaS/E0MnogyXRkifCKg0quw/sx6tLS2CB3r4Wg+PpEY3lvu9lzBSY0k11J1\nrQq6Ah3umu/CkGpA/LB4oSN1RfPxiUAkM8/dnmOEj0v45dY9JPXXwxjDx6UfY/HhxXgh5gWsUK9A\n/779hY5FiOAkU9ztwUcjSW5nB3J5PQ13G/DGoTfwxf9+gbVT1mJG+Az69ifi0mRV3PkghlsGc0lu\nr8d4yQhdvg5h3mHYoNmAAK8AoSMRIgjJDKiKBZ83+OKD2F9PTwfJ1YFqlOpKkTA8AfGGeLx/8n1Y\nWi1Oz0mIXNhsuRcVFSE7OxstLS14+eWXsXjx4i7LGI1GvPrqqzCbzRgyZAiMRmPXDYms5U741Ztu\no+rr1VhQsAANdxtgSDUgSZnklIyEiJHDtZNZYbFYWHBwMKupqWHNzc1MpVKxysrKDsvcuHGDRUZG\nsrq6OsYYY9euXet2XTY2RUQmI4OxlBTGpk5l7MaN3q9v6lTGAMYSEhxbX2trK9tRuoMNfW8oyyzI\nZI33GnsfihAJcLR2Wu2WKSkpQUhICAIDA+Hh4YH09HTs37+/wzJ5eXn49a9/DaVSCQAYMmRIzz9h\nZMKergepzOHn+jtqe9ttpFAoMCt2FiozK2FptSAyNxK7/7lbGmeDUtnpRFbcrT1ZX18Pf3//9sdK\npRJnzpzpsEx1dTXMZjPGjRuH27dv45VXXsGLL77Y7fqWLVvW/m+1Wg21Wu14chG6XxCBtuO5u64H\ne5YRA66v2Rk4kJvXOshzEAypBhTXFkNXoMNHpR9ho2YjggYFiXc+qFR2OhEFo9HYbdd2T1kt7vZM\nQTObzTCZTPjiiy9w9+5dJCcn44knnkBoaGiXZR8s7nJ0vyB6eLS1UCdOBP72t441RioXOublifua\nnTEBY2Cab8KaU2uQ+EEifpf8O/yu+jz+49jxtgXEVESlstOJKHRu+C5fvtyh9VjtlvHz80NdXV37\n47q6uvbul/v8/f0xadIkeHp6wtvbG2PHjkVpaalDYaQuLw/w8QHMZuDmTeDw4a5dGmKf1XLf/Za2\nmDN69PHA4icX42zGWRyvPY74xHM4EQDxFVGp7HQiL9Y65M1mMwsKCmI1NTWsqamp2wHVqqoqNn78\neGaxWNidO3dYdHQ0q6io4GxQQGruDxwCjP3qV9wMRhLbWltb2Scl29jwtx5hL//tN+z63etCRyKE\nE47WTqstd3d3d2zYsAGTJ09GZGQk/uu//gsREREwGAwwGAwAgPDwcEyZMgWxsbFISkpCRkYGIiMj\nefhYEqe8PGDGDOA//xM4coQaanxRKBRIS5iDyiU/oG+//ojcGIkdZTukMeBKiBO47BWqRN5K6kug\nzdfC29MbudNyEeYdJnQkQhwiqytUaeYY6a1Ev0SczTgLTagGo7eMxopjK9BkaRI6FiG8EWXLXS43\nweKCWGf3SUntzVosOrAI5xvOQ5+qhzpQLXQkQuwmq5Y7zRz7GdcXE7miAK8A7Evfh1UTVuE3n/4G\ns/fNxrU714SORYhTibK408yxn9EHHXdmhM9ARWYFvD29Eb0pGlvPbaVxICJbouyWIT+jLwByDtMV\nE7T5WvTz6Af9ND0ifCKEjtQV9ckR0P3cCemxltYWbPpqE5YfWw7t41q89dRb8PTwFC5Q52I+YwYN\nPhF59bkTwoc+bn2wMHEhSnWluHD9AmI2xeDgxYPCBeo8wEJ9cqQXqOVOyL8VVhciqzALycpkrJm8\nBkMfG8rdyu3pYun8dVr3f4/65FwatdwJ6SVNqAYVmRUI8ApAzKYY6L/So5W1crNye6Y9dZ5JIIUb\n/BDRopY7z2iMTBrKr5ZDm68FAOhT9Yj1je3dCuX2JbeEN9Rylwiaty4NMb4xOPHSCcyJm4MJ2yfg\nzUNv4k7zHcdXSPN7Cc94L+5ivbUAX7lojEw63BRumP/4fJQvKEf97XpEb4pGwYUCx1ZGXSyEZ7x3\ny4j11gJ85aJ569J16OIhZBZmQuWrwtopa+E3wE/oSMQFSKZbRqwtV75yUQNOuiYGT0SZrgwRPhFQ\n6VVYf2Y9Wua/zP0pn1hPb4mk8N5yF2vLVay5eoMGb52n6loVdAU63C0/B8Ou24i/Au5O+cR6eksE\nQVeoki6oRjgXYwwfz1ZhsW85Xvg/X6z481n0/4W/7V+0hWbWkAdIoluGzjT5JdYuMLlQKBSYs+5L\nVFyejsbU8YjaOQafVn3a+0YMzawhHOC15Q60bYpakfyQY1eTmBkvGaHL1yHMOwwbNBsQ4BUgdCQi\nA5LolgEYnWkSWWuyNOHd4nex9sxa/OHJP+CVJ16Bu5u70LGIhEmiuKelMVG3ImkAknCl+no1FhQs\nQMPdBhhSDUhSJgkdiUiUJIq72AdUXWkAkj7InI8xhrzyPLx+6HXMjJiJnKdz4PWIl9CxiMRIYkBV\n7FxpALK3t0Ggqdjd6PRHUSgUmBU7C5WZlbC0WhCZG4nd/9wt+kYOkQdquT/AlQYgezvbTqizHFGf\ncdj4oxTXFkNXoINygBIbNRsRNCiI/4xEcqjlzgFXunq0t7PthDrLEfWN12z8UcYEjIFpvgnqX6qR\n+EEiVh5fieaWZp5DEldBLXfiEKHOckR9fU8P/ig1N2qQVZiF2pu10Kfq8WTAkzyFJFJDA6rEJcip\n64wxhj2Ve5D9eTY0oRqsnrAagz0HCx2LiAx1yxCXIKeuM4VCgbSoNFRmVqJvn76Iyo3CjrId1Agi\nnKCWOyEiUVJfAm2+Ft6e3sidlosw7zChIxERcFrLvaioCOHh4QgNDcXq1asfutzZs2fh7u6Ov//9\n7z0OQUh3XG26ZaJfIs5mnIUmVIPRW0ZjxbEVaLI0CR2LSJTV4t7S0oKFCxeiqKgIlZWV2LVrF6qq\nqrpdbvHixZgyZQq1zglnRD0zxknc3dzxWvJrMGlNMF0xQaVXwXjJKHQsIkFWi3tJSQlCQkIQGBgI\nDw8PpKenY//+/V2WW79+PZ599ln4+Pg4LSjhhpRaw650UVlnAV4B2Je+D6smrMJvPv0NZu+bjWt3\nrgkdi0iI1Tsa1dfXw9//5/tTK5VKnDlzpssy+/fvx5EjR3D27Nl/3yCse8uWLWv/t1qthlqtdiw1\n6ZY9F/jcbw3fX17Mt1jIy5PPzJiHsrHTZoTPwPgR47HUuBTRm6KxcvxKzI2ba/U4I9JmNBphNBp7\nvR6rxd2eN1B2djZWrVrV3ulvrVvmweJOuGdP4ZZSa/j+zBhZs2On9e/bH2smr8F/x/43tPlafFz6\nMfTT9IjwieA5LOFD54bv8uXLHVqP1W4ZPz8/1NXVtT+uq6uDUqnssMzXX3+N9PR0jBgxAnv37kVm\nZiY+++wzh8KQ3rGncNP3QIhMDz5t44fF4/S800iLTMPYj8ZiyZEluGe+x0NIIkVWp0JaLBaMHDkS\nX3zxBYYPH47ExETs2rULERHdtxjmzp2LZ555BjNnzuy6IZoK6XRyusDHZTi4076//T2yi7JhumJC\n7rRcTAqe5MSQREiO1k6r3TLu7u7YsGEDJk+ejJaWFsybNw8REREwGAwAAK1W61haJxP1zaWcyCW6\nMeTGwZ02vP9wfJL2CQqrC6HN1yJZmYw1k9dg6GNDnRCSSJGsLmK6X9TLy4Eff2z7f3K/LzsRgMha\nD3fNd7Hi2ApsObcF74x7B/Mfnw83BV18Lhd0bxl0vOMqINKbSxEAoquPPSPSb3Upv1oObX7b2bQh\n1YAY3xiBExEu0L1l8PPY1K9+BcyYIa/CLqX56faQ9AVKtgZBBdpZMb4xOPHSCcyJm4Px28fjzUNv\n4k7zHd62T0SG8YSPTd24wVhaWtt/5SYlhTGg7SctTeg0vTd1attrSUiQ4P6y9UYTwc764fYP7IW9\nL7DAvwSy/G/yBclAuOFo7ZRVt4ycifo+5g6Q9cweEe2sQxcPIbMwEypfFdZOWQu/AX6CZSGOoT53\nmZN1MZQbke2se+Z7yDmRg01nN2FpylJkJmSij1sfoWMRO1FxJ4RYVXWtCroCHe6a78KQakD8sHih\nIxE7UHGXEEnPFCGSxhjDx6UfY/HhxXgh5gWsUK9A/779hY5FrKDZMhIi6ZkiRNIUCgXmxM1BRWYF\nGn9qRGRuJPad3yd0LOIE1HIXgIjG24iLM14yQpevw8ghI7F+6noEeAUIHYl0Qi13kepuyjPdvIuI\nhTpQjVJdKUYNG4V4QzzeP/k+LK0WoWMRDlDL3clEejGjS6KxDuuqr1djQcECNNxtgCHVgCRlktCR\nCKjlLlpSun+63NFYh3Wh3qE49OIhvDH6DczYPQNZhVm4+dNNoWMRB1FxdzLqghEP+qC1TaFQYFbs\nLFRmVsLSakFkbiR2/3O3S551Sx11yxCXYevaIuq26aq4thi6Ah2UA5TYqNmIoEFBQkdyOTTPXSBU\nEOSjt+Mjcn0vmFvMWHNqDf588s94ffTr+F3y7+DRx0PoWC6DirtAaMBU2h4syGYzcPiw41NU5f5e\nqLlRg6zCLNTerIUh1YAxAWOEjuQSaEBVINSPK20PDrI+9ljvxkfk/l4YMWgECl4owNKUpXhuz3PI\n+EcGfrz3o9CxyENQce8lGjCVtgcL8rZtba1tR/ejK7wXFAoF0qLSUJlZib59+iIqNwo7ynbI8qxc\n6qhbhrg0kd3AUXJK6kugzdfC29MbudNyEeYdJnQk2aE+d0KIICytFqw7sw45x3OwKGkRFo9ZjL7u\nfYWOJRtU3Akhgqq9WYtFBxbhfMN56FP1UAeqhY4kC1TcCSGisO/8Piw6sAhPj3ga7016D0P6DRE6\nkqTRbBkJktuXXtviaq/XVc0In4GKzAoM9hyMqNwobD23lRp2AqCWu4DkPi+6M1d7vQQwXTFBm69F\nP49+0E/TI8InQuhIkkMtdwmS+7zozlzt9RIgflg8Ts87jbTINIz9aCyWHFmCe+Z7QsdyCdRyF5Cr\nTcNztddLOvr+9vfILsqG6YoJudNyMSl4ktCRJMHlBlTleh8PYh/a/9JVWF2IrMIsJCuTsWbyGgx9\nbKjQkUTN5bpl6N7cro32v3RpQjWoyKxAgFcAYjbFQP+VHq2sVehYsmNXcS8qKkJ4eDhCQ0OxevXq\nLs/v3LkTKpUKsbGxGDNmDMrKyjgP2hn137o22v/S1s+jH1ZNWIUjvzmC7aXb8eTWJ1F+tVzoWPLC\nbLBYLCw4OJjV1NSw5uZmplKpWGVlZYdlTp48yRobGxljjB04cIAlJSV1WY8dm+qRGzcYS0tr+6+c\nZGQwlpLC2NSp8nttXJLr/ndFLa0tzPCVgfm868PeOPgG+1fTv4SOJCqO1k6bLfeSkhKEhIQgMDAQ\nHh4eSE9FEHX+AAAMJUlEQVRPx/79+zssk5ycDC8vLwBAUlISLl++7IzPoQ4GDuzdTZ7Eirob7CPX\n/e+K3BRumP/4fJQvKEf97XpEb4pGwYUCoWNJns3iXl9fD39///bHSqUS9fX1D11+y5Yt0Gg03KRz\nQdTdIE90AZdtvo/5YufMndicuhnZn2fj2U+eRf2th9caYp27rQUUCoXdKzt69Ci2bt2K4uLibp9f\ntmxZ+7/VajXUarXd63YVeXk0XbAzOcyMuX9GBrS9HpsXcMnhRTtoYvBElOnKkHMiByq9CktTliIz\nIRN93PoIHY0XRqMRRqOx9yuy1W9z6tQpNnny5PbHOTk5bNWqVV2WKy0tZcHBway6uprTfiNCUlIY\nA9p+0tKETuOYqVPb8ick2DlOIIcXzYHK/6tkY7eNZaM2j2Jff/+10HEE4WjttPlbZrOZBQUFsZqa\nGtbU1NTtgOp3333HgoOD2alTpzgPSOShNwPFPS6MItTjAWA5vGiOtLa2sm3ntrFf/PkXLLsom936\n6ZbQkXjltOLOGGOFhYUsLCyMBQcHs5ycHMYYY3q9nun1esYYY/PmzWODBw9mcXFxLC4ujiUkJHAW\nkMhDbxqiLjkzxiVftHXX7lxjc/bNYco1SvZp1adCx+GNo7VTsleoEmnRaNpmADn65dOE3Ge8ZIQu\nX4eRQ0Zi/dT1CPAKEDqSU7ncFapEWlzh+0UJP9SBapTqSjFq2CjEG+Lx/sn3YWm1CB1LdKjlTgiR\nrOrr1VhQsAANdxtgSDUgSZkkdCTOudyNw+TMhWfBEdJjjDHklefh9UOvY2bETOQ8nQOvR7yEjsUZ\n6paREbpKlRD7KRQKzIqdhcrMSlhaLYjMjcQnFZ+4fGOSWu4iRIOP3aMzGmKP4tpi6Ap0UA5QYqNm\nI4IGBQkdqVck2XKnS7K7R4OP3aMzGmKPMQFjYJpvgvqXaiR+kIhVJ1bB3GIWOhbvBG2503dqkp6Q\n0xkNnYXwo+ZGDbIKs1B7sxaGVAPGBIwROlKPSbLlTjfJcj29OVuT0xkNnYXwY8SgESh4oQBLU5bi\nuT3PIeMfGfjx3o9Cx+KFoMVdTgcrsU9vipqcbvNLDRv+KBQKpEWloTKzEn379EVUbhR2lO2Q/Rig\n4AOqdHrqWuTUtdIb9GXhwimpL4E2XwtvT2/kTstFmHeY0JGskuw8d+p3dy1U1IgYWFotWHdmHXKO\n52BR0iIsHrMYfd37Ch2rW5It7tSSI4QIpfZmLRYdWITzDeehT9VDHagWOlIXki3u1JIjhAht3/l9\nWHRgEZ4e8TTem/QehvQbInSkdpKcLQPIa5CMECJNM8JnoCKzAoM9ByMqNwpbz22V/ICr4C13QggR\nE9MVE7T5WvTz6Af9ND0ifCIEzSPZljshhIhJ/LB4nJ53GmmRaRj70VgsObIE98z3hI7VY6Is7lzf\nloBucyAPtB8JX/q49cHCxIUo1ZXiwvULiNkUg4MXDwodq0dE2S3D9fRImm4pD7QfiVAKqwuRVZiF\nZGUy1kxeg6GPDeVt25LolrG35cX11Xt0NaA80H4kQtGEalCRWYEArwDEbIqB4SsDWlmr0LGs4rXl\nnpLC7Gp5cT09kqZbygPtRyIG5VfLoc3XAgAMqQbE+MY4dXuSmOc+dSqjC5YIIZLXylrxoelDLDmy\nBHPi5mBpylI8+h+POmVbkuiWoRuFEULkwE3hhvmPz0f5gnLU365H9KZoFFwoEDpWB6IcUCWEECk5\ndPEQMgszofJVYe2UtfAb4MfZuiXRcieEEDmaGDwRZboyRPhEQKVXYf2Z9WhpbRE0k8u03OnWwoQQ\nPlRdq4KuQIe75rswpBoQPyy+V+ujlrsN9M03hBA+RPhEwDjbiKyELEzdORWvfv4qbjfd5j2HyxR3\nmiNNCOGLQqHAnLg5qMisQONPjYjMjcS+8/v4zeAq3TKNjUB8PDB8ODBgAHXNEEL4Y7xkhC5fh5FD\nRmL91PUI8Aqw+3ed1i1TVFSE8PBwhIaGYvXq1d0us2jRIoSGhkKlUuHcuXM9DsGHgQOBgACguNh6\n14zRaOQ1l6PEktPWVcdiyWmNFDIClJNrfOZUB6pRqivFqGGjEG+Ix/sn34el1eLUbVot7i0tLVi4\ncCGKiopQWVmJXbt2oaqqqsMyhYWF+Pbbb1FdXY3NmzdjwYIFTg3cG/Z0zdAbs429t4qwNZYhhr+n\nHD6AAMrJNb5z9nXvi7dT3sapeadw4NsDGLV5FM5cPuO07Vkt7iUlJQgJCUFgYCA8PDyQnp6O/fv3\nd1jms88+w+zZswEASUlJaGxsxNWrV50WuDfoIir72TsALYWxDBpMJ2IS6h2KQy8ewhuj38CM3TOQ\nVZiFmz/d5Hw7Vot7fX09/P392x8rlUrU19fbXOby5cscx+QGfeuT/ewt2lL4wJTCBxBxLQqFArNi\nZ6EysxKWVgsicyNhumLidiPMij179rCXX365/fFf//pXtnDhwg7LpKamshMnTrQ/Hj9+PPv666+7\nrAsA/dAP/dAP/Tjw4wh3WOHn54e6urr2x3V1dVAqlVaXuXz5Mvz8ul56S7ceIIQQ/ljtlhk1ahSq\nq6tx6dIlNDc3Y/fu3Zg+fXqHZaZPn47t27cDAE6fPo2BAwfC19fXeYkJIYTYZLXl7u7ujg0bNmDy\n5MloaWnBvHnzEBERAYPBAADQarXQaDQoLCxESEgIHn30UWzbto2X4IQQQqxwqDPHigMHDrCRI0ey\nkJAQtmrVqm6X+e1vf8tCQkJYbGwsM5lMXEewi62cO3bsYLGxsSwmJoaNHj2alZaWii7jfSUlJaxP\nnz5s7969PKb7mT05jx49yuLi4lhUVBRLSUnhN+C/2cp57do1NnnyZKZSqVhUVBTbtm0b7xnnzp3L\nfvGLX7Do6OiHLiOG48dWTjEcP4zZ9/dkTPhjyJ6cPT2GOC3uFouFBQcHs5qaGtbc3MxUKhWrrKzs\nsExBQQGbOnUqY4yx06dPs6SkJC4jcJbz5MmTrLGxkTHWVhT4zmlPxvvLjRs3jk2bNo3t2bOH14z2\n5rxx4waLjIxkdXV1jLG2IirGnEuXLmW///3v2zMOHjyYmc1mXnN++eWXzGQyPfQgF8Pxw5jtnEIf\nP/fZysmY8McQY7ZzOnIMcXpvGanMi7cnZ3JyMry8vNpz8j29056MALB+/Xo8++yz8PHx4TXfffbk\nzMvLw69//ev2wfghQ4aIMuewYcNw69YtAMCtW7fg7e0Nd3erPZece+qppzBo0KCHPi+G4wewnVPo\n4+c+WzkB4Y8hwHZOR44hTou7VObF25PzQVu2bIFGo+EjWjt7/5b79+9vvypYoVDwmvF+Bls5q6ur\n8eOPP2LcuHEYNWoU/vrXv/Id066cGRkZqKiowPDhw6FSqbB27Vq+Y9okhuOnp4Q4fuwlhmPIHo4c\nQ5w2S+z9w7BO0yL5/oP2ZHtHjx7F1q1bUVxc7MREXdmTMTs7G6tWrWq/sVDnvysf7MlpNpthMpnw\nxRdf4O7du0hOTsYTTzyB0NBQHhK2sSdnTk4O4uLiYDQacfHiRUycOBGlpaXo378/DwntJ/Tx0xNC\nHT/2EsMxZA9HjiFOizuX8+KdyZ6cAFBWVoaMjAwUFRXZPLXjmj0Zv/76a6SnpwMAGhoacODAAXh4\neHSZrip0Tn9/fwwZMgSenp7w9PTE2LFjUVpaymtxtyfnyZMn8dZbbwEAgoODMWLECHzzzTcYNWoU\nbzltEcPxYy8hjx97ieEYsodDxxBH4wGMMcbMZjMLCgpiNTU1rKmpyeaA6qlTpwQZaLEn53fffceC\ng4PZqVOneM9nb8YHzZkzR5CRfntyVlVVsfHjxzOLxcLu3LnDoqOjWUVFhehyvvrqq2zZsmWMMcZ+\n+OEH5ufnx65fv85rTsYYq6mpsWtAVajj5z5rOYU+fh5kLeeDhDqG7rOW05FjiNOWu1TmxduTc8WK\nFbhx40Z7X5yHhwdKSkpElVEM7MkZHh6OKVOmIDY2Fm5ubsjIyEBkZKTocv7xj3/E3LlzoVKp0Nra\ninfffReDBw/mNefzzz+PY8eOoaGhAf7+/li+fDnMZnN7RjEcP/bkFPr4sTenWNjK6cgxxNuXdRBC\nCOGPy3zNHiGEuBIq7oQQIkNU3AkhRIaouBNCiAxRcSeEEBmi4k4IITL0/wFbSsINJOntCgAAAABJ\nRU5ErkJggg==\n"
}
],
"prompt_number": 284
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Linear Threshold Units as Measures of Similarity"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(linear threshold units as measures of similarity)\n",
"\n",
"Assume both inputs and weights are normalized.\n",
"\n",
"$$ ||w||=1, ||x||=1 $$\n",
"\n",
"The dot product becomes the cosine of the angle, the _cosine similarity_ between the vectors $w$ and $x$:\n",
"\n",
"$$ y_l = w\\cdot x = cos \\angle(w,x) $$\n",
"\n",
"Alternatively, we also get a relation to the Euclidean distance:\n",
"\n",
"$$ ||w-x||^2 = w^2 - 2w\\cdot x + x^2 = 2-2w\\cdot x $$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(pattern recognition)\n",
"\n",
"Perceptrons therefore detect pattern as follows:\n",
"\n",
"- store normalized target patterns in the weights $w$ of each neuron\n",
"- normalize the input vectors $x$\n",
"- an output value above some threshold indicates that the pattern is likely present\n",
"\n",
"This is implicitly the basis of much of machine learning, and it explains why perceptrons and \"neural networks\" with McCulloch-Pitts neurons are so successful."
]
}
],
"metadata": {}
}
]
}