mirror of
https://github.com/saymrwulf/nvidia-AI.summerschool17-notebooks.git
synced 2026-11-22 10:03:18 +00:00
435 lines
136 KiB
Text
435 lines
136 KiB
Text
|
|
{
|
||
|
|
"metadata": {
|
||
|
|
"name": "08-associative-memories"
|
||
|
|
},
|
||
|
|
"nbformat": 3,
|
||
|
|
"nbformat_minor": 0,
|
||
|
|
"worksheets": [
|
||
|
|
{
|
||
|
|
"cells": [
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Author: Thomas M. Breuel\n",
|
||
|
|
"Title: Associative Memories\n",
|
||
|
|
"Institution: UniKL"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"from pylab import *\n",
|
||
|
|
"import tables"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 4
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Input Patterns\n",
|
||
|
|
"=============="
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"As input patterns, we use images from the MNIST database.\n",
|
||
|
|
"We're not performing character recognition with this database here,\n",
|
||
|
|
"but just using the digits as simple patterns.\n",
|
||
|
|
"(We will be performing true character recognition later.)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"h5 = tables.openFile(\"mnist.h5\",\"r\")\n",
|
||
|
|
"images = h5.root.images[:100,:,:].copy()\n",
|
||
|
|
"images = array([1.0*(image>0.5) for image in images])\n",
|
||
|
|
"w,h = images[0].shape\n",
|
||
|
|
"n = w*h\n",
|
||
|
|
"def cshow(image): imshow(image.reshape(w,h))\n",
|
||
|
|
"gray()\n",
|
||
|
|
"subplot(131); cshow(images[0])\n",
|
||
|
|
"subplot(132); cshow(images[1])"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAPYAAAB/CAYAAADcrT/YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnWtsm9X9x7+OYztOfEkcX2LHaXyJnfutjVoKFDqgsDfr\nQJ2AaYNqhW3ixS4qGiBNG2wvWPuiYh3bCzR1qNMktL1hIA0qhqZ25Rpo0+bqJE1s14kvcRKniZ34\nEvv5v+j/OYtzcWLHTh475yM9EvTxc3Lsr79+nnPO7/x+PIZhGFAolIKiaLc7QKFQsg81NoVSgFBj\nUygFCDU2hVKAUGNTKAUINTaFUoBkbOxLly6hoaEBFosFZ8+ezWafKLsI1bVAYDJgeXmZMZvNjN1u\nZ6LRKNPe3s4MDg6S8wDowaEjW7pSbbl1pCKjO3Z3dzfq6upgMBggEAjw9NNP47333sukKQqHoLoW\nDhkZe3JyEjU1NeT/9Xo9Jicns9Ypyu5AdS0cMjI2j8fLdj8oHIDqWjhkZOzq6mq4XC7y/y6XC3q9\nPmudouwOVNcCIr1ps7vEYjHGZDIxdrudiUQidPKM40e2dKXacutIRTEyoLi4GH/84x/x2GOPIR6P\n47nnnkNjY2MmTVE4BNW1cOD9/69wdhulYzVOkU2JqbbcIZWuNPKMQilAqLEplAKEGptCKUCosSmU\nAoQam0IpQDJa7qJQChWZTAalUgmlUomKiooNX5dIJDA9PU2OpaWlHezl5lBjUygrkMlkMJvNqK+v\nh9Fo3PB18XgcQ0NDsNlsCIVC1NgUCpeRyWQwmUw4dOgQ9u/fv+HrotEoxGIxFhcXcfv27R3s4dbI\n2NgGgwEymQx8Ph8CgQDd3d3Z7FdeIRAIUFJSArFYjJKSkqRzy8vLCIfDWFpaQjgchlAoJK8TCoUb\ntplIJBAOh8m18Xg8128DwN7TtaioiOhRUlICo9GIuro61NfXo6mpacProtEopqam4Ha74XK5iF6s\nzrtNxsbm8Xi4fPkyFApFNvuTl0gkEuj1enKsZGFhARMTE+RQKBTkdUqlcsM2Y7EYucblciEUCuX6\nbQDYe7oKBAJUVVWhpqYGer0e9fX1qKurSzm+Bu7+IKjVajQ1NYFhGOh0uiSdd5ttPYrnIBo1Lykr\nK4PBYEBHRwfa29uTzvn9fvT09CAWi8HtdqOiogIWiwWdnZ0px3DhcBg3btwAAExPT++YsYG9patA\nIIBOp0Nrays6OztRW1sLjUazZWM3NzdDqVRCq9Wip6cHS0tL+W1sHo+HRx55BHw+Hz/+8Y/xwx/+\nMJv9yiskEgkMBgO6urrw8MMPJ527ffs2otEoPB4P+Hw+FAoFrFYrDh8+jI6Ojg3bDIVC4PF4mJ6e\nhs1my/VbIOw1XQUCAbRaLVpbW/HQQw9BrVaDz+eDz+envK6oqAgajQaVlZWor6+HWq3G0tJS0rbX\n3SRjY3/66afQarXw+/04duwYGhoacOTIkWz2LSdUVFSgoqIC5eXlKC0tzUqbBoMBVqsVWq0WZWVl\na/5ebW0tWltbsbi4iMbGRphMJqhUqjWvXQnDMBCJRBAIBDu68SJfdU0HqVRKvgc6nQ5NTU2oqamB\nXC6HWCzecjsrfwDEYjGEQuGmPwg7RcbG1mq1AACVSoUnnngC3d3dnP8C8Hg8KJVKWCwWWCyWlGPc\ndFCpVDCZTKisrFxzTiwWo6amBrFYDDKZDDqdDiaTCTKZLCt/O9vko67pUl5ejrq6OvI9MJvNqK6u\nTjmZmW9kZOzFxUXE43FIpVKEQiF89NFHePXVV7Pdt5ygUqnQ2NiIe+65BwaDISttisViyGQyyOXy\ndc+xdwOTyYSysjLIZDJOGjufdU0HuVyOuro6HD58GO3t7US7PW9sn8+HJ554AsDd5Zzvfe97ePTR\nR7PasXQoLi6GQCAgx0bweDzo9XpYrVa0t7fDYrHkpD+xWAzRaBSxWAzxeBxFRUUoLy9HeXk5ec3S\n0lLKoIZQKIRgMIhwOIxEIpGTfq6Ga7pmi6KiIvLdEAqFqK6uRl1dHVpaWlLOc6wmHo8TXZeXl5Pa\n5PP55Ae+srKSvI79Duw0GRnbaDSSGVsuIJfLodVqodVqoVarU762vb0dtbW1Kce328Xv98Pj8cDj\n8eDOnTsZtRGJRNDb24uJiYkdWxflmq7ZQigUQqfTQavVkjG12WxO+qHdCgsLC3C73XC73Zienk5q\nUyKRoLa2Fl1dXRAKhXC73fB4PHC73VhYWMjRO9uYgog8k8vlMJvNaG1thdVq3fB1PB4POp2OCJEL\nGIbB9PQ0hoaG0NfXB4/Hk1E7sViMfDG4Fq6Yb4hEIuh0OrS1taGtrQ21tbXQ6XTrDp1SsbCwALvd\njt7eXoyOjpL2KioqyJKnUCiEVqtFb28vent7MTc3R42dKSvDAA8dOpTytUKhkBy5wu/3Y2hoCFev\nXsWtW7cyaoNhGESjUXJQMod9/G5vb8dDDz0EpVKZ0XeANXZ3dze++uorxONxEpdQUVEBg8EAnU6H\nlpYWCAQC3LlzB2NjYzl6V6kpCGMLBAKUlZWhvLwcKpVqt7uDaDSKUCiEQCAAv9+/293ZEwgEAjIJ\nJpPJUFT0vx3JCoUC9fX1JPhEKpVm9DeWl5exuLiIO3fuYHp6GvPz8yTclx1ji8ViSKVS7Nu3DxaL\nBX6/H2KxGPPz8+TYiQCggjA2hVJSUgK9Xg+z2QyTyZR0N5ZIJLBYLNBqtSknV7MFu6za2NiI4uJi\naLVajI+PY3x8HPPz8zn/+wA1NqVAYI3d2dmJgwcPJm3GEQqFqKiogEKhQHFx7r/yPB4PKpUKxcXF\nqKqqglarRUlJCRYWFjA+Pk7v2FslkUggFoshEolgaWkJfD4fRUVFJAooHo8jkUiQRyb2XKqILoZh\nyDWJRCJ1qteioi21SckdIpEIWq0WjY2NuO+++7a86rFaZx6PR7Rc+Ti/HvF4HLFYjOzCYyPRioqK\noFAooFAoYDabUVFRgZmZGdy6dWvHvh8FYWx2kkImk2FxcRFqtRoajQYajQbLy8vw+Xzw+Xzw+/3k\nnFqtTjnWYrflsdemmsCSyWSkXS6M8SlbJxaLJeksFovJd2e9SEKWRCIBn8+HwcFBiEQiTExMkOs2\nW3LdCQrG2OPj44jFYvD5fGhqakJzczPkcjnC4TAcDgcGBgYwNDSE5uZmNDc3o6ysbFNju91uDAwM\nYGBgIOXuKq1Wi6amJvD5fGrsPIPddTc4OIj+/n4oFAo0NzdDKBRuauypqSkMDAxgfn4ebrcbzc3N\n4PP51NjZYn5+HmNjY3C73RgdHUU0GiVr28FgEE6nE9euXcMnn3yCSCRCgglSwRq7t7cXV65cwdzc\n3IavraurA5/PR1VVVbbfGiXHxGIxeL1e9PX14fLly9DpdBCJRKiurk55HWtsdglsamqKU9+BlMY+\ndeoU/vWvf0GtVqOvrw8AMDs7i6eeegpOpxMGgwH/+Mc/0o7gyTaRSASRSAR37tzB/Pw89Ho9dDod\nqqqqEAqFMDo6ivHxcTgcDjQ0NGB+fh6xWGxNO4uLiwiFQgiFQuRHYmxsDA6HI6Wx+Xw+9Ho9tFot\nlEolJiYmMDs7y9n153zRdTNKS0shkUhQVlaGffv2oaqqClKpdNOx8dLSEtHZ6/Um6RyJRIiWq4OY\n7HY7/H4/FhcXwTAMFhcXsbi4CODujrFAIMCZYKKUxv7BD36An/zkJ3j22WfJv505cwbHjh3DSy+9\nhLNnz+LMmTM4c+ZMzju6VeLxOHw+HwYGBhCPxxGJRDA8PIzp6elNrw0EAnA6nXA4HBgfH4fNZoPH\n41n3R2AloVAIDoeDzHzabDY4nU4Eg8Fsva2sko+6rkdlZSUMBgNqa2thNpvR2NgIjUaz6dbJO3fu\nwOFwJOnsdrsRjUbJ059QKMTU1FTSdR6PB8PDw5iZmcnl28oKKY195MgROByOpH97//33ceXKFQDA\nyZMncfToUU59AdjJMtbg8Xgc
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 8
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Willshaw Associative Memory\n",
|
||
|
|
"===========================\n",
|
||
|
|
"\n",
|
||
|
|
"Willshaw's associative memory (or perhaps Steinbuch's Lernmatrix) is a very\n",
|
||
|
|
"simple example of an associative learning rule, developed in the early 1960's."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"We're considering a grid of n binary inputs $x\\in \\\\{0,1\\\\}^n$\n",
|
||
|
|
"and n binary outputs $y\\in \\\\{0,1\\\\}^n$.\n",
|
||
|
|
"\n",
|
||
|
|
"The output is determined as \n",
|
||
|
|
"\n",
|
||
|
|
"$y = \\theta(C \\cdot x)$\n",
|
||
|
|
"\n",
|
||
|
|
"Where $\\theta$ is a threshold function and $C$ is an $n\\times n$ matrix."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Given training patterns $\\xi^{(k)}$, the entries of $C$ are determined as:\n",
|
||
|
|
"\n",
|
||
|
|
"$C_{ij} = \\min(1,\\sum_k \\xi_i^{(k)} \\xi_j^{(k)})$\n",
|
||
|
|
"\n",
|
||
|
|
"The goal of this associative memory is that if a partial or corrupted\n",
|
||
|
|
"version of one of the training vectors $\\xi^{(k)}$ is presented,\n",
|
||
|
|
"the original training vector is reproduced."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Note the close connection with two aspects of neurons:\n",
|
||
|
|
"\n",
|
||
|
|
"- output values are a thresholded sum of the inputs, just as in the McCulloch-Pitts neurons\n",
|
||
|
|
"- the training rule for the $C_{ij}$ is a _Hebbian rule_: the value increases if input and output are active simultanously\n",
|
||
|
|
"\n",
|
||
|
|
"There has been a lot of work on this particular memory, determining\n",
|
||
|
|
"its capacity etc."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"We are just going to train two patterns here to keep things simple and robust.\n",
|
||
|
|
"\n",
|
||
|
|
"The capacity of this associative memory is fairly limited in this example because\n",
|
||
|
|
"the patterns don't satisfy the randomness and sparsity criteria that make\n",
|
||
|
|
"Willshaw's associative memory work better than this. "
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"C = zeros((n,n))\n",
|
||
|
|
"for i in range(2): C += outer(images[i].flat,images[i].flat)\n",
|
||
|
|
"C = minimum(1,C)\n",
|
||
|
|
"imshow(C)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 9,
|
||
|
|
"text": [
|
||
|
|
"<matplotlib.image.AxesImage at 0x300d350>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAQMAAAD9CAYAAAC4PhIWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGxpJREFUeJztne1vFNcVxh+78CmK7Coyi+VFNTH2Ghszu8HEXyJtFsdY\naoITBEmhEqQJSj+gKgqVEM0fgNckKq5R2w+tQLIaCadqpQaqYgxa2yCgIMUvUbAUkFgaY2y32N4E\nnBcb+/QDu5P1eubOrHfHXnufn3QEM8/ee+fMy/HM3DP35oiIgBCS9eQu9QYQQjIDBgNCCAAGA0JI\nFAYDQggABgNCSBQGA0IIAIeCQXt7O8rLy1FaWopjx4450QQhJM3kpDvPYGZmBh6PBxcvXkRRURG2\nbt2K06dPY+PGjelshhCSZtJ+Z3Djxg1s2LABxcXFWL16Nfbs2YNPPvkk3c0QQtLMqnRXODQ0hHXr\n1unLbrcb169f15dzcnLS3SQhxCaqB4G03xnwYidkeZL2YFBUVITBwUF9eXBwEG63O93NEELSjaSZ\n6elpefbZZyUcDsv3338vmqbJwMCArgOg0WhLZCrS/s5g1apV+P3vf4/6+nrMzMzgwIED7EkgZBmQ\n9q5Fywb5ToGQJWNRXyASQpYnDAaEEAAMBoSQKAwGhBAADAaEkCgMBoQQAAwGhJAoDAaEEAAMBoSQ\nKAwGhBAADAaEkCgMBoQQAAwGhJAoDAaEEAAMBoSQKAwGhBAADAaEkCgMBoQQAAwGhJAoymDw9ttv\nw+VyoaqqSl83Pj6Ouro6lJWVYfv27YhEIroWDAZRWlqK8vJydHR0OLfVhJD0oxo6+dKlS9LT0yOb\nNm3S1x0+fFiOHTsmIiJNTU1y5MgRERG5efOmaJomU1NTEg6HpaSkRGZmZubViQwYLppGy1ZTYTlv\nQjgcnhMMPB6PjIyMiIjI8PCweDweERFpbGyUpqYm/Xf19fVy7do1BgMaLYNMRdLvDEZHR+FyuQAA\nLpcLo6OjAID79+/PmTnJ7XZjaGgo2eoJIUtESi8Qc3JylPMgcI4EQpYPSQcDl8uFkZERAMDw8DDW\nrFkDYP4ci/fu3UNRUVGaNpMQ4jRJB4OGhga0trYCAFpbW/Haa6/p69va2jA1NYVwOIzbt2/j+eef\nT+/WEkKcQ/VCYc+ePVJYWCirV68Wt9stp06dkrGxMamtrZXS0lKpq6uTiYkJ/fdHjx6VkpIS8Xg8\n0t7eblgnMuAlCo2WraaCcy0SkkWoLndmIBJCADAYEEKiMBgAyM/PR15enqnu9/vh9/tT0jVNU5ZV\n6fn5+cq6zcjPz5+jJ7aRWDbdejLLRv7HrzPy065udnxjZcz2YezYWOlmWOmZxqql3oBMQfUuIzdX\nHTPt6Ga/ia230tOxbYk+JpZNt57MspEf6dSNju9i7vvlAF8gEpJF8AUiIcQSBgNCCAAGA0JIFAYD\nQggABgNCSBQGAwCBQEDZHyxPBoFJSe/q6lKWVemBQEBZtxmBQGCOHgqFlGXTrSezbOR/vG7kp13d\n7PjGypjtw9ixsdLNsNIzDQaDJPB6vfPWxV+oiXogEFBeyCoNgDLZKF34fD7L3xj5HY+VH4FAwNKX\nVHy1at/r9Zr6acf/hWBVr9W5sSQoP2NyAGTAl1sLMU3TpLm5ed76rq4u3RL1eM2oTivd5/OZavF1\npKK3tLRY+m7kd7LboGmaY36o9qGV3tLSYlnWzH879Vod+1TPy2RNBZOOCMkiVJc7HxMIIQAYDAgh\nURgMCCEAGAxs4/f753WNAZjT/WTU9SY2uqbM9MSuQbM6UtHNujTjMfI72W2w+pQ3FT9U+9BK7+rq\nsixr5r+deq2OfSbBYAB7eQZ2LhoVVuVV+kK7oBLLZXKegRFO63YDUMZ1AToEgwEhBIDF4CaDg4PY\nv38//vvf/yInJwe//OUv8e6772J8fBw/+9nP8J///AfFxcX461//qieNBINBnDp1Cj/60Y9w4sQJ\nbN++fVEcSYXOzk6lnpOTA03T8Itf/GKe1t3drf+/v7/fVHvxxReVZY30SCSC7u5u5fbF15FIZ2fn\nHH3btm1z9JycHLS0tCh14Ae/zPRYG4l6bLm7uxuRSMRUN/PDrm62D6z0RP+N9K6uLvT29hrqqn2f\neC4kU3bJUCUhDA8PS29vr4iIPHz4UMrKymRgYCClyVexyEkWNBrtB1ORVAbiq6++KhcuXEhp8tWl\n3hk0WjabCtvvDO7evYve3l7U1NRw8lVCViC2gsGjR4+wa9cutLS04Omnn56jZcvkq+xaTG0b2LWY\n3DYvBZbBYHp6Grt27cK+ffv0eRU5+SohKw9lMBARHDhwABUVFXjvvff09Stt8lXmGTijM89geaHs\nWrxy5Qo++ugjbN68Wf8+OxgM4je/+Q3eeOMNnDx5Uu9aBICKigq88cYbqKiowKpVq/DHP/5xWTwm\nTExMKE+MWNdeYtcY8OQxKHbSmXW9hUIhw66m+LJGemdnJzo7OzExMWG5bUZMTEzM0fv6+uaVjd9m\nIx34wS8zPeZnoh5bttIB4He/+9287Y/XjfyMr9/owo/pPp/P8PjG/DcLGlZdzir9xRdfVAajTLwu\n+AkzIVmE6nJnBiIhBACDASEkCoMBIQQAg4FtmGeQ2jYwzyC5bV4KGAzArkWndHYtLi8YDAghACzy\nDLIF5hmY6wDzDMxgnkGqDWbgTiAkW2CeASHEEgYDQggABgNCSBQGA5vk5eVB07R56/1+v26Jerxm\nhJWen59v2T+fqm7kU7K/sbMNeXl5Kdeh0haqa5pmWdbMfzv1Wh37jEI5DpIDIAOGfkq0QCAgfr/f\ncqioUCikHEYqUY/HaJJNKz0QCIiISCAQUG6byq943Wj74ts10uPXm+mJv0usT0TE7/eb6mZ+2NXN\nzi0rPea/2T60Om9VWqzehZR10lQwGACSn58veXl5prrf71cGCzu62SzEsbIqPT8/X1m3yq94PbGN\nxLLp1pNZNvI/fp2Rn3Z1s+MbK2O2D2PHxkpf6HmxFKaCXYuEZBGqy53vDAghABgMCCFRGAwIIQAY\nDAghUZTB4LvvvkNNTQ28Xi8qKirw/vvvAwDGx8dRV1eHsrIybN++HZFIRC8TDAZRWlqK8vJydHR0\nOLv1hJD0YdUVODk5KSIi09PTUlNTI5cvX15xcy0yz8Bcj1/PPANj3Uhb0XkGk5OTUl1dLZ9//jnn\nWqTRlqmpsHxnMDs7C6/XC5fLhUAggMrKSs61SMgKxHJwk9zcXPT19eGrr75CfX39vAEdsmWuRUJW\nOrZ7E/Ly8vDyyy/j008/5VyLhKxAlMHgwYMHek/Bt99+iwsXLsDn8624uRYJIVC/Ufjss8/E5/OJ\npmlSVVUlH3zwgYiIjI2NSW1trZSWlkpdXZ1MTEzoZY4ePSolJSXi8Xikvb19Xp3IgJcoNFq2mgp+\ntQh+teiUzq8Wl9dXiwwGYJ6BSo9fzzwDY91IW455BvyEmZAsQnW589sEQggABgMATybZUI3z19XV\npZz+zI7e0tKiLKvSfT6fsm4zfD7fHL25uVlZNt16MstG/sfrRn7a1c2Ob6yM2T6MHRsr3QwrPdPg\nYwIhWQQfEwghljAYEEIAMBgAsDclu+r2yo5u9uwYK6vSVVOCq9oNBAJz9Eyekt3I/3jdyE+7utnx\njZUx24exY2Olm2GlZxzKjkcHwCL3q6bLNE2T48ePG/ZlxyxRj9eM6rTSvV6vqRZfRyp6c3Ozpe9G\nfie7DWZJVenwQ7UPrfTm5mbLsmb+26nX6tinel4mayr4ApGQLEJ1ufMxgRACgMEAAPMMnNKZZ8A8\nA3WDfEwgZMngYwIhxBIGA0IIAAYDAMwzcEpnnsHyyjPgOwNCsgi+M0gDmqbNexsOzH3jbPS23c7b\naDM9sTfArI5UdLNejHiM/E52G1S9NXbrUGkL1VtaWizLmvlvp96F9kQsBbwzICSL4J2BBcwzcEZn\nnkHm/fVXYud7gsePH4vX65VX
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 9
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Now we can reconstruct patterns from corrupted inputs."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"subplot(131); cshow(images[0])\n",
|
||
|
|
"corrupted = (images[0].flat+0.5*randn(n)>0.5)\n",
|
||
|
|
"subplot(132); cshow(corrupted)\n",
|
||
|
|
"out = dot(C,corrupted)\n",
|
||
|
|
"subplot(133); cshow(out>0.7*amax(out))"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAWwAAAB/CAYAAADLqvg4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnWtwY3d5/79H95vlm2RZsrySZcv3a9bsZsMu5EICvCCF\noQN0Wsg00DK8aKcTpsAbaNoXkLxgOpR2pgyknTCdYeAFFGaAJKUhIWyS3WTXt/VVlixZsu62btb9\ncv4v9v/7VfJFlmTZlr3nM+OB7Dk656fznN+jc57f83wfhmVZFhwcHBwcDQ/vrAfAwcHBwVEZnMPm\n4ODgOCdwDpuDg4PjnMA5bA4ODo5zAuewOTg4OM4JnMPm4ODgOCfU7LBfeeUVDA4Owmw248UXX6zn\nmDjOEM6uFxPOrhcEtgZyuRzb29vLbmxssJlMhp2YmGCXlpbodgDcXwP91cuunG0b64+z68X8K0dN\nT9i3b99GX18fjEYjhEIhPve5z+GXv/xlLYfiaCA4u54+ly5dwpe//GX8/Oc/RzKZxO9//3tcv34d\nExMT0Gq1ePbZZ/HTn/4UkUgEf/zjH/HNb34T09PTVZ2Ds+vFQVDLh7a2ttDd3U3/W6/X49atW3Ub\nFMfZ0Ih2VSqV9I9hGESjUfrHFhXpCoVCKJVKNDc3Q6lUIpFI0P0SiUTF55PJZPR8Uqm05HzZbJbu\nxzAMPZdSqUShUEA0GkUgEEA6na74fJlMBsFgEDabDbOzs7BYLNjd3UUymSzZ76233sLPf/5zrK+v\nw+12V3x8oDHtylEbNTlshmHqPQ6OBqAR7apSqWAymdDb2ws+nw+bzQar1Yrd3V3k83m6n0QigV6v\nR29vL0wmE3w+H6xWK2w2W1UOW6lU0mNoNBp6DKvVWuKweTweOjo6YDKZYDKZkMvlYLPZsLu7W5XD\nTqVScDqdkEgkiMfjcDqdCIVCyGQyJfvduHEDLS0tePXVV5FIJKpy2o1oV47aqMlhd3V1wel00v92\nOp3Q6/V1GxTH2dCIdlWpVBgaGsLVq1chEAggkUgQi8Vgt9sPdNhTU1O4cuUK1tbWwLIsgsEgvF5v\nxedrbm5Gb28vrly5gr6+PjQ1NSGTyWBrawu7u7t0P+KwR0ZGcOXKFaTTaYhEIjidToTD4YrPl0wm\n4XQ6sbu7C4fDQd8IEokElEplxccpRyPalaM2anLY09PTsFgssNvt0Ol0+OlPf4qf/OQn9R4bxylT\nqV2lUikAoFAooFAoIJ/Po1AonMiYmpub0d3djbGxMQiFQvh8Pqyuru57ahSJRFCr1ejt7cVDDz0E\nHo+Hzc1NyGSyqs4nk8mg1WoxODiIiYkJGq4QCoUl+/F4PLS2tsJgMGB8fBypVAoejwd6vR6rq6sV\nny+dTsPj8cDj8ezb1tTUhFwuh0wmg1QqhXQ6jWw2W/W15ubrxaEmhy0QCPCv//qv+OhHP4p8Po8v\nfvGLGBoaqvfYOE6ZSu36sY99DAAQCoXg8/ng8/mws7NzImMKBoNYWVmBTCaDQCDA0tIS/H7/PqeV\nSqXgcrkwOzsLhmFgs9lgs9kQjUarOl80GoXVaoVSqYTf78fs7CxcLte+MEehUIDP58PS0hLEYjHS\n6TRWVlawvb197O9MSKfT2NrawtzcHHg8HhwOB6xWKyKRSFXH4ebrxYFh2frLq3Ixs8ainiZmGAb/\n/u//DgCw2+1YXFzE4uIibDZb3c5RTHd3N/3j8XhwuVzY3NyE0+kscdpyubxk32AwCKfTCafTiWAw\nWPH5VCoVLl26hO7ubrS3t9NjbG5ulsTC+Xx+yflyuRzd1+Vy1eW7y2SyknPs7OzQcwQCgbrblaMx\nKGdXzmE/ANR7Ym9sbAAA5ubm8Oabb+LNN9/E3bt363aOYqRSKf1jGAbJZJL+FX8vPp8PqVQKmUwG\nqVSKdDpN99u7gFcOkUhEzycSiUrOVxwzZximZGwsy5bsWw/IdyJ/2WwWyWQSiUQCmUyGc9gXlHJ2\nrSkkwvFgk0qlANxPScvlckfGVHk8HuRyOf0rjgezLIt4PI54PI7d3d2STAwAZR2gTCaDQqGAXC4H\nj8ejxwkGgyVjEggEJefPZDJ0373HzmQyyGQyR4YdWJali4MnRT6fx+7ubsliJ8eDDeewOarmlVde\nAQBsbm5ifX39yKwIHo8HjUYDo9EIg8GApqYmui2fz8PhcMDhcMBut1cVn21vb6fHFIvF9BgOh6PE\nYYtEIuh0OhiNRhiNRuzs7ND96vU0zMFxGnAOm6NqXnvtNQBAOBxGMBg80mHz+Xx0dHRgeHgY09PT\nUKlUdFs+n8f7779PF/Gqddj9/f24fPkyFAoF7ty5g2QyCZfLVfKkLhKJ0NXVhfHxcUxPT8PpdILP\n5yMcDleV8sfBcdZwDpujasgTNuGoWCqfz4dGo8Ho6Cg+/OEPl1TdZbNZsCyLQCCApaWlqsbR1taG\ngYEBfPCDH0RrayvS6TR1xsWIxWLodDpMTEzgiSeewMrKCsLhMKxWa1Xn4+A4aziHzVE11S525fN5\n+P1+LC0tQSqVoqenByqVCiqVihaHVLLoJZVK6edUKhUMBgMymQwWFhYgkUgQiUSg0+nw2GOPwe/3\nIxgMIhgMIpPJwO12Y35+HmKxGE6n89D0OKVSSY/f1NREjxEMBiESiei2lpaWkm1caIXjNOAcNseJ\nk8/n4fP5sLi4iHg8DrfbjaGhITAMU1U1n1QqRXd3NwYHBzE4OIhcLodUKoX5+XmwLAuxWIzOzk4Y\njUbY7XYsLy8jk8lgZ2eHOuxoNIpQKISNjY0DQzmkNH1gYABdXV1YXl7G6uoqYrEY5HI5jEYjhoaG\nYDQasby8jJWVlQMXLzk4TgLOYXOcOCQ+HY/HYbfb4fP5wDAMVCoVenp6Kj4OcdgPPfQQrl+/jpWV\nFczOzmJ+fh7xeByTk5MwmUyYnJzEvXv3kMlk4PF44PV64Xa7EYlEYLVakclkDs2+UCqVMJlMuHr1\nKgYHByEWi7G7u4uNjQ00NTWhp6cHH/jABzA5OQmpVIpEIoHNzc16Xi4OjkOp2WEbjUYolUrw+XwI\nhULcvn27nuM6VwiFQkgkEkilUkgkkpJt5CkwmUwilUrRPF+JRAKRSHToMQuFAlKpFP1scQ7wSVIv\nu4pEInpNSD5zLBaD3+8HwzDo7OxEV1cXNBoNAoEAYrEYcrkcBAIBvT6kBJ6g1Wqh0WjQ1taGpqYm\niEQisCyLVCqFRCKBfD5PU/ikUimEQiF4PB5yuRzC4XBFGh98Pp/aqKmpCW1tbdBoNNDr9VCpVOjr\n68PAwACGhoZgt9vR2toKoVB45D1A7H+WT+LcnL0PyaEn9ipe8yD3E7EXwzCH2nUvxZ/bm55aL2p2\n2AzD4I033kBbW1s9x3MuUSgU0Ov19K+YWCwGl8tF/9ra2uh+xdkSe8lms/QzTqcT8Xj8pL8GgPrZ\ntampCd3d3dDr9VCr1bQCkAgd2e12yOVyRKNRLC4uwul0IplM0opFco2KY9tisRhNTU1wuVyIxWKI\nxWIQiUQYHx9HLpdDS0sLPB4Pfv/738Nms2FjY6Om0nSbzQapVIpgMIhIJIKWlhY8/PDD6OjoQH9/\n/4HXRi6XQ6/X07EXE4lESmx5ArVqFcHN2fsIBAJoNBpqK7lcTrfl83l6r7pcLvD5/EPtupfieR4K\nhU5m7Mf58FndeI0GiW1OTk5iYmKiZFsgEMDMzAyy2SzcbjdaW1thNpsxNTVVNhyQSqUwOzsL4L6e\nxmk5bKA+dlUqlejp6cHU1BT6+vowMzMD4P53ISGGZDJJQyQ+nw+JRALNzc0wGAyYnJzE1NRUyTHj\n8Tg8Hg9cLhfee+89qNVqaLVaTExMQCQSwe12w+Px4O7du/B6vfB6vVU7bBI2Iep5Wq0WOp0Ow8PD\n6OzspE/4e1EoFDAajZiamtp3D3i9XszOziKdTsPlcp3pvOHm7P23qM7OToyOjmJqagqtra10Wy6X\nw927d8EwDLa3tyESiQ61617m
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 10
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"subplot(131); cshow(images[1])\n",
|
||
|
|
"corrupted = (images[1].flat+0.5*randn(n)>0.5)\n",
|
||
|
|
"subplot(132); cshow(corrupted)\n",
|
||
|
|
"out = dot(C,corrupted)\n",
|
||
|
|
"subplot(133); cshow(out>0.7*amax(out))"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAWwAAAB/CAYAAADLqvg4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXlsY9d1/7/cSZEiRY5EihQlUaQ4onZxRjPjGc94nzj5\nI0GCAFnQNm6dpEULtChcNElRoHULtLH/MIo2BdqgTVMHBdIGLdIEKODEhjO2Y8ezaqU2SlzETdxJ\ncV/f74/53RtSOylqtAw/gP6Y4ePj5TvvHt537jnfw2IYhkGTJk2aNDnxsI97AE2aNGnS5GA0HXaT\nJk2anBKaDrtJkyZNTglNh92kSZMmp4Smw27SpEmTU0LTYTdp0qTJKaFuh/3WW2/BZDLBaDTi9ddf\nb+SYmhwjTbueTZp2PSMwdVAsFhmDwcDY7XYmn88z4+PjzMLCAn0dQPPvBP01yq5N256sv6Zdz+bf\nXtS1wr5z5w76+/uh0+nA4/HwpS99CT/5yU/qOVWTE0Stdr18+TL+4i/+Ah999BFisRj+8z//E7/9\n27+Nzs5OmM1m/Nmf/Rlu3bqFVCqF//mf/8HXv/51dHd3P5LvcuPGDfzN3/wN7t+/j42NDfz7v/87\nvvzlL6Otre2RfP5Jojlfzw7cet7k8XiqJp5Wq8Xt27cbNqgmx0OtdrXb7bh//z70ej0KhQLsdjvC\n4TAKhQLS6TQ2NjZgtVrB4/Fgs9kQCoWQz+erzsFisSCVSukfwzDY3Nykf/WSTCbh9XqxtLSESCQC\np9OJSCSCUqlU9zlFIhEdZ2tra9Vr2WwWm5ubCAaDyOVydX+GRCIBi8WCVqsFl8ul1yEej4PD4UAg\nEEAoFCIUCh34nM35enaoy2GzWKxGj6PJCaBWuwoEAmQyGXz88cew2WxYWVmBx+NBPp9HIpGAzWYD\nl8uFx+OB1WqF2+1GNpvd9plKpRJ6vR56vR7lchk2mw02m+1QDjsajcJqtQIApFIprFYrfD4fCoVC\n3edsbW1FX18fDAYDent7q14LhUKw2WxIJpOHctgKhQJsNhvXrl2DWCzG2toabDYb0uk0Ojo6oNfr\nYTAY8P3vf//A52zO17NDXQ67q6sLLpeL/tvlckGr1TZsUE2Oh1rtGo/Hsbq6ikgkgpaWFkSjUUQi\nEeRyOTAMA5vNhlgsBolEQl/b6rDZbDY6OjowODiIJ554AqVSCXw+H5ubm7DZbGDqlLqJRqNYWVlB\nKBQCn8+nn18sFus6H/Brh33p0iWYzeaq15xOJ/h8PlwuF2KxWN2foVAo0N3djSeffBJtbW0QiURI\np9NYX1+HXC6H0WjElStXanLYzfl6dqjLYU9OTsJqtcLhcECj0eC//uu/8MMf/rDRY2vyiKnVrolE\nAolEAuvr69teKxaLyGazCAQCYLN/vVXCZrMhEAhQLpdRKpXAYrHQ1taG3t5ejI6OolQqwev1Ynl5\n+VDfJR6PIx6Pw+FwbHuNzWaDw+GAzWaDxWLRsZTL5T1/IMRiMbq7uzE2NoannnqKvqdUKkEqlcLn\n80Gr1R5q7FKpFJcvX8bw8DA6OjoQCASwuroKNpsNqVSK3t7ebT8W+9Gcr2eHuhw2l8vFP/7jP+LF\nF19EqVTCV7/6VQwODjZ6bE0eMY20a2trK1QqFVQqFdrb26te29zchN/vh9/vRyQSQSAQwMLCAoRC\nIUqlEpaWlmqK0dZKW1sbVCoVlEolJBIJ/H4/AoEA/H7/gcMZ5XKZvsfv92NlZQVLS0sIh8OHGlsk\nEsHKygrEYjFkMhnm5+exsbFxqNh7c76eHepy2ADwqU99Cp/61KcaOZYmJ4BG2VUqlUKv12N4eBhG\no7HqNZ/Ph/n5eRSLRYTDYQSDQSwuLiKZTKJcLsPlciEYDNYdDtkPmUyG/v5+DA8PQ6lUwmKxwGKx\n0HDOQSiXy/D7/VhYWIDFYsHq6ipcLtehf2iIw85kMhCJRFhfX4fP5ztUKAdoztezQt0Ou0mTvZBK\npTTee+XKlarXrFYrCoUCNjY2wDAMgsEgkskkHA4HGIZBJpNBJpM5srG1tbXBYDDgypUr0Ol04HA4\nNOZ9UMgKe2FhAe+//z7sdntDxh2JRJDJZOByucDhcOg5D+uwm5wNmg67SUNgs9mQSCQQi8UQi8Xo\n6emBQqEAl8vdttGYy+VQLBZRLpfpe7lcLgQCARiGQbFYPFSmxX6Uy2UUi0Xk83nkcjkUCgUachAK\nhRCLxZBIJBAIBEilUkgmk0ilUsjlcgiHw3C5XFhaWoLVasXq6irsdjs8Hs+un8fn8+l1EYvFSKfT\nSKVS9JyVZLNZZLNZRKPRI/v+TU4vTYfdpCHweDx0dnZCp9Oht7cXcrkcXC6Xhgoq2djYoPFektbX\n29sLnU6HcrkMh8MBp9OJVCp1JGONxWJYXV0Fj8fD6uoqlpaW4PV6kc/nIZVK6Xfo6OiA0+mEw+GA\nw+FAIpGA3W6HUCikYRyXy7XvqrqlpQU9PT30O7pcLnreo/xhanL2aDrsJg2By+Wis7MTIyMjuHjx\nIorFIpxOJ1ZXV7etPhOJBEKhEMLhcFVa3+TkJEqlEoRCIVKpFJxO55HEsePxONbW1hCPxyEWixEM\nBmlRT2dnJ/R6PSYnJ6HX63Hv3j0avkkkEnA4HEgmk1hdXaXvO6jDvnDhAiYnJzEzMwPgYe52JBJp\n+PdrcnZpOuwmDYE47NHRUTz77LPY2NigBTPvv//+ju9hGAY8Hg8dHR0YGhrCU089hWKxSJ31URGL\nxegqu3IswMMNSYPBgMuXL2NiYgLFYhEbGxuYn59HOBymq+yt79sL4rAvXryIF198EXw+H6FQ6NCp\ni00eP5oOuw44HA7a29vpn1Ao3PXYTCaDUChE/0jc9iwgk8noNVAqlVCpVIhGo/j444/h8/lgtVoR\niUS2OTWJRELf19HRge7ubqRSKdy/fx/JZLIh6XEHYSdnSwp27t69i0gkgpmZGXg8nqrQRa2r/nQ6\nDZfLhampKXA4HMzMzMDpdCKZTB76OzQ5GFKplN5zcrl81+PK5XLVfD3Kze96aDrsOuByuVCpVDCZ\nTBgYGNhTUCgajWJpaQnLy8uIRqNnzmEbDAaYTCb09PQgm80iEonA6/ViY2MDDodjx80ziUQCnU5H\n5T6z2SySySTu3buHUCgEu91+pGl9e0Ecdrlcht1uh8Ph2Oawa4VUKrLZbEQiERrDridG3ywzrw+p\nVAqDwYCBgQH09fXtelypVMLi4iKWlpaQSqWaDvsswOFwoFKpMDw8jOvXr6Ozs3PXY71eL00bI+ls\nZwXisJ944gkYjUZMT09jamoK09PT2NjYQDKZ3HEVKZFIaMrfpUuXqt7ncrl2fd+jgMS3/X4/hEIh\nHcvWTJdaIA47HA5jeXkZ6XT6WL/j4wipC7hy5QouXLiw63H5fL5KDuCkUbfD1ul0kEql4HA44PF4\nuHPnTiPHdeLg8/kQCoUQiUSQy+XQ6/UwGo0YHByEWq3e9X1SqRR+vx9erxculwvxeByZTAbZbHab\nct1JoBa7stls8Pl8iEQiiEQi5PN5BAIBWK1WBIPBPd/H4/FoCl2hUEA4HMba2lpNk0QoFFKbsNls\nel0zmUzV6pzD4dDjhEIhCoUCPW6rDcrlMvL5PDKZDMrlMi0Jl0qlKBQK9DNyuVzVORmGoefc6tzz\n+TwikciOG4xcLrfqe+RyOXqerT/uuVwOsVgMGxsbB75GlTxOc5bNZlPbCIVC9PX1ob+/HwMDAxga\nGtr1feQeJvO1XC7vatfjoG6HzWKxcOvWLSgUikaO58Qik8mg1Wqh1WrR29uL4eFhaLXaPePXwENJ\nTq1Wi4mJCfB4PKyvr8PtdsPtdh9p+XW91GLXzc1NrK2t0Yo8i8UCr9e7b/iAFMmIxWJsbm7CYrEc\nKD1uK0QoSavVQiAQwOVywe12w+VyVRWa8Pl8aDQaemw8HqfHBgKBqnNKpVJ63NYnJxLOcLvd8Pv9\nUCqV9NhisUjt6na7D/wdhEIh
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 11
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"\"Hopfield\" Network\n",
|
||
|
|
"=================="
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Hopfield Networks)\n",
|
||
|
|
"\n",
|
||
|
|
"The Hopfield network is another associative memory. \n",
|
||
|
|
"\n",
|
||
|
|
"Like Willshaw's memory, it is composed of McCulloch-Pitts neurons, and its learning rule is an associative learning rule.\n",
|
||
|
|
"\n",
|
||
|
|
"Unlike Willshaw's memory, it uses _recurrent connections_.\n",
|
||
|
|
"\n",
|
||
|
|
"That is, the output of the units at time $t$ becomes an input to the units at time $t+1$."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Update Rule)\n",
|
||
|
|
"\n",
|
||
|
|
"The update rule is:\n",
|
||
|
|
"\n",
|
||
|
|
"$s(t+1) = \\hbox{sgn}(W \\cdot s(t))$\n",
|
||
|
|
"\n",
|
||
|
|
"The learning rule is as before:\n",
|
||
|
|
"\n",
|
||
|
|
"$W_{ij} = \\frac{1}{N}\\sum_k \\xi^{(k)}_{i} \\xi^{(k)}_j$"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Note)\n",
|
||
|
|
"\n",
|
||
|
|
"However, input patterns are now assumed to be in $\\\\{-1,1\\\\}$. \n",
|
||
|
|
"They are also assumed to be random vectors with expected mean zero.\n",
|
||
|
|
"These conditions, however, are violated for the training examples below,\n",
|
||
|
|
"which is why the Hopfield network here has a much lower capacity."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Differences)\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"The version implemented here is not the original Hopfield memory but a similar variety (it uses parallel updates\n",
|
||
|
|
"and a slightly different weight matrix)."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"simages = [2*(images[i].flat>0.5)-1 for i in range(2)]"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 12
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Let's now create the weight matrix."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"W = 0.5*sum(array([outer(simages[i],simages[i]) for i in range(len(simages))]),axis=0)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 13
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"And now we perform the associative retrieval.\n",
|
||
|
|
"The images are the original pattern, the corrupted pattern, and the reconstructed pattern."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"subplot(131); cshow(simages[0])\n",
|
||
|
|
"r = sign(simages[0]+randn(n))\n",
|
||
|
|
"subplot(132); cshow(r)\n",
|
||
|
|
"for i in range(100): r = sign(dot(W,r))\n",
|
||
|
|
"subplot(133); cshow(r)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAWwAAAB/CAYAAADLqvg4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnVlsY9d9/7/cV1EiJZISRUnUQu2jZUae8dhje7KMnTzE\nTZAiSdE6RpO0RVC0KJw2yUPTun1I7AejSNM+BI0TuCgQJCiQxkATJ3Udr3FG9oz2ZUSRIkWK+ybu\n+/0/zP+ckqJEiRQlUZr7AQjM6N57eO793vPjvef8Fg7DMAxYWFhYWBoe7ll3gIWFhYXlaLAGm4WF\nheWcwBpsFhYWlnMCa7BZWFhYzgmswWZhYWE5J7AGm4WFheWcULPBfu211zA8PAyj0YgXX3yxnn1i\nOUNYXS8mrK4XBKYGcrkc09/fz2xtbTGZTIaZnJxkVldX6XYA7KeBPvXSldW2sT6srhfzU4manrBn\nZ2cxMDAAg8EAgUCAL3zhC/j5z39eS1MsDcRp6To2Noa//uu/xjPPPINMJoNXX30Vf/7nf46+vr66\nf1cjcuvWLbz00ktYWVkBwzAln9/85je4ceMGJicn6/Z97Hi9OPBrOWhnZwddXV30/3q9Hrdv365b\np1jOhtPSNZlMwuPxIBgMYm5uDmazGT6fD+l0uuJxQqEQCoUCzc3NUCgUiMViiEQiCAQCyOVyNfWF\nz+dDoVCgUCigu7sbmUwGkUgEkUgEsVispjYPIxKJYGdnB2tra0ilUiXbTCYTYrEYkslk3b6PHa8X\nh5oMNofDqXc/WBqA09I1Fotha2sL8Xgcv/71r7GxsYHt7e1DjZRUKkV3dzf6+vrQ398Pu90Oi8WC\naDRas8EWCATo6OhANpvFRz/6Uezu7sJiscBsNp+YwQ4EAlhfX0ehUMC9e/dKtm1tbSEUCiGTydTt\n+9jxenGoyWB3dnbCbrfT/9vtduj1+rp1iuVsOC1dY7EYrFYrBAIBfv3rXyMYDCIUCiGRSFQ8TiKR\noKurC5cvX8a1a9ewuLiIQqEAm8126LEHQQy2VqvFRz7yEXi9XvB4PASDQTgcjpraPAxisL1eL6RS\nacm2cDiMRCJR8/nsBzteLw41GeyZmRmYTCZYrVbodDr85Cc/wY9//ON6943llDktXYsN0tbWVsV9\neTweuFwueDwempub0dnZieHhYVy5cgXpdBo2mw0qlQput7tk32IYhkGhUEA+n0ehUABTlO+Mz+dD\no9HgySefxPXr17G9vQ2324319fW6nS+HwynpWyKRQDQaxdbWVklfTgp2vF4cajLYfD4f//Iv/4Kn\nnnoK+XweX/7ylzEyMlLvvrGcMo2oa1tbGzQaDbRaLTo6OtDU1ASHw4HXX38dKysrsFqtiMViEIvF\n0Gq10Gq10Gg04PP/79ZOpVLwer3weDzweDzIZrOneg5SqZT2TavVwufz0b5Eo9ET//5G1JWlNmoy\n2ADwyU9+Ep/85Cfr2ReWBqDRdG1ra8PIyAjGxsagVCoRiUTgcDiwuroKh8MBu92OaDQKsVgMvV6P\nsbExjI2NQSgU0jai0ShWVlbA5XIRDAbPxGB3d3djbGwM4+PjWF9fx8rKCuLx+KkYbKDxdGWpjZoN\nNgvLadDW1obh4WHcuHEDEokEH3zwAVZXVzE7O0u9KZLJJFQqFTo7OzE1NYUnnngCEomEtuH3+8Hh\ncBAMBssW+U4DYrAvX76MmzdvQi6XIxaLwWaznXpfWM43rMFmqRmxWAy5XA6ZTFZiIIH70xDxeBzx\nePxYC2iFQgHZbBbpdBqFQgHBYBButxs2m63sSTmfzyOTySCVSpV4RqTTaWSzWeTzeTAMA4lEAplM\nBplMBrVaDZlMhkQiga2tLdjtdvj9/n37LBKJIJPJIJfLIZFIEIvF6DlyOBx6LWQyWck2hmFK+pbJ\nZOh8OgtLNbAGm6VmWlpa0NPTA4PBgI6OjpJtXq8XNpsNVqv1WAbb5/NhfX2dLtwR74q9xi6VSsHh\ncGB+fh6pVKpkSiQWi2FtbQ0ulwu5XA7Nzc0wGAwwGAzQ6XSQSCQIBAJ499134XQ6YTKZEA6Hy/rS\n1NREz1en08FqtcJqtcJms4HP59Nt3d3d9O/k/Le3t3H37l0kk0m6ABiPx2u+LiwPJqzBZqkZhUKB\n/v5+zMzMYHh4uGSb2WyGUCikQSK14vf7sb6+jkAgAA6HA7/fD5/PV+ZdQQx2Op3Gzs5OiadIOp2m\nx+VyObS0tGBgYABXrlyBwWCAzWajH7fbDZ/Ph1AoVNaXpqYm9Pb2YmZmBiMjI/jwww+Rz+fh9Xoh\nFArR09ODK1euYHp6Gnfu3AHDMPB6vQiHw9je3kYikYDdbkcgEIDP5zsxP2+WiwtrsFlqpqWlBf39\n/bh27RquX79esm1ubo66rh0Hv99P/ZYJ+7nCEYNd6ceBHEcM9sMPPwyj0YhEIoG7d+/i3Xffhc/n\nO/A7iMG+evUqrl+/jkKhAK/Xi9XVVUgkEhgMBjz00EP42Mc+BuD+W8ba2hqcTidsNhu2t7crngML\ny2GwBpulZkhU4OzsbJm3g9lshslkQjAYrNgGh8NBW1sb/TAMA7/fTz/A0YybWCwuaScSidA2UqlU\nybbu7m4AwL179+BwOLC0tASn04l0Ol3xu6LRKKxWKz744APE43HMz8/D4XAgmUzSAJ47d+6gUCjg\n7t279KmacJTzUKlUtJ9CobDkWtQazclycWANNkvN7O7uwmw2I5/Pl0TSAfefLq1W675TC8VwuVxo\nNBoMDQ1hZGQE+Xwe6+vrdBrkqE+iIpEInZ2dGBkZwdDQEM3VkclkwDAMdDodhoeHMTIyAi6Xi1Qq\nhfX1dUQiEVitVjidzkPDwYvfGHZ2dmC1WmG325FMJpHNZmGz2cDlcuHz+WC1WrG9vV31PLVSqYTR\naMTw8DCampqwtraGe/fuIRwOswabhTXYLLVDDLbH49nXSyQWix06T8vhcKBWqzE6OorHHnsM+Xwe\nHA4HgUCgKhc84oc9NTWFGzduYHV1FZlMBi6XC8lkEp2dnZicnMRjjz0Gt9uNubk5rK2tYX19nfbz\nMINNcqD4fD5IpVJ6XDKZBJfLhdVqhd/vx9raGt1W7YKrSqXC4OAgHn30UahUKvD5fOzu7mJzc7Oq\ndlguJjUbbIPBAIVCAR6PB4FAgNnZ2Xr261whEAggFoshkUggFotLtuVyOaRSKSSTSeq9QPYr9mTY\nS6FQQCqVosfm8/mTPg0A1elKXNUSiQSy2WzJeR61vxwOB01NTWhvb8fAwAA4HA5cLhd2dnbolAK5\nDnvd+MRiMb3u3d3d6O/vx+DgIH1Sd7lccDgcEAgE0Gq1UKlUaGpqgtvtxu7uLmw2W8UfBQ6HQ7WS\nSCR0IZNhGMRiMdonEvpOrgFBLpdDLpcjl8vR63JYgisejweRSASpVAqZTAaRSFQStXkc2DF7n4N0\nBe5rW3wfczicA8f2XoqPO6ngrJrvBA6HgzfffBMqlaqe/TmXyOVy6PV6+ikmGo3C4XDQj0qlovu1\ntbUd2GY2m6XH2O32U3MBq0bX5uZmdHV1Qa/XQ6lUwm630/7Wmh5UKpVSbwuRSFTS5u7ubsm+KpWK\nfn9fXx/Gxsag1WrB4/Ho1EImk4HT6URLSwt8Ph/efvttbG9vw2w2l7W3Fz6fD61WS79DJpPRbWQa\niGjE4/Gg1+vpvsXs7u6WaFlpmicUCmFjYwNCoRBNTU1YXl6G2+2uy3QIO2bvUy9d91I8zg+bCqy5\n78c5mF3pvo9MJoPBYMDU1FRZ4nmfz4e5uTlks1k4nU5qSKanp9Hb23tgm6lUCvPz8wDue0qcps/u\nUXVtbm5Gf38/pqenodfrMT8/T13ZajXYMpkMPT09dIpjbm6OZs/ba2BbW1sxNDSEqakpDA0Nob29\nvcRgDw4OQqFQwO12w+VyweVyYX5+Hh6Phz5lV4LH46G9vR3j4+OYnp6GUqmk23K5HO7evUunb4RC\nIQwGA6anp8vuAbfbjfn5eaTT
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 14
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"subplot(131); cshow(simages[1])\n",
|
||
|
|
"r = sign(simages[1]+randn(n))\n",
|
||
|
|
"subplot(132); cshow(r)\n",
|
||
|
|
"for i in range(100): r = sign(dot(W,r))\n",
|
||
|
|
"subplot(133); cshow(r)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAWwAAAB/CAYAAADLqvg4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvVlsm+l1///lvooUtVGkKJISRYkStVi2LI89dsbxxJPk\nIkGCtEmLtgmapA1ykaKYAJlcNWkvmpmLoAjSmwBJf5igQNpcJGiAIpNJMLUzk5nYY1uyJEqUKO6b\nuG/ivv0v/H+ekFooal/MD0BgxhRfPu973ufwec9zzvcwqtVqFS1atGjR4szDPO0BtGjRokWL5mg5\n7BYtWrQ4J7QcdosWLVqcE1oOu0WLFi3OCS2H3aJFixbnhJbDbtGiRYtzwoEd9ltvvQWDwQC9Xo83\n3njjKMfU4hRp2fVi0rLrBaF6AEqlUlWn01Xtdnu1UChUp6amqsvLy/R9AK3XGXodlV1btj1br5Zd\nL+arEQdaYT98+BBDQ0PQarXgcDj4i7/4C/zP//zPQQ7V4gzRsuvJo1ar8bWvfQ2/+MUvkM1m8X//\n93+4efMmpqamjuw7Wna9OLAP8iGv14v+/n76/yqVCg8ePDiyQbU4Hc66XblcLiQSCaRSKSQSCTY3\nN5FMJhGJRFAqlQ50TDabDYlEgkqlArVajUKhgGQyiWQyiUwmQ79LIpGgVCrR9zY3N+veq1QqSCaT\nCIVCyOfzR3zmh+Os27VF8xzIYTMYjKMeR4szwFm3q1AohFqtxuDgIHQ6HdxuN2w2G1Kp1IEdNofD\ngUKhQLFYxJ07d5BIJGCz2WC1WlEoFCCXy6HT6TA4OIhsNkvfy2Qy6OnpweDgIAYHB1EqlWCz2bC5\nuXnmHPZZt2uL5jmQw+7r64Pb7ab/73a7oVKpjmxQLU6Hs25XgUCA/v5+XL58GdeuXcPCwgIqlQqc\nTicymcyBjkkctlwux0c/+lEEg0GwWCxEo1FEIhH09vZifHwc165dQyKRAJvNRiwWg8/nQ09PD4xG\nI2ZnZ5HP58HlcuF2uxGPxw91nmq1GiaT6VDHqOWs27VF8xwohj0zMwOLxQKHw4FCoYD//u//xqc/\n/emjHluLE+as25XP50OpVMJoNOLmzZuYmJhAf38/urq6DnxMFouFzs5OfPSjH8X09DTGxsbQ19eH\ntrY2sFgsdHR0QKvVYmpqCkajEf39/WhrawOTyYRMJoNGo8Hk5CQdy06OkMFggM1mg8vlQiAQgMfj\ngc1mg8FgoFqtolQqoVAoIJvNolAoQKFQoFKpHOZS1XHW7dqieQ60wmaz2fj3f/93fPzjH0e5XMZX\nvvIVjI6OHvXYWpwwz6Ndi8UifD4fFhYWwGazEQ6HYbFYEIvFUCqVsLGxgaWlJXC5XCSTSayuriIa\njaJSqSAQCGB5eRk8Hg/5fB5msxmRSGTbdwiFQsjlcvoKhUIIBAIIBALIZrNwu92Ym5sDg8HA2toa\n7HY7UqnUkZ3j82jXi8qBHDYAfPKTn8QnP/nJoxxLizPA82bXYrEIv98PNpuNeDyOzc1NuN1uxGIx\nlMtl6rATiQRyuRzcbjd12MFgECaTCclkEqVSCW63G+FweNt3kNi70WjE+Pg4zGYzTCYT0uk0UqkU\nDVeEw2EEg0G43e4jddjA82fXi8qBHXaLFhcB4rDj8TjW19dRLpeRyWSQzWZRKpUQCASQTCZhs9lQ\nLpeRzWaRzWapw06lUrDb7ahWq/S9rRCHffnyZdy+fRtisRibm5twOp0IhULweDyIRCJYXV1FPp/f\n9TgtWrQcdotjh8lkQiQS0VelUkE6naavZikWi4jFYvB6vVhdXUUkEgGLxYJSqUQmk6k7ZrMxYAaD\nARaLBQ6HAz6fj2KxiGKxiHw+j2q12nCMmUyGbnZyuVyIxWJ0dXVBJBJhc3OTfrZaraJcLqNQKCCX\ny6FQKKBcLqNSqaBUKiGVSh35irrFxaTlsFscO0wmE3K5HFqtFhqNBoVCAQ6HA06nc18OO5PJwO12\n48mTJ8hkMigWi+ByuZicnIRKpYLT6aTHbdZhkywRMrZUKkWPs59VrlgshkajgVarhVqtpuNwOBzI\nZDJwuVx48uQJstks3QDcz7m3aAG0HHaLE4DFYqGnpwdjY2OYmZlBJpMBj8dDKpWCy+Vq+jhkg47E\nktVqNTQaDfR6ParVKh4/foxsNguPx4NisdjUMYnDHh8fx8zMDEKhEDgcDhKJBHw+X9NjE4lE0Gg0\nuHLlCqanp/H48WNUq1UEg0HE43G4XC76gxOJRBAKhbC5udn08Vu0AFoOu8UJwGKxIJfLMT4+jpde\negnJZJLGfvdDNpuFy+Wim3R37tyBXq/H1NQU2tvbkc/n4Xa7wWKxmj4mcdiTk5O4c+cOXC4XLZ7Z\nD2KxGFqtFlevXsXLL78MAAgGg1hZWYHP54PT6az7caq2Wqm2OAAth30AWCwWurq66IvP5+/6t9ls\nFuFwmL6OMr/2vFAulxEMBrG8vAyBQIBMJoO1tTVEo9F9HUcgENRddxJeWVhYAACYTCYEg0GUy+Wm\nj1ksFmkmCJ/PRz6fR6VSgV6vh0gk2vVzlUqlzq7pdBpOpxOPHz9GpVLBkydP6KqaQJx0Z2cnPQcW\ni/Xc3x8ngUQioddcJpPt+ndb7XrWNn9bDvsAsNlsyOVyGAwGjIyMoL29fde/jcViMJvNWF1dRSwW\ney4nZLlcRiAQoKls+XweDodjx5zlRpBKR4PBAIPBgFKphFwuh4WFBcRiMTgcDgQCgX07bL/fj8XF\nRaRSKUilUvD5fAwPD2N8fLzhOa2srMBsNiOdTtOsDyaTiVAoBIfDAZfLtWOcurOzEyMjIzAYDODx\neDCbzTCbzTRdsMXRI5FIoNPpMDIygoGBgV3/bqtdWw77AkAe8UnFXW9v765/6/P5wGKxEIvFYLFY\nmo6tXiRIkUk6nYbD4UClUsHm5ua+Y7i1pek3b96E2WzG/Pw8FhYWYLVaaWbGQRw2Sd0bHh7G9PQ0\nhoeHYTAYdv1coVCgTwsulwvhcBgOhwPhcBgrKyv0/HYqme/s7ITBYMDNmzchFArBZDIRjUaxtrZ2\nYE2UFo2RSCQYHBzEtWvXcPny5V3/bqtdzxoHdtharRYSiYSmRD18+PAox3Xm4HK54PP5EAgEkMlk\nGBwchF6vx+joKBQKxa6fk0gkCAQC8Pl8cLvdSCQSyGazNL3rrNGMXdVqNQDQcupcLtdQ8KhSqdSl\nrnE4HAgEAohEInR0dCCXy9HjNHJYTCaTflYsFoPL5aJarSKXyyGTySCfz6NUKm2LD7NYLGo7krpH\nvrNUKqFUKiGfz4PBYNB0PqIMyOfz6WdrY+OFQgHBYJDatVKpIJfLIRgM7jnR2Ww2eDwehEIhfVRX\nKBTo7+9HMpmkYzvqH/fnac4ymUxqbz6fj4GBAQwNDWFkZARjY2O7fm43u5L787Q5sMNmMBi4d+8e\nOjo6jnI8ZxapVAqVSgWVSgWNRgOj0QiVStUwfg08WxWqVCpcunQJHA4HLpcLHo8HHo9nx6q406YZ\nu3784x8H8GxTzePxwO12IxgMNv0dYrGY6m4oFAq43W56nEb5yCRL5MmTJygUCkin0zStT6lU1h2n\n1vFzuVwolUr6nYlEgv5tIpFAb28vfa+zsxMsFgsWiwXhcJj+e39/P4RCIT0mk8mkmS/VahVKpZLa\n1ePxNDx/UiTDZrPR2dmJZDIJpVKJO3fuwOv10nPYb4x/L56nOcvhcOrsOjIygqGhoYbxa+Bwdj0J\nDhUSeZ52uqVSKQYHBzE9PY3x8XGqC9GMw+7v76dOY3FxkepSnEWHDextV+Kw19fXMT8/j1QqtW+H\nrdVqqdjS3NwcmEwmIpFIUw67UCjA5/NBoVBAoVBgamoKpVIJ8/PzVBtkq8Pu6+vD5OQkLl++DK/X\nCy6Xi1QqhUwmA6VSiYmJCUxPT6NUKsHv92NtbQ2pVArT09MAgJ6enh0dttFopCvkubk5mlbYiHA4\nDLPZjFgsht7eXiiVSiiVSszM
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 15
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"Significance\n",
|
||
|
|
"============\n",
|
||
|
|
"\n",
|
||
|
|
"Both Willshaw's and Hopfield's associative memories have generated enormous numbers\n",
|
||
|
|
"of citations and follow-on publications.\n",
|
||
|
|
"\n",
|
||
|
|
"From a computational and statistical point, this is hard to understand.\n",
|
||
|
|
"Both approaches solve a problem that amounts to little more than a nearest\n",
|
||
|
|
"neighbor lookup, where \"nearest neighbor\" is defined not necessarily\n",
|
||
|
|
"by Hamming or Euclidean distance, but by some other generally reasonable\n",
|
||
|
|
"but fairly haphazard distance.\n",
|
||
|
|
"Furthermore, even out of McCulloch-Pitts neurons, it is much easier to\n",
|
||
|
|
"create better kinds of associative memories.\n",
|
||
|
|
"\n",
|
||
|
|
"However, both approaches show how very simple Hebbian learning rules\n",
|
||
|
|
"can be used to build potentially useful pattern recognition systems\n",
|
||
|
|
"that learn from data (think about: why is associative memory useful \n",
|
||
|
|
"for pattern recognition?). \n",
|
||
|
|
"This means that they show that neural hardware, even in its most simplified\n",
|
||
|
|
"form, could be capable of learning, association, and memory tasks.\n",
|
||
|
|
"\n",
|
||
|
|
"The extreme simplicity of either algorithm also means that very simple\n",
|
||
|
|
"analog hardware might be used for implementing it. This was\n",
|
||
|
|
"particularly motivating for the original Lernmatrix approach.\n",
|
||
|
|
"But it might also become important again in nanotechnology, where\n",
|
||
|
|
"we may be able to generate very large computational structures\n",
|
||
|
|
"out of molecules, but without the kind of fine-grained control\n",
|
||
|
|
"that we have for regular VLSI chips.\n",
|
||
|
|
"\n",
|
||
|
|
"Hopfield's network also illustrates that neural computation can be viewed\n",
|
||
|
|
"not as the computation of a static function, but as convergence\n",
|
||
|
|
"of a _dynamical system_ towards fixed points or attractors.\n",
|
||
|
|
"Hopfield networks also have mathematical connections to much more powerful\n",
|
||
|
|
"statistical methods as well as models in statistical mechanics, \n",
|
||
|
|
"such as Boltzman machines, Ising models and Bayesian networks.\n",
|
||
|
|
"In addition, they are Turing universal in some sense.\n",
|
||
|
|
"Hopfield networks are also interesting as dynamical systems,\n",
|
||
|
|
"and their behavior can be described using Lyapunov functions.\n",
|
||
|
|
"\n",
|
||
|
|
"Much of the work on these models has focused on exploring these mathematical\n",
|
||
|
|
"connections, as well as determining the \"capacity\" of these memories.\n",
|
||
|
|
"\n",
|
||
|
|
"From a practical point of view, these models are of limited value.\n",
|
||
|
|
"Associative memories in real-world pattern recognition systems are\n",
|
||
|
|
"usually implemented using nearest neighbor methods, PCA, and similar \n",
|
||
|
|
"techniques. If you want to use these models for building \"useful\" \n",
|
||
|
|
"associative memories, you need to be a lot more careful with the\n",
|
||
|
|
"conditions on the input vectors, as well as the details of the\n",
|
||
|
|
"implementation, than in the rough examples above."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"(Significance)\n",
|
||
|
|
"\n",
|
||
|
|
"History and utility:\n",
|
||
|
|
"\n",
|
||
|
|
"- enormous academic interest, analysis\n",
|
||
|
|
"- computationally, little more than nearest neighbor\n",
|
||
|
|
"- little practical value\n",
|
||
|
|
"- little biological plausibility\n",
|
||
|
|
"\n",
|
||
|
|
"Main contributions:\n",
|
||
|
|
"\n",
|
||
|
|
"- shows how Hebbian learning can be used to create nearest neighbor systems out of random networks\n",
|
||
|
|
"- Hopfield demonstrates learning and computation in dynamical systems"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [],
|
||
|
|
"language": "python",
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": []
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"metadata": {}
|
||
|
|
}
|
||
|
|
]
|
||
|
|
}
|