nvidia-AI.summerschool17-no.../teaching-ncso/98-sequence-learning-tasks.ipynb

563 lines
326 KiB
Text
Raw Permalink Normal View History

2017-09-22 17:38:09 +00:00
{
"metadata": {
"name": "sequence-learning-tasks"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Title: Sequence Learning Tasks\n",
"Author: Thomas Breuel\n",
"Institution: UniKL"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"\n",
"import urllib2"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 45
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"default_n = 29\n",
"default_ninput = 2"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 2
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is a number of sequence prediction tasks. For each of them...\n",
"\n",
"- think about how you would solve them at all\n",
"- how you would learn them with a dedicated machine learning algorithm\n",
"- how an AR and MA model would solve them\n",
"- what kind of complexity learning might have"
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Non-Sequence Tasks"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def generate_threshold(n=default_n,ninput=default_ninput,threshold=0.5,example=0):\n",
" \"No temporal dependencies, just threshold of the sum of the inputs.\"\n",
" x = rand(n,ninput)\n",
" y = 1.0*(sum(x,axis=1)>threshold*ninput).reshape(n,1)\n",
" return x,y\n",
"\n",
"x,y = generate_threshold()\n",
"ylim([-.1,1.1]); plot(y,linewidth=3,alpha=0.5); plot(x)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 17,
"text": [
"[<matplotlib.lines.Line2D at 0x4da5590>,\n",
" <matplotlib.lines.Line2D at 0x4da5d50>]"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD5CAYAAADY+KXfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXmYG9WZ7n/qRUvv++JWd9tttzdsbMDGxg7ELAFMwBAg\nY8hKkps4BEJyh4TcTGYCzGTyhGSWZIYJ4SZzycISmEAABzCLjcPqxvvW7cZL273vq7q1dEu6f5wu\nqSSVpCqp1C0bvc/Dg1t1dOqolrfees/3fcfg9Xq9pJBCCimkcE4gbbYHkEIKKaSQgn5IkXoKKaSQ\nwjmEFKmnkEIKKZxDSJF6CimkkMI5hBSpp5BCCimcQ0iRegoppJDCOYSMmdqRwWCYqV2lkEIKKZxT\n0BJ5PqNK3ev1nrP/3X///bM+htTvS/221O879/7TipT9kkIKKaRwDiFF6imkkEIK5xBSpK4TNmzY\nMNtDSCjO5d93Lv82SP2+jxoM3lhMm1h2ZDDE5A+lkEIKKXyUoZU7U0o9hRRSSOEcQorUU0ghhRTO\nIaRIPYUUUkjhHEKK1FNIIYUUziGkSD2FFFJI4RxCitRTSCGFFM4hpEg9hRRSSOEcQorUU0ghhRTO\nIaRIPYUUUkjhHEJUUv/yl79MeXk5y5cvD9vmnnvuob6+nhUrVrB//35dB5hCCimkkIJ6RK2n/qUv\nfYlvfvObfOELX1Dc/vLLL3PixAmOHz9OQ0MDd955J7t27dJ9oDMJrxfa2sBuV9c+LQ1qasBk0ncc\nNht0dKhvn58PFRX6jsHjEcfC4VDXPi0NamvBaNR3HFpgt4sxJ6IqhdEofl/aWfKOOzEh/isp0bdf\ntxt6esT1pvex6OmB4WF9+5RQUgLFxfr2eeiQuPfNZsEDs710RFRSv/TSSzl9+nTY7S+++CJf/OIX\nAVizZg3Dw8P09PRQXl6u2yBnGlu3wr592r6TnQ3f+pZ+ZNbfD7/8pSBVLdi4Edas0WcMAM89B0eO\naPtOfj5885uQMWNLsPgxMQG/+AU4nYnbR309fPazietfL4yNwX/8B0xOwi23QISXbc144gk4dQqW\nLoW/+Rv9+t29G156Sb/+lHDbbbB4sT59TU2JewQgPR3+4R/06TcexH3bdXR0UF1d7fvbarXS3t6u\nSOoPPPCA798bNmxI2upqTU3avzM+LtTh/Pn6jOHDD7UTOoix60XqXm9sx2JkBDo7hWqZabS0JJbQ\nAY4fFzfzbDy0tODkSUHoIM6jXqTudApCl/r1evVTp7Fcb7HsQy9Sl7/Bms369Llz50527twZ8/d1\nuSyDK4iFW7pOTurJCq838EQtXBi5fVeXUESgL5nIx1BSAkVF4ds6nXDmTOj34sXUlHjNBnHT1tdH\nbt/RIR5u0phmA/Lfn58Per4wnjoljom0n5wc/fpOBOTHQs/rQt6X1wsul37Wo7zv2lr9+h0fh97T\nE2S6xnE6S/XplMSQerDgffDBBzV9P25Sr6qqoq2tzfd3e3s7VVVV8XY7a3C5/F5sZiZ85jOR2z//\nPBw4IP6dqBtn1SpYuzZ828FB8ZqdqDG8z79zheUePvOZ9Ijt/+d/4OhR/cehBfL9Ll0K11yjX9//\n8R/iWEv7SZG6/+9EkPoNN+g3F9DSAqe/9DMKRs5w8Lz/p0+nBI5X7zm1WBH3FMemTZv4/e9/D8Cu\nXbsoKCg4q/10rU9eeRs9bxy50o02Dvl2PRWy0wke3LzGvTiM7VHbJ+pYaIGW46YViTrOiYJ8jHpf\nF5H+1qtvPc+f2QxLm57FYh+ctft0phBVqd9+++389a9/pb+/n+rqah588EEmp426LVu2cN111/Hy\nyy+zYMECsrOzeeyxxxI+6ERC60lK1I2uRQHItzsc+nmcDgc4GQGDF1tGK1AbsX0ykF4ilVPwcU52\nzKRS1wPB1qee58/SfpyC3sPYLUUJu0/PGlJ/6qmnonby8MMP6zKYZECyKHUt40hPF1bR5KS+HqfD\nARMMADCW1hq1fTIo9UTeZMnw+7TgbCN1+RxOerq+E9FZr/2ZNutazI7hhB2Lc8Z+Odeg9SQlSr0l\nwxuD0wl2hIk8SluU1smhZFP2ix/yMbrd/klePftV+luPfs1mfeO9M7c+x/6VX8bsGMLp1C+HIRnt\nlxSpByEZyBSS4+HicPhJfcijTamn7JfZR6IU9Uz0q+u5a2/HcPw4J5ffhNkxjMfjD/WMF8lov6RI\nPQhno/2SqHEIUh/A6M1l0J2yX5Lh92nB2Uzqup67P/8Zrr8eT2ExRpcNg8ed/A+iOJAi9SAkg0L2\negOVrppxJNJ+qWAFA5Mp+yUZ3kS0YCZskkT1q+u5e+45uPlmzFlpuIy5mJyjul2fKfvlLEAy2C/B\nsfLpkcPDgUTaLwOUs4JeZ8p+SYaHlloER5LA2aXUdTt3fX2i5sfVV2MygcNcgNkxrNv1mbJfzgIk\ni+2hZQyJHIedQYpZyKTXyZhzbMbHoAXBbzgfZftFHkki4Wwidd3O3Ysvigw0iwWz2U/qSf0gihMp\nUg9CvPaLHrPqsRBTouyXCQawUExlVg1to5EtmNlWsvI3HKNR/+qByfAmohZK4/tI2i/T1ovUp91S\nqCupp+yXswBa1YIUIw7+GHE9x6D26Z/I6BcLRczJqaZ1JLIFYzL5w9BcrtgKksWDRKum2X5oaYHS\n+M4mpa7L+RsZgbffhuuu8/WZSKWeIvUkRTKo5OSyXwbIopjqvBraRiIrdYMh8GacaTWb6BvsbLJf\nznZS1+X8vfQSXHYZ5OX5+nSYC7BMx6rrAYcD0qecrN3185T9kqxIBpWcDA8WqR9JqdcWVNM6Gn2y\ndDbVbKJfhVP2i3I/SWu/yKwXqU89lbo0h5M10c/6d3+aUurJimRQyckwBqkfyVOfWxhdqSdqHGox\n0/ZLIlZW0guJUupud6jFmJT2y8QEvP46bNoU0KfDpB+pS3M4FvsgjqyipFkNK0mGkTxIBpWcDG8L\nXi/YnVO4sGEmn7rimqieOsyumk20/ZKI+ZNEIVGkrnRO9Zo/0fX8vfYaXHRRQO1euVLX8z61OIZw\nZhfG36FOSJG6DB6P/0Y1GNQvTXcu2i8uF9i9w5gpwGRME/aLClI/l+2X4H6T2YJJlP0Srg+9+477\n/D33nFjDTwa97RdpvGb7EJM5KVJPSgQrZLUFhRJlvwzRwhnn/lkbg7BeijCboTq/mvbRdjzeyJIs\nWeyXmSD1ZJ4sTZRSD9fHbNmOinC54C9/gZtuCvhYb1L3KXX7IJO5EZYmm2GkSF2GWJVCouyXwzzJ\nM20/V/UdvRWyPJzRZIKszCxyjDn0jfdF/F6y2C+JikQ4W8Iaz3ZSj+v8vfkmLFoEQSuwmUzgmI5T\n19t+ceellHpSItaLKlH2i40ueh3RJyelMegZIy4iX0Q4o0TUNfnJnYCUsl/8ONvsl1jqHYVFUNSL\nBL9SH9LXfnEM4clPKfWkRKyvf4myX2x006OS1PWOEZcrdTmpR/PVU/ZLciCcUo83YidRSj04G1hN\nvSNFuN1i4eBPfSpkUyLtF29BSqknJZLNfrHRTfd4O16Vd6KeKlkezij1W50ffbI0Zb8kB5TGpkfE\nTqJIXbdz9957UFEBCxaEbDIawWERpK7H26yf1IegMEXqSYlksV/kpO5wO+if6Ff1PT1VpNMJjmCl\nnpeyX85m+yXS52oR7pzq2W9c504h6kWCwQCGnBwyJydI80zFPWbp+xb7IBSl7JekRLLYL04nePFi\no4u6gvmqQgmDx6HHTSYp9ZT9otzv2aLU5Wt96vGwT3S/MZ87rzesn+7rOysNpykfk2NEt7cLs2OI\ntOKUUk9KJIP9IsXKu7CBAc4rWxpVHUvQ236RR7+AsF+iZZWm7JfkgHxs+fnKnydrvzGfu337xJfP\nOy9sEz1rqsvtl/TSFKknJZJBqcutl1wqVBFpIsYRLvpFTaVGvcagFSn7RSA4kkROvnraJInqN+Zz\n9+yzQqVHSDDRc7JUPlGaUZqy
}
],
"prompt_number": 17
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Sequence Generation Tasks"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def generate_mod(n=default_n,ninput=default_ninput,m=3,example=0):\n",
" \"Generate a regular beat every m steps. The input is random.\"\n",
" x = rand(n,ninput)\n",
" y = 1.0*(arange(n,dtype='i')%m==0).reshape(n,1)\n",
" return x,y\n",
"x,y = generate_mod()\n",
"ylim([-.1,1.1]); plot(y,linewidth=3,alpha=0.5); plot(x)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 18,
"text": [
"[<matplotlib.lines.Line2D at 0x4eda150>,\n",
" <matplotlib.lines.Line2D at 0x4eda910>]"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD5CAYAAADY+KXfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd4HNW5/z+ruurVlmRJlq3ibiwXMG5gWkwxEAIJJQEC\nN5QEQhICXCA3F8O9IaEkIUAK3PAjwI0JF0wgBtsJGERcKG6Suy1Zli2r974raXd/f4xm68zszOzs\nanH2+zw8RjuzM++eOfM97/me97yvyeFwOIgggggiiOC0QNR4GxBBBBFEEIFxiJB6BBFEEMFphAip\nRxBBBBGcRoiQegQRRBDBaYQIqUcQQQQRnEaIkHoEEUQQwWmEmFDdyGQyhepWEUQQQQSnFbREnofU\nU3c4HKftf4888si42xD5fZHfFvl9p99/WhGRXyKIIIIITiNESD2CCCKI4DRChNQNwsqVK8fbhKDi\ndP59p/Nvg8jv+1eDyaFHtNFzI5NJlz4UQQQRRPCvDK3cGfHUI4gggghOI0RIPYIIIojgNEKE1COI\nIIIITiNESD2CCCKI4DRChNQjiCCCCE4jREg9gggiiOA0QoTUI4gggghOI0RIPYIIIojgNEKE1COI\nIIIITiP4JfVbb72VnJwc5s6dK3vOPffcQ1lZGfPmzWPPnj2GGhhBBBFEEIF6+CX1W265hU2bNske\n37BhAzU1NVRXV/Piiy/y3e9+V/bczk59RhoBmw0aG2G8MxV0dkJ///jaMDoaHm3R3g6Dg+Nrw8gI\nNDWNf1u0tcHQ0PjaMDwcHm3R2goWy/jaYLVCc/P4t4Ue+C2SsWLFCurq6mSP/+1vf+Pmm28GYPHi\nxXR3d9PS0kJOTo7PuceOQWamfmP1wuGAP/0J6uth3jy46qrQ2wBw9CisXQvR0XDXXePXFn/8o9Bh\nFy2C1atDbwPAgQPw5psQEwP33AOpqaG3wW6HF14QBpclS2DVqtDbAFBZCe+8A3Fx8IMfQFJS6G2w\n2eB3v4PubjjnHDj//NDbALBjB7z/PpjN8MMfCv+GGiMj8Pzz0NcHF14Iy5eH3oZAEHDlo4aGBgoL\nC51/FxQUcOrUKUlS//3v1/D++8L/r1y5MmTZ1QYGBEIHOHRo/Ej98GHhX5sNqqth8eLQ29DdLRA6\nCG0xXqR+6JDw7+go1NTAggWht6G9XfhPtGe8SF1si+FhqK0FBaUzaGhuFvqGaM94kbrYFhYL1NXB\njBmht6GhQSB00Z5Qk3pFRQUVFRW6v29IOTvvDGJypetuuGENF15oxB21wX0qNzwseGhR47BE7G6H\n4dNLu114K/24/+73tVoNtkEDgtoWOmyItIXr/yNtMb42eDu8jz76qKbvB0xt+fn51ItuMHDq1Cny\n8/Mlzw2HhwTj12mD2lmeflqVe+V+39FR4b/xwHi/OFI2jJd+Go5tMV4IBzvCwYZAEDCpX3HFFbz6\n6qsAfPbZZ6Snp0tKLzB+DeRN4uHQWQwdWDo64Kmn4PhxOHlS8dRwaQt3O8JhkHU4hFnceCDc2kKc\nzY4Hwq0txnPWohd+5Zfrr7+eTz75hPb2dgoLC3n00UcZGRkB4I477uDSSy9lw4YNlJaWkpSUxMsv\nvyx7rXB4SONph/t9DSXT//5v+MY3oKcHNmyAO++UPVWqLZKTDbRFJcLBG/LuB1YrxMeH3o5wbYuE\nhNDa4HCEX1uIs9kYQ4Tq0MCvqa+//rrfizz//POqbhYOHnK42GGYDbW18NprcPAgbN4shNdoIPXT\nqi0CsEH8O9RROA5HEAd7DZBqi1CT+uioEEQgZ1OoINUW4+H46EVIlwvDYeQdTzuCMq17+GEh9mvi\nRCF845NPFAOeg90W33jzGxxsO6h4js0mhI3J2RQqhMMANzzsqeX/K7eF1GzBH/qH+7GOGtto4TKz\n14uQknq4dNjxsCMoXsgXX8CWLfCjHwl/Z2ZCeTl8/LHsV4LZFk19Tbx58E02Vm9UPC9cBlk9JGI0\nwoFM4cvbFt98+5vcs+keQ+0Il/6pF/8Snno4vDiG2+BwwAMPwGOPee5WWb0a3nsvdHa44f3q98lM\nyKTiRIXieeHwPMLFjnCwIVzs0GrDobZDfFr/KW8eeJOmvqZxsyPcEFJSH7WMjsuqejiMvIZ7yO+9\nJ+ycGdvN68Rllwlb8mTi84LZFu8dfY//WPEfbDmxBZvdJnteOHiFEB4vb6QtXNDaFr/89Jfcfdbd\nfOuMb/Hrz35tmB3hMLMPBCEl9eT+5rCY1o2HDd73DChGfHQU/v3f4YknfJflZ80SdlYdOCD51WC1\nhWXUwsd1H3PTvJsoSC1gT7N8YjcpAhmPGPFwINRwIFP48rVFc38z6w6t43tnfo8fL/kxL+15ia6h\nLkPsCAcnMBCElNRTehvCwksOBxsCsuPllyE3Fy691PeYySR46zISTLDa4qPjHzEvZx5ZiVmsnLKS\niroK2XO97zleMeLh2C/GK0Y8HNtCyYbnvniOG+beQHZiNkXpRayetprf7/x9yO0IR4SU1FP7xofU\nw2HklbqnLm+ovx/WrIEnnxQIXAqiBCOBYLXF+qPrWT1NSCTjj9Slfne4e4bBQqQtXPD+3Tab9Gy2\nf7ifF3a+wL1n3+v87IGlD/Ds588yNBJ4qstwmNkHgpB76v+qHdYwT/1Xv4JzzxVSLMrhvPOgqkrY\naernnkY8D4fDwXtH3+PyaZcDcE7ROWw9uZVRu7S+ZOisJQCEi+QwzAB1XT9lYOSQ87NQwjtWHsJj\nYJH77KXdL7FyykpKMkucn82eOJvFBYt5uVJ+86MaeEepydkQzviX8NTDYeSVuqfmtmhpgWefhZ/9\nTPk8sxlWroS//93vPY14HpXNlcRHxzMjW0ipNzFpIgWpBVQ2V0qeHw6kbrP5Sj6htmHYNswbx5/n\nOUq55u9PkXPgsfGxY9h3TSMc3lOpz0bto/z6s19z/9L7fc59cNmDPLX9KVlnwigbwh2h9dTHgdSl\nvJBw6bCaB5c1a4Rol6lT/Z+7erWPBBMsL2T90fVcPv1yj+ycShJMOEgOhgyyOmGz23i16lWmPz+d\nzzo2cAMbmDo0mbTO/bK2BRPhQmRq+sVbB99ictpkFhf45q1eUriEyWmTeWP/G7ptMEwmHUeE1lMf\nh4XSL5MXoojDh+Gtt+AnP1F3/qWXwqZNHqJksF7e9UfXO6UXEUqkHg4kMh42OBwO3jn8DvP+MI8X\nd73Iq199lQcLN5DHfNKH7EzsqA+JHd4Ih+ehxg6Hw8FT25+S9NJFPLT8IX6x7Rc+6cCNsuHLgJB7\n6v5GvbX71rLyTyvpHzam5lu4jLwBe4YPPSSEMaotl1RQAJMnw2efKd4v0LZo7GukprOGFZNXeHyu\npKuHw4sT6tnCR8c/YslLS3ik4hGeuPAJttyyhRVFK5y/O8UyRGFnL6NYTvu2kIO/flFRV8HgyCCX\nTbtM9hqrSlYRbYpmQ/UGXTaM5wzOKITeUx9SHkG3nNxCXXcdV7x+hSEr2VIPaTzyiAdEqFu2wO7d\ncPfd2m7qFdoo12EDiRHfUL2BVSWriI2O9fhcSVcPBxIJ1cCyo2EHF712Ebevv50fLP4Be+7Yw2XT\nLnNKVeLvThjqYlqniTbHgdO2LfzBX794avtT/HjJj4kyydOWyWTiweUP8ottv9BlQ7g4gYEg5PV/\nRjt7FY9XNlfyp6/+idzkXK558xqGbYEFMMt1zi/NFNfhgPvvFxZHtRZs9AptlLpfoDHiUtKLCDkJ\nJhxIROp+RseI377+dq564yqumXkNh+46xPVzr/chJIsFoketRNlHGYmOZWjgn2HRFuEmv+xv3c+e\n5j1864xv+b3ONbOuobGvka0ntxpqw5cFISX13tR8opsbZI/b7Db2texjfu58XvnqK8RGxXLDuhsM\nX82GL9HC3Lp1AtvccIP2m551llB88sQJxfvpbYuhkSE+Pv4xl5RdInk8nEld7jcb1S8+rf+Ufxz7\nB0e/f5Q7Ft3hM5MRYbFAgqWL
}
],
"prompt_number": 18
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Generalization:\n",
"\n",
"- periodic function generator\n",
"- pattern generator with triggers"
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Counting Tasks"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def generate_imod(n=default_n,ninput=default_ninput,m=3,p=0.2,example=0):\n",
" \"\"\"Generate an output for every m input pulses.\"\"\"\n",
" if example:\n",
" x = array(arange(n)%4==1,'i')\n",
" else:\n",
" x = array(rand(n)<p,'i')\n",
" y = (add.accumulate(x)%m==1)*x*1.0\n",
" x = array(vstack([x]*ninput).T,'f')\n",
" y = y.reshape(len(y),1)\n",
" return x,y\n",
"x,y = generate_imod()\n",
"ylim([-.1,1.1]); plot(y,linewidth=3,alpha=0.5); plot(x)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 22,
"text": [
"[<matplotlib.lines.Line2D at 0x56b9b10>,\n",
" <matplotlib.lines.Line2D at 0x56be310>]"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD5CAYAAADY+KXfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvWuMJNlZ9/mLvMXJe1Vm1qWrqoexPbM7bQwGNGAQC+qX\nF3bc1jJIXORBr7TIeM3IYhYhYQmJL8wgLcLsl1fLfBlkLuI2Gj6gHZBMs2svDS8ge7j4gtZeNGCG\nqaruumZW5fWczIyM/RAV1XXJy4nIyEtVxU8aqbs6KismMs9Tz3nO8/z/hm3bNiEhISEh14LIrG8g\nJCQkJCQ4wqAeEhISco0Ig3pISEjINSIM6iEhISHXiDCoh4SEhFwjwqAeEhISco2ITesHGYYxrR8V\nEhIScq3w0nk+1Uzdtu1r+98v//Ivz/wewv+/8P8t/P+7fv95JSy/hISEhFwjwqAeEhISco0Ig3pA\n3L17d9a3MFGu8//fdf5/g/D/76Zh2H6KNn5+kGH4qg+FhISE3GS8xs4wUw8JCQm5RoRBPSQkJOQa\nEQb1kJCQkGtEGNRDQkJCrhFhUA8JCQm5RoRBPSQkJOQaEQb1kJCQkGtEGNRDQkJCrhFhUA8JCQm5\nRowM6j/90z/NysoK3/It3zLwmp/7uZ/j6aef5oMf/CBf+tKXAr3BkJCQkBB9Rgb1j33sY9y/f3/g\nv3/2s5/lX//1X3n77bf5zd/8TT75yU+OdUONBhwejvUSITPCtmFnB9rtWd/J7LFtePQIOp1Z30nI\nTWOkScb3fd/38c477wz89z/90z/lp37qpwD40Ic+xNHREbu7u6ysrHi+maMj+D/+94fYcoePfuI7\neOYZzy8RGN/4wn3e+90fnt0NzBH//H/+Jq2dzZHX/cd/wDvvgGnCd30XRDSKe3d+/JNkS2vj3+QZ\nZP2Ig3//f9n4lu8N9HW98PnPw9+9+Rfk3/dD/K8/F9F6FrOmZ3X5j3/4PO/50HOBv/Y3vnCf93zX\n/4hxFR7EFWds56Pt7W1u3759+veNjQ22trb6BvWXX3759M937969pK729ttQ/Mqn2Nj+e77+P7w9\n06Ce/88fYedL/8Dqf/cds7uJOeGpn3iRf/7AMowwr1pqQbHn/Nn+MiP3gWvvHPLlnS2+7+XfCeQ+\nXb70mf+N6O/9Hhv/tBvo63rh61+Hn3z9ed74iT9kd/fHuXVrZreizf/3f/0Rsf/lRdhuBf7a5r3/\niXf/77/gm579z4G/9nXjwYMHPHjwwPf3B2Jnd1FBbJB13dmg3g8pwWwdkm7VOVBB3Jk/elaXhZbN\nv+9swg0P6rJ+RMSG7/rS6AD5X/+rs9sC+C//BZ5+evj1D370O6BcDuAuz9M93CNdCz4weUFKyMsO\n0dY2aoafZS+09h5yqzmZelGuafHu7tZEXvu6cTHhfeWVVzx9/9h7ofX1dTY3H2/Nt7a2WF9f9/Va\nUoJQR2RkCynHvTP/1Ms7RG1oHjyc3U3MCdXdd6km9fxlz75nWu9fPg9HFX83NgSrUibZmF1h37bd\noG4Tk7sz/Sx7oX24R7ZpBf663bYk2wZ58Cjw1w65zNhB/fnnn+f3fu/3APjCF77AwsKCr3o6gFKQ\nlDXSqj3boL637dzP4d7sbmJOqO8/pJ4cvaGzbc5lpDrvn5FfwKhWx7i7ARwfkW51g39dTbpdsNQR\npgUJdXBlgnq3cki2DVYn2F+ItX1nPbUr+4G+bkh/Rq7Wn/zJn+Sv/uqvODg44Pbt27zyyit0To70\nX3zxRT7ykY/w2c9+lqeeeop0Os3v/I7/+qiUkFINsrIz0y1r4yRDb5cPZncTc0Lr4BGkEyOva7ed\nwO6i8/5FC0X46j+PcXf9MapVMq3gM05dlAKkU2qIy/KVKb9YJ7um2v42C2vvCex16/vbLALdctjW\nNg1GBvXXX3995Iu8+uqrgdyMlLAkm+SkNdPspnWyTbQqYVCXBzuQNkdfJ4f/vR+xxSJGreHzzgYT\nPa6TU07GGY2P/oUUNFJC5CSoC1m5Mpm6Wwqr7W0GGtQbJztfqxL8+UnIZeaqv0gpSEtJTtnIVo9Z\nud/Jk7JLbwL13qtGu7xPO5MceZ2foG4WlknUg4948XoTeLztnzZSQlQ5iYFQ1SsT1N1SWDPg2rcs\nnxyyHx8F+roh/ZmroC4lZFSHmA12e29mgxsdt/Z3fDybG5gjupVDurn0yOsulhh0Sg5iaRXRCL42\nYdadzpf6wWyCulIQlU4gS6ralSm/RKt1AFqHO4G+rns2NZHzk5BLzF1Qz8kOPcCQmzPLcLrlQ3oG\nRI7DD6FVKWPlsiOv85Opp4q3SE+ghS7ZUPSMx9v+aSMlxOUePSAl61cmU4/XmvQMUAfB9vd3yvv0\nDIge1wJ93ZD+zFVQVwpyssfDXJSIfDizxWAfVdjJR4lOoN575Tg+glxu5GV+gnpmaX0iB5qpZped\nfDTwjFMXKSGhDnmYi5KacXuuF8x6i518lE7AZ0nWUZmdfPS0LBYyWeYmqPd60JYS0YW9bJKo2pnd\ntvX4mINSKvwQ4myZjcXFkdf5Kb/klm+TkzZ2r+fz7vqTaVkclFKo8mxa6JSChKzwKJ8io9SVKb+I\nZpuDUirwA027UuGglJrI+UnIZeYmqEsJyE2qpkEtKYjJnZllOJHjKrVbBUT4ISR6XCe6UBh5nZ9M\nPS5SqCg0joKbB7B7PbLKprZaoDOjOQNniK7C3kKBrJztzIUX0o0OtVsF7KAbBKrOekpO4Pwk5DJz\nE9SVgojapiaiNM0kcTm7oY1orUFnY22mU4nzQrzeJL5YGnmdn6AOUEtGqO0FNz5eL+8gY9BdzM2s\nhU5KELJKOb9GdsbtuV7ItCw6G2sQ8FlStFqjs7FGakISBCHnmZug7vT2blMzYzRFhsQMg3qi3iLy\nTU+SCT+EmPUWZmn0hPDFEkO77ZTURlFPxQI90KztblITEexcLviMUxOlnAPSav5J8rJ3Jcov7g4n\n8sQ3EQ24SyVWbRD5picnIkEQcpm5CupRuUtdmLREFlNVZrYYRL1F8n13yLSCrfVeRZINhSiMDur9\nfgHrvH+tVCLQA83GwUMa6TgsLAaeceriTEY3aaVvYwAdWaU7O9UCLerlHVoxiC2tEgu4QSBRb5F8\n739PVgV/fhJymbkJ6kpBXO3TEEmUmcdUxzPL1JPNDvmnvplUxxEjusmkml1SS6N1Y/u9Vzrvn8yI\n02GvIGgd7tBKxYksLBKpzaaFTkpIyxYdscKxaWCo2bXn6lLf36aejJAolEjUg1W4FA1F6tYTgZ+f\nhPRnboK6lJCQ+zTNFMpcJClnN4mXbnTILK1TN2c3lTgvZFsW2eXbI6/zG9Q7mSTtAIO6PNhFpgWx\nxSLxk2GaaSMlZKTCEivURIyI3Jr/oL63RSMVRxRXEPVgt8ipRptk6ZZzfrI72mwlZDzmKqjHZZmm\nyKBEkaRszKz8kpU9siu3qSWj1G9wULd7PTLKJrs0Wkq533ul8/51cxmso+CEnjqVfdrZFGZxOfCM\nUxelIKu69MQ6NREnKmfYnqtJ63CHZjpBammNZDPYBoFMq0t2ecM5P9kP5awnTSAmGUGglKOlLkWO\nriiRVk2OZpDddNtOr3y0sMrDVBxusAZ0vbwDMcgmxMhr/WbqvVw2UE11q1KGbBqzuAIzaqGTErLS\nomeu0zBNYmr+NdXV4R6kBSulNYyAB8Ky0oblDXZSCZjRQNhNYm6CutMGdkwtu0bHXCYt5UwWQnV3\nk4hpsBCJINMJ7Bsc1Gu7mxjJCKNFAvwHdccoIzihp16lDLksqdIqzKAl1THI6JFXNra5TkMkr4RR\nRru8B5kk2eUNIgE2CMj6EYYNZmYBlTaxD8KgPmnmqvySVDWUKGCJNTKyPZMta/1gm1oy6txTJnmj\njTIaBw9ppOIjr7Ms+oqv6bx/Rn4RoxrggebxMSwskFnemImmersNtnI6SYxoiqZIE5eHc19+cTR+\nMiSzBSI2qEYwnUO1va1T56x2
}
],
"prompt_number": 22
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def generate_smod(n=default_n,ninput=default_ninput,m=3,r=0.5,example=0):\n",
" \"\"\"Generate an output for every m input pulses. The input\n",
" is band limited, so it's a little easier than generate_imod.\"\"\"\n",
" x = rand(n)\n",
" x = filters.gaussian_filter(x,r)\n",
" x = (x>roll(x,-1))*(x>roll(x,1))\n",
" y = (add.accumulate(x)%m==1)*x*1.0\n",
" x = array(vstack([x]*ninput).T,'f')\n",
" y = y.reshape(len(y),1)\n",
" return x,y\n",
"x,y = generate_imod()\n",
"ylim([-.1,1.1]); plot(y,linewidth=3,alpha=0.5); plot(x)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 25,
"text": [
"[<matplotlib.lines.Line2D at 0x5b31c90>,\n",
" <matplotlib.lines.Line2D at 0x5b35490>]"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD5CAYAAADY+KXfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztvXmMXNd97/mp9Z7au7uqN3a3Q1ukhrQlkXpDRzbyHNBJ\nFJnOhHZiZ8A8xBBsxiE0EQwDdmAkeJ5QGYxhGcgMbDODUeIldhZB+SOJFEBmYmfcseM8ixlbtjJP\negYdm8/dzWav1V3rObXd+eP2bXazu2u9xb51eT6AAFbVrdOnVOf3rXPPb/OZpmmi0Wg0Gk/gP+wJ\naDQajcY5tKhrNBqNh9CirtFoNB5Ci7pGo9F4CC3qGo1G4yG0qGs0Go2HCN6tP+Tz+e7Wn9JoNBpP\n0Unk+V3dqZum6dn/fv/3f//Q56A/n/5s+vN5779O0ccvGo1G4yG0qGs0Go2H0KLuEGfPnj3sKfQV\nL38+L3820J/vXsNndnNo080f8vm6Oh/SaDSae5lOtVPv1DUajcZDaFHXaDQaD6FFXaPRaDyEFnWN\nRqPxEFrUNRqNxkNoUddoNBoPoUVdo9FoPIQWdY1Go/EQWtQ1Go3GQ7QU9Q984AOMj4/z4IMPHnjN\nhz70IY4fP86pU6d4+eWXHZ2gRqPRaNqnpai///3v5+rVqwe+/uKLL/LDH/6Q69ev88d//Mc88cQT\njk7Q7ZgmLC2BUoc9E41G04xiEdbWDnsW/aelqL/tbW9jeHj4wNdfeOEFHn/8cQAeeeQRNjY2WFpa\ncm6GLue//Bf4v//3/4dP/x8lKpXDno1Gs5tidpmb//Wlvoz9o28fvNlzGxsb8IefuMmVT3yX//bf\nDns2/aXnzkcLCwvMzMxsP56enmZ+fp7x8fE9116+fHn732fPnvVEdbXXXoN3/u27+cezH2du7ne4\n777DnpFGc5uX/6//Ff+LL3LkWz9xdNzC+i2m/uM5zEodn9/9rrnr1yH9/Y8yvfCvvPYfr3PixGHP\n6GBmZ2eZnZ3t+v2OtLO7s4LYQa3rdoq6V5ASErJCSN7SRzAa11FfXyGSKzo+bn5pjsk6FDaWiY9M\nOD6+00gJRnmNWLnAqsvt9M4N71NPPdXR+3v+iZ2ammJubm778fz8PFNTU70OOzBICUlZJSxXkfKw\nZ6PR7MbMZhEF51WssDwPWOI+CEgJQm0Ql2XP22nPon7+/Hm+/OUvA/Dtb3+boaGhfY9evIpSkJQN\nwjLr+cWiGUByOSIl55095bVbABRXbjo+dj9QCiIyT0xVPG+nLY9ffv3Xf51/+qd/YnV1lZmZGZ56\n6imq1SoAly5d4p3vfCcvvvgix44dIxaL8cUvfrHvk3YLjQZUpCRetXYB+vhF4zYCuTzxct3xcdXa\nMnBb3N2OlBBVRRKy6nk7bSnqzz77bMtBrly54shkBg0pAWndfgq56fkdgGbwCOaKJMsNx8etrFui\nLlcHR9RHZYmkrHveTt3vtnYxSoFfLQAQUQXPLxbN4BEulBE1UMWco+PWs+sAVNZXHR23XygFMSlJ\nKhNZbuDlzppa1HtASvDLRQCisuT52zrN4CGK1qLMrTjr0GxsZAGobwxGNo+UEFdVgiaYlWW2TpA9\niRb1HpASAnKRBtwTXnXN4BEtVmj4oLA07+zAGxs0fNDY2rG7HTtKrQH45JynbVWLeg8oBSG1ws1k\n4J7wqmsGj3i5xq1UgJLDZ9/+XJ5bqQBsbjo6br+wo9RuJgP45U1P26oW9R6QEkJylcVU9J7wqmsG\nj4Q0Wc1EUevOlu4I5AqsZqL4cnlHx+0HdpSaqMFyIkJAeTtRUIt6D0gJYbnG8tDIPeFV1wwWsrAB\nQHEkgVpzVtTDhTL5yRGCuYKj4/YDO0otZ/jIRwRBecvTtqpFvQeUsuLT11OTJJVJ2eNedc1gkV+e\nJy981BIxauvOOjSNoqQ6fYRQoeTouP3AjlLLiwAlI0LI49nfWtR7QEorPr0cyaCCYKpbulKjxjXk\nl+cpRILUk/HtEESniBYr+H/qKKLgfnW0o9TyRpCSiHu+pIcW9R6QEiIqjxIjbAo/frXg6bM6zWBR\nXl2kFAtDKgWbG46OHStVidx3kkjR/bsYO0qtIAzKIoGhsp62Uy3qPWDVkyhRMdLkjaDnveqawUKu\nLSFjBr7UEL6cs8lH8XKD4ROniZVrjo7bD+wotaKIoIwUhvJ29rcW9R6QEuKyRE2MUhBhAlrUNS6i\nsrZMJR7BPzRCYNO5KJVaRRKpweixh0iUnK8r4zRWQMMKJSOKMoaJyJyn7dSReur3KlJCTClqYoKC\nEATVsqdv6zSDRS27BskYoXQGn4MOzfzKAhg+UqNTmFWoVysEQmHHxncaK/R4nZKIo0SaiCwiPWyn\neqfeA0pBXFapG5P3hFddM1jUN7LUkwmMkTHCDjo0CysLFCIB/IEgBWNL5F2MHaUmRZKayBBTJU/b\nqRb1HrBSj2uY4sg94VXXDBgbWUgmEelxIkXntqbF5QVKUesmPx8JkF92d6MMO0pNGUNUjTFiUnra\nTrWo94CUkFImpnjdPeFV1wwWvlwO3/Aw0dEjREvOVbCS60uUYwYAxWiI0uqiY2P3g51RanVxhLis\neNpO9Zl6l9RqUFdbYWLBoXvCq64ZLAK5AgyNEB+dwudgowy1ugTxCAAyFsZ0eaMMpSC1FaXWEJMk\nVM3Tdqp36l0iJfjkT9g0rCbb94JXXTNYhPIlQsMZkmMzJKWJ2XCmWUY1u0o1HgVAxiOWyLuYnVFq\nDWOKlMdLemhR7xJL1G+SE9bNju1V9/JtnWawMApljMw4IRFFBaC4sezIuPWNdeqpOADVRJRq1t2N\nMnZGqfmMI0SrVoEvh37jXIcW9S6xstRuUhAhAMurLr3tVdcMFpGiQoxYTeDzET/5ZWdqqpvZLGYy\nCUA9EaeedXejjJ1RaviC5MOAh7O/tah3iVIQVEtWfHoQamKCmPK2V10zWERLNWJjUwAUokGKyw6F\nHuZyMDQMgJlKYm44W4LAaewotYaYJhiEnAjgkwuetVUt6l0iJQTlCkUjQioFNWPC8151zWARL9eJ\nj1qiXo6GKTvk0Azk8gS2RN2qK+NsCQKnsaPUENOkUpAXIQJq0bO2qkW9S6zU41XKIkoqBQ0xSVJW\nPfvrrxkszEaDhDJJbIm6ihlIh7ofBXNFgsNpAKsEgcN1ZZxkZ5RawBgiFoOCESYoFz1rqzqksUuU\nAkNlKYskQ0NgihmSquHZhaIZLArrtyAIibAAoJKIYq4749AMF8qQHgMgOJLBly86Mm4/2I5SEz6E\nACGgICIE5LJnbVXv1LtESjDUBtJIkkyCz5jwvFddMzhYqfy3zbuWjFPfcMahKYoKkbYcsOGRjCXy\nLmU7Ss0Ibot6WUQJq1V9/KLZjZQQkZtIYwghwBDe96prBofC8jzFaGj7cSMRp+FQo4xosUIkMwmA\nSI8jCu5d8Duj1AzDEvWSEScs1/ROXbMbq5Z6kYpIW6JueN+rrhkcymu3rAYZNkNDsLnpyNjxco3E\n2DQA0dEjRErubZSxM0rNtlMpkhhyw7N2qkW9S6SEqCpRFZnt2zqve9U1g4NaW0bFxPZjX2oYX86Z\nmurJsrkt6rHMEeIOliBwmp1RaradSiOFUDnP2qkW9S6REmKyTNUY3b6t87pXXTM4VNatBhk2wZE0\nwVyh53FlwYokEfEhAJLjryNZdq8TaWeUmm2nFTHi6ZIeLUX96tWrnDhxguPHj/P000/veX11dZV3\nvOMdnD59mgceeIA//dM/7cc8XYeVpaaoi8nt2zqve9U1g0M9u049Gd9+HBrJEHKgUUZ+eZ5cxLf9\nWMSH8Ju3xd5t7IxSs+20IjJEVdGzdtpU1Ov1Ok8++SRXr17l1Vdf5dlnn+W1117bdc2VK1d4+OGH\n+d73vsfs7Cwf+chHqNXc37ewV6wstSqNLVHf9qrLFc8uFs3g0Miu00gmth8bI2MIBxpl5JfnKURu\nR0L7/H5ywudYCQKn2RmlZttp
}
],
"prompt_number": 25
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(sum/product problem)\n",
"\n",
"Input consists of two sequences:\n",
"\n",
"- value sequence (values in $[-1,1]$)\n",
"- marker sequence\n",
"\n",
"Output:\n",
"\n",
"- sum of the values marked by the marker sequence\n",
"- product of the values marked by the marker sequence"
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Language Recognition Tasks"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def generate_anbn(ninput=default_ninput,n=default_n,k=default_n//3,example=0):\n",
" \"\"\"A simple detector for a^nb^n. Note that this does not\n",
" train the network to distinguish this langugage from other languages.\"\"\"\n",
" inputs = zeros(n)\n",
" outputs = zeros(n)\n",
" if example:\n",
" l = n//3\n",
" else:\n",
" l = 1+int((k-1)*rand())\n",
" inputs[:l] = 1\n",
" outputs[2*l] = 1\n",
" outputs = outputs.reshape(len(outputs),1)\n",
" return vstack([inputs]*ninput).T,outputs"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 34
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"x,y = generate_anbn()\n",
"ylim([-.1,1.1]); plot(y,linewidth=3,alpha=0.5); plot(x)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 35,
"text": [
"[<matplotlib.lines.Line2D at 0x6d2e590>,\n",
" <matplotlib.lines.Line2D at 0x6d2ed50>]"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD5CAYAAADY+KXfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHfZJREFUeJzt3XtMXNe9L/DvHmZ4zDCAxzxiZriiMVjg2AbuxSU5VapJ\nW9ePKiStU4n2NvJxaMKxakWRcnuiVr0q9EhRiFRdpaFH13k2aVqL/pEWVxfT1lWmdtNrUyV2fM61\nFeEmqAOxsTFghuExMKz7x5QJ2MMw77X3mu9HsjIDe2Z+Wzv5Zvm316ylCSEEiIhICSbZBRARUeow\n1ImIFMJQJyJSCEOdiEghDHUiIoUw1ImIFGLO1AdpmpapjyIiUko8M88zOlIXQij754c//KH0Gnh+\nPDeen3p/4sX2CxGRQhjqREQKYainiNvtll1CWql8fiqfG8DzyzaaSKRpk8gHaVpC/SEiomwWb3Zy\npE5EpBCGOhGRQhjqREQKYagTESmEoU5EpBCGOhGRQhjqREQKYagTESmEoU5EpJANQ/2xxx5DRUUF\ndu7cue4xTz75JGpra9HQ0IDz58+ntEAiIordhqF++PBhDAwMrPv7/v5+XLlyBUNDQ3jppZdw5MiR\nlBZIci0vA598AgSDsisholhsuEnG/fffj+Hh4XV/f+LECRw6dAgA0NLSgqmpKYyNjaGiouKOYwf/\n9/9MvNIMM9vsaPrv/wOaKbs7VG+/DfznfwLV1cA//7PsaohoI0nvfDQ6Ooqqqqrwc5fLhZGRkYih\n/nLX/wo//q+FFvw3e26yH5829f9xHdd2fwFb6ppllyKNEMDly6HHw8PA3BxQUCC1JCLleTweeDye\nhF+fku3sbl9BbL2t616+OpOKj8uIv92VB3F9FMjiUF9aWtt2mZ9nqBOlm9vtXrOccFdXV1yvTzrU\nnU4nvF5v+PnIyAicTmeybyvdrC0XYvwT2WVINT8f/TkR6U/SDePW1la8+eabAICzZ8+ipKQkYuvF\naBZseVi4eV12GVItLER/TkT6s+FI/Rvf+Ab+9Kc/YXx8HFVVVejq6sLi4iIAoKOjAwcOHEB/fz9q\nampgs9nw+uuvp73oTAjYrRAT47LLkIojdSLj2TDUjx8/vuGb9PT0pKQYPVkqKgQmGerRnhOR/qTk\nRqmKlu2FwNSk7DKkYvuFyHiyexJ2NCUlwK1bsquQiiN1IuNhqK9DK94E061p2WVIxVAnMh62X9Zh\n3uQAfH7ZZUjF9guR8TDU12HZXAbMzMouQyqO1ImMh6G+jjxHObSZ7E4xhjqR8TDU11FQehc0f0B2\nGVKx/UJkPAz1ddjKnDDNLsouQyqO1ImMh6G+jsIyJ0xzy7LLkIqhTmQ8nNK4DntpJayLwFIge5OM\n7Rci42Gor8OUY8ZMHuC7MSq7FGluH5kvLYX+EJF+MdSj8BXkYCZLQ12IyCNztmCI9I2hHoXfasHs\n+FXZZUgRCISC/XZswRDpG2+URjFvy4XI0lBfb0TOkTqRvjHUo1iw5UNk6UYZDHUiY2KoRxEosmXt\nmurrtVnYfiHSN4Z6FEF7ITB5U3YZUnCkTmRMDPUoRJEdmJqSXYYUDHUiY+Lsl2iyeKMMtl+IjImh\nHoWpxIGcaZ/sMqRYPSI3myP/nIj0h6EehdlRCnOWbpSxOryLiyP/nIj0h6EeRa6jFLm+OdllSLG6\nzbI61Nl+IdI33iiNIn9zBTR/dqbY6hF5SUnknxOR/jDUo7CWVQKz2blRBtsvRMbEUI/CVloJ02x2\nLkvIkTqRMTHUo7CXu2Caj7CqVRZgT53ImHijNIoCuwMmAczPZN8XkKK1XyKt3khE+sBQj0IzmTCd\nr8F3fUR2KRm3OtStVsBiCT0WIrQsLxHp04ahPjAwgLq6OtTW1qK7u/uO34+Pj2Pfvn1obGzEjh07\n8LOf/SwddUrjz8KNMoJBYPEfe25rGpCbC+Tnf/p7tmCI9CtqqAeDQRw9ehQDAwO4dOkSjh8/jsuX\nL685pqenB01NTbhw4QI8Hg+efvppLCm055nflovZG5/ILiOjVod2Xl4o2FeHOm+WEulX1FAfHBxE\nTU0NqqurYbFY0NbWhr6+vjXHbNmyBdPT0wCA6elpbN68GWazOvdf5215mL85JruMjFod2ithnpcX\n+fdEpC9R03d0dBRVVVXh5y6XC+fOnVtzzOOPP44vfOELqKyshM/nw69+9at136+zszP82O12w+12\nJ1Z1BgVs+RCTN2SXkVG3j9QBtl+IMsXj8cDj8ST8+qihrmnahm/w7LPPorGxER6PB3/729+wZ88e\nfPDBB7Db7XccuzrUjWKxqBC4mV0bZUQaqbP9QpQZtw94u7q64np91PaL0+mE1+sNP/d6vXC5XGuO\n+ctf/oKvf/3rAICtW7fiM5/5DD788MO4itCzYFEhglOTssvIKLZfiIwraqg3NzdjaGgIw8PDCAQC\n6O3tRWtr65pj6urqcOrUKQDA2NgYPvzwQ9x9993pqzjTiouBW9k1T311eyXSSJ3tFyL9itp+MZvN\n6Onpwd69exEMBtHe3o76+nocO3YMANDR0YHvf//7OHz4MBoaGrC8vIznn38eDocjI8VnglZcAnz8\nkewyMmr1SDxST50jdSL92nCayv79+7F///41P+vo6Ag/Li0txW9/+9vUV6YTphIHtOmLssvIKLZf\niIxLnbmHaWLZXAptZlZ2GRnF9guRcTHUN5DnKIeWZRtlsP1CZFwM9Q3kb66AlmVrqrP9QmRcDPUN\nWMsqoc0uyi4jo9h+ITIuhvoGCsucMM0GZZeRUWy/EBkXQ30DReVV0BYExPIyNFN2rFTM9guRcWVH\nSiXBkm/FQg4wM3FNdikZE6n9srJaIxBaT315OfN1EdHGGOox8BWYsmZNdSEit180be1onX11In1i\nqMdgxmqGP0vWVF9aCm2SAQA5OcDqVZTZgiHSP4Z6DOasuZgdz45Qv731snqhTs6AIdI/3iiNwYIt\nD+LmddllZESk1ssKzoAh0j+GegwCdivERHasqR5p5ssKtl+I9I+hHoOlokJgkqHOkTqR/jHUY7Bs\nLwSyZKOMSFvZrWBPnUj/eKM0FiUlwK1bsqvICLZfiIyNoR4DrXgTTLemZZeREWy/EBkb2y8xMG9y\nAD6/7DIygu0XImNjqMfAsrkMmi87NsrgSJ3I2BjqMchzlEPzZ0eKsadOZGwM9RgUlFVC82fHRhls\nvxAZG0M9BrbSLTBlyUYZbL8QGRtDPQaFZU6Y5rJjrVm2X4iMjVMaY2AvrYR1EVgKqJ9kkdZSj/Sc\n7RcifWKox8CUY8ZMHuDLgjXVoy3oZTaHluMFQkv0Li1lri4iig1DPUa+ghzlN8oQIvqN0ts3ymAL\nhkh/GOox8lstmB2/KruMtAoEQsEOALm5n47KV2MLhkjfeKM0RvO2XAjFQz1a62UFZ8AQ6RtDPUYL\ntnzlN8qINvNlBdsvRPq2YftlYGAAdXV1qK2tRXd3d8RjPB4PmpqasGPHDrjd7lTXqAuBIhsWJ27I\nLiOtos18ifRztl+I9CfqSD0YDOLo0aM4deoUnE4ndu/ejdbWVtTX14ePmZqawne+8x387ne/g8vl\nwvi4mptJBO2FwNSE7DLSiu0XIuOLOlIfHBxETU0NqqurYbFY0NbWhr6+vjXH/PKXv8TBgwfhcrkA\nAKWlpemrViJRZIeYmpJdRlqx/UJkfFFDfXR0FFVVVeHnLpcLo6Nrp/UNDQ1hYmICDzzwAJqbm/Hz\nn/88PZXKlgUbZbD9QmR8UdsvmqZt+AaLi4t4//338cc//hGzs7O47777cO+996K2tvaOYzs7O8OP\n3W63ofrvphIHtKErsstIK7ZfiOTzeDzweDwJvz5qqDudTni93vBzr9cbbrOsqKqqQmlpKQoKClBQ\nUIDPf/7z+OCDDzYMdaMxO0qhKb5RBtsvRPLdPuDt6uqK6/VR2y/Nzc0YGhrC8PAwAoEAent70dra\nuuaYhx56CH/+858RDAYxOzuL
}
],
"prompt_number": 35
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"General symbol coding (input or output)\n",
"\n",
"- unary coding: if there are $n$ symbols, the vector is $n+1$ dimensional\n",
"- symbol $i$ is represented as a vector that is all $0$, except for entry $1$\n",
"- this is the same way we represent classes in other neural network classification problems"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Other language recognition tasks:\n",
"\n",
"- Reber grammar\n",
"- embedded Reber grammar\n",
"- $a^nb^n$ - context free language\n",
"- $a^mb^nB^nA^m$ - embedded counters, context free\n",
"- $a^nb^nc^n$ - context sensitive language\n",
"- long time lag probem $x,a_1,...,a_n,x$ vs $y,a_1,...,a_n,y$\n",
"- simple 2 sequence problem: $a_1,a_2,...,a_N$ signal followed by $t$ noise elements\n",
"- temporal order ` .*A.*B.*` vs `.*B.*A.*`\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"imshow(imread(\"reber.png\"),cmap=cm.gray)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 49,
"text": [
"<matplotlib.image.AxesImage at 0x78453d0>"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAADzCAYAAACbv53AAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xtczuf/B/DXHUVHp1QUoqNDKiyntnIoY1RkFZPQDmZj\nNiPsa+hLcph9HWdzGrMtOXWaki035ZRZfqJREepWIaKiovv6/dG3z9e9Tnf1ue9P9937+Xjcj0f3\n53Bd7z7dvbu6PtfnukSMMQZCCCEqS0PoAAghhDQNJXJCCFFxlMgJIUTFUSInhBAVR4mcEEJUHCVy\nQghRcQpJ5HFxcbC1tYWVlRXWrl2riCoIIYT8l4jvceQVFRWwsbHB77//DlNTU7zxxhv49ddf0bt3\nbz6rIYQQ8l+8t8iTk5NhaWkJc3NzaGpqws/PD5GRkXxXQwgh5L9a812gRCJBt27duPdmZma4ePGi\nzDEikYjvagkhpEWoqROF90Qub5Jevnw597WrqytcXV35DkWlrFixAitWrBA6jGaDrkd1dE1ktYTr\nIRaLIRaLufcrV66s8TjeE7mpqSmys7O599nZ2TAzM6t2nLr/AAghpKn+2citLZHz3kc+aNAgZGRk\n4M6dOygvL8fBgwfh4eHBdzWEEEL+i/cWeevWrbF161aMGTMGFRUVCAwMpBErcmjpXUv/RNejOrom\nsuh6/A/vww/lqlQkqrHDnhBCSO1qy530ZCchhKg4SuSEEKLiKJETQoiKo0ROCCEqjhI5IYSoOErk\nhBCi4iiRkwZ58uQJRCIR2rdvD2NjY+jo6OD27dsQiUR4/vy50OER0iJRIidyu3btGuzs7GBubo5L\nly4hPz8fERERsLCwEDo0Qlo0SuREbunp6ZBIJDA1NYWVlRUAwN3dHU+ePBE4MkJaNkrkRG6Wlpbo\n2rUrzp49i8DAQOzduxcAoK2tjc8//xytW/M+4wMhRA70iD5pkOHDh+PcuXMAKufVMTExkZntkhCi\nOPSIPuHF2bNncePGDZiamuLVq1fIycmBSCSCj48PKioqhA6PkBaJWuSkUTIzM5GamopJkyZx20pK\nSqCjoyNgVISot9pyJyVy0iDDhw/H2bNnZbZVrQpFiZwQxaKuFcKbxYsXc19v374dADB16lS0adNG\nqJAIadGoRU4axM/PDykpKTLbdHV18ddff1U79t69eygtLcXHH3+MpKQkmJubc/syMjK4z8DNmzfR\nvXt3tG3bVqGxE6LqqGuFKM1XX32FgoICHDt2DAYGBvjyyy/Rvn17+Pr6csfs2rWLuzk6e/ZsTJw4\nEc7Ozvjiiy+ECpuQZo8SOVGKjRs3YsGCBZg2bRp++uknuc9zdXXFpUuX8Pz5cxQXF0NXV1eBURKi\nmiiRE4VbvXo1IiMj8c4772D58uUNPv/OnTsYM2YMHj9+jPj4eDg6OiogSkJUFyVyolCrV6/Gv/71\nL5SWljbppmdpaSm0tbXRpk0blJaW8hghIaqPRq0Qhdm8eTMiIiJQVlbW5JErbdu2RW5uLnr27IlO\nnTrxFCEh6o0SOWmyzz77DB4eHtDS0uKlPBMTE4jFYjx9+hQ7duzgpUxC1Bl1rZAmmT17Nn755Rc8\ne/YMv/32G9555x3eyn78+DE6deqE8vJyaGpq8lYuIaqKulYIb1JTU+Hi4gKJRIKoqCjExsYCAHx8\nfASOjJCWiRI5aTCpVIqrV68iOTkZbdq0wfDhw7F8+XLeVwjq2LEjli5dyluXDSHqqkmJ3NzcHP37\n94ejoyOcnJwAVP477ObmBmtra7i7u6OwsJCXQEnzYW9vD1dXV0yaNEnmcX1F8PPzQ6tWrRRaByGq\nrkmJXCQSQSwWIyUlBcnJyQCA0NBQuLm5IT09HaNGjUJoaCgvgZLmpaYRJaamprzXY2dnBw0NDXz+\n+ee8l02Iumhy18o/O96joqIQEBAAAAgICEBERERTqyDN0K5du6ptS09PFyASQkiTW+SjR4/GoEGD\nsHPnTgBAfn4+jI2NAQDGxsbIz89vepSkWWrVqhX3h1wqlSq0rqqpcgkh1TVp+GFubi66dOmChw8f\nws3NDVu2bIGHh4fMYrwdO3bE48ePZSsViWQe4XZ1dYWrq2tjwyACyczMhJWVFZ49ewYDAwOFzUeu\npaWF8vJy3sslpLkTi8UQi8Xc+5UrVyr2Ef2VK1dCT08PO3fuhFgshomJCXJzczFixAjcuHFDtlIa\nR64WKioqMGvWLGzdupUSOSFKwPs48ufPn6OoqAhA5cow8fHxsLOzg4eHB/bt2wcA2LdvH7y8vBpb\nBWnmWrVqBW1tbRgYGMDBwQGtW7fmvY6UlBSFd9sQouoa/ZuXn5+PiRMnAgBevXqF9957D+7u7hg0\naBB8fHywe/dumJubIzw8nLdgSfPzr3/9C99//z0mTZqkkPHehw8fpkWdCakHPaJPmkwkEiE4OBjL\nli3jtVx6RJ8QWfSIPlGYo0ePKuwP84wZMxTSZUOIOqFETprM3d0dRkZGvJdrZGSEYcOG0dBDQupB\nXSuk2cnNzcVXX32FyMhIFBQUCB0OIc0Gda0QlVBUVISuXbvi0KFDjU7ienp6EIlE3Mvb2xsA0KFD\nB4hEIhgaGmLr1q18hk2IoCiRk2YjKysLAwYMQOfOnZGUlNTochISEmBgYMC9P3LkCADgzJkzmDx5\nMh49eoRPP/20yfES0lxQ1wppNjQ1NfHq1SteRqm8fPmSGw759ttvIzY2FgMGDMCff/4JDQ1qvxDV\nVFvupOEARKHu37/PtYgBICgoCGvXruXez5gxA/r6+gCAb775BvPmzeOlXk1NTbi7uyM+Ph5xcXEQ\niUQICQlpFkk8Li4OGRkZACqfjo2NjcX48eO5/XPnzhUqNKKiqEVOFMbLywsXL15EXl5erceYmprC\n09MT27Zt473+kpISODk5IS0tDUD1mTqFYG5ujocPH9a5CEePHj0QHh7OzfFPSBW62UmU6ssvv0Rk\nZGSdSRwAJBIJtm/fLtNK54uurq7M06YnT57kvQ555ebmol+/frh79269KyndvXsXgwcPRkJCgpKi\nI6qOEjnh3aRJk/DNN9806JzFixfjk08+UUg8mzdvBlD5H0LV+qLKNmLECFy/fr1B53h6elIyJ3Kh\nRE54tWDBAhw7dqxR527fvh3r16/nLZatW7ciPDwcQ4cORfv27fH8+XOcO3eOt/Ll8eDBA/Ts2RM3\nb95s8LnFxcUYNWoULl++rIDIiDqhPnLCKz6ewuTjsxEYGIiysjIcOHAAAHDu3DkMHz4cQOUN1r17\n9za5Dnns2bMHgYGBTSrD3NwcWVlZPEVEVFltuZMSOeGNrq5uvf2/8ujduzd3g7IxLCwsuK9jYmLw\n+PFjTJ8+XeaYxYsX44MPPpCrvIKCghrXKJUHX9MLbNmyhca+Exp+SBQrJiaGt8Uf8vLycO7cOQwb\nNqxR59+6dUuubfJas2YNXrx4wb2Xd4TNokWLGl3nP4WHh1MiJ7WiFjnhxfLlyxEcHMxbebt27Wpy\nlwSfpFIp9PT0AFSO/S4vL4e7uzu3uLiWlhZatWolc46hoSGvc8XQ7wyhFjkhTaChocF1G6WkpHA3\ndPv27YusrCy8++67sLOzAwDe52UnpD7UIie84LtFrqOjo5AVh/hWXFyMV69eyWwzNDTEw4cPqUVO\neEctcqJSNm/e3Ky6VgDg6tWrSE5OBgCsWLECEokE48aNQ9euXQEAO3fuFDI80oJRIiekHuvWrcOP\nP/6IgoICPHjwAF5eXtxTot27d4eurq7AEZKWjrpWCG+ay/BDvu3evRsRERGIjo5u8Lk0/JDwieZa\nIQp3/PhxoUNQiMDAwEYlcQBYuHAhz9EQUh0lcsIbFxcXdOvWrUllWFhYNKvWeFMNHz4cbdu2bVIZ\n77zzDrXGSZ2oa4XwKj09Hc7Oznj48GGjzo+OjpaZm1sdxMTEYMKECY0+n35XSBXqWiFKoa+vj+7d\nu6NDhw4NOs/Q0BCTJk1SuyQO
}
],
"prompt_number": 49
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What does \"learning\" actually mean?\n",
"\n",
"- training strings of length less than 100\n",
"- testing strings of up to length 1000\n",
"\n",
"Tests may mean one of several things:\n",
"\n",
"- detect when the language is accepted\n",
"- take an entire input sequence and output TRUE/FALSE\n",
"\n",
"Does it generalize?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Claim: LSTM networks learn non-regular languages\n",
"\n",
"Discuss..."
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Timing Tasks"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def generate_timing(ninput=default_ninput,n=default_n,t=5,example=0):\n",
" \"\"\"A simple timing related task: output a spike if no spike occurred within\n",
" t time steps before.\"\"\"\n",
" x = 0\n",
" inputs = []\n",
" while x<n:\n",
" inputs.append(x)\n",
" x += max(1,0.5*t*randn()+t)\n",
" inputs = [-999990]+inputs\n",
" outputs = []\n",
" for i in range(1,len(inputs)):\n",
" if inputs[i]-inputs[i-1]>t:\n",
" outputs.append(inputs[i])\n",
" inputs = inputs[1:]\n",
" xs = zeros((n,ninput))\n",
" xs[inputs,:] = 1.0\n",
" ys = zeros((n,1))\n",
" ys[outputs,:] = 1.0\n",
" return xs,ys"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 37
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"x,y = generate_timing(n=100)\n",
"ylim([-.1,1.1]); plot(y,linewidth=3,alpha=0.5); plot(x)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 39,
"text": [
"[<matplotlib.lines.Line2D at 0x6fd56d0>,\n",
" <matplotlib.lines.Line2D at 0x6fd5e90>]"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD5CAYAAAAtBi5vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvW3MLddVJvjsj6o67+vrtIGgDPG1FIg9slEgiWQG+BH6\nwohJTDeG4WNk5kdDEkVWBotGIho0SIj4BxmCNEg0np5JACUNEp4wIySDOjgiqC8joBMDIemhk3EM\niXscQ9K4Q3DwfU9V7Y/5sWpX7fo459T+OG9y3WdJEdx7j/epqrP3qmc9a61nMWutxclOdrKTnexF\nZ/xLfQEnO9nJTnay49jJwZ/sZCc72YvUTg7+ZCc72clepHZy8Cc72clO9iK1k4M/2clOdrIXqZ0c\n/MlOdrKTvUhNXtYXMcYu66tOdrKTnexFZbHV7JeK4K21p/9Zi5/5mZ/5kl/Dl8v/Ts/i9CxOz2L/\n/1LsRNGc7GQnO9mL1E4O/mQnO9nJXqR2cvBfArt27dqX+hK+bOz0LAY7PYvBTs8ijzGbSvKs/SLG\nkvmkk53sZCf7z81SfOcJwZ/sZCc72YvUTg7+ZCc72clepHZy8Cc72clO9iK1k4M/2clOdrIXqZ0c\n/MlOdrKTvUjt5OBPdrKTnexFaicHf7KTnexkL1I7OfiTnexkJ3uR2snBn+xkJzvZi9QOOvg3velN\neNnLXoZv+IZv2PmZH/uxH8Ndd92FV7/61fjzP//zrBd4spOd7GQni7ODDv6Nb3wjHn/88Z3//v73\nvx9/+Zd/iaeeegrvfve78da3vnXnZz//+biLnNoXvwh84Qt51vpytS98ge7zGKYU8Dd/AxxLOeJv\n/xa4uDjO2r5ZS/ehVL71PvtZoG3zrHcM+0//CbhxI996x9xnX+5mLfDXfw1o/aW9ji9+Efi7vzvO\n2gcd/Ote9zp8xVd8xc5//+3f/m388A//MADgm7/5m/GFL3wBn/vc5xY/+8lPRl6lZ5/7HPALP/tp\n/OI7/j2efjp+nWc++n+jufiHg5979i/+LS6eX34zfepDu198KfbpTwO/+I5/j1/42U/jP/7HvGtb\nC/zqrwK//I4P4P3vz7s2AHz0o8C/fMe/xS+88/OLjijnM/vX/5ru41d/Nc/L6g/+APiVh/8v/G9v\n+Wl86F/+NJ74338an/3kR4LX+fSHP5B+MQv28Y8Dv/Szf4r/5R2fxfPPp6/39NPDPttxZNet88Tv\nwejlt2zMs/jMv/tD1C9kuMED9thjwK/8zx/Ae98bv4bRCk8/8XvR//3f/i3wC+/4D/gX7/gLfPrT\n8dexy5I5+GeffRZ33HFH/+erV6/iM5/5zOJnf/mX3463v53+d/369ajve+op4Gv+7CG88o/fhKee\niloCAPC5f/Z9+H/e90sHP/f/vfkH8LF/9c7Ff3v56+5b9ZIItU9+Erjzj96I/+Ij/xx/+Zd5137+\neeBvnvxTvOnX/gmefDLv2gDw5JPAt3zgB7H5xM8tvoBf/rr7sh3eJ58E3vRr/wR/8+SfZkGhTz4J\n3P/YP8M3/5t/Afvud+MrH/55/L8/9WDQGn//2f+A2779vvSL2XF9r/03P4SX/MVP41OfSl/vk58E\nXvnHb8LX/NlDSfvshf/ue/Hk7/0fs7//3FMfxea/+a7g9f76h78P/+7RX4y/oJX25JPA//Cr9+GZ\nT/5VdFT01PXfwvYHvif6Gp56CnjZR/45/ss/+pEeAF+/fr33k29/+9uj1wYyjeybKp3tGs/3Iz/y\ndrzudWnfpTUg1RZSNUmhlWwUmu3hX1U0LdR2zjcYrbBRwAvbGyjPrsRfyIJpDRRqC6m32cNHrQGm\nnkepzFFCU62BUrXg6mK2vjUGGwX8Q30D1S0vyfJdlTLg+h+y3As9d4PHvve9+JH/6b/Fp37h+8A+\n9VdBazQ3vohb1HG4L7rfBiLTvqD7rSF1nbRe0WrUFy/M/r554XmcRzwL2Si0K85mqmkNbJQF1Bej\n77+98Q/YtPEPj/zZBYTnz65duzaSS3744Yej109G8LfffjueeeaZ/s+f+cxncPvtty9+1pjUb6M1\nuFXgRidtSq4NTNus+pxdIGVVsx3935xmDMCNBjcquxM2BmCmgTTH4R6NAYTRYKad/d7uWemmzvZd\n0gAwaQ7KmdaANBbgFV27LAAVtrBqtnRNRzD/2ea431z7TGi7eJZ0U9PzDDSuDYw6fDZTjX5vgJk6\n2jeZtobQ8S90+g0UxBHOOpDBwd9///34tV/7NQDAhz70Idx222142ctetvjZXA5e6BbCqKT1uDYw\nzeFNJJRe/JxDGKrOn02ke1TgC04yx9rMXKAwNvvabn1hDPjCoWlrema50JkxQKEtmN5m21vSGBi+\noedUlmCBGVzdbFFqilZyG+0LDa6bjGcpfZ9JbaAXgI5qtpARTksoDVMf38EbAxTdeYh28E0Nmejg\nhUn3Z7vsIEXzQz/0Q/iDP/gDPPfcc7jjjjvw8MMPo+0Q7YMPPojv+q7vwvvf/37ceeeduOWWW/Ce\n97xn51q5UBY3CkKnPRChDOwKJMmVgV1AJ86xH8PBaw0IoyB0fgevNcD0FoXO88JdWl92Tmj6e/cv\nxSbPM9PaHdCMDl5bQJzRcypKsDbMwTs6T6sGstykX5RnhDg1RKYXf659JrSFWThLur5AEbEunbk8\nUd4usxawRkFYAAmUl663kDr+4ZE/ayG0hvpSOPhHH3304CKPPPLIqi/LhjqMgjBpD0RoC6sO18Nx\nbRY/l5tu8M0YoDQa3OgjIXiiaI6L4OcvYPesclM0KSH2fL2BomFFARZ48nXnmFSzze7gKZw32SK7\nXPtsF0Vj2iaKrhJm3dlMMWMAGDrDaRRNC5Hw7AaaLP9ZBy65kzUbB6/TOSuhDewaikYbYOFzPYLP\nhEZ9c6G4yMS1TtdmZgtpAa1N9lp4R3NwM6cR3LPKEfXkQmC+OQ7e8jPaZ2UJFsjBO6riGElCijA0\nuM7HwQutkveZ3IXgmy24BfSKXJdvFF0fl6LRGrCafqOUCNC0GSga3UJ+uXLwIZY1rExFHStRgtiR\nZM2NRkdra0qmHeOtrjUheABAJmpjur4wdpGi6Z9Z4IHf9T0DAsvHSUsDQJx3FE0V7uDbY+8LA27z\n8LVED5jkfSYNFs+IQ/WhhQhCW9hAaizUpgg+1rnatklKstJvoMGNufkRfDbUYRSETtyUK1GC0BZY\npGg6rrU+ThWNMDo5z7Brbabdxr5xlPWlXqZoHHLXGRC8MQMCW0roxq5Z6IGi4UUBHurga3LsbXMk\nBG9MttwMrZe+z6S2MAucuTsbLrm+1oQ2QAYQsM9csQEAICFJb5o6Ks/gX4cwCjLRn+2ymw7BO/pC\nJpZJEkpYgeCVWexdV7Xj4I/j4KXWRymdIs6PDqPVN46yfmHMYiJwcPDpz8w/oCxnXbgBLD/vKJoq\n2MEbVz670DuRav7LMx9Fk77PCoNFsORom9BnIbVdLGzIaVoD0HRdKQDBNg2KhGc30GRp/myX3ZQO\nPkdII80yMp+aMHZR7MT0dMNxkqzCmGQaatfajqLJlZycri+MXU6ydoc2B0UzDrFzIloAXZkkL0rw\nwAoJ09/jMfdFnsiujxS/3CiaHWcup5kuOQ+kUXxWtfSCi1xg+A1OFE2/hswQ0kiNVapSUtvFzznk\nfgyKxnGtx6BoqEzSbez4+t9960ttwRcSd46aMRminlwIzJlL2jILMF4SN1pWEMEUzXH3hTQWPNO+\noPVMEt1pjekc/B4EH1iIsOvM5bRcEaDrkdGRjVkup3iiaLw1iKJJa7Vfu4mIg5+jCUfRrOmGDbWB\noskftlEE1F2znssJ5FhfGrv4cnIvxRzPbHxA0ztZXVVFK4b1RbUJR/DOqR2pAU5qk426y7HPHL++\nSNG0cc+iuAQHTwCB9mMSQOhYgNiqqYEmO450yM3p4I1JDmmkwaowUO5w8G7zLpWHpZqjOY4RtvkU\nDY5E0cgdFE1PX2RA8H6InaMu
}
],
"prompt_number": 39
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def generate_revtiming(ninput=default_ninput,n=default_n,t=5,example=0):\n",
" \"\"\"A simple timing related task: output a spike if no spike occurs within\n",
" t time steps after. This cannot be learned using a causal model (it requires\n",
" a reverse model).\"\"\"\n",
" x = 0\n",
" inputs = []\n",
" while x<n:\n",
" inputs.append(x)\n",
" x += max(1,0.5*t*randn()+t)\n",
" inputs = inputs+[999999]\n",
" outputs = []\n",
" for i in range(len(inputs)-1):\n",
" if inputs[i+1]-inputs[i]>t:\n",
" outputs.append(inputs[i])\n",
" inputs = inputs[:-1]\n",
" xs = zeros((n,ninput))\n",
" xs[inputs,:] = 1.0\n",
" ys = zeros((n,1))\n",
" ys[outputs,:] = 1.0\n",
" return xs,ys"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 40
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"x,y = generate_revtiming(n=100)\n",
"ylim([-.1,1.1]); plot(y,linewidth=3,alpha=0.5); plot(x)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 41,
"text": [
"[<matplotlib.lines.Line2D at 0x711aa10>,\n",
" <matplotlib.lines.Line2D at 0x711d210>]"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD5CAYAAAAtBi5vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvX3QJdddJvb0Oaf73ve1LBsj1sEzCrItVSQwspWSMYbC\njAlEGlcxpPhasUnKaytmokLLR2EgIR9YVEIhllSKoK1EYDAFGK1IhaxMlRnXChgoDJYIYDvEXnls\nIzwadg3CaGU8773d5yN//Pp0n/667+3T59zpV9ynymW9M+/06du3+9fPeZ7fR2KMMdhjjz322OMF\nB3atT2CPPfbYY4842Af4PfbYY48XKPYBfo899tjjBYp9gN9jjz32eIFiH+D32GOPPV6g2Af4PfbY\nY48XKMSuFkqSZFdL7bHHHnu8oOCbzb5TBm+M2f/PGPzYj/3YNT+Hufxvfy3212J/LTb/bwr2Es0e\ne+yxxwsU+wC/xx577PECxT7AXwOcOXPmWp/CbLC/FjX216LG/lqEQWKmijzbLpQkk/WkPfbYY49/\naJgSO/cMfo899tjjBYp9gN9jjz32eIFiH+D32GOPPV6g2Af4PfbYY48XKPYBfo899tjjBYp9gN9j\njz32eIFiH+D32GOPPV6g2Af4PfbYY48XKPYBfo899tjjBYpjA/zb3/52vPzlL8dXfuVXDv7O937v\n9+KWW27Ba1/7WvzZn/1Z0BPcY4899tjDD8cG+Le97W24cOHC4N+///3vxyc/+UlcunQJP/dzP4f7\n7rtv8Hc/9zm/k4yNq1eBv/3b3az1uc8BX/jCbtbaFbQG/uqv6P9DwBg6nlJhjhcDn/0skOe7X/dv\n/5bu19gwBvi3/xYoivhrbULoZ3O9Bv76r6cd47nngM9/Psz5uJAS+PjHgaeeAj796TDHPHbgx9d9\n3dfh6aefHvz7973vfXjrW98KAHjDG96A5557Dp/97Gfx8pe/vPO7n/gE8NVf7X+yMfD888DP/NS/\ng7r6DL7zv7oTX/7l8db6xCeAX3v4w2CLG/DPfug0vuiL4q0VCn/23v8Fxeefox8Yx+3/5PuxvO6l\njd9573uBv3jiAm590934zu+cvua/+lfA//s7H8DpO+7C298+/Xih8cEPAo8/+js4eMXX4Ad+cIk0\nnX7MK3/+R3jZf/gf4eD6lw3+zsc+Bvz6u/8f8MPT+L4f/g9w/fXT1x3C448Df/S+D+ClN38T7v9n\nDMxTzP2LJz6AV77hrtH/5m/+7A+wXgN//CTw+Ru+Cd/+3W/CV3yF3zlYrNfA//rTf4/1X/8pzv7n\nb8JXfdX4Y/zlXwK/9C/+P4Ad4r/+gVeiJ8x5Y7UCHn3vczj4y/8d7Pb/Fj/0Q9OPOXmi05UrV3Dj\njTdWP58+fRrPPPNMb4D/+Z9/F+xm4MyZM7PoGPepTwHX//n/gJv+8iI+8aZL0QP8V/z+f4G/fvnt\n+OQnfw2vf328tULgub/6C3zFW9+JD9/+jwAAr/zE3+DPhcCdb//vq9+REvjkU8/jv3n4LH765WGa\nyT31FHD/u8/if/u+v8DR0Zfh4CDIYYPhqaeAu973rfidN/9P+Ku/uh9f9mXTj/mX3/2d+Mx33oM3\nfv8/37juHb/7XXj6y87g05/+ebzuddPX3bTWPf/yHP7lP34Mn/vc3bjhhvHH+OylD2P5n74F+Pfj\ntmLPnv8vcfj8VRTXvQjf+Defx2f+0Xvx1Nd/enKAv3wZwL95CG/+o5/GU3c+6xXgP/EJ4NV/eC+O\nDl6GS5feHzTAaw0kVy/h1f/Xf4ef/+t1kJ1+kJF97U5nQ+P5/uk/fRe+7utCrBgOSgFcXoWQBVaR\nJQGlgFTmYHI1a/nBIj/6PIqDBF/1p58FADz5H78cZr1q/I5SANTf40ACUmoYwzB1OqNSwFIaQH5+\nltdJKWAhJbi6Guz8+LqAXG/WXmjdHFzFv39oLQ2m/t57rfwLz+NQjn/p80Iif+B/xMFX/DAe/4nv\nw23/5v8O8nnpWf8CMim9j6cUIOQaXK6DfwdaA4le46syhst3vws/8AP05w888ID3MSdn0Zw6dQqX\nL1+ufn7mmWdw6tSp3t8NpdGGhNYA0xJcq508NFwrMC1neS3aUPkaktXRWgsOUzSFZ3tTAkBi8iCf\nSylAaAB6NcvrpDXAtUai82D3DFMa5hjBm9ZVSHSxo3vVANo/kKl8DaHHB3imNFiaQSnAsAxcqyD3\nAd2rObjWkwI80wrMhH+GtQagjiBZAs7DHHNygD937hx++Zd/GQDwoQ99CC996Ut75RlgzgE+D3YT\nHbcW1xJchwmEsSHXR1C8GeB1vm78jtYANDFPo65O/lzGAEppZBpgMw7wQhkkah3s/JhUMMe4tloD\nXCkwFf/+0RpIJ35Gma8gPAIplwosW0BrQPMFuAoX4JnOIbT2Pl71DKsi+HegFAC9huT+nkcbx0o0\n3/Vd34Xf+73fw7PPPosbb7wRDzzwAIqSaZw/fx5vectb8P73vx8333wzXvSiF+E973nP5g8wMygF\nMJVDBLqJNqF+QOMzsBCQ6yMkor7TjOCd1BGSaI4AAIm+Cq2HTcJtYAztBOjgR7O8TrTD0GA6XIDn\nUnV2R/3rKnAdPri0oTUgtJn0klXrI6Qe/5ZJDWRLaA0YtoAItLtWCkjUGlxNDPBKgml/mWfTsZle\nQbFkdwH+kUceOfYgDz300FaLzZWNWYlmNww+zvYuBlSRI3EkGsM5IJsygivRIEDA06U0E+p4MUDf\no0ESMNAypWHk8RIN0xpshwE+mbDb1EVOUttIcG1gsowYPEvBJzDuxvlogOkCQptJAZ5esrEkmhUK\nFo7B77SSdb4PazFJl9sWFQOLsL2LAbU+guIOg08FdN6nwZcBOQDjVoqkHmDeEk2qNBIVzmhjSsPk\nxwd4saMdoFJAqgBMMHRVvgIzRBTGgEsNnh3QunwZbHftSjRTNHiuJZgKH+CVIo9AMTYfDX4M5viw\nkmkig7GETagZWPjtXQzoIocSDoMXosPg6aa0DH56QG7vCOZ4nawBGZJJc6U717Z/Xb2THSAxVUz6\njLoM7DJfHfObTXBlIBbLksEvgpEvetYL8IkMnpcMPpZEE1KD32mAn+PDWm3bdqTBCxVnexcDKl81\nGfxAFg0UPcAhGLdr2oY0MUNCayDVBiygWc6lObZslIKu3skO0BrdyQSZTJUptcUx6Z9tcKXB04Wj\nwYeTaBKVI52swceTaCyD35kGHxJzfVi5kpO2bduiMucibO9iQK1XSBoBPgXyPg3+qPwHYSQaa9rO\nWaIRpTkfLA9e6e0CvIq/A6SXLL3I2QQZymZcydXRqH8nlIFZLEmu44tJpmjjfEo5VmgzLQ9eK3Cl\nIGNk0ahVUIlmH+Ark9V/2zZmLa71iWHwusgbAR5iyGQtWX0gk7XKq59pOmltQAaUaLShsuBj1t3F\n/eMa3VNNVsBDotEGpmTwYItJpmjjfAKZrCSzht/x07HXkJyfTAY/R4nGmiYiEEvYBMvAYuh3MaDz\nNQV1izTtsExKPQsr0dSm7TwrfpUC0pINhrpnhDyewRN7NNF3gFq7RveEAG8ZfO7B4JcHpQa/DLa7\nppToAkL5k00rs4pIGnyii71EExJWokl3KNHE2N7FgMrXSI4J8JZ10D+YHpCbEs2MNfjgEs12DH4X\nBME1zqd8Rl2UAX49LsCnysBkh2UWzUEw8mUlmnSyRKODFV+1j83UGorxfRZNKFCAUjuUaEyU7V0M\nmCKHbkg0ohOEXIkmhKTSPN780kltpa0wCNpyQmiDZJsAr82OJJrS6J7wHdjWC6pV/XwcuAZ4Fkui\nkRB6GoMnmSz8jt/e+5KLfRZNKNAbWUKo3QR4oXSU7V0M6GINvZVEU7K9YBJNyeAD5pmHgqtPB5Vo\nFIBic4C3Ek0M9uiCMqPsLmqKREMvajVSg08VkC4PHYnGn3G7sGmS6WSJRkfpXUUvoBx6X+gUDpbB\nh2IJx61F5d/x9f4QMEVB7Qks
}
],
"prompt_number": 41
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def generate_biditiming(ninput=default_ninput,n=default_n,t=5,example=0):\n",
" \"\"\"Like generate_timing and generate_revtiming combined. Requires\n",
" bidirectional training.\"\"\"\n",
" x = 0\n",
" inputs = []\n",
" while x<n:\n",
" inputs.append(x)\n",
" x += max(1,0.5*t*randn()+t)\n",
" inputs = [-999999]+inputs+[999999]\n",
" outputs = []\n",
" for i in range(1,len(inputs)-1):\n",
" if inputs[i+1]-inputs[i]>=t and inputs[i]-inputs[i-1]>=t:\n",
" outputs.append(inputs[i])\n",
" inputs = inputs[1:-1]\n",
" xs = zeros((n,ninput))\n",
" xs[inputs,:] = 1.0\n",
" ys = zeros((n,1))\n",
" ys[outputs,:] = 1.0\n",
" return xs,ys"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 42
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"x,y = generate_biditiming(n=100)\n",
"ylim([-.1,1.1]); plot(y,linewidth=3,alpha=0.5); plot(x)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 43,
"text": [
"[<matplotlib.lines.Line2D at 0x7354e50>,\n",
" <matplotlib.lines.Line2D at 0x7359650>]"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD5CAYAAAAtBi5vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvX/sNdldHvacHzP3ft/dNYaYWGF3FQNetE4dbNQlgNSg\nF1pkb9osFT+qpWkLxqUrikNRsBqFCsWbVIBpU1XKpsJIiQlJcU0r1MWSWRQTvYkQwUsNNgQbZwFv\nsl4nxhtwFu/3e2fOr/5x5sw9c+acmXNm5nv95t37SJb87jvv3LnnnvOZ5/N8fhFjjMEZZ5xxxhl3\nHOjn+wHOOOOMM864HpwN/BlnnHHGHYqzgT/jjDPOuENxNvBnnHHGGXcozgb+jDPOOOMOxdnAn3HG\nGWfcoeCn+iBCyKk+6owzzjjjjsLSbPaTMnhjzPl/xuCv//W//nl/htvlf+e1OK/FeS2m/7cGZ4nm\njDPOOOMOxdnAn3HGGWfcoTgb+M8Dbt68+fl+hNsG57U44rwWR5zXYhsQs1bkyf0gQlbrSWecccYZ\nLzessZ1nBn/GGWeccYfibODPOOOMM+5QnA38GWecccYdirOBP+OMM864Q3E28GecccYZdyjOBv6M\nM8444w7F2cCfccYZZ9yhOBv4M84444w7FGcDf8YZZ5xxh2LWwH/3d383Xv3qV+PP/tk/m7zm+7//\n+/HAAw/gDW94A37jN35j0wc844wzzjhjGWYN/Fve8hY89dRTyb9///vfj9/93d/FM888g5/8yZ/E\n937v9276gGf8+42rK+Azn/l8P8UZZ7w8MWvg//yf//P4wi/8wuTf//zP/zy+8zu/EwDwNV/zNfjs\nZz+LT3/604se5t/+q4/jD597pujfGK3x+7+afgGl8NyH/ynaq88V/ZurF/8Qz//zfxb9u3/9sV/D\nS3/0B7P3+PQzH8aLn/lk0efmwmiNT3zwF7Ouzb1uDS4vgf/tnX+I/+NH/hk+/OFt7inbA/7lh/5x\n9O8+8+xv47Of+kT07z7xwV+E0XqbhwDwLz/0jyHbQ/Kz1mBqn/k4fO6zePon34Gnf+KH8fRP/DB+\n4//8W6s+N8SSNfvspz6Bzzz729G/m1ozH5/8zV9G89KLRZ+bu2bt1efw3If/adG9p6CVxLNP/6Pi\nf/cvbv1c/7uF//vQT/3oZs+3eqLT888/j/vvv7//83333YdPfvKTePWrXz269h3veEf//2/evDnq\nGPdbP/zfAZzj5t/9pezP/zcf/xB2D/9nwB/Jouf+9H/zLfiDv/KD+A+/669l/5sP/70fhfnZ/wv3\n/spzo7/7vf/+UeA/+Sb8R//TT0ze43d+4L8C+cqvxNf/6M8UPW8Onv21fwTzyCPAp5vJ6/717/x/\n2L35Py1es+LneRbYf+zH8Prf+hn8i6/7JN74xvX3/Oj7/h7MD/01/OmP/7vR3/32//gW4E++Gjef\neN/o78wjj+DZn/95fOnXvGn9QwD4o7/0Lfjs3/yf8YZvf9vgv/+7f/Mv8cpveBi4XP4y+cjffyfM\nP/yHuPeDz09e98//77+DL/0rfwOf+IovBgD8B7/5B/ijm/85vvDeL1/82T70Nz+CT/y/T+LLvvbN\n2f/mwz/2A8CnPoWb/8+vjf7uj/7St+Czf+Nv4g3/xV+evMenvvNb8K+/7y/jq//bH87+3N/86f8F\n5qf/Pu59+lOT1/3Wzz4B87f+V9z/my9k33sKz9z6OZi3fBfwry6L/p34rv8aux1Hc9d+9He/ohne\n9+z0Gc7FJiP7wk5nqfF8voGP4nAAqqros9uXXsQNWX6YeCvRHsp+FH31EngbN4q0FVCHq9l70KaF\nPsyzmCUQl3+Mi4y1aF96ETfEdmw2BaUAKl9CpQSU2uae8vJzqEX8ZqRpYJr42tZC4fLqpW0eAun9\n017+Me6S67qmqstLVIl9NrjucIVPfMUX48/9uvWYP3MPhTls9x1roXFV6OXicAXSxo0TbxWajN+A\ntxKi8Gyqq5fARcaaXb2EOmNtcyEuP4d9Yj9OoRIa1bv/Lt7wDd82+rs/F/z58ccfX/h0Gxj4e++9\nF889d2S0n/zkJ3Hvvfcuu5lSQOHsViUasAW2iioNLdqif2OkAElYKiIVjBSz97DXXQ9ztmsxb1yU\naMGv375bA68FmNabGXgtWjCVeHipgMTaMm2gE4ZnCVhi/8j2sHptjRSgGQtmRAvD2fGzKQG2/I7a\nQInC+0kJIuPPTrWGFvNnhCoNLQvPphCgqX3hQYs267pcaNGAqfIXOtUaqHabPUfyc9be4JFHHsFP\n//RPAwB+9Vd/Fa985Suj8kwOiBDJA5qCbA7gSxZYaei2cBO1LVhq80oFk3E/IiVQ+GLJhWqarLVQ\nzdWiNSuF1gBRLbhS2Er+1m0DlmDIdm3jBoQrA5Wh/+aCyfgLQ7UH1Aqr9H4jmjxj1TbQnoFXjEA2\n815kLrgyUE2hgRfC/g4RMKmhM14YTCropuyM6ImzGV63qYFv885cCK4MeD2WZ7bGLIP/ju/4DvyT\nf/JP8MILL+D+++/H448/DtEdosceewx/4S/8Bbz//e/Ha1/7Wtx1111497vfvfhhSIaLFUK1V6gW\nsEMmNUwh2zGiBU1IIFQqmBx2IlXSCK2FbvNedqo5oDoBg9caYLoF03pbA584oFTIJEPkCtsaeKWj\nBl52Mp2S7eIDPEUkwuvgM3hOYTY28Lp0zYRInmOmNHTGC4NKDVPoOUydzcF1bQO2QNJNQTUH8AUv\nDC41sL/Y7DmSnzN3wXve857ZmzzxxBObPMwSiUa3y+QGpkyWpOJjyg2kSgMZ96MqLSOshZUvbh+J\nRmuAdBLNZgZeiKQMRZRCSgti2hRLclOw9xv/3k7SkO1huYEXMotlGikA5jF4SmC2/I5qwZpJlZQx\n7Zmbvx/TC86mzJNojBSLJJUU7H4s/3dMm38/JJotQYVMuncpqNay0VKXmCmdJan4MEIk3/4sk5mT\na2TwucxcdzrxlmmD0edRAFUCXG2nwZu2sewnAiKVlfkiqBQ21eB5wgN0XkJpkHCATDaq2xamOnI0\nxSjUhgy+0nZPlYBIkdTguTJZZ85614UvlnYiNuPBZF6XCy1WSDS762fwt5WBJ0olN0cKjmGowqDM\nEpYAKcASlsoy+PmXU+51S6CFyGLmbs1ycpJXPY8+Blm3epcYOcHgpbKB1gi4xuYMPrZ/HINXK14m\nRsqsYDmkgOGegedk0+9o16z0jCgrQ0aQe+aYMjCFcq2RMs/AT+yfJTCZXnMIpvEyNPBSgRYaP8fK\nShkTX8IShABN/JhU6SxmziYyPdZCiyYrwNezzGYFy8x5Hg0wLVBtaeDbNsmYqFRRD9BojUpvzOAT\nbNQFJUW7jsHn6MRhFo1idLM4g9EatUJWUNQHETJp4K2mn8HglS5PRMg0tKZtkx7gEui2WRTP4sqA\n1S9LiaaMwfcHqtBYWZZQyE6ESLIEJvMMPJV6UTA5By6ANefNLH0pluIo0ZjtJBrRgifuRaUCjayt\n81S2NfCI6t0uKCkzaiKSmNhnPkwrYPixbkQzWiyppOD2UE5Q1AeR0wY+J0aQe90AE/KpDyO21eBN\n2y5K8qgUUO1ubPYcKdxWBp4oXWzgXdCm1CXm2mQFRQeQMrk5mDLJAJ8PqnRxnCEXbi3mpJde1trQ\n4EU/RwPUyG0lmokgK1U6GuDrDfyGsQ+m4wShX9vS/HEfE/tseJ0YZNFoRjeTaNya5QRFfRClksFO\npvLOHNOm3MstkLX4lhKNFItigFzjJGmSt5WBpxP6XQqOlZXm/3KF8mCnEEl5gCmdDPCNrrsuA9+5\nv3PMvF+zdruAXPRzNMCUPVBbGvgUY2IyHsNx3l1pWuwUKoWogXcMehWTnthn4XV+5bfibDMvxe2h\nUhmTTmQAVdp6HXPgymx6NsfXld16Ck5yKokBOsnw5WfglT6hgV+yiWQy55WrPOmFSQ2yoLQ5B/1m\nmznk7rpVMkIGlAKYluB6O4kG
}
],
"prompt_number": 43
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Other examples:\n",
"\n",
"- measuring spike delays, discriminate a delay of $t_0$ from $t_0+1$\n",
"- more generally, detect any delay $t_0+t_d$ where $t_d$ is from some set of delays\n",
"- generalizing timing spikes: the input is $t_d$, the output is a corresponding spike delay"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [],
"language": "python",
"metadata": {},
"outputs": []
}
],
"metadata": {}
}
]
}