nvidia-AI.summerschool17-no.../teaching-ncso/22-mlp-oop.ipynb

803 lines
122 KiB
Text
Raw Permalink Normal View History

2017-09-22 17:38:09 +00:00
{
"metadata": {
"name": "22-mlp-oop"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Title: Multilayer Architectures\n",
"Author: Thomas Breuel\n",
"Institution: UniKL"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"\n",
"import scipy,pylab,numpy\n",
"from numpy import *\n",
"from scipy import *\n",
"from pylab import *\n",
"from urllib import urlopen\n",
"from gzip import GzipFile\n",
"from scipy.spatial import distance\n",
"from scipy.spatial.distance import cdist\n",
"\n",
"def sigmoid(x):\n",
" return 1/(1+exp(-x))"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 188
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Multilayer Architectures"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the previous worksheet, we looked at backpropagation from a mathematical point of view. Let's look at it from a software architecture point of view.\n",
"\n",
"Let's start by defining a single layer, equivalent to logistic regression. We define three main methods:\n",
"\n",
"- forward: take an input vector and compute the predicted output vector\n",
"- delta: take a vector of errors, compute weight updates, and output the delta for the previous layer\n",
"- update: actually perform the weight update"
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"# MLP layer\n",
"class Layer:\n",
" def __init__(self,ninput,noutput,eps=0.1):\n",
" # random weight initialization\n",
" self.ninput = ninput; self.noutput = noutput\n",
" self.x = zeros(ninput); self.y = zeros(noutput)\n",
" self.A = eps*randn(noutput,ninput); self.a = eps*randn(noutput)\n",
" self.DA = zeros((noutput,ninput)); self.da = zeros(noutput)\n",
" def forward(self,x):\n",
" # forward propagate x through the network\n",
" self.x[:] = x\n",
" y = sigmoid(dot(self.A,x)+self.a)\n",
" self.y[:] = y\n",
" return y\n",
" def backward(self,delta):\n",
" # compute the pre-nonlinearity delta\n",
" delta_y = delta*self.y*(1-self.y)\n",
" # compute the weight updates\n",
" self.DA += outer(delta_y,self.x)\n",
" self.da += delta_y\n",
" # propagate the delta backwards\n",
" delta_x = dot(delta_y,self.A)\n",
" return delta_x\n",
" def update(self,eta):\n",
" # update the weights with the accumulated deltas\n",
" self.A -= eta*self.DA\n",
" self.a -= eta*self.da\n",
" self.DA[:,:] = 0\n",
" self.da[:] = 0"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 156
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that we could update the weights directly in the `backward` method.\n",
"\n",
"Separating the two steps is useful for problems where we need to perform a number of computations using the existing weight matrix before actually updating the weights.\n",
"\n",
"Let's try this out."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"# make a linear classification problem\n",
"def make_linear(n):\n",
" data = random_sample((1000,2))\n",
" labels = 1*(data[:,0]*0.7+data[:,1]*0.4>0.5)\n",
" return data,labels"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 157
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"# linear classification problem\n",
"def plotdata(data,labels):\n",
" d0 = data[labels==0]\n",
" d1 = data[labels==1]\n",
" figure(figsize=(6,6)); xlim((0,1)); ylim((0,1))\n",
" plot(d0[:,0],d0[:,1],\"bo\")\n",
" plot(d1[:,0],d1[:,1],\"ro\")\n",
" \n",
"data,labels = make_linear(1000)\n",
"plotdata(data,labels)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAFtCAYAAAD8jSoDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvX1wFNedLvxgD2YGEHgskQxCeG1EkmVl7Gje4JWr7pWI\n943GXiW5SXavBAkWsZERkcyX38S7RhIjg/FNmSrfQqAY72tXrjckWXa3atebGQfELsvIddcEKqGC\nBSRry+QNIGSwLjbgSBjh8/4x6lF3zzndp0+f090j9FS5jGZ6zld3/87vPL+vKYQQgklMYhKTmMSE\nwi1+D2ASk5jEJCYhH5PCfRKTmMQkJiAmhfskJjGJSUxATAr3SUxiEpOYgJgU7pOYxCQmMQExKdwn\nMYlJTGICwla4P/bYY/j0pz+NxYsXM69Zt24dPvOZz+C+++7DsWPHpA5wEpOYxCQm4Ry2wv3RRx/F\nvn37mN+//vrreOedd/D222/jb/7mb/Cd73xH6gAnMYlJTGISzmEr3P/rf/2viEajzO//5V/+BStX\nrgQA/Omf/ik++OADvPfee/JGOIlJTGISk3AM15z7uXPnMH/+/NzfZWVlOHv2rNtmJzGJSUxiEi4Q\nktGIOYPBlClT8q6hfTaJSUxiEpOwh0iWGNea+7x583DmzJnc32fPnsW8efOo1xJCbpr/kjU1IEDe\nf8maGiSTSa42MqkUNpWXG36/qbwcmVTK9/lJWyfKWrTV1lLXrh1AeyLB3XYmlUJbbS2SNTVoq631\nZd2c3ENHa+FgHbqTSdQXF2Pl7NmoLy5G91g/+vVZVVmJlspK27Wyujei4+N9LhyNx2X/QXqOROFa\nc//qV7+KXbt2YdmyZTh8+DBuv/12fPrTn3bbbMFjdNo06uc3wmHcytlGT1cXtvX3Gz7b1t+Pjp07\nUV1X53KEwUXo2jXq57cCwMgIVxu96TT2r19vWL81b7yBHy9YgDnz5qF23TpP1tDtPWSuBec6/KCz\nE8e3bcPe0dHcZ2u2bcNf/ed/InTkCLb196MXwH4A23S/axsbs3mMtevWoa2/3zCn5lAI39K1v6m8\nHA+tXcs1Php602kc3LMHOHQIo9OmWd4rt+vDMxbzc8Ram6DBVrgvX74cmUwG77//PubPn49nnnkG\n169fBwA0Nzfjz//8z/H6669j4cKFmDFjBn74wx8qH3QhgPYSaA/9waNHudpQ/eAGFcyNEQDCYa42\naEJ19/AwOk6cwNYTJzx7QZ3ew950Gj1dXQhdu4bRadNw6fJl6nU3ONchs2uXQbADwO7RUXzlH/4B\nPxv7vAdGwQ6wNyDt746dO3HryAhuhMO4r6oKBw4fxsGxvx9au1Z4XTVh+mB/PzrH7pHVvbJSomRA\ntYJlvt8ylQ5b4f7Tn/7UtpFdu3ZJGUwhgnVzaC+B9tB/MmMGV9uqH1y/0ZtOoz+dRqdJQ6tdtw5P\nHj+OFwYHc9duAjAYi+HbnBqhpfYP705ATu5hNBzO0xJXxWJ4MhYzroUDzThiEuwaZuiO+ywhwNqA\n9M+3bGjC9JDuM6t7ZaVEyYBKBUv1qUCKQfVmBevo3/fUU2jp7GS+BEuXLuVq3+mD25tOY29HB67+\n7ne4BmDmXXehcevWQB4ftbX7EevBfvlltG7ejKunT+NjADPvvhvf3rLFci76jfZUXx/1mhu6f3tx\nAnJyDy/++7/naYmvDA7i8XgcHffdl6ck8GA4RH/FP9I5ONDFvz9KhCZMl5o+t9poALoSJQMqFSzl\ntCvxCB525RnaamsJAfL+q49ESCaVktJHJpUi7YkESdbUkPZEgtluJpUiG2Mxwzg2AeSxWEzaWGSC\ntXbtiYRQe5lUimwqL8+1kwFIcyhkaPvpsc/d9iUyNp57mKypoa5JYzQqfA+7k8m8dVgdCpGnli/P\nrVdm7FkxrFV5uZLnJpNKkbbaWpKsqSFttbV5fch+LmSMV/9cyVwb1v1O1tQYrhOVnZ5q7olEO9at\nq0VdXTXS6V50dfXg2rUQpk0bzX1uBZHfqATryLZoeBgHOHZfHr6N9wjc09VlOLoDWR61Y3CQayxe\nQ/Zx16wFVQPA6CiWlZRg7ty5GHjnHbQOD0N7WmQe3e3Aew9ZWuKdly5h//r1ubacoKWzEz8AsKy7\nG+Hr1zEydSqqW1vR0tmJ3nQ6p/FeuHIFrQDmFBUJab88zzIPDaGaZnEKlScD5bSr6+2HEwAIQEh5\n+SaSTHaT8vJNhg2rvHwTSaUyzN+nUhnHv1ENppZB2X3NoGkEm1xoBEwtgGMsTmCnefFCREOz6ttO\nC+LVnv0EVUvUnTb80l7tQBt3cyRCupNJw3W891zFvZL13MoE76lAVEx7LtwBQoqL62n3mBQX15Oa\nmiSprW3LE9q1tW3U3yQS7V5NIQ+ZVIo0RyLUl9FOSNUXF5MkQNokUQVWG42owKRdK2tDcnrctes7\naMd5UWRSKbIsGiXJsXuXGfuvDSArZ88OjGDSg5ee5KUh9JAhlGUrUjLBs5EVlHCfPXslVVADSaZW\nXlOTpP6mpibp1RSo6E4mSX0kYngZHQspnYAX1bJpnPvTAHnUgnN3+tCr4Ml5NTS7vlVyo7xQcaqh\n8eFBEUwarE6N+mfD6fPD+3wWGo/vFAUl3FmaO9DO1MqDqLlrkCKkJDxwmVSKtMTjpDEaJcuiUdIU\nj7sSmGaIaF6ywNO3H9SLJlg2LF6c1VQlCGG9UGujvyiGe+Q35cBLTzrdgHmeT54NwM1z6/faEiIu\n3D13hSwv34QVK2qwZ08b+vv1oRObADxkuHZkZDyWc926WvT3G39TXr4Ja9caf+MEsgy0Tvx+rfyv\n3RqOnPofOzVq+ul3z9O3Sv9rGqgGQm0sEHdr0xvxzh4+DHz4Yd412j0KQgRl7bp1WPPGG9g9PJz7\nTHubD5juD8BvnOR5PnncCUWf2yCsrStI3mSYAEASifYc1ZJKZUgi0U5qapJjmnzGVivX/8bcVm1t\nG5Ovp0HUQCvSlx4sbaShpCQ4GpeDY7JX1EcQaBcz7E5hMk41dvfIK8rBToN1Sk/ygGduvCc6kWcn\nKHSOqJj2VLizQBe0TysV0iI0jwyPnaAIqUwqRVZVVpI14bCjsfjpdRI0jxcrrlmWILB7Xrygypxw\n3zLvD8+7otIDx08aUo+CFu6EsLVyO4hy8SIGWlm8v99CSv/SdAOkHiANt9xCvlFUlOe+djND2FAn\nedM221JWVVYyvYQ0z5rGaFQaR+ynBmv3rqhUltzOWxZfX/DCXRSiXjQigjqoHjtOoT20heCJ4Rd4\ntFXaNasjEbK6okLqpm01Fv13qu5nUDRYFlQpS242DpnulzetcBfVpkWooCB77DiB9rLyeGLcrPAz\n4MbpWLQxNESjSu5nULhnPyB6f2Wumajs9NRbRkX6AFEvGq3fnTs7MDJyK8LhG1i79iHL8ajw2PED\nmveA02yAhQIZaVR5PYm88NCxG4s2hs6lS4FMhnmdKIKWEsBLiN7fIKTr9lS4r1+/3yAY+/uzjmNu\nBLyIkNb/1knfbvoKErSXdYrJhUxDkFMK2wluWe5rQUm33JtO41RfHzqRzd5YC+Ty45jHomrMehfG\nC2fP4oPBQZROn46eri7D93ZQmbvci/adIBDPj5C+LwAAtFNKwVEaEwWZVIo0xeOW3jJBCOAwj9mO\nx5R1HFbt1cSztlbRzLSxeDFmUR5ZdQqAoKUYYN2L7mTS8TslKqZ9F+6FZoycaGBxikF7WQiR5/fM\nCy8NdbS1FYmJUGkDcLNxquLttU1Slb3B7dj096I7mRRKpyAq3H0v1hEO37C/CMFJ9xuUccgCi1MM\nYv1WHh5T5nFYFZ/Ou7as+f5xRQVzXCptAG54ZBUctJ6C62Rco5rjtqKCtP/3dHUhNDKCzK5daB0a\nMvzefN9ptKK5BCIvPBXu5eVixsh0ulcJX+8UssYhyg16ySkGwSBkBo/gLgTjH+/aBoK31cHNeFTM\nRb9J+lFNys6+Y5eeQoNdOgVh
}
],
"prompt_number": 158
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Single Layer Training"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In order to use our logistic regression layer, we need to invoke `forward`, `backward`, and `update` in the right order.\n",
"\n",
"This is handled by the `Network` class.\n",
"\n",
"Let's start with a simple version for one layer."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Network1:\n",
" def __init__(self,ninput,nclasses):\n",
" self.ninput = ninput\n",
" self.nclasses = nclasses\n",
" self.layer = Layer(ninput,nclasses)\n",
" self.eta = 0.1\n",
" def train(self,x,c):\n",
" # compute target vector\n",
" target = zeros(self.nclasses)\n",
" target[c] = 1\n",
" # perform forward computation\n",
" pred = self.layer.forward(x)\n",
" # compute delta\n",
" delta = pred-target\n",
" # perform backward propagation\n",
" self.layer.backward(delta)\n",
" # update the weights\n",
" self.layer.update(self.eta)\n",
" def pred(self,x):\n",
" pred = self.layer.forward(x)\n",
" return argmax(pred)"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 189
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's create and train a network.\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"network = Network1(2,2)\n",
"\n",
"for i in range(50000):\n",
" index = randint(0,len(data))\n",
" network.train(data[index],labels[index])"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 160
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Use the network for prediction and compute the error rate.\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pred = [network.pred(x) for x in data]\n",
"print 1*labels[:10]\n",
"print pred[:10]\n",
"sum(labels!=pred) * 1.0/len(labels)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"[1 0 1 0 1 1 1 0 0 0]\n",
"[1, 0, 1, 0, 1, 1, 1, 0, 0, 0]\n"
]
},
{
"output_type": "pyout",
"prompt_number": 161,
"text": [
"0.010999999999999999"
]
}
],
"prompt_number": 161
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"print network.layer"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"<__main__.Layer instance at 0x3863c20>\n"
]
}
],
"prompt_number": 162
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Multilayer Training"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For multilayer training, we need a slightly more complicated network object, performing three steps during training and prediction.\n",
"\n",
"- forward step: propagate the input vector through all the layers\n",
"- delta computation\n",
"- backward step: propagate the delta vector backwards through all the layers\n",
"- weight update for all the layers"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Network:\n",
" def __init__(self,layers):\n",
" self.layers = layers\n",
" self.eta = 0.1\n",
" def train(self,x,c):\n",
" target = zeros(self.layers[-1].noutput)\n",
" target[c] = 1\n",
" temp = x\n",
" # forward propagation\n",
" for l in self.layers:\n",
" temp = l.forward(temp)\n",
" # output delta\n",
" delta = temp-target\n",
" # backward propagation\n",
" for l in self.layers[::-1]:\n",
" delta = l.backward(delta)\n",
" # weight update\n",
" for l in self.layers:\n",
" l.update(self.eta)\n",
" def pred(self,x):\n",
" temp = x\n",
" for l in self.layers:\n",
" temp = l.forward(temp)\n",
" return argmax(temp)"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 163
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Single Layer Learning"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The new `Network` class should work for single layer training as well, so let's give that a try.\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"network = Network([Layer(2,2)])\n",
"\n",
"for i in range(50000):\n",
" index = randint(0,len(data))\n",
" network.train(data[index],labels[index])"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 164
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And the error rate is...\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pred = [network.pred(x) for x in data]\n",
"print 1*labels[:10]\n",
"print pred[:10]\n",
"sum(labels!=pred) * 1.0/len(labels)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"[1 0 1 0 1 1 1 0 0 0]\n",
"[1, 0, 1, 0, 1, 1, 1, 0, 0, 0]\n"
]
},
{
"output_type": "pyout",
"prompt_number": 165,
"text": [
"0.010999999999999999"
]
}
],
"prompt_number": 165
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Multilayer Learning"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now the real purpose of the `Network` class: multilayer training.\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"network = Network([Layer(2,4),Layer(4,2)])\n",
"\n",
"for i in range(50000):\n",
" index = randint(0,len(data))\n",
" network.train(data[index],labels[index])"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 166
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And the error rate...\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pred = [network.pred(x) for x in data]\n",
"print 1*labels[:10]\n",
"print pred[:10]\n",
"sum(labels!=pred) * 1.0/len(labels)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"[1 0 1 0 1 1 1 0 0 0]\n",
"[1, 0, 1, 0, 1, 1, 1, 0, 0, 0]\n"
]
},
{
"output_type": "pyout",
"prompt_number": 168,
"text": [
"0.001"
]
}
],
"prompt_number": 168
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Nonlinear Problem with Multilayer"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The multilayer network should be able to solve nonlinear classification problems.\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def make_nonlinear(n):data,labels = make_linear(1000)\n",
"\n",
" data = random_sample((n,2))\n",
" labels = ((data[:,0]-0.3) * (0.7-data[:,1]) > 0.0)\n",
" return data,labels\n",
"data,labels = make_nonlinear(1000)\n",
"plotdata(data,labels)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAFtCAYAAAD8jSoDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvX10VFWeLvygEaraECgTpssADhK6bYyRSU3jwLvWJFxn\ndWJ3ur3Xd3oItggKETDIl6s/RpJQaWm7+4W1XJdoWnoG14wt0z3Mxzu351bZEHsYKs4aEd6WNQgy\nfTXgCMRSzEQBJ0GD+/2jclLnnNr7nL332fucUzHPWixN1Tn765z67d9+fl+TCCEEE5jABCYwgXGF\na4IewAQmMIEJTEA9JoT7BCYwgQmMQ0wI9wlMYAITGIeYEO4TmMAEJjAOMSHcJzCBCUxgHGJCuE9g\nAhOYwDiEq3BftWoVPv/5z6OmpoZ5zcaNG/GFL3wBCxYswLFjx5QOcAITmMAEJiAOV+H+4IMPYv/+\n/czvX3jhBbz55pt444038Gd/9md4+OGHlQ5wAhOYwAQmIA5X4f6Hf/iHiMVizO//8R//EStXrgQA\n/MEf/AE++OADvPvuu+pGOIEJTGACExCGZ879/PnzmD179tjfs2bNwrlz57w2O4EJTGACE/CAEhWN\n2DMYTJo0qeAa2mcTmMAEJjABd8hkifGsuc+cORNnz54d+/vcuXOYOXMm9VpCCPVfJpXC1qoqEGDs\n39aqKmRSKeY9Yfxnn0cGwNJoFJtratDW0DA2n2QyGfhY7f9SqQwaGtpQX59EQ0MbqqvXmJ5G/l99\nPXvs9jZSqYxrv2FciyDWvqpqK4Dk2DpXVW3lWj+Rfw0NbdRn2tjYzvW9n/9434v82hHXtZN5P1X9\nq69PCv+ejH+y8Ky533333Xj66aexbNkyHD58GNOnT8fnP/95oTZ6urrwRF+f5bMn+vrQ8dRTqGtq\n8jpE32CeRy+AAwD2DQ0Br70GvPYa2ka/+4833kB7YyNKrlzByJQpaNi4MdB5ptO92LTpAPr6nhj7\nLBpdh9ws6izXRiJXudvo62sDADQ11VHvKWak073o6urBlSslmDJlBBs3NkjPs6urZ3TdOsc+6+t7\nAk891UFtU7bvK1foP/fh4Wu5vg8j8muXB23tgn4/p0wZoXzaixMnTmHJkk7P7xANrsL93nvvRSaT\nwfvvv4/Zs2fj+9//Pj755BMAwNq1a/G1r30NL7zwAubNm4frr78ef/EXfyE+iCtXqJ9fOzws3FaQ\nMM+jB8ATtu+f6OvDQ9u2oe/MGfzF4ODY54bQD0rA034gQ0O7EY02Y2go/7JVVW3Fhg13cbfhJKCK\nGaoFhYhQ9dI3XcDkN2y378MI3rUL+v3cuLEBfX1tpjH8BMBvMDCwD5mMMR61m42rcP/FL37h2sjT\nTz/taRAjU6ZQP78aiXhq12+Y58Fa2EtnzuBxk2AHgj+lsH4g8+ZVorKyA8PD1yISuYoNG+5ivniy\nWt+SJUuExhoGqBYUOaHaC6APOe19BEADVah66btQwFg3bLfvaVB5gjHDeC/c2ufdkII+lRhjfuqp\nDpw79x5OnXoPn376D5ZrVG82SgyqXtGwcSPa+vos1MzWqirctWFDgKMSh3ke9FcOmAJgCeXzIE8p\nrB9IZWUp9u/f7qkNN62vGIW7akGxeHElDh78OUZGnh/7rKRkHRYtul1p32YBQ9uw3b63QyfVsWTJ\nEq72eTck1vt56dJ5NDa2K9+caGhqqkNTUx0aG9tx8uSN1GtUbjahEO6Gxtrx1FO4dngYVyMR3LVh\nQ1Hx7YB1HhfOncO606exe2ho7PutVVUoLSsDbJo74O8pxa4NLV5cKayx2VH4I+tFNNqN/v5KNDa2\na/3R+A3V9MXLL/djZGS35bORkd04fLhDed+GgJH93gzdVAdP+7wbEm0TiMdXob9/Ol599Qem9vXz\n8LkNWj8FFgrhDuQEY7EJcxrM8+hNpws2LABo27QpsFMKSxtavnwmDh/m09hosB87T5+ehKGhfYYt\neVwZV2XoCyeIaOOq+2aBh27RTXXwts+zIdE2gffei+LYsSct1/nBw+c26AYAbTBb5qLRtdiw4T5l\n/QQm3HvTafR0dYXGY0QHnDasoE4pLG3o8OEObgqGBeux8weW7+w/GlVcrdHO+fMXkM1+gMrKSpSU\nXAIwBWVlM7QctUXpCzeIaOOq+6aBl25hjVuVB4jqE5J9E1iypJN6nQ4e3vy+X7yYRTz+v5DN/g8A\nHQCuRTR6Ct/9br3aTYX4BAAkk0qRtoYGsqa6mqyNRgkBxv5traoimVTKr+F8ZlFfnzQv+9i/+vqk\n9j5qajYTQghJpTKkqmqr5buqqq0klcoI9ZNvJ0MAoz3z/8u3rQupVIY0NLSR+vokaWhoI6lUhrEe\njwU25oaGNurza2xsL5iLfdwlJWtGn0GGAG0kEllBamtbx+ZCmz8LMusi0j7vPL2CNo94fBVJJFpJ\nfX2SNDa2O45TVkz7Kty3VlURApA22ooCpL2x0a/hfGbhxwvN6iMaXTr241Mxhnw75vb8+cHKwGlT\nSya7SXn5UjJt2kpSXr6UJJPdgY1TRAFIpTKksbGd1NcnSXn5UpNgL5xnMtktvKmb23cTgqJKA+/m\nIbJh0OD1fS8K4W7MKskQ7sn6er+G85mFH1piKpUh0eha2+N9jAD5H6qK00O+HXN7+k8msj921o88\nkWgl8fiq0Y0pSYA2Eo+v8k1zt8+ntrZVShjlnwd9njnhr2/jlRGibpuHilOm1/ddVrgHwrmz3ASL\nza9dF3TaI/zgbJua6jB37l/h5MkcnwhcBXAXgDoMDx9UxqXm2zG3p9cLwYv7H8tA+NvfnsNHH90G\nIG+nyGbbsG3b89oN0LT5xOOrEY8/imw2b2zkMdrmnwd9niMjUernfhtgzXAzxqrwCAoqOCwQ4V5o\nJy5Ov3YR8Ars3nQaB2zeNG2m/1ch9EXc3WQxc+YMnDxZaKDNbSZqPD7yniONyL9RhW+XSm8SLz92\n1o/8yhUCSjwzzpy518NI+UCbTzb7LBKJh7BggZgCkH8e9CSBJSVD1M9VCTkvQpRl4FfhEeSXh5Md\nvgr3tqoqPNHXN5atpDkaReW8eSitrCxKv3ZeOAls+5xZeXYe2rYNv/Phh1xthAFOL7Sq00O+nRdx\n7tz7ePfdZbjxxhtx3XWXAazH1KkzlJ9MrD/2XuQSTZTgyJE3kE73OvbDWpN3370ely/T7pisZMxO\nYAmvqVNnYv/+zoLPaTESL7/cP/b38uUz8b//9zG8/vrDGB5+Zuy+eHwLPve5EXz0kfVzlUJOVog6\nncacNgxejy8/TstUSJE5EgBy3jLtjY0kWV9P2hsbPzPeMW0NDdwG5GR9PfXa5lis6IzQIsawYkGe\nj5bzyqGtCYvjTiRatc/Hjac28/G1tatJPL7Fcl1JydrRtbCugXmeeZuCsW7tJBJZQRKJVuXvhMw7\nV7gGOU+fWCzn6ZMfe95GJWMcloWsmPbXoBoSGC6Zyfp60tbQoH2TYQlsmgGZtRGsYAj3z7IR2qsX\ngwxqa1ePCnV1XjmpVKZAaMbjm32Zj5OBvfA7+pyBdsc18MvlUBZWg2fhph2PbyGJRItlw/BzTrKy\nMzQRqn5BhCJRBZHEaKw8O2FIWxAmdHb+BDt2HMfQUD5s348o2LKyWQDuBPAs9XsZ42BTUx327LEf\n2+/xJZrXiTJobGy38fEscWGds30NeHhrXQnIeGClXgrzuWazT2LBgg4LTbVz50FqW6xMnkHM7TMn\n3IPIHS+SGI2VZwcINm2BG/x8gdPpXuzYkcHQ0D7L5/6FjtchJwQKoSp60k+w+i4Uykw/N8tf9jVw\nM3QGnWvdytXzGVB5jbfWueVsNL29z2L+/H3Yvr1Z6/w+c8KdJ3e8aldE0cRoYUxb4AS/f5xdXT0Y\nGppP/U53Cle6h04OfnhAiMDrhlsowAo9ka69dg2uXl0+9rexBtZw+0HE46uRzT5bcB0QfK518+nl\nyJE3aAfkAqHNa7zNz80o3/ME
}
],
"prompt_number": 169
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Training (with 10 hidden units).\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"network = Network([Layer(2,10),Layer(10,2)])\n",
"network.eta = 1.0\n",
"\n",
"for i in range(100000):\n",
" index = randint(0,len(data))\n",
" network.train(data[index],labels[index])"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 170
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Prediction.\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pred = [network.pred(x) for x in data]\n",
"print 1*labels[:10]\n",
"print pred[:10]\n",
"sum(labels!=pred) * 1.0/len(labels)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"[0 0 1 0 1 0 0 0 1 0]\n",
"[0, 0, 1, 1, 1, 0, 0, 0, 1, 0]\n"
]
},
{
"output_type": "pyout",
"prompt_number": 171,
"text": [
"0.025000000000000001"
]
}
],
"prompt_number": 171
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Defining a New Layer Type"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Training is now completely encapsulated in the `forward`, `backward` and `update` methods.\n",
"We can therefore easily define different layer types.\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"# new layer type with a different output transformation\n",
"class LinearLayer(Layer):\n",
" def forward(self,x):\n",
" # forward propagate x through the network\n",
" self.x[:] = x\n",
" y = dot(self.A,x)+self.a # UPDATED FUNCTION\n",
" self.y[:] = y\n",
" return y\n",
" def backward(self,delta):\n",
" # compute the pre-nonlinearity delta\n",
" delta_y = delta # UPDATED DERIV\n",
" # compute the weight updates\n",
" self.DA += outer(delta_y,self.x)\n",
" self.da += delta_y\n",
" # propagate the delta backwards\n",
" delta_x = dot(delta_y,self.A)\n",
" return delta_x"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 190
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's try single layer training.\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"data,labels = make_linear(1000)\n",
"\n",
"network = Network([LinearLayer(2,2)])\n",
"network.eta = 0.01\n",
"\n",
"for i in range(50000):\n",
" index = randint(0,len(data))\n",
" network.train(data[index],labels[index])"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 191
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pred = [network.pred(x) for x in data]\n",
"print 1*labels[:10]\n",
"print pred[:10]\n",
"sum(labels!=pred) * 1.0/len(labels)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"[1 1 0 1 0 0 0 1 1 0]\n",
"[1, 1, 0, 1, 0, 0, 0, 1, 1, 0]\n"
]
},
{
"output_type": "pyout",
"prompt_number": 192,
"text": [
"0.021000000000000001"
]
}
],
"prompt_number": 192
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And now let's try the new layer type in multilayer training.\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"network = Network([LinearLayer(2,4),Layer(4,2)])\n",
"network.eta = 0.01\n",
"\n",
"for i in range(50000):\n",
" index = randint(0,len(data))\n",
" network.train(data[index],labels[index])"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 193
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pred = [network.pred(x) for x in data]\n",
"print 1*labels[:10]\n",
"print pred[:10]\n",
"sum(labels!=pred) * 1.0/len(labels)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"[1 1 0 1 0 0 0 1 1 0]\n",
"[1, 1, 0, 1, 0, 0, 0, 1, 1, 0]\n"
]
},
{
"output_type": "pyout",
"prompt_number": 194,
"text": [
"0.014"
]
}
],
"prompt_number": 194
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that inserting a `LinearLayer` into an MLP doesn't actually do anything in this case; it can still only solve the same problem as a single logistic regression layer (why?).\n",
"\n",
"But a linear layer can be useful for dimensionality reduction, in a network like this:\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"network = Network([LinearLayer(1000,4),Layer(4,4),Layer(4,2)])"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 195
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Networks with Branches"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(layers and bundles)\n",
"\n",
"The above examples are still just layers stacked on top of one another.\n",
"There is a lot you can do with such network architectures.\n",
"\n",
"The most general case, however, is networks with branches. In such networks, you need a bit more abstraction than what is shown above. A good architecture is the following:\n",
"\n",
"- `Area` - holds activations and deltas\n",
"- `Bundle` - connects two layers, holds weights, propagates activations and deltas\n",
"- `Network` - collects bundles and layers and updates them in the right order\n",
"\n",
"What we used to call a `Layer` is now a combination of an `Area` and a `Bundle`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(branches)\n",
"\n",
"For a split (i.e., a `Area` used as an input to multiple `Bundle` instances), the bundles just propagate their deltas back and add them up in the layer.\n",
"\n",
"For a merge (i.e., multiple outputs turning into a single `Layer`), in simple cases, the outputs can also just add up.\n",
"\n",
"In the more general case, you can define a `Bundle` with multiple inputs and non-linear internal functions (e.g., multiplication)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(update order)\n",
"\n",
"Layers and bundles for a directed graph.\n",
"\n",
"The update order is given by a topological sort of that graph."
]
}
],
"metadata": {}
}
]
}