nvidia-AI.summerschool17-no.../teaching-lw/140-summary-classifiers.ipynb

1112 lines
162 KiB
Text
Raw Permalink Normal View History

2017-09-22 17:26:46 +00:00
{
"metadata": {
"name": "90-summary-classifiers"
},
"nbformat": 2,
"worksheets": [
{
"cells": [
{
"cell_type": "markdown",
"source": [
"Summary - Pattern Recognition",
"===============================",
"",
"This notebook is a summary of many of the techniques we have looked at in pattern recognition,",
"as well the interfaces and classes used to implement them in the `scikit.learn` package."
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"from pylab import *",
"from scipy.spatial import distance",
"import tables",
"import random as pyrandom"
],
"language": "python",
"outputs": [],
"prompt_number": 6
},
{
"cell_type": "markdown",
"source": [
"MNIST Training and Test Data",
"======================",
""
]
},
{
"cell_type": "markdown",
"source": [
"We have tried out a lot of these algorithms on the MNIST data set.",
"The MNIST data set has been widely used for experiments in pattern",
"recognition.",
"Being based on images, it's actually easy to see what the various",
"preprocessing methods do with the data."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"mnist = tables.openFile(\"mnist.h5\",\"r\")"
],
"language": "python",
"outputs": [],
"prompt_number": 7
},
{
"cell_type": "markdown",
"source": [
"Most algorithms work on 1D vectors, while input images are 2D.",
"Furthermore, training data sets are, by convention, matrices whose rows represent training data vectors.",
"The `make2d` function is useful for converting."
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"def make2d(data):",
" return data.reshape(len(data),prod(data.shape[1:]))"
],
"language": "python",
"outputs": [],
"prompt_number": 8
},
{
"cell_type": "markdown",
"source": [
"This is the training data..."
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"data = make2d(array(mnist.root.deskewed,'f'))",
"labels = array(mnist.root.labels)"
],
"language": "python",
"outputs": [],
"prompt_number": 9
},
{
"cell_type": "markdown",
"source": [
"And this is a separate test set..."
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"test_data = make2d(array(mnist.root.test_deskewed,'f'))",
"test_labels = array(mnist.root.test_labels)"
],
"language": "python",
"outputs": [],
"prompt_number": 10
},
{
"cell_type": "markdown",
"source": [
"k-Nearest Neighbors",
"==================="
]
},
{
"cell_type": "markdown",
"source": [
"One of the simplest and most intuitive algorithms for classification",
"is the nearest neighbor classifier.",
"You should always try it out to get some idea of how difficult",
"a classification problem is.",
"For large amounts of training data, the k-NN classifier can be slow,",
"but the implementation in SciKit has various optimizations that can",
"make it a lot faster than the implementation you created yourself."
]
},
{
"cell_type": "markdown",
"source": [
"Scikit classifiers generally start out as an object that is initialized with some parameters."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from sklearn.neighbors import KNeighborsClassifier",
"knn = KNeighborsClassifier(n_neighbors=1)"
],
"language": "python",
"outputs": [],
"prompt_number": 11
},
{
"cell_type": "markdown",
"source": [
"Training is performed with the `fit` method."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"knn.fit(data[:1000],labels[:1000])"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 16,
"text": [
"KNeighborsClassifier(algorithm='auto', leaf_size=30, n_neighbors=1,",
" warn_on_equidistant=True, weights='uniform')"
]
}
],
"prompt_number": 16
},
{
"cell_type": "markdown",
"source": [
"Classification is carried out with the `predict` method."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pred = knn.predict(test_data)",
"print pred[:10]",
"print test_labels[:10]"
],
"language": "python",
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"[7 2 1 0 4 1 9 4 9 9]",
"[7 2 1 0 4 1 4 9 5 9]"
]
}
],
"prompt_number": 13
},
{
"cell_type": "markdown",
"source": [
"In the past, we have evaluated predictions directly by comparing labels."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"sum(pred!=test_labels)*1.0/len(test_labels)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 17,
"text": [
"0.076100000000000001"
]
}
],
"prompt_number": 17
},
{
"cell_type": "markdown",
"source": [
"However, the package `sklearn.metrics` contains",
"various evaluation functions.",
"The `zero_one_score` function computes the fraction of _correct_ results."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from sklearn import metrics",
"metrics.zero_one_score(test_labels,pred)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 14,
"text": [
"0.92390000000000005"
]
}
],
"prompt_number": 14
},
{
"cell_type": "markdown",
"source": [
"Often, the error rate is more useful."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"1-metrics.zero_one_score(test_labels,pred)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 15,
"text": [
"0.076099999999999945"
]
}
],
"prompt_number": 15
},
{
"cell_type": "markdown",
"source": [
"Linear Discriminant Analysis",
"=============================="
]
},
{
"cell_type": "markdown",
"source": [
"Linear discriminant analysis (LDA) is one way of obtaining a _linear classifier_.",
"It is actually based on parametric models, however, not perceptron learning or other",
"_discriminative_ methods."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from sklearn import lda",
"lc = lda.LDA()"
],
"language": "python",
"outputs": [],
"prompt_number": 18
},
{
"cell_type": "markdown",
"source": [
"As before, the `fit` method does the training."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"lc.fit(data,labels)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 19,
"text": [
"LDA(n_components=None, priors=None)"
]
}
],
"prompt_number": 19
},
{
"cell_type": "markdown",
"source": [
"The `predict` method does the prediction, and we use the `zero_one_score` function to evaluate."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pred = lc.predict(test_data)",
"1-metrics.zero_one_score(test_labels,pred)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 123,
"text": [
"0.081999999999999962"
]
}
],
"prompt_number": 123
},
{
"cell_type": "markdown",
"source": [
"Preprocessing",
"============"
]
},
{
"cell_type": "markdown",
"source": [
"We had talked about various forms of preprocessing.",
"Often, feature vector normalization is important.",
"For example, we might transform $v$ into $v/||v||$.",
"",
"Some preprocessing methods need to be trained with a `fit` method,",
"just like a classifier,",
"but instead of a `pred` method, there is a `transform` method.",
"",
"The `fit_transform` method fits a model and transforms the input data at the same time."
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"from sklearn import preprocessing",
"normalizer = preprocessing.Normalizer(norm='l2')",
"normalized_data = normalizer.fit_transform(data)"
],
"language": "python",
"outputs": [],
"prompt_number": 124
},
{
"cell_type": "markdown",
"source": [
"Now we can fit a classifier in terms of the transformed data."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"lc = lda.LDA()",
"lc.fit(normalized_data,labels)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 125,
"text": [
"LDA(n_components=None, priors=None)"
]
}
],
"prompt_number": 125
},
{
"cell_type": "markdown",
"source": [
"We also need to remember to transform the test data prior to classification."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pred = lc.predict(normalizer.transform(test_data))",
"1-metrics.zero_one_score(test_labels,pred)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 126,
"text": [
"0.077500000000000013"
]
}
],
"prompt_number": 126
},
{
"cell_type": "markdown",
"source": [
"Preprocessing is important for achieving low error rates,",
"but many attempts at preprocessing make classification worse, not better.",
"Finding good preprocessing methods is tricky and requires both experience",
"and experimentation."
]
},
{
"cell_type": "markdown",
"source": [
"Feature Extraction",
"=================="
]
},
{
"cell_type": "markdown",
"source": [
"Feature extraction is a particular form of preprocessing.",
"There really isn't any clear distinction between feature extraction,",
"preprocessing, normalization, etc.",
"Normally, feature extraction aims at reducing the amount of data",
"and information that needs to be processed by a classifier,",
"and often it attempts to enhance features of the input that are",
"relevant to classification and discard parts of the input that",
"are noise or irrelevant."
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"from scipy.ndimage.filters import gaussian_filter",
"def extract(data):",
" return array([gaussian_filter(x.reshape(28,28),2.0).ravel() for x in data])"
],
"language": "python",
"outputs": [],
"prompt_number": 127
},
{
"cell_type": "markdown",
"source": [
"Feature extraction is often hard-coded by hand."
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"extracted = extract(data)"
],
"language": "python",
"outputs": [],
"prompt_number": 128
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"lc = lda.LDA()",
"lc.fit(extracted,labels)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 129,
"text": [
"LDA(n_components=None, priors=None)"
]
}
],
"prompt_number": 129
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pred = lc.predict(extract(test_data))",
"1-metrics.zero_one_score(test_labels,pred)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 130,
"text": [
"0.080600000000000005"
]
}
],
"prompt_number": 130
},
{
"cell_type": "markdown",
"source": [
"Confusion Matrix",
"==================="
]
},
{
"cell_type": "markdown",
"source": [
"We'd looked already at one function in the `metrics` package, but there",
"are other useful functions for evaluation and getting insight in how",
"classifiers work.",
"",
"The _confusion matrix_ tells us which kinds of classes are most often confused.",
"For example, '9' and '7' are confused fairly frequently, as are '8' and '3',",
"due to significant similarity.",
"",
"Based on confusions, we can often conclude what kind of preprocessing might",
"be helpful in reducing the error rate."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"metrics.confusion_matrix(test_labels,pred)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 131,
"text": [
"array([[ 951, 0, 0, 2, 1, 12, 3, 0, 9, 2],",
" [ 0, 1105, 2, 5, 1, 2, 4, 0, 16, 0],",
" [ 9, 0, 912, 20, 16, 6, 8, 3, 55, 3],",
" [ 2, 0, 5, 953, 1, 17, 1, 4, 16, 11],",
" [ 0, 4, 8, 1, 922, 5, 3, 2, 5, 32],",
" [ 5, 0, 0, 19, 16, 816, 13, 0, 18, 5],",
" [ 11, 3, 10, 1, 15, 39, 870, 0, 9, 0],",
" [ 2, 5, 15, 3, 16, 1, 0, 899, 10, 77],",
" [ 13, 1, 7, 19, 10, 28, 3, 2, 879, 12],",
" [ 5, 1, 3, 22, 40, 5, 0, 38, 8, 887]])"
]
}
],
"prompt_number": 131
},
{
"cell_type": "markdown",
"source": [
"Quadratic Discriminant Analysis",
"================================"
]
},
{
"cell_type": "markdown",
"source": [
"Quadratic Discriminant Analysis (QDA) is what we were looking",
"at under parametric classifiers in previous lectures."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from sklearn import qda",
"qc = qda.QDA()",
"qc.fit(data,labels)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 132,
"text": [
"QDA(priors=None)"
]
}
],
"prompt_number": 132
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pred = qc.predict(test_data)",
"1-metrics.zero_one_score(test_labels,pred)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 133,
"text": [
"0.21189999999999998"
]
}
],
"prompt_number": 133
},
{
"cell_type": "markdown",
"source": [
"Note that performance isn't very good, since this attempts to infer",
"a complete covariance matrix from not very much data.",
"",
"Again, this is different from learning a polynomial classifier discriminatively."
]
},
{
"cell_type": "markdown",
"source": [
"Support Vector Machines",
"=============="
]
},
{
"cell_type": "markdown",
"source": [
"Support vector machines are a popular machine learning technique.",
"You can think of them as combining ideas from perceptron learning with",
"non-linear transformations of the input space based on kernels.",
"As such, the method is quite versatile and can solve lots of problems.",
"",
"However, both training and classification with SVMs can be kind of slow, and",
"other methods are often significantly faster.",
"",
"Algorithmically, they are a bit more complicated than that, but",
"the numerical details are not all that important for using these methods."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from sklearn import svm",
"sv = svm.SVC()",
"sv.fit(data,labels)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 135,
"text": [
"SVC(C=1.0, cache_size=200, coef0=0.0, degree=3, gamma=0.00127551020408,",
" kernel='rbf', probability=False, scale_C=False, shrinking=True,",
" tol=0.001)"
]
}
],
"prompt_number": 135
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pred = sv.predict(test_data)",
"1-metrics.zero_one_score(test_labels,pred)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 136,
"text": [
"0.033599999999999963"
]
}
],
"prompt_number": 136
},
{
"cell_type": "markdown",
"source": [
"Note that performance is fairly good compared to the other methods.",
"There are a lot of parameters you can tune to improve them further."
]
},
{
"cell_type": "markdown",
"source": [
"Logistic Regression",
"==================="
]
},
{
"cell_type": "markdown",
"source": [
"Another linear classifier that we had mentioned briefly is _logistic regression_.",
"",
"It computes a linear decision boundary, but uses different methods from LDA or SVM."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from sklearn.linear_model import LogisticRegression",
"lr = LogisticRegression()",
"lr.fit(data,labels)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 137,
"text": [
"LogisticRegression(C=1.0, dual=False, fit_intercept=True, intercept_scaling=1,",
" penalty='l2', scale_C=False, tol=0.0001)"
]
}
],
"prompt_number": 137
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pred = lr.predict(test_data)",
"1-metrics.zero_one_score(test_labels,pred)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 138,
"text": [
"0.04930000000000001"
]
}
],
"prompt_number": 138
},
{
"cell_type": "markdown",
"source": [
"ROC Curve",
"========="
]
},
{
"cell_type": "markdown",
"source": [
"As shown above, classifiers just give a single error rate.",
"But often, there are different error rates we can achieve.",
"For a binary classifier, there is the _true positive rate_",
"(the percentage of true positives classified as such),",
"and the _false positive rate_ (the percentage of negative",
"samples classified as positive), and there",
"is a tradeoff between the two."
]
},
{
"cell_type": "markdown",
"source": [
"The basic idea is to use the value of a _discriminant function_ to make different tradeoffs.",
"By choosing different cutoff values, we can increase the true positive rate or decreate the",
"false positive rate.",
"For this, we need to ask the classifier to return a _degree of confidence_ or the value of",
"a discriminant function.",
"There are multiple candidates for this, including `predict_log_proba`."
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"scores = lr.predict_log_proba(test_data)"
],
"language": "python",
"outputs": [],
"prompt_number": 139
},
{
"cell_type": "markdown",
"source": [
"The function `metrics.roc_curve` will then generate an ROC curve. ",
"It only does this for binary classification tasks, so here we look at",
"0 vs. everything else."
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"fpr,tpr,thresholds = metrics.roc_curve(test_labels==0,scores[:,0])"
],
"language": "python",
"outputs": [],
"prompt_number": 140
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"plot(100*fpr[:1000],100*tpr[:1000])"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 141,
"text": [
"[<matplotlib.lines.Line2D at 0x99ca150>]"
]
},
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD9CAYAAAC/fMwDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEldJREFUeJzt3XtsU+Xjx/FPzZZovMCKrBM2szBZdlE25CYqWoOdUQOZ\niouYkIroH94xJooxRr7+IcVLoniJiRfSRKMSTcZUmJdoiQIyRCImiPMCAnOtl64iXrjM5/cHvxUR\ngZ7Ts55tz/uVLCldn54PD+yzJ09PTwPGGCMAwJB3nN8BAACFQeEDgCUofACwBIUPAJag8AHAEhQ+\nAFjiqIV//fXXKxQK6ayzzsrel06nFYlEVF1draamJmUymez3Fi1apLFjx6qmpkbvvvtu/6UGADh2\n1MKfO3eu2tvbD7kvFospEomos7NT06dPVywWkyRt3rxZr732mjZv3qz29nbdfPPN+vvvv/svOQDA\nkaMW/rRp01RSUnLIfW1tbYpGo5KkaDSq1tZWSdLy5cs1e/ZsFRcXq7KyUmeccYY6Ojr6KTYAwKki\npwNSqZRCoZAkKRQKKZVKSZJ++OEHnXPOOdnHlZeXq6ur65CxgUAgn6wAYC0vLoqQ14u2gUDgqCX+\nX98zxgy4rwceeMD3DGQik425yJTbl1ccF34oFFIymZQkdXd3q7S0VJI0evRo7dixI/u4nTt3avTo\n0R7FBADky3Hhz5w5U/F4XJIUj8fV3Nycvf/VV1/V3r17tXXrVn399deaPHmyt2kBAK4ddQ9/9uzZ\nWrVqlX7++WdVVFTowQcf1IIFC9TS0qIXXnhBlZWVWrZsmSSprq5OLS0tqqurU1FRkZ555plBs2cf\nDof9jnAYMuWGTLkbiLkGWqZgUOrpCet//+vf43i4S+NIwHi5QXSsgwUCnu5HAcA/HShs9+NLSqR0\n2rs8XvGqOx2fpQMA/yXfsvVCSYl/q+fBgMIHLFCIMqZsBz4KHxgEvNiqoIxB4QPHwFYFhgoKH1bL\npcwpWwwVXB4ZQ1owKAUCR/6SDpT50b4G4lkbgBsUPga1fAudModNOA8fg1ogwHYLhj6vupMVPga0\nY63g/3X1bgBHwQofAxoreIAVPgDAIQofACxB4aPfHGv/PZcv9ugB77CHj37D/jvgDfbw4TvOoAEG\nFwofR8SbmoChhS0dHBFbMsDAwJYO8pLLC6psyQBDCyt8S7F6BwYPVvg4Kl5QBfBvrPCHKFbwwNDB\nCh8A4AiFP0ixZQPAKbZ0Bim2bAB7sKUDAHCEwvcBFxUD4Ae2dHzAdgwAJ9jSAQA4QuEDgCUofACw\nBIUPAJag8AHAEhQ+AFiCwgcAS1D4/YDr3AAYiFwX/qJFi1RfX6+zzjpL1157rfbs2aN0Oq1IJKLq\n6mo1NTUpk8l4mXXQ6Onhs14BDDyuCn/btm167rnn9Nlnn+mLL75Qb2+vXn31VcViMUUiEXV2dmr6\n9OmKxWJe5x0QWMEDGIxcFf4pp5yi4uJi/fHHH9q/f7/++OMPjRo1Sm1tbYpGo5KkaDSq1tZWT8MO\nFKzgAQxGRW4GBYNB3XXXXTr99NN1wgkn6JJLLlEkElEqlVIoFJIkhUIhpVKpw8YuXLgwezscDisc\nDrsKDgBDVSKRUCKR8Px5XV087dtvv9WMGTP00UcfadiwYbr66qt11VVX6bbbblNPT0/2ccFgUOl/\nLHeHysXTuPgZgELy9eJpn376qc4991yNGDFCRUVFuvLKK7V27VqVlZUpmUxKkrq7u1VaWpp3QACA\nN1wVfk1NjT755BP9+eefMsbo/fffV11dnWbMmKF4PC5Jisfjam5u9jQsAMA919fDf/jhhxWPx3Xc\nccfp7LPP1vPPP6/ffvtNLS0t2r59uyorK7Vs2TINHz784MHY0gEAx7zqTj4AxQUKH0Ah8QEoAABH\nKHwAsASFDwCWoPABwBIUPgBYgsIHAEtQ+P+Bq2ECGIo4D/8/cJ49gIGE8/ABAI5Q+ABgCQofACxB\n4QOAJSh8ALAEhQ8AlqDwAcASFD4AWMK6wj/Wu2h5Jy2Aocq6d9ryLloAgw3vtAUAOELhA4AlKHwA\nsASFDwCWoPABwBIUPgBYgsIHAEtQ+ABgCQofACxB4QOAJSh8ALAEhQ8AlqDwAcASFD4AWILCBwBL\nUPgAYAkKHwAsQeEDgCVcF34mk9GsWbNUW1ururo6rVu3Tul0WpFIRNXV1WpqalImk/EyKwAgD64L\n/4477tBll12mL7/8Ups2bVJNTY1isZgikYg6Ozs1ffp0xWIxL7MCAPLg6kPMf/31V40fP17ffffd\nIffX1NRo1apVCoVCSiaTCofD2rJly8GD8SHmAOCYV91Z5GbQ1q1bNXLkSM2dO1eff/65JkyYoMcf\nf1ypVEqhUEiSFAqFlEqlDhu7cOHC7O1wOKxwOOwqOAAMVYlEQolEwvPndbXC//TTTzV16lStWbNG\nkyZN0vz583XyySfrqaeeUk9PT/ZxwWBQ6XT64MFY4QOAY151p6s9/PLycpWXl2vSpEmSpFmzZumz\nzz5TWVmZksmkJKm7u1ulpaV5BwQAeMNV4ZeVlamiokKdnZ2SpPfff1/19fWaMWOG4vG4JCkej6u5\nudm7pACAvLja0pGkzz//XDfccIP27t2rqqoqLV26VL29vWppadH27dtVWVmpZcuWafjw4QcPxpYO\nADjmVXe6LnxXB6PwAcAxX/fwAQCDD4UPAJYYcoUfDB7YtjnSV0mJ3wkBwB9Dbg+fPXoAQw17+AAA\nRyh8ALAEhQ8AlqDwAcASg67wOQsHANwZdGfpcBYOANtwlg4AwBEKHwAsQeEDgCUofACwBIUPAJag\n8AHAEhQ+AFiCwgcAS1D4AGAJCh8ALEHhA4AlKHwAsASFDwCWoPABwBIUPgBYgsIHAEtQ+ABgCQof\nACxB4QOAJSh8ALAEhQ8AlqDwAcASFD4AWILCBwBLUPgAYAkKHwAs4brwe3t7NX78eM2YMUOSlE6n\nFYlEVF1draamJmUyGc9CAgDy57rwn3jiCdXV1SkQCEiSYrGYIpGIOjs7NX36dMViMc9CAgDy56rw\nd+7cqRUrVuiGG26QMUaS1NbWpmg0KkmKRqNqbW31LiUAIG9FbgbdeeedeuSRR7Rr167sfalUSqFQ\nSJIUCoWUSqX+c+zChQuzt8PhsMLhsJsIADBkJRIJJRIJz583YPqW6Dl66623tHLlSj399NNKJBJ6\n7LHH9Oabb6qkpEQ9PT3ZxwWDQaXT6UMPFgjI4eEODxyQ8nwKABhUvOhOycUKf82aNWpra9OKFSv0\n119/adeuXZozZ45CoZCSyaTKysrU3d2t0tJSV4GCQekfvzcOU1Li6mkBwHqOV/j/tGrVKj366KN6\n8803dffdd2vEiBG65557FIvFlMlkDnvhNpffUqzgAeBQXq3w8z4Pv+8snQULFui9995TdXW1Pvjg\nAy1YsCDvcAAA7+S1wnd8MFb4AODYgFnhAwAGBwofACxB4QOAJSh8ALAEhQ8AlqDwAcASFD4AWILC\nBwBLUPgAYAkKHwAsQeEDgCUofACwBIUPAJag8AHAEhQ+AFiCwgcAS1D4AGAJCh8ALEHhA4AlKHwA\nsASFDwCWoPABwBIUPgBYgsIHAEsUFfqAgcDRv19SUpgcAGCbghe+MYU+IgBAYksHAKxB4QOAJSh8\nALAEhQ8AlqDwAcASFD4AWILCBwBLUPgAYAkKHwAs4arwd+zYoYsuukj19fU688wztWTJEklSOp1W\nJBJRdXW1mpqalMlkPA0LAHAvYIzzix0kk0klk0k1NjZq9+7dmjBhglpbW7V06VKdeuqpuvvuu7V4\n8WL19PQoFosdPFggIBeHAwCredWdrlb4ZWVlamxslCSddNJJqq2tVVdXl9ra2hSNRiVJ0WhUra2t\neQcEAHgj74unbdu2TRs3btSUKVOUSqUUCoUkSaFQSKlU6rDHL1y4MHs7HA4rHA7nGwEAhpREIqFE\nIuH587ra0umze/duXXjhhbr/
}
],
"prompt_number": 141
},
{
"cell_type": "markdown",
"source": [
"Cross Validation",
"================"
]
},
{
"cell_type": "markdown",
"source": [
"We have seen several times that evaluating a classifier on the training",
"set yields lower error rates than evaluating it on new data."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from sklearn import cross_validation",
"from sklearn import linear_model",
"",
"cl = linear_model.LogisticRegression()",
"cl.fit(data[:1000],labels[:1000])",
"pred = cl.predict(data[:1000])",
"print metrics.zero_one_score(labels[:1000],pred)"
],
"language": "python",
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"0.999"
]
}
],
"prompt_number": 142
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pred = cl.predict(test_data)",
"print metrics.zero_one_score(test_labels,pred)"
],
"language": "python",
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"0.9108"
]
}
],
"prompt_number": 143
},
{
"cell_type": "markdown",
"source": [
"If we don't have a separate test set, we can use _cross validation_ for estimating the",
"true error rate of a classifier.",
"",
"Cross-validation splits the training data itself into a separate test and training set,",
"but it does so multiple times and then reports the error rate for each split."
]
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"scores = cross_validation.cross_val_score(cl,data[:1000],labels[:1000],cv=10)"
],
"language": "python",
"outputs": [],
"prompt_number": 144
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"print scores",
"print mean(scores)"
],
"language": "python",
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"[ 0.94 0.94 0.94 0.93 0.93 0.95 0.93 0.97 0.9 0.9 ]",
"0.933"
]
}
],
"prompt_number": 145
},
{
"cell_type": "markdown",
"source": [
"Dimensionality Reduction via PCA",
"=================="
]
},
{
"cell_type": "markdown",
"source": [
"We have talked about manual preprocessing and feature extraction steps.",
"We also saw one important automatic feature extraction step.",
"",
"Remember that PCA is a linear transformation that reduces the dimensionality of the input,",
"while preserving distances as much as possible.",
"It is an example of _unsupervised learning_."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from sklearn import decomposition",
"pca = decomposition.PCA(n_components=50)",
"pca.fit(data)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 146,
"text": [
"PCA(copy=True, n_components=50, whiten=False)"
]
}
],
"prompt_number": 146
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from sklearn.neighbors import KNeighborsClassifier",
"knn = KNeighborsClassifier(n_neighbors=3)",
"knn.fit(pca.transform(data),labels)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 147,
"text": [
"KNeighborsClassifier(algorithm='auto', leaf_size=30, n_neighbors=3,",
" warn_on_equidistant=True, weights='uniform')"
]
}
],
"prompt_number": 147
},
{
"cell_type": "code",
"collapsed": true,
"input": [
"pred = knn.predict(pca.transform(test_data))"
],
"language": "python",
"outputs": [],
"prompt_number": 148
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"1-metrics.zero_one_score(test_labels,pred)"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 149,
"text": [
"0.01529999999999998"
]
}
],
"prompt_number": 149
},
{
"cell_type": "markdown",
"source": [
"Clustering with k-Means",
"======================="
]
},
{
"cell_type": "markdown",
"source": [
"A second form of unsupervised learning we looked at is $k$-means clustering.",
"",
"$k$-means clustering attempts to identify representative patters in the input data.",
"",
"$k$-means clustering has a number of applications:",
"",
"- we can use it to visualize the structure of the data",
"- we can use it to create \"pseudoclasses\" that can then be used with, say, a QDA (this gives a classifier somewhat similar to a kernel-based classifier)",
"- we can use it as kernel centers for kernel-based methods"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from sklearn import cluster",
"km = cluster.KMeans(40)",
"km.fit(data[::10])"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 150,
"text": [
"KMeans(copy_x=True, init='k-means++', k=40, max_iter=300, n_init=10,",
" precompute_distances=True,",
" random_state=<mtrand.RandomState object at 0x7fadbbe4a498>, tol=0.0001,",
" verbose=0)"
]
}
],
"prompt_number": 150
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"for i,v in enumerate(km.cluster_centers_):",
" subplot(5,8,i+1)",
" imshow(v.reshape(28,28))"
],
"language": "python",
"outputs": [
{
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD6CAYAAACBIq1uAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvXeUXNWV7/+5t3Ku7uqc1bmlbgVoZQkkASKDwGDAYAw4\nYOzxzBiPx3iejcf+zTD4zeBnz9hj+9nGxtnkjFCWUEY5Ialb6pxzha587++PU1ddkrvVQc28pbW0\n16pVt27d8L1777PP2fvss6+kqqrKFbpCV+gKXaHLhuT/1wCu0BW6QlfoCk2OrhjuK3SFrtAVuszo\niuG+QlfoCl2hy4yuGO4rdIWu0BW6zOiK4b5CV+gKXaHLjK4Y7it0ha7QFbrMaMqGe+3atVRWVlJW\nVsb3v//96cQ07XS5YL1ccMIVrB8XXS5YLxeccHlhnTCpU6BYLKaWlJSoDQ0NaiQSUefMmaOeOHFi\nKpf62OlywXq54FTVK1g/LrpcsF4uOFX18sI6GdJPxdjv3buX0tJSioqKALj//vt54403qKqqAkCS\npGnrWKaLLhesM2bMQFXVv8IJV7BeCl0u8ofLB+vlJP/LCas6gTWRUzLcbW1t5Ofnn/udl5fHnj17\nLjiqEChKbBclbW8BVoxx5bH+m+w5jYn9RUAPcGIcrKWJj7Zdlth+D7h5jPuO9d9k99cBawE70Au0\nXgQnTJ6vY+2fyjmvA24ET/uBznGwrk7aLmGEx+8DN15wrJTYvxpQEx8p8VkH3AbEgVhin0ajXas+\ncU4J0AUcPvfPxyd/JekZ7kjgjyf2X+xamvxLEfw8NA7W/4l2Ndb+xsR/NiYm/+LE58LtDcD1oxw/\n1v6pnPMSkIKQfx/QPg7Wm5K2J6IDa4FbR9n/9hjHj3WtZPlr1x2fpmS4J9ZDFTG2Io15ZUTYXWu4\n2r6x7icn/Z/cSxUl3f8EyYZ7dCplbGZPhJKnCibbe5chDE02cBLNcI9NRUyer9NFbkZ4egat4Y5N\nFxrUsUiToYzowEwI1TSD5AD9IWAWxIZBHQJCCAMeTpynS1xH05vSBL4bgSMkG+7RaSryT9Y7GcgA\npxP0B6B6JsgKdMehLQhDAwmMhsTxCiMGXZP/zQijfejCG11ARfy/k39R4pPBxORfzNiGeKIkcb5+\naJRsI0ajlMS9jwKn0Qz32DTV9p+MQUr6TrYDCmNTsvzhYzXcubm5tLS0nPvd0tJCXl7eVC6VoOQH\n1gx3PLGd/FEvOF5OOic+xjWd5+29dKwXkoYtWTjaaHEy5AYGzv2afpzTSU5g6NyvS8OqyU9GqGMh\nuNPBbkU2OZAMmSiB7WC7FrXXD94OiLQn7t+PMIhGRnQmzvmNyXXe3aaPrxLCEOsAC6TPwTnThRRd\nT8aXs/FZ7Hh3uRheG4aDRxMYjYlzI4xueNwfE9bppumW/2ik8UaXtO/CYxX+mocXDuJcTB/W5Hto\neqtjxA6B0GFDYlu94DN9NCXDXVtby8aNG6mqqsJkMnH69Gn2799/wVFFY5x94X7t4UG4gckNz4RQ\n9iqEexZjxCjqgCBQwF8bbRnR20tADsA4WEsZncbar/2nGZw44Evgy0n81nG+oo13rXxECAJqamqo\nr6+nrKyMb3zjGxccWzTGNSa7/1KulQM0ADB37twxeHoxKkna1gy2HVgJ0rXwSRndLRGMs0JYbV3E\nds8i5AwRfscObxVBvQcx6h8GKhG8DnN+aEK7h2ik0y//CoShzQXK4Ssubrr1FeY3beaBWS/xduYN\n/MH2MB90L4CDHmBWEjYdQlcuvE/+BLAWjYFnrP1TOWe8a+UgOs3xdLWY0Sl5f/KgTNuf7JFo27lA\nYJTzihL/a6NxXeK3dq1cxIh2qrp6oQ5ontMchLwKgdTE/YcQgy8X4AeijC7n8e4xPk3JcOv1ejwe\nD9FolGg0yre//e3zgv2CisY4ewbnuzxaw9UDsxECjCU+hsRnFqJRGjnnQmNFGMZShAH3IQQbTVyj\nkGSBXhxrGaPTaPu1nnZmAo9HbH86hnN+P0YlTO/6PNjZDwPtiPhaFChn7F5Xu889wM/x+/08/fTT\nfPOb3xzl2KIxrjGR/Rp2LRRxFWBBGM0wgoc+BE8vHLkmX0tOnBfA7/ePwdOLUVkCixYXtgP5kLkS\n1phZ9cD7zJ+1h1x3C2G9ib13LWD7nn46InkwKCWwRhKfosR3stGGkcYgBgXTJ3+NKgATeJywLJVv\nLXmGW0+9Q3lnA5Zrhtnpu4bTB6pgr+YJzETwdjS3WbvPRLAWjYFnrP0gjJg2MtS8Q4WRwYeOEYOk\nB9ITmDUea20q+T43A38cR1cnYrhJ3NuECAHZAA8YUyANEZVxqKKpaWYjirCL3ZKI1kQCwCDCcPoA\nb9I9dAhb4b9EXdUwZkFqNlTfCLcaWZW9g6uNRylUm7DEvbTqU9iOnSMHq+l61w1HmxFx9iBCZy4m\n/4nTlAw3gMViYc+ePXg8nkmeqTEhOcxhQfRaqYn/QgjmGxkZucYRjLMBdtDZIEUPJhUiIRjyQrQD\n1D6EAdeMv2DU1LCORZqyG8DmgJIy7lz5JhXpH6HzReh4JJt3leV075OhJ5x4lovFuTSaCcC+ffum\nESuMNEwjYjSQA5INjAZwmQEb+CMwPIBQsgFEIwgydmcj3MH6+vop4JEu+LjBmY1ttpWyh/dxb8pL\nLB3YgSkW5Eh6NVv6rie60wgHhmCoC9FqtZGM5oZenL/TL38ZdE5Scg0seGQdt59+hXkfHaXLlsPv\n5Af44I1FdL1tgLp2BD81XbzQO/y4sCa78ybAgQhxmBGWT9tvBqsObDqw6cGiF38Nx2EwBANDQAfC\nu9FGkAqasZm6rkoI8+MC0sCUDtkupLkGstNDzLD2kJbai8c1gMkQEb6rJmoZwnEj/V4XZ3pKaO7P\nYrA9D+q90OGHSAMiph1OYL0UXSXBKyuQC/mFZC9RmHnNKSoK6lnYu42igSM44i1ElTAOXTYSQ1Sm\nddB2UyWvVi2GjXboa0PowUTswPg0ZcMtSRLXX389Op2Oxx9/nM9//vMXHLElabuIkZ5aM9wGBEPM\nQCpIJWApBKMZ5GGI94JeBzoV9HHQK2ByIOvtyDozOoMOXYmC6oD4kJ7IST10NIK/GaKbgAMkuyoX\nx/pe0nbyjPJYlBS3SpXgTvh7xw9Z+N4WQi0yyr9bOH3ne/QGMlB6HIx0JKMZwTo0V06jqfF1NNIa\nrhmwgJQChmJIrcTgsmD2RLDmB1CRCHTYGW6SULu7IXwalEYE/6JJ12tMfEB0riJsNjpOEBkWGiVn\nlcBIyMsAeJBmpJKxapCHZr7AneveIUPp5kjJLLbqV7Jt2yp8fzLAh6eBU4nnkREdkSmBMTm2eSbx\nGeH39Mlf018zuqwUiuYO82T5/yb7uWb6s53suGcJ/3Dk+8R/NgTH6hHhrxAjHsyFcdmPQ/4aRi2u\n7gFDDliywWoDUxTJaEQ22DDIVgxpUfQZMXQZMfSeGJIM4T4jwTo9wSNh6D8O4R5Q9iAme0e8sYtj\n3ZC0nZxVosndDXIhOMowFGbhWd5L6ie6WVi4i5XKJiqVk5RFzuIMBJADjDgAVvCl2DiTUsD7KatZ\n672NgwdqGXolGzao0KuD+EFgfwLrRHR1NB3QdEoPkgt0FcjLPNQ8soUvVf2YO19YS+RdaO+HRp1E\nKxJBfzNze5opXvIK/Y/N5eSjv6eho5jghyqEhjk/dAKjyX8iNGXDvWPHDrKzs+np6eGGG26gsrKS\n5cuXJx2x4iK3NCOYYkSM/spAVwI1eiiWwGqFgVTwSMJdSlchE5ihYMoK4HT04tYN4dB5USWZvq50\nGneWw+tFsCcX2nMRcfEAYrS7mYMHD14E62RmlLVQTiJ045JhGbABTu+ErgaFFY9ESP/PDux1Nrzb\nDBe/HGWcbyjWjoN1xQRxyojGYUHMsKeCKReya+AxPWmrW6gpO8QC3R5i6Nnmv4Y9OxYT/5c8qB+G\nYc3tTDbcRYwYiquBH/Dee++N
}
],
"prompt_number": 151
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pseudoclasses = km.predict(data)",
"pseudoclasses[:10]"
],
"language": "python",
"outputs": [
{
"output_type": "pyout",
"prompt_number": 152,
"text": [
"array([11, 23, 24, 1, 2, 19, 1, 31, 32, 13])"
]
}
],
"prompt_number": 152
}
]
}
]
}