mirror of
https://github.com/saymrwulf/nvidia-AI.summerschool17-notebooks.git
synced 2026-11-19 09:50:53 +00:00
1112 lines
162 KiB
Text
1112 lines
162 KiB
Text
|
|
{
|
||
|
|
"metadata": {
|
||
|
|
"name": "90-summary-classifiers"
|
||
|
|
},
|
||
|
|
"nbformat": 2,
|
||
|
|
"worksheets": [
|
||
|
|
{
|
||
|
|
"cells": [
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Summary - Pattern Recognition",
|
||
|
|
"===============================",
|
||
|
|
"",
|
||
|
|
"This notebook is a summary of many of the techniques we have looked at in pattern recognition,",
|
||
|
|
"as well the interfaces and classes used to implement them in the `scikit.learn` package."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"from pylab import *",
|
||
|
|
"from scipy.spatial import distance",
|
||
|
|
"import tables",
|
||
|
|
"import random as pyrandom"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 6
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"MNIST Training and Test Data",
|
||
|
|
"======================",
|
||
|
|
""
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"We have tried out a lot of these algorithms on the MNIST data set.",
|
||
|
|
"The MNIST data set has been widely used for experiments in pattern",
|
||
|
|
"recognition.",
|
||
|
|
"Being based on images, it's actually easy to see what the various",
|
||
|
|
"preprocessing methods do with the data."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"mnist = tables.openFile(\"mnist.h5\",\"r\")"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 7
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Most algorithms work on 1D vectors, while input images are 2D.",
|
||
|
|
"Furthermore, training data sets are, by convention, matrices whose rows represent training data vectors.",
|
||
|
|
"The `make2d` function is useful for converting."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"def make2d(data):",
|
||
|
|
" return data.reshape(len(data),prod(data.shape[1:]))"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 8
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"This is the training data..."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"data = make2d(array(mnist.root.deskewed,'f'))",
|
||
|
|
"labels = array(mnist.root.labels)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 9
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"And this is a separate test set..."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"test_data = make2d(array(mnist.root.test_deskewed,'f'))",
|
||
|
|
"test_labels = array(mnist.root.test_labels)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 10
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"k-Nearest Neighbors",
|
||
|
|
"==================="
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"One of the simplest and most intuitive algorithms for classification",
|
||
|
|
"is the nearest neighbor classifier.",
|
||
|
|
"You should always try it out to get some idea of how difficult",
|
||
|
|
"a classification problem is.",
|
||
|
|
"For large amounts of training data, the k-NN classifier can be slow,",
|
||
|
|
"but the implementation in SciKit has various optimizations that can",
|
||
|
|
"make it a lot faster than the implementation you created yourself."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Scikit classifiers generally start out as an object that is initialized with some parameters."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"from sklearn.neighbors import KNeighborsClassifier",
|
||
|
|
"knn = KNeighborsClassifier(n_neighbors=1)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 11
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Training is performed with the `fit` method."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"knn.fit(data[:1000],labels[:1000])"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 16,
|
||
|
|
"text": [
|
||
|
|
"KNeighborsClassifier(algorithm='auto', leaf_size=30, n_neighbors=1,",
|
||
|
|
" warn_on_equidistant=True, weights='uniform')"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 16
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Classification is carried out with the `predict` method."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"pred = knn.predict(test_data)",
|
||
|
|
"print pred[:10]",
|
||
|
|
"print test_labels[:10]"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "stream",
|
||
|
|
"stream": "stdout",
|
||
|
|
"text": [
|
||
|
|
"[7 2 1 0 4 1 9 4 9 9]",
|
||
|
|
"[7 2 1 0 4 1 4 9 5 9]"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 13
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"In the past, we have evaluated predictions directly by comparing labels."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"sum(pred!=test_labels)*1.0/len(test_labels)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 17,
|
||
|
|
"text": [
|
||
|
|
"0.076100000000000001"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 17
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"However, the package `sklearn.metrics` contains",
|
||
|
|
"various evaluation functions.",
|
||
|
|
"The `zero_one_score` function computes the fraction of _correct_ results."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"from sklearn import metrics",
|
||
|
|
"metrics.zero_one_score(test_labels,pred)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 14,
|
||
|
|
"text": [
|
||
|
|
"0.92390000000000005"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 14
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Often, the error rate is more useful."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"1-metrics.zero_one_score(test_labels,pred)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 15,
|
||
|
|
"text": [
|
||
|
|
"0.076099999999999945"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 15
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Linear Discriminant Analysis",
|
||
|
|
"=============================="
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Linear discriminant analysis (LDA) is one way of obtaining a _linear classifier_.",
|
||
|
|
"It is actually based on parametric models, however, not perceptron learning or other",
|
||
|
|
"_discriminative_ methods."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"from sklearn import lda",
|
||
|
|
"lc = lda.LDA()"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 18
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"As before, the `fit` method does the training."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"lc.fit(data,labels)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 19,
|
||
|
|
"text": [
|
||
|
|
"LDA(n_components=None, priors=None)"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 19
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"The `predict` method does the prediction, and we use the `zero_one_score` function to evaluate."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"pred = lc.predict(test_data)",
|
||
|
|
"1-metrics.zero_one_score(test_labels,pred)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 123,
|
||
|
|
"text": [
|
||
|
|
"0.081999999999999962"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 123
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Preprocessing",
|
||
|
|
"============"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"We had talked about various forms of preprocessing.",
|
||
|
|
"Often, feature vector normalization is important.",
|
||
|
|
"For example, we might transform $v$ into $v/||v||$.",
|
||
|
|
"",
|
||
|
|
"Some preprocessing methods need to be trained with a `fit` method,",
|
||
|
|
"just like a classifier,",
|
||
|
|
"but instead of a `pred` method, there is a `transform` method.",
|
||
|
|
"",
|
||
|
|
"The `fit_transform` method fits a model and transforms the input data at the same time."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"from sklearn import preprocessing",
|
||
|
|
"normalizer = preprocessing.Normalizer(norm='l2')",
|
||
|
|
"normalized_data = normalizer.fit_transform(data)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 124
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Now we can fit a classifier in terms of the transformed data."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"lc = lda.LDA()",
|
||
|
|
"lc.fit(normalized_data,labels)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 125,
|
||
|
|
"text": [
|
||
|
|
"LDA(n_components=None, priors=None)"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 125
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"We also need to remember to transform the test data prior to classification."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"pred = lc.predict(normalizer.transform(test_data))",
|
||
|
|
"1-metrics.zero_one_score(test_labels,pred)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 126,
|
||
|
|
"text": [
|
||
|
|
"0.077500000000000013"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 126
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Preprocessing is important for achieving low error rates,",
|
||
|
|
"but many attempts at preprocessing make classification worse, not better.",
|
||
|
|
"Finding good preprocessing methods is tricky and requires both experience",
|
||
|
|
"and experimentation."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Feature Extraction",
|
||
|
|
"=================="
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Feature extraction is a particular form of preprocessing.",
|
||
|
|
"There really isn't any clear distinction between feature extraction,",
|
||
|
|
"preprocessing, normalization, etc.",
|
||
|
|
"Normally, feature extraction aims at reducing the amount of data",
|
||
|
|
"and information that needs to be processed by a classifier,",
|
||
|
|
"and often it attempts to enhance features of the input that are",
|
||
|
|
"relevant to classification and discard parts of the input that",
|
||
|
|
"are noise or irrelevant."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"from scipy.ndimage.filters import gaussian_filter",
|
||
|
|
"def extract(data):",
|
||
|
|
" return array([gaussian_filter(x.reshape(28,28),2.0).ravel() for x in data])"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 127
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Feature extraction is often hard-coded by hand."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"extracted = extract(data)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 128
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"lc = lda.LDA()",
|
||
|
|
"lc.fit(extracted,labels)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 129,
|
||
|
|
"text": [
|
||
|
|
"LDA(n_components=None, priors=None)"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 129
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"pred = lc.predict(extract(test_data))",
|
||
|
|
"1-metrics.zero_one_score(test_labels,pred)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 130,
|
||
|
|
"text": [
|
||
|
|
"0.080600000000000005"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 130
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Confusion Matrix",
|
||
|
|
"==================="
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"We'd looked already at one function in the `metrics` package, but there",
|
||
|
|
"are other useful functions for evaluation and getting insight in how",
|
||
|
|
"classifiers work.",
|
||
|
|
"",
|
||
|
|
"The _confusion matrix_ tells us which kinds of classes are most often confused.",
|
||
|
|
"For example, '9' and '7' are confused fairly frequently, as are '8' and '3',",
|
||
|
|
"due to significant similarity.",
|
||
|
|
"",
|
||
|
|
"Based on confusions, we can often conclude what kind of preprocessing might",
|
||
|
|
"be helpful in reducing the error rate."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"metrics.confusion_matrix(test_labels,pred)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 131,
|
||
|
|
"text": [
|
||
|
|
"array([[ 951, 0, 0, 2, 1, 12, 3, 0, 9, 2],",
|
||
|
|
" [ 0, 1105, 2, 5, 1, 2, 4, 0, 16, 0],",
|
||
|
|
" [ 9, 0, 912, 20, 16, 6, 8, 3, 55, 3],",
|
||
|
|
" [ 2, 0, 5, 953, 1, 17, 1, 4, 16, 11],",
|
||
|
|
" [ 0, 4, 8, 1, 922, 5, 3, 2, 5, 32],",
|
||
|
|
" [ 5, 0, 0, 19, 16, 816, 13, 0, 18, 5],",
|
||
|
|
" [ 11, 3, 10, 1, 15, 39, 870, 0, 9, 0],",
|
||
|
|
" [ 2, 5, 15, 3, 16, 1, 0, 899, 10, 77],",
|
||
|
|
" [ 13, 1, 7, 19, 10, 28, 3, 2, 879, 12],",
|
||
|
|
" [ 5, 1, 3, 22, 40, 5, 0, 38, 8, 887]])"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 131
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Quadratic Discriminant Analysis",
|
||
|
|
"================================"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Quadratic Discriminant Analysis (QDA) is what we were looking",
|
||
|
|
"at under parametric classifiers in previous lectures."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"from sklearn import qda",
|
||
|
|
"qc = qda.QDA()",
|
||
|
|
"qc.fit(data,labels)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 132,
|
||
|
|
"text": [
|
||
|
|
"QDA(priors=None)"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 132
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"pred = qc.predict(test_data)",
|
||
|
|
"1-metrics.zero_one_score(test_labels,pred)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 133,
|
||
|
|
"text": [
|
||
|
|
"0.21189999999999998"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 133
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Note that performance isn't very good, since this attempts to infer",
|
||
|
|
"a complete covariance matrix from not very much data.",
|
||
|
|
"",
|
||
|
|
"Again, this is different from learning a polynomial classifier discriminatively."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Support Vector Machines",
|
||
|
|
"=============="
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Support vector machines are a popular machine learning technique.",
|
||
|
|
"You can think of them as combining ideas from perceptron learning with",
|
||
|
|
"non-linear transformations of the input space based on kernels.",
|
||
|
|
"As such, the method is quite versatile and can solve lots of problems.",
|
||
|
|
"",
|
||
|
|
"However, both training and classification with SVMs can be kind of slow, and",
|
||
|
|
"other methods are often significantly faster.",
|
||
|
|
"",
|
||
|
|
"Algorithmically, they are a bit more complicated than that, but",
|
||
|
|
"the numerical details are not all that important for using these methods."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"from sklearn import svm",
|
||
|
|
"sv = svm.SVC()",
|
||
|
|
"sv.fit(data,labels)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 135,
|
||
|
|
"text": [
|
||
|
|
"SVC(C=1.0, cache_size=200, coef0=0.0, degree=3, gamma=0.00127551020408,",
|
||
|
|
" kernel='rbf', probability=False, scale_C=False, shrinking=True,",
|
||
|
|
" tol=0.001)"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 135
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"pred = sv.predict(test_data)",
|
||
|
|
"1-metrics.zero_one_score(test_labels,pred)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 136,
|
||
|
|
"text": [
|
||
|
|
"0.033599999999999963"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 136
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Note that performance is fairly good compared to the other methods.",
|
||
|
|
"There are a lot of parameters you can tune to improve them further."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Logistic Regression",
|
||
|
|
"==================="
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Another linear classifier that we had mentioned briefly is _logistic regression_.",
|
||
|
|
"",
|
||
|
|
"It computes a linear decision boundary, but uses different methods from LDA or SVM."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"from sklearn.linear_model import LogisticRegression",
|
||
|
|
"lr = LogisticRegression()",
|
||
|
|
"lr.fit(data,labels)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 137,
|
||
|
|
"text": [
|
||
|
|
"LogisticRegression(C=1.0, dual=False, fit_intercept=True, intercept_scaling=1,",
|
||
|
|
" penalty='l2', scale_C=False, tol=0.0001)"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 137
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"pred = lr.predict(test_data)",
|
||
|
|
"1-metrics.zero_one_score(test_labels,pred)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 138,
|
||
|
|
"text": [
|
||
|
|
"0.04930000000000001"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 138
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"ROC Curve",
|
||
|
|
"========="
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"As shown above, classifiers just give a single error rate.",
|
||
|
|
"But often, there are different error rates we can achieve.",
|
||
|
|
"For a binary classifier, there is the _true positive rate_",
|
||
|
|
"(the percentage of true positives classified as such),",
|
||
|
|
"and the _false positive rate_ (the percentage of negative",
|
||
|
|
"samples classified as positive), and there",
|
||
|
|
"is a tradeoff between the two."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"The basic idea is to use the value of a _discriminant function_ to make different tradeoffs.",
|
||
|
|
"By choosing different cutoff values, we can increase the true positive rate or decreate the",
|
||
|
|
"false positive rate.",
|
||
|
|
"For this, we need to ask the classifier to return a _degree of confidence_ or the value of",
|
||
|
|
"a discriminant function.",
|
||
|
|
"There are multiple candidates for this, including `predict_log_proba`."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"scores = lr.predict_log_proba(test_data)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 139
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"The function `metrics.roc_curve` will then generate an ROC curve. ",
|
||
|
|
"It only does this for binary classification tasks, so here we look at",
|
||
|
|
"0 vs. everything else."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"fpr,tpr,thresholds = metrics.roc_curve(test_labels==0,scores[:,0])"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 140
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"plot(100*fpr[:1000],100*tpr[:1000])"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 141,
|
||
|
|
"text": [
|
||
|
|
"[<matplotlib.lines.Line2D at 0x99ca150>]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD9CAYAAAC/fMwDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEldJREFUeJzt3XtsU+Xjx/FPzZZovMCKrBM2szBZdlE25CYqWoOdUQOZ\niouYkIroH94xJooxRr7+IcVLoniJiRfSRKMSTcZUmJdoiQIyRCImiPMCAnOtl64iXrjM5/cHvxUR\ngZ7Ts55tz/uVLCldn54PD+yzJ09PTwPGGCMAwJB3nN8BAACFQeEDgCUofACwBIUPAJag8AHAEhQ+\nAFjiqIV//fXXKxQK6ayzzsrel06nFYlEVF1draamJmUymez3Fi1apLFjx6qmpkbvvvtu/6UGADh2\n1MKfO3eu2tvbD7kvFospEomos7NT06dPVywWkyRt3rxZr732mjZv3qz29nbdfPPN+vvvv/svOQDA\nkaMW/rRp01RSUnLIfW1tbYpGo5KkaDSq1tZWSdLy5cs1e/ZsFRcXq7KyUmeccYY6Ojr6KTYAwKki\npwNSqZRCoZAkKRQKKZVKSZJ++OEHnXPOOdnHlZeXq6ur65CxgUAgn6wAYC0vLoqQ14u2gUDgqCX+\nX98zxgy4rwceeMD3DGQik425yJTbl1ccF34oFFIymZQkdXd3q7S0VJI0evRo7dixI/u4nTt3avTo\n0R7FBADky3Hhz5w5U/F4XJIUj8fV3Nycvf/VV1/V3r17tXXrVn399deaPHmyt2kBAK4ddQ9/9uzZ\nWrVqlX7++WdVVFTowQcf1IIFC9TS0qIXXnhBlZWVWrZsmSSprq5OLS0tqqurU1FRkZ555plBs2cf\nDof9jnAYMuWGTLkbiLkGWqZgUOrpCet//+vf43i4S+NIwHi5QXSsgwUCnu5HAcA/HShs9+NLSqR0\n2rs8XvGqOx2fpQMA/yXfsvVCSYl/q+fBgMIHLFCIMqZsBz4KHxgEvNiqoIxB4QPHwFYFhgoKH1bL\npcwpWwwVXB4ZQ1owKAUCR/6SDpT50b4G4lkbgBsUPga1fAudModNOA8fg1ogwHYLhj6vupMVPga0\nY63g/3X1bgBHwQofAxoreIAVPgDAIQofACxB4aPfHGv/PZcv9ugB77CHj37D/jvgDfbw4TvOoAEG\nFwofR8SbmoChhS0dHBFbMsDAwJYO8pLLC6psyQBDCyt8S7F6BwYPVvg4Kl5QBfBvrPCHKFbwwNDB\nCh8A4AiFP0ixZQPAKbZ0Bim2bAB7sKUDAHCEwvcBFxUD4Ae2dHzAdgwAJ9jSAQA4QuEDgCUofACw\nBIUPAJag8AHAEhQ+AFiCwgcAS1D4/YDr3AAYiFwX/qJFi1RfX6+zzjpL1157rfbs2aN0Oq1IJKLq\n6mo1NTUpk8l4mXXQ6Onhs14BDDyuCn/btm167rnn9Nlnn+mLL75Qb2+vXn31VcViMUUiEXV2dmr6\n9OmKxWJe5x0QWMEDGIxcFf4pp5yi4uJi/fHHH9q/f7/++OMPjRo1Sm1tbYpGo5KkaDSq1tZWT8MO\nFKzgAQxGRW4GBYNB3XXXXTr99NN1wgkn6JJLLlEkElEqlVIoFJIkhUIhpVKpw8YuXLgwezscDisc\nDrsKDgBDVSKRUCKR8Px5XV087dtvv9WMGTP00UcfadiwYbr66qt11VVX6bbbblNPT0/2ccFgUOl/\nLHeHysXTuPgZgELy9eJpn376qc4991yNGDFCRUVFuvLKK7V27VqVlZUpmUxKkrq7u1VaWpp3QACA\nN1wVfk1NjT755BP9+eefMsbo/fffV11dnWbMmKF4PC5Jisfjam5u9jQsAMA919fDf/jhhxWPx3Xc\nccfp7LPP1vPPP6/ffvtNLS0t2r59uyorK7Vs2TINHz784MHY0gEAx7zqTj4AxQUKH0Ah8QEoAABH\nKHwAsASFDwCWoPABwBIUPgBYgsIHAEtQ+P+Bq2ECGIo4D/8/cJ49gIGE8/ABAI5Q+ABgCQofACxB\n4QOAJSh8ALAEhQ8AlqDwAcASFD4AWMK6wj/Wu2h5Jy2Aocq6d9ryLloAgw3vtAUAOELhA4AlKHwA\nsASFDwCWoPABwBIUPgBYgsIHAEtQ+ABgCQofACxB4QOAJSh8ALAEhQ8AlqDwAcASFD4AWILCBwBL\nUPgAYAkKHwAsQeEDgCVcF34mk9GsWbNUW1ururo6rVu3Tul0WpFIRNXV1WpqalImk/EyKwAgD64L\n/4477tBll12mL7/8Ups2bVJNTY1isZgikYg6Ozs1ffp0xWIxL7MCAPLg6kPMf/31V40fP17ffffd\nIffX1NRo1apVCoVCSiaTCofD2rJly8GD8SHmAOCYV91Z5GbQ1q1bNXLkSM2dO1eff/65JkyYoMcf\nf1ypVEqhUEiSFAqFlEqlDhu7cOHC7O1wOKxwOOwqOAAMVYlEQolEwvPndbXC//TTTzV16lStWbNG\nkyZN0vz583XyySfrqaeeUk9PT/ZxwWBQ6XT64MFY4QOAY151p6s9/PLycpWXl2vSpEmSpFmzZumz\nzz5TWVmZksmkJKm7u1ulpaV5BwQAeMNV4ZeVlamiokKdnZ2SpPfff1/19fWaMWOG4vG4JCkej6u5\nudm7pACAvLja0pGkzz//XDfccIP27t2rqqoqLV26VL29vWppadH27dtVWVmpZcuWafjw4QcPxpYO\nADjmVXe6LnxXB6PwAcAxX/fwAQCDD4UPAJYYcoUfDB7YtjnSV0mJ3wkBwB9Dbg+fPXoAQw17+AAA\nRyh8ALAEhQ8AlqDwAcASg67wOQsHANwZdGfpcBYOANtwlg4AwBEKHwAsQeEDgCUofACwBIUPAJag\n8AHAEhQ+AFiCwgcAS1D4AGAJCh8ALEHhA4AlKHwAsASFDwCWoPABwBIUPgBYgsIHAEtQ+ABgCQof\nACxB4QOAJSh8ALAEhQ8AlqDwAcASFD4AWILCBwBLUPgAYAkKHwAs4brwe3t7NX78eM2YMUOSlE6n\nFYlEVF1draamJmUyGc9CAgDy57rwn3jiCdXV1SkQCEiSYrGYIpGIOjs7NX36dMViMc9CAgDy56rw\nd+7cqRUrVuiGG26QMUaS1NbWpmg0KkmKRqNqbW31LiUAIG9FbgbdeeedeuSRR7Rr167sfalUSqFQ\nSJIUCoWUSqX+c+zChQuzt8PhsMLhsJsIADBkJRIJJRIJz583YPqW6Dl66623tHLlSj399NNKJBJ6\n7LHH9Oabb6qkpEQ9PT3ZxwWDQaXT6UMPFgjI4eEODxyQ8nwKABhUvOhOycUKf82aNWpra9OKFSv0\n119/adeuXZozZ45CoZCSyaTKysrU3d2t0tJSV4GCQekfvzcOU1Li6mkBwHqOV/j/tGrVKj366KN6\n8803dffdd2vEiBG65557FIvFlMlkDnvhNpffUqzgAeBQXq3w8z4Pv+8snQULFui9995TdXW1Pvjg\nAy1YsCDvcAAA7+S1wnd8MFb4AODYgFnhAwAGBwofACxB4QOAJSh8ALAEhQ8AlqDwAcASFD4AWILC\nBwBLUPgAYAkKHwAsQeEDgCUofACwBIUPAJag8AHAEhQ+AFiCwgcAS1D4AGAJCh8ALEHhA4AlKHwA\nsASFDwCWoPABwBIUPgBYgsIHAEsUFfqAgcDRv19SUpgcAGCbghe+MYU+IgBAYksHAKxB4QOAJSh8\nALAEhQ8AlqDwAcASFD4AWILCBwBLUPgAYAkKHwAs4arwd+zYoYsuukj19fU688wztWTJEklSOp1W\nJBJRdXW1mpqalMlkPA0LAHAvYIzzix0kk0klk0k1NjZq9+7dmjBhglpbW7V06VKdeuqpuvvuu7V4\n8WL19PQoFosdPFggIBeHAwCredWdrlb4ZWVlamxslCSddNJJqq2tVVdXl9ra2hSNRiVJ0WhUra2t\neQcEAHgj74unbdu2TRs3btSUKVOUSqUUCoUkSaFQSKlU6rDHL1y4MHs7HA4rHA7nGwEAhpREIqFE\nIuH587ra0umze/duXXjhhbr/
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 141
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Cross Validation",
|
||
|
|
"================"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"We have seen several times that evaluating a classifier on the training",
|
||
|
|
"set yields lower error rates than evaluating it on new data."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"from sklearn import cross_validation",
|
||
|
|
"from sklearn import linear_model",
|
||
|
|
"",
|
||
|
|
"cl = linear_model.LogisticRegression()",
|
||
|
|
"cl.fit(data[:1000],labels[:1000])",
|
||
|
|
"pred = cl.predict(data[:1000])",
|
||
|
|
"print metrics.zero_one_score(labels[:1000],pred)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "stream",
|
||
|
|
"stream": "stdout",
|
||
|
|
"text": [
|
||
|
|
"0.999"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 142
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"pred = cl.predict(test_data)",
|
||
|
|
"print metrics.zero_one_score(test_labels,pred)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "stream",
|
||
|
|
"stream": "stdout",
|
||
|
|
"text": [
|
||
|
|
"0.9108"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 143
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"If we don't have a separate test set, we can use _cross validation_ for estimating the",
|
||
|
|
"true error rate of a classifier.",
|
||
|
|
"",
|
||
|
|
"Cross-validation splits the training data itself into a separate test and training set,",
|
||
|
|
"but it does so multiple times and then reports the error rate for each split."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"scores = cross_validation.cross_val_score(cl,data[:1000],labels[:1000],cv=10)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 144
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"print scores",
|
||
|
|
"print mean(scores)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "stream",
|
||
|
|
"stream": "stdout",
|
||
|
|
"text": [
|
||
|
|
"[ 0.94 0.94 0.94 0.93 0.93 0.95 0.93 0.97 0.9 0.9 ]",
|
||
|
|
"0.933"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 145
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Dimensionality Reduction via PCA",
|
||
|
|
"=================="
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"We have talked about manual preprocessing and feature extraction steps.",
|
||
|
|
"We also saw one important automatic feature extraction step.",
|
||
|
|
"",
|
||
|
|
"Remember that PCA is a linear transformation that reduces the dimensionality of the input,",
|
||
|
|
"while preserving distances as much as possible.",
|
||
|
|
"It is an example of _unsupervised learning_."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"from sklearn import decomposition",
|
||
|
|
"pca = decomposition.PCA(n_components=50)",
|
||
|
|
"pca.fit(data)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 146,
|
||
|
|
"text": [
|
||
|
|
"PCA(copy=True, n_components=50, whiten=False)"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 146
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"from sklearn.neighbors import KNeighborsClassifier",
|
||
|
|
"knn = KNeighborsClassifier(n_neighbors=3)",
|
||
|
|
"knn.fit(pca.transform(data),labels)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 147,
|
||
|
|
"text": [
|
||
|
|
"KNeighborsClassifier(algorithm='auto', leaf_size=30, n_neighbors=3,",
|
||
|
|
" warn_on_equidistant=True, weights='uniform')"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 147
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": true,
|
||
|
|
"input": [
|
||
|
|
"pred = knn.predict(pca.transform(test_data))"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [],
|
||
|
|
"prompt_number": 148
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"1-metrics.zero_one_score(test_labels,pred)"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 149,
|
||
|
|
"text": [
|
||
|
|
"0.01529999999999998"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 149
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"Clustering with k-Means",
|
||
|
|
"======================="
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"source": [
|
||
|
|
"A second form of unsupervised learning we looked at is $k$-means clustering.",
|
||
|
|
"",
|
||
|
|
"$k$-means clustering attempts to identify representative patters in the input data.",
|
||
|
|
"",
|
||
|
|
"$k$-means clustering has a number of applications:",
|
||
|
|
"",
|
||
|
|
"- we can use it to visualize the structure of the data",
|
||
|
|
"- we can use it to create \"pseudoclasses\" that can then be used with, say, a QDA (this gives a classifier somewhat similar to a kernel-based classifier)",
|
||
|
|
"- we can use it as kernel centers for kernel-based methods"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"from sklearn import cluster",
|
||
|
|
"km = cluster.KMeans(40)",
|
||
|
|
"km.fit(data[::10])"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 150,
|
||
|
|
"text": [
|
||
|
|
"KMeans(copy_x=True, init='k-means++', k=40, max_iter=300, n_init=10,",
|
||
|
|
" precompute_distances=True,",
|
||
|
|
" random_state=<mtrand.RandomState object at 0x7fadbbe4a498>, tol=0.0001,",
|
||
|
|
" verbose=0)"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 150
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"for i,v in enumerate(km.cluster_centers_):",
|
||
|
|
" subplot(5,8,i+1)",
|
||
|
|
" imshow(v.reshape(28,28))"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "display_data",
|
||
|
|
"png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD6CAYAAACBIq1uAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvXeUXNWV7/+5t3Ku7uqc1bmlbgVoZQkkASKDwGDAYAw4\nYOzxzBiPx3iejcf+zTD4zeBnz9hj+9nGxtnkjFCWUEY5Ialb6pxzha587++PU1ddkrvVQc28pbW0\n16pVt27d8L1777PP2fvss6+kqqrKFbpCV+gKXaHLhuT/1wCu0BW6QlfoCk2OrhjuK3SFrtAVuszo\niuG+QlfoCl2hy4yuGO4rdIWu0BW6zOiK4b5CV+gKXaHLjK4Y7it0ha7QFbrMaMqGe+3atVRWVlJW\nVsb3v//96cQ07XS5YL1ccMIVrB8XXS5YLxeccHlhnTCpU6BYLKaWlJSoDQ0NaiQSUefMmaOeOHFi\nKpf62OlywXq54FTVK1g/LrpcsF4uOFX18sI6GdJPxdjv3buX0tJSioqKALj//vt54403qKqqAkCS\npGnrWKaLLhesM2bMQFXVv8IJV7BeCl0u8ofLB+vlJP/LCas6gTWRUzLcbW1t5Ofnn/udl5fHnj17\nLjiqEChKbBclbW8BVoxx5bH+m+w5jYn9RUAPcGIcrKWJj7Zdlth+D7h5jPuO9d9k99cBawE70Au0\nXgQnTJ6vY+2fyjmvA24ET/uBznGwrk7aLmGEx+8DN15wrJTYvxpQEx8p8VkH3AbEgVhin0ajXas+\ncU4J0AUcPvfPxyd/JekZ7kjgjyf2X+xamvxLEfw8NA7W/4l2Ndb+xsR/NiYm/+LE58LtDcD1oxw/\n1v6pnPMSkIKQfx/QPg7Wm5K2J6IDa4FbR9n/9hjHj3WtZPlr1x2fpmS4J9ZDFTG2Io15ZUTYXWu4\n2r6x7icn/Z/cSxUl3f8EyYZ7dCplbGZPhJKnCibbe5chDE02cBLNcI9NRUyer9NFbkZ4egat4Y5N\nFxrUsUiToYzowEwI1TSD5AD9IWAWxIZBHQJCCAMeTpynS1xH05vSBL4bgSMkG+7RaSryT9Y7GcgA\npxP0B6B6JsgKdMehLQhDAwmMhsTxCiMGXZP/zQijfejCG11ARfy/k39R4pPBxORfzNiGeKIkcb5+\naJRsI0ajlMS9jwKn0Qz32DTV9p+MQUr6TrYDCmNTsvzhYzXcubm5tLS0nPvd0tJCXl7eVC6VoOQH\n1gx3PLGd/FEvOF5OOic+xjWd5+29dKwXkoYtWTjaaHEy5AYGzv2afpzTSU5g6NyvS8OqyU9GqGMh\nuNPBbkU2OZAMmSiB7WC7FrXXD94OiLQn7t+PMIhGRnQmzvmNyXXe3aaPrxLCEOsAC6TPwTnThRRd\nT8aXs/FZ7Hh3uRheG4aDRxMYjYlzI4xueNwfE9bppumW/2ik8UaXtO/CYxX+mocXDuJcTB/W5Hto\neqtjxA6B0GFDYlu94DN9NCXDXVtby8aNG6mqqsJkMnH69Gn2799/wVFFY5x94X7t4UG4gckNz4RQ\n9iqEexZjxCjqgCBQwF8bbRnR20tADsA4WEsZncbar/2nGZw44Evgy0n81nG+oo13rXxECAJqamqo\nr6+nrKyMb3zjGxccWzTGNSa7/1KulQM0ADB37twxeHoxKkna1gy2HVgJ0rXwSRndLRGMs0JYbV3E\nds8i5AwRfscObxVBvQcx6h8GKhG8DnN+aEK7h2ik0y//CoShzQXK4Ssubrr1FeY3beaBWS/xduYN\n/MH2MB90L4CDHmBWEjYdQlcuvE/+BLAWjYFnrP1TOWe8a+UgOs3xdLWY0Sl5f/KgTNuf7JFo27lA\nYJTzihL/a6NxXeK3dq1cxIh2qrp6oQ5ontMchLwKgdTE/YcQgy8X4AeijC7n8e4xPk3JcOv1ejwe\nD9FolGg0yre//e3zgv2CisY4ewbnuzxaw9UDsxECjCU+hsRnFqJRGjnnQmNFGMZShAH3IQQbTVyj\nkGSBXhxrGaPTaPu1nnZmAo9HbH86hnN+P0YlTO/6PNjZDwPtiPhaFChn7F5Xu889wM/x+/08/fTT\nfPOb3xzl2KIxrjGR/Rp2LRRxFWBBGM0wgoc+BE8vHLkmX0tOnBfA7/ePwdOLUVkCixYXtgP5kLkS\n1phZ9cD7zJ+1h1x3C2G9ib13LWD7nn46InkwKCWwRhKfosR3stGGkcYgBgXTJ3+NKgATeJywLJVv\nLXmGW0+9Q3lnA5Zrhtnpu4bTB6pgr+YJzETwdjS3WbvPRLAWjYFnrP0gjJg2MtS8Q4WRwYeOEYOk\nB9ITmDUea20q+T43A38cR1cnYrhJ3NuECAHZAA8YUyANEZVxqKKpaWYjirCL3ZKI1kQCwCDCcPoA\nb9I9dAhb4b9EXdUwZkFqNlTfCLcaWZW9g6uNRylUm7DEvbTqU9iOnSMHq+l61w1HmxFx9iBCZy4m\n/4nTlAw3gMViYc+ePXg8nkmeqTEhOcxhQfRaqYn/QgjmGxkZucYRjLMBdtDZIEUPJhUiIRjyQrQD\n1D6EAdeMv2DU1LCORZqyG8DmgJIy7lz5JhXpH6HzReh4JJt3leV075OhJ5x4lovFuTSaCcC+ffum\nESuMNEwjYjSQA5INjAZwmQEb+CMwPIBQsgFEIwgydmcj3MH6+vop4JEu+LjBmY1ttpWyh/dxb8pL\nLB3YgSkW5Eh6NVv6rie60wgHhmCoC9FqtZGM5oZenL/TL38ZdE5Scg0seGQdt59+hXkfHaXLlsPv\n5Af44I1FdL1tgLp2BD81XbzQO/y4sCa78ybAgQhxmBGWT9tvBqsObDqw6cGiF38Nx2EwBANDQAfC\nu9FGkAqasZm6rkoI8+MC0sCUDtkupLkGstNDzLD2kJbai8c1gMkQEb6rJmoZwnEj/V4XZ3pKaO7P\nYrA9D+q90OGHSAMiph1OYL0UXSXBKyuQC/mFZC9RmHnNKSoK6lnYu42igSM44i1ElTAOXTYSQ1Sm\nddB2UyWvVi2GjXboa0PowUTswPg0ZcMtSRLXX389Op2Oxx9/nM9//vMXHLElabuIkZ5aM9wGBEPM\nQCpIJWApBKMZ5GGI94JeBzoV9HHQK2ByIOvtyDozOoMOXYmC6oD4kJ7IST10NIK/GaKbgAMkuyoX\nx/pe0nbyjPJYlBS3SpXgTvh7xw9Z+N4WQi0yyr9bOH3ne/QGMlB6HIx0JKMZwTo0V06jqfF1NNIa\nrhmwgJQChmJIrcTgsmD2RLDmB1CRCHTYGW6SULu7IXwalEYE/6JJ12tMfEB0riJsNjpOEBkWGiVn\nlcBIyMsAeJBmpJKxapCHZr7AneveIUPp5kjJLLbqV7Jt2yp8fzLAh6eBU4nnkREdkSmBMTm2eSbx\nGeH39Mlf018zuqwUiuYO82T5/yb7uWb6s53suGcJ/3Dk+8R/NgTH6hHhrxAjHsyFcdmPQ/4aRi2u\n7gFDDliywWoDUxTJaEQ22DDIVgxpUfQZMXQZMfSeGJIM4T4jwTo9wSNh6D8O4R5Q9iAme0e8sYtj\n3ZC0nZxVosndDXIhOMowFGbhWd5L6ie6WVi4i5XKJiqVk5RFzuIMBJADjDgAVvCl2DiTUsD7KatZ\n672NgwdqGXolGzao0KuD+EFgfwLrRHR1NB3QdEoPkgt0FcjLPNQ8soUvVf2YO19YS+RdaO+HRp1E\nKxJBfzNze5opXvIK/Y/N5eSjv6eho5jghyqEhjk/dAKjyX8iNGXDvWPHDrKzs+np6eGGG26gsrKS\n5cuXJx2x4iK3NCOYYkSM/spAVwI1eiiWwGqFgVTwSMJdSlchE5ihYMoK4HT04tYN4dB5USWZvq50\nGneWw+tFsCcX2nMRcfEAYrS7mYMHD14E62RmlLVQTiJ045JhGbABTu+ErgaFFY9ESP/PDux1Nrzb\nDBe/HGWcbyjWjoN1xQRxyojGYUHMsKeCKReya+AxPWmrW6gpO8QC3R5i6Nnmv4Y9OxYT/5c8qB+G\nYc3tTDbcRYwYiquBH/Dee++N
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 151
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"collapsed": false,
|
||
|
|
"input": [
|
||
|
|
"pseudoclasses = km.predict(data)",
|
||
|
|
"pseudoclasses[:10]"
|
||
|
|
],
|
||
|
|
"language": "python",
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"output_type": "pyout",
|
||
|
|
"prompt_number": 152,
|
||
|
|
"text": [
|
||
|
|
"array([11, 23, 24, 1, 2, 19, 1, 31, 32, 13])"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"prompt_number": 152
|
||
|
|
}
|
||
|
|
]
|
||
|
|
}
|
||
|
|
]
|
||
|
|
}
|