Stanford cs231n Assignment #1 (a) 实现KNN

好久没有好好学机器学习了，上学期写过的CNN, SVM, 自编码器都已经生疏了。打算发愤图强了……已经看了cs231n的４个lecture了，可以完成第一个assignment了。视频在youtube上，感兴趣的人可以自己搜。

http://cs231n.github.io/assignments2016/assignment1/

1. Visualize CIFAR-10

CIFAR的图本身是32×32×3的RGB图，读入的时候把32×32×3直接vectorize了

Screenshot from 2016-12-08 12:21:02.png

2. KNN

The kNN classifier consists of two stages:

• During training, the classifier takes the training data and simply remembers it
• During testing, kNN classifies every test image by comparing to all training images and transfering the labels of the k most similar training examples
• The value of k is cross-validated

说白了knn的思路很简单，没有所谓的训练过程。只需要记录下所有的train数据，当test数据到来的时候，与train数据进行比较，利用vote机制得到最后最符合的那个train数据所在的类就可以了。k的值代表的含义是，计算出distance之后，取前k个距离最小的，在这之中再选出出现次数最多的类别。k大一些会使得输出结果的偶然性小一些。

• 方法１：两层循环计算test和train数据之间的欧式距离

We would now like to classify the test data with the kNN classifier. Recall that we can break down this process into two steps:
First we must compute the distances between all test examples and all train examples.
Given these distances, for each test example we find the k nearest examples and have them vote for the label. Lets begin with computing the distance matrix between all training and test examples. For example, if there are Ntr training examples and Nte test examples, this stage should result in a Nte x Ntr matrix where each element (i,j) is the distance between the i-th test and j-th train example.

num_test = X.shape[0]
    num_train = self.X_train.shape[0]
    dists = np.zeros((num_test, num_train))
    for i in xrange(num_test):
      for j in xrange(num_train):
        dists[i][j] = np.sqrt(np.sum(np.square(i-j)))

• 方法２：单层循环实现（此处不赘述了，只是在np.linalg.norm的时候加一个axis参数罢了）

• 方法３：Vectorization Implementation
  怎么用纯矩阵运算来计算这个确实花费了我一会儿功夫才想出来...上图说话！！

Photo_1208_1a.jpg

5555疑似写错了，不是1024哦，是3096...因为是RGB图……写错啦> <

用python实现的具体代码：(其中X就是A, self.X_train就是B啦)

  def compute_distances_no_loops(self, X):
    """
    Compute the distance between each test point in X and each training point
    in self.X_train using no explicit loops.

    Input / Output: Same as compute_distances_two_loops
    """
    num_test = X.shape[0]
    num_train = self.X_train.shape[0]
    dists = np.zeros((num_test, num_train)) 

    testSqr = np.sum(X*X, axis=1).reshape(X.shape[0],1)
    trainSqr = np.sum(self.X_train*self.X_train, axis=1).transpose()
    dists = -2*np.dot(X, self.X_train.transpose()) + np.tile(trainSqr,(X.shape[0],1)) + np.tile(testSqr, (1, self.X_train.shape[0]))
    dists = np.sqrt(dists)

    return dists

三种方法最后比较的的结果

Two loop version took 25.155940 seconds
One loop version took 30.616855 seconds
No loop version took 0.274438 seconds

hyper-parameter K 通过cross-validation来决定

Paste_Image.png

num_folds = 5
k_choices = [1, 3, 5, 8, 10, 12, 15, 20, 50, 100]

X_train_folds = []
y_train_folds = []
################################################################################
# TODO:                                                                        #
# Split up the training data into folds. After splitting, X_train_folds and    #
# y_train_folds should each be lists of length num_folds, where                #
# y_train_folds[i] is the label vector for the points in X_train_folds[i].     #
# Hint: Look up the numpy array_split function.                                #
################################################################################
X_train_folds = np.array_split(X_train, num_folds)
y_train_folds = np.array_split(y_train, num_folds)

# A dictionary holding the accuracies for different values of k that we find
# when running cross-validation. After running cross-validation,
# k_to_accuracies[k] should be a list of length num_folds giving the different
# accuracy values that we found when using that value of k.
k_to_accuracies = {}


################################################################################
# TODO:                                                                        #
# Perform k-fold cross validation to find the best value of k. For each        #
# possible value of k, run the k-nearest-neighbor algorithm num_folds times,   #
# where in each case you use all but one of the folds as training data and the #
# last fold as a validation set. Store the accuracies for all fold and all     #
# values of k in the k_to_accuracies dictionary.                               #
################################################################################
for k_candi in k_choices:
    k_to_accuracies[k_candi] = []
    for i in range(num_folds):
        X_test_hy = X_train_folds[i]
        y_test_hy = y_train_folds[i]
        
        # print len(y_train_folds)
        # print X_train_folds[0:i], X_train_folds[i+1:]
        X_train_hy = np.vstack(X_train_folds[0:i]+X_train_folds[i+1:])
        y_train_hy = np.hstack(y_train_folds[0:i]+y_train_folds[i+1:])
        
        classifier.train(X_train_hy, y_train_hy)
        dists_hy = classifier.compute_distances_no_loops(X_test_hy)
        
        y_test_pred_hy = classifier.predict_labels(dists_hy, k=k_candi)
        num_correct_hy = np.sum(y_test_pred_hy == y_test_hy)
        accuracy_hy = float(num_correct_hy) / len(y_test_hy)
        k_to_accuracies[k_candi].append(accuracy_hy)

# Print out the computed accuracies
for k in sorted(k_to_accuracies):
    for accuracy in k_to_accuracies[k]:
        print 'k = %d, accuracy = %f' % (k, accuracy)

最终根据这个cross-validation的结果可以得到最佳k的值，但是即使是最佳值其实正确率也只有28%，我们还需要更好的算法来解决问题。