自监督学习知识迁移-FlyAI

自监督学习知识迁移

图像分类

最后更新 2020-11-06 14:48 阅读 5328

最后更新 2020-11-06 14:48

阅读 5328

图像分类

##FlyAI文献翻译
**英文原文：**[：Knowledge Transfer in Self Supervised Learning](https://amitness.com/knowledge-transfer/)
> **标签** ：图像分类

Self Supervised Learning is an interesting research area where the goal is to learn rich representations from unlabeled data without any human annotation.

This can be achieved by creatively formulating a problem such that you use parts of the data itself as labels and try to predict that. Such formulations are called pretext tasks.

pretext tasksFor example, you can setup a pretext task to predict the color version of the image given the grayscale version. Similarly, you could remove a part of the image and train a model to predict the part from the surrounding. There are many such [pretext tasks](https://amitness.com/2020/02/illustrated-self-supervised-learning/ "pretext tasks").

>自监督学习是一个非常有趣的研究领域，其目标是在没有任何人工标注的情况下，从未标注的数据中学习丰富的表示。

>我们可以通过创造性地提出一个问题来实现，这样你就可以使用数据本身的一部分作为标签并尝试进行预测。这样的表述被称为前置任务

>例如，你可以设置一个前置任务，给定灰度图情况下来预测图像的彩色版本。同样，您可以删除图像的一部分，并训练一个模型从周围预测该部分。有很多类似这样的前置任务。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602224813191536.png)

By pre-training on the pretext task, the hope is that the model will learn useful representations. Then, we can finetune the model to downstream tasks such as image classification, object detection, and semantic segmentation with only a small set of labeled training data.

>通过对前置任务的预训练，希望模型能够学习到有用的表示。之后，我们可以只用一小组标记的训练数据，可以将模型微调到下游任务，例如图像分类、对象检测和语义分割。

###Challenge of evaluating representations

So pretext tasks can help us learn representations. But, this poses a question:

How to determine how good a learned representation is?

Currently, the standard way to gauge the representations is to evaluate it on a set of standard tasks and benchmark datasets.

- Linear classification: ImageNet classification using frozen features

- Low Data Regime: ImageNet Classification using only 1% to 10% of data

- Transfer Learning: Object Classification, Object Detection and Semantic Segmentation on PASCAL VOC

We can see that the above evaluation methods require us to use the same model architecture for both the pretext task and the target task.

>###评估表示的挑战

>所以，前置任务可以帮助我们学习表征。但是，这就提出了一个问题：

>目前，衡量表征的标准方法是在一组标准任务和基准数据集上进行评估。

>如何判断一个表示学习的好坏？

>- 线性分类：利用固定特征进行ImageNet分类
- 小样本设定：ImageNet分类只使用1%至10%的数据。
- 迁移学习：在PASCAL VOC上进行目标分类、目标检测和语义分割。

>我们可以看到，上述评价方法要求我们对前置任务和目标任务使用相同的模型架构。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602224813299650.png)

This poses some interesting challenges:

1. For the pretext task, our goal is to learn on a large-scale unlabeled dataset and thus deeper models(e.g. ResNet) would help us learn better representations.

3. But, for downstream tasks, we would prefer shallow models(e.g. AlexNet) for actual applications. Thus, we currently have to consider this limitation when designing the pretext task.

5. It’s harder to fairly compare which pre-text task is better if some methods used simpler architecture while other methods used deeper architecture.

7. We can’t compare the representations learned from pretext tasks to handcrafted features such as HOG.

9. We may want to exploit several data domains such as sound, text, and videos in the pretext task but the target task may limit our design choices.

11. Model trained on pretext task may learn extra knowledge that is not useful for generic visual recognition. Currently, the final task-specific layers are ignored and weights or features only up to certain convolutional layers are taken.

>这就会带来了一些有趣的挑战：

>1. 对于前置任务，我们的目标是在大规模的无标签数据集上学习，因此更深的模型(如ResNet)将会帮助我们学习更好的表示。

>3. 但是，对于下游任务，我们则更倾向于实际应用的浅层模型(如AlexNet)。因此，我们目前在设计前置任务时就必须考虑这个限制。

>5. 如果有些方法使用了较简单的架构，而其他方法使用了较深的架构，那么就很难公平地比较哪种前置任务更好。

>7. 我们无法将从前置任务中学习到的表征与HOG等人工设计的特征进行比较。

>9. 我们可能希望在前置任务中利用多个数据域，如声音、文本和视频，但目标任务可能会限制我们的设计选择。

>11. 在前置任务上训练的模型可能会学习到对通用视觉识别无用的额外知识。如今，最后一层的任务特定层会被忽略，实际只取权重或特征到一定的卷积层。

###Knowledge Transfer

Noroozi et al. proposed a simple idea to tackle these issues in their 2018 paper “[Boosting Self-Supervised Learning via Knowledge Transfer](https://arxiv.org/abs/1805.00385 "Boosting Self-Supervised Learning via Knowledge Transfer")”.

**Intuition**

The authors observed that in a good representation space, semantically similar data points should be close together.

>###知识迁移

>Noroozi等人在他们2018年的一篇论文 “Boosting Self-Supervised Learning via Knowledge Transfer”中提供了一个非常简单的想法去解决这些问题。

>**直觉**

>作者观察到，在一个良好的表示空间中，语义相似的数据点应该是紧密相连的。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602224813786294.png)

In regular supervised classification, the information that images are semantically similar is encoded through labels annotated by humans. A model trained on such labels would have a representation space that groups semantically similar images.

Thus, with pre-text tasks in self-supervised learning, the objective is implicitly learning a metric that makes the same category images similar and different category images dissimilar. Hence we can provide a robust estimate of the learned representation if we could encode semantically related images to the same labels in some way.

>在常规的有监督分类中，图像在语义上相似的信息是通过人类注释的标签进行编码的。在这种标签上训练的模型一般都会有一个表示空间，对语义相似的图像进行分组。

>因此，对于自监督学习中的前置任务，目标是隐式学习一个使相同类别图像相似而不同类别图像不相似的度量。因此，如果我们能够以某种方式将语义相关的图像编码到相同的标签上，我们就可以对学习的表示提供可靠的估计。

###General Framework

The authors propose a novel framework to transfer knowledge from a deep self-supervised model to a separate shallow downstream model. You can use different model architectures for the pretext task and downstream task.

**Key Idea:**

Cluster features from pretext task and assign cluster centers as pseudo-labels for unlabeled images. Then, re-train a small network with target task architecture on pseudo-labels to predict pseudo-labels and learn a novel representation.

The end-to-end process is described below:

**1. Pretext task**

Here we choose some deep network architecture and train it on some pretext task of our choice on some dataset. We can take features from some intermediate layer after the model is trained.

>###通用框架

>作者提出了一个新颖的框架，可以将知识从一个深层的自监督模型转移到一个独立的浅层下游模型。你可以使用不同的模型架构来完成前置任务和下游任务。

>**核心想法：**

>将前置任务中的特征进行聚类，并将聚类中心分配给未标记的图像作为伪标签。然后，在伪标签上重新训练一个具有目标任务架构的小型网络来预测伪标签，并学习一种新的表示方法。  
这种端到端的处理流程介绍如下：

>**1.前置任务**

>在这里，我们选择一些深度网络架构，并在一些数据集上对我们选择的一些前置任务进行训练。在模型训练完成后，我们可以从一些中间层获取特征。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602224814341918.png)

Figure: Training on Pre-text Task ([Source](https://arxiv.org/abs/1805.00385 "Source"))

>图：在前置任务上训练（资源）

**2. K-means Clustering**

For all the unlabeled images in the dataset, we compute the feature vectors from the pretext task model. Then, we run K-means clustering to group semantically similar images. The idea is that the cluster centers will be aligned with categories in ImageNet.

>**2.K-means聚类**

>对于数据集中所有未标记的图像，我们计算前置任务模型中 的特征向量。然后，我们基于K-means聚类来分组语义上相似的图像。我们的想法是，聚类中心将与ImageNet中的类别对齐。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602224814152932.png)

Figure: Clustering Features (Source)

In the paper, the authors ran K-means on a single Titan X GPU for 4 hours to cluster 1.3M images into 2000 categories.

>图：聚类特征（资源）

>在论文中，作者使用的K-means在单台Titan X GPU上运行了4个小时，将130万张图片聚成2000个类别。

**3. Pseudo-labeling**

The cluster centers are treated as the pseudo-label. We can use either the same dataset as the above step or use a different dataset itself. Then, we compute the feature vectors for those images and find the closest cluster center for each image. This cluster center is used as the pseudo-label.

>**3. 伪标签**

>聚类中心被当作伪标签。我们可以使用与上一步相同的数据集，也可以自己使用不同的数据集。然后，我们计算这些图像的特征向量，并为每个图像找到最接近的聚类中心。这个聚类中心被用作伪标签。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602224815831828.png)

Figure: Generating Pseudo-labels ([Source](https://arxiv.org/abs/1805.00385 "Source"))

>图：生成伪标签（来源）

**4. Training on Pseudo-labels**

We take the model architecture that will be used for downstream tasks and train it to classify the unlabeled images into the pseudo-labels. Thus, the target architecture will learn a new representation such that it will map images that were originally close in the pre-trained feature space to close points.

>**4. 在伪标签上训练**

>我们采取将用于下游任务的模型架构，并利用其将未标记的图像分类为伪标签。因此，目标架构将学习一个新的表示，这样它将把原本在预先训练的特征空间中接近的图像映射到接近的点上。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602224814891419.png)

Figure: Re-training on pseudo-labels (Source)

**Advantage of Knowledge Transfer**

We saw how by clustering the features and then using pseudo-labels, we can bring the knowledge from any pretext task representations into a common reference model like AlexNet.

As such, we can now easily compare different pretext tasks even if they are trained using different architectures and on different data domains. This also allows us to improve self-supervised methods by using deep models and challenging pretext tasks.

>在伪标签数据上重新训练（来源）

>**知识迁移的优势**

>我们看到了如何通过对特征进行聚类，然后使用伪标签，我们可以将任何前置任务表征中的知识带入一个共同的参考模型中，比如AlexNet。

>因此，我们现在可以轻松地比较不同的前置任务，即使它们是使用不同的架构和在不同的数据域上进行训练的。这也让我们可以通过使用深度模型和具有挑战性的前置任务来改进自监督方法。

**How well does this framework work?**

To evaluate the idea quantitatively, the authors set up an experiment as described below:

**a. Increase complexity of pretext task (Jigsaw++)**

To evaluate their method, the authors took an old puzzle-like pretext task called “Jigsaw” where we need to predict the permutation that was used to randomly shuffle a 3*3 square grid of image.

>**这个框架的效果如何？**

>为了定量评价这个想法，作者设置了一个实验，如下所述：

>**a. 增加前置任务的复杂性(Jigsaw++)**

>为了评估他们的方法，作者采取了一个名为 "拼图 "的老式拼图式前置任务，我们需要预测用来随机打乱3 * 3正方形图像网格的排列。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602224815205791.png)

Image Modified from [Paper](https://arxiv.org/abs/1805.00385 "Paper")

They extended the task by randomly replacing 0 to 2 number of tiles with tile from another random image at some random locations. This increases the difficulty as now we need to solve the problem using only the remaining patches. The new pretext task is called “Jigsaw++”.

>  图来自这篇论文

>他们通过在某些随机位置用另一个随机图像中的图块随机替换0到2个图块来扩展任务。这增加了难度，因为现在我们只需要使用剩余的补丁来解决这个问题。新的前置任务被称为 "Jigsaw++"。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602224815563256.png)

Image Modified from [Paper](https://arxiv.org/abs/1805.00385 "Paper")

In the paper, they use 701 total permutations which had a minimum hamming distance of 3. They apply mean and standard deviation normalization at each image tile independently. They also make images gray-scale 70% of the time to prevent the network from cheating with low-level statistics.

>  图来自这篇论文

>在本文中，他们使用了701个总排列，其最小汉明距离为3。他们对每个图像块分别应用均值和标准差归一化。他们还将图像的灰度化率提高到70%，以防止网络利用低浅显的统计信息而作弊。

**b. Use a deeper network to solve pretext task
**
The authors used VGG-16 to solve the pretext task and learn representations. As VGG-16 has increased capacity, it can better handle the increased complexity of the “Jigsaw++” task and thus extract better representation.

**c. Transfer Knowledge back to AlexNet**

The representations from VGG-16 are clustered and cluster centers are converted to pseudo-labels. Then, AlexNet is trained to classify the pseudo-labels.

**d. Finetune AlexNet on Evaluation datasets**

For downstream tasks, the convolutional layers for the AlexNet model are initialized with weights from pseudo-label classification and the fully connected layers were randomly initialized. The pre-trained AlexNet is then finetuned on various benchmark datasets.

>**b. 用更深层次的网络来解决借口任务**

>作者使用VGG-16来解决前置任务并学习表征。由于VGG-16的容量增加，它可以更好地处理 "Jigsaw++"任务带来的额外复杂性，从而提取更好的表征。

>**c. 将知识迁移至AlexNet**

>对来自VGG-16的表示进行聚类，并将聚类中心转换为伪标签。然后，AlexNet被训练来对伪标签进行分类。

**d. 在评估数据集上对AlexNet进行微调**

>对于下游任务，使用伪标签分类的权重初始化AlexNet模型的卷积层，并随机初始化完全连接的层。然后，在各种基准数据集上对预训练的AlexNet进行微调。

**e. Results**

Using a deeper network like VGG-16 leads to better representation and pseudo-labels and also better results in benchmark tasks. It got state of the art results on several benchmarks in 2018 and reduced the gap between supervised and self-supervised methods further.

>**e. 结果**

使用像VGG-16这样的更深层网络，可以得到更好的表示和伪标签，也可以在基准任务中获得更好的结果。它在2018年的几个基准上得到了最优秀的结果，并进一步缩小了监督和自监督方法之间的差距。

1. Transfer Learning on PASCAL VOC

The authors tested their method on object classification and detection on PASCAL VOC 2007 dataset and semantic segmentation on PASCAL VOC 2012 dataset.

>1. PASCAL VOC的转移学习

>作者在PASCAL VOC 2007数据集上测试了他们的方法，在PASCAL VOC 2012数据集上测试了他们的对象分类和检测，在PASCAL VOC 2012数据集上测试了语义分割。

Insights

- Training Jigsaw++ with VGG16 and using AlexNet to predict cluster gives the best performance.

- Switching to a challenging pretext task "Jigsaw++" improves performance than "Jigsaw".

- Knowledge transfer doesn't have a significant impact when using the same architecture AlexNet in both Jigsaw++ and downstream tasks.

>洞察

>使用VGG16训练 "Jigsaw++"，并使用AlexNet预测集群可获得最佳性能。

>切换到更具挑战性的前置任务 "Jigsaw++"比 "Jigsaw "提高了性能。

>当在Jigsaw++和下游任务中都使用相同架构的AlexNet时，知识迁移并没有明显影响。

![](https://dataset.flyai.com/1604644888.png)

**2. Linear Classification on ImageNet**

In this, a linear classifier is trained on features extracted from AlexNet at different convolutional layers. For ImageNet, using VGG-16 and transferring knowledge to AlexNet using clustering gives a substantial boost of 2%.

>**2. 在ImageNet上的线性分类**

>在这种设定下，在从不同卷积层的AlexNet中提取的特征上训练线性分类器训练。对于ImageNet来说，使用VGG-16并通过聚类将知识转移到AlexNet中，可以大幅提升2%。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602224814156728.png)

**3. Non-linear classification on ImageNet**

For a non-linear classifier, using VGG-16 and transferring knowledge to AlexNet using clustering gives the best performance on ImageNet.

>**3. ImageNet上的非线性分类**

>对于非线性分类器来说，使用VGG-16将知识迁移到AlexNet上并使用聚类可以在ImageNet上获得最佳性能。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602224814170947.png)

###Additional Insights from Paper

**1. How does the number of clusters affect the performance?**

The network is not significantly affected by the number of clusters. The authors tested AlexNet trained on pseudo-labels from a different number of clusters on the task of object detection.

>###来自论文的其他见解

>**1. 簇的数量对性能有什么影响？**

>网络并没有受到聚类数量的显著影响。作者在物体检测的任务上测试了在不同簇数的伪标签上训练的AlexNet。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602224814686014.png)

**2. How is this different from Knowledge Distillation?**

Knowledge transfer is fundamentally different from knowledge distillation. Here, the goal is to only preserve the cluster association of images from the representation and transfer that to the target model. Unlike distillation, we don’t do any regression to the exact output of the teacher.

**3. Can you use different datasets in clustering vs predicting pseudo-labels?**

Yes, the method is flexible and you can pre-train on one dataset, cluster on another, and get pseudo-labels for the third one.

The authors did an experiment where they trained clustering on representations for ImageNet and then calculated cluster centers on the “Places” dataset to get pseudo-labels. There was only a small reduction (-1.5%) in performance for object classification.

>** 2. 这与知识蒸馏有什么不同？**

>知识迁移与知识蒸馏有着本质的区别。在这里，我们的目标是只保留表征中图像的簇关联，并将其转移到目标模型中。与蒸馏不同的是，我们不对teacher的精确输出做任何回归。

>**3. 在聚类与预测伪标签中可以使用不同的数据集吗？**

>可以，这个方法很灵活，你可以在一个数据集上进行预训练，在另一个数据集上进行聚类，然后得到第三个数据集的伪标签。

>作者做了一个实验，他们在ImageNet的表示上训练聚类，然后在 "Places "数据集上计算聚类中心，得到伪标签。对象分类的性能仅有小幅下降（-1.5%）。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1602224814607944.png)

###Conclusion

Thus, Knowledge Transfer is a simple and efficient way to map representations from deep to shallow models.

>###结论

>因此，知识迁移是一种简单有效的方式，可以将表征从深层模型映射到浅层模型。

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可，转载请附上原文出处链接和本声明。
本文链接地址：https://flyai.com/article/749

讨论

500字

表情

发送

删除确认

是否删除该条评论？

取消删除