NLP 数据扩增的可视化综述-FlyAI

NLP 数据扩增的可视化综述

自然语言处理

最后更新 2020-11-11 14:10 阅读 5414

最后更新 2020-11-11 14:10

阅读 5414

自然语言处理

##FlyAI文献翻译
**英文原文：**[：A Visual Survey of Data Augmentation in NLP](https://amitness.com/2020/05/data-augmentation-for-nlp/)
> **标签** ：自然语言处理

Unlike Computer Vision where using image data augmentation is standard practice, augmentation of text data in NLP is pretty rare. This is because trivial operations for images like rotating an image a few degrees or converting it into grayscale doesn’t change its semantics. This presence of semantically invariant transformation is what made augmentation an essential toolkit in Computer Vision research.

>与使用图像数据扩增是标准做法的计算机视觉不同，NLP 中文本数据的扩增非常罕见。这是因为图像的琐碎操作（如旋转图像几度或将其转换为灰度）不会更改其语义。这种语义不变转换的存在使得扩充成为计算机视觉研究中必不可少的工具包。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798758403992.png)

I was curious if there were attempts at developing augmentation techniques for NLP and explored the existing literature. In this post, I will share my findings of the current approaches being used for augmenting text data.

>我很好奇，是否有尝试开发NLP的扩增技术，并探索现有的文献。在本帖子中，我将分享我关于当前用于扩充文本数据的方法的发现。

###Approaches

1. Lexical Substitution
This approach tries to substitute words present in a text without changing the gist of the sentence.

Thesaurus-based substitution
In this technique, we take a random word from the sentence and replace it with its synonym using a Thesaurus. For example, we could use the [WordNet](https://wordnet.princeton.edu/ "WordNet") lexical database for English to look up the synonyms and then perform the replacement. It is a manually curated database with relations between words.

>###方法

>1.词汇替换

>此方法尝试在不改变语义的情况下替换词汇

>基于同义词库的替换

>在此技术中，我们从句子中随机抽取一个单词，并使用同义词库将其替换为同义词。例如，我们可以使用 WordNet 词汇数据库查找同义词，然后执行替换。这是一个手动整理的数据库，词与词之间有一定关联。在此技术中，我们从句子中随机抽取一个单词，并使用同义词库将其替换为同义词。例如，我们可以使用 WordNet 词汇数据库查找同义词，然后执行替换。这是一个手动整理的数据库，词与词之间有一定关联。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798758617477.png)

[Zhang et al](https://arxiv.org/abs/1509.01626 "Zhang et al"). used this technique in their 2015 paper “Character-level Convolutional Networks for Text Classification”. [Mueller et al](https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/download/12195/12023 "Mueller et al"). used a similar strategy to generate additional 10K training examples for their sentence similarity model.

NLTK provides a programmatic [access](https://www.nltk.org/howto/wordnet.html "access") to WordNet. You can also use [TextBlob API](https://textblob.readthedocs.io/en/dev/quickstart.html#wordnet-integration "TextBlob API"). There is also a database called [PPDB](http://paraphrase.org/#/download "PPDB") containing millions of paraphrases that you can download and access programmatically.

>Zhang等人在2015年的论文《文字分类的字符级卷积网络》中使用了这一技术。Mueller等人使用类似的策略为其句子相似性模型生成额外的 10K 训练示例。

>NLTK 支持对 WordNet 的程序化访问。你还可以使用 TextBlob 的API接口。还有一个叫PPDB的数据库，其中包含数百万个支持程序化访问和下载的语料。

**Word-Embeddings Substitution**

In this approach, we take pre-trained word embeddings such as Word2Vec, GloVe, FastText, Sent2Vec, and use the nearest neighbor words in the embedding space as the replacement for some word in the sentence. Jiao et al. have used this technique with GloVe embeddings in their paper “TinyBert” to improve generalization of their language model on downstream tasks. Wang et al. used it to augment tweets needed to learn a topic model.

>**单词嵌入替换**

>在此方法中，我们通过诸如 Word2Vec、GloVe、FastText、Sent2Vec等预先训练的嵌入词库，使用嵌入空间中接邻的单词，来替换句子中某些单词。Jiao等人在论文《小贝特》中使用了GloVe嵌入这项技术，用于改进其语言模型对下游任务的概括。Wang等人用它来扩充学习主题模型所需的推文。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798758450363.png)

For example, you can replace the word with the 3-most similar words and get three variations of the text.

>举个例子，你可以将某个单词替换为 3 个最相似的单词，并获得文本的三个变体。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798758562820.png)

It’s easy to use packages like Gensim to access pre-trained word vectors and get the nearest neighbors. For example, here we find the synonyms for the word ‘awesome’ using word vectors trained on tweets.

>使用 Gensim 这样的包来访问预先训练的单词矢量并获取接邻词十分简单。如下操作，我们通过在推文中训练的单词矢量，找到5个“awesome”的同义词。

```
# pip install gensimimport gensim.downloader as apimodel = api.load('glove-twitter-25')  model.most_similar('awesome', topn=5)
```
You will get back the 5 most similar words along with the cosine similarities.

>你将得到5个最相似的单词和应对应的余弦相似值。

```
[('amazing', 0.9687871932983398),
 ('best', 0.9600659608840942),
 ('fun', 0.9331520795822144),
 ('fantastic', 0.9313924312591553),
 ('perfect', 0.9243415594100952)]
 ```
Masked Language Model
Transformer models such as BERT, ROBERTA and ALBERT have been trained on a large amount of text using a pretext task called “Masked Language Modeling” where the model has to predict masked words based on the context.

This can be used to augment some text. For example, we could use a pre-trained BERT model and mask some parts of the text. Then, we use the BERT model to predict the token for the mask.

>掩码语言模型

>使用前置任务进行大量语料训练的转换模型，被称为“掩码语言模型”，例如BERT, ROBERTA及ALBERT等。这些模型需要根据语境，预测被掩码的单词。

>这可以扩充某些文本。例如，我们可以先拿训练好的BRERT模型，对其中部分文本惊醒掩码，再使用BERT模型来预测掩码的特征。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798758253097.png)

Thus, we can generate variations of a text using the mask predictions. Compared to previous approaches, the generated text is more grammatically coherent as the model takes context into account when making predictions.

>于是，通过对掩码的预测，我们就能生成许多文本的变体。与前面的方法相比，这个方法生成的文本更加符合语法逻辑，因为模型在进行预测时参考了上下文。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798759223067.png)

This is easy to implement with open-source libraries such as transformers by Hugging Face. You can set the token you want to replace with <mask> and generate predictions.

>使用Hugging Face等开源库，实现起来非常简单。你可以将（上述示意图中的）<mask>替换成任何特征，并生成预测。

```
from transformers import pipelinenlp = pipeline('fill-mask')nlp('This is <mask> cool')
```

```
[{'score': 0.515411913394928,
  'sequence': '<s> This is pretty cool</s>',
  'token': 1256},
 {'score': 0.1166248694062233,
  'sequence': '<s> This is really cool</s>',
  'token': 269},
 {'score': 0.07387523353099823,
  'sequence': '<s> This is super cool</s>',
  'token': 2422},
 {'score': 0.04272908344864845,
  'sequence': '<s> This is kinda cool</s>',
  'token': 24282},
 {'score': 0.034715913236141205,
  'sequence': '<s> This is very cool</s>',
  'token': 182}]
  ```
However one caveat of this method is that deciding which part of the text to mask is not trivial. You will have to use heuristics to decide the mask, otherwise the generated text will not retain the meaning of original sentence.

>然而，此方法的缺陷之一，是对被掩码的文本有一定要求。你必须使用探索法来决定文本被掩码的部分，否则，生成的文本将无法保留原文本的语义。

TF-IDF based word replacement

This augmentation method was proposed by [Xie et al](https://arxiv.org/abs/1904.12848 "Xie et al"). in the Unsupervised Data Augmentation paper. The basic idea is that words that have low TF-IDF scores are uninformative and thus can be replaced without affecting the ground-truth labels of the sentence.

>基于TF-IDF的单词替换

>这种扩充方法由Xie等人在关于无监督数据增强的论文中提出。其基本思想是，TF-IDF得分较低的单词缺少信息价值，因此在替换后，不会影响语句的基本事实标签。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798759357390.png)

The words to replace with are chosen from the whole vocabulary that have low TF-IDF scores in the whole document. You can refer to the implementation in the original paper from [here](https://github.com/google-research/uda/blob/master/text/augmentation/word_level_augment.py "here").

>我们可以从通篇中，选取TF-IDE得分较低的词，用于替换。这里可以参阅原论文中的具体实现。

**2. Back Translation**

In this approach, we leverage machine translation to paraphrase a text while retraining the meaning. Xie et al. used this method to augment the unlabeled text and learn a semi-supervised model on IMDB dataset with only 20 labeled examples. The method outperformed the previous state-of-the-art model trained on 25,000 labeled examples.

The back-translation process is as follows:

- Take some sentence (e.g. in English) and translate to another Language e.g. French

- Translate the french sentence back into English sentence

- Check if the new sentence is different from our original sentence. If it is, then we use this new sentence as an augmented version of the original text.

>**2.回译**

>在这种方法中，我们利用机器翻译来解释文本，同时重新训练意思。谢等，在IMDB数据集上，用该方法对未标记文本进行扩充，并学习了一个半监督模型。该方法优于先前在25000个标记样本上训练的最新模型。

>回译过程如下：

>- 把一些句子（如英语）翻译成另一种语言，如法语

>- 把法语句子译成英语句子

>- 检查新句子是否与我们原来的句子不同。如果是这样的话，我们就用这个新句子作为原文的扩充版。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798759683604.png)

You can also run back-translation using different languages at once to generate more variations. As shown below, we translate an English sentence to a target language and back again to English for three target languages: French, Mandarin and Italian.

>你还可以使用不同的语言同时运行翻译以生成更多变体。如下图所示，我们将一个英语句子翻译成目标语言，然后再翻译成英语。三种目标语言的英语：法语、汉语和意大利语。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798759637825.png)

This technique was also used in the [1st place solution](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52557 "1st place solution") for the “Toxic Comment Classification Challenge” on Kaggle. The winner used it for both training-data augmentations as well as during test-time where the predicted probabilities for English sentence along with back-translation using three languages(French, German, Spanish) were averaged to get the final prediction.

For the implementation of back-translation, you can use TextBlob. Alternatively, you can also use Google Sheets and follow the instructions given [here](https://amitness.com/2020/02/back-translation-in-google-sheets/ "here") to apply Google Translate for free.

>这项技术也被应用于Kaggle的“Toxic Comment Classification Challenge”的第一名解决方案中。胜利者将其用于训练数据扩充，以及在测试期间使用三种语言（法语、德语、西班牙语）对英语句子和反译的预测概率进行平均，以获得最终预测。

>对于反向翻译的实现，可以使用TextBlob。或者，你也可以使用谷歌工作表，并按照这里给出的说明免费应用谷歌翻译。

**3. Text Surface Transformation**

These are simple pattern matching transformations applied using regex and was introduced by Claude Coulombe in his paper.

In the paper, he gives an example of transforming verbal forms from contraction to expansion and vice versa. We can generate augmented texts by applying this.

>**3.文本表面转换**

>这些是使用正则表达式应用的简单模式匹配转换，由克劳德·库仑在他的论文中介绍。

>在这篇文章中，他举了一个例子来说明语言形式从收缩到扩张，反之亦然。我们可以通过应用这个来生成增强文本。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798759376936.png)

Since the transformation should not change the meaning of the sentence, we can see this can fail in case of expanding ambiguous verbal forms like:

![](由于转换不应该改变句子的意思，我们可以看到，如果扩展模糊的动词形式，例如：)

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798759921496.png)

To resolve this, the paper proposes that we allow ambiguous contractions but skip ambiguous expansion.

>为了解决这个问题，这篇文章提出允许歧义收缩，从而跳过歧义扩展。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798759515512.png)

You can find a list of contractions for the English language [here](https://en.wikipedia.org/wiki/Wikipedia%3aList_of_English_contractions "here").

>你可以在这里找到英语的缩略语列表。

**4. Random Noise Injection**

The idea of these methods is to inject noise in the text so that the model trained is robust to perturbations.

>**4. 随机噪声注入**

>这些方法的思想是在文本中注入噪声，使训练的模型对扰动具有鲁棒性

Spelling error injection

In this method, we add spelling errors to some random word in the sentence. These spelling errors can be added programmatically or using a mapping of common spelling errors such as this list for English.

>拼写错误注入

>在这种方法中，我们在句子中随机添加一些单词的拼写错误。这些拼写错误可以通过编程方式添加，也可以使用常见拼写错误的映射来添加，例如此英语拼写错误列表。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798760315580.png)

QWERTY Keyboard Error Injection

This method tries to simulate common errors that happen when typing on a QWERTY layout keyboard due to keys that are very near to each other. The errors are injected based on keyboard distance.

>QWERTY键盘错误注入

>此方法尝试模拟在QWERTY布局键盘上键入时由于按键彼此非常接近而发生的常见错误。错误是根据键盘距离注入的。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798760581369.png)

Unigram Noising
This method has been used by Xie et al. and also the UDA paper. The idea is to perform replacement with words sampled from the unigram frequency distribution. This frequency is basically how many times each word occurs in the training corpus.

>单格噪声

>该方法已被谢等人采用。还有UDA论文。其思想是用从单字图频率分布中抽取的单词进行替换。这个频率基本上就是每个单词在训练语料库中出现的次数。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798760920897.png)

Blank Noising

This method has been proposed by Xie et al. in their paper. The idea is to replace some random word with a placeholder token. The paper uses “_” as the placeholder token. In the paper, they use it as a way to avoid overfitting on specific contexts as well as a smoothing mechanism for the language model. The technique helped improve perplexity and BLEU scores in the paper.

>空白噪音

>这种方法是由谢等提出的。在他们的论文上。这个想法是用占位符标记替换一些随机单词。论文使用“_”作为占位符标记。在论文中，他们使用它来避免对特定语境的过度拟合，以及语言模型的平滑机制。这项技术有助于改善论文中的疑问点和BLEU分数。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798760901035.png)

Sentence Shuffling

This is a naive technique where we shuffle sentences present in a training text to create an augmented version.

>换位

>这是一种简单的技巧，我们将训练文本中出现的句子打乱，以创建一个扩大的版本。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798760796672.png)

**Instance Crossover Augmentation**

This technique was introduced by Luque in his paper on sentiment analysis for TASS 2019. It is inspired by the chromosome crossover operation that happens in genetics.
In the method, tweets are divided into two halves and two random tweets of the same polarity(i.e. positive/negative) have their halves swapped. The hypothesis is that even though the result will be ungrammatical and semantically unsound, the new text will still preserve the sentiment polarity.

>**实例交叉扩充**

>这项技术是由Luque在其关于TASS 2019情绪分析的论文中介绍的。它的灵感来自遗传学中的染色体交叉操作。

>在该方法中，tweet被分成两半，两个具有相同极性（即正/负）的随机tweet被交换。假设是，即使结果不符合语法和语义，新的文本仍然会保留情感的极性。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798760129367.png)

This technique had no impact on the accuracy but helped with the F1 score in the paper showing its impact on minority classes such as Neutral class with fewer tweets.

>这项技术对准确性没有影响，但有助于F1分数的提高，这篇论文显示了它对少数民族阶级的影响，如中立阶级，推特数量较少。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798761385718.png)

**Syntax-tree Manipulation**

This technique has been used in the paper by Coulombe. The idea is to parse and generate the dependency tree of the original sentence, transform it using rules and generate a paraphrased sentence.
For example, one transformation that doesn’t change the meaning of the sentence is the transformation from active voice to the passive voice of sentence and vice versa.

>**语法树操作**

>库仑在本文中已经使用了这一技术。其思想是解析并生成原语句的依存树，利用规则对其进行转换，生成一个复述句。

>例如，一个不改变句子意思的转换是句子的主动语态到被动语态的转换，反之亦然。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798761963297.png)

MixUp for Text
Mixup is a simple yet effective image augmentation technique introduced by Zhang et al. in 2017. The idea is to combine two random images in a mini-batch in some proportion to generate synthetic examples for training. For images, this means combining image pixels. The method acts as a form of regularization.

**MixUp for Text**

Mixup is a simple yet effective image augmentation technique introduced by [Zhang et al](https://arxiv.org/abs/1710.09412 "Zhang et al"). in 2017. The idea is to combine two random images in a mini-batch in some proportion to generate synthetic examples for training. For images, this means combining image pixels. The method acts as a form of regularization.

>**文本混合**

>Mixup是一种被zhang等提出的简单而有效的图像增强技术。2017年。其思想是将两幅随机图像按一定比例组合成一个小批量，生成用于训练的综合示例。对于图像，这意味着合并图像像素。该方法作为正则化的一种形式。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798761385138.png)

Bringing this idea to NLP, [Guo et al](https://arxiv.org/abs/1905.08941 "Guo et al"). modified Mixup to work with text. They propose two novel approaches for applying Mixup to text:

wordMixup:

In this method, two random sentences in a mini-batch are taken and they are zero-padded to the same length. Then, their word embeddings are combined in some proportion. The resulting word embedding is passed to the usual flow for text classification. The cross-entropy loss is calculated for both the labels of the original text in the given proportion.

>单词混合：

>在这种方法中，两个随机句子在一个mini-batch中被抽取，并且它们被零填充到相同的长度。然后按一定的比例组合词。生成的单词嵌入被传递到文本分类的常规流。在给定的比例下，计算两个原文标签的交叉熵损失。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798761337698.png)

sentMixup:

In this method, two sentences are taken and they are zero-padded to the same length. Then, their word embeddings are passed through LSTM/CNN encoder and we take the last hidden state as sentence embedding. These embeddings are combined in a certain proportion and then passed to the final classification layer. The cross-entropy loss is calculated based on both the labels of original sentences in the given proportion.

>句子混合：

>在这种方法中，取两个句子，它们被零填充到相同的长度。然后，通过LSTM/CNN编码器将它们的单词嵌入传递到最后一个隐藏状态作为句子嵌入。这些嵌入按一定比例组合，然后传递到最终的分类层。交叉熵损失是根据原句的两个labels按一定比例计算的。

![](https://static.leiphone.com/uploads/new/sns/article/202005/1589798761936360.png)

Implementation

Python libraries like nlpaug and textattack provide simple and consistent API to apply the above methods. They are framework agnostic and can be easily integrated into your pipeline.

>应用

>像nlpaug和textack这样的Python库提供了简单而一致的API来应用上述方法。它们与框架无关，可以很容易地集成到您的pipline中。

Conclusion

My takeaway from the literature review is that many of these augmentation methods are very task-specific and their impact on performance has been studied for some particular use-cases only. It would be an interesting research to systematically compare these methods and analyze their impact on performance for many tasks.

>结论

>我从文献综述中得到的结论是，这些增强方法中的许多都是非常特定于任务的，它们对性能的影响只针对一些特定的用例进行了研究。系统地比较这些方法并分析它们对许多任务性能的影响将是一项有趣的研究。

**Citation Info (BibTex)**

If you found this blog post useful, please consider citing it as:

>**引文信息（BibTex）**

如果你觉得这篇博文有用，请考虑引用它：

```
@misc{chaudhary2020nlpaugment,
  title   = {A Visual Survey of Data Augmentation in NLP},
  author  = {Amit Chaudhary},
  year    = 2020,
  note    = {\url{https://amitness.com/2020/05/data-augmentation-for-nlp}
}
```

**参考文献**

Qizhe Xie, et al. “[Unsupervised Data Augmentation for Consistency Training](https://arxiv.org/abs/1904.12848 "Unsupervised Data Augmentation for Consistency Training")”

Claude Coulombe “[Text Data Augmentation Made Simple By Leveraging NLP Cloud APIs](https://arxiv.org/abs/1812.04718 "Text Data Augmentation Made Simple By Leveraging NLP Cloud APIs")”

Xiaoqi Jiao, et al. “[TinyBERT: Distilling BERT for Natural Language Understanding](https://arxiv.org/abs/1909.10351 "TinyBERT: Distilling BERT for Natural Language Understanding")”

Xiang Zhang, et al. “[Character-level Convolutional Networks for Text Classification](https://arxiv.org/abs/1509.01626 "Character-level Convolutional Networks for Text Classification")”

Franco M. Luque “[Atalaya at TASS 2019: Data Augmentation and Robust Embeddings for Sentiment Analysis](https://arxiv.org/abs/1909.11241 "Atalaya at TASS 2019: Data Augmentation and Robust Embeddings for Sentiment Analysis")”

Ziang Xie, et al. “[Data Noising as Smoothing in Neural Network Language Models](https://arxiv.org/abs/1703.02573 "Data Noising as Smoothing in Neural Network Language Models")”

Hongyu Guo, et al. “[Augmenting Data with Mixup for Sentence Classification: An Empirical Study](https://arxiv.org/abs/1905.08941 "Augmenting Data with Mixup for Sentence Classification: An Empirical Study")”

Hongyi Zhang, et al. “[mixup: Beyond Empirical Risk Minimization](https://arxiv.org/abs/1710.09412 "mixup: Beyond Empirical Risk Minimization")”

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可，转载请附上原文出处链接和本声明。
本文链接地址：https://flyai.com/article/756

讨论

500字

表情

发送

删除确认

是否删除该条评论？

取消删除