强化学习的10个现实应用-FlyAI

强化学习的10个现实应用

机器学习深度学习

最后更新 2020-11-07 15:33 阅读 6017

最后更新 2020-11-07 15:33

阅读 6017

机器学习深度学习

##FlyAI文献翻译
**英文原文：**[：10 Real-Life Applications of Reinforcement Learning](https://neptune.ai/blog/reinforcement-learning-applications)
> **标签** ：强化学习

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603694359448449.png)

In Reinforcement Learning (RL), agents are trained on a reward and punishment mechanism. The agent is rewarded for correct moves and punished for the wrong ones. In doing so, the agent tries to minimize wrong moves and maximize the right ones.

>在强化学习中，我们使用奖惩机制来训练agents。Agent做出正确的行为会得到奖励，做出错误的行为就会受到惩罚。这样的话，agent就会试着将自己的错误行为最少化，将自己的正确行为最多化。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603694360398295.png)

In this article, we’ll look at some of the real-world applications of reinforcement learning.

>本文我们将会聚焦于强化学习在现实生活中的实际应用。

###Applications in self-driving cars

Various papers have proposed[ Deep Reinforcement Learning](https://arxiv.org/pdf/2002.00444.pdf " Deep Reinforcement Learning") for autonomous driving. In self-driving cars, there are various aspects to consider, such as speed limits at various places, drivable zones, avoiding collisions — just to mention a few.

Some of the autonomous driving tasks where reinforcement learning could be applied include trajectory optimization, motion planning, dynamic pathing, controller optimization, and scenario-based learning policies for highways.

For example, parking can be achieved by learning automatic parking policies. Lane changing can be achieved using [Q-Learning](https://towardsdatascience.com/simple-reinforcement-learning-q-learning-fcddc4b6fe56 "Q-Learning") while overtaking can be implemented by learning an overtaking policy while avoiding collision and maintaining a steady speed thereafter.

[AWS DeepRacer ](https://aws.amazon.com/fr/deepracer/ "AWS DeepRacer ")is an autonomous racing car that has been designed to test out RL in a physical track. It uses cameras to visualize the runway and a reinforcement learning model to control the throttle and direction.

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603694359682430.png)

Wayve.ai has successfully applied reinforcement learning to training a car on how to drive in a day. They used a deep reinforcement learning algorithm to tackle the lane following task. Their network architecture was a deep network with 4 convolutional layers and 3 fully connected layers. The example below shows the lane following task. The image in the middle represents the driver’s perspective.

>Wayve.ai已经成功应用了强化学习来训练一辆车如何在白天驾驶。他们使用了深度强化学习算法来处理车道跟随任务的问题。他们的网络结构是一个有4个卷积层和3个全连接层的深层神经网络。例子如图。中间的图像表示驾驶员视角。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603694362895563.png)

###Industry automation with Reinforcement Learning

In industry reinforcement, learning-based robots are used to perform various tasks. Apart from the fact that these robots are more efficient than human beings, they can also perform tasks that would be dangerous for people.

A great example is the use of AI agents by [Deepmind to cool Google Data Centers](https://deepmind.com/blog/article/safety-first-ai-autonomous-data-centre-cooling-and-industrial-control "Deepmind to cool Google Data Centers"). This led to a 40% reduction in energy spending. The centers are now fully controlled with the AI system without the need for human intervention. There is obviously still supervision from data center experts. The system works  in the following way:

>###强化学习在工业自动化中的应用

>在工业自动化中，基于强化学习的机器人被用于执行各种任务。这些机器人不仅效率比人类更高，还可以执行危险任务。

>Deepmind使用AI智能体来冷却Google数据中心是一个成功的应用案例。通过这种方式，节省了40%的能源支出。现在，这些数据中心完全由人工智能系统控制，除了很少数据中心的专家，几乎不再需要其他人工干预。该系统的工作方式如下：

- Taking snapshots of data from the data centers every five minutes and feeding this to deep neural networks

- It then predicts how different combinations will affect future energy consumptions

- Identifying actions that will lead to minimal power consumption while maintaining a set standard of safety criteria

- Sending  and implement these actions at the data center

The actions are verified by the local control system. 
 
>- 每五分钟从数据中心获取数据快照，并将其输入深度神经网络

>- 预测不同组合将如何影响未来的能源消耗

>- 在符合安全标准的情况下，采取具有最小功耗的措施

>- 向数据中心发送相应措施并实施操作

>当然，具体的措施还是由本地控制系统操作完成。

###Reinforcement Learning applications in trading and finance

Supervised [time series](https://arxiv.org/ftp/arxiv/papers/1803/1803.03916.pdf "time series") models can be used for predicting future sales as well as predicting stock prices. However, these models don’t determine the action to take at a particular stock price. Enter Reinforcement Learning (RL). An RL agent can decide on such a task; whether to hold, buy, or sell. The RL model is evaluated using market benchmark standards in order to ensure that it’s performing optimally.

This automation brings consistency into the process, unlike previous methods where analysts would have to make every single decision. [IBM](https://medium.com/inside-machine-learning/reinforcement-learning-the-business-use-case-part-2-c175740999 "IBM") for example has a sophisticated reinforcement learning based platform that has the ability to make financial trades. It computes the reward function based on the loss or profit of every financial transaction.

>###强化学习在金融贸易中的应用

>有监督的时间序列模型可用来预测未来的销售额，还可以预测股票价格。然而，这些模型并不能决定在特定股价下应采取何种行动，强化学习（RL）正是为此问题而生。通过市场基准标准对RL模型进行评估，确保RL智能体正确做出持有、购买或是出售的决定，以保证最佳收益。

>通过强化学习，金融贸易不再像从前那样由分析师做出每一个决策，真正实现机器的自动决策。例如，IBM构建有一个强大的、面向金融交易的强化学习平台，该平台根据每一笔金融交易的损失或利润来调整奖励函数。

###Reinforcement Learning in NLP (Natural Language Processing)

In NLP, RL can be used in text summarization, question answering, and machine translation just to mention a few.

The authors of this paper Eunsol Choi, Daniel Hewlett, and Jakob Uszkoreit propose an RL based approach for question answering given long texts. Their method works by first selecting a few sentences from the document that are relevant for answering the question. A slow RNN is then employed to produce answers to the selected sentences.

>###强化学习在自然语言处理NLP中的应用

>RL可用于文本摘要、问答和机器翻译等NLP任务。

>Eunsol Choi、Daniel Hewlett和Jakob Uszkoret在论文中提出了一种基于RL的长文本问答方法。具体而言，首先从文档中选出几个与问题相关的句子，然后结合所选句子和问句通过RNN生成答案。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603694359710005.png)

A combination of supervised and reinforcement learning is used for[ abstractive text summarization in this paper](https://arxiv.org/pdf/1705.04304.pdf " abstractive text summarization in this paper"). The paper is fronted by Romain Paulus, Caiming Xiong & Richard Socher. Their goal is to solve the problem faced in summarization while using Attentional, RNN-based encoder-decoder models in longer documents. The authors of this paper propose a neural network with a novel intra-attention that attends over the input and continuously generates output separately. Their training methods are a combo of standard supervised word prediction and reinforcement learning.

>该论文结合监督学习与强化学习生成抽象文本摘要。论文作者Romain Paulus, Caiming Xiong和Richard Socher等人希望解决基于注意力的RNN编解码模型在摘要生成中面临的问题。论文提出了一种新的内注意力神经网络，通过该注意力可以关注输入并连续生成输出，监督学习和强化学习被用于模型训练。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603694360420677.png)

On the side of machine translation, [authors from the University of Colorado and the University of Maryland](http://users.umiacs.umd.edu/~jbg/docs/2014_emnlp_simtrans.pdf "authors from the University of Colorado and the University of Maryland"), propose a reinforcement learning based approach to simultaneous machine translation. The interesting thing about this work is that it has the ability to learn when to trust the predicted words and uses RL to determine when to wait for more input.

>斯坦福大学、俄亥俄州立大学和微软研究所的研究人员提出Deep-RL，可用于对话生成任务。Deep-RL使用两个虚拟智能体模拟对话，并学习多轮对话中的未来奖励的建模，同时，应用策略梯度方法使高质量对话获得更高奖励，如连贯性、信息丰富度和简洁性等。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603694360206430.png)

More NLP applications can be [found here](https://github.com/adityathakker/awesome-rl-nlp "found here").

>[点此链接](https://github.com/adityathakker/awesome-rl-nlp "点此链接")查看更多RL在NLP中的应用。

###Reinforcement Learning applications in healthcare

In healthcare, patients can receive treatment from policies learned from RL systems. RL is able to find optimal policies using previous experiences without the need for previous information on the mathematical model of biological systems. It makes this approach more applicable than other control-based systems in healthcare.

RL in healthcare is categorized as [dynamic treatment regimes(DTRs)](https://arxiv.org/pdf/1908.08796.pdf "dynamic treatment regimes(DTRs)") in chronic disease or critical care, automated medical diagnosis, and other general domains.

>###强化学习在医疗保健中的应用

>医疗保健领域，RL系统为患者只能提供治疗策略。该系统能够利用以往的经验找到最优的策略，而无需生物系统的数学模型等先验信息，这使得基于RL的系统具有更广泛的适用性。

>基于RL的医疗保健动态治疗方案（DTRs）包括慢性病或重症监护、自动化医疗诊断及其他一些领域。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603694360442990.png)

In DTRs the input is a set of clinical observations and assessments of a patient. The outputs are the treatment options for every stage. These are similar to states in RL. Application of RL in DTRs is advantageous because it is capable of determining time-dependent decisions for the best treatment for a patient at a specific time.

The [use of RL in healthcare](https://arxiv.org/pdf/1908.08796.pdf "use of RL in healthcare") also enables improvement of long-term outcomes by factoring the delayed effects of treatments.

RL has also been used for the discovery and generation of optimal DTRs for chronic diseases.

You can dive deeper into RL applications in healthcare by exploring this [paper](https://arxiv.org/pdf/1908.08796.pdf "paper").

>DTRs的输入是一组对患者的临床观察和评估数据，输出则是每个阶段的治疗方案。通过RL，DTRs能够确定患者在特定时间的最佳治疗方案，实现时间依赖性决策。

>在医疗保健中，RL方法还可用于根据治疗的延迟效应改善长期结果。

>对于慢性病，RL方法还可用于发现和生成最佳DTRs。

>通过本文，您可以深入研究RL在医疗保健中的应用。

###Reinforcement Learning applications in engineering

In the engineering frontier, Facebook has developed an open-source reinforcement learning platform — [Horizon](https://engineering.fb.com/ml-applications/horizon/ "Horizon"). The platform uses reinforcement learning to optimize large-scale production systems. Facebook has used Horizon internally:

- to personalize suggestions

- deliver more meaningful notifications to users

- optimize video streaming quality.

>###强化学习在工程中的应用

>在工程领域，Facebook提出了开源强化学习平台 —— Horizon，该平台利用强化学习来优化大规模生产系统。在Facebook内部，Horizon被用于：

>- 个性化指南

>- 向用户发送更有意义的通知

>- 优化视频流质量

Horizon also contains workflows for:

- simulated environments

- a distributed platform for data preprocessing

- training and exporting models in production.

>Horizon主要流程包括：

>- 模拟环境

>- 用于数据处理的分布式数据平台

>- 模型的训练与输出

A classic example of reinforcement learning in video display is serving a user a low or high bit rate video based on the state of the video buffers and estimates from other machine learning systems.

Horizon is capable of handling production-like concerns such as:

- deploying at scale

- feature normalization

- distributed learning

- serving and handling datasets with high-dimensional data and thousands of feature types.

>一个典型例子是，强化学习根据视频缓冲区的状态和其他机器学习系统的估计可选择的为用户提供低比特率或高比特率的视频。

>Horizon还能够处理以下问题：

>- 大规模部署

>- 特征规范化

>- 分布式学习

>- 超大规模数据的处理和服务，如包含高维数据和数千特征的数据集。

###Reinforcement Learning in news recommendation

User preferences can change frequently, therefore[ recommending news](http://www.personal.psu.edu/~gjz5038/paper/www2018_reinforceRec/www2018_reinforceRec.pdf " recommending news") to users based on reviews and likes could become obsolete quickly. With reinforcement learning, the RL system can track the reader’s return behaviors.

Construction of such a system would involve obtaining news features, reader features, context features, and reader news features. News features include but are not limited to the content, headline, and publisher. Reader features refer to how the reader interacts with the content e.g clicks and shares. Context features include news aspects such as timing and freshness of the news. A reward is then defined based on these user behaviors.

>###强化学习在新闻推荐中的应用

>在新闻推荐领域，用户的喜好不是一成不变的，仅仅基于评论和（历史）喜好向用户推荐新闻无法一劳永逸。基于强化学习的系统则可以动态跟踪读者反馈并更新推荐。

>构建这样一个系统需要获取新闻特征、读者特征、上下文特征和读者阅读的新闻特征。其中，新闻特征包括但不限于内容、标题和发布者；读者特征是指读者与内容的交互方式，如点击和共享；上下文特征包括新闻的时间和新鲜度等。然后根据用户行为定义奖励函数，训练RL模型。

###Reinforcement Learning in gaming

Let’s look at an application in the gaming frontier, specifically AlphaGo Zero. Using reinforcement learning, AlphaGo Zero was able to learn the game of Go from scratch. It learned by playing against itself. After 40 days of self-training, Alpha Go Zero was able to outperform the version of Alpha Go known as Master that has defeated [world number one Ke Jie](https://deepmind.com/alphago-china "world number one Ke Jie"). It only used black and white stones from the board as input features and a single neural network. A simple tree search that relies on the single neural network is used to evaluate positions moves and sample moves without using any [Monte Carlo](https://en.wikipedia.org/wiki/Monte_Carlo_method "Monte Carlo") rollouts.

>###强化学习在游戏中的应用

>RL在游戏领域中的应用备受关注，且极为成功，最典型的便是前些年人尽皆知的AlphaGoZero。通过强化学习，AlphaGoZero能够从头学习围棋游戏，并自我学习。经过40天的训练，AlphaGoZero的表现超过了世界排名第一的柯洁。该模型仅包含一个神经网络，且只将黑白棋子作为输入特征。由于网络单一，一个简单的树搜索算法被用来评估位置移动和样本移动，而无需任何蒙特卡罗展开。

###Real-time bidding— Reinforcement Learning applications in marketing and advertising

In this [paper](https://arxiv.org/pdf/1802.09756.pdf "paper"), the authors propose real-time bidding with multi-agent reinforcement learning. The handling of a large number of advertisers is dealt with using a clustering method and assigning each cluster a strategic bidding agent. To balance the trade-off between the competition and cooperation among advertisers, a Distributed Coordinated Multi-Agent Bidding (DCMAB) is proposed.

In marketing, the ability to accurately target an individual is very crucial. This is because the right targets obviously lead to a high return on investment. The study in this paper was based on [Taobao ](http://taobao.com/ "Taobao ")— the largest e-commerce platform in China. The proposed method outperforms the state-of-the-art single-agent reinforcement learning approaches.

>###实时竞价——强化学习在广告营销中的应用

>该论文提出了一种基于多智能体强化学习的实时竞价策略。对大量广告商进行聚类，然后为每个聚类分配一个策略投标智能体实现竞标。同时，为了平衡广告主之间的竞争与合作，论文还提出了分布式协同多智能体竞价（DCMAB）。

>在市场营销中，选择正确的目标全体才可带来高回报，因此个人精准定位至关重要的。论文以中国最大的电子商务平台淘宝网为研究对象，表明上述多智能体强化学习优于现有的单智能体强化学习方法。

###Reinforcement Learning in robotics manipulation

The use of deep learning and reinforcement learning [can train robots](https://ai.googleblog.com/2018/06/scalable-deep-reinforcement-learning.html "can train robots") that have the ability to grasp various objects — even those unseen during training. This can, for example, be used in building products in an assembly line.

This is achieved by combining large-scale distributed optimization and a variant of [deep Q-Learning](https://en.wikipedia.org/wiki/Q-learning "deep Q-Learning") called [QT-Opt](https://arxiv.org/abs/1806.10293 "QT-Opt"). QT-Opt support for continuous action spaces makes it suitable for robotics problems. A model is first trained offline and then deployed and fine-tuned on the real robot.

Google AI applied this approach to robotics grasping where 7 real-world robots ran for 800 robot hours in a 4-month period.

>###强化学习在机器人控制中的应用

>通过深度学习和强化学习方法训练机器人，可以使其能够抓取各种物体，甚至是训练中未出现过的物体。因此，可将其用于装配线上产品的制造。

>上述想法是通过结合大规模分布式优化和QT-Opt（一种深度Q-Learning变体）实现的。其中，QT-Opt支持连续动作空间操作，这使其可以很好处理机器人问题。在实践中，先离线训练模型，然后在真实的机器人上进行部署和微调。

>针对抓取任务，谷歌AI用了4个月时间，使用7个机器人运行了800机器人时。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603694361225807.png)

In this experiment, the QT-Opt approach succeeds in 96% of the grasp attempts across 700 trials grasps on objects that were previously unseen. Google AI’s previous method had a 78% success rate.

>实验表明，在700次实验中，QT-Opt方法有96%的概率成功抓取陌生的物体，而之前的方法仅有78%的成功率。

###Final thoughts

Whereas reinforcement learning is still a very active research area significant progress has been made to advance the field and apply it in real life.

In this article, we have barely scratched the surface as far as application areas of reinforcement learning are concerned. Hopefully, this has sparked some curiosity that will drive you to dive in a little deeper into this area. If you want to learn more check out this awesome repo — no pun intended, and this one as well.

>###总结

>强化学习是一个非常有趣且值得广泛研究的领域，RL技术的进步及其在现实各领域的应用势必将取得更大的成功。

>在本文中，我们粗略介绍了强化学习的不同领域应用。希望这能激发起你的好奇心，并引起你对RL的热爱和研究。如果想了解更多，推荐查看这两个项目：

https://github.com/aikorea/awesome-rl，https://github.com/dennybritz/reinforcement-learning。

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可，转载请附上原文出处链接和本声明。
本文链接地址：https://flyai.com/article/750

讨论

500字

表情

发送

删除确认

是否删除该条评论？

取消删除