终极版AlphaGo，DeepMind新算法MuZero作者解读-FlyAI

终极版AlphaGo，DeepMind新算法MuZero作者解读

深度学习机器学习

最后更新 2021-01-04 15:39 阅读 13872

最后更新 2021-01-04 15:39

阅读 13872

深度学习机器学习

##FlyAI文献翻译

**英文原文：**[：MuZero Intuition
](http://www.furidamu.org/blog/2020/12/22/muzero-intuition/)

> **标签** ：强化学习

To celebrate the publication of our MuZero paper in Nature, I've written a high level description of the MuZero algorithm. My focus here is to give you an intuitive understanding and general overview of the algorithm; for the full details please read the paper.

MuZero is a very exciting step forward - it requires no special knowledge of game rules or environment dynamics, instead learning a model of the environment for itself and using this model to plan. Even though it uses such a learned model, MuZero preserves the full planning performance of AlphaZero - opening the door to applying it to many real world problems!

>为庆祝Muzero论文在[Nature](https://www.nature.com/articles/s41586-020-03051-4 "Nature")上的发表，我特意写了这篇文章对MuZero算法进行详细介绍，希望本人能让你对该算法有一个直观的了解。更多详细信息请阅读[原文](http://www.furidamu.org/publications/ "原文")。

>MuZero是令人振奋的一大步，该算法摆脱了对游戏规则或环境动力学的知识依赖，可以自行学习环境模型并进行规划。即使如此，MuZero仍能够实现AlphaZero的全部功能——这显示出其在许多实际问题的应用可能性！

###It's all just statistics

MuZero is a machine learning algorithm, so naturally the first thing to understand is how it uses neural networks. From AlphaGo and AlphaZero, it inherited the use of policy and value networks1:

>MuZero是一种机器学习算法，因此自然要先了解它是如何使用神经网络的。简单来说，该算法使用了AlphaGo和AlphaZero的策略网络和值网络：

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608861677673583.jpg)

Both the policy and the value have a very intuitive meaning:

>策略和值的直观含义如下：

The policy, written p(s,a), is a probability distribution over all actions aa that can be taken in state s. It estimates which action is likely to be the optimal action. The policy is similar to the first guess for a good move that a human player has when quickly glancing at a game.

The value v(s) estimates the probability of winning from the current state ss: averaging over all possible future possibilities, weighted by how likely they are, in what fraction of them would the current player win?

Each of these networks on their own is already very powerful: If you only have a policy network, you could simply always play the move it predicts as most likely and end up with a very decent player. Similarly, given only a value network, you could always choose the move with the highest value. However, combining both estimates leads to even better results.

>策略p(s,a)表示在状态s时所有可能的动作a分布，据此可以估计最优的动作。类比人类玩家，该策略相当于快速浏览游戏时拟采取的可能动作。

>值v(s)估计在当前状态s下获胜的可能性，即通过对所有的未来可能性进行加权平均，确定当前玩家的获胜概率。

>这两个网络任何一个都非常强大：只根据策略网络，能够轻易预测每一步的动作，最终得到良好结果；只依赖值网络，始终选择值最高的动作。但是，将这两个估计结合起来可以得到更好的结果。

###Planning to Win

Similar to AlphaGo and AlphaZero before it, MuZero uses Monte Carlo Tree Search2, short MCTS, to aggregate neural network predictions and choose actions to apply to the environment.

MCTS is an iterative, best-first tree search procedure. Best-first means expansion of the search tree is guided by the value estimates in the search tree. Compared to classic methods such as breadth-first (expand the entire tree up to a fixed depth before searching deeper) or depth-first (consecutively expand each possible path until the end of the game before trying the next), best-first search can take advantage of heuristic estimates (such as neural networks) to find promising solutions even in very large search spaces.

MCTS has three main phases: simulation, expansion and backpropagation. By repeatedly executing these phases, MCTS incrementally builds a search tree over future action sequences one node at a time. In this tree, each node is a future state, while the edges between nodes represent actions leading from one state to the next.

>###取胜之路

>与AlphaGo和AlphaZero相似，MuZero也使用蒙特卡洛树搜索方法（MCTS）汇总神经网络预测并选择适合当前环境的动作。

>MCTS是一种迭代的，最佳优先的树搜索过程。最佳优先意味着搜索树的扩展依赖于搜索树的值估计。与经典方法（如广度优先或深度优先）相比，最佳优先搜索利用启发式估计（如神经网络），这使其在很大的搜索空间中也可以找到有效的解决方案。

>MCTS具有三个主要阶段：模拟，扩展和反向传播。通过重复执行这些阶段，MCTS根据节点可能的动作序列逐步构建搜索树。在该树中，每个节点表示未来状态，而节点间的边缘表示从一个状态到下一个状态的动作。

Before we dive into the details, let me introduce a schematic representation of such a search tree, including the neural network predictions made by MuZero:

>在深入研究之前，首先对该搜索树及逆行介绍，包括MuZero做出的神经网络预测：

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608861721457576.jpg)

Circles represent nodes of the tree, which correspond to states in the environment. Lines represent actions, leading from one state to the next. The tree is rooted at the top, at the current state of the environment - represented by a schematic Go board. We will cover the details of representation, prediction and dynamics functions in a later section.

>圆圈表示树节点，对应环境状态；线表示从一个状态到下一个状态的动作；根节点为当前环境状态，即围棋面板状态。后续章节我们会详细介绍预测和动力学函数。

Simulation always starts at the root of the tree (light blue circle at the top of the figure), the current position in the environment or game. At each node (state s), it uses a scoring function U(s,a) to compare different actions aa and chose the most promising one. The scoring function used in MuZero would combine a prior estimate p(s,a) with the value estimate for v(s′):

>模拟：从树的根节点出发（图顶部的淡蓝色圆圈），即环境或游戏的当前位置。在每个节点（状态s），使用评分函数U(s,a)比较不同的动作a，并选择最优动作。MuZero中使用的评分函数是将先前的估计p(s,a)与v(s')的值结合起来，即

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608861925696616.png)

where c is a scaling factor3 that ensures that the influence of the prior diminishes as our value estimate becomes more accurate.

Each time an action is selected, we increment its associated visit count n(s,a), for use in the UCB scaling factor c and for later action selection.

Simulation proceeds down the tree until it reaches a leaf that has not yet been expanded; at this point the neural network is used to evaluate the node. Evaluation results (prior and value estimates) are stored in the node.

Expansion: Once a node has reached a certain number of evaluations, it is marked as "expanded". Being expanded means that children can be added to a node; this allows the search to proceed deeper. In MuZero, the expansion threshold is 1, i.e. every node is expanded immediately after it is evaluated for the first time. Higher expansion thresholds can be useful to collect more reliable statistics4 before searching deeper.

Backpropagation: Finally, the value estimate from the neural network evaluation is propagated back up the search tree; each node keeps a running mean of all value estimates below it. This averaging process is what allows the UCB formula to make increasingly accurate decisions over time, and so ensures that the MCTS will eventually converge to the best move.

>其中c是比例因子，随着值估计准确性的增加，减少先验的影响。

>每选择一个动作，我们都会增加其相关的访问计数n(s,a)，以用于UCB比例因子c以及之后的动作选择。

>模拟沿着树向下进行，直到尚未扩展的叶子。此时，应用神经网络评估节点，并将评估结果（优先级和值估计）存储在节点中。

>扩展：一旦节点达到估计量值后，将其标记为“扩展”，意味着可以将子级添加到节点，以便进行更深入的搜索。在MuZero中，扩展阈值为1，即每个节点在首次评估后都会立即扩展。在进行更深入的搜索之前，较高的扩展阈值可用于收集更可靠的统计信息。

>反向传播：最后，将神经网络的值估计传播回搜索树，每个节点都在其下保存所有值估计的连续均值，这使得UCB公式可以随着时间的推移做出越来越准确的决策，从而确保MCTS收敛到最优动作。

###Intermediate Rewards

The astute reader may have noticed that the figure above also includes the prediction of a quantity r. Some domains, such as board games, only provide feedback at the end of an episode (e.g. a win/loss result); they can be modeled purely through value estimates. Other domains however provide more frequent feedback, in the general case a reward r is observed after every transition from one state to the next.

Directly modeling this reward through a neural network prediction and using it in the search is advantageous. It only requires a slight modification to the UCB formula:

>细心的读者可能已经注意到，上图还包括r的预测。某一情况（如棋盘游戏）在完全结束后提供反馈（获胜/失败结果），这样可以通过值估计进行建模。但在另外一些情况下，会存在频繁的反馈，即每次从一种状态转换到另一种状态后，都会得到回报r。

>只需对UCB公式进行简单修改，就可以通过神经网络预测直接对奖励进行建模，并将其用于搜索。

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608861886446692.png)

where r(s,a) is the reward observed in transitioning from state s by choosing action a, and γ is a discount factor that describes how much we care about future rewards.

Since in general rewards can have arbitrary scale, we further normalize the combined reward/value estimate to lie in the interval [0,1] before combining it with the prior:

>其中，r(s,a)是指在状态s时执行动作a后观察到的奖励，而折扣因子γ是指对未来奖励的关注程度。

>由于总体奖励可以时任意量级的，因此在将其与先验奖励组合之前，我们将奖励/值估计归一化为区间[0,1]：

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608861855650631.png)

where qmin and qmax are the minimum and maximum r(s,a)+γ⋅v(s′) estimates observed across the search tree.

>其中，q_min和q_max分别是整个搜索树中观察到的最小和最大r(s,a)+γ⋅v(s')估计。

Episode Generation

The MCTS procedure described above can be applied repeatedly to play entire episodes:

Run a search in the current state stst of the environment.

Select an action at+1 according to the statistics πt of the search.

Apply the action to the environment to advance to the next state st+1 and observe reward ut+1.

Repeat until the environment terminates.

>过程生成

>重复执行以下过程可实现上述MCTS：

>在当前环境状态下进行搜索；

>根据搜索的统计信息π_t选择一个动作a_(t+1);

>根据该动作更新环境，得到新的状态s_(t+1)和奖励u(t+1)；

>重复上述过程。

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608862045760305.jpg)

Action selection can either be greedy - select the action with the most visits - or exploratory: sample action aa proportional to its visit count n(s,a), potentially after applying some temperature tt to control the degree of exploration:

>动作的选择可以是贪心的（选择访问次数最多的动作），也可以是探索性的：通过一定的温度t控制探索程度，并对与访问次数n(s,a)成比例的动作a进行采样：

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608862079421626.png)

For t=0, we recover greedy action selection; t=inf is equivalent to sampling actions uniformly.

>当t = 0时，等效贪婪采样；当t = inf时，等效均匀采样。

Training
Now that we know how to run MCTS to select actions, interact with the environment and generate episodes, we can turn towards training the MuZero model.

We start by sampling a trajectory and a position within it from our dataset, then we unroll the MuZero model alongside the trajectory:

>现在，我们已经学会了运行MCTS来选择动作，并与环境互动生成过程，接下来就可以训练MuZero模型了。

>首先，从数据集中采样一条轨迹和一个位置，然后根据该轨迹运行MuZero模型：

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608862114411079.jpg)

You can see the three parts of the MuZero algorithm in action:

the representation function h maps from a set of observations (the schematic Go board) to the hidden state ss used by the neural network

the dynamics function g maps from a state stst to the next state st+1 based on an action at+1. It also estimates the reward rtrt observed in this transition. This is what allows the learned model to be rolled forward inside the search.

the prediction function f makes estimates for policy pt and value vt based on a state st. These are the estimates used by the UCB formula and aggregated in the MCTS.

>可以看到，MuZero算法由以下三部分组成：

>表示函数h将一组观察值（棋盘）映射到神经网络的隐藏状态s；

>动态函数g根据动作a_(t + 1)将状态s_t映射到下一个状态s_(t + 1)，同时估算在此过程的回报r_t，这样模型就能够不断向前扩展；

>预测函数f根据状态s_t对策略p_t和值v_t进行估计，应用UCB公式并将其汇入MCTS过程。

The observations and actions used as input to the network are taken from this trajectory; similarly the prediction targets for policy, value and reward are the ones stored with the trajectory when it was generated.

You can see this alignment between episode generation (B) and training (C) even more clearly in the full figure:

>根据轨迹选择用于网络输入的观测值和动作。相应地，策略、值和奖励的预测目标是在生成存储的轨迹。

>从下图可以看到过程生成（B）与训练（C）之间的一致性：

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608862182291049.jpg)

Specifically, the training losses for the three quantities estimated by MuZero are:

policy: cross-entropy between MCTS visit count statistics and policy logits from the prediction function.

value: cross-entropy or mean squared error between discounted sum of N rewards + stored search value or target network estimate and value from the prediction function5.

reward: cross-entropy between reward observed in the trajectory and dynamics function estimate.

>具体问言，MuZero估计量的训练损失为：

>策略：MCTS访问统计信息与预测函数的策略logit之间的交叉熵；

>值：N个奖励的折扣和+搜索值/目标网络估计值与预测函数的值之间的交叉熵或均方误差；

>奖励：轨迹观测奖励与动态函数估计之间的交叉熵。

Reanalyse
Having examined the core MuZero training, we are ready to take a look at the technique that allows us to leverage the search to achieve massive data-efficiency improvements: Reanalyse.

In the course of normal training, we generate many trajectories (interactions with the environment) and store them in our replay buffer for training. Can we get more mileage out of this data?

>重分析

>在了解了MuZero的核心思想后，接下来我们将介绍重分析技术，这将显著提高模型对大量数据的搜索效率。

>在一般训练过程中，通过与环境的相互作用，我们会生成许多轨迹，并将其存储在重播缓冲区用于训练。那么，我们可以从该数据中获得更多信息吗？

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608862196159620.jpg)

Unfortunately, since this is stored data, we cannot change the states, actions or received rewards - this would require resetting the environment to an arbitrary state and continuing from there. Possible in The Matrix, but not in the real world.

Luckily, it turns out that we don't need to - using existing inputs with fresh, improved labels is enough for continued learning. Thanks to MuZero's learned model and the MCTS, this is exactly what we can do:

>很难。由于需要与环境交互，我们无法更改存储数据的状态、动作或奖励。在《黑客帝国》中可能做到，但在现实世界中则不可能。

>幸运的是，我们并不需要这样。只要使用更新的、改进标签的现有输入，就足以继续学习。考虑到MuZero模型和MCTS，我们做出如下改进：

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608862206144747.jpg)

We keep the saved trajectory (observations, actions and rewards) as is and instead only re-run the MCTS. This generates fresh search statistics, providing us with new targets for the policy and value prediction.

In the same way that searching with an improved network results in better search statistics when interacting with the environment directly, re-running the search with an improved network on saved trajectories also results in better search statistics, allowing for repeated improvements using the same trajectory data.

Reanalyse fits naturally into the MuZero training loop. Let's start with the normal training loop:

>保持轨迹（观测、动作和奖励）不变，重新运行MCTS，就可以生成新的搜索统计信息，从而提供策略和值预测的新目标。

>我们知道，在与环境直接交互过程中，使用改进网络进行搜索会获得更好的统计信息。与之相似，在已有轨迹上使用改进网络重新搜索也会获得更好的统计信息，从而可以使用相同的轨迹数据重复改进。

>重分析适用于MuZero训练，一般训练循环如下：

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608862217182153.jpg)

We have two sets of jobs that communicate with each other asynchronously:

a learner that receives the latest trajectories, keeps the most recent of these in a replay buffer and uses them to perform the training algorithm described above.

multiple actors which periodically fetch the latest network checkpoint from the learner, use the network in MCTS to select actions and interact with the environment to generate trajectories.

To implement reanalyse, we introduce two jobs:

>设置两组异步通信任务：

>一个学习者接收最新轨迹，将最新轨迹保存在重播缓冲区，并根据这些轨迹进行上述训练；

>多个行动者定期从学习者那里获取最新的网络检查点，并使用MCTS中的网络选择动作，与环境进行交互生成轨迹。

>为实现重分析，引入两个新任务：

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608862229651270.jpg)

a reanalyse buffer that receives all trajectories generated by the actors and keeps the most recent ones.

multiple reanalyse actors 6 that sample stored trajectories from the reanalyse buffer, re-run MCTS using the latest network checkpoints from the learner and send the resulting trajectories with updated search statistics to the learner.

For the learner, "fresh" and reanalysed trajectories are indistinguishable; this makes it very simple to vary the proportion of fresh vs reanalysed trajectories.

>重分析缓冲区，用于接收参与者生成的所有轨迹并保留最新轨迹；

>多个重分析行动者从重分析缓冲区采样存储的轨迹，使用学习者的最新网络检查点重新运行MCTS，并将生成的轨迹和更新的统计信息发送给学习者。

>由于学习者无法区分新轨迹和重分析的轨迹，这使得新轨迹与重分析轨迹的比例更改变得简单。

What's in a name?
MuZero's name is of course based on AlphaZero - keeping the Zero to indicate that it was trained without imitating human data, and replacing Alpha with Mu to signify that it now uses a learned model to plan.

Digging a little deeper, we find that Mu is rich in meaning:

夢, which can be read as mu in Japanese, means 'dream' - just like MuZero uses the learned model to imagine future scenarios.

the greek letter μ, pronounced mu, can also stand for the learned model.

無, pronunced mu in Japanese, means 'nothing' - doubling down on the notion of learning from scratch: not just no human data to imitate, but not even provided with the rules.

>MuZero的命名基于AlphaZero，其中Zero表示是在没有模仿人类数据的情况下进行训练的，Mu取代Alpha表示使用学习模型进行规划。

>更研究一些，Mu还有其他丰富的含义：

>夢，日语中读作mu，表示“梦”的意思， 就像MuZero通过学习的模型来想象未来状况一样；
希腊字母μ（发音为mu）也可以表示学习的模型；

>無， 日语发音为mu，表示“无、没有”，这强调从头学习的概念：不仅无需模仿人类数据，甚至不需提供规则。

###Final Words

I hope this summary of MuZero was useful!

If you are interested in more details, start with the full paper. I also gave talks about MuZero at NeurIPS (poster) and most recently at ICAPS.

Let me finish by linking some articles, blog posts and GitHub projects from other researchers that I found interesting:

A Simple Alpha(Go) Zero Tutorial

MuZero General implementation

How To Build Your Own MuZero AI Using Python

>###结语

>希望本文对MuZero的介绍对你有所启发！

>如果想了解更多细节，可以阅读[原文](https://www.nature.com/articles/s41586-020-03051-4 "原文")，还可以查看我在[NeurIPS](https://www.youtube.com/watch?v=vt5jOSy7cz8&t=2s "NeurIPS")的[poster](https://storage.googleapis.com/deepmind-media/research/muzero_poster_neurips_2019.pdf "poster")以及在[ICAPS](https://www.youtube.com/watch?v=L0A86LmH7Yw "ICAPS")上发表的关于MuZero的演讲。

>最后，分享给你一些其他研究人员的文章，博客和GitHub项目：

>[A Simple Alpha(Go) Zero Tutorial](https://web.stanford.edu/~surag/posts/alphazero.html "A Simple Alpha(Go) Zero Tutorial")

>[MuZero General implementation](https://github.com/werner-duvaud/muzero-general "MuZero General implementation")

[>How To Build Your Own MuZero AI Using Python](https://medium.com/applied-data-science/how-to-build-your-own-muzero-in-python-f77d5718061a ">How To Build Your Own MuZero AI Using Python")

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可，转载请附上原文出处链接和本声明。
本文链接地址：https://flyai.com/article/798

讨论

500字

表情

发送

删除确认

是否删除该条评论？

取消删除