Fundamental Iterative Methods of Reinforcement Learning

栏目: IT技术 · 发布时间: 5年前

Fundamental Iterative Methods of Reinforcement Learning

Leading towards reinforcement learning

Value Iteration

Learn the values for all states, then we can act according to the gradient. Value iteration learns the value of the states from the Bellman Update directly. The Bellman Update is guaranteed to converge to optimal values, under some non-restrictive conditions.

Learning a policy may be more direct than learning a value. Learning a value may take an infinite amount of time to converge to numerical precision of a 64bit float (think about a moving average averaging in a constant at every iteration, after starting with an estimate of 0, it will add a smaller and smaller nonzero number forever).

Policy Iteration

Learn a policy in tandem to the values. Policy learning incrementally looks at the current values and extracts a policy. Because the action space is finite , the hope is that it can converge faster than Value Iteration. Conceptually, the last change to the actions will happen well before the small rolling-average updates end. There are two steps to Policy Iteration.

The first is called Policy Extraction , which is how you go from a value to a policy — by taking the policy that maximizes over expected values.

Policy extraction step.

The second step is Policy Evaluation . Policy evaluation takes a policy and runs value iteration conditioned on a policy . The samples are forever tied to the policy, but we know we have to run the iterative algorithms for way fewer steps to extract the relevant action information .

Policy evaluation step.

Like value iteration, policy iteration is guaranteed to converge for most reasonable MDPs because of the underlying Bellman Update.

Q-value Iteration

The problem with knowing optimal values is that it can be hard to distill a policy from it. The argmax operator is distinctly nonlinear and difficult to optimize over, so Q-value Iteration takes a step towards direct policy extraction . The optimal policy at each state is simply the max q-value at that state.

Q-learning of an MDP.

The reason most instruction starts with Value Iteration is that it slots into the Bellman updates a little more naturally. Q-value Iteration requires the substitution of two of the key MDP value relations together . After doing so, it is one step removed from Q-learning, which we will get to know.


以上所述就是小编给大家介绍的《Fundamental Iterative Methods of Reinforcement Learning》,希望对大家有所帮助,如果大家有任何疑问请给我留言,小编会及时回复大家的。在此也非常感谢大家对 码农网 的支持!

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

计算机程序设计艺术卷1:基本算法(英文版.第3版)

计算机程序设计艺术卷1:基本算法(英文版.第3版)

Donald E.Knuth / 人民邮电出版社 / 2010-10 / 119.00元

《计算机程序设计艺术》系列著作对计算机领域产生了深远的影响。这一系列堪称一项浩大的工程,自1962年开始编写,计划出版7卷,目前已经出版了4卷。《美国科学家》杂志曾将这套书与爱因斯坦的《相对论》等书并列称为20世纪最重要的12本物理学著作。目前Knuth正将毕生精力投入到这部史诗性著作的撰写中。想了解本书最新信息,请访http://www-cs-faculty.stanford.edu/~knut......一起来看看 《计算机程序设计艺术卷1:基本算法(英文版.第3版)》 这本书的介绍吧!

图片转BASE64编码
图片转BASE64编码

在线图片转Base64编码工具

Markdown 在线编辑器
Markdown 在线编辑器

Markdown 在线编辑器

html转js在线工具
html转js在线工具

html转js在线工具