Action-Value Function

Loading...

En provenance du cours de New York University Tandon School of Engineering

Reinforcement Learning in Finance

22 notes

New York University Tandon School of Engineering

Reinforcement Learning in Finance

22 notes

Cours 3 sur 4 dans Specialization Machine Learning and Reinforcement Learning in Finance

This course aims at introducing the fundamental concepts of Reinforcement Learning (RL), and develop use cases for applications of RL for option valuation, trading, and asset management. Prerequisites are the courses "Guided Tour of Machine Learning in Finance" and "Fundamentals of Machine Learning in Finance". Students are expected to know the lognormal process and how it can be simulated. Knowledge of option pricing is not assumed but desirable.

À partir de la leçon

MDP model for option pricing: Dynamic Programming Approach

MDP Formulation10:49

Action-Value Function5:40

Optimal Action From Q Function6:46

Backward Recursion for Q Star8:17

Rencontrer les enseignants

Igor Halperin

Now, let's talk about how we can compute the optimal policy in this model.

We already know that the optimal policy, pi*,

is a policy that maximizes the value function Vt.

And the optimal value function, V*,

satisfies the Bellman optimality equation, shown here.

If we know the dynamics, then this equation can be solved

using methods such as Value Iteration that we reviewed in our previous course.

But especially for teaching the second and

more interesting case when the dynamics are unknown, it turns out that

another version of the value function is more useful for such tasks.

This another version is called the action-value function.

The action-value function Q at times t is the function of two arguments,

rather than one argument Xt, as was the case for value function V of Xt.

The first argument in the Q-function is the same Xt,

while the second argument is the time t action at.

The Q-function is determined as an expectation of the same expression that

determines the value function V.

But this time it's conditioned on the first action at to be specifically

equal a, while all subsequent actions at will follow policy pi.

Now by using the same splitting in the last term here into the time t term and

sum over future values of t, we can obtain the Bellman equation for

the action-value function.

Now, this equation holds for the Q-function for any policy pi,

but we are more interested in finding the optimal policy, pi*.

This is defined in the same way as before, the optimal policy pi* should maximize

the action-value function as shown in the last equation here.

Now, the Bellman equation for the Q-function relates it

not with itself, but rather with the value function V.

But if we look at the optimal Q-function, Q*,

we can obtain the Bellman equation that involves only this function.

To this end, let's note that we actually have two equations here.

The first one is an equation for Q* that is obtained by replacing

pi by pi* in the Bellman equation for the Q-function.

But there is also a second equation here that says that the optimal value function,

V*, is obtained from Q* when its second argument at is chosen optimally.

So if we substitute the second of these equations into the first one,

we obtain an equation that contains only Q*.

This is called the Bellman optimality equation for the action-value function,

and it plays a central role in reinforcement learning.

The Bellman optimality equation is a backward equation that should be solved

backward in times starting from time capital T- 1.

Those who have a terminal condition for Q* in terms of the distribution of

the portfolio value, pi T at maturity T.

The optimal action at time t is therefore given by a value of

at that maximizes the Q-function.

This is sometimes referred to as a greedy policy.

It just picks a current action that maximizes the Q-function without

worrying how this action will impact other actions in the future.

Now, if we substitute the explicit form of the reward function

r into the Bellman optimality equation, we get this equation.

What this equation shows that the Q-function is quadratic in action at.

Therefore, it's easy to maximize with respect to at.

We can also check what happens with these formulas if we

take the limit of this conversion lambda going to 0.

In this limit, we get the first equation shown here.

But now we can replace the optimal Q-function with minus

the portfolio value whose expectation is exactly the mean

option price C hat as we discussed earlier.

Therefore, by using this and flipping the overall sign,

the first equation can be written as the second one.

But the second equation is something that we already saw.

It's a recursive relation for the mean option price that has right

Black-Scholes limit when the time steps delta t are very small.

Therefore, we see that our formulas reduce to the Black-Scholes

model when both these conversion lambda and time steps delta t vanish.

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2018 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play