In the last course of our specialization, Overview of Advanced Methods of Reinforcement Learning in Finance, we will take a deeper look into topics discussed in our third course, Reinforcement Learning in Finance. In particular, we will talk about links between Reinforcement Learning, option pricing and physics, implications of Inverse Reinforcement Learning for modeling market impact and price dynamics, and perception-action cycles in Reinforcement Learning. Finally, we will overview trending and potential applications of Reinforcement Learning for high frequency trading, cryptocurrencies, peer-to-peer lending, and more.
Modeling Feedback Frictions
Loading...
En provenance du cours de New York University Tandon School of Engineering
Overview of Advanced Methods of Reinforcement Learning in Finance
13 notes
New York University Tandon School of Engineering
13 notes
Cours 4 sur 4 dans Specialization Machine Learning and Reinforcement Learning in Finance
À partir de la leçon
Black-Scholes-Merton model, Physics and Reinforcement Learning
Rencontrer les enseignants
Igor Halperin
In this video, I would like to talk about feedback effects from trading.
Let me start with specifying what exactly we are going to discuss here.
It's known that large trades on stocks move the market.
Therefore, we need to incorporate such effects in our stock trading models,
that we apply for portfolio of stocks,
if we really want to deal with such big trades.
We talked about how such effects for a stock trading in last week of our previous course.
Now, I want to talk about price impact effects
from stock trading as applied to option trading.
A link between these two topics is provided by
the fact that an option derives its value from a stock,
and it's also hedged by the stock.
Now, this video is based on a few papers that I recommend
if you want to know more about models of such type.
One of them is a model by Li and Almgren from 2016 called,
Option Hedging with Smooth Market Impact.
This paper deals with intraday hedging of an option position by a large trader,
who has to worry about marketing of trades and made in a hedged portfolio of the option.
Almgren and Li tell the following remarkable story.
On the morning of July 19th, 2012,
in unusual sawtooth pattern was absorbed in US equity markets.
Four large stocks exhibited substantial price swings,
on the regular half-hour schedule.
Each stock hit local minimum price on each hour,
and a local maximum on each half hour,
as shown in this figure on the slide.
Now, no significant news were released on this day.
But CBOE options expiration,
was the next day.
Now, the most plausible explanation for this phenomenon is that,
this was the result of delta-hedge strategy,
executed by a large option position holder,
with no regard for market impact.
Each half hour, he or she evaluated
the necessary trades to obtain delta neutral position,
and executed this trade across the next half hour.
Market impact caused the price to move and the next evaluation,
at the new price the position was partially reversed.
In fact, this section is quite similar to
a forward Euler discretization of an ordinary differential equation,
which can introduce instability of exactly similar sort.
Similar phenomena are also observed in treasury markets.
So, what this phenomenon demonstrate is that there are
feedback effects from the option market to the stock market.
Such phenomena cannot be explained by
standard models for option prices such as the Black-Scholes model,
or its straightforward extensions because in these models,
stock market drives an option market,
but not vice versa.
A closely related effect is the so-called stock pinning.
Pinning at the strike refers to the likelihood that the price of a stock coincides with
a strike price of an option written on
the stock immediately before the expiration date of the option.
A graph that you see on the slide is taken from a paper by Avellaneda, Kasyan,
and Lipkin from 2013,
that extends the model proposed by Avellaneda and Lipkin in 2003.
This graph schematically shows trajectories of two stocks, A and B.
Stock A does not have an option, while stock B,
has an option with strike of 1200.50,
that expires on the fourth Friday of the month.
As this graph illustrates,
the price of stock be concise in this example with the strike of the option,
at the option expiry date.
In real markets, this phenomenon can be a bit less pronounced,
so that the two prices would not exactly coincide,
but still would be very close near the option maturity.
This is why this phenomenon is called stock pinning.
The stock price may become pinned at the option strike when there are
minutes left or less than a day left to the option maturity.
This is very remarkable,
because it provides a strong trading signal at this time.
In other words, stop dynamics becomes quite
predictable during such periods and therefore also potentially exploitable.
The Avellaneda and Lipkin model,
is based on the behavior of
option market makers that impact the stock price by delta-hedging their positions.
In their first paper,
they considered a linear impact function.
While in the second paper,
they extended the analysis to a general power-law.
The impact model that they analyzed is shown in the formula on the slide.
It says that the price impact or relative change over
a stock price is proportional to the size of the trade delta x,
raised to power p, and multiplied by an elasticity parameter E. What they found is
that there are qualitative differences in the model behavior for
different values of parameter p. If p is less or equal than one half,
there is no pinning phenomenon anymore.
It only exists if p is larger than half.
Using the language of physics,
this means as a phase transition in the system.
In the sense that the behavior of the model
is qualitatively different in these two regimes.
This is all implies infects on the very profound,
that the classical financial theory,
that corresponds to the limit,
when p goes to zero is likely to be a non-analytical limit.
This limit describes different system from the actual financial markets.
Finally, I want to briefly mention a few more related models.
The first one, is a model of small region with market impacts,
by Li and Almgren,
from 2016, which I already mentioned earlier.
They built a model similar to models for optimal stock of execution,
that we discussed in the first week of the previous course.
In their searching to target portfolio is defined
by the value of the Black-Scholes Delta.
Risk function is quadratic and is based on a market,
it's like mean variance analysis.
The same approach that was used in the QLBS model that I presented earlier.
While Li and Almgren focus on analytically tractable settings,
a similar framework, but with a more general setting was
developed by Gueant and Pu in 2015.
Their approach relies on a utility based approach,
and the amounts numerically to solving an HJD equation,
which is computational heavy.
All these topics that we discussed in this lesson,
finally bring us back to what we
started with in this lesson namely, reinforcement learning.
As the whole field of using reinforcement learning for
option pricing in hedging is still in its infancy.
There are no yet papers that would compare
reinforcement learning with classical methods on these types of tasks.
But, at least the direction of how it can be done,
is shown by a simple framework of the QLBS model,
that I presented in the previous course.
This model uses a very simple quadratic utility,
that does not take into account feedback effects and transaction costs,
but it can be extended in a straightforward way to incorporate this effects.
Indeed, feedback defects are in general assumed in reinforcement learning tasks.
Hedging, market impact, and transaction costs simply
amounts to modifying the reward function in the reinforcement learning setting,
keeping the rest of the methodology intact.
This is exactly why reinforcement learning appears to
be a very natural and computationally attractive approach to option pricing,
hedging can and trading.
On this note, I would like to round up this week of our course.
In the next week,
we will finally leave the option world,
and talk more about the stock world. See you there.
Explorer notre catalogue
Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.
Coursera propose un accès universel à la meilleure formation au monde,
en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.
© 2018 Coursera Inc. Tous droits réservés.
