Xiao-Yang Liu
ABSTRACT
Deep reinforcement learning (DRL) has been envisioned to have a competitive edge in quantitative finance. However, there is a steep development curve for quantitative traders to obtain an agent that automatically positions to win in the market, namely to decide where to trade, at what price and what quantity, due to the error-prone programming and arduous debugging. In this paper, we present the first open-source framework FinRL as a full pipeline to help quantitative traders overcome the steep learning curve. FinRL is featured with simplicity, applicability and extensibility under the key principles, full-stack framework, customization, reproducibility and hands-on tutoring.
Embodied as a three-layer architecture with modular structures, FinRL implements fine-tuned state-of-the-art DRL algorithms and common reward functions, while alleviating the debugging workloads. Thus, we help users pipeline the strategy design at a high turnover rate. At multiple levels of time granularity, FinRL simulates various markets as training environments using historical data and live trading APIs. Being highly extensible, FinRL reserves a set of user-import interfaces and incorporates trading constraints such as market friction, market liquidity and investor's risk-aversion. Moreover, serving as practitioners' stepping stones, typical trading tasks are provided as step-by-step tutorials, e.g., stock trading, portfolio allocation, cryptocurrency trading, etc.
- Joshua Achiam. 2018. Spinning up in deep reinforcement learning. https://spinningup.openai.comGoogle Scholar
- Andrew Ang. August 10, 2012. Mean-variance investing. Columbia Business School Research Paper No. 12/49. (August 10, 2012).Google Scholar
- Wenhang Bao and Xiao-Yang Liu. 2019. Multi-agent deep reinforcement learning for liquidation strategy analysis. ICML Workshop on Applications and Infrastructure for Multi-Agent Learning (2019).Google Scholar
- Stelios D Bekiros. 2010. Fuzzy adaptive decision-making for boundedly rational traders in speculative stock markets. European Journal of Operational Research 202, 1 (2010), 285--293.Google ScholarCross Ref
- Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI gym. arXiv preprint arXiv:1606.01540 (2016).Google Scholar
- Hans Buehler, Lukas Gonon, Josef Teichmann, Ben Wood, Baranidharan Mohan, and Jonathan Kochems. 2019. Deep hedging: Hedging derivatives under generic market frictions using reinforcement learning. Swiss Finance Institute Research Paper 19--80 (2019).Google ScholarCross Ref
- Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. 2018. Dopamine: A research framework for deep reinforcement learning. http://arxiv.org/abs/1812.06110 (2018).Google Scholar
- Ltd China Securities Index Co. 2017. CSI 300. http://www.csindex.com.cn/uploads/indices/detail/files/en/145_000300_Fact_Sheet_en.pdfGoogle Scholar
- Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren, and Qionghai Dai. 2016. Deep direct reinforcement learning for financial signal representation and trading. IEEE Transactions on Neural Networks and Learning Systems 28, 3 (2016), 653--664.Google ScholarCross Ref
- Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. 2017. OpenAI baselines. https://github.com/openai/baselines.Google Scholar
- Hao Dong, Akara Supratak, Luo Mai, Fangde Liu, Axel Oehmichen, Simiao Yu, and Yike Guo. 2017. TensorLayer: A versatile library for efficient deep learning development. In Proceedings of the 25th ACM International Conference on Multimedia. 1201--1204.Google ScholarDigital Library
- Shanghai Stock Exchange. 2018. SSE 180 Index Methodology. http://www.sse.com.cn/market/sseindex/indexlist/indexdetails/indexmethods/c/IndexHandbook_EN_SSE180.pdfGoogle Scholar
- Thomas G. Fischer. 2018. Reinforcement learning in financial markets-asurvey. FAU Discussion Papers in Economics. Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics.Google Scholar
- Scott Fujimoto, Herke Van Hoof, and David Meger. 2018. Addressing function approximation error in actor-critic methods. International Conference on Machine Learning (2018).Google Scholar
- Prakhar Ganesh and Puneet Rakheja. 2018. Deep reinforcement learning in high frequency trading. ArXiv abs/1809.01506 (2018).Google Scholar
- Sumitra Ganesh, Nelson Vadori, Mengda Xu, Hua Zheng, Prashant Reddy, and Manuela Veloso. 2019. Reinforcement learning for market making in a multi-agent dealer market. NeurIPS'19 Workshop on Robust AI in Financial Services.Google Scholar
- Mao Guan and Xiao-Yang Liu. 2021. Explainable deep reinforcement learning for portfolio management: an empirical approach. ACM International Conference on AI in Finance (ICAIF) (2021).Google ScholarDigital Library
- Chien Yi Huang. 2018. Financial trading as a game: A deep reinforcement learning approach. arXiv preprint arXiv:1807.02787 (2018).Google Scholar
- Hang Seng Index. 2020. Hang Seng index and sub-indexes. https://www.hsi.com.hk/eng/indexes/all-indexes/hsiGoogle Scholar
- Zhengyao Jiang and J. Liang. 2017. Cryptocurrency portfolio management with deep reinforcement learning. Intelligent Systems Conference (IntelliSys) (2017), 905--913.Google Scholar
- Zhengyao Jiang, Dixing Xu, and J. Liang. 2017. A deep reinforcement learning framework for the financial portfolio management problem. ArXiv abs/1706.10059.Google Scholar
- Prahlad Koratamaddi, Karan Wadhwani, Mridul Gupta, and Sriram G. Sanjeevi. 2021. Market sentiment-aware deep reinforcement learning approach for stock portfolio allocation. Engineering Science and Technology, an International Journal.Google Scholar
- Mark Kritzman and Yuanzhen Li. 2010. Skulls, financial turbulence, and risk management. Financial Analysts Journal 66 (10 2010).Google Scholar
- Zechu Li, Xiao-Yang Liu, jiahao Zheng, Zhaoran Wang, Anwar Walid, and Jian Guo. 2021. FinRL-Podracer: High performance and scalable deep reinforcement learning for quantitative finance. ACM International Conference on AI in Finance (ICAIF) (2021).Google ScholarDigital Library
- Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. 2018. RLlib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning (ICML).Google Scholar
- Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2016. Continuous control with deep reinforcement learning. ICLR (2016).Google Scholar
- Siyu Lin and P. Beling. 2020. A deep reinforcement learning framework for optimal trade execution. In ECML/PKDD.Google Scholar
- Xiao-Yang Liu, Zechu Li, Zhaoran Wang, and Jiahao Zheng. 2021. ElegantRL: A scalable and elastic deep reinforcement learning library. https://github.com/AI4Finance-Foundation/ElegantRL.Google Scholar
- Xiao-Yang Liu, Zechu Li, Zhuoran Yang, Jiahao Zheng, Zhaoran Wang, Anwar Walid, Jian Guo, and Michael Jordan. 2021. ElegantRL-Podracer: Scalable and elastic library for cloud-native deep reinforcement learning. Deep RL Workshop, NeurIPS 2021 (2021).Google Scholar
- Xiao-Yang Liu, Jingyang Rui, Jiechao Gao, Liuqing Yang, Hongyang Yang, Zhaoran Wang, Christina Dan Wang, and Guo Jian. 2021. Data-driven deep reinforcement learning in quantitative finance. Data-Centric AI Workshop, NeurIPS.Google Scholar
- B. G. Malkiel. 2003. Passive investment strategies and efficient markets. European Financial Management 9 (2003), 1--10.Google ScholarCross Ref
- Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning. 1928--1937.Google ScholarDigital Library
- John Moody and Matthew Saffell. 2001. Learning to trade via direct reinforcement. IEEE Transactions on Neural Networks 12, 4 (2001), 875--889.Google ScholarDigital Library
- J. Moody, L. Wu, Y. Liao, and M. Saffell. 1998. Performance functions and reinforcement learning for trading systems and portfolios. Journal of Forecasting 17 (1998), 441--470.Google ScholarCross Ref
- Abhishek Nan, Anandh Perumal, and Osmar R Zaiane. 2020. Sentiment and knowledge based algorithmic trading with deep reinforcement learning. ArXiv abs/2001.09403 (2020).Google Scholar
- Quantopian. 2019. Pyfolio: A toolkit for portfolio and risk analytics in Python. https://github.com/quantopian/pyfolio.Google Scholar
- Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. 2019. Stable Baselines3. https://github.com/DLR-RM/stable-baselines3.Google Scholar
- Francesco Rundo. 2019. Deep LSTM with reinforcement learning layer for financial trend prediction in FX high frequency trading systems. Applied Sciences 9 (10 2019), 1--18.Google Scholar
- Jonathan Sadighian. 2019. Deep reinforcement learning in cryptocurrency market making. arXiv: Trading and Market Microstructure (2019).Google Scholar
- Svetlana Sapuric and A. Kokkinaki. 2014. Bitcoin is volatile! Isn't that right?. In BIS.Google Scholar
- Otabek Sattarov, Azamjon Muminov, Cheol Lee, Hyun Kang, Ryumduck Oh, Junho Ahn, Hyung Oh, and Heung Jeon. 2020. Recommending cryptocurrency trading points with deep reinforcement learning approach. Applied Sciences 10.Google Scholar
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).Google Scholar
- William F Sharpe. 1970. Portfolio theory and capital markets. McGraw-Hill College.Google Scholar
- David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529, 7587 (2016), 484.Google Scholar
- Richard S. Sutton, David Mcallester, Satinder Singh, and Yishay Mansour. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12. MIT Press, 1057--1063.Google Scholar
- Evgeni B. Tarassov. 2016. Exchange traded funds (ETF): History, mechanism, academic literature review and research perspectives. Microeconomics: General Equilibrium & Disequilibrium Models of Financial Markets eJournal (2016).Google Scholar
- Nelson Vadori, Sumitra Ganesh, Prashant Reddy, and Manuela Veloso. 2020. Risk-sensitive reinforcement learning: a martingale approach to reward uncertainty. International Conference on AI in Finance (ICAIF) (2020).Google ScholarDigital Library
- Svitlana Vyetrenko, David Byrd, Nick Petosa, Mahmoud Mahfouz, Danial Dervovic, Manuela Veloso, and Tucker Hybinette Balch. 2020. Get real: Realism metrics for robust limit order book market simulations. International Conference on AI in Finance (ICAIF) (2020).Google ScholarDigital Library
- Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine Learning 8, 3--4 (1992), 279--292.Google ScholarDigital Library
- Zhuoran Xiong, Xiao-Yang Liu, Shan Zhong, Hongyang Yang, and Anwar Walid. 2018. Practical deep reinforcement learning approach for stock trading. NeurIPS Workshop (2018).Google Scholar
- Hongyang Yang, Xiao-Yang Liu, Shan Zhong, and Anwar Walid. 2020. Deep reinforcement learning for automated stock trading: An ensemble strategy. ACM International Conference on AI in Finance (ICAIF) (2020).Google ScholarDigital Library
- Daochen Zha, Kwei-Herng Lai, Kaixiong Zhou, and X. X. Hu. 2019. Experience replay optimization. International Joint Conference on Artificial Intelligence (IJCAI).Google Scholar
- Yong Zhang and Xingyu Yang. 2017. Online portfolio selection strategy based on combining experts' advice. Computational Economics 50, 1 (2017), 141--159.Google ScholarDigital Library
- Zihao Zhang, Stefan Zohren, and Stephen Roberts. 2020. Deep reinforcement learning for trading. The Journal of Financial Data Science 2, 2 (2020), 25--40.Google ScholarCross Ref
Index Terms
- FinRL: deep reinforcement learning framework to automate trading in quantitative finance
Recommendations
FinRL-podracer: high performance and scalable deep reinforcement learning for quantitative finance
ICAIF '21: Proceedings of the Second ACM International Conference on AI in FinanceMachine learning techniques are playing more and more important roles in finance market investment. However, finance quantitative modeling with conventional supervised learning approaches has a number of limitations, including the difficulty in defining ...
Synthetic Data Augmentation for Deep Reinforcement Learning in Financial Trading
ICAIF '22: Proceedings of the Third ACM International Conference on AI in FinanceDespite the eye-catching advances in the area, deploying Deep Reinforcement Learning (DRL) in financial markets remains a challenging task. Model-based techniques often fall short due to epistemic uncertainty, whereas model-free approaches require ...
A synchronous deep reinforcement learning model for automated multi-stock trading
AbstractAutomated trading is one of the research areas that has benefited from the recent success of deep reinforcement learning (DRL) in solving complex decision-making problems. Despite the large number of researches done, casting the stock trading ...
Comments