Automatic Market Making System with Offline Reinforcement Learning

Market making is an important research topic in quantitative finance. Market makers need to continuously optimize their ask and bid prices to provide liquidity and make profits, which can be viewed as a continuous control problem. Reinforcement learning is a common method for solving sequential decision-making problems, in which an agent learns from reward signals through interactions with the environment to maximize the cumulative return. However, traditional online reinforcement learning methods can be inefficient in practice as they require the agent to interact with the environment to collect training data, which could be unstable. Additionally, exploration in financial trading can be very expensive. To address these issues, we apply offline reinforcement learning methods which use historical experience to train agents. In this paper, we present ORL4MM (Offline Reinforcement Learning for Market Making), a novel market making agent using offline training and online fine-tuning to mitigate potential losses and instabilities. We demonstrate the effectiveness of our method through experiments, where our agent outperforms all baseline models, including traditional models and online RL agents. To the best of our knowledge, we are the first to explore the application of offline reinforcement learning in market-making tasks, and we provide valuable practical experience for the deployment of reinforcement learning in financial scenarios.