Enhance TraderX Database Performance and Scalability

[lucaborella89]

TraderX Support for Bitemporal Data

07/16/24: FINOS Tech Sprint 2024 Kick-off

Watch the recording
Presentation
Work will be happening on a fork here: https://github.com/xtdb/traderx

Business Problem

Databases are the backbone of reliable software and data infrastructure, however they come in many shapes and sizes. Identifying the characteristics of a database that are most important for a system like TraderX is non-trivial, and often the correct choice can be to use multiple database systems, each with their own strengths and weaknesses. The rise of new software & hardware technologies and cloud databases has only complicated matters.

Within many real world trading systems, time-related business requirements are a huge source of complexity and various specialised databases exist to help where general purpose SQL databases fall short in various dimensions. For example, the ‘bitemporal model’ is a common pattern seen in financial systems, and various databases have first-class support which TraderX might benefit from. In the words of Kent Beck:

You may have heard the word “bi-temporal”. What’s that about?
In a nutshell, we want what’s recorded in the system to match the real world. We know this is impossible (delays, mistakes, changes) but are getting as close as we can. The promise is that if what’s in the system matches the real world as closely as possible, costs go down, customer satisfaction goes up, & we are able to scale further faster.

Goal

Identify and implement a bitemporal database solution that can effectively handle the growing volume and complexity of trading transactions in TraderX, to address both functional concerns like correctness and reporting flexibility, and non-functional concerns like performance, scalability, and reliability.

High-Level Architecture

TraderX currently uses H2 exclusively. H2 is a lightweight SQL database that runs on the JVM. In principle any part of the schema may be adapted and outsourced into one or more specialised, scalable databases.

Roadmap - Bitemporal Data and Risk Reporting PoC:

1. Extend TraderX schema with historical trade-related data using a bitemporal model and supporting database system
2. Demonstrate historical reporting capabilities that allow the user to answer regulatory/audit questions
3. Validate how ‘Day 2’ development requirements and schema changes interact with the model

Key people helping to facilitate this hackathon stream:

1. Jeremy Taylor (@refset) - Head of Product at JUXT, working on XTDB, an open source bitemporal database
2. Matt Butler (@mbutlerw) - Senior Software Engineer at JUXT, also working on XTDB

[refset]

[lucaborella89]

@refset thanks, description updated

[refset]

Help needed (even if you can't contribute time to help with actual development!)

We are looking to hear from people who have felt "bitemporal pains" previously (or have worked on trade-related reporting) and can help collaborate with us to prioritise what this team should work on - please do get in touch 🙏 (here / Slack / jdt@juxt.pro)

Pitch deck presented at the 16 July "FINOS Tech Sprint Kick-off" session: https://drive.google.com/file/d/1wik7XyrV8uVK6MhvsDO0xQiaj4S-jo-D/view?usp=sharing

Work will be happening on a fork here: https://github.com/xtdb/traderx

Other ideas we could explore

1. a "reporting module" that looks at speaking to CDM's ISDA/DRR requirements (or is that overkill, and does TraderX just need a CDM API?)

[refset]

Hi folks, we will be arranging another public call for this PoC on Thursday afternoon (UTC). On the this week's call will we review progress against the backlog and discuss priorities for the coming week. We are aiming to run twice-weekly calls between now and the OSFF for any newcomers to get involved 🙂

So please check back here for details on the repeating call schedule once we have it setup! And in the meantime, a warm welcome to @mpisanko who has also joined the PoC team and is currently working on a new pricing and reporting service for TraderX 🚀