jjmunro
Joined 2 years ago
Karma
1
Submissions
7
Member since
Aug 25
Recent submissions
Comment · 2 years ago
Will improve.
Comment · 2 years ago
We do have a specialist time-series index, optimised for things like tick-data. It compresses fairly well but we generally optimise for read-time. Not all over the place random-access, but slicing out date-ranges. There are two layers of index, a high level index of the data-objects, and the index in each object in S3.
A built-in as-of join is something we want to build.
Comment · 2 years ago
So, yes, separating the use-cases out...
You would use some convention for naming and parametrising backtests, 'different' backtests would get stored separately. But once you start updating backtests, running them in a loop with changing data, that's when the time-travel feature starts to be useful.
Comment · 2 years ago
It's a primary use-case.
Comment · 2 years ago
That's fair feedback. A direct parquet comparison in the presentation would have been useful.
Comment · 2 years ago
100% makes sense, it depends what you're looking for.
There are however lots of time-series that do change in Finance, e.g. valuations, estimates, alt-data (an obvious one is weather predictions). The time-travel feature can being super useful outside of external data changing as well, as an audit-log, and as a way to see how your all-time evaluations have changed (say backtests).
Comment · 2 years ago
Hi. I'm the presenter. Thanks for the interest. Opinions here are my own.
I'll put in a TLDR as the presentation is quite long. The other thing I'd like to say was that QCon London impressed me, the organisers spent time ensuring a good quality of presentation. The other talks that I saw were great. Many conferences I've been to recently are just happy to get someone, or can choose and go with well known quantities. I first attended QCon London early in my career, so it was interesting coming back after over a decade to present.
TLDR:
Why did we build our own database? In effort terms, successful quantative trading is more about good ideas well executed than it is about production trading technology (apart from perhaps HFT). We needed something that helped the quants be the most productive with data.
We needed something that was:
- Easy to use (I mean really easy for beginner/moderate programmers). We talk about day 1 productivity for new starters. Python is a tool for Quants not a career.
- Cost effective to run (no large DB infra, easy to maintain, cheap storage, low licensing)
- Performant (traditional SQL DBs don't compare here, we're in the Parquet, Clickhouse, KBD, etc space)
- Scalable (large data-science jobs 10K+ cores, on-demand)
A much shorter 3 min intro from PyQuantNews: https://www.youtube.com/watch?v=5_AjD7aVEEM
GitHub repo (Source-available/BSL): https://github.com/man-group/ArcticDB