traK6Dcm

HFT may not, but trading itself definitely provides positive value. People's utility function is not purely monetary over an infinite time horizon. Trading allows you to trade off price and risk over multiple time horizons.

For example, a trade where someone needs to convert his assets into cash due to a family emergency benefits both sides. The person with the emergency takes liquidity from the market and pays a premium because the trade is time-sensitive - he needs cash the next day. Other liquidity traders may profit from such "uninformed" flow in the long term, but both parties are happy because they got what they want.

Another example is trading off risk and hedging against certain changes in the world that would affect you.

Yeah, fair enough. The truth is that I'm just a little frustrated by people touting the horn of "making the world a better place" when in reality most end up doing nothing of the sort but then criticize people working in Finance :)

Some people argue that HFT provides liquidity for retail investors, but it's debatable whether that liquidity is real or not since it'll be gone during black swan events. At the same time HFT profit from uninformed/retail flow. So it's debatable wether the actual activity provides values. I'd say probably not.

But there can be indirect value in working in HFT, just like there is with other demanding jobs. There's interesting systems and ML research going on in the field - that's knowledge you can take with you and share with others. Some time ago, HFT was driving a lot innovation in networking, infrastructure, and systems.

Also, over time I've come to realize that "providing value" is really difficult to quantify. Does working in Academia provide value? Maybe if you get lucky and publish something that actually turns out useful. The majority of academics publish papers that are nothing more than noise - negative value. Does working at FAANG provide positive value to society? Again arguable. You may inadvertently end up promoting ads and misinformation. Not everyone does - but it's not guaranteed that you don't without realizing.

"Providing value" is subjective and most of the time it's nothing more than a story people tell themselves so they can feel good about themselves.

I think he means smart as in "how long your model prediction takes". If your neural net (haven't actually met anyone who uses these in trading) takes 5ms to make a prediction that'll lock you out of a whole lot of trading opportunities/strategies.

Speed always matters, no matter where on the smartness spectrum you are, but it's relative. If your model prediction takes 5ms you're not getting much ROI out of investing $1M into shaving off 50ns in your data processing. But if your end-to-end latency without prediction is 1ms, you better invest in getting that down.

> I understand they are not making the world a better place, but are they making it worse?

Most people would argue that they are not directly making the world a worse place, but rather that they are sucking in talent that could have used their skills in other places to make the world a better place. For example, many talented Physicists ended up in trading companies a few decades ago. Today, you can make a similar argument for companies such as Google or FB who probably have tons of extremely intelligent scientists working on ads.

> Something that I have heard is that they are providing a service by providing liquidity to the market.

It's unquestionable that they provide liquidity, at least most of the time. That's good. What's questionable is whether they are doing it in an efficient way. Their primary goal is not to provide liquidity - it's to make money. Liquidity is just kind of a side effect of their activities, which is an elaborate race to "beat" other firms through speed and algorithms. Most of the resources go towards that, but don't result in additional liquidity, most of which is probably not even needed. It's like saying "FB provides an important service by allowing you to message your friends" - sure, that's not wrong, but that's not their primary goal. And you don't need a trillion dollar to make a messaging app. The difference is that FB also does lots of other things, so it's unclear how much of their resources are actually wasted vs. potentially used for "making the world a better place" - it's not quite as obvious that they are wasting a lot of talent and capital.

The other question is what the alternative would be. Markets run by human traders were significantly more unfair than anything we have now, but HFT was never meant to happen, it was just result of markets becoming electronic and transparent, and people trying to extract more money.

Even if that was true (it's difficult to prove either way since there is little data on black swan events), the fact that they provide liquidity during stable times can make up for not providing liquidity during black swans. The impact could still be net-positive.

Also, it's unclear what the alternative is. While HFT may not be needed, markets run by human traders were significantly more unfair than anything we have now. It's just that fewer people complained about it because nothing was transparent. Or do you believe human traders don't pull quotes?

It hasn't really, at least not in production. Academics are now publishing a lot of papers using Deep Learning or RL, but you won't usually see those in live systems.

In live systems, latency is usually more important than a "better" model - A model that takes milliseconds to make slightly better predictions is too slow when you're working on nano- to microsecond scales, often on specialized hardware. Really, the "AI" part is less important in HFT than you may think. It's often more system/infrastructure.

This is for HFT specifically, perhaps it has had more impact on longer time horizons, or something like portfolio management. My impression is (but I may be wrong) that there aren't that many people doing something in between HFT and much longer (minutes to days) time horizons, something like milliseconds to seconds. Maybe there is an opportunity there for some of the newer AI techniques.

What people are complaining about is not the overfitting, but the unrealistic assumptions in the backtest. In the real world there is slippage, latencies/jitter, special market open regimes, hidden orders, market impact, front-running, variable fees, and all kinds of other complexities. Their transaction costs are apparently also an unreasonable assumption. Sophisticated simulators used in professional trading firms can account for such things to some extent, but most academic papers conveniently ignore these complexities and just assume they can trade at whatever price the data tells them. It's completely unrealistic.

To answer your original question about overfitting, they can still overfit to test data by running a lot of experiments with different hyperparameters, architectures and parts of the data, and only report what has worked. There are also more complex ways that test data can leak into training data (see the book Advances in financial ML for a good overview). You can already see this is likely the case just from the variance in their results and trades. They also don't compare to baselines. It's not unlikely that the results are just random and they fail to report those experiments that didn't work. Of course, you cannot prove this without having an exact log of all things they ever did to the data. But again, that's not the main issue here.

Data distribution shift. The market changes over time and your current data does not come from the same distribution as old data. That limits the amount of data you can use for training and testing. You need to be very careful not to overfit. That's especially true for something like daily or hourly data - there isn't much data to begin with and you won't have much left if you look at only a few weeks or months. Market data already has a low signal/noise ratio to begin with, so you need a good chunk of data to learn from.

As you go to shorter time scales you get more usable data, but then you also need to deal with other issues such as latencies/jitter, market impact, complex order types, order book queues, etc. It becomes a different game.

I don't think daily OHLCV data is a good data source. First of all, it's too little because of the data distribution shift over time. It's also driven significantly by outliers and events outside of the data (news, etc). There's way too much noise in daily prices that most of the signal is drowned out (longer time horizons = more uncertainty). I don't believe you can find any edge looking at daily data. This kind of data is would be equivalent to what MNIST is in ML. Nice for some playing around, but nobody who is serious would use it for production or benchmarking, at least not by itself.

There is a good reason trading firms pay a lot of money (sometimes millions) for fine-grained historical data from exchanges. It's not only about speed. For interesting experiments you IMO need L2 or L3 order book data, ideally somewhere on second or sub-second scales. That's not HFT (which is nano and micros), but somewhere in the "middle" - it's a different world than what you are talking about.

By simulators he means market simulators for L2/L3 data with a matching engine, latencies, queue positions, jitter, complex order types, etc. You can't simulate other market participants (at least not fully, but there are techniques to even estimate this based on live trading feedback), but there are still many things left that you can simulate in a realistic way during training and backtesting. Trading companies typically have their own high-performance simulators built in house. Some of these are incredibly complex. Good simulators can give you a huge edge and are absolutely necessary.

Is 200-500k still true? It used to be, but I think it has decreased significantly over the last decade. I'd say most junior people in this field are making about the same or less than software engineers these days.

But like you said, the range here is incredibly wide and largely depends on how well your strategies do and if you have your own desk/fund.

In this simple example, yes, but you are assuming that monetary value = utility. That's not always the case. People have all kinds of different incentives for participating in the markets.

Let's say I am a market maker offering to buy Apple shares at $99 and sell them at $100. Let's take an ex-Apple employee who owns some shares. He just had a family emergency and wants to liquidate his shares to get cash, and he needs it quickly. He doesn't care about paying a few dollars extra in exchange for a quick trade because he needs to pay a bill tomorrow. I buy his shares for $99. He is happy because he immediately got his cash.

On the other side, there is a a retail investor doing long-term investment and wants to add Apple to their portfolio. They also don't care about a few cents because they're holding the stock for a decade and love the new CEO. They buy my Apple shares from me for 100.0. They are happy because I can guarantee them a stable price for a decent number of shares.

All participants are happy. I just made $1 from the spread for providing liquidity, the investor got the long-term investment they wanted, and the ex-Apple employee got his cash.

Sure, both sides of the market could have made more optimal trades if they had put in more effort and "optimized" their trades with algos and somehow skipped the middle-man, but they would've sacrificed convenience and time, which may be worth more to them than the little bit of extra $ they paid. Aren't we all winners?

When you go buy bananas in your grocery store you also don't complain about them taking a cut for providing liquidity. You don't say the farmer has "lost" money because the consumer paid more than what the farmer originally sold for to the grocery store. The farmer is happy because otherwise he may not have traded at all or his bananas may have gone bad (= needs to trade quickly). This is no different.

You make a good point. I've also gotten into trading because I enjoy the algorithmic and mathematical aspects, and I would love to share more of what has been working for me and write extensively about it. And there are probably more people like that out there. However, trading has such a bad reputation and uncertain future that I am not sure that's a good career move. I'm torn.

You're right that there are probably some gems and people writing up good posts and articles. However, 99% of what comes to my inbox, which is certain newsletters and arXiv subscriptions, is clearly BS. I'm particularly disappointed with arXiv/academia, because in other fields like biology and CS/ML/AI, published papers tend to be of higher quality than your average blog post. In trading the opposite seems to be true. Seeing a good trading paper on arXiv is incredibly rare. I would even go as far as saying that reddit is a significantly better source of information than arXiv for this field.

Yup, it's probably just the output of an optimizer and then tested on held-out future data. Not overfitting is the key here and what's really hard. You need to be careful about the number of parameters and the amount of validation data you have.

These alphas will likely be only profitable for a short time period as long as the market data distribution (i.e. strategies of other market participants) doesn't change. So you would need to continually optimize and update them.

The way I think about it is that you are essentially finding the right parameters to "exploit" the combination of algorithms of all other participants, where algorithm could also be a human looking at charts and following certain rules, with a lot of random noise from retail traders thrown in.