hcarlens

The Wikipedia article doesn't have that much detail on M5 or M6, but the M5 papers are in the International Journal of Forecasting[1] and M6 should be published later this year (there's already a preprint on arxiv [2]).

I recently spent some time looking into the history and results of the M competitions and had a chance to speak to Professor Makridakis about them, as well as the winners of each of the M6 competition tracks [3]. While the methods have become more sophisticated, some conclusions from M1 still seem to hold: in particular, that there is no overall "best" method, and that the winning method tends to be different for different types of data, time horizons, and evaluation metrics.

[1]: https://www.sciencedirect.com/science/article/pii/S016920702... [2]: https://arxiv.org/abs/2310.13357 [3]: https://mlcontests.com/state-of-competitive-machine-learning...

Comment · 2 years ago

XGBoost, LightGBM, and Catboost are all used quite frequently in competitions. LightGBM is actually marginally more popular than the other two now, but it's pretty close. In the M5 forecasting competition a few years back, many of the top solutions used primarily LightGBM.

Comment · 2 years ago

Yeah, this book was incredible and the tech in it has aged extremely well. Have you tried any of Ted Chiang's books? They're also great hard sci-fi. Another one that plays with similar ideas to Permutation City is the Bobiverse series by Dennis E. Taylor.

Comment · 2 years ago

Also relevant: https://aimoprize.com/

($10m prize for models that can perform well at IMO)

Comment · 2 years ago

Agreed, and not only do they not compare their model to Phi-2 directly, the benchmarks they report don't overlap with the ones in the Phi-2 post[1], making it hard for a third party to compare without running benchmarks themselves.

(In turn, in the Phi-2 post they compare Phi-2 to Llama-2 instead of CodeLlama, making it even harder)

[1]: https://www.microsoft.com/en-us/research/blog/phi-2-the-surp...

My AI Timelines Have Sped Up (Again) alexirpan.com

50 hcarlens 2 years ago 95 comments

Comment · 2 years ago

There have been quite a few interesting Kaggle competitions in recent years, as well as other interesting ML/data science competitions on other platforms.

Platforms like Kaggle, DrivenData, Zindi, AIcrowd, CodaLab and others are running dozens-hundreds of competitions a year in total, including ones linked to top academic conferences. One interesting recent one is this one on LLM efficiency - trying to see to what extent people can fine-tune an LLM with just 1GPU and 24h: https://llm-efficiency-challenge.github.io/

Or the Makridakis series of challenges, running since the 80s, which are a great testbed for time-series models (the 6th one finished just last year): https://mofc.unic.ac.cy/the-m6-competition/

Comment · 2 years ago

An interesting approach I came across at NeurIPS a few weeks ago is called "ML with Requirements"[1]: https://arxiv.org/abs/2304.03674

My basic understanding is that it combines "standard" supervised learning techniques (neural nets + SGD) with a set of logical requirements (e.g. in the case of annotating autonomous driving data, things like "a traffic light cannot be red and green at the same time"). The logical requirements not only make the solution more practically useful, but can also help it learn the "right" solution with less labelled data.

[1] I don't know if they had a NeurIPS paper about this; I was talking to the authors about the NeurIPS competition they were running related to this approach: https://sites.google.com/view/road-r/home