poomer

As someone who's spent a lot of years in an academic social science field and in analytics/modeling more generally, I feel strongly that most educated people have way too much trust in academic literature.

Outright generated data like this is rare and dumb, instead it's very common to play around with data cleaning/variable selection/modeling assumptions to get whatever result you want. I've seen it from grad students, tenured academics, DS/MLE in big tech, basically anywhere where you mix high pressure & neurotic personalities with analytics work.

I'm very skeptical that this will succeed. As with other extremely hyped social medias (anyone remember Clubhouse?) it's already filled with influencers and e-celebs trying to get in early, but these people always poison the well with insincere, low-quality, monetization-oriented content. AFAIK, every successful social media platform grew more or less organically and only had these folks latch on later (usually causing the degradation of the platform when they did).

Having worked in analytics at various tech companies including Fb, the reason is almost always just that app users have way higher usage rates and ad clickthrough rates. Of course if you know basic stats you will realize this is spurious correlation (social media addicts prefer apps, using an app doesn't make one a social media addict), but somehow management never cares.

I find this an extremely odd and uninterpretable article for this reason. When people say that "x drives y" they usually mean that "x causes y". And inflation is by definition price hikes. So article seems to be saying that "corporate profits causes price hikes", which is meaningless since the causal direction should be in the reverse (price changes cause profit changes).

The fact that labor costs (which are part of a company's profit calculation) are considered a separate "driver" makes this even more confusing.

There is a stop sign in front of my house and probably around 95% of cars stop, and the amount of cars that completely ignore it (ie don't even slow down) is easily <1%. I think this is just more of an issue with local cultural norms.

This class of startup, "build domain specific LLMS using your own data", is extremely crowded right now but I am not optimistic about their future. For large companies, the actual modeling work for this is already easy for any ML team, thanks to existing FOSS work on stuff like PEFT and LoRA. The hard part is figuring out what data goes into the fine tuning process and how to get this data in a usable form, but this is very business specific and can't be automated in a SaaS process.

For SMBs, the value would be in using the LLM to generate responses to customer Q&A/search queries. But these companies aren't going to integrate some external third party service, they'll only use it if it's already baked into their CMS - Wordpress/Shopify/Wix/etc. I just don't see who the final consumer for this product would be.

Thank you, I appreciate your transparency with this work.

Most time series models assume you've already deseasonalized your data in advance. Typically, seasonality is obvious to the human doing the modeling (e.g. sales being up near Christmas), so it's usually preferable for the human to deseasonalize the data in advance using a separate model that bakes in some of their human knowledge of how the world works. Forcing the model to learn seasonal trends fully on its own adds another layer of estimation error.

Prophet is popular because it works off the shelf with non-deseasonalized data and mixed frequency data, which makes it great for quick forecasting exercises. But IMO it is never the ideal model if you have a lot of time and expertise to work with.

At work we were facing this dilemna. Our team is working on a model to detect fraud/scam messages, in production it needs to label ~500k messages a day at low cost. We wanted to train a basic gbt/BERT model to run locally but we considered using GPT-4 as an label source instead of our usual human labelers.

For us human labeling is suprisingly cheap, the main advantage of GPT-4 would be that it would be much faster, since scams are always changing we could general new labels regularly and be continuously retraining our model.

In the end we didn't go down that route, there were several problems:

- GPT-4 accuracy wasn't as good as human labelers. I believe this is because scam messages are intentionally tricky, and require a much more general understanding of the world compared to the datasets used in this article which feature simpler labeling problems. Also, I don't trust that there was no funny business going on in generating the results for this blog, since there is clear conflict of interest with the business that owns it.

- GPT-4 would be consistently fooled by certain types of scams whereas human annotators work off a consensus procedure. This could probably be solved in the future when there's a larger pool of other high-quality LLMs available, and we can pool them for consensus.

- Concern that some PII information gets accidentally sent to OpenAI, of course nobody trusts that those guys will treat our customers data with any level of appropriate ethics.

You could make a nice living covertly selling the AI "startups" to wealthy folk who just want bragging rights about being founders.

I do not remember the specific metrics, but there was a big focus on the fact that their CPC/CPM/etc was lower than alternative social media, meaning advertisers put less value on clicks/views from Reddit than other platforms. And they believed this was because their ads were not targeted enough since there was (is?) no fancy ML models behind it, advertisers could just chose some basic rules for what subreddits they want to target.

To solve this they were building out a huge ad relevancy team to target ads at users using posting history, similar to Meta/LinkedIn/etc.