ej88

i think the thread shows:

- most people's perception of art is heavily affected by the framing (and to a lot of people ai = bad, and so they start seeing technical issues with it that could /never/ be made by Monet despite it being a Monet)

- but I think the critique here is more: even if someone recreated a Monet stroke-for-stroke, what's the value of this copy? I think the artist's personal life and context around the painting adds so much more to it compared to just being a pretty painting (perhaps this is the single most important part of what makes a painting interesting and valuable)

Ya, the demos were pretty contrived (feels like a running theme amongst the labs...)

swe bench pro has a public and private test set, where the private eval is from proprietary codebases only

This is cool!

I used to work on post-training & evals. it's really hard to make a good eval set and catch all forms of reward hacking. Excited to see more from poolside!

An omni model seems very useful for real-time human-computer interaction, off the top of my head:

- Voice assistants

- Customer experience

- Gaming

- Meeting assistants

- Real-time coach or user assistant for using software

- Translation

- Real-time work on a computer controlled by voice (frontend / mobile dev, CAD, 3D modeling, etc)

Traditionally a lot of these use cases with LLM agents are higher latency because the model needs to wait for the speaker to finish, then decide to call a tool or respond - if they call a tool they need to process the tool result and decide if they want to call a tool or respond, etc...

i would argue its the opposite

farming hit a ceiling because of demand

software today is heavily, heavily constrained by supply. demand is basically infinite for actually good software that solves problems people have (and people always have problems).

"She rejected several applicants with PhDs and engineering backgrounds, reasoning that their level of education could not compensate for a lack of hands-on specialty coffee experience."

This is depressing.

1. part of the moat is their guardrails and obviously they are audited and tracked. there are agents issuing refunds and more at scale right now so not sure where the skepticism comes from.. you're free to try and jailbreak them

2. another part of the value prop of these companies is figuring out how to construct the proper harness to take advantage of the lower latency of faster models while shoring up the weaker intelligence, how you blend deterministic and non-deterministic behaviors, compliance etc.

its a hard problem which is why f500 is willing to pay up

1&2 are already happening, these startups take on brand liability and trust to do so

3 depends on how companies want to measure it, but lack of user submitting satisfaction score is not a good thing

you can use a model w/o reasoning, + use various tricks to simulate low latency

that's fair, most implementations in the industry are in the early stages and implementing a full powered agent with access to all the tools it needs is hard (very political as you can imagine). i hope over the next year you notice them getting better!

adding some context as someone who works in this space

1. most people (average, non-tech people) reach for the phone to call in for easily solvable problems. Plus, if the agent is integrated deep enough & has tools to interact with crms, you can raise the ceiling on the types of problems it can solve.

You're trying to avoid the bad customer experience of human 1 reading off their script, then they transfer you to some other department who may or may not know how to solve your problem, and the entire interaction cost the company way more than the value created, so the company is disincentivized to help customers.

2. All the companies in this space start with the outsourced BPO market for cx (multi billion market still) but the next market is going to be in revenue generation and churn prevention at scale, i.e. how do you proactively avoid customer issues, how do you upsell and generate revenue instead of reducing cost, how do you keep customers happy?

3. I think more companies will pivot to outcome based pricing on the contrary, makes it so much more measurable than seat-based and protects margins better than usage based. Plus cx is one of the few industries with very well known metrics

4. Kind of? Most companies in this space don't use native voice models which are noticeably dumber, they use transcription + a stronger text model + TTS. The majority of customers can be handled with the latest SOTA text model and you need smart context engineering to handle the long tail of more complicated asks