luke14free

I'd argue there is a bias for using mostly open models through open router as using anthropic/oAI models from open router makes little sense (more expensive).

Hey HN,

We built a benchmark that tests how LLMs behave when they have to hold a difficult debate position against an adversary.

We took 6 frontier models, paired them in structured disputes (business conflicts, ethics dilemmas, property disputes, family disagreements), and forced them to argue opposing sides before a third LLM mediator. Each model gets a position to defend and a fixed number of turns. A separate judge panel scores the outcome.

The interesting part isn't who "wins" but rather what the disputes reveal about post-training behavior. Some models fold almost immediately, conceding points they shouldn't. Others hold firm on weak positions when a smarter move would be strategic compromise.

We ran this as a Swiss tournament (like chess) - 10 rounds, ~300 matches total, every case played twice with sides swapped to cancel position bias. Three independent frontier judge LLMs score each ruling, majority vote decides the outcome.

A couple of things we noticed: - models tuned hardest to be agreeable are the ones that lose most, they tend to concede points mid-argument even when holding a strong position - some models argue much better when they're on the "sympathetic" or "morally comfortable" side of a dispute than when they're assigned the harsher position. E.g., a model might crush it defending a tenant against eviction but argue poorly when it has to defend the landlord's right to evict

P.S. For every match read the full argument transcript.

search with a - instead of a space, I get more results (i.e. F1-car returns one icon, and so does thumbs-up)

Cool product, could be good for people that want to move from constantly disturbing notifications to daily digests. What is the roadmap for supporting more sources and for creating customized digests (i.e. create digests from custom sources that other people can follow)?

There is a public ranking of apps ranked by downloads in the app store. If you know how many downloads correspond to what position in the chart (e.g. number 100 makes ~1000 downloads), you can fit a log regression to determine the number of downloads given the ranking in the category. I am oversimplifying because you also need to take into account seasonality and time trends (e.g. the app store is always expanding), but overall fits are very very precise (unless you get near the top positions, which are those for which they have more problems in estimating). Ah, and if you wonder where the data comes from, App Annie and Sensor Tower get data for downloads to fit their models directly from tens of thousands of developers that share it with them in exchange for free analytics,

I work on models like those of Sensor Tower and App Annie use to estimate downloads. They are extremely precise, the cross-validated error is << 10% for apps that are this small, and in the US (it increases with much bigger apps in countries where little data is available).

Fair point. I am editing the entry name to reflect your suggestions. Static typed -> type checked at runtime