noambrown
Joined 8 years ago
Karma
505
Submissions
57
Member since
Dec 27
Recent submissions
Comment · 4 years ago
It's in the supplementary material of the 2019 paper: http://www.cs.cmu.edu/~noamb/papers/19-Science-Superhuman_Su... . Look at the "Variance reduction via AIVAT" section.
Comment · 4 years ago
The four humans were getting $120,000 between them. Their share of that was dependent on how much better they did than the other humans. That means there was no incentive to collude.
Top pro poker players understand the value of money. They weren't treating it as a freeroll and anyone that has seen the hand histories can confirm that.
Comment · 4 years ago
That's not true in practice for poker. Pluribus showed that if you run CFR in multiplayer poker you get a solution that works great in practice. Multiple equilibria are certainly a theoretical issue for many games, but poker conveniently isn't one of them.
Comment · 4 years ago
Bots are superhuman in self-play Hanabi: https://ai.facebook.com/blog/building-ai-that-can-master-com...
The remaining challenge is getting it to play well with human partners. Doing that requires modeling human conventions rather than learning weird bot conventions. That's hard because while you can collect essentially unlimited data through self play, it's hard to collect a lot of data playing with humans using reinforcement learning. AI algorithms are really bad at sample efficiency.
Comment · 4 years ago
Normally 10,000 hands would be too small a sample size but we used variance-reduction techniques to reduce the luck factor. Think things like all-in EV but much more powerful. It's described in the paper.
Comment · 4 years ago
Bots are superhuman in no-limit Texas hold'em. Libratus beat top humans in two-player in 2017 and Pluribus beat top humans in six-player in 2019:
https://www.science.org/doi/abs/10.1126/science.aao1733
https://www.science.org/doi/abs/10.1126/science.aay2400
It's shocking that the reporter didn't mention these results or anything else more recent than 2015.
Comment · 7 years ago
Bridge has a similar challenge, though from what I understand Bridge AIs are not superhuman yet. I suspect our techniques could be applied to Bridge, though they may need to be adapted a bit.
The imperfect information in Hanabi absolutely matters a ton. It's not an interesting game without it.
Comment · 7 years ago
Thanks! We're looking in a few different directions, but one thing I'm excited about is mixed cooperative/competitive settings. In poker, there is no room for cooperation. In Hanabi, you are 100% cooperating with your teammates. But most real-world situations, like a negotiation, are somewhere in between. The AI techniques for these settings are not too strong yet.
Comment · 7 years ago
Open source it, learn from it, and build upon it to continue to push forward the frontier of AI.
Comment · 7 years ago
Definitely!
Comment · 7 years ago
In terms of Hanabi, this bot arrived at conventions that are pretty different from how humans play the game. We invited an advanced Hanabi player to play with the bot and he pointed out a few things in particular that he'd like to start using. For example, humans usually have a rule that if your teammate hints multiple cards of the same color/number, you should play the newest one. The bot uses a more complicated rule: if the card you just picked up was hinted then play that card, otherwise play the oldest hinted card. That gives you way more flexibility to hint playable cards that would otherwise be tough to get played.
I think one important general lesson is that search is really, really important. Deep RL algorithms are making huge advancements, but Deep RL alone can't reach superhuman performance in Go or poker with search. Here, too, we see that search was the key to conquering this game, and I think that will hold true in more complex real-world settings as well. Figuring out how to extend search to more complex real-world settings will be a challenge, but it's one worth pursuing.
Comment · 7 years ago
The search algorithm shares a lot in common with our Pluribus poker AI (https://ai.facebook.com/blog/pluribus-first-ai-to-beat-pros-...), but we added "retrospective belief updates" which makes it way more scalable. We also didn't use counterfactual regret minimization (CFR) because in cooperative games you want to be as predictable as possible, whereas CFR helps make you unpredictable in a balanced way (useful in poker).
The most surprising takeaway is just how effective search was. People were viewing Hanabi as a reinforcement learning challenge, but we showed that adding even a simple search algorithm can lead to larger gains than any existing deep RL algorithm could achieve. Of course, search and RL are completely compatible, so you can combine them to get the best of both worlds, but I think a lot of researchers underestimated the value of search.
Comment · 7 years ago
Hi! I'm one of the authors on the paper. We'd be happy to answer any questions. Ask us anything!
Comment · 7 years ago
The humans knew the whole time which player was the bot.
Comment · 7 years ago
The hand logs from the 5 humans + 1 AI experiment are included in the supplementary material of the Science paper.