fun story[0] about Bard's
June update - implicit code execution (remember?
https://news.ycombinator.com/item?id=36229782) - the example that they advertised on their blogpost/video demo now no longer works.
Here's what I got as of today: https://g.co/bard/share/2580222b6fb2 (screenshot from a couple days ago: https://twitter.com/swyx/status/1678495067663925248?s=20)
> What are the prime factors of 15683615?
> (after outputting some code) The output of the code is the list [5, 11, 23, 43]
this result is hallucinated - Bard tells you it is running the code, but it doesnt. If you run the code, you get [5, 151, 20773], which actually is the correct answer
now lets all acknowledge that maintaining stable performance of LLMs is an unimaginably hard problem, but* i suggest some regression testing to keep up the baseline. Would fuzz the test (really, its just another eval) because of course this test is going to be memorized in the weights soon
0: i first discussed this on monday's update https://www.latent.space/p/code-interpreter#%C2%A7misc-notes...
* Edited based on feedback from commenters, thanks.
Here's what I got as of today: https://g.co/bard/share/2580222b6fb2 (screenshot from a couple days ago: https://twitter.com/swyx/status/1678495067663925248?s=20)
> What are the prime factors of 15683615?
> (after outputting some code) The output of the code is the list [5, 11, 23, 43]
this result is hallucinated - Bard tells you it is running the code, but it doesnt. If you run the code, you get [5, 151, 20773], which actually is the correct answer
now lets all acknowledge that maintaining stable performance of LLMs is an unimaginably hard problem, but* i suggest some regression testing to keep up the baseline. Would fuzz the test (really, its just another eval) because of course this test is going to be memorized in the weights soon
0: i first discussed this on monday's update https://www.latent.space/p/code-interpreter#%C2%A7misc-notes...
* Edited based on feedback from commenters, thanks.