mlb_hn

Joined 8 years ago

Karma

726

Submissions

162

Member since

Dec 29

playing with little space aliens at julius.ai

View on Hacker News

Recent submissions

Comment · 1 year ago

Our GCP is down

Comment · 2 years ago

nice overview of progress over time. are there quant metrics for the sim capabilities or is it mostly vibes?

Comment · 2 years ago

he was good people

Accelerating Development with LLMs by Hallucinating Functions engineering.hyperwriteai.com

3 mlb_hn 3 years ago 0 comments

Comment · 4 years ago

yeah, that's a very big caveat - haven't checked neo 20b yet. I've had a hard time getting the AI21 models to use it and those are also pretty big so it's interesting why sometimes it works and sometimes it doesn't. (and Davinci > Codegen Davinci > Curie > J-6B). Fine tunes can also learn to do the inner monologue as well which is really cool - not sure how much is architecture vs. training parameters.

Comment · 4 years ago

definitely. also works on text translation/comprehension like emojis! https://aidungeon.medium.com/introducing-ai-dungeon-translat.... For actual benchmarks, scratchpad improves GPT-Davinci WIC from 50% accuracy (chance) to nearly 70%.

I think the check and validate is a different sort of scratchpad but maybe not. Seems like at least 3 types - soe for pulling implicit info out of the network viz wic, sometimes for intermediary steps viz coding, sometimes for verification like here.

Comment · 4 years ago

I get the tokenization argument and it may influence it a bit, but I suspect the n-digit math issue has to do more with search the way it samples (in the bpe link gwern references some experiements I'd done with improving n-digit math by chunking using commas, http://gptprompts.wikidot.com/logic:math). I think since it samples left to right on the first pass, it's not able to predict well if things carry from right to left.

I think can mitigate the search issue a bit if you have the prompt double-check itself after the fact (e.g. https://towardsdatascience.com/1-1-3-wait-no-1-1-2-how-to-ha...). Works different depending on the size of the model tho.

Comment · 4 years ago

Couple things there where you can see if it improves with the prompt/formatting. E.g. with Davinci (and J a bit but didn't test too much) you can get bette results by:

  - Using few-shot examples of similar length to the targets (e.g. 10 digit math, use 10 digit few shots)

  - Chunking numbers with commas

  - Having it double check itself

and here it's not doing any of those things.

Comment · 5 years ago

An alternative of that is as Oppenheimer once said, sometimes things are secret because a man doesn't like to know what he's up to if he can avoid it

How We Accidentally Gave Our Bots Their Personalities aidungeon.medium.com

2 mlb_hn 5 years ago 0 comments

Comment · 6 years ago

This has been a systemic issue reported on for years; e.g. reported by the Intercept in 2017 [1] and Atlantic in 2019 [2]. Not really made clear from the story considering the Economist headline is almost identical to the Atlantic one.

[1]https://theintercept.com/2017/11/02/war-crimes-youtube-faceb...

[2]https://www.theatlantic.com/ideas/archive/2019/05/facebook-a...

Comment · 6 years ago

I think that goes back to a Karpathy quote [1], don't know where he got it from

[1] https://twitter.com/karpathy/status/961307010246492160

Comment · 6 years ago

My thinking there wasn't because of BPEs, I think it's a graph traversal issue.