seahyinghang8

Joined 8 years ago

Karma

Submissions

Member since

Sep 16

Recent submissions

founder of castform here, we believe that as agent deployment moves from experimentation phase where cost doesn't matter as much to deployment (what's the margin of serving the request), there will be a rise in interest in optimized models.

Comment · 3 months ago

<founder of castform here> tldr: we generated synthetic training questions from the gitlab product handbook.

totally agree that this larger corpus with harder to search information would be a good way to stress test - i'm sure we will encounter more interesting problems to solve. love to hear any suggestions of corpus to search against that is not just the public internet

the training run link is also a little buried but here, you can see the comparison against the various models and their exact traces: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

Comment · 3 months ago

we actually have the test benchmark against luna but no deepseek flash (we haven't added DSFlash into our benchmarking model pipeline)

you can check out the full comparison against all the other models here: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

- founder of castform

Comment · 3 months ago

founder of castform here again - slightly unrelated to retrieval but on the topic that folks are discussing here, i was actually collecting benchmarking various coding traces for the purpose of training a model router and surprisingly, luna held up very well against sol and terra. it was able to solve close to >95% the that sol can handle at a fraction of the cost. have not benchmarked the OSS models yet but will add the popular ones to the list like Deepseek Flash and Kimi k3 to see how they fare.

will share the full results soon!

Comment · 3 months ago

we actually have the test benchmark against luna too! it's just not in our title but you can see it in the first diagram below the title. luna does pretty well tbh but sol is just a tad bit better. but luna is way cheaper.

if you want to dive down into the various traces of the benchmark, you can check this out: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

- founder of castform

Comment · 3 months ago

(founder of castform here) - we didn't get to dive too deep into the dataset we were using for the retrieval in the blogpost for brevity, but we did link the training run (which shows the dataset) here: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

the page shows the exact trace of all the models we are comparing against and the aggregate scores

we generated the question & answer pair from gitlab product handbook (https://handbook.gitlab.com/) since the point is to show that you can generate training questions from raw data corpus (something a company already has today)

Comment · 3 months ago

(founder of castform here!) - having worked at FAANG / big tech, i totally get this. our example was on gitlab's open source company handbook but i think a real company's corpus is way more messy and has many sources of truth.

a few ideas i have yet to validate are: - prioritize recently updated docs when generating the training questions (assumption those docs are more correct than others) - actually including contradicting documents that talks about the exact same topic might be a good training example - ideally the model should surface all the relevant info it can find, and explain what it has found. (usually contradiction comes from the fact that the later document is the updated stance) - you could also mine high quality Q&A from public slack / communication channels where questions were asked and someone else in the team linked some docs / answer. those are strongly validated "ground truth" answers

Comment · 5 months ago

i really hope that's the case

Comment · 5 months ago

you should check out https://openai.com/index/gpt-5-5-bio-bug-bounty/

Comment · 6 months ago

i use zed 1.0 agent interface and it's pretty great. cool to see that pokegent is modeled after zed's ACP. tho zed has been bugging out recently where the agents just stop working at all so i might give this a try

Comment · 2 years ago

Xcode 16 recently released swift predictive completion but it requires ARM processor with more than 8GB memory. We made an Xcode "extension" that provides AI completion that works with both Intel and Apple Silicon macs. You can either connect to a model hosted by us for free or a local ollama model of your choice.

If you develop on Xcode, give our extension a try and let us know what you think!