salamo
Joined 3 years ago
Karma
408
Submissions
138
Member since
Nov 27
https://blog.lukesalamone.com
View on Hacker NewsRecent submissions
Comment · 17 days ago
Sam Altman having direct access to my chat messages is a close second.
Comment · 2 months ago
> We have paused some frontier RL training to ensure that we can meet the appropriate alignment, security and monitoring standards for the new level of capabilities in front of us. Model progress is now extremely rapid, and we always said we would take action if we felt that model capabilities were outstripping the pace of safety and alignment.
> We care very deeply about AI safety. We believe the entire field will have to coordinate on shared safety standards, but will act unilaterally in the meantime.
> We expect confidence in safety to increasingly set the pace of AI progress. We are optimistic about the alignment work we are doing, and we remain committed to making frontier capabilities widely available.
My guess is this is related to the HuggingFace incident last month, not Sam Altman announcing the downfall of OAI.
Comment · 2 months ago
This matches my experience with Sol. Read and thought for a while, and edited files, tested, edited again, then ran out of budget in a relatively short time. But its solution was very good and was done quickly, so all things equal I prefer that over something much more verbose like Deepseek.
Comment · 2 months ago
As someone who's done something similar (https://blog.lukesalamone.com/posts/creating-tiny-semantic-s...) the expensive part wasn't the training itself but the data curation and evaluation post-training. For this, getting a reasonable distribution of tool calls when the tool call can be anything isn't easy.
Once you have that, the model is small enough batch sizes are probably enormous and training can probably be done on a consumer-grade GPU in a week or less. Or even faster on a bigger GPU.
Comment · 3 months ago
Possible reasons:
- They might be dynamically adjusting these at inference time [1]. For example, start with a low temperature and generate samples with increasingly high temperatures until one of them passes some quality gate.
- They don't want you to fine-tune on high temperature completions (rejection fine-tuning). You could call this "rejection fine-tuning rejection".
[1] https://rlhfbook.com/c/09-rejection-sampling#related-best-of...
Comment · 3 months ago
Yeah, the model is small enough that inference is already basically instant for my usecase (only 6 transformer layers for the blog search).
Comment · 3 months ago
Yeah, I also found that for ultra low footprint models ORT is a big portion of the total payload, because it contains logic for general ONNX graph operations. In my case I found that ORT alone was 3.4MB over the wire, so I swapped it out for a tiny wasm that was 850x smaller and only contained the operations I needed: https://blog.lukesalamone.com/posts/creating-tiny-semantic-s...
Comment · 3 months ago
On the one hand, compacting at 300k saves OpenAI 40%. That's great.
On the other hand, $10 for 1M tokens still seems really high? It's not too hard to blow through that in an hour or two.
Comment · 3 months ago
I come at it from a slightly different angle.
I write technical blog posts with visualizations and live demos. That usually means embedding a bit of custom javascript in the page for the demo. Or shipping custom wasm to enable extreme semantic model compression.
I do this by pushing content from my machine to github pages which is wired up to my subdomain.
If github pages stops being a good, free option for this, I will find another. Not sure I would call this "hardcore" really.
Comment · 3 months ago
$1.7 billion is small potatoes. My bill is over $155 billion and growing. I'm worried if the trend continues I'll have depleted my rainy day fund.