ramesh1994

Sorry I don't know how hackernews notifications work, didn't see this reply. But if you see this here's another reminder for you to clean this up into a public demo :)

This is really cool! Appreciate sharing this work and the explanation.

You mentioned that massaging the data into shape as one of the problems, which I think is possibly one of the best applications of LLMs in my opinion. Creating a pipeline of a data feed (still hard) into an LLM which outputs JSON with the fields of interest would be amazing.

Hey this sounds pretty cool! If this is public would you mind sharing a link?

I just found about this project from this comment, absolutely excited to try this out.

As someone who's never used any of the infrastructure tools, I'm thinking of pyinfra as a way to run shell commands + install dependencies on hosts (declaratively?) on a bunch of hosts via ssh.

Inventory is to sort of take a self-defined inventory on a bunch of hosts?

One final question on usage, would it be possible to sync or reference files from the machine running pyinfra on the remote hosts? Or would that have to be done indirectly via running shell commands to sync?

I think distillation in the original sense isn't being done anymore but finetuning on outputs from larger models like GPT-4 is a form of distillation (top-1 logit vs all logits and a curated synthetic data instead of the original dataset)

On quantization though its still weird how just the weights are quantized in methods like gptq / int8 while there are other methods which quantize the activations as well. There's also the matter of KV cache still being in original 16bit precision regardless which is also unsolved here. Do you have any thoughts or insights into this?

Head over to the elueuther.ai discord and discuss with some of the folks there. Tons of small experiments with LLMs can use the $10k in compute

It prohibits anything that competes with OpenAI services i.e as long as you're not literally providing an LLM API commercially you should be fine

> This means it is way cheaper to look something up in a vector store than to ask an LLM to generate it. E.g. “What is the capital of Delaware?” when looked up in an neural information retrieval system costs about 5x4 less than if you asked GPT-3.5-Turbo. The cost difference compared to GPT-4 is a whopping 250x!

In a narrow use-case of a strict look-up. This seems to exaggerate the cost difference while having completely different trade-offs.

I think parts of the write-up are great.

There are some unique assumptions being made in parts of the gist

> 10: Cost Ratio of OpenAI embedding to Self-Hosted embedding

> 1: Cost Ratio of Self-Hosted base vs fine-tuned model queries

I don't know how useful these numbers are if you take away the assumptions that self-hosted will work as well as API.

> 10x: Throughput improvement from batching LLM requests

I see that the write up mentions memory being a caveat to this, but it also depends on the card specs as well. Memory Bandwidth / TFLOPs offered by say 4090 is superior while having the same amount of VRAM as 3090. The caveat mentioned with token length in the gist itself makes the 10x claim not a useful rule of thumb.

I've been looking for a course like this! Especially great given how much of the recent progress in training large models is made possible with the aid of flash attention and fused kernels