I don't understand how a "CSV file/database/model" of 70,000,000,000 (70B) "parameters" of 4-bit weights (a 4 bit value can be 1 of 16 unique numbers) gets us an interactive LLM/GPT that is near-all-knowledgable on all topics/subjects.
edit: did research, the 4-bit is just a "compression method", the model ends up seeing f32?
> Quantization is the process of mapping 32-bit floating-point numbers (which are the weights in the neural network) to a much smaller bit representation, like 4-bit values, for storage and memory efficiency.
> Dequantization happens when the model is used (during inference or even training, if applicable). The 4-bit quantized weights are converted back into floating-point numbers that the model's computations are actually performed with. This is done using the scale and zero-point determined during the initial quantization, or through more sophisticated mapping functions that aim to preserve as much information as possible despite the reduced precision.
so what is the relationship to "parameters" and "# of unique tokens the model knows about (vocabulary size)"?
> At first glance, LLAMa only has a 32,000 vocabulary size and 65B parameters as compared to GPT-3,
> The 65 billion parameters in a model like LLAMA (or any large language model) essentially function as a highly intricate mapping system that determines how to respond to a given input based on the learned relationships between tokens in its training data.
edit: did research, the 4-bit is just a "compression method", the model ends up seeing f32?
> Quantization is the process of mapping 32-bit floating-point numbers (which are the weights in the neural network) to a much smaller bit representation, like 4-bit values, for storage and memory efficiency.
> Dequantization happens when the model is used (during inference or even training, if applicable). The 4-bit quantized weights are converted back into floating-point numbers that the model's computations are actually performed with. This is done using the scale and zero-point determined during the initial quantization, or through more sophisticated mapping functions that aim to preserve as much information as possible despite the reduced precision.
so what is the relationship to "parameters" and "# of unique tokens the model knows about (vocabulary size)"?
> At first glance, LLAMa only has a 32,000 vocabulary size and 65B parameters as compared to GPT-3,
> The 65 billion parameters in a model like LLAMA (or any large language model) essentially function as a highly intricate mapping system that determines how to respond to a given input based on the learned relationships between tokens in its training data.