Comment

Comment by c6401

IMO the simplest option is llamafile (it is multiplatform using "cosmopolitan" lib so should run on Windows too, but I haven't tried)

    wget https://huggingface.co/Mozilla/Llama-3.2-1B-Instruct-llamafile/resolve/main/Llama-3.2-1B-Instruct.Q6_K.llamafile
    chmod +x Llama-3.2-1B-Instruct.Q6_K.llamafile
    ./Llama-3.2-1B-Instruct.Q6_K.llamafile --server

parent c6401 2 years ago

Comment by c6401

Discussion