Comment by amaiya

The latest release of https://github.com/amaiya/onprem (a simple Python package to help use/apply local LLMs based on llama.cpp) includes a customizable, streamlit-based Web app for both general prompting and retrieval augmented generation (i.e., to talk your documents) and can be setup fairly easily:

  # run at Python prompt
  from onprem import LLM
  llm = LLM()
  llm.ingest('/your/document/folder')

  # run at command line
  onprem --port 8000

More information and example use cases are in the documentation: https://amaiya.github.io/onprem