Fine Tuning Language Models

I want to fine tune a language model to summarize the action items from meeting transcriptions. Due to privacy concerns, I need to run on the model on my local machine. I also need to be able to fine tune the model for our specific need. How do I pick a model to start with? There are so many options, and I feel lost.