I built and trained a BERT on my gaming laptop (3070 RTX) to ~94% of BERT-base's performance in ~17 hours* (BERT-base was trained on 4 TPUs for 4 days). This notebook goes over the whole process, from implementing and training a tokenizer, to pretraining, to finetuning. One feature that makes this BERT different from most (though not unique) is the use of relative position embeddings.
Edit - for anyone unsure about what "BERT" is or its relevance, it's a transformer based natural language model just like GPT. However, where GPT is used to generate text, BERT is used to generate embeddings for input text that you can then use for predictive models (e.g. sentiment prediction), and that process is also demonstrated in the notebook.
*Edit 2 - The 17 hours are pretraining only, not including the time to train the tokenizer, or finetuning.
Edit - for anyone unsure about what "BERT" is or its relevance, it's a transformer based natural language model just like GPT. However, where GPT is used to generate text, BERT is used to generate embeddings for input text that you can then use for predictive models (e.g. sentiment prediction), and that process is also demonstrated in the notebook.
*Edit 2 - The 17 hours are pretraining only, not including the time to train the tokenizer, or finetuning.