anakin87
Joined 3 years ago
Karma
-1
Submissions
12
Member since
Apr 4
Recent submissions
Comment · 7 months ago
Comment · 1 year ago
LLMs improve when they can practice and reason in interactive environments.
Recent work (DeepSeek-R1, GRPO) shows RL can teach models to prefer better outputs by giving rewards.
But most RL environments for LLMs are fragmented or closed. That makes it hard for the community to experiment or reproduce results.
Environments Hub is a new open platform by Prime Intellect where anyone can share RL environments for training or evaluating LLMs. Think of them as software packages: data, harness and scoring rules.
Agents today incorporate models and tools (from APIs to a terminal), so environments need to capture that complexity.
I wrote a hands-on walkthrough covering:
- RL + LLM basics
- Navigating the Environments Hub
- Evaluating models and agents
- GRPO-style training of a tiny model on an alphabetical sort task
Comment · 1 year ago
I experimented with GRPO lately, since I am fascinated by models learning from prompts and rewards - no example answers needed like in Supervised Fine-Tuning.
After the DeepSeek boom, everyone is trying GRPO with GSM8K or the Countdown Game, but I wanted a different challenge.
So I opted for teaching a model to create a schedule from a list of events and priorities.
Choosing an original problem forced me to think about the problem setting, generate data, choose the base model, design reward functions, and run multiple rounds of training, hoping that my model would learn something.
A fun and rewarding experience :-)
I learned a lot of things, that I want to share with you.
---
- Blog post: https://huggingface.co/blog/anakin87/qwen-scheduler-grpo
- Code: https://github.com/anakin87/qwen-scheduler-grpo
- Hugging Face collection (dataset and model): https://huggingface.co/collections/anakin87/qwen-scheduler-g...
---
Some hot takes from my experiment
- GRPO is cool for verifiable tasks, but is more about eliciting desired behaviors from the trained model than teaching completely new stuff to it. https://arxiv.org/abs/2504.13837
- Choosing the right base model (and size) matters.
- "Aha moment" might be over-hyped. https://oatllm.notion.site/oat-zero
- Reward functions design is crucial. If your rewards are not robust, you might experience reward hacking (as it happened to me).
- Unsloth is great for saving GPU, but beware of bugs.
Comment · 1 year ago
I experimented with GRPO lately, since I am fascinated by models learning from prompts and rewards - no example answers needed like in Supervised Fine-Tuning.
After the DeepSeek boom, everyone is trying GRPO with GSM8K or the Countdown Game, but I wanted a different challenge. So I opted for teaching a model to create a schedule from a list of events and priorities.
Choosing an original problem forced me to think about the problem setting, generate data, choose the base model, design reward functions, and run multiple rounds of training, hoping that my model would learn something.
A fun and rewarding experience. :-)
I learned a lot of things, that I want to share with you.
Blog post: https://huggingface.co/blog/anakin87/qwen-scheduler-grpo
Code: https://github.com/anakin87/qwen-scheduler-grpo
Hugging Face collection (dataset and model): https://huggingface.co/collections/anakin87/qwen-scheduler-g...
Comment · 3 years ago
I recently conducted some experiments with Llama2 and Haystack (https://github.com/deepset-ai/haystack), the NLP/LLM framework.
The notebook can be helpful for those trying to load Llama2 on Colab.
1) Installed Transformers from the main branch (and other libraries)
2) Loaded Llama-2-13b-chat-hf on Colab using 4-bit quantizazion, thanks to the material shared by Younes Belkada
3) Disabled Tensor Parallelism, which caused some issues
4) Installed a minimal version of Haystack
5) Found a hacky way to load the model in Haystack's PromptNode
6) Had a fun chat session with the model, discussing everything from David Guetta to Don Matteo (an Italian TV series)!
Comment · 3 years ago
In my experience, Argilla is a good open source platform for datacentric NLP. And these features are a great addition...
Have you tried it?
Over the past year, we've seen a shift in LLM Post-Training. Previously, Supervised Fine-Tuning was the most important part: making models imitate curated Question-Answer pairs. Now with RLVR and GRPO, we can make models learn through trial and error in dynamic environments, which are software artifacts.
But how to effectively build RL environments?
In the repo, I cover:
- Mapping core RL concepts (Agents, Environments) to the LLM domain.
- Using the Verifiers open-source library to construct single-turn, multi-turn, and tool-use environments.
- Hands-on: taking a small language model (LiquidAI's LFM2-2.6B) and turning it into a Tic-Tac-Toe master that beats GPT-5-mini. Build the game Environment, ese it to generate synthetic data for SFT warm-up, then Group-based Reinforcement Learning.
---
Links
Course: https://github.com/anakin87/llm-rl-environments-lil-course
Video walkthrough: https://www.youtube.com/watch?v=71V3fTaUp2Q
Play against the trained model: https://huggingface.co/spaces/anakin87/LFM2-2.6B-mr-tictacto...
Datasets and Models on HF: https://huggingface.co/collections/anakin87/lfm2-26b-mr-tic-...
---
I'm fascinated by the idea of building these "little worlds" where LLMs can learn, so I hope it's useful.
Feel free to share opinions...