Today we released our paper on arXiv. I also wrote a thread with some of the key details.

Interesting Tidbits

We find that Reasoning Gym is a highly effective evaluation of frontier models’ problem solving capabilities. Here are results of several models on RG hard configs:

Additionally, we run some small RL experiments and find that RLVR on RG mathematics data can improve models on common LLM benchmarks at the 3B scale.

RG has already been adopted for training and evaluation by NVIDIA researchers in their ProRL paper.

It’s also possible to use RG data in William Brown’s popular verifiers RL library, since he built a direct integration. Thanks to Will!

The Future

We want to build an even bigger library of reasoning environments. If you have ideas, please feel free to let us know, or even contribute datasets generators directly via pull requests to our repository.

Running more evaluations is also of interest. If anyone has OpenRouter or other API provider credits, we’d love to find out how more frontier models fare on RG tasks.

Background

The open-source release of DeepSeek-R1 stirred up a lot of interest, in both international markets and the open-source AI community. Two of the key innovations justifying the excitement are:

• GRPO, originally published in 2024 with DeepSeekMath, is more cost-effective than the dominant PPO for reinforcement learning on LLMs. GRPO removes the value function, reducing resource usage since the value function was typically a neural model of similar size to the policy model.
• Algorithmically verifiable problems as training data, allowing rules-based rewards in the RL process rather than a (difficult to train and often unreliable) neural reward model.

Goal

Reasoning Gym seeks to take advantage of both of the mentioned DeepSeek innovations by building a procedural dataset generator for algorithmically verifiable problems. This will allow anyone to train reasoning models using an RL technique like GRPO, without going through the process of curating huge volumes of data.

Many datasets have configurable complexity or difficulty levels, allowing virtually infinite datasets of varying challenge levels.

To achieve this, contributors can write their own dataset classes in a range of categories such as games, logic puzzles, algorithms, coding, geometry, and others. My own first contribution to the project was a Futoshiki generator.

Further Work

Going forwards a key aim is to build a “curriculum”, where data is generated with progressing levels of complexity, depending on the learning progress being made by the model being trained.

The repository for the project also contains examples for training LLMs using RL with various frameworks, including OpenRLHF, trl, and veRL.

Finally, building supplementary reasoning datasets, which are not procedurally generated, will be explored in order to augment the procedural generators.

Contributing

Anyone can contribute to Reasoning Gym by commenting on the issues in the GitHub repository and raising pull requests. The project readme also details how to join the community on Discord.

Dataset

We release the first version of the OpenAssistant Conversations Dataset, a fully volunteer-contributed dataset consisting of over 50,000 high-quality instruction and response pairs across multiple languages. The dataset is released under a permissive Creative Commons license, meaning it can be used for a wide range of purposes, including commercial. The instruction and response pairs have been used successfully to tune a pretrained large language model (LLM) to follow instructions using supervised finetuning (SFT), demonstrating their value.

In addition to the instruction and response pairs, we release accompanying label data from community moderation, with each message being rated by several users for categories such as whether it is spam, its quality, its helpfulness, and others. Finally, each prompt has multiple response options, and we release the accompanying human ranking data for relative ranking of the responses for each prompt. This data can be used to train reward models for reinforcement learning from human feedback (RLHF).

A more in-depth breakdown and analysis of the dataset will be available in the upcoming Open Assistant Dataset version 1 research paper.

Models

We release a range of models tuned from two different base models, Pythia 12B, and LLaMa 30B.

Our Pythia-based 12B parameter model is released under the Apache 2.0 license, making it available for a wide range of uses including commercial. This is the most capable fully open-source instruction-tuned model available, having been trained on more data than others such as Databricks’ Dolly. While larger public instruction-tuned models exist, such as Vicuna, they are derived from LLaMa and therefore not fully open-source.

Our LLaMa-based 30B model will be released in the form of weight deltas, meaning you must have a copy of the original LLaMa weights in order to use it. The restrictions placed on the use of LLaMa models by Meta must also be followed, meaning the model is available only for research use.

We additionally release the reward model used for RLHF training under a permissive Apache 2.0 license.

Chat Interface and Safety

You can chat with an OpenAssistant LLaMa-based 30B model using the free chat interface here. This will also help us build better datasets and models as responses and feedback are recorded, so please use the thumbs up and down ratings liberally!

In conjunction with OpenAssistant, safety techniques for LLMs were developed under the blade2blade project. Our chat interface will soon use blade2blade to mitigate harms.

Future Work

We will in future release improved models and models tuned with RLHF. We are additionally considering several streams of future work, including:

• Tuning other open-source models to follow instructions using OpenAssistant data, such as Cerebras-GPT, GPT-J, GPT-NeoX.
• Developing integrations for Open Assistant, similar to the plugins proposed for ChatGPT.
• Expanding the Open Assistant data and preparing a corpus for pretraining a large language model (LLM) from scratch.

Contributors

A fantastic team has been working on this project, and you can see an incomplete list of those who contributed to development here!

If you wish to get in touch with me to discuss the project or otherwise, you can message me on LinkedIn (link below) or email me at oliver ge stanley (_at_) gmail (_dot_) com (address obfuscated to avoid spam).