October 9, 2026 · 5 min read

A Long-Term Goal for Quartz: Beat SmolLM-135M

We have a long-term goal for the Quartz series: a model that beats SmolLM-135M, Hugging Face's small base model, on the same benchmarks. SmolLM has 35% more parameters than ours and saw about 90 times more text, and we train on one 6 GB GTX 1660 Ti. This will take several releases, and we do not expect the model training right now, V3, to get there. It is the next step, and it will show us how far away we are. This post says what the goal is, what would count as reaching it, and what V3 is testing.

The target

SmolLM-135M has 135 million parameters, was trained on 600 billion tokens on 64 H100 GPUs, and is released under Apache 2.0. Its model card lists only one benchmark, so the scores below are our own zero-shot runs. Quartz will go through exactly the same setup. The random-guess column shows how much of each score is signal.

Where Quartz stands today

Quartz Micro Preview V2 has 100.09 million parameters. It uses the stock Llama architecture with a 32,768-token vocabulary, so it loads in Transformers, MLX and llama.cpp without custom code. It went public yesterday as a base model and an instruction-tuned model.

V2 made four passes over a 100-million-token mix, about 0.4 billion tokens in all. It writes fluent short text and often states wrong facts with confidence. The instruction-tuned model passed 5 of 15 keyword fact checks in our own test. We have not run Quartz on the five benchmarks yet, so today there is no score to compare. V3 is the working name for the next one.

The size of the gap

Per parameter the difference is larger. SmolLM-135M saw about 4,400 tokens for every parameter. V3 will see about 65. The web and textbook text in V3 is read once, with no repeats. Even so, our whole run covers about 1% of what SmolLM trained on, so V3 will not close the gap by volume. Over the next releases we will need more tokens and more compute. V3 tests whether better data gets us closer for the same amount of training.

What V3 tests

V3 is the first run built with this goal in mind. It tests three ideas.

• A mix chosen for small models. About 98% of V3's 6.5 billion tokens is filtered educational web text (FineWeb-Edu 2.6B, DCLM 0.5B), textbook-style synthetic text (Cosmopedia 1.3B), Wikipedia (0.8B), StackExchange (0.45B), math (FineMath 0.39B) and Python (Python-Edu 0.32B). The other 2% is question-answering and chat data, including examples where the right reply is that the context does not contain the answer. That slice includes the training splits of SQuAD and TriviaQA. Neither is one of the five benchmarks in the table, but we are listing them so you know exactly what the model has seen.
• Test sets kept out. We remove any training text that shares a 10-word window with the test or validation splits of ARC, HellaSwag, PIQA, WinoGrande, LAMBADA, OpenBookQA, SciQ, BoolQ, TriviaQA, MMLU and CommonsenseQA. In FineWeb-Edu alone that removed 63,750 documents.
• Settings chosen by validation loss, never by benchmark score. We ran three 300-step pilots. Final validation loss was 5.287 at a learning rate of 3e-4, 5.339 at 6e-4 and 5.460 at 1.2e-3. The run uses 6e-4. The pilots are too short to settle the choice.

What counts as reaching it

We will report all five benchmarks, with the same harness and the same settings for both models, and reaching the goal means beating SmolLM-135M on at least three of the five by more than the evaluation's own margin of error. At this size some scores sit close to random guessing: SmolLM-135M's WinoGrande score is only 3 points above it, so small differences there are mostly noise. We will also say where we lose.

We do not expect V3 to win. When it finishes we will run it through the five benchmarks and publish the numbers, whatever they are. That gives us the starting gap, and each later Quartz can be measured against it.

SmolLM-135M's size, training tokens and license come from its model card. Its benchmark scores are from our own zero-shot runs. Token counts for V3 are unique tokens from the dataset manifest, rounded.