Earlier this month, we released cribl-decision-1.0, our first open-weight decision model. Today, we're following that up with two new releases from the Cribl AI Research Lab:
Telemetry Decisions Bench: A public benchmark and leaderboard for evaluating decision models on more than 4,800 production-like IT, security, and telemetry tasks.
cribl-decision-2: A new family of open-weight decision models trained specifically for telemetry, with significant improvements over our first release.
Telemetry for IT and security is full of decisions. Is an alert suspicious? Does an event match a known attack technique? Should a log be retained, dropped, or routed somewhere else?
These use cases are good candidates for decision models, which select from a set of options and return confidence scores rather than generating text. But they also present challenges that aren't well represented in most general-purpose AI benchmarks. Models need to understand messy, semi-structured data, specialized schemas, and security context, often while operating under tight latency and cost constraints. We wanted to understand which models actually perform well on these tasks, and how much we could improve performance by training specifically for the domain.
Introducing Telemetry Decisions Bench
We built Telemetry Decisions Bench to measure how well models handle the kinds of decisions that come up in real IT and security environments.
The benchmark includes 4,806 tasks across 10 families, covering everything from observability and alert triage to detection and AI agent telemetry. A few examples:
A few details about how we designed the benchmark:
Held out from training. None of the benchmark tasks, their source datasets, or derivatives were used to train our cribl-decision models.
Labels are provided at inference time. Models receive the available options with each request rather than predicting from a fixed set of labels. This lets us test how well they understand a task instead of how well they've memorized particular labels.
We measure more than accuracy. Alongside accuracy, we report negative log-likelihood (NLL), expected calibration error (ECE), latency per decision, and cost for API models.
That last point is important. In telemetry workflows, a model's confidence score may determine whether an event is dropped, routed, or escalated for investigation. A model that's accurate overall but overconfident when it's wrong can be difficult to use safely. And even a highly accurate model may not be practical if it's too slow or expensive to run millions of times.
What we found
To date, we’ve evaluated 40 models, from small open-weight decision models to general-purpose LLMs hosted by the major AI labs. One of the clearest findings was how much domain-specific training helped smaller models.
Our new cribl-decision-2-4B scored 86.5% accuracy, ranking first among models under 5 billion parameters. That's a 10.1-percentage-point improvement over cribl-decision-1.0 at the same size (76.4%). It also outperformed every model under 10B we tested except elephant-1 (86.9%) and came within 0.7 percentage points of our previous 27B model (87.2%), despite being more than six times smaller. These results suggest that a relatively small model trained on the right data can get surprisingly close to much larger models on specialized telemetry tasks.
Introducing cribl-decision-2
For cribl-decision-2, we built on what we learned from the first release, making changes to the architecture, training approach, and, most importantly, the training data.
An architecture better suited for typed decisions. We are still building on the same Qwen base but have replaced LoRA with a full or partial, layer-wise fine-tune along with adding bidirectional attention, reverse scans, and an improved output head.
Domain data, built from scratch. We collected a large corpus of domain-specific documents (e.g. public log datasets, vendor docs, schema standards, etc.) and turned it into a wide range of typed decision tasks. Rather than training on a fixed list of tasks, we generated many task framings over the same data, so the model learns the domain, not a handful of label sets.
General skills kept intact. Domain data was added in a final fine-tuning stage, about 20k questions split roughly half IT/security/telemetry and half general questions replayed from the earlier general training. On the 26-task general suite from the 1.0 release, cribl-decision-2-4B scores 79.4% task-averaged accuracy, versus 78.6% for 1.0.
Technical specifications:
Other things we learned
Beyond the overall accuracy rankings, a few other results stood out.
Confidence matters as much as correctness. In many telemetry use cases, a model's confidence score determines what happens next. For example, a high-confidence decision might trigger an automated action, while a lower-confidence prediction gets sent for further analysis. cribl-decision-2 (4B) achieved an ECE of 0.036, compared with 0.102 for gpt-5.6-luna. Luna was slightly more accurate overall (87.7%), but its confidence scores were less well calibrated. This is an important distinction for applications that use confidence thresholds to automate decisions.
Smaller models saw some of the biggest gains from domain training. Compared with cribl-decision-1.0, the new models gained 12.8 percentage points at 0.8B (60.2% to 73.0%) and 10.1 points at 4B (76.4% to 86.5%). Smaller models have less capacity to absorb domain knowledge during pretraining, so it's particularly interesting to see how much targeted training can improve their performance.
Speed matters at telemetry scale. These decisions may need to run millions of times a day, so even relatively small differences in latency can add up. In our evaluation, cribl-decision-2-4B took 78 ms per decision, compared with 265 ms for elephant-1, the nearest higher-scoring self-hosted model. It was also about four times faster than Microsoft Decision-1 (316 ms), an API model it outscored. Our smaller cribl-decision-2-0.8B took just 20 ms per decision. Latency isn't the same as cost, but these differences matter when thinking about the compute and infrastructure needed to serve models at high volume.
Get started
Both cribl-decision-2 models are available as open weights, and the full benchmark results are public.
Explore the leaderboard: Telemetry Decisions Bench
Try the models: Download cribl-decision-2 from Hugging Face.
What's next
We're working on a larger cribl-decision-2-27B model targeting performance competitive with leading models in the 27B range. We're also continuing to improve Telemetry Decisions Bench with more difficult and realistic tasks, and exploring how to bring these models into Cribl products.
In future releases, these decision models will be available in Cribl StreamAI, our enterprise AI gateway, both as standalone models and built into new and existing features, making Cribl smarter, faster, and easier to use.
If this is the kind of problem you like working on, we're hiring.
