Following the latest runs of Cribl’s SecIT Bench, GLM 5.3 and GPT 6 Astra sit atop our leaderboard with statistically indistinguishable results: GLM at 81.4 +/- 2.9% and Astra at 81.0 +/- 3.4%. On the surface, the two models look pretty similar. But once we dug into how they actually worked through the investigations, the differences became clear.
That’s part of what makes benchmarking agents interesting. The final score only tells you so much. How a model gets to an answer, how much work it takes to get there, and what happens when you give it more data can tell you just as much about how it will perform in the real world.
Technical tradeoffs
While the outcomes across the tasks were similar, the models took very different paths to get there. Astra was far more token efficient, averaging 21 turns and 749,000 input tokens per trial, while GLM 5.3 averaged 45 turns, 2.7x as many tool calls, and 2.2 million input tokens. Despite this, GLM 5.3 came out approximately 3.5x less expensive.
Astra’s fewer turns and tool calls also made it much faster, coming in at just over 8 minutes compared with 14 minutes for GLM 5.3. Looking task by task, Astra outperformed GLM 5.3 on 13 and tied it on eight.
The gap also started to widen as we increased the amount of data available to the models. During our analysis, we observed a statistically significant trend where Astra performed better relative to GLM 5.3 as dataset size increased. GLM 5.3 showed a negative trend in accuracy as the datasets grew, while Astra stayed relatively consistent.
So while the headline scores are nearly identical, that similarity starts to break down once the models have more data to work through.
A tale of two approaches
The models’ investigation styles also differed quite substantially. GLM 5.3, broadly speaking, focused on exhaustive detail. It spent the early part of its investigations digging into individual signals and inspecting the structure of the data with a fine-toothed comb. We often found it running queries specifically to understand individual fields or small pieces of the dataset before moving on.
It is most comparable to a forensic analyst. Much like a CSI investigator working through a crime scene piece by piece, GLM 5.3 tends to inspect everything it can before settling on a conclusion. Below are the most common types of commands GLM 5.3 runs:
Inventory:
cat /data/trigger.md; echo "---"; ls -la /dataFile sizes and line counts:
du -sh /data/logs/*; wc -l /data/logs/logsimulator-*/*.log | sort -nSample a single source:
head -50 /data/logs/logsimulator-a7f2e3c9-storefront-1/stdout.logCompare boundaries:
tail -5 storefront.log; head -3 order-service.log; tail -3 order-service.logCount categorical fields:
grep -o '"status":[0-9]*' storefront.log | sort | uniq -c
Astra’s approach was more focused on connecting the dots early. Rather than inspecting each clue one at a time, it tended to pull together evidence from multiple sources and look for the broader pattern. It spent the early part of its investigations consolidating evidence across structures and was able to reach a conclusion in far fewer turns because of it. That also seems to be part of why Astra held up better as the amount of data increased. It was doing more work across multiple sources at once instead of drilling into each one separately. Astra’s commands demonstrate this approach, with the model often doing multiple analyses in a single turn:
Inventory:
ls -la; cat trigger.md; find logs -type f | sortMulti-source sampling:
wc -l logs/*/* ...; for f in ...; do head ...; tail ...; doneJSON aggregation:
jq -s 'map(select(...)) | group_by(...) | map({...})'Service-log rollups:
awk '{ ... } END { ... }' inventory.log | sortGrouped counts:
cut ... | jq -r 'select(...) | ... | @tsv' | sort | uniq -cCross-source checks in one command:
grep ...; awk ...; jq ...; echo '=== ... ==='
Those different approaches also came with different failure modes. Because GLM 5.3 was generally better at picking up minute details, it often surfaced very specific signals that Astra missed. But we also saw that detail work against it. In some cases, GLM 5.3 latched onto a narrow signal and became more confident in the conclusion than the evidence really justified. Astra had the opposite problem. By focusing more on the bigger picture, it sometimes missed the kind of granular evidence GLM 5.3 was good at finding. That led to penalties in scenarios where a more specific or heavily supported conclusion was required.
The main takeaway for us is that similar benchmark scores do not necessarily mean two models behave the same way. GLM 5.3 and Astra got to roughly the same place, but they got there in very different ways, and each approach came with its own strengths and weaknesses.
That’s also where we think benchmarks like SecIT Bench can get more useful. We don’t just want to know whether a model got the answer right. We want to understand how it investigated the problem, where it struggled, and what happens as the task gets harder. Future versions of SecIT Bench will push further in that direction, with larger datasets and more complex scenarios designed to expose more of those differences.
Have a scenario, dataset, or idea for how AI agents should be tested in IT or security? Let's build the next benchmark together. Contact us and join our Slack community. And if this is the kind of problem you like working on, we’re hiring. Come help us figure out how to build, test, and evaluate AI for real-world IT and security work.
