How we turned an agentic vulnerability research workflow into a continuous security testing system
In AI-assisted vulnerability research at Cribl, I wrote about how our Product Security team brought AI into the vulnerability research loop.
We got the agent to become a really helpful part of the workflow. It read product code, traced user-controlled data across modules, identified trust boundaries, built focused proofs of concept, validated behavior in testing environments, and helped turn confirmed issues into remediation-ready findings.
We wanted to take the same workflow and make it continuous.
The result is a security testing harness that turns code changes, research requests, coverage gaps, and known vulnerabilities into effective research tracks. It uses agentic reasoning for the parts of vulnerability research that require context and judgment, then uses deterministic validation to prove what happened on the target.
This was not as simple as running the original prompt on a schedule. We had to build the state, scheduling, proof requirements, safety controls, and evaluation needed to make the workflow repeatable.
Why we built it
AI helped us shorten the loop between code review and validation, but each investigation was still a point-in-time effort. The context, next research path, and earlier results had to be carried from one session to the next.
That becomes difficult across a large product surface. New code is constantly being added. Existing features change. A trust boundary reviewed six months ago may behave differently today. Some areas receive a lot of security attention while quieter parts of the product remain untested.
Earlier research matters too. If a path was ruled out because of a validator, the harness should remember that. If the validator changes later, that assumption may be worth testing again. Confirmed vulnerabilities also need to be re-tested against new releases so we know the fix continues to hold.
We needed a system that could preserve that context, decide what needed attention, validate findings consistently, and carry evidence forward.
Turning the workflow into a harness
The core research loop stayed close to the process from the first post:
research brief → hypotheses → source-to-sink trace → focused proof of concept → target validation → remediation-ready finding
The difference is that each step now belongs to a durable research track. A track records the source revision, research goal, trust boundary, deployment targets, authorized actors, hypotheses, active trace, proof of concept, attempts, target evidence, cleanup, and final result.
The work moves through five lanes:
Intake turns a report, code change, or research goal into a structured brief.
Hunt prioritizes recall failures, changed trust boundaries, unreviewed areas, and fair rotation across repositories.
Validate traces the strongest hypothesis, builds a focused proof, and runs it against a local target.
Cloud validates eligible paths in a dedicated Cribl-owned security testing organization.
Release re-runs confirmed proofs against new product versions.
This gives the harness a clear history. It knows what changed, what was already tested, what evidence supported the result, whether the behavior was reproduced locally or in the hosted product, and what path should be investigated next.
Agentic research with deterministic proof
The agent is most useful where vulnerability research requires context. It can search a large codebase, connect behavior across modules, follow a value through validators and asynchronous jobs, compare several attack paths, and build a test around the strongest hypothesis.
Changed files are useful starting points, but they are not the review boundary. A vulnerability may begin in an API handler and end in a background worker, dependency wrapper, expression engine, filesystem operation, or another repository. The agent can form several ranked hypotheses, but it works through one at a time so the research stays focused on completing the path from controlled input to security impact.
Once a proof is ready, a deterministic runner handles validation. This is the same standard we apply to vulnerability research generally. A strong source trace is useful, but the claimed behavior still needs to be reproduced on the target.
The model explores. The target provides the proof.
Every new proof includes:
a benign positive control to confirm the intended surface is reachable;
a negative control, such as a safe input, insufficiently privileged actor, invalid credential, or patched version;
target-produced evidence tied to a unique nonce for the current run; and
verified cleanup of any user, configuration, package, file, or other test artifact.
These controls make sure the proof matches the claim. For example, some web applications return their main HTML page with a 200 status even when the requested API route does not exist. Instead of treating the status code as proof, the harness verifies the expected API response and confirms that the claimed effect actually occurred.
Local and Cloud results also stay separate. A path may be vulnerable in one deployment and unreachable, disabled, or patched in another. The same applies to test failures. An expired token, unhealthy target, broken listener, or wrong workspace is not a vulnerability result. The harness records those conditions separately so it can distinguish a ruled-out hypothesis from a test that could not run.
Coverage should create work
One of the biggest changes was treating coverage as an input to the scheduler instead of only a dashboard metric.
Code changes create research work, especially when they touch APIs, authorization, parsers, package handling, expression evaluation, outbound requests, or other trust boundaries. The harness also looks for product surface that has never been exercised, the age of the oldest untested endpoint, trust-boundary families without recent review, recall failures, and confirmed vulnerabilities that need regression testing.
Those gaps become queue entries.
We still use lexical patterns and known high-risk paths to help rank code, but they do not define the whole discovery surface. Authorization problems, business-logic flaws, cross-file data flow, and new abstractions may not contain an obvious security keyword. Broader sweeps and product-aware reasoning help cover those paths, while fair rotation keeps the busiest repository from consuming all of the research time.
Coverage is useful when it changes what the system tests next. A dashboard by itself does not reduce dark surfaces.
Evidence-driven iteration and safety
Proofs do not always work on the first attempt. A request may reach the target but use the wrong trigger. A listener may be available locally but unreachable from the container. A product version may have changed the API contract. The original hypothesis may depend on an assumption that the source trace does not support.
The harness allows bounded iteration, but every retry must test a new fact. That fact can come from a response body, target log, proof file, callback, metric, or additional source inspection. Infrastructure retries use a separate budget so a temporary target or authentication problem does not consume the attempts reserved for research.
Testing stays inside disposable local labs and dedicated security testing environments. Proofs run with only the credentials and environment variables declared for the target, under an unprivileged operating-system user without access to host or Cloud credentials.
The system also has a kill switch for autonomous research. Disabling autonomous scheduling does not stop intake, validation, release regression, or cleanup. It only prevents the harness from creating new research tracks on its own.
Measuring recall
The first post focused on how AI helped us move from suspicious code to a tested security finding. Once that workflow became continuous, we needed a way to measure what it missed.
We built a blind historical recall suite using known vulnerabilities and pinned source revisions. The harness receives the product context and research goal without being handed the answer. It has to rediscover the controlled input, trace the path across modules, identify the required actor and trigger, build the proof, and reproduce the behavior on the correct target.
The benchmark measures more than whether the model mentioned the right file. A successful result needs the complete trace, correct deployment context, target-produced evidence, negative controls, and verified cleanup.
We use that evidence as an enablement gate for autonomous scheduling. When recall fails, we can see whether the problem was discovery, source tracing, artifact handling, actor selection, target setup, proof design, or cleanup. It turns “the agent seems good at vulnerability research” into something we can test.
How you can help
If you’re interested in helping Cribl improve our security, we encourage ethical security research and responsible reporting through our Vulnerability Disclosure Program.
Takeaways
The biggest shift in the first project was that AI moved from being a support tool to being part of the vulnerability research loop.
The shift here was turning that research loop into a system.
The agent still reads code, connects behavior across modules, identifies trust boundaries, forms hypotheses, builds proofs of concept, and uses target feedback to improve the test. The harness preserves that context, schedules the next research path, creates work from coverage gaps, validates proofs against controlled targets, and measures blind recall.
Building it reinforced a few things. Durable state matters more than a long transcript. Target evidence needs to be independent from the proof that generated it. Cleanup is part of the proof. A wrong hypothesis, patched target, expired credential, broken listener, and incomplete cleanup all require different next actions.
Recall gives us a way to improve the complete workflow instead of tuning prompts based only on the findings that made it through.
The result is the same research process made repeatable, measurable, and able to keep up with a product that continues to change.
