If you use REST Collector, you already know how powerful it can be. REST Collector gives you a flexible way to pull data from HTTP-based APIs that Cribl does not support out of the box, which makes it an important bridge for customers trying to bring more data into Cribl without waiting on a packaged integration.
The problem has been that sometimes things can go sideways. When that happens it can be very difficult to see what and why, requiring combing through logs, using tools like Postman / Webhook.site, or even standing up your own webhooks.
No more. Our new REST Collector Debugging changes everything!
The existing experience
Prior to the new debugging, you had very little visibility into failures at the HTTP request and response level. If authentication failed, discovery broke, paging behaved unexpectedly, or collection returned something you did not expect, there often was not an obvious place to go to understand what happened.
There were ways to piece things together. You could inspect logs. You could expand log details. You could sometimes use external tools or mock services. But that workflow was hard to discover, hard to follow, and often still incomplete. In many cases, teams were left guessing, trying different changes without enough visibility into what Cribl actually sent or what the upstream API returned. We can do a lot better.
What is new in REST Collector Diagnostics
With REST Collector Diagnostics, you can now run a collector job in debug mode and inspect the request and response details interactively. That includes visibility across the major phases of the collector, including authentication, discovery, and collection. We’re making it obvious so you can get the data you need when you need it.
As you can see in the screenshot below, instead of digging through logs and trying to infer what happened, you can now enable debugging and quickly see everything you’ve wanted; headers, payload, urls, status code. Heck, we’re even giving you a curl!
Using it in practice
One example we showed in the video is an authentication issue.
A collector run captures no events, which by itself does not tell you much. With diagnostics enabled, you can open the job, go to the debug view, and immediately see that the request failed with a 400 bad request during authentication. That kind of visibility changes the debugging experience from “I think something is wrong” to “I know exactly where to start.”
Another important scenario is when the collector partially works. Discovery might succeed, but collection might fail. Or data may come back, but not in the way you expect. Diagnostics now lets you inspect each discovery step and each collection step independently, so you can drill into exactly where behavior diverges from what you intended.
That level of step-by-step visibility is especially important for real-world APIs, where rate limiting, pagination behavior, header issues, or auth details can create failure modes that are not obvious from a top-level result.
Secure by design
When we designed this, it was important that we improve visibility without simply dumping sensitive details into logs.
The diagnostic data is intended for interactive display and is not ingested into Cribl, rather it is encrypted and stored locally. We also sanitize the data to remove tokens and secrets before it is displayed. In the experience itself, redaction is enabled by default and is configurable.
That balance matters. You want enough information to solve the problem quickly, but you do not want to create new risk in the process.
Our favorite feature, the curl command.
One part we love and which we think you’ll love is the generated curl command.
You likely already use curl as part of your debugging workflow for REST Collector. The new debugging makes it easy to see the equivalent curl statement so you can run it yourself from the shell or even import it into a tool like Postman.
This is just the beginning
We are extremely excited about our new diagnostics because it is going to help you fully harness the power of REST Collector within your organizations. This is a big step in giving you a much better experience.
REST Collector is an incredibly powerful tool and there are many other areas we plan to invest in to make it even better. This IS just the beginning.
