Data Collection – Listening To Our Customers

September 17, 2020
Written by
Clint Sharp's Image

As Co-Founder and CEO, Clint leads the Cribl team and oversees product and engineering, s... Read Moreales and marketing, and general and administrative functions. In his role, he has led the team to several straight years of triple digit customer and ARR growth, achieved $100 million in ARR in less than four years–becoming one of the fastest infrastructure companies to reach centaur status–and secured more than $400M in funding from the world’s top investors. Clint brings a passion for bringing innovative products to market that deliver unmatched value to customers, which comes from his two decades leading product management and IT operations at technology and software companies like Splunk and Cricket Communications. His experience as a practitioner means he has deep expertise in network issues, database administration, and security operations, and he personally understands the fundamental challenges that enterprise IT and Security teams face. Read Less

Categories: Announcements, Learn

From the very start, the Cribl founding team came in with some strong assumptions, which you can even see baked into the name of our first product: LogStream. The founders have been in the logging ecosystem for 30+ combined years, having worked as customers and with customers. We knew organizations wanted to work with log data in motion to route the right data to the right store, in the right format. We knew logging use cases demanded their own fit-for-purpose solution, with a data centric experience that makes it easy to work with gritty log data. We knew in order to solve customers’ pain, we needed to meet customers where they were at and support all their existing agents and collectors.

Organizations are really struggling to get data into logging tools, time series databases, and data lakes, in the right shapes, structured properly for the data store. Solving this problem isn’t just about data in motion. Over the last year, we’ve gotten numerous requests to be able to collect data. Requests usually come in the form of a question, like: “how can I collect back what I put to rest in cheap storage?”. We fulfilled that request in our 2.2 release with Ad-Hoc Data Collection tools. Now we can easily replay data in cheap storage to any destination.

But, we also received a lot of questions like: “how do I get data from Office 365 APIs?” or “how can I collect from a REST API on an interval?” There’s no category name for this kind of problem, but working with our customers, it’s become clear that reliably and scalably collecting data from APIs is a huge pain point. Today, they’re forced to run dozens of different collectors, each with their own unique configuration and administration. Many of these collectors are custom scripts written by in-house engineers, because no vendor had even built a collector for that type of data. 

Keeping all these little collectors running is simply operations toil. Most collectors leave scaling to the administrator: hand configured individual nodes, each handling a slice of the workload. Each has to be documented and operationalized. They need monitoring. Each collector comes with its own failure modes, and when they fail, someone has to diagnose and resolve the issues. One node fails, you lose data. 

LogStream 2.3 takes away this toil. Now, as part of the same platform that has consolidated receiving from all of your deployed agents, you can easily collect data from anywhere, on an arbitrary interval. Run your custom scripts. Collect from REST APIs. The system handles sharding and scaling, transparently to the administrator. You can reuse the same infrastructure you have for receiving, or you can create different worker groups that might have different IAM/security roles to make authentication and authorization easier and more secure.

Scheduled Data Collection is a great example of how Cribl’s innovation is driven by our customers-first culture. An Observability Pipeline demands streams processing, as well as batch and mini-batch processing, in the same engine. It blurs lines typically drawn in data processing. Cribl meets customers where you are at, and we solve problems others are ignoring. 

There’s so much more coming. What’s your biggest pain point we’re not solving? Our product is inspired by our customers and we want nothing more than to solve problems for you. Hit me up in our community Slack, I’d love to hear from you!

Feature Image

The Evolution of Data Archiving: How to Get Immediate Access to Archived Data

Read More
Feature Image

The Stream Life Podcast Episode 105: Exploring Cribl Copilot!

Read More
Cribl Copilot

Cribl Copilot: Your Trusted AI Wingman for Deploying, Configuring & Troubleshooting

Read More

Try Your Own Cribl Sandbox

Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.


So you're rockin' Internet Explorer!

Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari

Got one of those handy?