New to observability? Find out everything you need to know.
Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn More >Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn More >Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn More >The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn More >Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief >AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn More >Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
Get this Gartner® report and learn why telemetry pipeline solutions represent a robust and largely untapped source of business insight beyond event and incident response.
Download Report >Escaping Data Lock-In Amidst Industry Takeovers
Learn how IT & Security engineers increase resilience & provide more options for analysis to make decisions faster with better data.
Watch On-Demand >Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now >Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories >Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study >Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now >Take Control of Your Observability Data with Cribl
Learn More >Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide >Stay up to date on all things Cribl and observability.
Visit the Newsroom >Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders >Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More >Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert >November 29, 2022
Picture this! The coffee is hot, the keyboard is ready to rock, the bandwidth is unused, and the software is deployed (or the cloud is waiting patiently)…. but the data is missing! That’s right, most of us have been there. In our industry, it is very common for data to be the lowest common denominator for many projects. Whether you’re testing something new in your home lab, or you’re troubleshooting a new Cribl Pack in your dev environment before pushing to production, having data is pretty important.
Now, we have been in that situation more times than we can count, and since you probably already know one of our most important value is Customers First, Always, we build products to help customers first AND then everything else. So let’s talk about a very powerful feature that comes right out-of-the-box with Cribl Stream. That feature is a built-in data generator (Datagen) that can be useful for testing and/or troubleshooting.
Cribl Stream ships with several built-in datagen templates, but you’re also able to use your own data samples to create datagens. We’ll walk through both below or you can watch the included video.
First, navigate to Data/Sources, and filter on “System and Internal”.
Next, choose Datagen, then Add New. Below, you can see some built-in datagen templates. Give it a name, choose a datagen file, and event rate.
After performing a Commit & Deploy, click Live to see the datagen events.
Congratulations, you’ve configured a datagen! Now you can treat this as any other source, send the data through a custom pipeline, or a pack, then send it off to a destination.
Now using Cribl’s built-in datagens is all well and good, but what about using your own data sample to create a datagen that is more relevant to your needs? Well, let’s walk through how we do that.
First, navigate to Processing/Pipelines. On the right hand side of the page, you should see the Sample Data pane. Assuming you have a set of sample data in a file, you can either upload the file directly, or copy/paste:
If you decide to go the upload route, you can upload events in any format you’d like as long as you can correctly identify where the timestamp and event breaks are located.
Below, I’ve pasted my sample in. This is where you would configure event breaker rules, as well as timestamp settings. You can validate these are working by comparing the “IN” view and the “OUT” view. In my sample below, you can see the timestamps properly detected and highlighted in purple, and the event break in red:
Once we’re happy with the timestamp and event breaker settings, choose “Create A Datagen File” in the bottom right corner. In this step, we can instruct Stream to replace the sample timestamp with a new timestamp and the desired timestamp format. In this case, we’ll leave it as is (3-letter month, two digit day, two digit hour, minute, second). Don’t forget to check the “File Name,” before moving to the next step. Finally, choose “Create A Datagen File.”
Finally, same as before, we’ll navigate to Data/Sources, and choose Datagen. Create a new Datagen, and our newly added datagen file should be listed in the dropdown:
Save this, and be sure to Commit & Deploy. Once the worker picks up the new config, we should be able to see the events flowing under live data:
This should get you started with your own custom datagens. These are perfect for testing a new config, troubleshooting a Pack, and probably many more scenarios we haven’t thought of yet. If you have any questions, be sure to hit up our docs and or Cribl Curious.
The fastest way to get started with Cribl Stream and Cribl Edge is to try the Free Cloud Sandboxes.
David Maislin Dec 7, 2023