In today’s environments, the number of endpoints seems to be endless. Simultaneously, with more advanced bad actors and increasingly complex systems, it is more important than ever that no endpoint goes unmonitored. However, many solutions simply can’t keep up with this growing scale of data collection at the edge.

Luckily for you, there is Cribl Edge! Cribl Edge is a highly scalable, vendor-neutral agent that allows you to collect and process IT and security data at the edge from Linux and Windows machines, applications, microservices, etc., and deliver it to your destination of choice. As a modern and intelligent agent, Cribl Edge offers a rich and interactive UI, real-time change validation, configuration version control, built-in fleet management, automatic discovery of host data, and much more!

Endless endpoints also bring unprecedented volumes of data at the edge. While plenty of this data is incredibly valuable, there are also ample amounts of data dispersed within it that do not need to be collected or analyzed. Many vendors will still charge you for bringing in this data you don’t ultimately want to keep, which may force you to make compromises on whether you should monitor certain endpoints altogether. But here at Cribl, we believe in empowering you with choice, control, and flexibility over your data, so we have introduced a new feature where data dropped in Cribl Edge does not count against the ingested volume (available in release 4.6). So let’s walk you through a few scenarios on using this new feature to make the most of your Edge ingest and get even more out of your Cribl license!

How to Drop Data at Edge

Data (events) at Edge Nodes can be dropped in Pipelines. A Pipeline is a list of functions that work on data. Edge supports 3 different Pipelines:

• Pre-processing Pipeline: These Pipelines are attached to the Source and are used to process or drop events before they are delivered to the Processing Pipeline
• Processing Pipeline: These Pipelines are normal event Processing Pipelines and are attached to QuickConnect (or Routes).
• Post-Processing Pipeline: These Pipelines are attached to the Destination and are used to process / drop events before they are sent out to the Destination.

Pipelines can be part of a Pack too. Any original events dropped in any of the pipelines above is summed up and subtracted from the total ingest to arrive at Net Ingest for that Edge Node, but keep in mind this only does not apply to cloned events (as in when you are sending to multiple destinations) and only to the original data. Any data sent to DevNull Destination is also subtracted from total ingest. Also, as always, any data sent from Edge to Cribl Stream via Cribl HTTP or Cribl TCP Destinations is only counted once as Net Ingest from Edge so you won’t be billed for it again.

Let’s walk through a few examples.

Example 1: Sending to DevNull Destination

When testing Source configurations, Pipelines, or when working in Staging environments, you don’t have to worry about ingest costs anymore. You can just route all the data to the DevNull Destination and it won’t be counted as ingest. This is the default out-of-box configuration for any fleet.

If you are ingesting 4 GB from a Source (e.g, File Monitor) and routing it all to the DevNull Destination, then you wouldn’t be charged for any ingest.

Net Ingest is 0 GB

Example 2: Drop events from Windows Event Log

We see many customers interested in collecting events from Windows Security Event Logs. While a lot of information from Security Event Logs is useful, there are some event codes that customers typically would like to drop before routing to a destination (e.g, Event ID 4662, which logs access to Directory Service and can result in a lot of events as read access to Directory Service is very common).

In the example below, I created a Pipeline to drop Event ID 4662 and attached it to the QuickConnect Route (as Processing Pipeline) between Windows Event Logs & Grafana Cloud. You can also attach this pipeline as a Pre-Processing Pipeline to Windows Event Logs. The result would be the same.

Net Ingest is 3.5 GB

Example 3: Drop Events when sending to Multiple Destinations

Often, customers would like to maintain full fidelity raw logs in an archive while sending out a subset to their analytics tool or SIEM. Let’s expand on the above scenario where Windows Event Logs are sent to Grafana Cloud after dropping Event ID 4662. In addition to sending to Grafana Cloud, let’s say I want to archive all the events (including Event ID 4662) to an external storage like S3.

In this scenario, since you are cloning the data before dropping, all ingest is counted against license. However, if you drop data in a Pre-Processing Pipeline (at Source, before cloning), then, any dropped data will be deducted from ingest.

Net Ingest is 4 GB

I hope these examples help you get some ideas on how to take advantage of dropping events in Cribl Edge to get more out of your Cribl license. Maybe you can free up enough ingestion to deploy more Edge Nodes?!

Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.

We offer free training, certifications, and a free tier across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started and continue to build and evolve. We also offer a variety of hands-on Sandboxes for those interested in how companies globally leverage our products for their data challenges.