Cribl Search is a powerful tool that allows users to search and analyze data at rest, quickly and efficiently. But what if you need to send your search results to a different system for further analysis, audit, or compliance purposes?
For instance, consider the following use cases:
Incident Analysis: A full copy of event log data from various sources is stored in AWS S3, but you need to send specific data to a SIEM system for further analysis.
Compliance audit: You need to find and report which devices are accurately sending logs, and identify any devices that may be missing.
Handling Massive Cardinality: When dealing with high cardinality data, you can use Cribl Search to first decide what data is useful, and then move it to an analytics tool.
AWS Security Logs Processing and Forwarding: You store AWS Logs in S3, and need to send a focused and concentrated version of it to another tool for further analysis.
AWS S3 to S3 Hairpin: You need to find specific data in one S3 bucket, and then transfer the distilled search results back into a file (or set of files) in the S3 bucket.
That’s where send operator comes in.
Introduction
In the recent v4.1 release, the addition of the send operator to Cribl Search has unlocked new possibilities for data routing and analysis. In this blog post, we will discuss what the send operator does, and how it works, provide examples, and share some best practices to help you fine-tune your search results before sending them out.
What It Does
The new send operator in Cribl Search allows users to find with surgical precision exactly what they need and leverage native integration with Cribl Stream to easily send the search results to another system without having to configure anything.
How It Works
The syntax for send operator is very straightforward. Once you completed your search, pipe to the send operator and specify various parameters for it:
send [ tee=Boolean ] ( [ group=WorkerGroup ] | [ URL ] )
The simplest way to use this operator is to add | send to your search.
This data will automatically be routed to the Stream Cribl HTTP Source, which is a Source in Stream.
That Source can then be connected to any Destination via Routes or QuickConnect (see Stream docs for more details on how to accomplish this.)
A few considerations
Today, you can send results only to Stream groups managed by the same Cribl.Cloud Organization.
tee defaults to false to view progress stats. To see events on the UI, use tee=true
group and URL are exclusive to each other.
The group parameter is used as a great shortcut to address Cribl.Cloud Stream Group(s) by name.
The URL is used to address the Cribl HTTP Source when sending to non-Cribl.Cloud-hosted group(s).
Examples
Now, let’s review a few simple examples and use cases:
1. Send status events to a default Worker Group:
dataset=myDataset | limit 100 | send
2. Send limited events, including results:
dataset=myDataset | limit 100 | send tee=true
3. Send events to a Worker Group named “myGroup”:
dataset=myDataset | send group=myGroup
4. Send events to a hybrid (non-Cribl.Cloud-hosted) Worker Group:
dataset=myDataset | send "https://in.your-tenant.com:10200"
Better Practices
Before sending out your search results, it’s important to ensure the accuracy and relevance of the data. The count and limit operators in Cribl Search can help you achieve this by fine-tuning your search results.
The ‘count’ Operator
count: This operator returns the total number of events in a dataset. Use count to verify the volume of data before sending it out.
Example:
dataset=myS3VPCFlows earliest=1hr | count
The ‘limit’ Operator
limit: The limit operator returns only the specified number of events from the dataset. Use this to review a sample of your search results and ensure they match your expectations before sending the data.
Example:
dataset=myS3VPCFlows earliest=1hr | limit 10
Once you’ve verified the accuracy and relevance of your search results using the count and limit operators as guardrails, you can proceed to send the data using the send operator, as described in the examples section.
Conclusion
With the new send operator in Cribl Search, users can now effortlessly send search results to different systems for advanced analysis, audit, and compliance. By following the best practices outlined in this post, and adhering to the rules governing the send operator, you can ensure the accuracy and relevance of your search results before sending them out.
The fastest way to get started with Cribl Stream, Edge, and Search is to try the Free Cloud Sandboxes.
