We know the old adage: All data is security-relevant. But at what cost? Many organizations are still trying to get their arms around existing data flows and tooling to say nothing of new apps and data sources coming into play as we continue to migrate to the cloud. Working to get a complete picture of their security environments, many CISOs are forced to make painful decisions between staying within budget and getting complete security event visibility.
By now, many security professionals are familiar with the ways in which Cribl is helping to maximize customers’ investment in their security analytics/security information and event management (SIEM) and observability tools. Cribl Stream gives QRadar admins choice and control over their data, routinely reducing data volumes by 35% or more.
- Observability tools are traditionally based on daily data-volume ingestion. Cribl Stream can optimize SIEMs by reducing data volumes by 30% or more.
- Even with new workload-based licensing models, Cribl optimizes an Observability tool’s performance, and decreases CPU utilization, providing comparable savings.
- Data reduction can be accomplished by (1) reducing the size of the events themselves, and/or (2) reducing the overall number of events.
QRadar and Splunk – Dominant Security Analytics Platforms
While SIEM tools are a very important piece of a security portfolio, each tool approaches licensing differently. IBM QRadar and Splunk are the two most widely deployed SIEM solutions among major enterprises and government agencies today.
QRadar has the reputation of being a reliable SOC platform for threat detection and incident response built for large enterprises. QRadar has a large deployment base and an extensive set of service providers. However, QRadar has both a radically different architecture and licensing model than Splunk.
While Splunk leverages a well-known volume-based licensing model, QRadar is licensed based upon event count – Events Per Second (EPS). So how, if at all, can Stream optimize a customer’s use of QRadar?
QRadar Data Pipeline Architecture
QRadar’s Event Collector collects events from log sources and normalizes raw log source events to repurpose them into the proprietary log format required by QRadar (LEEF). The Event Collector provides some event optimization for customers by bundling or coalescing identical events before sending them to the QRadar Event Processor. The Event Collector is assigned to an EPS license that matches the Event Processor that it is connected to.
The processing rate for events is determined by your events per second (EPS) license. If you exceed the EPS rate, events are buffered and remain in the Event Collector source queues until the rate drops. However, if you continue to exceed the EPS license rate, and the queue fills up, your system drops events – and QRadar issues a warning about exceeding your licensed EPS rate.
QRadar and EPS Pricing
QRadar is available as hardware virtual appliances and software, or as a cloud-SaaS model called QRadar on Cloud (QRoC). Both are licensed based on a customer’s event velocity (the EPS of data sources in scope). The cost relationship is straightforward and linear – the higher the EPS count, the higher the cost.
What Challenges Does QRadar’s Pricing and Architecture Present?
IBM QRadar’s architecture provides some built-in advantages for event reduction, with the de-duplication functionality within the Event Collection tier. However, QRadar also architects in some constraints.
QRadar still requires data to strictly adhere to its predefined schema prior to ingestion: Log Event Extended Format (LEEF), a customized event format for QRadar. As a result, you cannot reformat events, nor can you reduce the size of an event by removing null-value key-value pairs or headers, etc. The event format and size need to remain unaltered, or else the correlation rules within the QRadar Event Processor will simply break!
Additionally, the event reduction functionality with the QRadar Event Collectors is fairly rudimentary, limiting reduction to data deduplication and aggregation. No fine-grained controls are present for selecting which events to retain or drop. Practitioners have only the option of managing events at the endpoint, such as customizing WinCollect or firewall logging. That approach limits control and adds administrative burden since a typical enterprise could have tens of thousands of endpoints.
And of course, the QRadar Event Collectors can send data only to QRadar and only in LEEF format; the pipeline cannot provide another stream of data in a different format (i.e., JSON or syslog), nor to another system of retention or analysis (i.e., Elastic).
How Can Cribl Stream Help to Optimize QRadar?
Stream is a universal event pipeline that gives you the flexibility to route, shape, restructure, and enrich data from any source to any destination without adding new infrastructure or agents. Stream is the best way to get multiple data formats into your analytics tools, including sending LEEF to QRadar. Use the Stream universal receiver to collect from any observability data source and to schedule batch collection from multiple APIs and data stores.
Stream can provide fine-grained control over what events are sent to the QRadar Event Processor, without breaking the expected LEEF format. With Stream, the QRadar administrator can choose to keep or drop events based upon limitless criteria. Stream gives administrators the control to decide which events are highly security-relevant and which events merely consume license and compute. For example, Stream empowers a user to reduce event count by dropping, sampling, and/or suppressing events.
- Dropping events: Based on specific event codes that are low-security-value (e.g., certain Windows, Cisco ASA, or Palo Alto events).
- Sampling events – 1 out of N of this type of data is delivered, while the rest are discarded.
- Dynamic sampling – low-volume data of this type is delivered at 100%, but as volume increases, the % of dropped data increases, and less is discarded.
- Suppression – No more than N copies of this type of data are delivered per unit of time.
- Aggregation – Aggregations perform calculations on the stream of data, and emit outputs of those calculations every window of X seconds. Aggregations are frequently used with time series databases, to convert log data to metrics for faster queries and more-affordable storage.
With each of these methods, the administrator specifies the matching criteria to be used – meta-information such as hostname, source, sourcetype, or log level – in combination with content extracted from the events themselves. LogStream gives QRadar admins the ability to be independent, by controlling what events are being sent to the QRadar event processor. How great is that?!
One global real estate firm recently deployed LogStream to feed QRadar, and achieved a >50% reduction of events (EPS), without losing any security-relevant events (as defined by their security team).
Get Started Using LogStream
Do you want to learn how you can reduce EPS for QRadar and gain control over your data? It’s easy to get started on using LogStream to route and reduce your events, by signing up for Cribl.Cloud and instantly provisioning an instance of Stream Cloud. Once connected to data sources, you can process up to 1TB per day for free!
If you have questions, sign up for our Community Slack. If you just want to test the waters of Stream, head on over to the Stream Sandbox to try a full version in the cloud with sample data.
