Event Hubs is Microsoft’s cloud-native real-time event streaming service. For Event Hubs to work, data must be pushed to or pulled from it. That is where Cribl Stream comes in. Event Hubs is a source and destination inside Cribl Stream and the control for how you route, shape, and transform your data from Event Hubs. But, one does not simply Stream into (or from) Event Hubs. There is a lot that goes into architecting an Event Hubs Source. The first step is to review how much data the producer (log source) will generate and size your event hubs accordingly. Microsoft has excellent sizing documentation for Event Hubs here.
Sizing Event Hubs in Cribl Stream
Using the standard subscription, each Through Put Unit (TU) can accommodate an ingress of 1MB/s or 1000 eps (whichever comes first) and an egress of 2MB/s or 4096eps (whichever comes first). A Premium Event Hubs namespace will be required to handle heavier log volumes. A premium namespace offers much higher thruput. A premium Event Hubs namespace with 1 PU and 1 event hub (100 partitions) can ~5-10 MB/s ingress and ~10-20 MB/s. It is also essential to consider the number of partitions you will need to transfer the logs to Cribl Stream efficiently. With a standard subscription, there is a limit of 32 partitions per Event Hub. With a Premium subscription, there is a limit of 100 partitions per Event Hub and 200 partitions at the namespace level.
Further, Mircosoft limits ingress to 1 MB/s per partition. For example, if you have a log source that generates 1.29TB per day, you will need a Premium Event Hubs namespace with at least 2 PUs to meet the ingress requirements or a Standard Event Hubs namespace with 15 Throughput Units. In both cases, the Event Hub will need to have 15 partitions.
Third, on the Cribl Stream side, one worker process to one partition is not necessary, but sizing for data volume and having at least 3 worker nodes available is essential for critical workloads to provide resiliency for ingesting your Event Hubs data. Standard calculations apply, 400GB throughput (in+out) per CPU core @3.0 GHz with 2GB RAM. Please review our docs here for more sizing information on sizing, including details on Graviton2/ARM64. As a guideline, we have seen better performance for standard-level namespaces if the worker has a 3:1 (partitions to worker process ratio) as your largest Event Hub. For example, if you have 3 Event Hubs and the largest Event Hub has 32 partitions, you will want at least 12 worker processes (32 Intel/AMD CPU). However, for premium-level namespaces, worker processes can quickly become the bottleneck if you fully saturate a 100-partition Event Hub. Due to this, we recommend a nearly 1:1 ratio if you are saturating a 100-partition Event Hub.
In terms of Architecture, if you have a mix of Event Hubs and other sources, consider a separate worker group to, at a minimum, ingest the Event Hubs data.
Setup
Configure the Azure Event Hubs Source in Cribl Stream
Login to Azure and navigate to your Event Hubs Namespace
Write down the Hostname.
Finding the Hostname and Shared access policies
Click Shared Access Policies to open the page where you can select policies for your Event Hubs Namespace, and then click RootManageSharedAccessKey to show details for that policy.
Viewing Shared access policies
Copy and securely store the Connection String-primary key.
Click on Event Hubs on the left-hand column and copy the Event Hubs name you wish to ingest.
Event Hubs Name
From a Cribl Stream instance’s or Group’s Manage submenu, select Data > Sources, then select Azure Event Hubs from the Manage Sources page’s tiles or left nav. Click New Source to open the Azure Event Hubs > New Source modal.
In the General Settings tab, enter the following values:
InputId: Stream
Brokers: Enter the Host Name you wrote down earlier, append port 9093, and enter the result. For example:
CriblTest.servicebus.windows.net:9093.Event Hub Name: The name of your Azure Event Hub, for example:
defender.Group ID: This value can be Cribl. However, the groupID used in Cribl cannot be shared with other technologies. Failing to follow these guidelines may cause unexpected behavior, such as failing to collect logs reliably.
The General Settings tab
In the Authentication tab, enter or select the following values:
SASL mechanism: PLAIN (the only supported option).
Username: $ConnectionString (the default generated by Azure).
Authentication Method: Select Manual to use the Connection String Key generated by Azure Event Hubs.
Password: Enter the Connection String-primary key that you copied earlier.
The Authentication tab
Verify that Data is Flowing from your Azure Event Hub to Cribl Stream
Before you can verify that data is reaching Cribl Stream, you must ensure that it is flowing out of your Azure Event Hub in the first place.
One option is configuring a Datagen and a Route to send data to the Event Hub destination. We’ll assume you have done that or gotten data flowing from your Azure Event Hub in another way.
In Cribl Stream, open the Sources > Azure Event Hubs > Cribl Stream page. This should show your Source, with a message confirming it is working properly.
A Working Source
Open the Live Data tab. You should see the data that is flowing from your Azure Event Hub to Cribl Stream.
Viewing Live Data
Common Errors
If the Event Hubs source will not connect after initial configuration, check to make sure :9093 is appended to the broker’s hostname in the Event Hub Source configuration tab, and your Connection String-primary key is entered correctly on the authentication page.
Brokers Hostname and Port
If Event Hubs is still having trouble connecting, check your Event Hubs firewall rules. You may need to allow the Cribl workers to connect to the Event Hub Namespace. More information can be found in Microsoft’s documentation here.
Rebalance Errors
‘Rebalancing’ log messages are expected behavior when deploying a new configuration bundle to a worker group. However, if you notice a high number of ‘rebalancing’ log messages, you may need to tune some additional timeout settings.
On the Event Hub source, go to the advanced settings and increase the Session and Rebalance Timeout. Doubling the default values is a good place to start. Increasing max retries to 10 can also help.
Session timeout, rebalance timeout, and max retries settings
Collection lag or low thruput
If you notice collection lag, or lower than expected thruput for your Event Hubs subscription, increasing the Max bytes per partition and Max bytes settings will help alleviate the issue.
This server does not host this topic-partition
If you are receiving “This server does not host this topic-partition” Errors check to ensure the Event Hub Name entered in your Source or Destination exist in your Event Hub Namespace.
Destination Side – Server Busy Error
If you are receiving Server Busy Errors on an Event Hubs destination, this is an indication that more Through Put or Compute Units are required to ingest the volume of data that Cribl is sending if the Server Busy Errors are preventing sending the observability data in a timely manner.
Wrap up
I hope this troubleshooting guide has been helpful! If you’re new to Cribl’s products, I invite you to sign up for a free Cribl.Cloud account for up to 1TB/day for free! You can also check out our cloud-hosted Sandbox to try Cribl Stream with sample data.
