Syslog server best practices: setup, tuning, and scaling with Cribl

How do you set up Cribl Stream as a syslog server?

Setup is fast: one set of configs gets the port listening. Give the Cribl Stream syslog Source a name, a port, and your protocol choice (TCP, UDP, or both). By default, Cribl parses the basic syslog structure, and from there you can route, transform, filter, reshape, aggregate, and re-deliver the data any way you please.

TIP: Don't change the 0.0.0.0 entry unless you know precisely what you're doing. All Worker Nodes in the group receive this config, and 0.0.0.0 tells the system to listen on all interfaces and addresses. A 127.0.0.1 entry would mean listening on localhost only. No two Nodes should share IPs outside of 0.0.0.0 and 127.0.0.1.

TIP: Some sources send using RFC 6587, which defines the syslog message length at the front of the payload for absolute accuracy in event breaking. Good news: as of Cribl Stream 4.7.0, RFC 6587 messages are auto-recognized. No manual toggle needed.

TIP: Establish Cribl Stream instances as close as possible to your syslog producers. With UDP this is vital, since there's no guarantee of delivery. Even with TCP, long communication channels create more places to bottleneck, so get logs into Stream ASAP where they're safe. Most syslog senders have no queuing capability, so if the destination doesn't respond quickly, events could be lost. Persistent queuing (covered below) remedies this.

TIP: There are intentionally no Event Breaker configs in Cribl syslog Sources, because the protocol defines what an event is. If a "syslog" source breaks events improperly, consider a Raw TCP or Raw UDP Source with a custom event breaker of your own making.

Why enable TLS for syslog?

Encrypt everything you can. We support Transport Layer Security (TLS) across deployments and recommend enabling it when possible. Setup is straightforward: upload your certs into a Certificates config object under Group Settings, Security, Certificates. Then in your TCP-enabled syslog Source, go to TLS, flip the enable switch, and select your cert. Save, commit, deploy. Done. (Bonus pro tip: set a reminder to renew those certificates.)

TIP: Cloud Worker Nodes come with TLS ready to roll, but we don't normally recommend delivering syslog data over the Internet. It's available if absolutely needed via the Data Sources page on the Cloud landing portal.

TIP: By default, Cloud Worker Groups have TLS syslog on port 6514, and you have 11 open, unused ports from 20000 to 20010. Any active TLS source can be disabled and repurposed. For self-hosted or hybrid Workers, use any open port you want, though ports under 1024 need special permissions.

TIP: Palo Alto's Panorama service requires TLS to be active on the receiving end, and not in a standard manner. We cover the details in our docs.

How do you load balance and scale syslog?

Most syslog-producing apps and appliances have no concept of load balancing. You give them a hostname and port, and they blast data at it. Spreading those events across multiple hosts is up to you, so you'll need a load balancing service or appliance in front of your Cribl Worker Nodes, like HAProxy, nginx, F5, Cisco, or AWS ELB.

One problem with TCP load balancing is "pinning" to a connection. Most load balancers aren't syslog aware, they don't know where one event ends and another begins, so they hold a connection and keep sending data into it. Result: your data isn't distributed across your resources.

TIP: As of Cribl Stream 4.6.0, you can enable native syslog load balancing. This dedicates one process on each Worker Node to receive TCP syslog, and fans events out across all Worker Processes on the Node. With a big enough instance, this solution scales to roughly 100 MB/s of ingested syslog data per Node, equivalent to about 8.6 TB/day, according to Cribl's own performance testing.

TIP: If you enable this feature, account for the LB process. Say you have a 16-core system running the default -2 setting: 14 processes handle processing, 2 are reserved for system overhead. Without the internal load balancer, the external load balancer risks pinning to a single process, limiting throughput to one thread. With it active, an extra process is dedicated to receiving syslog traffic and spraying events evenly across all available Worker Processes. Your throughput limit now equals 14 cores, but you've taken one of the 2 reserved cores. Change the reserve to -3 and add cores if 13 processing cores aren't enough.

TIP: Scaling recommendations:

• For HA, don't drop below 2 (preferably 3) Nodes in a Group, and size them to handle running at least 1 instance short.

• At or below 8 cores per instance, scale up (add cores) before scaling out (add instances).

• If syslog is the primary volume source in the Worker Group, consider a dedicated Worker Group just for syslog.

TIP: Load balancers introduce a side effect, all connections appear to come from the load balancer's IP instead of the original sender. Depending on your load balancer type, enabling proxy protocol support provides the original IP to Cribl Stream.

System Tuning for UDP Syslog

This section applies only to UDP-based syslog, not TCP, and is summarized from our docs. It doesn't apply to Cribl-managed Cloud Workers, where Cribl's engineers handle it for you.

Incoming UDP traffic sits in a Linux kernel buffer, and Cribl drains it as resources allow. At lower throughput this isn't an issue, but as you scale syslog, the default buffer size is almost always too small.

TIP: Check the current buffer size with sysctl net.core.rmem_max. The default is often a little over 200 KB, easily overwhelmed by even a moderately busy syslog source.

TIP: Check (and monitor) packet receive errors with netstat -su | grep 'receive errors'. If receive errors exceed zero, you've almost certainly lost events, and a rapidly increasing count is a serious problem.

TIP: Bump the key settings on a live system, for example to 25 MB, with sysctl -w net.core.rmem_max=26214400 and sysctl -w net.core.rmem_default=26214400. To make the change permanent, add both values to /etc/sysctl.conf.

When should you use different ports, labels, or dedicated Worker Groups?

Syslog events are painfully simple, and the message doesn't always identify what kind of data you have. Usually you can set up pattern or host matching to identify and route event types, either in the Fields config area of the Source or in a pre-processing Pipeline. Alternatively, use different ports per data type: Cisco ASA on 9514, NXLog on 9515, and so on. Even then, lean toward using the Fields config to label the syslog data type.

TIP: For lighter loads (under roughly 200 GB/day), use a single port for all types, then qualify and label data via the Fields config or a pre-processing Pipeline using host lookups or patterns. For heavier loads or special cases, set up a dedicated port but still use the Fields config to set the same label. This keeps your routing table sane.

TIP: For extra-large sources, consider an entirely isolated Worker Group. If 90% of your traffic is syslog, or one particular syslog source dominates, give it its own Worker Group dedicated to optimizing syslog delivery.

How do you dead letter unwanted syslog data?

Syslog is prone to receiving unwanted logs. Validate the host sending the logs, or even the content itself. If validation fails, send the data to object store for short retention only, and periodically check your dead letter dataset for valuable logs that fell through unintentionally.

For example, if you expect only ASA data on a syslog Source, label events containing %ASA as 'asa'.

Now I can use __syslogType in my routing rules, presenting a uniform set of rules:

Finally, data that does not match would fall to the default rule at the bottom, either dropping the data or sending it to a “dead letter” storage pool in your long-term retention (object store, data lake, etc):

This validation tip extends beyond syslog, always actively validate which logs end up in your analysis tier. You can also use Cribl's Schema Validation functionality.

How should you handle syslog timestamps and timezones?

Many syslog senders still don't include a timezone in their timestamps. RFC 5424 requires timezone identification, but earlier syslog specs did not. A timestamp without an explicit offset risks misinterpretation, landing events in the future or the past. Fortunately, Cribl gives you the tools to fix this.

1.21 jiggawatts can’t help your future-stamped events, but Cribl can!

TIP: Normalize your time now. Option A: use the Cribl Syslog Preprocessor Pack, or borrow the idea from it: lookup files align particular hosts to the correct timezone, and Cribl's Pipeline Functions adjust event time accordingly. Option B: use UTC for all servers and appliances. Even if you choose Option A, pursue UTC long term. Let the analysis and reporting tiers convert from UTC to local zones when they need to.

Why do you need persistent queues for syslog?

Being ready for unexpected downtime is crucial with syslog. Most syslog senders, and all UDP syslog senders, do not buffer data if the receiving side is down or blocking. There are a few scenarios where a syslog input risks losing events:

• The destination on the other side of Cribl is down or rejecting data.

• The destination is so slow that Cribl can't move events to it fast enough.

• Cribl itself is overwhelmed processing events and can't receive new data.

TIP: Cribl Stream's persistent queuing (PQ) handles these situations. Our recommendation is always-on source-side PQ for most environments. For Cloud-based Workers it's a simple enable flag. For hybrid and on-prem Workers, see the PQ planning guide for configs and sizing.

TIP: If you enable source PQ on a UDP syslog input, disable the event buffer under Advanced Settings by setting it to 0. You don't want both source PQ and event buffering active.

TIP: Use dedicated, fast storage for your PQ. Dedicated, so a full queue doesn't impact other parts of your system. Fast, so writing to and reading from storage never holds up processing.

How Cribl can help with syslog management

Syslog has its quirks, and scaling it successfully can be a journey. Cribl Stream puts management of all your logging sources in one place, including syslog, with a performant, flexible, and complete syslog service built in. Instead of hand-editing rsyslog configs across a fleet of servers, you get one interface to receive, validate, label, transform, and route syslog data to any destination you choose, with no lock-in and no data loss.

That vendor-agnostic approach matters because syslog feeds nearly every SIEM and analytics tool in your stack. Cribl gives IT and security teams the choice, control, and flexibility to decide what to collect, how to process it, and where to send it. Route high-value security events to your SIEM, send full-fidelity copies to low-cost object storage, and dead letter the noise, all from the same pipeline. Native TCP load balancing, persistent queues, and TLS support keep the whole flow resilient at enterprise scale.

Cribl is used by many organizations, including half of the Fortune 100. It helps teams reduce telemetry volume, cut costs, and accelerate SIEM migrations without adding agents or disrupting existing systems. Your syslog senders keep doing what they do, and Cribl organizes the data.

Ready to get hands-on? Visit Cribl Sandboxes and Cribl University to explore more ways Cribl can help you get your logging estate under control.

Log happy and s/chaos/order/ig

Syslog Server Best Practice FAQs