August 24, 2021
Cribl Stream has supported Syslog as a source, almost since its inception. Now, with the introduction of the Packs feature in the July release of Stream, you have the opportunity to handle inbound Syslog data even better than before. This article explores use of the Cribl Pack for Syslog Input to remove redundant data, handle timezone normalization, and set sourcetypes and other metadata based on lookups.
Syslog was developed in the earliest days of the Internet, as a means by which any OS, process, or device could deliver a log event to a syslog server using a well-defined transport protocol, with a well defined message format. (Syslog is one of the earliest examples of a remote API call!)
The importance of Syslog is in its ubiquity. Linux and Unix OSes support the Syslog protocol, as does nearly every enterprise-grade piece of networking equipment. While there are other options for shipping logs from applications, Syslog remains the de-facto protocol for networking gear including firewalls, switches, routers, and more. For many devices, Syslog is the only supported way to send logs.
To quote from Wikipedia:
Syslog was developed in the 1980s by Eric Allman as part of the Sendmail project. It was readily adopted by other applications and has since become the standard logging solution on Unix-like systems. A variety of implementations also exist on other operating systems and it is commonly found in network devices, such as routers.”
The Syslog protocol has continued to evolve over the years, adding support for TCP, TLS security, timezone specification, and larger message payloads.
Risk of Data Loss:
Syslog was initially based on UDP, not TCP. The UDP choice made sense at the time; it reduced overhead back when servers’ memory was measured in kilobytes, not gigabytes. UDP allowed Syslog senders to keep working just fine, even if the destination was down. Even today, some network devices only support sending Syslog on UDP. UDP delivery is potentially lossy – your data will probably get there, but it might not. And if you’re sending across the Internet, the chance of data loss goes up.
Lack of encryption:
Syslog over UDP doesn’t support encryption on the wire. TCP does allow Syslog to send using TLS, but among Syslog-sending devices this is a pretty rare feature. Sending unencrypted messages on a LAN might be fine for some situations, but it’s certainly not ideal.
While modern Syslog senders support RFC 5424 to accurate timezone, year, and day representation, most devices sending Syslog have a timestamp that looks like this:
May 12 21:20:39
Which timezone is that? What year? Is the sending system in UTC? Eastern daylight time? It’s impossible to tell.
Meta information & management:
When sending syslog data to Splunk, Elasticsearch, and other systems of analysis, it’s often important to set a sourcetype, index, or other fields that instruct that system on how to process and store those events for later retrieval.
Since this meta-information is not present in the original payload, it has to be added after the fact. This is typically done using a combination of configuration files for rsyslog or syslog-ng, in conjunction with Splunk forwarders or elastic beats agents. Management is tricky, as IP addresses and hostnames live within the syslog server’s config files, and these have to match up with the forwarding agent’s index and sourcetype settings.
When syslog data arrives in a destination such as Splunk or Elasticsearch, the human-readable timestamp – and the hostname – are superfluous. They’re already being delivered in the extracted timestamp field (as epoch time), and the host field. Having this data in the message field adds 25-30 bytes per event. This might sound small but in some cases this might represent more than 20% of the overall data volume.
Syslog data sent with RFC 5424’s TimeQuality feature adds up to an additional 63 bytes per event. This is all data that is rarely searched, and the extracted timestamp is already in _time, epoch time format, with millisecond precision if available. These 63 bytes of data are just an added storage + indexing burden on the destination system.
Reducing Management Complexity:
Stream is a native Syslog receiver, just like rsyslog and syslog-NG. You may be able to completely retire or repurpose your existing Syslog servers, replacing that functionality with Stream workers instead, and those same Stream workers also support receiving on other protocols as well. All of the issues mentioned above can be addressed within Stream, without hand-editing config files.
Minimizing possibility of data loss:
To help with UDP data loss, deploy Stream workers as close to the sender as possible. Optimally, this would be a pair of properly sized workers, behind a load balancer, on the same subnet or LAN. Configure backpressure for Stream destinations to use “Persistent Queueing” function, to ensure data is delivered when an off-line destination comes back.
Securing the connection:
Once the Stream worker receives the data via syslog (encrypted or not), delivery to any subsequent connection can be configured over TCP with TLS encryption. As with the UDP scenario, one would want to locate a Stream worker on the same subnet or LAN, to minimize the risk of sending unencrypted data.
Cribl Pack for Syslog Input:
Addressing the timezone, data cleanup, and enrichment uses are all handled by the Cribl Pack for Syslog Input. This pack is attached to your Syslog sources as a PreProcessing pipeline, allowing the benefits below to be applied to most or all inbound Syslog data. In some cases, this Pack’s volume reduction can remove over 100 bytes per event just by dropping redundant or unnecessary values.
_raw, saving about 16-25 bytes.
_raw, saving up to 63 bytes.
The pack includes data samples, to allow testing of the various options prior to putting it into production. Full documentation for the Pack is included within the pack itself.
By deploying Stream workers as replacements for existing Syslog servers (rsyslog, syslog-NG, etc), you can reduce management complexity while ensuring the reliable, secure delivery of Syslog data to your chosen systems of analysis and systems of retention.
By using the Cribl Pack for Syslog, you can easily enrich your inbound Syslog data with metadata while simultaneously removing redundant data for a savings of 20% or more. And finally see an end to timestamp and timezone extraction issues.
For more info, check out our docs on syslog data reduction or grab the Cribl Pack for Syslog Input. Join the #packs channel in our Slack Community to get the latest updates on this and other packs for Cribl Stream.
The fastest way to get started with Cribl Stream is to sign-up at Cribl.Cloud. You can process up to 1 TB of throughput per day at no cost. Sign-up and start using Stream within a few minutes.