Cribl Stream 7x more efficient than LogStash and Fluentd

Cribl Stream often gets compared to more general purpose streams processing engines or other open source log shippers. “How is this different from LogStash or Fluentd” is a frequent question from our prospects. In the past few months, we’ve been laying out our vision of an Observability Pipeline, and how one can achieve these benefits, even building your own. We’ve seen people build pipelines on top of log shippers like LogStash or Fluentd, but it is usually a long and expensive journey.

In this post we’ll compare the performance of Crib Stream vs LogStash and Fluentd for one of the simplest and common use cases our customers run into – adjust the timestamp of events received from a syslog server.

TL;DR Results

We’re interested in how efficiently can these systems perform. The chart below shows the CPU normalized throughput, (K events/sec) that each system was able to sustain. In the Parse test case Cribl Stream outperforms LogStash by a factor of 8.75x, in the parse and forward by about 6.5x and and in full test case by about 6.5x. In the Parse test case Cribl Stream outperforms Fluentd by about 26%, in the parse and forward by a factor of 4.3x and and in full test case by about 6.7x. If you’re using Apache NiFi we highly recommend checking out our earlier post showing a 55x outperformance.

New to Cribl LogStream?    Take our sandbox for a drive

Test Descriptions

Three separate test cases were performed

1. PARSE – data is sent to each system and each system is configured to drop the data
2. PARSE+FORWARD – data is sent to each system, which then forwards it to another destination
3. PARSE+UPDATE+FORWARD – data is sent to each system, the timestamp of events is modified then data is sent off to another destination

Environment & Settings

Testing was performed on AWS c5.2xlarge  (8CPU, 16GB RAM) instance running on AWS Linux (ami-0e8c04af2729ff1bb).

$ uname -a
Linux ip-172-31-14-129.us-west-2.compute.internal 4.14.165-131.185.amzn2.x86_64 #1 SMP Wed Jan 15 14:19:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

$ ./cribl/bin/cribl version
Version: 2.1.0-d58cfc78
Installation type: standalone

# LogStash 7.6.0 requires java 1.8
$ sudo yum install -y java-1.8.0

$ wget https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.6.0.tar.gz
$ tar -xf logstash-oss-7.6.0.tar.gz
$ cd logstash-7.6.0
$ bin/logstash-plugin install logstash-output-syslog

# install fluentd
# curl -L https://toolbelt.treasuredata.com/sh/install-amazon2-td-agent3.sh | sh

# test data
$ zcat syslog.log.gz | wc
15875552 269725643 1944436637

Methodology

Each system was ran with default out of the box configuration settings. Cribl Stream, LogStash and Fluentd were configured to listen on a local port and updated the configurations to perform each of the test cases. When sending data out, each system was configured to send data to another localhost listener that simply drops the data. Two concurrent senders were used to send the test dataset. We measure and report the CPU consumption of all target system processes before and after sending of the data completes. We then use the CPU time to compute the thruput instead of wall time as we’re interested in resource consumption normalized throughput.

normalized_thruput = sent_events / cpu_time

We’ve used the following scripts to measure the CPU consumption. Note, these measurements underestimate the CPU consumption by a small amount as the sender can complete before the destination system has fully processed the last buffered chunks of data.

top -bn1 | grep cribl
zcat syslog.log.gz | nc localhost 6666 &
zcat syslog.log.gz | nc localhost 6666 &
wait 
top -bn1 | grep cribl

Stream Pipeline Settings

asyncFuncTimeout: 1000
functions:
  - id: eval
    filter: "true"
    conf:
      add:
        - name: _time
          value: _time + 3600

LogStash Settings

input { 
  syslog { 
   port => 6666 
  } 
} 

filter {
  ruby {
    code => 'event.set("@timestamp", LogStash::Timestamp.new(Time.at(event.get("@timestamp").to_i + 3600)))'
  }
}

output { 
  syslog { 
   facility => "%{facility_label}"
   severity => "%{severity_label}"
   appname => "%{program}"
   sourcehost => "%{logsource}"
   procid => "%{pid}"
   codec => plain {
     format => "%{message}"
   }  

   rfc => rfc3164
   port => 5555 
   host => localhost 
   protocol => tcp
  } 
}

Fluentd Settings

<system>
  workers 2
</system>

<source>
  @type syslog
  port 6666
  bind 0.0.0.0
  tag system
  <transport tcp>
  </transport>
  <parse>
    message_format rfc3164
  </parse>
</source>

#update time 
<filter **>
  @type record_transformer
  enable_ruby
  renew_time_key new_time
  remove_keys new_time
 <record>
    new_time ${Time.at(time).to_f + 3600}
 </record>
</filter>

#send data out using papertrail plugin
<match **>
  @type papertrail
  papertrail_host 127.0.0.1
  papertrail_port 5555
</match>

Wrapping Up

This performance comparison only looks at one simple use case, pointing out that there’s a potential 85% savings in log shipping infrastructure costs. In the coming weeks we’ll post performance comparison results against a few other tools we’ve come across in the market. If we find we are slower in basic use cases, we will happily highlight it. We are by definition experts on our system and not others, if we’ve missed a “GoFaster = true” setting or if you have any other questions or suggestions please reach out!

One more thing, we’re hiring! If the problems above excite you, drop us a line at hello@cribl.io or better yet talk to us live by joining our Cribl Community.

The fastest way to get started with Cribl LogStream is to sign-up at Cribl.Cloud. You can process up to 1 TB of throughput per day at no cost. Sign-up and start using LogStream within a few minutes.