Cribl Stream often gets compared to more general purpose streams processing engines or other open source log shippers. “How is this different from LogStash or Fluentd” is a frequent question from our prospects. In the past few months, we’ve been laying out our vision of an Observability Pipeline, and how one can achieve these benefits, even building your own. We’ve seen people build pipelines on top of log shippers like LogStash or Fluentd, but it is usually a long and expensive journey.
In this post we’ll compare the performance of Crib Stream vs LogStash and Fluentd for one of the simplest and common use cases our customers run into – adjust the timestamp of events received from a syslog server.
TL;DR Results
We’re interested in how efficiently can these systems perform. The chart below shows the CPU normalized throughput, (K events/sec) that each system was able to sustain. In the Parse test case Cribl Stream outperforms LogStash by a factor of 8.75x, in the parse and forward by about 6.5x and and in full test case by about 6.5x. In the Parse test case Cribl Stream outperforms Fluentd by about 26%, in the parse and forward by a factor of 4.3x and and in full test case by about 6.7x. If you’re using Apache NiFi we highly recommend checking out our earlier post showing a 55x outperformance.
New to Cribl LogStream? Take our sandbox for a drive
Test Descriptions
Three separate test cases were performed
PARSE – data is sent to each system and each system is configured to drop the data
PARSE+FORWARD – data is sent to each system, which then forwards it to another destination
PARSE+UPDATE+FORWARD – data is sent to each system, the timestamp of events is modified then data is sent off to another destination
Environment & Settings
Testing was performed on AWS c5.2xlarge (8CPU, 16GB RAM) instance running on AWS Linux (ami-0e8c04af2729ff1bb).
$ uname -a
Linux ip-172-31-14-129.us-west-2.compute.internal 4.14.165-131.185.amzn2.x86_64 #1 SMP Wed Jan 15 14:19:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux$ ./cribl/bin/cribl version
Version: 2.1.0-d58cfc78
Installation type: standalone
# LogStash 7.6.0 requires java 1.8
$ sudo yum install -y java-1.8.0
$ wget https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.6.0.tar.gz
$ tar -xf logstash-oss-7.6.0.tar.gz
$ cd logstash-7.6.0
$ bin/logstash-plugin install logstash-output-syslog
# install fluentd
# curl -L https://toolbelt.treasuredata.com/sh/install-amazon2-td-agent3.sh | sh
# test data
$ zcat syslog.log.gz | wc
15875552 269725643 1944436637Methodology
Each system was ran with default out of the box configuration settings. Cribl Stream, LogStash and Fluentd were configured to listen on a local port and updated the configurations to perform each of the test cases. When sending data out, each system was configured to send data to another localhost listener that simply drops the data. Two concurrent senders were used to send the test dataset. We measure and report the CPU consumption of all target system processes before and after sending of the data completes. We then use the CPU time to compute the thruput instead of wall time as we’re interested in resource consumption normalized throughput.
normalized_thruput = sent_events / cpu_timeWe’ve used the following scripts to measure the CPU consumption. Note, these measurements underestimate the CPU consumption by a small amount as the sender can complete before the destination system has fully processed the last buffered chunks of data.
top -bn1 | grep cribl
zcat syslog.log.gz | nc localhost 6666 &
zcat syslog.log.gz | nc localhost 6666 &
wait
top -bn1 | grep criblStream Pipeline Settings
asyncFuncTimeout: 1000
functions:
- id: eval
filter: "true"
conf:
add:
- name: _time
value: _time + 3600LogStash Settings
input {
syslog {
port => 6666
}
}
filter {
ruby {
code => 'event.set("@timestamp", LogStash::Timestamp.new(Time.at(event.get("@timestamp").to_i + 3600)))'
}
}
output {
syslog {
facility => "%{facility_label}"
severity => "%{severity_label}"
appname => "%{program}"
sourcehost => "%{logsource}"
procid => "%{pid}"
codec => plain {
format => "%{message}"
}
rfc => rfc3164
port => 5555
host => localhost
protocol => tcp
}
}Fluentd Settings
<system>
workers 2
</system>
<source>
@type syslog
port 6666
bind 0.0.0.0
tag system
<transport tcp>
</transport>
<parse>
message_format rfc3164
</parse>
</source>
#update time
<filter **>
@type record_transformer
enable_ruby
renew_time_key new_time
remove_keys new_time
<record>
new_time ${Time.at(time).to_f + 3600}
</record>
</filter>
#send data out using papertrail plugin
<match **>
@type papertrail
papertrail_host 127.0.0.1
papertrail_port 5555
</match>Wrapping Up
This performance comparison only looks at one simple use case, pointing out that there’s a potential 85% savings in log shipping infrastructure costs. In the coming weeks we’ll post performance comparison results against a few other tools we’ve come across in the market. If we find we are slower in basic use cases, we will happily highlight it. We are by definition experts on our system and not others, if we’ve missed a “GoFaster = true” setting or if you have any other questions or suggestions please reach out!
One more thing, we’re hiring! If the problems above excite you, drop us a line at hello@cribl.io or better yet talk to us live by joining our Cribl Community.
The fastest way to get started with Cribl LogStream is to sign-up at Cribl.Cloud. You can process up to 1 TB of throughput per day at no cost. Sign-up and start using LogStream within a few minutes.
