LogStream is now available as a Cloud service! Learn More

Living the Stream

The Cribl blog covers Observability, Big Data Analytics, Data Streams Processing... and anything else we feel like writing about!

Filter by: Reset

L Tang Sep 2, 2020

Backpressure Behavior in Cribl LogStream

In this quick dive into backpressure topics in Cribl LogStream, we will also touch upon persistent queueing, how Cribl LogStream sends information out to destinations when backpressure options are selected, and how to approach troubleshooting systems with non-responsive destinations.  Sizing is an art and a science, born of expectations from back-of-napkin math, and refined through […]

Steve Litras Jul 14, 2020

Working with Data in LogStream 2.2

In case you couldn’t tell, we’re really excited about the 2.2 release of LogStream. In previous posts, I’ve covered the Data Collection Feature, and the improvements in manageability that come with 2.2. In this post, I’m going to cover the improvements we’ve made when it comes to working with data in the product. Preprocessing Data […]

Nick Romito Jun 24, 2020

Demystifying Collection Job Scheduling

As the tech lead behind Data Collection, the leading feature of LogStream 2.2, I can say we faced many challenges developing a scalable, accurate solution to batch collection of data at rest.  Among those issues was a resource load balancing issue that cropped up late in our development of the feature.  We’ll discuss this issue […]

Dritan Bitincka Mar 16, 2020

A Simple Guide to Scalable Data Collection from Amazon S3

Scalable data collection from Amazon S3 was introduced back in Cribl LogStream 2.0 and has been a real workhorse providing essential capabilities to many of our AWS customers. In this post we’ll take a look at how it works and how to configure it. If you’re new to Cribl LogStream you may want to take our sandbox for a […]

Dritan Bitincka Mar 12, 2020

Cribl LogStream Global Variables: A Practical Introduction

Global Variables were one of the most important features introduced with Cribl LogStream 2.1. In this short post we’ll take a look at what they are, how they work and how to use them to build more effective pipelines. If you’re new to Cribl LogStream it may make sense to take our sandbox for a drive before reading further. LogStream Global […]

Ledion Bitincka Mar 8, 2020

Cribl LogStream 7x more efficient than LogStash and Fluentd

Cribl LogStream often gets compared to more general purpose streams processing engines or other open source log shippers. “How is this different from LogStash or Fluentd” is a frequent question from our prospects. In the past few months, we’ve been laying out our vision of an Observability Pipeline, and how one can achieve these benefits, […]

Ledion Bitincka Feb 19, 2020

Cribl LogStream up to 55x faster than Apache NiFi

Cribl LogStream often gets compared to more general purpose streams processing engines. “How is this different from Apache NiFi?” is a frequent question from our prospects. In the past few months, we’ve been laying out our vision of an Observability Pipeline, and how one can achieve these benefits, even building your own. We’ve seen people […]

Ledion Bitincka Jan 27, 2020

Building for Multi-Petabyte Scale, Part 2

Scaling up with NodeJS This is the second post in a series aimed at describing how we designed Cribl LogStream to scale to process 10s of PBs of machine data per day. If you want to read why such scale is important check out part 1. In this post we’ll focus on scale-up, that is […]

Ledion Bitincka Jan 16, 2020

Building for Multi-Petabyte Scale, Part 1

Measure twice, cut once – understanding the requirements. This is the first of a series of posts where we’ll talk about architecture and implementation principles we’ve followed when building Cribl LogStream to be able to scale to processes 10s of Petabytes per day and at sub-millisecond latency. First, in this post, we’ll discuss one of […]

Dritan Bitincka Jan 7, 2020

Scaling Cribl LogStream Architectures: A Practical Approach

How does Cribl LogStream process data at scale? What does a distributed deployment look like? In this post we’ll discuss deployment patterns that allow for implementing scale up and scale out architectures. We’ll also look at features that align with best-practices approaches and strategies. Let’s get to it! The Need for Scale Most of Cribl […]

LogStream Sandbox

Learn about the features of Cribl LogStream in our interactive sandboxes!