August 2021 Updates to LogStream Include Notifications, New Integrations, and a Public Repository for Packs

Nick Tankersley
Written by Nick Tankersley

August 11, 2021

The team at Cribl is excited to announce the August 2021 updates to LogStream. We announced it in our community a few hours ago, and we just couldn’t wait to make all of the details publicly available. Let’s dive into what’s new in our latest release.

Notifications Take the Guesswork Out of Data Reliability

An important part of a mission-critical observability pipeline is being able to guarantee the flow of data to multiple destinations. But you don’t always have control of those destinations. When something goes wrong downstream, it can be hours before the team responsible for the destination knows there’s a problem.

The Notification features in LogStream enable users to identify a destination that is no longer receiving data, and to quickly notify the right team that the destination needs to be investigated.

You can send LogStream notifications to your on-call system, ticketing service, or ChatOps channels, enabling data teams to respond quickly and collaboratively. While your teams investigate data destination outages, they can simultaneously prevent data loss and poor user experience for those providing essential reports, dashboards, and analytics services.

The Cribl Pack Dispensary™ Is the Grooviest New Content Library in Town

What good is a way to build and share LogStream Pipelines, Samples, and Knowledge Objects – that is, LogStream Packs – without a place to share them?

Welcome to the grand opening of the Cribl Pack Dispensary! A community of data experts building and sharing LogStream Packs, to make working with streaming data easier, faster, and more valuable for everyone.

The Cribl Pack Dispensary integrates with the new Pack management tools that LogStream added in 3.1.0. Install directly from any of the dozen (plus!) Packs are already available in the Dispensary.

Not able to connect to GitHub directly from your LogStream environment? We got you. All of the Packs in the Dispensary have a .crbl file, accessible under Releases, that you can download directly and install in air-gapped environments.

For those looking to pull from private repos, branches, tags, and/or their own internal git systems: You can do all of those things with LogStream 3.1.0 – giving you secure and flexible control of Pack development and deployment in your LogStream environment.

Thinking about developing a Pack? Then join us at #packs in our Slack Community workspace.

Secrets Manager Available for Every Source and Destination

We’ve extended our Secrets Manager to work with every source, collector, and destination, making working with auth secrets, tokens, and username/password combos simple and secure. The Secrets Manager provides a central place to create, store (encrypted of course), and recall secrets for use in authenticating with all sources and destinations.

We’ve even thrown in the ability to call the Secrets Manager as part of certain LogStream functions, using the new C.Secret() internal method. This makes it safe to use auth tokens and/or key/value pairs outside of sources, collectors, and destinations, without worrying about exposing those values in plaintext somewhere in your config.

Google Chronicle, Grafana Cloud, and Prometheus Integrations

What would we be without our rich ecosystem of upstream and downstream data partners? We’d be an inert pipeline floating pointlessly in the infrastructure void. Luckily, we have many integrations, so we annoy the void with direction and focus. In this release, we’re adding 6 new integrations (bringing our total sources, collectors and integrations count to well above 70), including:

Each of these integrations provides an easy way to collect data, route it, and expand your observability reach – without sacrificing choice.

LogStream customers want to use their existing streams of data to try new security tools, or to enable other teams with valuable observability data by sending directly to the products and services those teams use daily. Our ongoing addition of new native integrations serves both goals. Some examples:

The Google Chronicle destination makes leveraging your existing security events with Chronicle something that can be achieved in a matter of minutes. This provides threat hunters with the sheer volume of data they need to do their job – without having to re-instrument their entire infrastructure.

The Grafana sources and destinations extend LogStream to provide a universal point for collecting, processing, and routing all your observability data. And, to make sure we covered all our bases, you can collect data from any Loki- or Prometheus-compatible source, and send it to any Loki-compatible destination. The same for the Prometheus metrics that are often used to power Grafana dashboards.

Other Items in the August 2021 Release

We’ve also added several updates to LogStream, beyond what we’ve talked about today, including:

  • The Code Function, for writing full JavaScript functions as part of a pipeline.
  • Configurable behavior for what to do when Persistent Queues fill up. And multiple updates to the UI, to account for all the new stuff in LogStream.

To get started with LogStream 3.1 in your cloud, or ours, sign up for LogStream Cloud. There, you can use LogStream for up to 1TB of throughput per day, free on accounts created through Jan. 1, 2022. Or, download LogStream and run it in your own environment. You can license more throughput as you need it.

Don’t forget to stop by the Cribl Pack Dispensary for a pick-me-up, and join us on Slack if you have any questions. Happy streaming!

Questions about our technology? We’d love to chat with you.