August 11, 2021
The team at Cribl is excited to announce the August 2021 updates to LogStream. We announced it in our community a few hours ago, and we just couldn’t wait to make all of the details publicly available. Let’s dive into what’s new in our latest release.
An important part of a mission-critical observability pipeline is being able to guarantee the flow of data to multiple destinations. But you don’t always have control of those destinations. When something goes wrong downstream, it can be hours before the team responsible for the destination knows there’s a problem.
You can send LogStream notifications to your on-call system, ticketing service, or ChatOps channels, enabling data teams to respond quickly and collaboratively. While your teams investigate data destination outages, they can simultaneously prevent data loss and poor user experience for those providing essential reports, dashboards, and analytics services.
What good is a way to build and share LogStream Pipelines, Samples, and Knowledge Objects – that is, LogStream Packs – without a place to share them?
Welcome to the grand opening of the Cribl Pack Dispensary! A community of data experts building and sharing LogStream Packs, to make working with streaming data easier, faster, and more valuable for everyone.
The Cribl Pack Dispensary integrates with the new Pack management tools that LogStream added in 3.1.0. Install directly from any of the dozen (plus!) Packs are already available in the Dispensary.
Not able to connect to GitHub directly from your LogStream environment? We got you. All of the Packs in the Dispensary have a .crbl file, accessible under Releases, that you can download directly and install in air-gapped environments.
For those looking to pull from private repos, branches, tags, and/or their own internal git systems: You can do all of those things with LogStream 3.1.0 – giving you secure and flexible control of Pack development and deployment in your LogStream environment.
We’ve extended our Secrets Manager to work with every source, collector, and destination, making working with auth secrets, tokens, and username/password combos simple and secure. The Secrets Manager provides a central place to create, store (encrypted of course), and recall secrets for use in authenticating with all sources and destinations.
We’ve even thrown in the ability to call the Secrets Manager as part of certain LogStream functions, using the new
C.Secret() internal method. This makes it safe to use auth tokens and/or key/value pairs outside of sources, collectors, and destinations, without worrying about exposing those values in plaintext somewhere in your config.
What would we be without our rich ecosystem of upstream and downstream data partners? We’d be an inert pipeline floating pointlessly in the infrastructure void. Luckily, we have many integrations, so we annoy the void with direction and focus. In this release, we’re adding 6 new integrations (bringing our total sources, collectors and integrations count to well above 70), including:
Each of these integrations provides an easy way to collect data, route it, and expand your observability reach – without sacrificing choice.
LogStream customers want to use their existing streams of data to try new security tools, or to enable other teams with valuable observability data by sending directly to the products and services those teams use daily. Our ongoing addition of new native integrations serves both goals. Some examples:
The Google Chronicle destination makes leveraging your existing security events with Chronicle something that can be achieved in a matter of minutes. This provides threat hunters with the sheer volume of data they need to do their job – without having to re-instrument their entire infrastructure.
The Grafana sources and destinations extend LogStream to provide a universal point for collecting, processing, and routing all your observability data. And, to make sure we covered all our bases, you can collect data from any Loki- or Prometheus-compatible source, and send it to any Loki-compatible destination. The same for the Prometheus metrics that are often used to power Grafana dashboards.
We’ve also added several updates to LogStream, beyond what we’ve talked about today, including:
To get started with LogStream 3.1 in your cloud, or ours, sign up for LogStream Cloud. There, you can use LogStream for up to 1TB of throughput per day, free on accounts created through Jan. 1, 2022. Or, download LogStream and run it in your own environment. You can license more throughput as you need it.