System admins have the unenviable task of getting API data into logging tools for analysis. They employ a wide range of tactics to do this –homespun Python scripts, hand-coded batch jobs, and bespoke vendor protocols.  

We’ve heard about this challenge from many customers, which is why we’ve expanded our Data Collection feature in Cribl LogStream.  In one tool, an easy way to collect data from multiple sources. Vendor agnostic.  Scheduled or ad-hoc.

Today, we’re pleased to announce the general availability of LogStream 2.3. With LogStream 2.3, we add scheduled data collection and collection from several new sources, including REST APIs.

Scheduled Data Collection

With the release of LogStream 2.2, Cribl offered the first tool purpose-built for the batch processing of logs.  This gave new freedom to security and IT professionals having to make decisions about what data to send to logging tools and how to more efficiently retain data for later investigations and long-term compliance.

Scheduled Data Collection enables you to set recurring schedules for the distributed collection of data from multiple sources. LogStream 2.3 allows you to configure collections based on resource filters and constraints.  You can also limit concurrent running instances of ad-hoc and scheduled jobs.

REST and Other APIs

Cribl continues to add new data sources for LogStream in our effort to be a universal receiver and collector of log and metrics data.  LogStream 2.3 adds ad-hoc and scheduled data collection from all REST endpoints. We offer several different ways to discover and retrieve REST data, with both known-structure and  schema-agnostic retrieval options.

LogStream can also receive data from other new APIs and sources. These include Kinesis Firehose via the Kinesis HTTP endpoint.  Raw HTTP data can be received on specified ports, which can create corresponding events and be pushed to Event Breakers. LogStream can also collect batch data from the Office 365 Service Communications API for service incidents on Microsoft cloud services, and from the Office 365 Management Activity API for actions and events on Azure Active Directory, Exchange, SharePoint, and other Microsoft servers.

Let LogStream be the universal collector to get all of these data sources into your analytics tools and other destinations.

Also New in LogStream 2.3

LogStream 2.3 introduces new functions including “Grok” for extracting fields from unstructured log data, based on patterns; “Rename” for efficient renaming of individual fields, especially in bulk; and “Metrics Rollup” for aggregating frequently generated metrics into wider time buckets.

LogStream can now route events and metrics to Wavefront analytics and SignalFx monitoring.  The 2.3 release also ships with more datagen sample files, representing several common use case scenarios.

LogStream 2.3 has improvements in monitoring and visualization, including sparklines, click/zooming, and new monitoring for data collection jobs.  We have improved the overall user interface and user experience, including enhancements to the Preview Pane.  We’ve added in-product documentation, which is available for offline reference – especially useful in air gapped environments. 

There are so many other improvements and new features to explore.  For a complete list of all the new stuff, read the LogStream 2.3 release notes.

Wrapping Up

We couldn’t deliver all of these great new features without your feedback.  Our products are designed to help you solve your biggest challenges in unlocking the value of security and machine data. Let us know what you’d like to see next. Here are a few ways you can get started.

• Download LogStream 2.3 today and process up to 5 TB of data per day at no charge
• Join our community Slack and interact with other LogStream users 
• Follow us on LinkedIn
• Test drive Logstream for yourself in our interactive, self-guided sandbox course on Data Collection
• If you have any questions about anything Cribl, use the chat widget here on the site