Source: Kafka
Apache Kafka is an open-source, distributed event streaming platform widely used for high-performance data pipelines, streaming analytics, metrics collection and monitoring, log aggregation, data integration, and mission-critical applications. As a durable message broker, Kafka enables applications to process, persist, and reprocess streamed data.
How to get data flowing to Kafka
This is a built-in integration between Cribl Stream and Kafka.
Kafka as Source and Stream as a destination
Create a Kafka topic, write to it, and export to LogStream via Kafka Connect.
Configure Stream to read data from Kafka via Sources > Kafka.
Specify the Kafka brokers, topics, and optional settings (Confluent Schema Registry, TLS certificate, and SASL authentication parameters).
Stream will start fetching data as KDS streams become available.
Destination: Amazon S3
Amazon Simple Storage Service (S3) offers storage of any amount of data, at any time, from anywhere on the web. S3 is accessible via a web services interface, a management console, SDKs for several languages and frameworks, and several APIs. Read our Solution Brief.
How to get data flowing to S3
This is a built-in integration between Cribl Stream and the Amazon S3 APIs. Stream pulls data from S3 buckets using event notifications through Amazon SQS. Stream’s S3 Destination can be adapted to send data to services for which Stream currently has no preconfigured Destination.
S3 as Destination and Stream as a source
Configure Stream to send data to S3 via Destinations > Amazon S3.
Configure bucket name, AWS Region, staging location, data format, key and file name prefixes, optional partitioning expression, and other details.
Authenticate via keys or IAM role.
Stream will start sending data as it becomes available.