Cribl

Destination

HDFS

Scale Hadoop clusters to handle large data sets on commodity hardware

Apache HDFS (Hadoop Distributed File System) is a distributed file system that handles large data sets running on commodity hardware. HDFS is highly fault-tolerant, and supports high-throughput access to application data. It can be used to scale a single Hadoop cluster up to hundreds or thousands of nodes. HDFS can support Apache HBase, a column-oriented, non-relational database management system that supports real-time data needs with in-memory processing.

How to get data flowing

This integration is facilitated through the Cribl Stream Filesystem/NFS Destination.

  • Mount HDFS on the target filesystem via FUSE (Filesystem in Userspace).

  • Configure Stream to send data to your Hadoop cluster via Sources > Filesystem.

  • Specify the output and staging locations, partitioning expression, data format, prefix expression, compression behavior, and backpressure behavior, and optional parameters.

  • Stream will start sending data to Hadoop (via the specified file system) as it becomes available.

View Documentation

Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.

We offer free training, certifications, and a free tier across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started and continue to build and evolve. We also offer a variety of hands-on-Sandboxes for those interested in how companies globally leverage our products for their data challenges.