x
edge.blog.headerBG.2X.1792x600

Updates From the Edge: Scalability for 250,000 Nodes and More

June 18, 2024
Written by
Judith Silverberg-Rajna's Image

Judith spearheads product marketing for Cribl Edge and our partners. With extensive exper... Read Moreience in data platforms and artificial intelligence, she is passionate about addressing the challenges of big data, utilizing Cribl products to solve the most complex issues. A proud alumna of UC Berkeley, Judith holds a degree in economics and carries the ‘Go Bears’ spirit in all her work! Read Less

Categories: Cribl Edge

We know endpoints can be endless and that you need an efficient, simple way to collect data from all of them— no matter the size of your environment. This is why, in the 4.7 release, our engineering team has worked hard to expand Cribl Edge’s scalability to support 250,000 Edge nodes!!! Whether you are a large enterprise, have a vast Kubernetes environment, or want to futureproof for inevitable growth (data volumes are growing at 28% CAGR, after all), Cribl Edge can handle your evolving data management needs at scale. As always, our goal is to partner in our customers’ journey to manage their data effectively, and we listen closely to our customers’ needs for today and the future.

As our customers continue to operationalize and expand their Cribl Edge deployment, we’ve made some changes to the UI to help them in their scaling journey.

  • Expanded List View: The health and status of all connected Sources and Destinations for each Node are now easily accessible in the List View.

  • Automated Connection Process Management: Manual configuration of connection processes is no longer necessary. Just enter the total number of Edge Nodes in the Edge Nodes Count setting within Global Settings, and Edge will automatically deploy the required connection processes to support your infrastructure.
  • Streamlined Map View: To improve performance in managing large deployments, the “Chart” and “During” fields have been removed.
  • Enhanced Status Filtering: The Status tab for Sources and Destinations now allows you to filter Edge Nodes by their health status. This helps you prioritize examining nodes that are malfunctioning or having errors.

What are you going to do with this expanded scalability? Well, now you can start monitoring your always-on workstations. As of 4.7, Edge nodes can be run on Windows 10 and 11 on desktops with a reliable, continuous network connection and power management features like sleep or hibernate disabled. Later this year, we’ll expand this to personal use desktops and laptops that aren’t always connected and turn on/off frequently. In the meantime, start running Cribl Edge on those 24/7 lobby computers, POS systems, and more.

As you expand your data collection at the edge, we want you to control your costs. To empower you with this control, in release 4.6, we introduced a new feature where the original dropped events do not count against the ingest volume. Specifically, they must be dropped in pipelines or sent to the DevNull Destination, from which they are summed up and subtracted from the total ingest to arrive at Net Ingest for that Edge node. Read some examples of how you can use this to get more out of your Edge ingestion in this blog.

But what’s scalability without performance? We understand your resources are precious, so we’ve reduced the memory footprint of Cribl Edge by streamlining its binary, resulting in improved performance, especially for deployments with limited resources. Earlier in 4.5, we also reduced the overall memory usage of Edge on Windows by 10-25% by enabling pointer compression in Windows builds for the Cribl package. We aren’t stopping here; we plan to do more work on our performance, from greater Windows optimization to further footprint reduction.

We hope you are as stoked as we are about these recent updates with Cribl Edge. If you haven’t already tried out Cribl Edge, make sure to check out the sandbox. If you’re ready to expand your Cribl Edge deployment, refer to the Scaling Edge Beyond 10k Nodes guide for a detailed explanation of these changes and best practices for scaling your environment.


 

Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.

We offer free training, certifications, and a free tier across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started and continue to build and evolve. We also offer a variety of hands-on Sandboxes for those interested in how companies globally leverage our products for their data challenges.

.
Blog
Default Image

How to Cut Through the Chaos of Custom App Log Management

Read More
.
Blog
Feature Image

Cribl’s Blueprint for Secure Software Development

Read More
.
Blog
Feature Image

Calling All MSSP’s and MDR’s! Cribl.Cloud is Here for You!

Read More
pattern

Try Your Own Cribl Sandbox

Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.

box

So you're rockin' Internet Explorer!

Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari

Got one of those handy?