Pipeline Profiling: Or How I Learned to Stop Worrying and Isolate the Problem

It’s that time of year again! If you’re not a procrastinator, you’ve probably already blown out your sprinklers for winter and are looking forward to the snow and holidays ahead. Well done, irrigation purists! I, on the other hand, am an olympic-level procrastinator and will usually wait until the last moment before NWS forecasts a 10″ snow for the night then frantically search for my air compressor. Needless to say, I have learned my lesson the hard way (but I’ll still procrastinate again next year).

We all know fixing an irrigation leak is easy once you know where the leak is. But the journey to spot the exact location is rocket-science-level stuff or finding-water-in-Mars-level stuff. Fast-forward, I spent 10 bucks at Home Depot for the replacement pipe parts… but it cost me another $800 to have someone come out, find the leak, isolate the problem to its exact location, dig it up, fix, and cover back up.

Moral of the story? Fixing the problem is easy but isolating the problem is hard.

But it doesn’t have to be that way. Not anymore at least.

Cribl Stream’s Pipeline Profiling feature is the panacea for all your Observability pipeline issues!

Pipeline Profiling Ensures Predictability and Moves Admins From Troubleshooting to Innovating:

According to a recent survey, data engineers spend two days every week on troubleshooting data issues, and ~60% of those surveyed mentioned the cause to be complex data pipelines.

Cribl Stream’s Pipeline Profiling feature aims to solve precisely this. For example, let’s take a pipeline that collects data from various sources such as Web-servers, Firewalls, and DNS Servers where the data is fed into Cribl Stream to undergo a series of data transformations before routing. Suppose the performance wasn’t meeting expectations. Sounds like it’s time to get the problem isolated. But with limited visibility, it’s hard to pinpoint and isolate exact issues. Also, successfully isolating the problem can be time-consuming and tedious in a production environment.

Pipeline Profiling, in Cribl Stream, helps identify performance issues with granular function-level stats. It comes with a Preview mode so you don’t have to worry about breaking production! I’ll say it again, with Pipeline Profiling, you don’t have to wait for pipelines to be deployed in production to determine the performance of pipelines. One last time, you can determine the performance of a pipeline BEFORE it is in production. Pipeline profiling provides resource consumption stats for pipelines and the individual functions within, including (but not limited to):

• Latency introduced by the function/pipeline.
• Throughput (in/out, events/bytes)
• Data reduction

Cribl Stream’s Pipeline Profiling feature can be enabled on-demand when there is a need to troubleshoot performance issues. With the advent of Pipeline Profiling, ITOps and SecOps teams can better utilize their time rather than wasting hours trying to isolate and fix problems. Take a look at these screenshots to see Pipeline Profiling in action.

Summary

Pipeline Profiling is but one of the exciting new announcements we have in store for you this Fall. If you’re interested, you can sign up at Cribl.Cloud today, where you can try our products for free up to 1TB of data per day. Take it for a spin, and we betcha you will love it! We’d love to hear from you in our community if you have any feedback. Join us to learn more about the newest features in our Cribl suite of products.

If you’d like to dig deeper, the Cribl Certified Observability Engineer (CCOE) certification program offers free industry-leading observability and security data education and knowledge. It exemplifies one’s commitment to mastering the skills and knowledge required to be a true observability engineer. Create your free Cribl University learner account and start your CCOE certification journey.

More on Cribl’s Fall Launch

• Advancing Observability: Cribl Search and New Product Enhancements Available Today
• Cribl is Redefining Search for Your Observability & Security Data
• What’s New With Cribl Stream 4.0?
• What’s New With Cribl Edge 4.0?
• Cribl’s New User Interface
• What’s New With Cribl.Cloud?
• The Stream Life Episode 66: Cribl Search is Here!