Recently, we launched a new Sandbox focused on handling syslog at scale with Cribl. The marketing messaging behind the Sandbox has been done a couple times already; therefore I wanted to let y’all see what we as Cribl Technical Marketing Engineers(TMEs) actually do in our daily lives. I’ll try to keep it engaging, with tales of danger and subterfuge, but I can only take so much artistic license.

What’s in a Sandbox and how the Sandbox platform functions (i.e. kubernetes, kustomize, and APIs Oh My!) will be covered in many future posts. For now, we’ll only peel back the curtain part way so to speak. I wanted to go into detail about what was tough to do making this Sandbox rather than what is tough about making Sandboxes in general.

In this Sandbox, then, we wanted a “real world” analog environment for admins using syslog and we settled on: two syslog sources (PAN and Apache), a Syslog-NG receiver, and a Splunk Universal Forwarder log shipper, all ending in a Splunk Indexer. Light work, right? Well for the most part, actually, yes. We maintain a library of previously used Sandbox building blocks (sand?) for the express purpose of making new sandboxes easier to build and deploy. This means that the two syslog sources, Splunk UF, and Splunk Indexer were pretty straightforward: just include their manifests in my kustomization.yml.

Syslog-NG would prove to be my pain point (which was kinda the point of this whole sandbox to begin with). Let’s start with the volume. No, not a spinal tap reference, a persistent volume claim (PVC). This would allow the Syslog-NG container to write out somewhere and have it be picked up by something else. The problem here is getting the “something else”s to read the data.

The Splunk UF is pretty easy, we just patch in a Splunk UF as a sidecar container to the Syslog-NG pod and Bobby’s your cousin. No issues having it ship off to the indexer or reading the files.

The script also called for users to install Cribl Edge on the Syslog-NG server though… Well, we’re not doing that because… Kubernetes. There isn’t really a syslog server. Also, we can’t give users access to the CLI for our entire Sandbox architecture; we’re not in the business of letting y’all mine Bitcoin. The solution would prove straightforward, yet painful (mostly because of my lack of Syslog-NG knowledge): mount the syslog PVC into the terminal container we use in Sandboxes!

I thought I was so clever until Edge couldn’t read any files in /var/log. Then I got them readable in the terminal, but they weren’t in the right directories. Then I got them in the right directories and named them properly, but I’m back to not being and to read them. It was a mess.

Luckily I’m good at Google (like any tech support person) and figured out the issue rather quickly. The end result looks like this:

source s_network {

default-network-drivers();

};

filter f_pan {

match("syslog", value("HOST"));

};

filter f_apache {

match("cribl\.io", value("HOST"));

};

destination d_apache {

file("/var/log/apache/${YEAR}-${MONTH}-${DAY}.log" perm(0644) create_dirs(yes) dir-perm(0755));

};

destination d_pan {

file("/var/log/pan/${YEAR}-${MONTH}-${DAY}.log" perm(0644) create_dirs(yes) dir-perm(0755));

};

log {

source(s_network);

filter(f_pan);

destination(d_pan);

};

log {

source(s_network);

filter(f_apache);

destination(d_apache);

};

Now to the experienced user, this looks messy but easy. To someone who has never picked up Syslog-NG, however, wow was this a journey? File names can’t separate tokens with periods? Do you have to tell Syslog-NG to create the directory? And you have to tell it the permissions to the directory? And permissions for the file?

This experience made me realize just how easy I have had it using Cribl as my introduction to the observability space. I rarely look up where or how to do something in the Cribl UI. I never have to edit the config and restart Cribl in order to see if it works. I’ve never missed a semicolon while configuring a Pipeline or Route… I digress. Now that the Syslog-NG config was done, I added my base Cribl configs to the course, booted it all locally, and triple-checked the script.

After that, the journey went quickly and smoothly. I opened a pull request to get my new sandbox merged into the test branch. This kicked off the workflow to notify the TME and Technical Writing teams to review the new Sandbox. Once I corrected all the regular spaces into non-breaking spaces, we made it live on the production site and the rest is history.

It’s not every day that I have to use a “competitive” product, but when I do, I love Cribl even more. Don’t take my word for it, though, go try out our new sandbox “Handling Syslog at Scale with Cribl” yourself. Cheers!

Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.

We offer free training, certifications, and a free tier across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started and continue to build and evolve. We also offer a variety of hands-on Sandboxes for those interested in how companies globally leverage our products for their data challenges.