Taming the Complexity of Windows Event Collection with Cribl Stream 3.4

OK, first things first. I have to admit that I am, first and foremost, an old-school UNIX systems administrator. I’m that grizzled sysadmin in your shop who soliloquizes wistfully about managing UUCP for email “back in the day.” Centralizing Logs? Yeah, we had syslog, and saved it all off to compressed files. When Windows hit the shop, we quickly realized that we didn’t have a whole lot of visibility into what was going on on our Windows machines if we didn’t actively log into the machines to find out. The introduction of the Windows Event Collection (WEC) service helped with that, by enabling the collection and forwarding of all of those events to a central location.

But, as is the case with so many otherwise great system tools, it wasn’t built to scale. Microsoft’s recommendations are to have a collector for every 2,000 or so systems (see Figure 1). While Windows has a significant footprint in most companies’ server environments, most of the time, it has a much, much bigger footprint in the end-user device community. With all of these systems, pretty soon you’ve got a small army of event collectors out there, consuming resources that really could be put to better use.

Additionally, there’s no good way to do load balancing in a native WEC environment. Sure, you can cluster your WEC servers, but that just ends up doubling your infrastructure to provide availability. More importantly, it’s extremely rare to only have Windows in your environment. More likely, you’ve also got some Linux, maybe a few straight UNIX variants out there, not to mention networking gear that’s likely very syslog-centric.

To be able to holistically manage all of this stuff (and the applications that run there), you really need a common way to look at events from each. There are a lot of options for forwarding your Windows events to a common log management system, but they often require you to jump through hoops, like installing an agent on each collector, or every Windows system.

Wouldn’t it be great if you could just configure your Group Policy Object (GPO) to send all events to one place, without a separate dedicated Windows event infrastructure, and then be able to transform and route that data anywhere you want?

That’s EXACTLY what Cribl Stream’s new Windows Event Forwarder Source provides you. It acts like a WEC instance, collecting events from any systems configured to send them. Since Stream worker groups usually live behind a load balancer and can be scaled horizontally (either manually or via autoscaling tools), the stream of events gets load-balanced along the way, providing availability benefits.

And since that data is going through Stream, you can use Stream’s full power. Clean up Windows events (which are notoriously big events) using the cribl-windows-events pack. Archive all events to cheap storage, and only send the ones you want to your end destination. Or even route them to one or more other tools for analysis, etc. And all of this on your existing Cribl Stream deployment!

The Windows Event Source is much more scalable than WEC, with no hard limit on the number of endpoints a given worker can support. Of course, “your mileage may vary” depending on variables like event size, events per batch, etc., but it’s likely that you already have capacity available in your existing environment to accommodate most workloads. Even if you don’t, adding a node or two to your Stream worker group is better than running a slew of WEC servers.

Figure 2 shows the same 8,000-node environment deployed using Cribl Stream’s Windows Event Forwarder Source. Even in this small environment, dropping 4 systems out of the environment simplifies your setup. (You don’t need to get tricky with your GPO’s to make sure the right systems point at the right WEC system.) Sure, you could deploy worker nodes close to the endpoints if you wanted to manage network traffic, but you don’t have to.

WEC Made Simple!

The Windows Event Forwarder is available in the 3.4.0 (and later) version of Cribl Stream. If you’re not familiar with using Cribl Stream to trim and clean up your windows events, try the Windows XML Event Reduction sandbox, which will allow you to learn how to do it in a full version of the product available to you for 24 hours – the course only takes about 10 minutes to go through.

The fastest way to get started with Cribl Stream and Cribl Edge is to try the Free Cloud Sandboxes.