Managing log data in Amazon Web Services is essential for modern IT and security teams, but keeping track of where those logs actually live is a constant headache. Between multi-region deployments, default destinations, and sprawling multi-account organizations, critical operational, performance, and security data can easily slip through the cracks. Getting a complete view of your logging footprint shouldn't require manual hunting or complex custom scripts.

AWS logging

Amazon Web Services (AWS) generates a wide range of logs across your environments. There are, of course, logs generated by applications running in these environments, but there is also a wealth of information about the environment itself, including configuration changes, authentication information, access attempts, traces, audit trails, and service issues.

This information is incredibly useful across multiple teams, with use cases including real-time operational monitoring, incident response, and performance debugging.

The problem

Searching these logs within AWS presents a familiar challenge for many teams when onboarding data into any log management platform: logs are often scattered across multiple locations, many of which aren't immediately obvious to an operator.

At first glance, CloudWatch seems like a central location. But, different services can send logs to a range of locations, not necessarily CloudWatch. Logs can be stored in Amazon S3, Firehose, and Firehose destinations. They’re often stored per region, meaning that there are even more locations to search. If an organization doesn’t have a complete view of which regions are enabled within their environment, they could miss critical logs for those locations.

This issue exists for any account, but organizations with numerous accounts compound the problem. Even for an organization that makes diligent use of Infrastructure as Code (IaC), it’s easy to miss some logging sources. Depending on the service, logs might route to a default destination that isn't specifically configured in the IaC templates, undermining its use as a source of truth for logging locations.

Ultimately, an organization can easily miss out on collecting valuable log data from their AWS environments without ever realizing it.

Enter Cloud Connections

To address these issues, Cribl created Cloud Connections to solve two core problems:

Enumerating the log destinations within your AWS organization or account(s).
Onboarding those log destinations to Cribl with minimal configuration required.

To streamline onboarding newly discovered logs, Cloud Connections generates CloudFormation configuration files, which can be applied as a CloudFormation stack to complete onboarding for those log sources to Cribl.

This isn't limited to organizations already using Cribl, either. The same tooling can onboard entirely new organizations or individual environments.

Using Cloud Connections shortens the effort required to onboard an AWS organization to Cribl and further reduces time-to-value.

How does it work?

To discover logs within an organization or environment, Cloud Connections needs appropriate access. At the beginning of the process, it generates a CloudFormation template that adds an appropriate AWS Security Token Service (STS) role to the environment. This permits read-only access to a number of AWS services. The service will then assume this role when connecting to an environment.

The first step in detection is to enumerate all service instances that generate logs, obtain their logging configurations, then identify the configuration of each associated logging destination.

This process relies on AWS APIs, which have different rate limits and quotas depending on the service. As a result, the time required to determine the location of all logs in an organization or environment will depend on both the size of the environment and the service composition within it.

The discovered logging information contains enough information to determine if a log source has already been onboarded to Cribl, as well as enough information to generate the CloudFormation configuration.

Closing

At the end of the day, gaining full visibility into your AWS environments shouldn't feel like searching for a needle in a haystack. Cloud Connections helps take the guesswork out of log discovery by automatically finding uncollected log sources and generating the CloudFormation templates needed to bring them straight into Cribl. Instead of spending hours hunting down missing log groups across regions and accounts, you can focus on analyzing your data, improving security, and keeping your systems running smoothly.