In today’s world of relentless data growth, security-relevant logs represent a small snapshot of an organization’s overall environment. Teams are beset with a variety of data types, including performance metrics and traces, asset configuration and state, audit logs, and much more. On top of that, teams are expected to scan all of this to compare against industry best practices and join this data with logs and metrics for added context. Logs and metrics can be solved, but integrating the data available behind multiple APIs is no small task.

Cribl Search can be an integral part of a comprehensive security strategy by abstracting 3rd party service API calls to enable users to query any and all of them in real-time. This blog will cover how to configure an API dataset and walk through a few example queries utilizing APIs.

Configuring an API Dataset Provider and Dataset

The first step is to configure your cloud service as a dataset provider, a configuration object that tells Cribl Search where to query and also contains access credentials.

ID is a unique identifier for the dataset provider. This is how you’ll reference it when assigning datasets to it.

1. Set Dataset Provider Type to AWS API.
2. Click Add Configuration to specify your AWS account. In the Account Configuration table, enter:
   • Name: AWS account name.
   • Access key: Long-term credentials for an IAM user
   • Secret key: The Secret key is combined with the access key to authenticate your requests

Next, define a dataset that tells Cribl Search what data to search from the dataset provider we just configured.

Set the Dataset Provider field to the ID of the AWS dataset provider you just configured. From here, you can also customize which API endpoints and regions within AWS you would like Cribl Search to query.

Searching the AWS API

Now that we have both a dataset provider and dataset configured, we can start searching.

dataset="aws_api" | limit 1000

This simple query selects from your AWS API dataset, limiting the search to the most recent hour, and to a maximum of 1,000 records. The first event returned in our list came from the ec2_security_groups endpoint and describes its configuration. Taking a deeper look at the IP permissions shows that it is currently open to 0.0.0.0/0 on port 22. Let’s build another query to see if anything else shares this configuration.

dataset="aws_api" endpointName="ec2_security_groups" IpPermissions[1].ToPort="22" IpPermissions[1].IpRanges[0].CidrIp="0.0.0.0/0" | summarize Groups= count(GroupName), GroupsIds=count(GroupId) by VpcId

Now we’re asking to search just the ec2_security_groups endpoint in the AWS API dataset for events that have the same configuration we found above. Notice, however, that we passed the results to the summarize operator. This will create a table, aggregating results by the specified fields. In this case, we’re creating two new fields representing the count of unique values for the fields GroupName and GroupId, then showing the VPC in which they exist. In just two queries we went from exploring randomly returned results to finding the needles in the metaphorical haystack of data available behind all AWS API endpoints.

Multi-cloud Searching

We could take examples like this even further by asking these questions of all our cloud-based solutions. Remember that Cribl Search is able to translate our generic query to utilize the appropriate API as defined in the dataset configuration, e.g. the ec2_security_groups API endpoint we recently used. This two-step abstraction allows us to easily extend our queries across multiple datasets, even if those datasets come from different cloud providers.

The following query will graphically represent the distribution of events across our various API datasets (you can change the graph type in the “Actions” dropdown):

dataset="*_api" | summarize count() by dataset

Cribl Search supports wildcards, so this query will evaluate any dataset that matches the pattern (prefix)_api. This can include not only Azure and GCP datasets, but even other SaaS providers like Zoom and Okta.

Wrapping Up

To recap, we’ve covered how to configure your first API dataset provider and dataset, how to build a few queries, and how to extend those queries to search multiple datasets simultaneously. There is a wealth of information available behind these endpoints that provide valuable insights into the current state, configuration, and activity in a cloud-based environment. Cribl Search makes identifying, correlating, and integrating this data easier than ever. Ready to try it for yourself? Set up a free Cribl.Cloud account to gain instant access to Cribl Search!

Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.

We offer free training, certifications, and a free tier across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started and continue to build and evolve. We also offer a variety of hands-on Sandboxes for those interested in how companies globally leverage our products for their data challenges.