This blog post discusses utilizing Cribl Search to pull and visualize data from the AWS API without ingesting data. This will allow you to collect, analyze, and visualize data from your AWS account in real time without ingesting the data first.

The Use Case

Traditionally, analytics platforms have only been able to analyze data that the platform has consumed in its data store. Data is fed into analytics platforms from APIs on a scheduled interval to ensure access to the latest or at least recent data. This results in hundreds, often thousands, of copies of the same data being stored, with the vast majority of those copies never being searched.

Cribl Search flips this paradigm on its head, allowing users to pull, filter, search, and analyze data from APIs without ever storing it. This allows for greater flexibility, better outcomes, and a more straightforward process.

Let’s look at this use case: an overview dashboard that shows the currently allocated AWS EC2 instances for a manager/team to manage their lab/sandbox space.

Setting up the Dataset Provider

The first step is to set up a dataset provider. This is the authenticated connection back to the source of data. Setting up the dataset provider is as simple as entering credentials for each AWS account.

Information on the permissions required for each endpoint is available here.

Setting up the Dataset

The dataset will define the endpoints and regions searched. Data for each endpoint across each region will be collected whenever you query this dataset. In this case, we will only search for ec2 instance data across four regions.

Searching Data

We will reference the dataset we just created to start searching our data. When searching data, the time selector references the field UsageOperationUpdateTime, so the time picker must be adjusted accordingly.

From here, we can start manipulating the data to find the items we seek. For example, if we want to see the longest-running instances. In that case, we can use a query similar to this:

We can also convert this data into visualizations to create our executive dashboard. Let’s look at how we can visualize which Owners have running EC2 instances.

Note: Owner names have been obscured

Using the “Actions” menu, you can save this visualization to a dashboard to save your search and visualization to be accessed later or turned into a full-blown executive dashboard.

Wrap-Up and Next Steps

You can continue adding panels and visualizations that fulfill your business requirements. Are you ready to try it for yourself? Sign up for Cribl.Cloud and gain instant access to Cribl Search today!

Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.

We offer free training, certifications, and a free tier across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started and continue to build and evolve. We also offer a variety of hands-on Sandboxes for those interested in how companies globally leverage our products for their data challenges.