In the ever-evolving world of data analysis, the ability to interact directly with live API endpoints is a significant advancement for practitioners. Cribl Search now offers this capability, enhancing your data analysis toolkit. This new feature allows you to gain broader visibility into the periphery of your infrastructure, enabling a more comprehensive analysis of user journeys and operational trends.

By querying live API endpoints, you can seamlessly integrate real-time data into your analysis, ensuring your insights are as current and relevant as possible. This development is especially valuable for connecting disparate data points across various platforms and applications. Whether you’re monitoring user interactions, evaluating system performance, or tracking application usage, directly integrating live API data into Cribl Search provides a more dynamic and holistic approach to data exploration.

This guide will walk you through setting up Azure API as a dataset provider to Cribl Search. These steps will leverage this enhanced capability to enrich your data analysis and decision-making processes.

Step 1: Registering the Application and Service Principal in Azure

Create an Azure service principal, an identity for your applications and tools to access Azure resources. Follow these steps to register:

1. Go to the Azure portal and access ‘App registrations.’
2. Select ‘New registration’ and provide the necessary details.
3. Note the Application (client) ID and Directory (tenant) ID for later use.

In this image, the Azure app search_api has a Service Provider with a display name search_api

Reference: Create a service principal in Azure.

Step 2: Assigning Roles to the Service Provider

Assign the right roles to your service principal for appropriate access levels. You can opt for the ‘Reader’ role or a custom role for specific permissions.

1. In Azure, navigate to ‘Subscriptions’ and select yours.
2. Go to ‘Access control (IAM)’ and choose ‘Add role assignment’.
3. Select the ‘Reader’ role or create a custom role.

In this image, the search_api Service Provider is assigned the Role Reader

Reference: You can assign the built-in role of Reader to the application so it has read access to all endpoints. To limit access to the current Search endpoints (listed in Cribl Search docs), create a custom role: Tutorial: Create an Azure custom role with Azure PowerShell – Azure RBAC

Step 3: Creating the Azure API Dataset Provider

1. In Cribl Search, navigate to Data → Dataset Providers.
2. Click ‘Create Provider’.
3. If prompted with a drop-down menu for Stream Worker Groups or Data Lake Amazon S3 Destinations, proceed by clicking ‘Create.’

Configuring the New Dataset Provider:

• Set the ID as a unique identifier for the dataset provider.
• The Description field is optional.
• Choose Azure API as the Dataset Provider Type.
• Click ‘Add Configuration’ to enter your Azure account details:
  • Account Name: Name of your Azure account.
  • Tenant ID: ID of your Azure Active Directory.
  • Client ID: ID of the application connecting to Azure Active Directory.
  • Client Secret: Secret key for the connection.
• Save your configurations.

Step 4: Creating the Dataset

Adding a New Dataset:

1. In Cribl Search, go to Data → Datasets.
2. Click ‘Add Dataset’.

Configuring the New Dataset:

• Set the ID as a unique identifier for the dataset.
• The Description field is optional.
• Choose the Azure dataset provider you created earlier as the Dataset Provider.
• Click ‘Add endpoint’ to select your desired endpoints (virtual machines, disks, networkSecurityGroups, web apps).
• Enter the Subscription IDs you wish to query.
• Under Processing, set up Datatypes for data organization and field definition.

Reference: Azure API | Cribl Docs

Step 5: Start Searching

With your dataset provider and dataset configured, you’re now ready to explore your data. Search results can appear in seconds, depending on the volume of data in your account.

Wrap up

Now that we’ve walked through that process, are you ready to try it yourself? We offer instant access to Cribl Search through Cribl.Cloud with a generous daily free usage. Check it out!

Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.

We offer free training, certifications, and a free tier across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started and continue to build and evolve. We also offer a variety of hands-on Sandboxes for those interested in how companies globally leverage our products for their data challenges.