AdobeStock_637546112 (1)

Conquering Data Lakes and Searching Google Cloud Storage Buckets With Cribl Search

March 14, 2024

What might you accomplish if you could easily search your data lakes without paying to move the data first? The most likely outcome is that you address a critical security incident quicker than ever, save your organization millions of dollars, get a promotion, and then go down in history as the best-looking, most talented analyst to have searched a storage bucket.

If not that, you’ll at least be able to consistently make better-informed security decisions, uncover actionable insights, improve outcomes, and find competitive advantages hidden within your data lake. Either way, you’re in good shape.

Connect Your Cribl Search Environment to Google Cloud Storage

Cribl Search is making futures brighter with native support for platforms like Amazon S3, Amazon Security Lake, Azure Blob, and Google Cloud Storage — enabling seamless data analysis right at its source. You can target specific data, create new datasets in minutes, and start searches with just one click.

Here’s how you can set up your Cribl Search environment to connect to your Google Cloud Storage (GCS) in just a few minutes. Check out this video or read below to get set up and start searching your data.

Start Creating a Google Cloud Storage Dataset Provider

The dataset provider tells Cribl Search where to query and what access credentials to use.

To set this up, navigate to Data Dataset Providers in your Cribl Search dashboard. Click Create Provider in the top right and select Google Cloud Storage as your dataset provider type. Give the dataset provider a unique ID and description.

Down below, you’ll see a field to put in the authentication key from the GCS side. First, you’ll have to create a service account and key on the GCS side — the account will also need access with storage admin role privileges to the storage bucket you’d like to search.

Set Up Google Cloud Storage Service Account and Key

Move Into the GCP console to set up your Google Cloud Storage service account and key.

First, you’ll need to create a service account that Cribl Search will use to access your GCS resources. Select IAM & Admin from the left navigation menu. Click Service Accounts and then Create Service Account. Give the service account a name and click Done at the bottom. Find the account you just created in the list and copy the email address of that service account for later.

Next, you’ll pivot to the GCS bucket to grant the service account access to your data. Select Cloud Storage from the left navigation menu and select your bucket from the list. Click PermissionsGrant Access and paste your service account email address into the New Principals field. Use the dropdown to assign the Storage Admin role and click Save.

The last item on the GCP side is to create a key for your service account. Navigate back to service accounts, click the one you created, then select the KEYSADD KEYCreate new key. Select JSON for the key type and click Create. You’ll see a download for your key in your downloads folder.

Finish Setting Up Your Dataset Provider

Now, you can pivot back to Cribl Search to finish setting up your dataset provider. Drag and drop the downloaded key into the Service Account Credentials field, click Save, and your dataset provider will be ready to use.

Set Up Your Google Cloud Services Dataset in Cribl Search

Next, you’ll set up your dataset so that Cribl Search knows exactly which data to search. Select Datasets from the left side menu and click Add Dataset. in the top right. Give it a name and description, then select the provider you created.

When you select a dataset provider, a couple of extra fields will pop up:

Bucket Path refers to any prefix within your bucket that you want to search. Here, I added my bucket name, an archive prefix, and some tokenized, time-based prefixes. Feel free to change this depending on what your partitions look like.

Path Filter accepts any Javascript expressions for path filters you want to add.
Once you click Save, your data set should be ready to use. Start searching by navigating back to the homepage and starting a search there, or leverage the Search Action Button at the end of the row for your data set.

Check out our documentation for more detailed info on how to set up Cribl Search to query Google Cloud Storage buckets, or visit our YouTube Channel to learn what else you can do with Cribl Search — including How to Search Azure Blob Storage Containers and How to Search Amazon S3 Buckets.

 

.
Blog
Feature Image

Cribl: Empowering Data Freedom with Open Standards and Unmatched Flexibility

Read More
.
Blog
Feature Image

Hello Vegas! Cribl @ AWS re:Invent 2024

Read More
.
Blog
Feature Image

Simplify OpenTelemetry Metrics with Cribl Edge OTLP Conversion

Read More
pattern

Try Your Own Cribl Sandbox

Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.

box

So you're rockin' Internet Explorer!

Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari

Got one of those handy?