Time to GIT Serious: Automate Your Cribl Pack Deployments

Cribl Packs provide a powerful way to manage and reuse configurations across Cribl products. While it’s straightforward to build and manage Packs manually in the Cribl UI, many teams may want to version-control their Packs and automate deployments across multiple environments.

In this post, we’ll demonstrate how to use GitHub Actions from the HelloPacks to automate the deployment of a Cribl Pack through a typical Dev → Test → Prod pipeline. By the end, you’ll have:

• A GitHub repository storing your Cribl Pack.

• An automated CI/CD pipeline that:

  • Builds and validates your Pack

  • Deploys it to the Dev environment for initial checks

  • Moves to Test after passing validations

  • Finally promotes the Pack to Production when ready

Why Automate with GIT repositories?

• Consistency: Ensures the same deployment steps happen every time a push or pull request is made to test or production branches.

• Collaboration: Multiple developers can branch the same pipelines, routes, or knowledge objects while leveraging GIT reviewers upon pull requests to ensure that changes are reviewed before deploying to higher environments.

• Scalability: One push can trigger deployments to multiple Worker Groups or multiple Cribl environments.

• Standardization: Unified approach for Dev, Test, and Prod.

Prerequisites

• Cribl Stream Environments: You should have two or three Workspaces running Cribl. There are several ways to do this, you can choose the deployment method that works best for you. This tutorial uses a Cribl.Cloud Org which consists of the following:

  • Development/Test: One Workspace with multiple Worker Groups. The plan will be to use one Worker Group within this Workspace as dev and the remaining Worker Groups within this environment as test. An example workflow would be to develop in one Worker Group and when things look good push to test which is represented by the remaining 2+ Worker Groups within the same Workspace to ensure things are continuously looking good after a more general deployment.

  • Production: One separate Workspace within the same org with multiple Worker Groups. The plan will be to only deploy production ready Packs to this Workspace.

• GitHub Repo: In this tutorial we’ll be using a GitHub repository and Github Actions but the same automated pipeline principles apply to other GIT providers like GitLab or BitBucket.

• Credentials/Secrets: You’ll need some way to authenticate to each environment.

  • For each Cribl.Cloud Workspace you need a Cribl ClientID and Cribl Client Secret. Refer to the docs for how to obtain the Cribl.Cloud credentials.

• Github Personal Access token- in the case you are using a private repository

Pack repo with .github/workflows

If you haven’t checked it out yet refer to this post here, which provides a step by step tutorial on how you can add an existing pack into Git. The directory structure below represents an uncompressed Pack from the HelloPacks.crbl file with the addition of a .github directory. The .github directory contains the Github Actions configurations we will review in this post. The cribl.yaml file (.github/workflows/cribl.yaml) demonstrates these steps.

Workflow Overview

Here’s the general approach:

• Push to Dev: A developer commits and pushes changes to the dev branch in GitHub.

• Promote to Test: Developer merges dev to test via a push or pull request which triggers the Pack to be distributed to multiple wWorker Groups

• Promote to Prod: After final checks are confirmed within the test environment Worker Groups, a push or pull request will merge the test branch into main triggering the deployment of the Pack to the Production environment’s Worker Groups.

GitHub Actions Secrets and Variables

Cribl’s API endpoint requires users to generate an authorization bearer token before requests can be made. In Cribl.Cloud this is done with a Client ID and Secret. In Github, navigate to the Settings tab of your Pack repository, click Secrets and variables then click Actions.

Add the following secrets (use the exact case and spelling if you want to utilize the cribl.yml and install.sh script with minimal changes):

• CRIBL_CLIENT_ID

  • This is the Cribl Client ID needed to use the Cribl API endpoint

• CRIBL_CLIENT_SECRET

  • This is the Cribl Secret key needed to use the Cribl API endpoint

• REPO_PERSONAL_ACCESS_TOKEN

  • This is the Github personal access token that Cribl needs in order to install private repository packs from Git.

Click the Variables tab and create the following Variables. Be sure to structure the endpoint variable as follows: https://[YOUR_URL].cribl.cloud/api/v1/m (do not include a trailing /)

• PROD_ENDPOINT

  • This is the API url endpoint for your production environment

• TEST_ENDPOINT

  • This is the API url endpoint for your test environment

GitHub Actions Workflow

Below is an example workflow file (.github/workflows/cribl.yaml) that demonstrates these steps. You can tailor it to fit your team’s preferences—particularly around how you want to handle when Pack installs are deployed (via push/pull and based on branch name).

A summary of the workflow:

1. The developer pushes updates to dev as the Pack is revised in a separate Worker Group.

2. Once the Pack is ready to be promoted to test, the developer pushes or creates a pull request from the dev branch to test.

3. Upon merge to test, the workflow installs the pack on all of the Worker Groups listed in the test_worker_groups.txt file

4. Upon merge to main, the workflow installs the pack on all of the Worker Groups listed in the prod_worker_groups.txt file

# This is a basic workflow to help you get started with Actions
name: Push to CriblWorkerGroups
# Controls when the workflow will run
on:
# Triggers the workflow on push or pull request events but only for the "test and main" branch
push:
branches:
- test
- main
pull_request:
branches:
- test
- main
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# A workflow run of the jobs needed to deploy to Cribl Worker Groups and Workspaces
jobs:
# This workflow contains a single job called "deploy-to-Test" which deploys to Cribl Test Worker Groups
deploy-to-test:
if: github.ref == 'refs/heads/test'
# The type of runner that the job will run on
runs-on: ubuntu-latest
defaults:
run:
working-directory: .github/workflows
steps:
# Checks-out your repository under $GITHUB_WORKSPACE so your job can access it
- name: Checkout out the repo to the runner
uses: actions/checkout@v4
# Updates the permission of the script such that it is executable
- name: Make the script executeable
run: chmod +x install.sh
# Installs packs across worker groups
- name: Install pack across worker groups
env:
CRIBL_CLIENT_ID: ${{ secrets.CRIBL_CLIENT_ID }}
CRIBL_CLIENT_SECRET: ${{ secrets.CRIBL_CLIENT_SECRET }}
CRIBL_ENDPOINT: ${{ vars.TEST_ENDPOINT }}
CRIBL_WG_LIST: test_worker_groups.txt
PACK_REPO_HTTPS_URL: git+${{github.server_url}}/${{github.repository}}.git
PACK_REPO_SSH_TOKEN: ${{ secrets.REPO_PERSONAL_ACCESS_TOKEN }}
REPO_OWNER: ${{ github.repository_owner }}
REPO_NAME: ${{ github.repository }}
# Set the install type to override the install if a pack exists in a destination worker group
OVERRIDE: true
# Set the branch of the pack to push to the worker groups
PACK_BRANCH: test
# Set to true if using a public repo and false if using a private repo
PUBLIC_REPO: false
run: |
./install.sh
deploy-to-prod:
if: github.ref == 'refs/heads/main'
# The type of runner that the job will run on
runs-on: ubuntu-latest
defaults:
run:
working-directory: .github/workflows
steps:
# Checks-out your repository under $GITHUB_WORKSPACE so your job can access it
- name: Checkout out the repo to the runner
uses: actions/checkout@v4
# Updates the permission of the script such that it is executable
- name: Make the script executeable
run: chmod +x install.sh
# Installs packs across worker groups
- name: Install pack across worker groups
env:
CRIBL_CLIENT_ID: ${{ secrets.CRIBL_CLIENT_ID }}
CRIBL_CLIENT_SECRET: ${{ secrets.CRIBL_CLIENT_SECRET }}
CRIBL_ENDPOINT: ${{ vars.PROD_ENDPOINT }}
CRIBL_WG_LIST: prod_worker_groups.txt
PACK_REPO_HTTPS_URL: git+${{github.server_url}}/${{github.repository}}.git
PACK_REPO_SSH_TOKEN: ${{ secrets.REPO_PERSONAL_ACCESS_TOKEN }}
REPO_OWNER: ${{ github.repository_owner }}
REPO_NAME: ${{ github.repository }}
# Set the install type to override the install if a pack exists in a destination worker group
OVERRIDE: true
# Set the branch of the pack to push to the worker groups
PACK_BRANCH: main
# Set to true if using a public repo and false if using a private repo
PUBLIC_REPO: false
run: |
./install.sh

Walkthrough

1. Merge pull request to test from dev branch

2. On the actions tab you can select the job and view the workflow that was triggered

1. Clicking into the deploy to test job, you can view all of the steps that were run including the verification that the script successfully deployed the Pack to the Test Workspace

1. In Cribl, you can also verify the Packs are now in the test workspace’s Worker Groups

Worker Group 1

Worker Group 2

1. Merge pull request to main from test branch and the same process from steps 1-4 applies.

Summary

By leveraging GitHub Actions, you can streamline the Dev → Test → Prod lifecycle for Cribl Packs. This approach provides:

• Consistency: Ensures the same deployment steps happen every time a push or pull request is made to test or production branches.

• Collaboration: Multiple developers can branch the same Pipelines, Routes, or Knowledge Objects while leveraging GIT reviewers upon pull requests to ensure that changes are reviewed before deploying to higher environments.

• Scalability: One push can trigger deployments to multiple Worker Groups or multiple Cribl environments.

• Standardization: Unified approach for Dev, Test, and Prod.

Give it a try in your own environment, and watch as your Cribl Pack changes move smoothly from Dev to Prod, saving you time and reducing deployment friction!