“ Shift left…”,

“Quickly scale your security team…”,

“Fix issues before production…“,

“Reduce time and money spent on Security…”

Wait! We’ve seen these promises somewhere before, haven’t we? Was it in the subject line of our never-ending spam folder, filled to the brim once again with some poor soul slinging the latest and greatest blockchain-driven, AI-enhanced Security tool or service? No, it can’t be that; it must have been one of the talks I saw at SuperSecureCon or DarkHat 2024. No, not it either; perhaps that was from the clickbait title of the YouTube creator we follow doling out Security wisdom in oh-so-digestible 13-minute clips?

Imaginably, each could be true. If you’ve spent any time in the Information Security industry, you’ve surely stumbled upon these claims before. In keeping with such traditions, I’m here to tell you how a successful Security Champions program can live up to each of those claims by getting ahead of cross-functional friction.

First, entertain me with our problem statement:

“Security Teams simply don’t scale at the same rate as the rest of the business, specifically when compared against sales, product, and engineering teams.”

It’s a common reflection here at Cribl that the size of data an organization generates to meet its business goals grows exponentially over time, and yet the budgets associated with managing this data rarely see such correlated investment. That’s why Cribl built Stream and Edge to solve this real-world customer problem. Directly related to that, we’ve observed the rapidly growing importance of securing organizational data in an ever-evolving threat landscape while lacking the same pace of scaling of investment in Information Security. It’s common for organizations to have a single Security Engineer for every 100 Software Engineers, many times even less (though I can proudly attest this is not the case at Cribl).

So, therein lies our problem statement:

How can an organization deliver secure, world-class software that solves customer problems while lacking the dedicated internal Product Security headcount and resources?

The solution seems simple, and we as an industry have attempted to embrace the adage “Security is everyone’s Job” with varying degrees of success. However, this adage alone hasn’t led to revelatory improvements in Security shifting left because of the various reasons that friction has existed between Engineering and Security teams.

Traditionally, security teams have had the unfortunate reputation of being the department of “no” and as a source of friction and delay. It’s no wonder that without healthy cross-functional collaboration between Security and Engineering, this perception will persist. Part of this perception is misaligned incentives, but I believe another part is a history of security knowledge being held behind closed doors, inaccessible to the folks with a vested interest and opportunity to benefit the most.

At Cribl, we recognized early on that with our pace of accelerated growth; we wouldn’t be able to scale security efforts without shifting left and enabling our engineers to identify and resolve issues independently before the implementation had even begun. We wanted to ensure interested Engineers were unrestricted in their opportunity to learn, partner, and engage with Security, so Cribl’s Product Security Team kicked off a Security Champions program. If Engineers are missing the important context and understanding of Security concepts introduced to them as blockers to their progress and team velocity, it’s no wonder there’s been a history of friction when remediating otherwise preventable vulnerabilities. Recognizing this, we have the opportunity to focus our Security Champion’s program’s activities on reducing this friction and ensuring our Engineers are interested and invited to learn about Security issues.

It is increasingly common that organizations require mandatory secure development training; however, it doesn’t have to breed discontent and contempt by missing the developer audience’s needs and the teams’ individual challenges. While we all agree the OWASP Top 10 has moved the industry forward, the specificity of each organization’s needs and goals makes it challenging for this training to feel relevant to engineers. One of the greatest benefits of having an in-house Security Champions program is the ability to tailor content and sessions toward the areas and topics of security that benefit your organization best.

At Cribl, while we are lucky to have engineers with deep experience in the Security industry, we still recognize that mandatory participation would greatly limit the long-term success of the program, and we have, since the earliest days of the program, encouraged participation voluntarily. While Cribl’s initial rollout of the program was limited to a dozen individuals, each has had a particularly keen interest in learning about the topics covered. By kicking off the program with the most invested and interested individuals on board, we’ve not only shifted security left but also towards improving security posture; more importantly, however, we’ve exchanged a source of friction for the opportunity to facilitate honest engagement and reward interest on a topic that often induces tension. As Cribl has grown and matured, so has its Security Champions program, which now boasts a membership of over 120 goats in a wide range of disciplines from Sales and Marketing to Engineering and Product Leadership. Growing beyond its initial focus on engineering has enabled the program to impact how we think and discuss security collectively, enabling many goats to consider security concerns in their daily activities.

As engineers, we recognize the value of automation: offloading mundane, repetitive tasks when possible to tools and systems that increase efficiency and free us to focus on more important tasks. This is just as true in modern security teams, where we’ve also embraced our suite of tools focused on alerting us to security issues where they would otherwise go unnoticed. These tools solve real problems and help us scale our impact to organization security posture, but they can also be a huge source of cross-functional friction. It’s not uncommon for organizations to integrate security tooling into their build and deployment pipelines; yet while this has obvious benefits, it can also introduce increased build time and, depending on configuration, introduce quality gating and merge blockers.

A deadline-focused Engineer, ill-equipped to contextualize the difference between a reachable critical vulnerability and an unreachable theoretical attack vector, is not only hindered by the introduction of this automation but sees little benefit in the absence of the ability to contextualize and interpret the severity of the finding. At Cribl, we’ve focused on ensuring that the tools we integrate into developer workflows and pipelines are tools these users can understand and use. We use our Security Champions program to create content and conduct enablement sessions, allowing Engineers to not only interpret findings but also write custom rules to suit their own use cases. By educating engineers, we’re ensuring the integration of security automation reduces friction and provides cross-functional value, resulting in a larger return on investment.

As Cribl has seen growth, so has our Security Champions program: from the early days with a dozen initial participants to its current state with formalized Lead Champions, hosting deep dives, back-to-basic series, and even a couple of internal CTFs. It’s hard to argue with the numbers. On more than one occasion, we’ve covered specific vulnerabilities, which led to the identification of various related critical findings by participants in the following weeks. We’ve seen the program scale from the initial dozen participants to the #security-champions channel, now boasting a membership of nearly a fifth of the entire company. With a library of recordings and write-ups, we’ve ensured that folks who are unable to attend still have the ability to participate asynchronously.

Ultimately, the Security Champions program has been about shifting left, but that doesn’t happen without focusing on the areas of friction challenging cross-functional collaboration. I often reflect on the phrase repeated by one of the founders and CEO:

”Software is a people business” – Clint Sharp

It’s a truth that resonates in so many different ways; specifically, when we think of building software securely at scale, teams of people collaborate to facilitate that kind of build. Reducing the cross-functional friction between security and engineering activities by truly investing in a Security Champions program has allowed us to tailor content to our organization’s needs as we grow and face new challenges. You’ll know your program is a success when you start hearing feedback like:

“…I feel like you guys have made security cool, I’m not used to that”. – Jenn M.

Additionally, you get the opportunity to reflect on the reach of your program when your CEO participates in your Security CTFs.

Software is a people business and Security is everyone’s job. By investing in a Security Champions program, you’re not only investing in the people behind the software; you’re slowly transitioning the various sources of friction into opportunities to build a cross-functional educational experience.

Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.

We offer free training, certifications, and a free tier across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started and continue to build and evolve. We also offer a variety of hands-on Sandboxes for those interested in how companies globally leverage our products for their data challenges.