Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and central access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›Cribl Copilot gets your deployments up and running in minutes, not weeks or months.
Learn more ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›Brendan Dalpe is the Chief Technology Officer and Co-Founder of SOI Solutions. Prior to h... Read Moreis role at SOI Solutions, he spent three years at Cribl in different roles including Solutions Architecture and Technical Marketing. Brendan has a strong background in technology, with specializations in security, cloud, and containerization. Read Less
It’s time to throw another birthday party for the Cribl Sandbox. The platform turns 4 on Nov 25! 🎉 Crazy how the time flies. Take a trip down memory lane with us as we recall how we leveled up the Sandbox platform after our earlier challenges. This is the first in a series of (upcoming) blogs detailing the Sandbox architecture.
Here’s the first git log from our Cribl Sandbox repo:
brendan cribl-sandbox % git log $(git rev-list --max-parents=0 HEAD)
commit de467bee72cf45d69f8c6e3e5d638d2b1ff72d9b
Author: Clint Sharp <redacted>
Date: Mon Nov 25 12:13:25 2019 -0800
initial commit
Since we launched, things have definitely changed. Initially, we compiled all Sandbox assets into a single Docker image and ran everything on the Amazon ECS Fargate platform. It was revolutionary. It was amazing. Sandboxes, however, took 5-10 minutes to start from cold boot. DNS issues plagued us. Ultimately, it wasn’t the ideal user experience we envisioned.
At this point, we were trying various things to improve speed, performance, and the internal developer experience. Somebody threw out Kubernetes, and we were off to the races rearchitecting the platform. The platform runs on a few EC2 instances that comprise the EKS cluster.
Our goal with the Sandbox platform is to keep the barrier to entry low. What’s the point of making your product hard to use and learn? Unfortunately, this mindset comes at a price in terms of security. Believe it or not, you could launch a Cribl Sandbox by just putting any email address in the form two years ago! And this led to the crypto mining bots taking advantage of the Sandbox platform. (Egress traffic filtering, what’s that?) After racking up a five-figure Sandbox bill in one month, we implemented an email-based one-time passcode system to ensure that email addresses were valid and that the requester was actually human.
Did you know that a six-digit number can generate one million codes? We’ve found this to be pretty effective at keeping the riffraff out of the Sandbox. Along with some pretty strict firewall rules and a WAF to prevent misbehaving folks from accessing the API.
So, how do you quickly implement an email verification system? Use commercial software that does this for you! As we were building Cribl.Cloud, we decided to use their SendGrid account.
We layered Twilio into the mix, et voilà, verification codes were sent and verified before the Sandboxes were started.
For historical reference, this is what Sandbox verification emails looked like:
Functional, but not elegant. I must lean into the screen to read that code even with my glasses on. Using Twilio and Sendgrid also created additional headaches for the Technical Marketing Engineering (TME) team. As a separate organization from Cribl Engineering that develops software and ships Cribl and the Cribl.Cloud platform, we (TME) don’t have administrative access to SendGrid and Twilio. It was nearly impossible to troubleshoot email delivery issues, and users were frustrated when they didn’t receive codes. Twilio also charges $0.10 per verification code. In the grand scheme of things, this isn’t a huge amount, but it starts to add up every month with hundreds of Sandbox users per week.
So we asked ourselves, what if we could do it better, more cost-effectively, and improve visibility into the email delivery process simultaneously? Time to level up!
We built our verification code system from the ground up using: Amazon’s Simple Email Service (SES) for email delivery, DynamoDB for temporary code storage, and nodemailer to format and send emails to SES. And with the magic of editing, it only took a few minutes!
The benefit of switching to nodemailer is that we have complete control over the message template. We took this as an opportunity to improve the aesthetics and usability of the message. Here’s what an email looks like today:
Much improved! Cribl branding, check. Readable code, check. Email address for help, check. Moving to SES, we can now obtain feedback notification messages through SNS topics. We’ve configured the SNS topic to invoke a very simple Lambda function that takes the feedback data and writes it to a new object in an S3 bucket. Feedback data is stored in an ordered prefix structure:
/ses/${domain}/${year}/${month}/${date}/${hour}/${notification_type}-${epochmillis}.json
With the introduction of Dashboards in Cribl Search this summer, it’s never been easier to visualize data. Our TME team used this feature addition to build a simple dashboard providing near-real-time visibility of email code delivery from Amazon SES. We’ll show you what that looks like below. With our data stored in S3 as JSON objects, all we needed to do was add a new Data source with the correct bucket path token syntax:
And with that, it is as easy as now writing some searches! Here are the actual queries powering the dashboard we built:
Total number of messages: dataset="cribl_sandbox_email_logs" domain="sandbox.cribl.io" | summarize count=count()
Total number of messages by type: dataset="cribl_sandbox_email_logs" domain="sandbox.cribl.io" | summarize Status=count() by notificationType
Top delivery domains: dataset="cribl_sandbox_email_logs" domain="sandbox.cribl.io" notificationType="Delivery" | extend recipient=delivery.recipients.0 | extract source="recipient" type=regex @'@(?<recipient_domain>.+)$' | summarize count=count() by recipient_domain | top 10 by count
Get bounce reasons: dataset="cribl_sandbox_email_logs" domain="sandbox.cribl.io" notificationType="Bounce" | extend recipient=mail.destination.0, reason=bounce.bouncedRecipients.0.diagnosticCode | project recipient, reason | render table
All put together, it looks like the screenshot below. We’ve even taken advantage of the new Dashboard Inputs functionality which makes inspecting different time ranges and data sets easy peasy lemon squeezy.
From zero visibility to full visibility with Cribl Search. 🙌 Our data has helped us identify that multiple users are receiving verification codes, but their email filters are blocking them. Bounced messages are logged with full explanations from SES, which allows us to reach out to users proactively instead of waiting for complaints.
Oh, and this whole setup costs us less than $0.25 per month to run. Excelsior!
We’ll be back with more blogs detailing the Sandbox architecture. We may be a little biased, but we think it’s a pretty neat platform. How many other companies encourage you to tool around in fully functioning free environments? There’s so much to cover, though, we’ll need… an entire series!
If you haven’t tried out our Sandboxes, or it’s been a while since you’ve taken a look, we encourage you to give them a try. We’re constantly updating them with new content as we add features to the Cribl product suite. Join us on our Community Slack if you would like to share your experience with our Sandboxes!
Sign up for a free Cribl.Cloud account and let us help you get control of and better visibility from your data – no matter what data store or format it’s in!
Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.
We offer free training, certifications, and a free tier across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started and continue to build and evolve. We also offer a variety of hands-on Sandboxes for those interested in how companies globally leverage our products for their data challenges.
Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?