Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and centralize access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
April 24 | 10am PT / 1pm ET
3 ways to fast-track your data lake strategy without being a data expert
REGISTER ›Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›AWS CloudTrail is a service that “enables governance, compliance, operational auditing, and risk auditing of your AWS account.” It continuously monitors accounts and it is one of the most valuable and probably the best data source for security analysis in AWS. Security and Operations teams love Cloudtrail because of the added visibility into user and resource activity across their AWS infrastructure.
Getting ahead of a potential compromises and being able to respond quickly is the name of the game when providing operational security. There are several industry-standard documents that outline how to best prepare for securing an AWS accounts. A very well-known one is the CIS Benchmark document which includes many best-practice security recommendations ranging from properly configuring IAM, to enabling Logging, to Monitoring etc. CloudTrail plays a central role here and its data can help reveal potential security gaps, assist in detecting early signs of compromise and help you stay compliant with your security policies and requirements. Therefore, extracting these insights as close to real-time as possible becomes a security necessity. This is where Cribl helps. With Cribl you can ingest CloudTrail logs natively using the Serverless Forwarding App and in real-time you can:
Once data has been categorized and checked/scored it can be sent downstream to analytics platforms like Splunk, Snowflake, Elastic, Microsoft Sentinel etc.
Let’s dive in and see how to collect, filter/route, categorize and check/score CloudTrail data with Cribl.
First, if you haven’t already done so, see our previous blog post: Serverless data forwarding to Cribl for AWS Services for a detailed walkthrough on how to collect CloudTrail data serverlessly.
Next, let’s de-noise our events so that we have a lean stream of data to work with. Everything is way more efficient and faster when working with clean data.
eventName
that fit this criteria: /(^Describe|^List|^Get)/.test(eventName)
. These voluminous, read-only events tend to be not that useful in security analysis. For example, here are top 10 eventNames from our sample/lab environment.eventName count ------------------------------ DescribeTargetHealth 311198 DescribeInstances 263879 DescribeRouteTables 263072 AssumeRole 149884 DescribeInstanceStatus 113441 DescribeServices. 87955 DescribeTable 86896 GetBucketLocation 44775 DescribeVolumes 44067 GetBucketEncryption 43641
An alternative to fully dropping events is to use the Dynamic Sampling or Suppress functions that allow some events through over a period of time.
Note: Additional trimming happens below.
Yet another alternative to dropping or suppressing events is routing them to different locations. E.g. you can send full fidelity stream to archive while sending an important subset to an analytics platform.
Not all CloudTrail data is created equal. Even when filtering or suppressing read-only events, there is still a lot left to sift through. One way to address this is to introduce categories for important or notable events. Think of this step as prep work that will pay dividends in the future; it will make your analysis easier and response faster.
Here we will use a lookup table to categorize events – in real-time – with labels that best describe the context in which that activity belongs to. For instance, events that indicate security-relevant S3 bucket configuration changes are labelled with S3BucketActivity
. Sample association table:
eventName,securityEventCategory ConsoleLogin,LoginActivity PutBucketAcl,S3BucketActivity PutBucketPolicy,S3BucketActivity ... ReplaceNetworkAclAssociation,NetworkACLActivity CreateCustomerGateway,GatewayActivity ... StopLogging,CloudTrailActivity DeleteGroupPolicy,IAMPolicyActivity
The context and what’s important naturally depends on threat model and from one account to another. The lookup table approach accommodates this by allowing editing. You can add/remove/change the categories per your needs.
To label the data, we use a Lookup function that enriches each event with a securityEventCategory
field if a matching eventName
is found.
CloudTrail events may contain other indications of security-relevant activity that cannot be done with a Lookup function. This needs conditional checking across multiple fields. For instance, checking for console authentication failures requires to evaluate eventName
and errorMessage
: eventName=='ConsoleLogin' && errorMessage=='Failed authentication'
Example checks:
To do this an Eval function is used to assign the securityEventCategory
field an appropriate value if a check returns true. The value can be a label (string) or, if a scoring model is introduced, a number. Numerical scoring is useful when further statistical analysis is done in downstream analytic platforms or when filtering is needed avoid alert fatigue.
With Cribl these checks can be articulated effortlessly given the expressibility of the language. In fact, most of the CIS Benchmark document recommendations that deal with CloudTrail events can be codified easily.
Some CloudTrail events contain fields with enormously sized payloads. For example, the sessionToken
value in AssumeRole
events is close to 1KB! With Cribl you can be very surgical and target these fields independently – we can either toss it out or trim it down. We can do this with the Parser function in reserialize mode; it has the capability to remove/keep any fields before converting the JSON object/event into a string and sending it downstream.
CloudTrail logs are critical for operating your AWS infrastructure. With added context and checks they can become invaluable for security operations. Cribl makes it super easy to bring in additional insights in real-time and enable you to get ahead of potential compromises, decrease response time and ultimately improve your security posture.
If you’d like to work with these configurations get them for free in GitHub.
If you’d like more details on configuration, see our documentation or join us in Slack #cribl, tweet at us @cribl_io, or contact us via hello@cribl.io. We’d love to help you!
Enjoy it! — The Cribl Team
The fastest way to get started with Cribl LogStream is to sign-up at Cribl.Cloud. You can process up to 1 TB of throughput per day at no cost. Sign-up and start using LogStream within a few minutes.
Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.