When monitoring your infrastructure, applications, users, devices, sensors etc. you’re likely to be using a variety of data sources and a number of monitoring tools to get the full picture. Oftentimes data is messy and chances are that you have instrumentation measurements embedded into log lines. That’s not necessarily a bad thing, but if you care principally about metrics those values can be extracted and send to a metrics store instead of a logging system. Metrics stores tend to be way leaner in terms of storage requirements and offer better analysis experience and vastly faster performance.

This is a no-frills, no-BS post on how you can use Cribl Logstream to visually author conversion of logs into metrics.

Let’s define a metric

A metric at its core is a record/event that contains a name, a value, a timestamp and one or more dimensions. The name is the name of the metric of interest, the value is its value at the given timestamp and dimensions represent metadata about the metric (i.e. groups that will be used to aggregate the values by). Here’s a meta-example:

timestamp:1565558300 | name:net.bytes | value:1024 | dimensions:OU=BigCo,Location=US,Region:us-east-1,AZ=us-east-1d

Steps to convert logs to metrics

1. Extract fields of interest using Regex Extract or Parser functions.
2. Select fields that will become metrics and dimensions using the Publish Metrics function

Example with VPC Flow Logs

VPC Flow Logs capture information about the IP traffic going to and from network interfaces in your AWS VPC. Flow log data looks like this:

2 496698333333 eni-0538333333deafbeef 10.0.2.112 10.255.81.70 8000 41056 6 10 4133 1565552093 1565552110 ACCEPT OK

version account_id interface_id srcaddr dstaddr srcport dstport protocol packets bytes start end action log_status

That’s it!!

If you’d like more details on how to get started, head over to our documentation or join us in Slack #cribl, tweet at us @cribl_io, or contact us via hello@cribl.io. We’d love to help you!

Enjoy it!  — The Cribl Team