August 3, 2021
Are you looking to correct misconfigured timestamps from logs in your Security Incident and Event Management tool? Let’s say you’ve just received a critical security alert from the SIEM tool regarding a brute-force attack that occurred on July 19, 2021, at 11:08AM. The cryptic event looks like this:
The Incident Response team was dispatched to investigate, but they can’t find any correlating events. Over the next few days, the SIEM continues to generate new security alerts, and management is now highly concerned that they will be on the front page of the Wall Street Journal as the next ransomware victim.
As their newest security analyst, you haven’t had time to carefully review the aging SIEM deployment yet, but your gut instinct triggers an overwhelming sense of dread about the root cause of the issue.
How would you feel if I told you that this event actually occurred on July 21, 2019?
Looking at the timestamp again,
"timestamp":"19-07-21 11:08:21.123", can you tell if the timestamp is formatted in day-month-year or year-month-day style?
This missing fact was revealed only after speaking with the application security team and reviewing current logs as they were captured in LogStream. The bottom line: You’ve got to get to work correcting and fixing misconfigured timestamps.
Is this timestamp in a day-month-year or year-month-day format?
Check out these date formats by country to demonstrate how challenging timestamp recognition can be. Let’s review a few approaches where LogSteam can assist with correcting misconfigured timestamps.
One of the first steps of processing, when an event arrives in LogStream, is to use the configured Event Breakers to determine event delineation and extract the correct timestamp. Sometimes custom event timestamps are received, and LogStream needs additional configuration to properly choose the real timestamp format. Event breaking is one of the first places that the correct timestamp can be set on an event.
Additional functions such as Auto Timestamp, and LogStream internal methods like C.Time.strptime(), can also help, especially if time has different locations and formats within the events. You can even see event output with the data preview. Let’s walk through a few ways that LogStream can help correct misconfigured timestamps.
You begin by inserting the time-challenged event as sample data, using LogStream’s Data Preview.
Once you paste the event, LogStream attempts to recognize and highlight the timestamp in purple.
Since the timestamp strptime format is not yet defined, LogStream used the current time for the event timestamp on the left side of the event preview.
When LogStream processes events in the Event Breakers, it must first locate a timestamp anchor. From there, the engine will try to do one of the following:
The closer an anchor is to the timestamp pattern, the better the performance and accuracy, especially if multiple timestamps exist within an event.
For the manually supplied option, the anchor must stop immediately before the timestamp pattern begins. Let’s define the timestamp anchor and strptime format of this event.
Set the timestamp anchor to
"timestamp":". The anchor will then be highlighted in blue in the preview pane. Regular expression syntax is supported in the Timestamp Anchor field.
The event from 2019 was showing up in the year 2021 because no one had applied a proper locale, such as a year-month-date format, to the timestamp. This mistake will certainly cause a misconfigured time stamp
The strptime reference provides a format of
%y-%m-%d %H:%M:%S.%L that must be added to the Timestamp Format > Manual Format field to correctly process the timestamp according to the locale we expect.
Since the timestamp format is now defined, LogStream extracts the proper timestamp from the event, placing the original event’s correct time on the left side of the preview.
If you find that the time of the event does not match the timestamp during a sample data preview, you can easily remedy the issue using various timestamp functions in LogStream pipelines. When we resort to the functions below, let’s assume that the Event Breakers did not already correct the misconfigured timestamps.
Once you add the Auto Timestamp function to the pipeline, create a regular expression capture group that matches the time in the event, and add the strptime format of
%y-%m-%d %H:%M:%S.%L. Notice that the event timestamp on the left is now correct. The capture group in the Regex field must only extract the timestamp you will apply
strptime against, and nothing else. Here is an excellent blog on this function: Using the Auto Timestamp Function in Cribl LogStream.
C.Time.strptime(_raw.timestamp,'%y-%m-%d %H:%M:%S.%L').getTime() / 1000
Once you identify and resolve the timestamp-related issues, you can choose to add these timestamp configurations to custom Event Breaker rulesets. We’ll cover Event Breaker rulesets in an upcoming blog.
If you want to level up your skills and learn how to fix your timestamps, try out our free Sandbox and brag about your certifications online! If you need to correct misconfigured timestamps in your own environment, process up to 1 TB of throughput per day at no cost using Cribl.Cloud.