The Quixotic Expedition into the Vastness of Edge Logs, Part 1: Analyzing Numerous Cribl Edge Nodes with Cribl Search

Cribl Search is a powerful tool that is designed to enhance your data search efficiency, irrespective of the location of your data. This blog will explore how this tool seamlessly integrates with numerous Cribl Edge Nodes in real time, simplifying the process of discovery and troubleshooting. An integral part of Cribl Search is the “teleport” feature, which enables users to access specific Edge Nodes for in-depth analysis, simply by clicking on a host field. All of this can be achieved without any need for complex ETL processes, unnecessary data movement, or time-consuming pre-processing.

Let’s dive in.

Use Case 1: Analyzing SSH Login Attempts in System Logs Across Multiple Edge Nodes

Scenario: You suspect unauthorized SSH login attempts on your Edge Nodes and want to examine the auth.log files for any suspicious activity across your entire network.

dataset="cribl_edge_system_logs" source=*auth.log sshd user 
| limit 1000

This query searches the cribl_edge_system_logs dataset for events containing sshd and user terms in auth.log files. It returns up to 1,000 results, enabling you to analyze login attempts and detect any unauthorized access. Once you identify an issue, you can teleport to the specific Edge Node to take appropriate action.

Use Case 2: Tracking Memory Usage Across All Edge Nodes

Scenario: You want to monitor memory usage across your vast network of Edge Nodes to optimize performance and identify potential issues.

dataset="cribl_edge_metrics" node_memory_Used_percent | summarize avg(node_memory_Used_percent) by host

In this query, we’re scanning the cribl_edge_metrics dataset specifically for data related to used memory percentage (node_memory_Used_percent). The search calculates the average percentage of memory used by each host and groups the results by host.

By monitoring average memory usage, you can identify hosts that consistently utilize a high percentage of memory – gaining invaluable insights for capacity planning, load balancing, and troubleshooting performance issues.

Use Case 3: Identifying High Network Traffic Events Across Thousands of Edge Nodes

Scenario: You want to identify high network-traffic events across your extensive network of Edge Nodes to detect potential issues and ensure optimal performance.

dataset="cribl_edge_metrics" node_network_receive_bytes_all_total>1000000 and node_network_transmit_bytes_all_total>1000000 
| timestats sum(node_network_receive_packets_all_total), sum(node_network_transmit_bytes_all_total) by host

In this query, we are searching the cribl_edge_metrics dataset for network traffic data. The search calculates the total bytes received and sent by each host (node_network_receive_bytes_all_total and node_network_transmit_bytes_all_total), and then filters the results to display only those hosts with more than 1,000,000 bytes (1 MB) of traffic.

Identifying high network-traffic events enables you to proactively address potential performance issues. With the teleport feature, you can instantly access the specific Edge Node of your choice, to investigate further and resolve any issues.

Stay tuned for the second part of this post, where we delve deeper into the investigation of unauthorized login attempts, and explore building a personal threat-intelligence solution using lookup tables. If you’d like to try out Cribl Search, you can create a Cribl.Cloud account to get started for free.