Migrating data from QROC with a custom Python script

Migrating SIEMs can be challenging and sometimes very difficult depending on the SIEM provider. In the case of IBM QRadar On Cloud (QROC), there are limited options to migrate historical data to other destinations. Many enterprises face the difficulty of migrating large datasets, especially if the data lives in proprietary formats. Recently at Cribl, we partnered with a large fortune 500 customer to address this exact scenario, building a custom Python script (co-authored by myself and Nate Zemanek) to facilitate data migration from QROC using a Cribl hybrid worker. This blog post summarizes the use case, outlines the process, explores the capabilities and options of the script (including Your Mile May Vary "YMMV" or "It Depends" factors), and shares lessons learned that might help others considering a similar migration path.

The Use Case: Why Migrate Away from QROC?

Many organizations find themselves wanting to switch SIEM platforms but are faced with a key obstacle: How do you move old logs, especially at scale (~TBs of data) without manual UI exporting, and route them to a new destination of your choice—whether it’s a new SIEM or a data lake like Cribl Lake or object storage.

In the original exercise with the customer, our main objectives were:

• Export a significant batch of historical QROC logs—roughly 50TB —into Cribl Lake for compliance, analytics, and long-term retention.

• Demonstrate flexibility, choice, and control so this strategy could serve a variety of destination systems and future data needs.

• Prove technical feasibility in an environment where "there's no way to get data out" is common lore.

The Script: How It Works

At its core, we built a Python script designed to hit QROC's APIs, retrieve logs over a user-defined period, and stream them out over HTTP (or other preferred methods) to a destination endpoint. The script is intended to run on a hybrid worker node—either connecting up to a Cribl Cloud leader or on-prem leader—ensuring secure, local control and maximizing transfer speed.

Most times collecting data from APIs can be achieved with Cribl’s REST Collector, however we were unable to use this for this use case due to the nature of the QROC API.

Key Technical Features:

• Parameterization: The script supports parameters for API token, start time, end time, destination host, port, and even a --zip option for compression.

• Multi-Threading: Empowers users to parallelize requests and handle much larger volumes of data quickly, providing near-linear speedups as additional threads are added.

• State Tracking: If the script crashes or needs to be stopped, it can resume where it left off, minimizing lost work and the need for restarts.

• Query Customization: Most parameters are flexible, but the actual query sent to QROC may need manual edits based on specific requirements.

• Destination Flexibility: While originally tested to send data to Cribl via HTTP locally, it's destination-agnostic; with minor tweaks, you can route logs to almost any endpoint supported by Cribl Stream.

The Migration Process: Step-by-Step

• Pre-flight Checks: Validate API access and permissions with QROC, ensure hybrid worker deployment is functioning, and estimate data volumes.

• Script Setup: Configure required script parameters according to environment, destination, and time windows. For example: python qroc_api_pull.py --api-token "..." --start-time "YYYY-MM-DD HH:MM:SS" --end-time "YYYY-MM-DD HH:MM:SS" --destination-host "localhost" --destination-port 8088 --num-threads 5 --zip

• Test Runs: Begin with small-scale tests (a few hours of logs) to validate throughput, API stability, and destination connectivity. Adjust num-threads and query block size as needed.

• Bulk Migration: Once tests are successful, expand to larger periods/data volumes. Monitor thread count, CPU resources on the hybrid worker, and network bottlenecks; ramp up hardware if needed.

• Post-migration Validation: Confirm data integrity and completeness at the destination; use Cribl Stream's capabilities for further transformations or enrichment as desired.

Options, Gotchas, and "YMMV"

Doing this sort of migration isn't perfectly turnkey. Here are some key variables to consider:

• QROC vs. On-Prem QRadar: The script was designed for QROC, but could be modified for on-prem installations. IBM's APIs may differ and, surprisingly, on-prem is reportedly more performant, although endpoint paths and authentication can vary. Always check API compatibility before starting.

• API Rate Limits & Scaling: Multi-threading greatly boosts throughput, but QROC's API can impose rate limits (at the time we received limited info regarding the QROC API). We advise customers to start small and scale cautiously—what works for one environment might fail for another.

• Script Location: For ultimate control, run the script on a dedicated server (the hybrid worker). For smaller tasks or where infra is limited, it may be possible to run locally on a laptop, but this isn't practical for very large exports.

• License Considerations: When migrating large historical volumes, be aware of licensing impacts—not all SIEMs bill gracefully for bulk transfers.

"It Depends" Scenarios:

• API endpoint differences between cloud and on-prem generations.

• Network bottlenecks, especially when exporting massive volumes over WAN/internet.

• Policy constraints: Some enterprises may limit bulk exports due to compliance, egress costs, or risk management.

• Post-export destination: Data replay and ingestion rules, enrichment pipelines, and retention policies can all introduce complexity downstream.

Real-World Impact and Outcomes

The exercise with our customer was a success. Once the customer saw the flexible, repeatable pattern for extracting data from QROC, they were empowered to handle the process themselves with minor tweaks.

If you're considering a similar migration:

• Keep your migration modular and parameterized.

• Plan for both technical and business "gotchas"—the process may be seamless for some, bumpy for others depending on scale.

• Always test before scaling and be creative; the only real "roadblocks" are those imposed by platform APIs and enterprise policies.

Wrap Up

Migrating historical data out of QROC (or similar SIEM platforms) is more than possible—it's practical with the right scripted API calls, a robust hybrid Cribl worker, and thoughtful planning. Custom python scripts like the one developed with this customer deliver flexibility, scale, and resilience, letting you route legacy logs wherever you need. The process is not perfectly uniform; sometimes “it depends” on your environment, requirements, and platform quirks. Customers moving off other SIEMs often use Cribl’s pull collector sources, such as the S3 Collector, to simplify and accelerate migration into their new SIEM. By combining careful testing with Cribl’s modular design, organizations can turn the challenge of migrating historical SIEM data into a predictable, manageable task.