At Cribl, one of our core values is Customers First, Always. This means that for every release, we ensure our software meets the highest standards for reliability and performance. As part of this process, we run extensive regression and performance tests, and if we detect any unexpected behavior, we investigate the root cause, resolve it, and, whenever possible, contribute our lessons learned to the community.
Understanding Higher RSS in Node.js 22
As we prepared to upgrade from Node.js 20 to 22.14.0, many of our memory performance tests began showing unexpected regressions. This prompted a deeper investigation into Node.js and V8 to understand whether these changes would affect our products or customers.
Our investigations revealed that in Node.js 22, V8 changed how it handles free memory pages. Instead of unmapping them immediately, it now caches free pages until an idle period or a memory-reducing garbage collection cycle occurs. During worker node startup, garbage generated increases both heap and resident set size (RSS), and because our performance tests apply maximum CPU pressure from the beginning, the workers have no idle time for V8 to release memory back to the OS. The result is a higher RSS, even though heap usage eventually stabilizes. This behavior is intentional—V8 avoids expensive memory system calls to optimize throughput. When memory pressure exists, either through available system constraints or V8 flags, V8 releases memory as expected. The higher RSS in Node.js 22 is therefore the result of performance optimization, not a regression or leak.
How We Verified the Root Cause and Its Impact
To confirm this behavior, we ran controlled experiments and monitored memory usage. Introducing short idle periods before generating load allowed V8 to release cached free pages, reducing RSS. Applying memory pressure or adjusting heap limits produced similar results. We also temporarily patched V8 to force early memory release, which significantly decreased the RSS in the performance tests, confirming that the elevated RSS was due to V8’s caching strategy rather than any change in our product’s memory requirements.
The outcome was reassuring: our workloads showed no leaks, regressions, or throughput issues. Higher RSS appears only when the runtime has available memory and sustained CPU activity—conditions under which V8 optimizes for performance. In typical usage, streams with occasional idle time naturally release memory back to the OS, and even under constant load, memory usage remains within expected limits.
Our Contribution to the Node.js Community
As we embarked on upgrading from Node.js 22.14.0 to 22.17.1, we faced a new challenge. This issue was separate from our previous memory investigations.
We observed that some of our performance tests showed a throughput reduction of around 10%. After analyzing the upstream commits between Node.js 22.14.0 and 22.17.1, we identified a regression to a specific change in the stream module where the upstream code always wraps the callback with bindAsyncResource, creating a new AsyncResource instance for every stream.finished() call. This behavior introduced unnecessary overhead, particularly for applications that utilize the stream module in hot paths, resulting in a noticeable performance drop. After identifying the root cause, we provided a patch that checks whether any async hooks or AsyncLocalStorage instances are active; if none are found, it safely skips creating a new async resource.
We first applied the patch internally at Cribl, verified that it passed all the performance tests, and monitored its behavior in production to ensure stability. Once confirmed, we submitted the patch to the Node.js upstream repository, including a performance test showing both the regression and the improvement, along with detailed context for future maintainers. Our merged upstream PR is here. This effort reflects our commitment to the Node.js community—we rely on Node.js every day, and when we identify issues, we contribute to help the broader ecosystem.
Closing Thoughts
Our investigation highlights why thorough performance testing is critical for every release. We closely monitor memory, CPU, and throughput metrics, and if we detect any irregularities, we dive deep to identify the root cause. By analyzing and validating these behaviors and contributing fixes upstream, we ensure that both Cribl and the wider Node.js community benefit. If this kind of work sounds exciting to you, we’re hiring!
