Linux Memory Paging: Monitor Swap Activity with /proc/vmstat in C++

How Do You Monitor Linux Memory Paging in a Real-Time FinTech Application?

While working on a high-frequency transaction processing platform for a FinTech client, we encountered a critical anomaly in our monitoring dashboards. The system, deployed on a cluster of Ubuntu-based cloud containers, relied heavily on in-memory processing to maintain microsecond-level latency. To monitor infrastructure health without introducing the overhead of heavy third-party monitoring agents, we had deployed a custom, lightweight C++ telemetry agent.

This agent was tasked with parsing Linux system files to collate real-time CPU and memory usage statistics. One of the core requirements was to calculate the page in/out ratio and the swap in/out ratio. Monitoring these metrics is vital in low-latency environments because excessive paging or swapping (memory thrashing) severely degrades transaction throughput.

During a stress-testing phase, the application began experiencing latency spikes, yet our telemetry agent reported zero swap activity. We realized the agent was reading the wrong data points. The engineering team had made a common assumption about how Linux exposes memory metrics. This challenge inspired this article, detailing how we corrected the telemetry architecture so other teams can avoid the same pitfall when designing system-level monitoring tools.

Why Was Our C++ Telemetry Agent Miscalculating Paging Ratios?

In our FinTech architecture, memory pressure is a constant threat. When physical RAM is exhausted, the Linux kernel moves (swaps) memory pages to disk. Disk I/O is exponentially slower than RAM access, making swap activity a leading indicator of impending performance degradation.

To detect this, the business use case required our C++ agent to calculate moving averages of page in/out and swap in/out ratios. By triggering alerts when the swap-out rate spiked, our orchestration layer could automatically route traffic to healthier nodes.

The issue surfaced in the agent’s data collection module. The developers were parsing /proc/meminfo and attempting to calculate swap ratios based on the SwapCached and SwapFree fields. However, the telemetry dashboard remained flat even when we intentionally induced memory starvation and forced the kernel to swap.

Why Doesn’t /proc/meminfo Track Page In and Swap Out Events?

The root of the problem lay in a fundamental misunderstanding of the Linux procfs (process filesystem). Our logs showed the agent successfully reading /proc/meminfo, but the metrics failed to reflect dynamic paging behavior.

The oversight was assuming that SwapCached and SwapFree represented cumulative swap activity.

• SwapFree: This simply indicates the total amount of swap space currently unused on the disk. It is a state gauge, not an event counter.
• SwapCached: This represents memory that was swapped out, successfully brought back into RAM, but has not yet been modified. It remains in the swap file as a backup. It does not tell you how many pages are actively moving in or out per second.

/proc/meminfo provides a snapshot of memory state at an exact millisecond. It does not provide cumulative counters of system events. Calculating an accurate “per-second” ratio of paging or swapping from state snapshots is virtually impossible because you miss all the microscopic events that occur between your polling intervals.

Where Can Developers Find Accurate Paging and Swap Metrics in Linux?

To fix the agent, we needed to identify the correct source of truth for cumulative system events. We stepped back to evaluate how the Linux kernel exposes these metrics. We considered several solutions to retrieve this data efficiently.

Can We Calculate Differentials from /proc/meminfo Snapshots?

Our first theoretical approach was to poll /proc/meminfo at very high frequencies (e.g., every 10 milliseconds) and calculate the delta of SwapFree. However, this approach was quickly discarded. SwapFree only changes when new swap space is allocated or freed. It does not capture pages that are swapped out and immediately replaced. Furthermore, high-frequency file I/O in a monitoring agent introduces the exact CPU overhead we were trying to avoid.

Should We Execute External Commands Like vmstat or sar?

We considered using C++ popen() to execute the standard vmstat command and parse its standard output. While vmstat provides exactly the data we needed, calling external processes via shell commands is heavily frowned upon in high-performance system engineering. Forking a new process every second creates unacceptable overhead and security risks. When companies hire linux developers for infrastructure management, they expect native, system-level integrations rather than brittle shell-script wrappers.

Is /proc/vmstat the Optimal Source for Cumulative Counters?

Our final and most efficient approach was to parse /proc/vmstat. Unlike /proc/meminfo, which tracks current state, /proc/vmstat tracks cumulative system events since the system booted. This file contains exactly the counters required to calculate accurate rates over time:

• pgpgin / pgpgout: The number of pages paged in and out of disk.
• pswpin / pswpout: The number of swap pages brought in and out of the swap partition.

By reading these cumulative counters at fixed intervals and calculating the difference, we could accurately determine the exact number of pages swapped per second.

How Did We Parse /proc/vmstat for Accurate Swap In and Out Ratios in C++?

We rewrote the agent’s data collection routine to read /proc/vmstat. The implementation was designed to be highly optimized, utilizing low-level C++ file streams and avoiding unnecessary string allocations.

Here is a sanitized, generic representation of how we extracted these metrics:

#include <iostream>
#include <fstream>
#include <string>
#include <unordered_map>
#include <thread>
#include <chrono>
struct VmMetrics {
    unsigned long long pgpgin = 0;
    unsigned long long pgpgout = 0;
    unsigned long long pswpin = 0;
    unsigned long long pswpout = 0;
};
VmMetrics read_vmstat() {
    std::ifstream vmstat_file("/proc/vmstat");
    std::string key;
    unsigned long long value;
    VmMetrics metrics;
    if (!vmstat_file.is_open()) {
        // Handle error gracefully
        return metrics;
    }
    while (vmstat_file >> key >> value) {
        if (key == "pgpgin") metrics.pgpgin = value;
        else if (key == "pgpgout") metrics.pgpgout = value;
        else if (key == "pswpin") metrics.pswpin = value;
        else if (key == "pswpout") metrics.pswpout = value;
    }
    
    return metrics;
}
void monitor_swap_ratio() {
    VmMetrics prev = read_vmstat();
    std::this_thread::sleep_for(std::chrono::seconds(1));
    VmMetrics curr = read_vmstat();
    // Calculate events per second
    unsigned long long swap_in_rate = curr.pswpin - prev.pswpin;
    unsigned long long swap_out_rate = curr.pswpout - prev.pswpout;
    std::cout << "Swap In Rate: " << swap_in_rate << " pages/secn";
    std::cout << "Swap Out Rate: " << swap_out_rate << " pages/secn";
}

Validation Steps: To validate the fix, we deployed the updated agent to an isolated test container. We artificially constrained the container’s memory limit using cgroups and executed a script that rapidly allocated memory. The updated telemetry agent instantly registered the spike in pswpout, successfully validating our architectural correction.

What Should Architects Remember About Linux System Telemetry?

This experience highlighted several critical lessons for engineering teams building custom infrastructure tooling:

• Know the Difference Between Gauges and Counters: /proc/meminfo provides gauges (current state). /proc/vmstat provides counters (cumulative events). Always use counters to calculate rates over time.
• Avoid Shell Execution in Telemetry: Native language file parsing is exponentially faster and safer than using popen() or system calls to execute standard utilities like vmstat.
• Understand Page vs. Swap: Not all paging is bad. Linux constantly pages executables and shared libraries from disk (tracked via pgpgin). However, swapping (pswpout) indicates RAM exhaustion. Ensure your monitoring alerts target the correct metric.
• Invest in Specialized Engineering: Building robust telemetry requires deep OS-level knowledge. When you hire c++ developers for system engineering, ensure they possess a thorough understanding of the Linux kernel interfaces.
• Test with Synthetic Loads: Telemetry tools must be validated against real stress conditions. If we hadn’t induced memory starvation in our testing environment, we wouldn’t have discovered the blind spot in our monitoring.
• Keep Parsing Lightweight: When reading procfs, optimize your string parsing. High-frequency polling with inefficient parsers can cause the monitoring tool itself to become a bottleneck.

How Can Proper System-Level Monitoring Prevent Production Outages?

Accurate system telemetry is the foundation of high-availability architectures. By correcting our misconception about /proc/meminfo and migrating our C++ agent to parse /proc/vmstat, we successfully established a reliable, low-latency early warning system for memory thrashing. This granular level of infrastructure monitoring empowers systems to auto-remediate before end-users experience latency. Whether you are building native agents or require experts to scale your infrastructure, having experienced engineers who understand the nuances of the operating system is critical. If your organization is facing similar architectural challenges or looking to scale your engineering capabilities, contact us.

Social Hashtags

#Linux #LinuxKernel #Cpp #CPlusPlus #DevOps #SystemProgramming #LinuxPerformance #PerformanceEngineering