Benchmark Analysis Tool: Tracking Performance in Valida zkVM

Performance tracking is absolutely critical for zkVMs, where execution time and memory usage directly determine whether proofs remain practical for real-world applications. Even seemingly minor regressions can snowball into significant overhead when scaling to production workloads. Our benchmark analysis tooling ensures Valida maintains predictable, optimized performance by enabling targeted improvements and rapid fixes when we detect either regressions or optimization opportunities.

By continuously monitoring these key metrics, we prevent small inefficiencies from compounding into major bottlenecks, keeping the zkVM performant across its entire operational range. This rigorous approach to performance management allows us to deliver consistent proof generation times while maintaining the flexibility to implement new features and optimizations.

To systematically monitor the performance of Valida zkVM, we developed a benchmark analysis tool, a Nim program that processes benchmark results from our CI pipeline.

This tool helps us:

• Detect performance regressions in pull requests
• Quantify the impact of optimizations
• Analyze the scaling behavior of Valida’s execution time and memory usage
  ‍

How the Benchmark Analysis Tool Works

The analyser makes use of many of the Nim libraries Sebastian wrote during his PhD that involved a lot of plotting, dataframe, curve fitting and more.
‍

Input: Benchmark Data from CI

Our benchmark CI runs on every pull request, executing a suite of programs (written in C and Rust) on Valida zkVM. These include:

• Fibonacci variants (serial/parallel, iterative/recursive)
• Hashing functions (SHA256, Keccak)

Each benchmark runs multiple times (default: 30 iterations, fewer for long-running programs), and the CI outputs a CSV file with execution time and memory usage per run:
‍

Additionally, we log trace sizes (key for zkVM performance analysis).

‍
Analysis Steps‍

1. Aggregate Statistics
   
   1. Computes min/max, mean/median, standard deviation
   2. Detects outliers (default: >3σ from mean) to flag anomalies (e.g., cloud resource contention)
2. Scaling Analysis
   
   1. Plots execution time/memory against trace size
   2. Fits a log-linear function (f(x) = a·x·ln(x) + b·x + c) to model Valida’s performance scaling:
      
      1. Linear term (b·x) from VM execution
      2. Log-linear term (a·x·ln(x)) from proving (e.g., FFTs in ZK proofs)
3. Comparison Between Runs
   
   1. Compares two benchmark results (e.g., PR vs. main branch)
   2. Highlights changes in mean/median performance and outliers:

Why This Matters

• Proactive Optimization: Catch regressions before they merge.
• Quantifiable Improvements: Measure the impact of changes (e.g., a 4.4% speedup in Keccak).
• Transparency: Future plans include public benchmark tracking.

Example Outputs

An example for an individual fit result (log-linear function):
‍

‍

An example for a combined grid plot of all traces for a single metric (log-linear function):

‍

As we evolve our benchmarking system, we're planning several key enhancements to provide even deeper insights into Valida's performance characteristics:

Planned Metrics & Features:

• Parallel Scaling Analysis
• Measure core utilization efficiency
• Quantify speedup from serial to parallel execution
• Calculate scaling factors (ideal vs. actual)
• More detailed reporting of fit results (covariance matrix)

→ Reveals whether adding more compute resources actually reduces runtime as expected

Hardware-Normalized Metrics:

• Document exact CPU specifications used in CI
• Develop FLOPs-equivalent efficiency numbers
• Create hardware-adjusted performance indices

→ Enables apples-to-apples comparisons with other zkVMs

‍

Energy Efficiency Metrics

• Integrate power draw measurements
• Calculate joules-per-proof metrics
• Estimate cloud compute costs

→ Provides true operational cost insights for large-scale deployments

‍

Public Benchmark Dashboard

• Automated result publishing
• Historical performance tracking
• Interactive scaling visualizations

→ Transparent performance tracking for our customers and community

These enhancements will position Valida as having one of the most comprehensive benchmarking systems in the zkVM space, providing both developers and users with unparalleled visibility into real-world performance characteristics. The data will not only help optimize Valida itself, but also establish meaningful benchmarks toolings and standards for the entire zkVM ecosystem.

‍

Try It Out

The tool is open-source: github.com/lita-xyz/benchmark-analyzer. Contributions welcome!