How can I generate JSON benchmark results for NumPy/SciPy to compare different Windows architectures?

Below is an explanation and a recommended workflow:

Why You’re Not Getting a JSON Output

When you run the benchmark with a simple command such as asv run, the results are printed to the console but are not saved to disk. This means that no JSON files are generated in your results folder for comparison.

How to Generate JSON Files for Comparison

To save your benchmark data (and thus obtain a JSON file), you need to publish your results using ASV. The typical workflow is:

1. Run the Benchmarks and Record the Results
   Change to the benchmarks directory and run:

   BASHCopied
   asv run -n -e --python=same

   Here, the -n flag is used for "recording" the results (i.e., no actual replication), and -e ensures you capture errors. The --python=same keeps the same Python environment.

2. Publish the Results
   After running the benchmarks, publish the recorded results using:

   BASHCopied
   asv publish

   This will generate the JSON file (along with HTML output if desired) in the results folder (inside a directory named after your machine). The JSON file is what you need for comparing scores across machines.

Comparing the Results

Once you have the published results on both machines, you can compare them using:

BASHCopied
asv compare machine1.json machine2.json

This command will output a plain-text comparison of the benchmark scores between the two JSON files. No Excel or manual extraction is required.

Summary

• Issue: Running asv run alone doesn’t save the JSON result files.
• Solution: Execute asv publish after running your benchmarks to generate the JSON file.
• Comparison: Use asv compare machine1.json machine2.json to get a numerical comparison without relying on the graphical view.

Following this workflow will allow you to focus on the numbers and directly compare the benchmark data from NumPy and SciPy across your different Windows machines.