Importing multiple versions/branches of a module to benchmark in Julia

Question

How can I use several different versions or branches of the same module in a single script in Julia?

e.g. If I wanted to benchmark each of the tagged releases.

(Someone asked a similar question recently and I answered the wrong one but though this might be useful anyway.)

Edit: I have answered this myself but I am sure their may be a better way!

Bourges

Answer 1

You can just git checkout a different version of the module and then use benchmarkTools.jl to benchmark. However it may be better to use multiple scripts though (or at least ignore the first trial) (See this comment Importing multiple versions of the same Module/Package for Benchmarking for more info).

e.g.

packagedir = Pkg.dir("DSP")
version2checkout = "v0.0.7"
run(`cd $packagedir`); run(`git checkout tags/$version2checkout`)
import DSP
# do all your benmarking stuff
# start again

Prevents you from having to copy the modules but still a little clunky I guess. You could even do it in a loop for lots of versions by capturing the output of git tag

for i in readlines(`git tag`)
    version2checkout = chomp(i)
    # checkout version and benchmark
end

Bourges

Answer 2

Note also that Pkg.checkout takes an optional branch argument:

help?> Pkg.checkout
  checkout(pkg, [branch="master"]; merge=true, pull=true)

  Checkout the Pkg.dir(pkg) repo to the branch branch. Defaults to checking
  out the "master" branch. To go back to using the newest compatible released
  version, use Pkg.free(pkg). Changes are merged (fast-forward only) if the
  keyword argument merge == true, and the latest version is pulled from the
  upstream repo if pull == true.

So you can do Pkg.checkout("MyPackage", "v0.6.0"). To ensure that modules are reloaded, the workspace() function might come in handy; or a new Julia process could be executed for each package version.

Cockspur

have you tried this? I couldn't get it to work. I thought you might need to use "tags/v0.6.0". Mind you I only tried on one package... – Bourges

@AlexanderMorley I don't know why you need the tags/. It seems to work for me without. – Cockspur

Answer 3

Available tools

There are several tools today that can be used for this:

• BenchmarkTools.jl
  • Contains fundamental benchmarking tools and terminology.
  • offers the @btime macro for quick basic benchmarking of functions
  • @benchmark macro for more detailed benchmarking,
  • Allows definition of BenchmarkGroup that is the basis for the benchmarks in the following packages
• PkgBenchmark.jl
  • Contains tools for benchmarking packages and their commits
  • Can automatically run BenchmarkGroups defined in package benchmark/benchmarks.jl file from outside the package for different commits
  • Can be used in CI, also usable for local benchmarking
  • Works ONLY with code that is committed, does not work with dirty directories.
• AirspeedVelocity.jl
  • terminal-based benchmarking tool for benchmarking commits inside published packages, can also be used in CI
  • not suitable for local benchmarking of in-development code.

Basic benchmarking when developing can be done with @btime macro from BenchmarkTools.jl.

Benchmarking package performance with PkgBenchmark.jl and BenchmarkTools.jl

The basic workflow would be to prepare a BenchmarkGroup in the package, and run the relevant group manually as you go to help development. Then, when you commit, run all the benchmarks with PkgBenchmark.jl with every commit, checking for performance regressions.

However, if you want to add new benchmarks to the package or upgrade the benchmarking code or compare new benchmark across different commits and branches, as you do, you will need to rebase the tested commits on top of the updated benchmarks to use PkgBenchmark.jl.

So, my workflow is following:

1. edit/create benchmarks in /benchmark/benchmarks.jl and commit them to a benchmarking branch
2. git rebase the branch with commits I want to evaluate on top of the changes to the benchmark. It's better to do this locally as rebase rewrites history.
3. ] dev MyPackage in the benchmarking environment, use PkgBenchmark.jl to evaluate the commits. Note that the package directory can't be dirty.

Example script for running benchmarks of a package that generates an overview of improvements/regressions of a single benchmark (BenchmarkGroup is defined in benchmark/benchmarks.jl):

using PkgBenchmark # installed in root environment
using BenchmarkTools 
using LibGit2
using Printf
# activate the package benchmarking environment and run following

# Define the commits to benchmark, obtained with `git rev-list <since_hash>..HEAD`
tags = [
"30242a78b4f6a2e3914610d608c602172e8968cb", # latest
"9c421b298a30deb338bc37e2dc43cdb4c17d09b7",
"76d103ee7a66e9f1619650b15d06912311212d3a",
"30c18da277921b64d70cc12271e5f83e7936e6bc",
"b10aa8696efa4a6559add82d0274412080bff4b8",
"98450a1cbcfc8ccbbbb288271729e9da19d867f7", # oldest
]
reverse!(tags)

baseline = "f6a815db026d21c144346c8b9e5c5df3548f0f61"
judgements = []
benchmarks = []

# Run the benchmarks of selected commits above, it's not necessary to store in the resultfile
baseline_benchmark = PkgBenchmark.benchmarkpkg("MyPackage", baseline; resultfile = "./dev/MyPackage/benchmark/results/pkgbnch_benchmark_$baseline.json")
for target in tags
    push!(benchmarks, PkgBenchmark.benchmarkpkg("MyPackage", target; resultfile = "./dev/MyPackage/benchmark/results/pkgbnch_benchmark_$target.json"))
end

# Collect information about the commits
r = LibGit2.GitRepo("./dev/MyPackage")
restable = """
| order | hash | time ratio | memory ratio | comment |
"""
for (i, benchmark) in enumerate([baseline_benchmark, benchmarks...])
    result = BenchmarkTools.ratio(median(PkgBenchmark.benchmarkgroup(benchmark)), median(PkgBenchmark.benchmarkgroup(baseline_benchmark)))
    trial_ratio = result["benchmark_suite_name"]["selected_benchmark"] # Select the benchmark for which you want to visualize the results

    commit_hash = PkgBenchmark.commit(benchmark)
    comment =  c = LibGit2.message(LibGit2.GitCommit(r, commit_hash))
    
    line = """
    | $i | $(commit_hash[1:8]) | $(@sprintf("%.2f", trial_ratio.time)) | $(@sprintf("%.2f", trial_ratio.memory)) | $(comment) |
    """
    restable *= line
end

print(restable)

Lebanon