Benchmark your NPM Package

Hello folks,

if you are interested in learning how to benchmark your NPM package, you are at the right place.

Today, you will learn to use benchmark.js and how to use it to benchmark all of your released versions. Doing so will allow you identify performance downgrade/upgrade for a particular version.

Some actual code

Before getting started with the fun part, the coding part, I want to first point out that I'm using TypeScript for my examples, but this would work almost just the same with vanilla JavaScript. For this example I will be using the npm package named ss-search, so you just need to replace that with your own library name to get the appropriate results.

If you are one of those that wants to see the final result right away, go ahead and take a look at the implementation I made for my search library ss-search.

1- Create a folder benchmark at the root of your package

2- Fetch all the published versions of our package

const packageVersions: string[] = JSON.parse(execSync("npm view ss-search versions --json").toString())

3- Fetch our existing benchmark results and check which one's are not benchmarked yet

const benchmarkResultPath = `${__dirname}/benchmarkResults.json`

let benchmarkResults: BenchmarkResult[] = existsSync(benchmarkResultPath) ? JSON.parse(readFileSync(benchmarkResultPath).toString()) : []

// Note that I'm using the difference function from lodash, but can easily be converted to pure js
const missingVersionsToBenchmark = difference(
    packageVersions,
    benchmarkResults.map((x) => x.version),
)

4- Run the benchmark and save the results

for (const version of missingVersionsToBenchmark) {
    // This installs the specific version of our library
    execSync(`cd ${__dirname} && npm i --prefix ./ ss-search@${version}`).toString()

    const suite = new Benchmark.Suite()

    // add tests
    suite
        // This add function allows us to add all the test cases we need for our benchmark
        .add(
            "search",
            function () {
                search(data, Object.keys(data[0]), "l o r a w e l l s Lesmuf")
            },
            { minSamples: 200 },
        )
        // add listeners
        .on("cycle", function (event: any) {
            const benchmarkResult: BenchmarkResult = {
                version,
                operationType: OperationType.Search,
                operationsPerSecond: event.target.hz,
                runsSampled: event.target.stats.sample.length,
            }
            benchmarkResults = [...benchmarkResults, benchmarkResult]
            // Save the results to a file
            writeFileSync(benchmarkResultPath, JSON.stringify(benchmarkResults, null, 2))

            console.log(`Benchmarked versrion ${version} - ${event.target}`)
        })
        .run({ minSamples: 200 })
}

5- Add a script to your NPM package to launch the benchmarks

// You might not need to pass in the compiler options, but since my tsconfig is targeting es6 and this one commonjs, I needed to overwrite those
"benchmark": "TS_NODE_COMPILER_OPTIONS='{\"module\":\"commonjs\"}' ts-node ./benchmark/benchmark.ts"

Conclusion

There you have it, you now have added historical benchmarking capabilities to your NPM package. Once again to see the full example in action go over here.