iso-bench - Isolate benchmark tests to avoid performance pollution.

iso-bench

Isolate benchmark tests to avoid performance pollution.

Pitch

iso-bench offers a unique approach to benchmarking by running tests in separate processes, thereby preventing optimization/deoptimization pollution. Unlike traditional libraries such as benchmark.js, iso-bench ensures consistent performance measurements by isolating test environments. This allows for accurate and dependable benchmarking results.

Description

iso-bench is a specialized benchmarking library designed to prevent optimization and deoptimization pollution between tests by executing them in isolated processes. This ensures that each benchmark runs in its own environment, providing consistent performance measurements regardless of the order of execution.

Motivation

Traditional benchmarking tools, such as benchmark.js and benny, often yield varying results due to test interdependencies caused by shared optimizations and deoptimizations in the V8 JavaScript engine. Such pollution can lead to misleading performance conclusions, impacting code quality and optimization efforts. iso-bench addresses this issue by utilizing process forking, allowing tests to run in separate V8 instances with individual memory spaces.

Pollution Examples

Consider the following example using benchmark.js:

const Benchmark = require("benchmark");
const functions = {
    method: function(buf:Buffer) {
        return buf.readUint8(0);
    },
    direct: function(buf:Buffer) {
        return buf[0];
    },
    method_again: function(buf:Buffer) {
        return buf.readUint8(0);
    }
};
const buffers = new Array(1000).fill(0).map(() => {
    const buf = Buffer.allocUnsafe(1);
    buf[0] = Math.floor(Math.random() * 0xFF);
    return buf;
});
const suite = new Benchmark.Suite();
for (const [type, fn] of Object.entries(functions)) {
    suite.add(`${type}`, () => {
        for (let i = 0; i < buffers.length; i++) {
            fn(buffers[i]);
        }
    });
}
suite.on("cycle", event => {
    console.log(String(event.target));
}).run({
    async: true
});

Running this code may return unreliable performance results depending on the order of execution. In contrast, with iso-bench, test outcomes remain stable:

import { IsoBench } from "..";
const bench = new IsoBench();
const functions = {
  method: function(buf:Buffer) {
      return buf.readUint8(0);
  },
  direct: function(buf:Buffer) {
      return buf[0];
  },
  method_again: function(buf:Buffer) {
      return buf.readUint8(0);
  }
};
const buffers = new Array(1000).fill(0).map(() => {
  const buf = Buffer.allocUnsafe(1);
  buf[0] = Math.floor(Math.random() * 0xFF);
  return buf;
});
for (const [type, fn] of Object.entries(functions)) {
  bench.add(`${type}`, () => {
      for (let i = 0; i < buffers.length; i++) {
          fn(buffers[i]);
      }
  });
}
bench.consoleLog().run();

0 comments

No comments yet.

New comment