<summary> About stdlib... </summary>

We believe in a future in which the web is a preferred environment for numerical computation. To help realize this future, we've built stdlib. stdlib is a standard library, with an emphasis on numerical and scientific computation, written in JavaScript (and C) for execution in browsers and in Node.js.

The library is fully decomposable, being architected in such a way that you can swap out and mix and match APIs and functionality to cater to your exact preferences and use cases.

When you use stdlib, you can be absolutely certain that you are using the most thorough, rigorous, well-written, studied, documented, tested, measured, and high-quality code out there.

To join us in bringing numerical computing to the web, get started by checking us out on GitHub, and please consider financially supporting stdlib. We greatly appreciate your continued support!

Benchmark

[![NPM version][npm-image]][npm-url] [![Build Status][test-image]][test-url] [![Coverage Status][coverage-image]][coverage-url]

Benchmark harness.

## Installation

bash
npm install @stdlib/bench-harness

## Usage

javascript
var bench = require( '@stdlib/bench-harness' );

#### bench( name[, options][, benchmark] ) Queues a benchmark to be run during a subsequent turn of the event loop. After running the benchmark, the function outputs benchmark results as Test Anything Protocol ([TAP][tap]) output.

javascript
bench( 'Math.sin', function benchmark( b ) {
    var x;
    var i;
    b.tic();
    for ( i = 0; i < b.iterations; i++ ) {
        x = Math.sin( Math.random() );
        if ( x !== x ) {
            b.fail( 'should not return NaN' );
        }
    }
    b.toc();
    if ( x !== x ) {
        b.fail( 'should not return NaN' );
    }
    b.pass( 'benchmark finished' );
    b.end();
});

A benchmark function has the following signature:

javascript
function benchmark( b ) {
    // Benchmark code...
}

where b is a Benchmark instance. Synchronous benchmarks should, at minimum, have the following structure:

javascript
function benchmark( b ) {
    var x;
    var i;

    // [1] Start timing:
    b.tic();

    // [2] Loop containing code to time...
    for ( i = 0; i < b.iterations; i++ ) {
        // [3] Code to time...

        // [4] A conditional verifying results to prevent certain compiler optimizations:
        if ( x !== x ) {
            b.fail( 'something went wrong!' );
        }
    }
    // [5] Stop timing:
    b.toc();

    // [6] Another conditional verifying results to prevent certain compiler optimizations:
    if ( x !== x ) {
        b.fail( 'something went wrong!' );
    }
    // [7] End the benchmark:
    b.end();
}

Asynchronous benchmarks should have a structure similar to the following:

javascript
function benchmark( b ) {
    var i = 0;

    // [1] Start timing:
    b.tic();

    // [2] Asynchronous code to time:
    return next();

    function next( error ) {
        if ( error ) {
            return b.fail( error.message );
        }
        i += 1;

        // [3] Exit condition:
        if ( i <= b.iterations ) {
            // Asynchronous task...
            return;
        }
        // [4] Stop timing:
        b.toc();

        // [5] End the benchmark:
        b.end();
    }
}

For both synchronous and asynchronous benchmarks, calling b.end() is mandatory, as failing to do so will cause the harness to hang. For example, the following benchmark will never complete.

javascript
function benchmark( b ) {
    var i;
    b.tic();
    for ( i = 0; i < b.iterations; i++ ) {
        // Synchronous task...
    }
    b.toc();
}

Avoid making assertions within timed code, as doing so will significantly affect raw performance numbers. For example, avoid the following:

javascript
function benchmark( b ) {
    var x;
    var i;
    b.tic();
    for ( i = 0; i < b.iterations; i++ ) {
        x = Math.sin( Math.random() );
        b.equal( x, x, 'does not return NaN' ); // Avoid doing this!
    }
    b.toc();
    b.equal( x, x, 'does not return NaN' );     // This is fine.
    b.end();
}

Additionally, ensure that all setup code executes before calling b.tic() and that all cleanup code executes after calling b.toc(). For example, avoid the following:

javascript
function benchmark( b ) {
    var x;
    var y;
    var i;

    // Start timing:
    b.tic();

    // Setup code:
    x = new Array( b.iterations );                  // Should be before b.tic()!
    for ( i = 0; i < b.iterations; i++ ) {
        x[ i ] = Math.random();
    }
    // Code to be timed...
    for ( i = 0; i < b.iterations; i++ ) {
        y = Math.sin( x[ i ] );
        if ( y !== y ) {
            b.fail( 'should not return NaN' );
        }
    }
    // Verify results:
    b.equal( x, x, 'does not return NaN' );         // Should be after b.toc()!

    // Stop timing:
    b.toc();

    b.end();
}

The function accepts the following options: - iterations: number of iterations. If null, the number of iterations is determined by trying successive powers of 10 until the total time is at least 0.1 seconds. Default: null. - repeats: number of repeats. Default: 3. - timeout: number of milliseconds before a benchmark is considered failed. Exceeding a timeout does not, however, end the benchmark. Ending a long running benchmark requires manual intervention. Default: 300000 (5 minutes). - skip: boolean indicating whether to skip a benchmark. By default, the harness will automatically determine an iteration number for each benchmark such that a benchmark runs for a length of time sufficient to accurately compute benchmark results. To require a specific number of iterations, set the iterations option.

javascript
var opts = {
    'iterations': 1e6
};

bench( 'require a specific number of iterations', opts, function benchmark( b ) {
    var i;
    b.tic();
    for ( i = 0; i < b.iterations; i++ ) {
        // Code to be benchmarked...
    }
    b.toc();
    b.end();
});

To ensure that benchmark results are reproducible, the harness runs each benchmark function multiple times. To specify a repetition number, set the repeats option.

javascript
var opts = {
    'repeats': 5
};

bench( 'repeat a benchmark multiple times', opts, function benchmark( b ) {
    var i;
    b.tic();
    for ( i = 0; i < b.iterations; i++ ) {
        // Code to be benchmarked...
    }
    b.toc();
    b.end();
});

To skip a benchmark, set the skip option.

javascript
var opts = {
    'skip': true
};

bench( 'skipped benchmark', opts, function benchmark( b ) {
    var i;
    b.tic();
    for ( i = 0; i < b.iterations; i++ ) {
        // Code to be benchmarked...
    }
    b.toc();
    b.end();
});

To fail benchmarks which take longer than a specified amount of time to complete, set a timeout option (in milliseconds).

javascript
var opts = {
    'timeout': 5000    // 5 seconds
};

bench( 'async benchmark', opts, function benchmark( b ) {
    var i = 0;
    b.tic();
    return next();

    function next( error ) {
        if ( error ) {
            return b.fail( error.message );
        }
        i += 1;
        if ( i <= b.iterations ) {
            // Asynchronous task...
            return;
        }
        b.toc();
        b.end();
    }
});

#### bench.onFinish( clbk ) Sets a listener which is invoked once the harness finishes running all benchmarks.

javascript
function onFinish() {
    console.log( 'Done!' );
}

bench.onFinish( onFinish );

#### bench.createStream( [options] ) Returns a results [stream][nodejs-stream].

javascript
var stdout = require( '@stdlib/streams-node-stdout' );

var stream = bench.createStream();

// Direct all results to `stdout`:
stream.pipe( stdout );

var opts = {
    'iterations': 1,
    'repeats': 1
};

bench( 'beep', opts, function benchmark( b ) {
    var i;
    b.tic();
    for ( i = 0; i < b.iterations; i++ ) {
        b.equal( 3.14, 3.14, 'should be equal' );
    }
    b.toc();
    b.end();
});

text
TAP version 13
# beep
ok 1 should be equal
  ---
  iterations: 1
  elapsed: 0.002985193
  rate: 334.98671610177297
  ...
#
1..1
# total 1
# pass  1
#
# ok

The results stream can be combined with any [Writable][nodejs-writable-stream] stream (e.g., network connection, file, stdout, etc). The function accepts the same options as [@stdlib/streams/node/transform][@stdlib/streams/node/transform]. For example, by default, the method returns a stream which produces [TAP][tap] output as text. To return an object stream,

javascript
var opts = {
    'objectMode': true
};

var stream = bench.createStream( opts );
stream.on( 'data', onRow );

function onRow( row ) {
    console.log( JSON.stringify( row ) );
}

opts = {
    'iterations': 1,
    'repeats': 1
};

bench( 'beep', opts, function benchmark( b ) {
    var i;
    b.tic();
    for ( i = 0; i < b.iterations; i++ ) {
        b.equal( 3.14, 3.14, 'should be equal' );
    }
    b.toc();
    b.end();
});

text
{"type":"benchmark","name":"beep","id":0}
{"id":0,"ok":true,"name":"should be equal","operator":"equal","actual":3.14,"expected":3.14,"benchmark":0,"type":"assert"}
{"ok":true,"operator":"result","iterations":1,"elapsed":0.00283753,"rate":352.41918147120913,"benchmark":0,"type":"result"}
{"benchmark":0,"type":"end"}

#### bench.createHarness( [options][, clbk] ) Creates a benchmark harness with a new pending stack and state.

javascript
var harness = bench.createHarness();

harness( 'beep', function benchmark( b ) {
    var i;
    b.tic();
    for ( i = 0; i < b.iterations; i++ ) {
        // Synchronous task...
    }
    b.toc();
    b.end();
});

To trigger an action when a harness finishes running all benchmarks, provide a callback function.

javascript
var harness = bench.createHarness( onFinish );

function onFinish() {
    harness.close();
}

harness( 'beep', function benchmark( b ) {
    var i;
    b.tic();
    for ( i = 0; i < b.iterations; i++ ) {
        // Synchronous task...
    }
    b.toc();
    b.end();
});

The method accepts the following options: - autoclose: boolean indicating whether to automatically close a harness after running all benchmarks. By default, a harness does not automatically close. To automatically close a harness once a harness finishes running all benchmarks, set the autoclose option to true.

javascript
var harness = bench.createHarness({
    'autoclose': true
});

harness( 'beep', function benchmark( b ) {
    var i;
    b.tic();
    for ( i = 0; i < b.iterations; i++ ) {
        // Synchronous task...
    }
    b.toc();
    b.end();
});

### Harness A harness has the following properties and methods... #### harness.createStream( [options] ) Returns a results [stream][nodejs-stream]. javascript var stdout = require( '@stdlib/streams-node-stdout' ); var harness = bench.createHarness(); var stream = harness.createStream(); // Direct all results to `stdout`: stream.pipe( stdout ); var opts = { 'iterations': 1, 'repeats': 1 }; harness( 'beep', opts, function benchmark( b ) { var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { b.equal( 'beep', 'beep', 'should be equal' ); } b.toc(); b.end(); }); text TAP version 13 # beep ok 1 should be equal --- iterations: 1 elapsed: 0.00166768 rate: 599.6354216636286 ... # 1..1 # total 1 # pass 1 # # ok The method accepts the same options as [@stdlib/streams/node/transform][@stdlib/streams/node/transform]. Note: benchmarks will not run until a destination stream has been created. #### harness.close() Closes a benchmark harness. Any pending benchmarks are cleared from the harness stack. javascript var stdout = require( '@stdlib/streams-node-stdout' ); var harness = bench.createHarness(); var stream = harness.createStream(); stream.pipe( stdout ); var opts = { 'iterations': 5, 'repeats': 5 }; harness( 'early close', opts, function benchmark( b ) { var i = 0; b.tic(); setTimeout( next, 0 ); function next() { i += 1; if ( i <= b.iterations ) { b.ok( true, 'should be truthy' ); return setTimeout( next, 10 ); } b.toc(); b.end(); } }); // Early close: setTimeout( onTimeout, 50 ); function onTimeout() { harness.close(); } text TAP version 13 # early close ok 1 should be truthy ok 2 should be truthy # WARNING: harness closed before completion. ok 3 should be truthy ok 4 should be truthy ok 5 should be truthy --- iterations: 5 elapsed: 0.05940291 rate: 84.17096064822414 ... Warning: a running benchmark may finish after closing a harness. #### harness.exit() Forcefully exits a benchmark harness. All pending benchmarks will generate failing assertions. javascript var stdout = require( '@stdlib/streams-node-stdout' ); var harness = bench.createHarness(); var stream = harness.createStream(); stream.pipe( stdout ); var opts = { 'iterations': 5 }; harness( 'force exit', opts, function benchmark( b ) { var i = 0; b.tic(); return next(); function next() { i += 1; if ( i <= b.iterations ) { b.ok( true, 'should be truthy' ); return setTimeout( next, 10 ); } b.toc(); b.end(); } }); // Forcefully exit: setTimeout( onTimeout, 20 ); function onTimeout() { harness.exit(); } text TAP version 13 # force exit ok 1 should be truthy not ok 2 benchmark exited without ending --- operator: fail TODO: include stack ... not ok 3 benchmark exited without ending --- operator: fail TODO: include stack ... ok 4 should be truthy ok 5 should be truthy ok 6 should be truthy ok 7 should be truthy --- iterations: 5 elapsed: 0.061504862 rate: 81.29438612511642 ... Warning: a running benchmark may finish after exiting a harness. #### harness.exitCode Read-only property whose value is the harness exit code. If all benchmarks run successfully (i.e., no failing assertions), the exit code is 0; otherwise, the exit code is 1. javascript var harness = bench.createHarness(); // Benchmarks only start running when results have a destination: var stream = harness.createStream(); function onFinish() { console.log( harness.exitCode ); // => 1 } var opts = { 'iterations': 1, 'repeats': 1 }; harness( 'exit code', opts, function benchmark( b ) { var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { b.fail( 'failing assertion' ); } b.toc(); b.end(); }); ### Benchmark A Benchmark instance has the following properties and methods... #### b.name Read-only property whose value is the benchmark name.

javascript
var str = b.name;
// returns <string>

#### b.iterations Read-only property whose value is the number of iterations.

javascript
var iter = b.iterations;
// returns <number>

#### b.tic() Starts a benchmark timer. In order to benchmark code, this method must always be called within a benchmark function.

javascript
function benchmark( b ) {
    var x;
    var i;

    // Start a timer:
    b.tic();

    for ( i = 0; i < b.iterations; i++ ) {
        x = Math.sin( Math.random() );
        if ( x !== x ) {
            b.fail( 'should not return NaN' );
        }
    }
    b.toc();
    if ( x !== x ) {
        b.fail( 'should not return NaN' );
    }
    b.end();
}

#### b.toc() Stops a benchmark timer. In order to benchmark code, this method must always be called within a benchmark function.

javascript
function benchmark( b ) {
    var x;
    var i;

    b.tic();
    for ( i = 0; i < b.iterations; i++ ) {
        x = Math.sin( Math.random() );
        if ( x !== x ) {
            b.fail( 'should not return NaN' );
        }
    }
    // Stop a timer:
    b.toc();

    if ( x !== x ) {
        b.fail( 'should not return NaN' );
    }
    b.end();
}

#### b.end() Explicitly ends a benchmark. In order to benchmark code, this method must always be called within a benchmark function.

javascript
function benchmark( b ) {
    var x;
    var i;

    b.tic();
    for ( i = 0; i < b.iterations; i++ ) {
        x = Math.sin( Math.random() );
        if ( x !== x ) {
            b.fail( 'should not return NaN' );
        }
    }
    b.toc();

    if ( x !== x ) {
        b.fail( 'should not return NaN' );
    }

    // Explicitly end the benchmark:
    b.end();
}

Warning: no assertions should follow a call to b.end(). Including assertions after b.end() may result in interleaved [TAP][tap] output or an output stream closing before a benchmark executes pending assertions. #### b.comment( msg ) Writes a message without breaking [TAP][tap] output.

javascript
b.comment( 'This is a comment.' );

text
# This is a comment.

To prevent confusing results parsers, avoid using comments. Comments are frequently used for demarcating the beginning of a new benchmark run and/or providing diagnostic information. Both of the aforementioned use cases typically fall under the domain of the harness, not the user. #### b.skip( value, msg ) Generates an assertion which will be skipped.

javascript
b.skip( false, 'This is skipped.' );
b.skip( true, 'This is skipped.' );

text
ok 1 This is skipped. # SKIP
ok 2 This is skipped. # SKIP

#### b.todo( value, msg ) Generates an assertion which should be implemented.

javascript
b.todo( false, 'This is a todo.' );
b.todo( true, 'This is a todo.' );

text
not ok 3 This is a todo. # TODO
  ---
  operator: todo
  TODO: include stack
  ...
ok 4 This is a todo. # TODO

While b.todo() assertions typically fail, they do not contribute to the failed assertion count. If a benchmark includes b.todo() assertions and no other failing assertions, the benchmark is considered successful. #### b.fail( msg ) Generates a failing assertion.

javascript
b.fail( 'This is a failing assertion.' );

text
not ok 5 This is a failing assertion.
  ---
  operator: fail
  TODO: include stack
  ...

#### b.pass( msg ) Generates a passing assertion.

javascript
b.pass( 'This is a passing assertion.' );

text
ok 6 This is a passing assertion.

#### b.ok( value[, msg] ) Asserts that a value is truthy.

javascript
b.ok( [] );

text
ok 7 should be truthy