Building a Testing Framework for JavaScript - Part Two

Recap

In the first part of our journey to build Fest, we laid the groundwork by implementing:

File path resolution for discovering test files
A test runner to execute the tests
A colorful logger for better output readability
An assertion library for validating test results

Now, let's continue our journey by implementing the remaining core features:

Handler for hooks like BeforeEach, AfterAll etc
A reporter for displaying test results

Hooks are functions that allows us to run custom code during the lifecycle of a function or program's execution. We use them to modify the behaviour of its parents' context without changing the core logic. It is important to clarify that hooks and callbacks are related concepts, but one is not a sub-type of the other.

We are going to implement the following hooks:

beforeEach: Run before each test case
afterEach: Run after each test case
beforeAll: Run once before all test cases
afterAll: Run once after all test cases

Things to note

These hooks are basically functions that take and execute a callback.
We need to keep track i.e. register all the hooks and call them in the right order.
We need to keep track of the current describe and it blocks because of the execution context/scope.
We are going to refer to it as spec.

The pseudocode looks something like this.

// Program stack
describe
  beforeAll
  afterAll
  beforeEach
  afterEach
  spec_1
  spec_2


// Execution order
describe
  beforeAll

  beforeEach
  spec_1
  afterEach

  beforeEach
  spec_2
  afterEach

  afterAll

Implementation

// src/hooks.js
const defaultContext = {
  name: "",
  specs: [],
  beforeAll: [],
  afterAll: [],
  beforeEach: [],
  afterEach: [],
};

let currentContext = null;

const beforeAll = (cb) => {
  if (currentContext) {
    currentContext.beforeAll.push(cb);
  }
};

const beforeEach = (cb) => {
  if (currentContext) {
    currentContext.beforeEach.push(cb);
  }
};

const afterEach = (cb) => {
  if (currentContext) {
    currentContext.afterEach.push(cb);
  }
};

const afterAll = (cb) => {
  if (currentContext) {
    currentContext.afterAll.push(cb);
  }
};

export { beforeAll, beforeEach, afterEach, afterAll };

Yup! That is all we need to do. We have our hooks. Now we need to implement the describe and spec functions.

Quick Detour

You might have noticed that we didn't check if our validate function gets an argument. We should do that and throw an error, if it doesn't.

// src/assertions.js
const validate = (actual) => {
  if (actual === undefined || actual === null) {
    throw new Error("Validation requires an actual value to be provided.");
  }

  // ... all other functions remain the same
};

describe

The describe function is going to take a name and a callback that contains the hooks and test cases. We are going to store the name and initialize our context with the defaultContext.

// src/hooks.js
import logger from "./logger.js";

// ... all other functions remain the same
const describe = (name, cb) => {
  try {
    currentContext = { ...defaultContext, name };
    cb();
  } catch (error) {
    logger.error("Failed to run test suite");
  } finally {
    logger.info(currentContext);
    currentContext = null; // cleanup
  }
};

export { beforeAll, beforeEach, afterEach, afterAll, describe };

Yeah, no. If we spread defaultContext into currentContext, we are going to eventually run into an issue where we are unable to clean up defaultContext properly because we are creating a shallow copy of currentContext. Any nested objects or arrays (like specs, beforeAll, etc.) are still references to the same objects in memory. This is going to prevent our tests from running in isolation.

Subsequent addition to the specs and hooks update the defaultContext object and when we try to mock a new instantation, we copy the previous state too. To fix this, we can either deep copy/clone the original object or create a function that returns a new object. The second is faster.

// src/hooks.js
const createFreshContext = () => ({
  name: "",
  specs: [],
  beforeAll: [],
  afterAll: [],
  beforeEach: [],
  afterEach: [],
});

// ... all other functions remain the same
const describe = (name, cb) => {
  try {
    currentContext = createFreshContext();
    currentContext.name = name;
    cb();
  } catch (error) {
    logger.error("Failed to run test suite");
  } finally {
    logger.info(currentContext);
    currentContext = null; // cleanup
  }
};

spec

The spec function is going to take a name and a test case as a callback. We are going to store the name and the test case in the currentContext.

// src/hooks.js

// ... all other functions remain the same
const spec = (name, cb) => {
  currentContext.specs.push({ name, cb });
};

export { beforeAll, beforeEach, afterEach, afterAll, describe, spec };

While we are not yet done, we are going to update our example test file with our new hooks.

// test/example.test.js
import validate from "../src/assertions.js";
import {
  beforeAll,
  describe,
  spec,
  beforeEach,
  afterAll,
  afterEach,
} from "../src/hooks.js";

describe("Example Test Suite 1", () => {
  beforeAll(() => {
    console.log("Setup before all tests");
  });

  beforeEach(() => {
    console.log("Setup before each test");
  });

  afterEach(() => {
    console.log("Cleanup after each test");
  });

  afterAll(() => {
    console.log("Cleanup after all tests");
  });

  spec("should fail", () => {
    validate(1 + 1).isEqual(3);
    validate({ a: 1 }).isEqual({ a: 1 });
  });

  spec("should pass", () => {
    validate({ a: 1 }).isEqual({ a: 1 });
  });
});

describe("Test Suite 2", () => {
  spec("should pass", () => {
    validate(3).isNotEqual({ a: 1 });
  });
});

Running this doesn't do anything, so what is missing and where do we go from here?

Next Steps

If you take a quick look at our features' checklist, the next thing we need to implement is the Test Reporter.

But we have a problem. Our tests aren't exactly running and we don't know if the tests are failing or passing. We need to figure out a way to run the tests and capture the output.

The next thing we have to do is to update our test runner.

Updating the Test Runner

We are going to rename the runTests function to main and we are going to create a new runTests function that is actually going to run each tests.

But first, we have to revisit the structure of our tests.

It is currently like this, Directory/Folder -> File -> Describe -> Spec. It is a many to many relationship. What does that mean? We can have multiple "specs" in a "describe" block and we can have multiple "describes" in a test file and so on.

We are already handling the Directory -> File and Describe -> Spec structures, so we are going to handle the File -> Describe. This is where we are going to add suites.

suites

We are going back to the hooks file. We are going to create a suites array and we are going to push each currentContext i.e. suite into the suites array.

// src/hooks.js

// ... all other functions remain the same
let currentContext = null;
let suites = [];

// ... all other functions remain the same
const describe = (name, cb) => {
  // ... all other functions remain the same
  finally {
    suites.push(currentContext);
    currentContext = null; // cleanup
  }
};

Nice nice. Now, we need to create a run function, which essentially goes through the suites array and runs all the hooks and specs. We are going to use a for loop to iterate through the suites, hooks and specs and we are going to wrap them a try/catch block.

Try to write the code for this. You can refer to the Execution order in case you need some help.

Done?

The first thing, we are going to do is create a getter and a setter for the suites array.

// src/hooks.js

// ... all other functions remain the same
const setSuites = (newSuites) => {
  suites = newSuites;
};

const getSuites = () => suites;

export {
  beforeAll,
  beforeEach,
  afterEach,
  afterAll,
  describe,
  spec,
  getSuites,
  setSuites,
};

Then we are going to create a run function in the testRunner.js file and import these two functions.

// src/testRunner.js

import { getSuites, setSuites } from "../hooks.js";

const run = async () => {
  const suites = getSuites();

  for (const suite of suites) {
    try {
      console.log("\n");
      logger.info(`Running suite: ${suite.name}`);
      console.group();

      // Run beforeAll hooks
      for (const hook of suite.beforeAll) await hook();

      console.log("");
      for (const spec of suite.specs) {
        try {
          logger.info(`Executing spec: ${spec.name}`);

          // Run beforeEach hooks
          for (const hook of suite.beforeEach) await hook();

          try {
            await spec.cb();
            logger.pass(`${spec.name}`);
          } catch (error) {
            logger.fail(`${spec.name}`, error);
          }

          // Run afterEach hooks
          for (const hook of suite.afterEach) await hook();
        } catch (error) {
          logger.fail(`${spec.name}`, error);
        }
        console.log("");
      }

      // Run afterAll hooks
      for (const hook of suite.afterAll) await hook();

      console.groupEnd();
    } catch (error) {
      logger.error("Error during suite execution", error);
    } finally {
      console.groupEnd();
    }
  }
};

Add the function just below the test file import in the testRunner file, and when you run this, you should see this in your terminal.

Running the tests

Just in case you are wondering why we didn't just export the suites array. In JavaScript, modules are evaluated only once when they are imported, and values that you export are cached. That means, if we change the suites array in the hooks files, our function isn't going to get the latest value. We are going to be getting a reference to the array as at the time of import, which would be an empty array.

Test Reporter

And we are finally here! Our test runner is up ~~and grateful~~. The final thing we are going to implement is the test reporter. If you have gotten here, you should realize how non-trivial this is.

We need two things.

A beautiful summary.
A way to keep track of results.

Test Results

We are going to add a stats object to our run function and keep track of total and failed tests and return thhat. In our main function, we are going to keep track of a summary object that contains more details about our tests.

// src/testRunner.js

const run = async () => {
  //...
  let stats = {
    total: 0,
    failed: 0,
  };

  //...
  for (const suite of suites) {
    try {
      //...
      for (const spec of suite.specs) {
        try {
          //...
          } catch (error) {
            stats.failed += 1;
            logger.fail(`${spec.name}`, error);
          }
          //...
        } catch (error) {
          logger.fail(`${spec.name}`, error);
        }
        stats.total += 1;
        console.log("");
      }
      //...
    }
    //...
  }
  // ...

  return stats;
};

const main = async () => {
  const testFiles = resolveTestFiles();

  const summary = {
    totalFiles: testFiles.length,
    filesWithErrors: 0,
    totalTests: 0,
    passed: 0,
    failed: 0,
  };

  if (testFiles.length > 0) {
    logger.info(`Found ${testFiles.length} test file(s)`);

    for (const testFile of testFiles) {
      logger.info(`Running test file: ${testFile}`);
      try {
        setSuites([]); // reset suites array for each import
        await import(path.resolve(testFile));
        const { total, failed } = await run();

        summary.totalTests += total;
        summary.failed += failed;
        summary.passed = summary.totalTests - summary.failed;
      } catch (error) {
        logger.error(
          `Failed to import test file: ${testFile}. Error: ${error.message}`
        );
        summary.filesWithErrors++;
      }
    }
  } else {
    logger.error("No test files found.");
    process.exit(1);
  }
};

Test Summary

Next up! The Test summary. We are going to create a new function in our logger object that receives our summary object and does some minor reporting.

// src/logger.js

const logger = {
  // ...
  summary: (summary) => {
    log("\n" + chalk.bold.underline("TEST SUMMARY"));
    log(chalk.cyan(`Files:`));
    log(chalk.cyan(`  Total:    ${summary.totalFiles}`));
    log(chalk.red(`  Errored:  ${summary.filesWithErrors}`));

    log(chalk.cyan(`\nTests:`));
    log(chalk.cyan(`  Total:    ${summary.totalTests}`));
    log(chalk.green(`  Passed:   ${summary.passed}`));
    log(chalk.red(`  Failed:   ${summary.failed}`));

    const passRate =
      summary.totalTests > 0
        ? ((summary.passed / summary.totalTests) * 100).toFixed(2)
        : 0;
    log(chalk.cyan(`\nPass Rate: ${passRate}%`));
    log("\n");
  },
};

And finally, we are going to call this function in our main function, just after the if/else block and you should see this in your terminal.

Test Results

I added an extra test file to the project to test our file resolution.

Conclusion

This has been a whole journey. We've explored how to build a functional testing framework from scratch. This is an insight into the inner working of a tool that we use everyday. A quick recap, we have built a testing framework with the following components:

File path resolution for discovering test files
A test runner to execute the tests
An assertion library for validating test results
Hook handlers for setup and teardown operations
A reporter for displaying test results

There are a lot more things a testing framework does, but this is a good start. Thanks for reading!

You can check out the full working implementation of this code on GitHub.

Notes

Using JSON.stringify to perform our isEqual and isNotEqual checks is not the best idea. It works well for simple objects but might not be sufficient for more complex data types like functions or nested objects.
You can extend this testing framework to include some other things like extensive assertions, watch mode, run tests in parallel, skipping tests etc. ~~I may or may not extend this framework to include these things.~~

Building a Testing Framework for JavaScript - Part Two

Recap

Hook Handlers

Things to note

Implementation

Quick Detour

describe

spec

Next Steps

Updating the Test Runner

suites

Test Reporter

Test Results

Test Summary

Conclusion

Notes

How to Build, Bundle, and Publish a JavaScript Package with CI/CD Automation

Building a Testing Framework for JavaScript - Part One