Ortmesh logo


Write code that matters

Takeaways from a JavaScript Testing Conference

Posted19 Sep, 2019
Read Time3 mins
Updated29 Sep, 2019

Key nuggets of knowledge from my favourite JavaScript testing conference, AssertJS 2019.



Making upfront architectural decisions can be costly for developer productivity, especially in the later stages of the software development lifecycle. Hard lessons are learned through implementation and failure. The root cause? Unable to predict the future.
Solution? Start simple. Build incrementally. Refactor rather than over-engineer prematurely. Each refactor should add value and come from the developer better understanding the nuances of the codebase. Breakthroughs, that bring massive value to codebases, come after several small refactorings. Over these small refactors, good developers intuitively understand the flow of changes and anticipate the future direction to make large profitable changes.
As good as refactorings are, they introduce the risk of regression. In regression, previously working code breaks due to newer changes. Constant refactoring are critical for breakthroughs. For codebases to remain antifragile to refactorings, they must be built on a solid foundation of tests. Test driven development, because of the Red-Green-Refactor approach, make refactoring baked into the developer’s workflow. Thus, large downstream breakthroughs result from many small cycles of writing tests and refactorings.

A Culture of 800 Deployments per Day

Shopify, multi-channel ecommerce platform, is deploying to production 800 times a day with 83k requests per day using state-of-the-art devops tools and processes. Despite that, they emphasized that their tools and processes don’t matter.
Tools and processes help get the job done but the real driver for this magnitude of velocity is culture. Shopify has a culture that gives developers high autonomy and responsibility for their work.
They have no QA teams, because QA is the responsibility of the developer. Once the pull request gets the green light, the developer ensures the change makes it to production without breaking. They follow their changes closely via automated monitoring tools. Everyday, 3 developers in Shopify, stop their regular duties to monitor for anomalies in operation.
The company heavily relies on automation techniques to empower developers to take ownership of their codebase from development to production to operations. This all stems from the fact Shopify expects its developers to be fully responsible for their decisions and actions.

High Intensity Interval Testing

Humans are creatures of habit. They do most things in life subconsciously, even the complicated things if they become regular enough. Test-driven development (TDD) is a habit that must be developed, with exercise, practice, and understanding. TDD has to become an automatic and natural way of coding. It should just happen.
Here’s an exercise to master TDD in 3 steps. Write a small codebase to solve some trivial problem (i.e. build a calculator).
  1. Set a countdown timer for 3 minutes. Within 3 minutes, write a failing test and check it in. If one fails to write the test, reset the git branch and repeat this step. Test should show the correct failing message.
  2. Start another 3 minute timer to pass the test.
  3. Now, spend another 3 minutes to refactor the code.
Do these steps in a loop until all features are complete.

The Power of 100%

Aiming for 100% test coverage can pressure developers to write testable code. It reveals dark corners of the codebase where bugs are more likely to stem from.
This green colored 3 digit percentage shown after every commit in a devops pipeline becomes a social contract for the team to write testable code. Coupled with code reviews to maintain high test quality, 100% test coverage forces a way of thinking that yields robust software. The team, even new developers, become highly confident in modifying it for newer features and refactorings.

Integration Testing are Underrated

Integration testing is the grey area between unit tests and end-2-end (E2E) test.
  • Unit tests validate small components in full isolation
  • E2E test run the full running app with all its dependencies to test
  • Integration testing tests two or more connected parts of a system
Unit tests are cheap and fast to write and test but offer least confidence. E2E are expensive to write, test, and maintain but offer most confidence. Integration testing may seem like it’s right in the middle, but it provides the most value.
Here’s why integration testing is so valuable (especially against E2E):
  • written in TDD style while developing
  • logically split to focus on different areas of architecture (i.e. third-party API, databases, front-end or any combinations)
  • catches large issues earlier in the development lifecycle and faster to fix
  • offers great balance between test flakiness and confidence
Side node: it’s very productive to record or capture actual test fixtures for integration testing.

Science of Flakiness

E2E tests can sporadically pass or fail because they require many dependencies to work together in harmony to function. This sporadic behaviour referred to as flakiness. Flakiness can be caused by:
  • inconsistent assertion timing
    • race conditions between assertions and the rest of the test
    • add waits to asynchronous code before assertions
  • test depend on each other
    • recreate or build test fixtures between each test
  • E2E tests are flaky
    • reliance on large amount of dependencies
    • write more integration than E2E tests.
Flakiness can be measured and tracked. This helps developers reduce the root cause for failing builds. It helps to differentiate build failure between infrastructure issues and poorly written code
Flakiness detection techniques can include:
  • (most okay) recording the percentage of fixed number of passed tests vs. executed runs
  • (most intense) doing the above scenario many times
  • (most interesting) measuring the delta between code coverage to delete variations in code flow