End-to-end Testing with JavaScript

Creating applications and developing custom software for the medical and healthcare fields in Toronto demands high quality code and speedy development cycles. At Evenset, we use End-to-end (E2E) testing throughout app development to provide results quickly without breaking builds in the process.

Having E2E tests as part of a full test suite for our products is important to us at Evenset. Not only does it ensure that our applications are built to their design standards and run error free, E2E testing also allows for smoother development workflows via test automation. Integrating E2E tests with our Continuous Integration and Continuous Deployment pipelines allow us to catch more errors before shipping to production. This lets our developers leave behind their worry of breaking the application while providing them with valuable information about issues within their application’s usage that manual or unit tests may miss entirely.

E2E tests help us determine whether the flow of an application is as designed from start to finish. It simulates our users’ experiences and validates the results of their actions within the application. It helps us answer the questions “Is the experience what we expect it to be?” and “Is everything working together as we expect them to?”.

In today’s JavaScript ecosystem, there are a multitude of choices with regards to E2E frameworks. They are primarily divided into two main categories: Selenium WebDriver driven frameworks, and browser-based frameworks. In this post, we discuss four different options we looked at during our search for a framework to use here at Evenset: NightwatchJS, Cypress, TestCafe, and Screenster.

Nightwatch.js

http://nightwatchjs.org/

Nightwatch has its own implementation of the selenium WebDriver to perform commands and assertions on DOM elements. It provides its own testing framework with a test server, assertions, and tools.

Advantages:

  • Flexible with frameworks
  • Syntax looks the simplest and the most readable
  • Built-in command-line test runner which can run the tests either sequentially or in parallel, together, by group, tags or single
  • Supports many browsers and devices and can be used with SauceLabs or BrowserStack
  • Free/open source

Disadvantages:

  • No typescript support and in general, this library seems to be slightly less supported.
  • Complicated environment configuration (Requires Selenium server and webdriver).

The main disadvantage of selenium-based tests is their flakiness (where tests fail erroneously), the complexity of setup and configuration, and slow speed. As nightwatch is a selenium-based framework, it also has these disadvantages.

Cypress.io

https://www.cypress.io/

Cypress injects tests into websites using the browser and runs them there. It comes bundled with a test runner that has powerful functionality and an optional web-based dashboard for use with continuous integration (such as Travis CI). The test runner can synchronously notify you of every single thing that happens inside the browser during your tests, as Cypress has native access to all objects within the application. Cypress allows writing of end-to-end tests while developing the app, focusing on test driven development.

Advantages:

  • Clear documentation, more mature support options
  • Pure JavaScript (using test suites such as Mocha and Chai)
  • Easy debugging and logging of the test process with the test runner
  • Has snapshot and video functionality (to record tests)
  • Easy to write tests and to automate as well as integrate with CI
  • Allows for parallel testing as well as group testing
  • Open source, as long as you don’t need the dashboard

Disadvantages:

  • No cross-browser support – only supports Chrome right now, but FireFox is in the works.
  • Test code is being evaluated inside the browser with JavaScript, but not in Node.js or any other server side language – no connection/importing of server-side libraries or modules directly
  • No multi-tab support or multi-browser support – only 1 tab/browser at a time
  • Native browser events are not fully supported – for example, file uploads are not supported in Cypress

Overall, Cypress has many upsides but also has some pitfalls. However, it is also very actively in development and many features requested by developers are constantly being added. There is a bright future for this framework that is worth following.

TestCafe

https://devexpress.github.io/testcafe/

TestCafe works by serving the test site via a proxy server. It injects itself into the website as JavaScript scripts instead of controlling the browsers themselves like Selenium or Cypress does. This allows TestCafe to run on any browser, including on mobile devices, and have full control over the JavaScript execution loop.

Advantages:

  • Fast to set up
  • Cross-browser and device support – supports many browsers and devices and can be used with SauceLabs or BrowserStack
  • Parallel testing – TestCafe can run your tests on several browser instances at once.
  • Convenient error reporting
  • Written in modern JS or TypeScript
  • Works with continuous integration
  • Free/open source

Disadvantages:

  • Young community since it is newer, less support (Went open source in 2015)
  • Lack of good documentation

With better documentation, TestCafe can rival Cypress’s ability for ease of test writing and functionality. What sets TestCafe apart is their use of JavaScript’s aysnc/await syntax, cross-browser support, and fast test execution with parallelization.

Screenster

https://screenster.io/documentation/demo-automation-of-gmail-with-screenster/

Screenster is a visual end-to-end and regression testing framework with tests done via a recording of web page interactions. Screenster records every action and user input and compares differences between releases. It is a useful software for automated UI regression testing.

Advantages:

  • Easy to run, can create tests either with code or through recording of actions
  • Can be used to easily spot differences and unexpected (or expected) changes in web UI
  • Can add, change, delete steps in the tests as required
  • It is standalone – easy to set up and is fully web-based
  • Has a plugin for continuous integration

Disadvantages:

  • Not the best tool for constantly changing, dynamic content
  • No way to easily test API layer
  • Can be slow on some pages

Screenster can be a very useful tool for UI testing, but has a distinct lack of control over back-end related tests. As such, it may be more useful as a regression framework than a true E2E framework.

In the end, we decided that Cypress was the best fit for our company based on our priorities of ease of use and code integration. Its ease of setup and seamless integration with our codebase as well as providing adequate testing functionality for our purposes solidify this decision.

To checkout some of our previous projects see: https://evenset.com/portfolio/ and reach out if you have any questions.

 

2 thoughts on “End-to-end Testing with JavaScript”

Post Your Comments

Your email address will not be published. Required fields are marked *