Evenset

End to end testing with javascript

End-to-end Testing with JavaScript

eventadmin

Update March 2019: A section was added to compare the most widely used testing libraries, TestCafe and Cypress

Creating applications and developing custom software for the medical and healthcare fields in Toronto demands high quality code and speedy development cycles. At Evenset, we use End-to-end (E2E) testing throughout app development to provide results quickly without breaking builds in the process.

Having E2E tests as part of a full test suite for our products is important to us at Evenset. Not only does it ensure that our applications are built to their design standards and run error free, E2E testing also allows for smoother development workflows via test automation. Integrating E2E tests with our Continuous Integration and Continuous Deployment pipelines allow us to catch more errors before shipping to production. This lets our developers leave behind their worry of breaking the application while providing them with valuable information about issues within their application’s usage that manual or unit tests may miss entirely.

E2E tests help us determine whether the flow of an application is as designed from start to finish. It simulates our users’ experiences and validates the results of their actions within the application. It helps us answer the questions “Is the experience what we expect it to be?” and “Is everything working together as we expect them to?”.

In today’s JavaScript ecosystem, there are a multitude of choices with regards to E2E frameworks. They are primarily divided into two main categories: Selenium WebDriver driven frameworks, and browser-based frameworks. In this post, we discuss four different options we looked at during our search for a framework to use here at Evenset: NightwatchJS, Cypress, TestCafe, and Screenster.

Nightwatch.js

http://nightwatchjs.org/

Nightwatch has its own implementation of the selenium WebDriver to perform commands and assertions on DOM elements. It provides its own testing framework with a test server, assertions, and tools.

Advantages:

  • Flexible with frameworks
  • Syntax looks the simplest and the most readable
  • Built-in command-line test runner which can run the tests either sequentially or in parallel, together, by group, tags or single
  • Supports many browsers and devices and can be used with SauceLabs or BrowserStack
  • Free/open source

Disadvantages:

  • No typescript support and in general, this library seems to be slightly less supported.
  • Complicated environment configuration (Requires Selenium server and webdriver).

The main disadvantage of selenium-based tests is their flakiness (where tests fail erroneously), the complexity of setup and configuration, and slow speed. As nightwatch is a selenium-based framework, it also has these disadvantages.

Cypress.io

https://www.cypress.io/

Cypress injects tests into websites using the browser and runs them there. It comes bundled with a test runner that has powerful functionality and an optional web-based dashboard for use with continuous integration (such as Travis CI). The test runner can synchronously notify you of every single thing that happens inside the browser during your tests, as Cypress has native access to all objects within the application. Cypress allows writing of end-to-end tests while developing the app, focusing on test driven development.

Advantages:

  • Clear documentation, more mature support options
  • Pure JavaScript (using test suites such as Mocha and Chai)
  • Easy debugging and logging of the test process with the test runner
  • Has snapshot and video functionality (to record tests)
  • Easy to write tests and to automate as well as integrate with CI
  • Allows for parallel testing as well as group testing
  • Open source, as long as you don’t need the dashboard

Disadvantages:

  • No cross-browser support – only supports Chrome right now, but FireFox is in the works.
  • Test code is being evaluated inside the browser with JavaScript, but not in Node.js or any other server side language – no connection/importing of server-side libraries or modules directly
  • No multi-tab support or multi-browser support – only 1 tab/browser at a time
  • Native browser events are not fully supported – for example, file uploads are not supported in Cypress

Overall, Cypress has many upsides but also has some pitfalls. However, it is also very actively in development and many features requested by developers are constantly being added. There is a bright future for this framework that is worth following.

TestCafe

https://devexpress.github.io/testcafe/

TestCafe works by serving the test site via a proxy server. It injects itself into the website as JavaScript scripts instead of controlling the browsers themselves like Selenium or Cypress does. This allows TestCafe to run on any browser, including on mobile devices, and have full control over the JavaScript execution loop.

Advantages:

  • Fast to set up
  • Cross-browser and device support – supports many browsers and devices and can be used with SauceLabs or BrowserStack
  • Parallel testing – TestCafe can run your tests on several browser instances at once.
  • Convenient error reporting
  • Written in modern JS or TypeScript
  • Works with continuous integration
  • Free/open source

Disadvantages:

  • Young community since it is newer, less support (Went open source in 2015)
  • Lack of good documentation

With better documentation, TestCafe can rival Cypress’s ability for ease of test writing and functionality. What sets TestCafe apart is their use of JavaScript’s aysnc/await syntax, cross-browser support, and fast test execution with parallelization.

Screenster

https://screenster.io/documentation/demo-automation-of-gmail-with-screenster/

Screenster is a visual end-to-end and regression testing framework with tests done via a recording of web page interactions. Screenster records every action and user input and compares differences between releases. It is a useful software for automated UI regression testing.

Advantages:

  • Easy to run, can create tests either with code or through recording of actions
  • Can be used to easily spot differences and unexpected (or expected) changes in web UI
  • Can add, change, delete steps in the tests as required
  • It is standalone – easy to set up and is fully web-based
  • Has a plugin for continuous integration

Disadvantages:

  • Not the best tool for constantly changing, dynamic content
  • No way to easily test API layer
  • Can be slow on some pages

Screenster can be a very useful tool for UI testing, but has a distinct lack of control over back-end related tests. As such, it may be more useful as a regression framework than a true E2E framework.

In the end, we decided that Cypress was the best fit for our company based on our priorities of ease of use and code integration. Its ease of setup and seamless integration with our codebase as well as providing adequate testing functionality for our purposes solidify this decision.

TestCafe vs Cypress – Updated March 2019

TestCafe and Cypress are the two of the biggest front runners for end-to-end Javascript testing in 2019. Despite their similarities mentioned above, these frameworks …

Browser Support

The most immediately noticeable difference between the two frameworks is their browser support. While Cypress only works with Chrome and Chrome-based browsers, TestCafe works by serving the test site via a proxy server, and can therefore test in practically any browser including mobile. With the gradual rise of Firefox, we believe that testing on all browsers is becoming increasingly more necessary.

Running in Node vs Running in Browser

Cypress runs your test code in the actual browser, whereas TestCafe runs it in Node. Both have their advantages. Running the code in the browser means that Cypress can access real DOM elements, but you’re also limited to communication with your app via HTTP. TestCafe on the other hand can call out to parts of your Node server application directly from the test, which is useful for setting up / clearing database fixtures or starting and stopping the test server.

Selectors

Some believe that an advantage of Cypress is its use of JQuery selectors, which have some extra capabilities such as :parent or :class. TestCafe only uses CSS selectors. However, TestCafe has framework-specific selectors, libraries of selectors dedicated to popular frameworks like React and Angular, which allows components to be selectors.

To checkout some of our previous projects see: https://evenset.com/portfolio/ and reach out if you have any questions.

you may also like

Subscribe to our newsletter

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

"*" indicates required fields

This field is for validation purposes and should be left unchanged.