Update March 2019: A section was added to compare the most widely used testing libraries, TestCafe and Cypress
Creating applications and developing custom software for the medical and healthcare fields in Toronto demands high quality code and speedy development cycles. At Evenset, we use End-to-end (E2E) testing throughout app development to provide results quickly without breaking builds in the process.
Having E2E tests as part of a full test suite for our products is important to us at Evenset. Not only does it ensure that our applications are built to their design standards and run error free, E2E testing also allows for smoother development workflows via test automation. Integrating E2E tests with our Continuous Integration and Continuous Deployment pipelines allow us to catch more errors before shipping to production. This lets our developers leave behind their worry of breaking the application while providing them with valuable information about issues within their application’s usage that manual or unit tests may miss entirely.
E2E tests help us determine whether the flow of an application is as designed from start to finish. It simulates our users’ experiences and validates the results of their actions within the application. It helps us answer the questions “Is the experience what we expect it to be?” and “Is everything working together as we expect them to?”.
Nightwatch has its own implementation of the selenium WebDriver to perform commands and assertions on DOM elements. It provides its own testing framework with a test server, assertions, and tools.
- Flexible with frameworks
- Syntax looks the simplest and the most readable
- Built-in command-line test runner which can run the tests either sequentially or in parallel, together, by group, tags or single
- Supports many browsers and devices and can be used with SauceLabs or BrowserStack
- Free/open source
- No typescript support and in general, this library seems to be slightly less supported.
- Complicated environment configuration (Requires Selenium server and webdriver).
The main disadvantage of selenium-based tests is their flakiness (where tests fail erroneously), the complexity of setup and configuration, and slow speed. As nightwatch is a selenium-based framework, it also has these disadvantages.
Cypress injects tests into websites using the browser and runs them there. It comes bundled with a test runner that has powerful functionality and an optional web-based dashboard for use with continuous integration (such as Travis CI). The test runner can synchronously notify you of every single thing that happens inside the browser during your tests, as Cypress has native access to all objects within the application. Cypress allows writing of end-to-end tests while developing the app, focusing on test driven development.
- Clear documentation, more mature support options
- Easy debugging and logging of the test process with the test runner
- Has snapshot and video functionality (to record tests)
- Easy to write tests and to automate as well as integrate with CI
- Allows for parallel testing as well as group testing
- Open source, as long as you don’t need the dashboard
- No cross-browser support – only supports Chrome right now, but FireFox is in the works.
- No multi-tab support or multi-browser support – only 1 tab/browser at a time
- Native browser events are not fully supported – for example, file uploads are not supported in Cypress
Overall, Cypress has many upsides but also has some pitfalls. However, it is also very actively in development and many features requested by developers are constantly being added. There is a bright future for this framework that is worth following.
- Fast to set up
- Cross-browser and device support – supports many browsers and devices and can be used with SauceLabs or BrowserStack
- Parallel testing – TestCafe can run your tests on several browser instances at once.
- Convenient error reporting
- Written in modern JS or TypeScript
- Works with continuous integration
- Free/open source
- Young community since it is newer, less support (Went open source in 2015)
- Lack of good documentation
Screenster is a visual end-to-end and regression testing framework with tests done via a recording of web page interactions. Screenster records every action and user input and compares differences between releases. It is a useful software for automated UI regression testing.
- Easy to run, can create tests either with code or through recording of actions
- Can be used to easily spot differences and unexpected (or expected) changes in web UI
- Can add, change, delete steps in the tests as required
- It is standalone – easy to set up and is fully web-based
- Has a plugin for continuous integration
- Not the best tool for constantly changing, dynamic content
- No way to easily test API layer
- Can be slow on some pages
Screenster can be a very useful tool for UI testing, but has a distinct lack of control over back-end related tests. As such, it may be more useful as a regression framework than a true E2E framework.
In the end, we decided that Cypress was the best fit for our company based on our priorities of ease of use and code integration. Its ease of setup and seamless integration with our codebase as well as providing adequate testing functionality for our purposes solidify this decision.
TestCafe vs Cypress – Updated March 2019
The most immediately noticeable difference between the two frameworks is their browser support. While Cypress only works with Chrome and Chrome-based browsers, TestCafe works by serving the test site via a proxy server, and can therefore test in practically any browser including mobile. With the gradual rise of Firefox, we believe that testing on all browsers is becoming increasingly more necessary.
Running in Node vs Running in Browser
Cypress runs your test code in the actual browser, whereas TestCafe runs it in Node. Both have their advantages. Running the code in the browser means that Cypress can access real DOM elements, but you’re also limited to communication with your app via HTTP. TestCafe on the other hand can call out to parts of your Node server application directly from the test, which is useful for setting up / clearing database fixtures or starting and stopping the test server.
Some believe that an advantage of Cypress is its use of JQuery selectors, which have some extra capabilities such as
:class. TestCafe only uses CSS selectors. However, TestCafe has framework-specific selectors, libraries of selectors dedicated to popular frameworks like React and Angular, which allows components to be selectors.
To checkout some of our previous projects see: https://evenset.com/portfolio/ and reach out if you have any questions.