Let's explore several options for bulding a test automation framework in all layers.
For CI/CD Integration and setting up pipelines, there are several options. Jenkins, Github Actions and Gitlab Runner are the most commonly used platforms.
For automating tests, depending on the programming language, there are multiple ways to choose. Focusing on JS, Jest, Mocha and Jasmine are possibly the best options. Mocha can be used with Chai assertion library, where the other two already have assertion pacakges inside.
Besides automating tests with Selenium on top of one of these, an E2E test framework can be choosen. The most powerful candidates are Playwright, Cypress and TestCafe. A table summarizing alternatives would look like:
E2E Test Frameworks
To be able to see the capabilities and how convenient each of them, I implement the same scenario with several JS based E2E frameworks. After the scenario is automated, can be executed via:
Playwright : playwright test testffolder --repeat-each=15 TestCafe : testcafe chrome :headless testffolder Nightwatch : nightwatch -e chrome –headless ./nightwatch/specs/*.js Cypress : cypress open --browser chrome run --spec "**/SearchButton.spec.js" Selenium + Jest : jest --config=selenium/tests/config.prod.js
Pro's and con's:
Playwright : (+) has device emulators Playwright : (+) multiple & parallel runs Playwright : (+) fast: using Browser content rather than browser itself (background: Playwright uses a WebSocket connection rather than the WebDriver API and HTTP.) Cypress : (-) does not support safari -> webkit support is just introduced but with Playwright Cypress : (+) has dashboard Cypress : (+) supports tags Nightwatch : (+) Fast Nightwatch : (+) Understandable Nightwatch : (-) Poor documentation Selenium : (-) Interceptions not easy Metacafe : (-) To type in text field, clicks
Execution durations are (in msec):
Overall, for me Cypress's dashboard is the biggest weapon for them. If Safari automation is needed, there are two options: Playwright and TestCafe. Not only for this, but for ease of use and performance, Playwright is my favorite. Then, Cypress in case of no Safari need, otherwise TestCafe.
Snapshots from test classes in each framework: