Sisense's Journey to Automated Testing with Playwright

Introduction

In today's fast-paced software development landscape, ensuring the reliability and quality of web applications is paramount. Automated testing has become a cornerstone of the software development life cycle, enabling teams to quickly identify and fix issues, maintain high code quality, and deliver robust software products.

As our company embarked on the journey to enhance our testing strategy, we faced the critical decision of selecting the most suitable automation test framework. After careful consideration and evaluation of various options, we chose Playwright, a modern and versatile testing framework, paired with TypeScript to leverage its powerful static typing capabilities.

This article aims to delve into the rationale behind our decision, explore the benefits of Playwright over other frameworks, and provide insights into the structure of our test framework based on Playwright. By sharing our experience, we hope to offer valuable guidance to other teams seeking to optimize their testing processes with a robust and efficient automation solution.

Overview of Playwright and its Benefits

Playwright is an open-source, modern automation framework developed by Microsoft and maintained by the community. It is designed to enable reliable end-to-end testing for web applications across different browsers. Playwright supports all major browser engines, including Chromium, WebKit, and Firefox. Playwright is not only ideal for UI testing but also for API and component testing.

Playwright offers several key features and benefits that make it a powerful tool for web automation. Its auto-waiting mechanism ensures stable and reliable test results by intelligently waiting for elements to be ready before performing actions, reducing flakiness. Extensive debugging tools, including screenshot capture, video recording, and DOM state inspection, along with the standout tracing feature, make troubleshooting failing tests highly effective. Playwright is fast-growing and actively developed, with continuous improvements and feature additions from an active Microsoft-backed development team, keeping it at the forefront of web automation technology. One of the major advantages is its compatibility with TypeScript, providing static typing for early error detection and improved code maintainability. Our developers' proficiency in TypeScript further eases the adoption and integration of Playwright into our workflows, enhancing development efficiency and reducing the learning curve. Additionally, Playwright benefits from a vibrant community and ecosystem, with comprehensive documentation and an active GitHub repository for bug reports and improvement tickets.

Our company chose Playwright because of these compelling features and benefits, recognizing it as the best fit for our automation needs and future growth. We have adopted a cutting-edge automation framework that aligns with our goals of achieving high-quality, reliable, and maintainable automated tests. In the following sections, we will delve deeper into the reasons behind our decision and provide an overview of our test framework structure.

Structure of the Test Framework Based on Playwright

Our test framework built on Playwright is designed to be modular, scalable, and maintainable. It follows best practices in software engineering to ensure that tests are easy to write, understand, and maintain. Below is a detailed look at the architecture and components of our test framework.

Layer 1: Foundation Classes

Page Object classes: Abstractions for the web pages of the application under test. They contain locators and methods for interacting with UI elements.

Utils: Utility class methods or functions that provide common functionality used across different tests, such as data and file system manipulation and configuration handling.

Controller classes: Abstractions for handling API interactions. They manage API requests, providing methods to interact with the backend services.

Layer 2: Step Classes

UI Step classes: Encapsulate the sequences of actions performed on the UI using methods from the Page Object classes. They define specific user interactions, such as logging in or navigating through the application, etc.

API Step classes: Similar to UI Step classes, these encapsulate the sequences of API interactions using methods from the Controller classes and handle responses. They define workflows like creating a user, fetching data, etc.

Layer 3: Test Classes

UI System tests: High-level tests that verify the entire system's functionality from the user’s perspective. They utilize the UI Step classes to simulate real user scenarios and validate the application's behavior.

API System tests: High-level tests that validate backend service functionality. They utilize the API Step classes to perform end-to-end API testing.

API Component tests: Granular tests focusing on individual components or units of the API. They ensure that specific functionalities of the API work as expected, often used for testing isolated services or endpoints.

Top Layer: High-Level Tests

UI tests: High-level integration tests focused on user interface validation, ensuring that the application behaves correctly from the user’s perspective.

API tests: High-level integration tests focused on backend services validation, ensuring that the APIs function correctly and meet their specifications.

Data Flow and Interaction:

From Layer 1 to Layer 2: Page Object and Controller Classes provide the foundational methods and interactions that are used by UI Step and API Step Classes to define specific workflows and actions.

From Layer 2 to Layer 3: The Step classes in Layer 2 are used by the test classes in Layer 3 to perform comprehensive system tests. The Step classes help abstract the test logic, making tests more readable and maintainable.

Cross-layer Interactions: Utils can be accessed across all layers to provide common functionalities, while the dependencies between classes ensure a modular and reusable structure.

This architecture promotes maintainability, readability, and reusability of test code, aligning with the best practices of modern test automation frameworks.

Importance of Preconditions and Postconditions in Tests

Preconditions and postconditions play a critical role in ensuring that tests run smoothly and reliably. They help establish the necessary initial state before a test begins and clean up afterward to maintain system integrity and test isolation. This practice prevents side effects that could affect subsequent tests, ensuring each test runs in a consistent environment.

Set up required data assets: Using preconditions, we create the necessary data assets required for the test. This might include creating users, setting permissions, or preparing specific configurations. We utilize API Step classes to handle these setups efficiently. By using API calls, we can quickly set up the required state without navigating through the UI, saving time and reducing complexity.

Clean up after tests: Postconditions ensure that any data created during the test is removed afterward. This helps maintain a clean state in the system and prevents interference with other tests. API step classes are also used for cleanup activities, such as deleting users or removing test data. This approach ensures a thorough and efficient cleanup process.

Reducing UI Steps with API Steps is to optimize our tests. We use API steps to set up the test environment, reducing the number of UI interactions needed. This makes tests faster and less brittle, as they rely less on the UI's state and more on direct API interactions.

API Steps for Preparation:

Efficient Setup: By using API calls to set up the test environment, we can ensure that all necessary preconditions are met quickly and reliably.
Minimizing UI Interactions: Reducing the number of UI steps in the setup phase minimizes the chances of UI-related issues affecting the test.

Directory Structure and Organization

The framework is organized into a well-defined directory structure that promotes clarity and ease of navigation. A typical structure looks like this:

/src: Main source code for the test framework, including controllers, models, configurations, constants, pages, steps, and utilities.

/api: Controllers for API requests, and models for structuring data.

/controllers: Classes with the API requests for different end-points.

users.ts: Manages API interactions related to users.

resClient.ts: Likely a client for handling the API context for each API controller request that provides base URL and handles authentication.

/models: Interfaces and types for structuring data.

User.ts: Data model for a user, used for structuring API data.

/config: Environment-specific configurations.

env.config.ts: Configuration file for environment-specific settings.

/constants: Constant values used across the framework.

roleName.ts: Constants for role names used across the tests.

/fixtures: Predefined data for tests.

pages.fixtures.ts: Fixture data for the tests, such as predefined sets of data.

/pages: Page Object classes for different pages of the application.

adminPage.ts: Page Object class for the admin page, containing locators and methods specific to this page.

/steps: UI and API Step classes to encapsulate sequences of actions.

adminPage.steps.ts: UI step definitions for the admin page, utilizing the Page Object class to perform actions.
admin.api.steps.ts: API step definitions for admin-related API interactions, utilizing the controllers.

/utils: Utility functions and classes.

/tests: The actual test scripts are organized by feature and type (UI/API).

admin.test.ts: Test scripts for the admin UI and API separately, utilizing the UI and API steps.

playwright.config.ts: Configuration file for Playwright, defining settings such as browsers, timeouts, and test project configurations.

Page Object Model (POM) Structure

The POM pattern helps encapsulate the web page structure and behaviors, making tests more maintainable and reusable.

Structure of Page Object Classes:

Properties: Locators for web elements.

Methods: Primitive actions with web elements such as click, fill, isVisible, waitForState, etc.

The main goal is to create straightforward methods that perform a single action, avoiding the combination of multiple actions involving web elements within a single method. All Page classes should extend the BasePage class. When creating a new locator (and the corresponding method to interact with it), ensure that you prioritize finding a unique locator (via the DOM) and create a unique method/step for the UI element. If it is not possible to find a unique locator through the DOM, you can identify dynamic elements by their text. In such cases, create an abstract method and an abstract step that accepts the element's text as a parameter.

Controller Classes

Controller classes in Playwright serve as a centralized way to manage and interact with different API endpoints of an application. They encapsulate the logic for making API requests, ensuring that each request is performed consistently and securely. This approach promotes code reuse, maintainability, and clarity, especially when dealing with multiple API endpoints.

The example code snippet for the UsersV1 class demonstrates several key concepts:

Static properties are used to define the API endpoint URLs. This practice centralizes the endpoint definitions, making it easy to manage and update URLs.
Static methods encapsulate the logic for interacting with the API. These methods handle tasks like setting up the request context, making the API call, and returning the response object.
Methods like getAuthorizedContext are used to obtain an authorized API request context. This ensures that each API request is made with the necessary authentication and authorization headers.
The controller class provides methods for different types of API requests (e.g., GET, PATCH). This makes it easy to extend the class with new methods for additional endpoints or request types.

By following this structure, the controller classes ensure that all interactions with the API are managed in a clean, maintainable, and scalable way, enhancing the overall robustness of the test framework.

Page Object Step Classes

The concept of UI step classes is to encapsulate sequences of actions, combining methods from different Page Object classes to create higher-level functionalities. This approach ensures modularity and reusability of test steps. Here’s an example to illustrate this:

Key Points:

Modularity: Each method performs a specific sequence of actions, making it reusable and maintainable.
Readability: Combining methods from different page objects into a single step method provides a clear, high-level overview of the steps involved in a process.
Reusability: These steps can be used across different tests, reducing redundancy and improving test maintainability.

This approach leverages the modularity of Page Object classes while providing a higher level of abstraction for test steps, making tests easier to write, read, and maintain.

API Step Classes

The concept of API Step classes is to encapsulate API interactions into reusable and maintainable methods. These classes combine various API calls and actions into high-level operations that can be used in tests. Below is a code snippet demonstrating an API Step class and an explanation of its components and functionality:

Key Points

Encapsulation: API Step classes encapsulate API interactions into high-level methods that can be reused across different tests.
Reusability: Methods like userLogIn and getUserIdByName can be used in multiple tests, promoting code reuse and reducing redundancy.
Modularity: Each method performs a specific API interaction, making the code modular and easier to maintain.
Test Steps: Each method uses test.step to provide descriptive steps, making the test execution logs more readable and easier to debug.
Response Handling: Each method handles the response status code, ensuring the expected status is received.
Data Extraction: Methods extract data from the response body based on predefined models (e.g., User), ensuring consistent and type-safe data handling.

API Step classes like UsersAPISteps abstract the complexity of API interactions, providing a clean and organized way to perform high-level operations. This approach enhances the readability, maintainability, and reusability of test code, aligning with best practices in test automation. Each method ensures the response status code is as expected and extracts data from the response body based on models, promoting robust and reliable tests.

Test Structure in Playwright

In our Playwright-based test framework, each test scenario is organized within a separate describe section. This structure ensures that each test can have its own setup and teardown procedures, allowing it to create and delete its own data as needed for the scenario. This approach helps maintain test isolation and reliability.

The following code snippet illustrates how we structure our tests:

Each test scenario is enclosed in a description section. This ensures that related tests and their setup/teardown processes are grouped together.
The beforeEach hook sets up the necessary data and state before each test case runs. In this example, it creates an admin user and verifies their presence.
The afterEach hook cleans up the data and states after each test case runs. In this example, it deletes the admin user and verifies their deletion.
Individual test cases are defined within the description section. Each test case focuses on a specific scenario or functionality. In this example, the test case logs in as the admin user navigates through various pages and verifies the data source title and cube build result.

In the test definitions, the custom fixtures are used as part of the test context. This allows tests to use pre-configured Page Object step instances (dataPageSteps, dataSourcePageSteps) without needing to create them within each test. With fixtures, tests are simpler and more focused on the actual testing logic rather than using additional code to initialize instances. In our tests, we also use the userContext fixture, which initializes the user data required for the test, including all necessary information for the test to run properly.

By structuring our tests in this way, we ensure that each test scenario is self-contained, with its own data setup and teardown processes. This promotes test reliability and makes it easier to diagnose issues when tests fail, as each test operates independently of others.

Conclusion

We chose Playwright with TypeScript because it offers powerful features like cross-browser testing, intelligent auto-waiting, and comprehensive APIs for both UI and API testing. TypeScript's static typing and widespread use among developers made it an excellent match for our needs. Compared to our previous Serenity/Selenium setup, Playwright is more modern, reliable, and easier to maintain.

We encourage you to consider Playwright with TypeScript for your testing needs. Its powerful features, modern capabilities, and strong community support make it an excellent choice for building reliable and maintainable test automation frameworks. This structure will help ensure their tests are reliable, maintainable, and efficient, just like ours.

By adopting Playwright, you can benefit from improved test coverage, enhanced reliability, and increased developer productivity. Our sample framework can serve as a valuable starting point, helping you to create your own efficient and effective tests.

We invite you to share your feedback and experiences with us. Your insights can help us all learn and improve together. Feel free to reach out and let us know how Playwright with TypeScript is working for you and any tips or best practices you discover along the way. Let's collaborate to achieve higher-quality software and more efficient development processes.

References

Link to official Playwright and TypeScript documentation

Link to playwright test framework example