grafana/contribute/style-guides/e2e-playwright.md

End-to-end tests

Grafana Labs uses a minimal homegrown solution built on top of Playwright for its end-to-end (E2E) tests.

Important notes:

• We generally store all element identifiers (CSS selectors) within the framework for reuse and maintainability.
• We generally do not use stubs or mocks as to fully simulate a real user.
• We also use Playwright for the plugins' end-to-end tests.

Framework structure

Our framework structure is inspired by Martin Fowler's Page Object.

• Selector: A unique identifier that is used from the E2E framework to retrieve an element from the browser
• Page: An abstraction for an object that contains one or more Selector identifiers with the visit function to go to the page.
• Component: An abstraction for an object that contains one or more Selector identifiers but without the visit function
• Flow: An abstraction that contains a sequence of actions on one or more Page abstractions that can be reused and shared between tests

How to run the Playwright tests:

Note: If you're using VS Code as your development editor, it's recommended to install the Playwright test extension. It allows you to run, debug and generate Playwright tests from within the editor. For more information about the extension and how to use reports to analyze failing tests, refer to the Playwright documentation.

Each version of Playwright needs specific versions of browser binaries to operate. You need to use the Playwright CLI to install these browsers.

yarn playwright install chromium

The following script starts a Grafana development server (same server that is being used when running e2e tests in CI) on port 3001 and runs all the Playwright tests. The development server is provisioned with the devenv dashboards, data sources and apps.

yarn e2e:playwright

You can run against an arbitrary instance by setting the GRAFANA_URL environment variable:

GRAFANA_URL=http://localhost:3000 yarn e2e:playwright

Note this will not start a development server, so you must ensure that Grafana is running and accessible at the specified URL.

Commands commonly used:

1 - To open the Playwright UI. It starts the Grafana server and then Playwright, which runs against this server.

yarn e2e:playwright --ui

2 - To run an individual test. It will run the test that matches the string passed to grep. Playwright will run all of them if you use a string that matches multiple tests.

yarn e2e:playwright --grep <testname>

You can also run all the tests matching a specific tag with @tagName.

yarn e2e:playwright --grep @<tagname>

3 - To run a project. It will run the entire project. You can find them in grafana/playwright.config.ts.

yarn e2e:playwright --project <projectname>

4- To open the last HTML report. It will open a Chrome window with the test list and the related info (success/error, name, time, steps, ...).

yarn playwright show-report

You can open an arbitrary report with yarn playwright show-report <reportLocation>. The reports are also downloadable from CI by:

• Clicking through to End-to-end tests/All Playwright tests complete.
• Clicking Summary.
• Download the playwright-html- artifact.
• Unzip.
• Run yarn playwright show-report <reportLocation>

You can see the full list in the Playwright documentation if you are curious about other commands.

How to create an end-to-end Playwright test

Basic example

Let's start with a simple JSX example containing a single input field that we want to populate during our E2E test:

<input className="login-form-input" type="text" />

It is possible to target the field with a CSS selector like .login-form-input. However, doing so is a brittle solution because style changes occur frequently.

Furthermore, there is nothing that signals to future developers that this input is part of an E2E test. At Grafana, we use data-testid attributes as our preferred way of defining selectors. See Aria-Labels vs data-testid for more details.

<input data-testid="Username input field" className="login-form-input" type="text" />

The next step is to add the Login page to the versionedPages export within <repo-root>/packages/grafana-e2e-selectors/src/selectors/pages.ts so that it appears when we type selectors.pages in your IDE.

export const versionedPages = {
  [...]
  Login: {
    url: {
      [MIN_GRAFANA_VERSION]: '/login',
    },
    username: {
      '10.2.3': 'data-testid Username input field',
      [MIN_GRAFANA_VERSION]: 'Username input field',
    },
    password: {
      '10.2.3': 'data-testid Password input field',
      [MIN_GRAFANA_VERSION]: 'Password input field',
    },
    submit: {
      '10.2.3': 'data-testid Login button',
      [MIN_GRAFANA_VERSION]: 'Login button',
    },
    skip: {
      '10.2.3': 'data-testid Skip change password button',
    },
  },
[...]

In this example, the username selector is prefixed with data-testid and specifies the minimum Grafana version for which the value is valid. The prefix is a signal to the framework to look for the selector in the data-testid attribute.

Now that we have a page called Login in our versionedPages const, use it to add a selector in our HTML as shown in the following example. This page really signals to future developers that it is part of an E2E test.

Example:

import { selectors } from '@grafana/e2e-selectors';

<input data-testid={selectors.pages.Login.username} className="login-form-input" type="text" />;

The last step in our example is to use our Login page as part of a test.

• Use the url property whenever you call the page.goto() in Playwright.
• Access any defined selector from the Login page by invoking it: page.getByTestId().

test(
  'Can login successfully',
  {
    tag: ['@acceptance'],
  },
  async ({ selectors, page, grafanaAPICredentials }) => {
    test.skip(grafanaAPICredentials.password === 'admin', 'Does not run with default password');

    await page.goto(selectors.pages.Login.url);

    await page.getByTestId(selectors.pages.Login.username).fill(grafanaAPICredentials.user);
    await page.getByTestId(selectors.pages.Login.password).fill(grafanaAPICredentials.password);

    await page.getByTestId(selectors.pages.Login.submit).click();

    await expect(page.getByTestId(selectors.components.NavToolbar.commandPaletteTrigger)).toBeVisible();
  }
);

Advanced example

Let's take a look at an example that uses the same selector for multiple items in a list for instance. In this example app, there's a list of data sources that we want to click on during an E2E test.

<ul>
  {dataSources.map(({ id, name }) => (
    <li className="card-item-wrapper" key={id}>
      <a className="card-item" href={`datasources/edit/${id}`}>
        <div className="card-item-name">{name}</div>
      </a>
    </li>
  ))}
</ul>

Like in the basic example, start by creating a page abstraction:

export const versionedPages = {
[...]
  DataSources: {
    url: {
      [MIN_GRAFANA_VERSION]: '/datasources',
    },
    dataSources: {
      [MIN_GRAFANA_VERSION]: (dataSourceName: string) => `Data source list item ${dataSourceName}`,
    },
  },
[...]
};

You might have noticed that instead of a simple string as the selector, there's a function that takes a string parameter as an argument and returns a formatted string using the argument.

Just as before, you need to add the DataSources url to the exported const versionedPages in packages/grafana-e2e-selectors/src/selectors/pages.ts.

The next step is to use the dataSources selector function as in the following example:

<ul>
  {dataSources.map(({ id, name }) => (
    <li className="card-item-wrapper" key={id}>
      <a className="card-item" href={`datasources/edit/${id}`}>
        <div className="card-item-name" data-testid={selectors.pages.DataSources.dataSources(name)}>
          {name}
        </div>
      </a>
    </li>
  ))}
</ul>

When this list is rendered with the data sources with names A, B and C ,the resulting HTML looks like this:

<div class="card-item-name" data-testid="data-testid Data source list item A">A</div>
<div class="card-item-name" data-testid="data-testid Data source list item B">B</div>
<div class="card-item-name" data-testid="data-testid Data source list item C">C</div>

Now we can write our test. The one thing that differs from the previous basic example is that we pass in which data source we want to click as an argument to the selector function:

test(
  'List test clicks on data source named B',
  {
    tag: ['@datasources'],
  },
  async ({ selectors, page }) => {
    await page.goto(selectors.pages.Datasources.url);
    // Select datasource
    const dataSource = 'B';
    await page.getByTextId(dataSource).click();
  }
);

Playwright for plugins

When end-to-end testing Grafana plugins, a best practice is to use the @grafana/plugin-e2e testing tool. The @grafana/plugin-e2e tool extends @playwright/test capabilities with relevant fixtures, models, and expect matchers. Use it to enable comprehensive end-to-end testing of Grafana plugins across multiple versions of Grafana.

Note: To learn more, refer to our documentation on plugin development and end-to-end plugin testing.

Add end-to-end tests for a core plugin

You can add Playwright end-to-end tests for plugins to the e2e-playwright/plugin-e2e directory.

1. Add a new directory that has the name as your plugin here. This is the directory where your plugin tests will be kept.

2. Playwright uses projects to logically group tests together. All tests in a project share the same configuration. In the Playwright config file, add a new project item. Make sure the name and the testDir subdirectory match the name of the directory that contains your plugin tests. Add 'authenticate' to the list of dependencies and specify 'playwright/.auth/admin.json' as the storage state to ensure that all tests in your project will start already authenticated as an admin user. If you want to use a different role for and perhaps test RBAC for some of your tests, refer to our documentation.

   {
      name: 'mysql',
      testDir: path.join(testDirRoot, '/mysql'),
      use: {
        ...devices['Desktop Chrome'],
        storageState: 'playwright/.auth/admin.json',
      },
      dependencies: ['authenticate'],
    },
   

3. Update the CODEOWNERS file so that your team is owner of the tests in the directory you added in step 1.

Commands

• yarn e2e:playwright runs all Playwright tests. Optionally, you can provide the --project mysql argument to run tests in a specific project.

  The yarn e2e:playwright command starts a Grafana development server on port 3001 and runs the Playwright tests.

  You can run against an arbitrary instance by setting the GRAFANA_URL environment variable:

  GRAFANA_URL=http://localhost:3000 yarn e2e:playwright

  Note this will not start a development server, so you must ensure that Grafana is running and accessible at the specified URL.

• You can provision the development server with the devenv dashboards, data sources, and apps.

Playwright tests in Grafana Enterprise

We are currently working on make Playwright available for creating end-to-end tests in grafana-enterprise.