End-to-End Testing

End-to-End Testing #

This page documents the end-to-end (e2e) testing setup for the BonsAI application using Playwright.

Overview #

The e2e test suite verifies application functionality from a user perspective, simulating real user interactions and workflows in a browser environment. Tests are written using TypeScript and the Playwright test framework.

Directory Structure # 

tests/e2e/
├── src/                   # Source code for tests
│   ├── .auth/             # Authentication state storage
│   ├── common/            # Common utilities and data
│   ├── configs/           # Test configuration
│   ├── core/              # Core test functionality
│   ├── helpers/           # Helper functions and utilities
│   ├── pages/             # Page Object Models
│   └── tests/             # Test specifications
├── screenshots/           # Test screenshots
├── playwright-report/     # HTML test reports
├── test-results/          # Test execution results
├── playwright.config.ts   # Playwright configuration
└── tsconfig.json          # TypeScript configuration

Setup #

Prerequisites #

  • Node.js v22.14.0 or later
  • PNPM v10.4.1

Installation # 

# From the monorepo root
mise run e2e-init

# Or manually
pnpm e2e install
pnpm e2e exec playwright install --with-deps chromium

Pull Test Data # 

mise run e2e-data-pull

Running Tests # 

# From the monorepo root
mise run e2e
mise run e2e --headed  # Run with browser visible
mise run e2e --debug   # Run with Playwright Inspector for debugging

# Or manually
doppler --no-check-version run -- pnpm e2e exec playwright test

Viewing Reports # 

# From the monorepo root
mise run e2e-report

# Or manually
pnpm e2e exec playwright show-report

Test Architecture #

Page Object Model #

Tests use the Page Object Model pattern to encapsulate page interactions and provide a clean API for test scripts. Page objects are located in the src/pages/ directory.

Example:

// Using a page object
const loginPage = new LoginPage(page);
await loginPage.navigate();
await loginPage.login(user);

Authentication #

Tests that require authentication use Playwright’s storage state feature. The auth setup creates a logged-in state that other tests can reuse:

  1. auth.setup.ts performs the login and saves the authentication state
  2. Test configurations use this state via the storageState property

Custom Test Context #

A custom test context (src/core/test.ts) handles specific requirements like Cloudflare Access headers:

test("Example test", async ({ ctx }) => {
  // ctx is a custom browser context with proper headers
  const page = await ctx.newPage();
  // ...
});

Test Configuration #

The playwright.config.ts file contains the following configuration:

  • Test timeouts: 30 seconds for tests, actions, navigation, and assertions
  • Browser: Currently configured for Chromium
  • Screenshots: Captured on test failure
  • Traces: Retained on failure for debugging

Writing Tests #

Tests should be organized by feature or workflow in the src/tests/ directory.

Example test structure:

import { test, expect } from "@core/test";

test.describe("Feature Name", () => {
  test("should perform expected action", async ({ ctx }) => {
    const page = await ctx.newPage();
    // Test steps...
    await expect(page).toHaveURL(/expected-url/);
  });
});

Best Practices #

  1. Use page objects for UI interactions
  2. Keep tests independent and isolated
  3. Include meaningful screenshots at key points
  4. Use descriptive test names that explain the expected behavior
  5. Prefer test helpers and abstractions over repetitive code

Environment Variables #

  • WEBAPP_HOST: Host for the application (defaults to localhost:3000)
  • CLOUDFLARE_ACCESS_CLIENT_ID: Cloudflare Access client ID
  • CLOUDFLARE_ACCESS_CLIENT_SECRET: Cloudflare Access client secret

Test Data Management with DVC #

The e2e tests use Data Version Control (DVC) to manage test data files. This approach allows us to version control large files without storing them in Git.

How DVC Works in Our Setup #

  1. Test data files are stored in the tests/e2e/test-data/ directory
  2. DVC tracks these files in test-data.dvc, which is committed to Git
  3. The actual data files are stored in an S3 bucket (bonsai-e2e-test-data)
  4. When you run tests, DVC ensures the right files are available locally

Setting Up AWS Credentials #

To access the DVC remote storage in S3, you need valid AWS credentials:

# Configure your AWS credentials
aws configure

You’ll need to provide:

  • AWS Access Key ID
  • AWS Secret Access Key
  • Default region name (eu-central-1)
  • Default output format (optional)

For temporary credentials (access keys starting with “ASIA”), you’ll also need to set up your AWS session token:

echo "aws_session_token = YOUR_SESSION_TOKEN" >> ~/.aws/credentials

Common DVC Commands #

All DVC commands should be run from the tests/e2e directory:

# Pull test data files from remote storage
poetry run dvc pull

# Check status of your DVC-tracked data
poetry run dvc status

# After adding new test files, track them with DVC
poetry run dvc add test-data

# Push tracked files to remote storage
poetry run dvc push

# Force operations if needed
poetry run dvc pull -f
poetry run dvc push -f

DVC Workflow for Adding New Test Data #

  1. Add your test files to the test-data directory
  2. Track them with DVC: poetry run dvc add test-data
  3. Commit the changes to Git: git add test-data.dvc
  4. Push the data to remote storage: poetry run dvc push
  5. Push the Git commit as normal

Troubleshooting DVC Issues #

  • Permission errors: Verify AWS credentials are correct and have access to the S3 bucket
  • Pull/push failing: Check network connectivity and firewall settings
  • Empty files: Ensure files were properly added and pushed to the remote

Troubleshooting #

  • Test failures: Check the HTML report for screenshots, traces, and error details
  • Authentication issues: Ensure Cloudflare credentials are set correctly
  • Timeouts: Consider increasing timeout values in playwright.config.ts

Adding New Tests #

When adding new tests, follow these guidelines:

  1. Create a new test file in src/tests/ with a descriptive name ending in .spec.ts
  2. For UI interactions, create or extend page objects in src/pages/
  3. For common data or utilities, add them to the appropriate directory under src/
  4. Add test screenshots at critical points with descriptive names
  5. Run tests locally before submitting changes

CI Integration #

The e2e tests are integrated into our CI pipeline, running on:

  • Pull requests to main
  • Scheduled runs against staging and production environments

CI runs are configured to:

  • Run tests in parallel
  • Upload test artifacts (reports, screenshots, traces)
  • Notify the team on failures