Drupal Test Traits or Playwright: which one should you choose?

To test this blog I run two test suites (and yes, I like being able to improve things without wondering whether I am about to break everything). The first is a Drupal Test Traits suite, which runs its tests over HTTP without ever opening a browser, and the second is a Playwright suite, which runs its own in a real browser. Both have to pass before a branch gets merged. I wrote both of them myself, so there is nobody else to blame for what follows ;)

Every time I add a test, I have to answer the same question: which of the two suites should take it? DTT or Playwright? For a long time I answered by instinct. Then I wrote down, in plain words, the rule I thought I was already following, applied it to the tests I had, and found that 15 of my 28 browser test methods had no need of a browser at all. Worse: seven of them were checking, line for line, what a DTT test already checked. That check has a name in testing, an assertion, and I will use both words from here on.

In this post we'll see how to take the two decisions in the order we actually meet them. First: does this check (this assertion) need a browser? There is a rule for that one, I'll walk you through it below, and it sends most tests to DTT. Then, when a browser really is needed: which of the two suites should carry the assertion? That one is a matter of cost and of preference, and I'll give you mine. On the way I'll show you what those two decisions deleted from my own suite.

No time to read all this? Here is what I'd tell a teammate

First decision, and it is the one that counts. Ask it of the assertion you are writing: does it depend on something only a real browser produces? If it doesn't, DTT can assert (verify) it over HTTP, however long the visitor's journey. If it does, you need a browser, and then comes the second decision.

Three elements to test need a browser: the JavaScript that changes what the visitor sees, computed CSS and the viewport, and browser-level signals like console errors or focus order. Everything else was already produced (rendered) by Drupal before the page left the server, and DTT reads it over HTTP faster and with less to go wrong.

Now the reverse, and this is where I often see test suites lose their way: a browser can read your canonical tag or your JSON-LD, but we still don't need a browser to test them. What is generated on the server gets proved on the server with DTT.

So DTT over HTTP is my default, and a browser test is the exception I justify. Applying that to my own suite deleted more tests than it wrote, and the last section of this post is the damage report.

Second decision, for the assertions that really do need a browser. DTT can drive one too, through the ExistingSiteSelenium2DriverTestBase class of its JavaScript mode, so the choice here is between two browser tools (DTT's JavaScript mode or Playwright). If you already run one of the two, start with the one you have. But DTT's JavaScript mode wins one case outright: when the test has to build its own content in PHP and then click on it. Why? Because Playwright drives the same site, with the same database, but only from the outside, through the URLs a visitor can reach, so it would have to create that content from the outside by logging in, and clean it up the same way. Doable, but tedious and slower.

For everything else that needs a browser test, I reach for Playwright, and for two reasons. Playwright waits for the JavaScript by default: DTT's JavaScript mode does have waitForElementVisible(), but you have to place every wait yourself, and the assertion that follows it does not retry on its own, so a wait you forgot is a flaky test, a test that passes or fails depending on the day, and a flaky test makes us lose confidence in our own suite. The worst case there is. The second reason is that Playwright records a failed run itself. DTT can dump the HTML and take a screenshot too, but you have to call for them at the line where you expect an issue, while Playwright keeps the whole run and lets you step back through it.

That is the recipe. The rest of this post is why, and what it cost me.

What are these two test suites?

Drupal Test Traits, DTT from here on, runs PHPUnit against the site that is already installed. No reinstall, no test database built from scratch. The test opens the real local site, with its real content and its real configuration, and it reads what Drupal would send to a browser, no more.

By default it browses that site through BrowserKit, over HTTP: it follows links, submits forms, keeps cookies and sessions, honors redirects, and never runs a single line of JavaScript. DTT also ships a second driver, that is, another way of browsing the site, and that one does run JavaScript. I'll come back to it when the second decision needs it.

Two names for DTT in this post, then. DTT on its own means that default driver, over HTTP, the one that never opens a browser. When I mean the other one, I'll say DTT's JavaScript mode.

Playwright, for its part, also starts a real browser, loads the page, and lets you work with what that browser actually produced: computed styles, hover states, viewport widths, and the elements that exist only once a script has run.

There is a second difference between the two, and it is about what each one can build rather than about what each one can see.

A DTT test, in both modes, is PHP running inside Drupal, after the bootstrap, once Drupal is up, so it speaks the same language as the site and calls the same APIs. It can call createNode(), read a config value, check a cache tag, create a user with one precise role, or read a field back from the database after a save. The DTT test further down builds its own article, adds a French translation to it, and deletes both at the end of the test, which is what we call the tear-down.

A Playwright test is JavaScript running outside of Drupal, in Node. It sees the site from the outside, like a visitor: a URL, some HTML, a DOM. It can NOT code "create me a node", because it has no Drupal function within reach. To get content, it would have to log in and fill the form by hand, or go through drush, or through JSON:API. And nothing cleans up behind it.

So the tool cannot be what decides which suite a test belongs to. What decides is what we are really checking.

First decision: do we really need a browser?

Here is the rule, and it fits on one line.

Does the assertion (verification) depend on something only a real browser produces?

If it does, it needs a browser. If it doesn't, it belongs to DTT over HTTP, however long the visitor's journey. Fairly simple, isn't it?

So DTT is the default, and a browser test is the exception we justify.

Speed is not what decides, but it is what makes DTT the default. On this site a DTT test takes about a third of a second, and a Playwright test a little over a second: 115 DTT tests run in 45 seconds, while 21 browser tests spend 24 seconds between them, brought down to 7 by four parallel processes (four workers). Three or four times per test is not a gap that changes our day. What changes our day is what the gap is made of: no browser to start, no rendering to wait for, no element to poll until it becomes clickable, so no timing to get wrong. The fastest test is the one that cannot be flaky.

That last part is where I was wrong. I saw Playwright as the tool for the complete visitor journey: navigate to the home page, open an article, run a search, switch language. But the length of the visitor's journey is not a criterion. A six-page walk without a single line of JavaScript can just as well be a DTT test, and it will be faster and steadier than its browser version.

Three families of assertion need a real browser:

1. The JavaScript that changes what the visitor sees. A lightbox, an AJAX pager, a search facet refreshing in place, an autocomplete.
2. Computed CSS and the viewport. Colors resolved through the cascade, hover states, layout at a given width, an element a media query hides.
3. Browser-level signals. Console errors, a 404 on an asset, focus order under the keyboard, images that load only once they are scrolled into view.

Everything else is markup that Drupal has already sent, and DTT reads it very well.

The reciprocal matters just as much, and it is the half we skip too often. A browser can read a <head>, so it can check your canonical, your hreflang alternates, your JSON-LD, a 404, a sitemap. But that is not a reason to test with a browser. Those facts are produced on the server! So the test that checks them belongs where they are produced: DTT, cheaper to run, more precise when it fails, and it will not go red the day a browser updates.

And there is a level below both suites, one I use a lot. A single drush php-eval line in the terminal (with or without Claude). Nothing to keep, nothing to maintain, an answer in two seconds. A good half of what I want to check never gets past that step.

Second decision: a browser it is, but which tool drives it?

This is where DTT's second driver properly comes on stage. ExistingSiteSelenium2DriverTestBase extends the same base class as every other DTT test, so all the entity helpers stay available, and it speaks WebDriver with core's JSWebAssert on top: waitForElementVisible(), assertWaitOnAjaxRequest(). It runs JavaScript, it clicks, it waits. So the second decision is not about capability. It is about what each tool brings along, and about how each one behaves when a test goes wrong.

There is also a setup cost, and it is worth being precise about when it applies. WebDriver is a separate, permanent process, Chromedriver or Selenium on a port, with a version that has to match the browser it drives, and that is true locally and in CI alike. Playwright installs its browsers once and drives them from inside the test process, so there is nothing to keep alive and nothing to match. On DDEV it also means one more container to start and to upgrade, while `--headed` and the trace viewer work without wiring a display into a container, though that last one is a local comfort: in CI both suites run in a container anyway.

That cost is not universal, though. If your project already runs Selenium for Drupal core's JavaScript tests, the service is already declared, already pinned to a version and already understood by the team, so DTT's JavaScript mode adds nothing new to your stack and none of this should weigh in your decision. On this blog the arithmetic runs the other way: I maintain exactly one browser stack (Playwright), and adding a second one (DTT's JavaScript mode) for a handful of client-side assertions would double the upkeep without adding any coverage.

Honestly, though, managing stacks is not what decided it for me. The next two reasons did.

The first is the waiting model, and it counts more than it sounds. Playwright waits for an element to be ready to receive an action before it acts, and its assertions retry until the element is ready or until they time out. With DTT's JavaScript mode you have to write the wait explicitly, one call at a time, waitForElementVisible() or assertWaitOnAjaxRequest(), each with a ten-second default, and on top of that the Mink assertion that follows never retries: it looks once at the DOM in front of it and throws if the element has not been rendered yet. Both work. But only Playwright forgives you for picking the wrong moment, and picking the wrong moment is exactly where flaky browser tests are born.

The second is what a failure leaves behind. On this site the Playwright config has trace: 'retain-on-failure', so a failed run keeps a recording of the whole test: the DOM before and after each action, the network, the console, plus a video and a screenshot, and --ui mode to walk back through it afterward. DTT has captureScreenshot() and an HTML dump (a copy of the page saved to disk), and they are useful, but you have to place those calls yourself (apart from drupalGet(), which writes one on its own when HTML output is enabled, as it is here), so what you get is the state you predicted rather than the run you had. Playwright's viewport work is native too, while we are here: three widths in a loop is the whole responsive suite on this site.

Now the other side, because there is one. DTT's JavaScript mode has what Playwright structurally cannot have: the Drupal API in the same process. A DTT test (in both modes) is PHP running inside Drupal, remember, so it can set the test up and then click on what it has just built. createNode(), createUser(), a config value, a cache tag, all inside the test that is driving the browser. The day I want to check AJAX facets on a node I have just created, DTT's JavaScript mode will be the right tool.

A flaky test is far worse than a failing one

I have just spent a whole section on the waiting model and on failure reports, and here is why: to my mind, a flaky test is the worst thing that can happen to a test suite. A failing test teaches us something. A flaky one leads us to doubt a run whether it passes or fails, and trust in the suite is probably a tester's main responsibility.

Once that trust is gone we have two ways out, and both are bad. We delete the flaky test, and we lose the coverage it was giving us. Or, worse, we leave it in place and start compensating around it, adding tests to cover what we no longer believe. The suite is now slower without telling us anything new, and a slow suite only gets run every other time. A suite we run every other time is not a safety net. It is an illusion.

That is the whole reason I care so much about which tool checks what, and it applies to both decisions: avoid flaky tests. It is why I want the browser level in the hands of a tool that waits for the JavaScript on its own, like Playwright, because that cuts the number of flaky browser tests right down. And when I don't need a browser test at all, so much the better: a test that never opens a browser has no moment to get wrong.

I got it wrong on my own site

Writing the rule down was the easy part. Applying it to the suite I already had was a good deal less comfortable.

My Playwright browser suite held 28 test methods. Fifteen of them did not need a browser, and seven were checking, line for line, what a DTT test already checked: the contact form submission, the hero, the search results, the tags facet. I was paying for those twice, once in DTT and once in Playwright, and the browser tests were the slower and the more fragile of the two. Nothing in my process had ever compared a new test file (a spec) against the suite I already had, and since everything stayed green, nothing ever flagged it. What a good idea for a Claude Code Skill...

What is left of my Playwright tests today is 5 files and 13 test methods. Not bad, is it?

One of those 13 does more work than the rest: it sits inside a loop over three widths and three pages, so on its own it produces 9 of the tests that actually run. That is why the suite reports 21 tests where I wrote 13 methods.

A trap of the same shape, while we are here. My ddev phpunit command defaults to DTT's fast config, phpunit.dtt.fast.xml, and that is the right default: 105 tests in 33 seconds instead of 115 in 45, because its bootstrap skips Drupal core's test infrastructure. But skipping that infrastructure is exactly what stops it from running a Kernel test, so the tests it leaves out did not fail, they were never executed. The fast config is for the short loop (the smoke tests, for example), the full one is the gate we have to pass, though that is worth a discussion of its own and we'll have it in another post.

Recap

Two decisions, in that order. First: does the assertion depend on something only a real browser produces? If it does, and if it falls into one of the three families: the JavaScript that changes the page, computed CSS and the viewport, or browser-level signals, then it needs a browser test. For everything else, DTT checks it over HTTP without JavaScript, however long the visitor's journey. Facts that are produced on the server must be tested with DTT over HTTP even when a browser could read them, because that is where they are produced, and that is where a failure is legible.

Then: which of the two browser tools drives it? DTT can, through ExistingSiteSelenium2DriverTestBase, and it is the better tool the day a test has to build its own content and then click on it. For that level I prefer Playwright, because it waits by default, so fewer flaky tests, because a failed run records itself, and because neither of those costs me a service to host on this setup.

Applied to my own suite, those two decisions deleted more tests than they wrote: 15 of my 28 browser test methods moved down a level, and what is left is 21 tests I actually trust.

Thanks to Moshe Weitzman and to everyone working on Drupal Test Traits. Testing a site that is already installed, with its real content and its real configuration, changed the way I test Drupal, and like everything in Drupal, DTT changed my life!