An untested codebase does not need a new architecture to gain reliable release checks. It needs a small, owned contract around behavior the business cannot afford to break. For a CTO choosing between an internal test stack and a vendor, my position is to build the first boundary tests in-house, then buy execution or maintenance capacity only where the pilot exposes a real constraint.
Buying test automation before defining ownership buys ambiguity
The build-versus-buy decision is often framed as a choice of testing tools. It is more consequential to decide who specifies expected behavior, creates test data, diagnoses failures, and removes obsolete checks. A vendor can supply browsers, dashboards, and support; it cannot reliably infer whether an unusual legacy response is a bug or a contractual behavior, because that knowledge usually lives with the people operating the system.
For a web application, compare Playwright maintained by your team with mabl as a managed testing platform. Playwright wins when engineers can edit the repository and need tests to travel with application changes, because selectors, fixtures, and assertions can be reviewed in the same pull request as the code. Its cost is internal time spent maintaining runners, browser versions, data setup, and flaky tests. Mabl can win when the team lacks browser-test operations capacity or needs broader participation in authoring checks; its cost includes subscription fees, platform-specific workflows, and the work of keeping recorded journeys aligned with the product. Neither option transfers responsibility for deciding what a passing checkout, account change, or export actually means.
I would not sign a platform contract on the promise that it will “cover the legacy system,” because coverage without an agreed failure response produces dashboards rather than a release decision. Ask the proposed owner of each test what happens when it fails at 4 p.m. on a deployment day: who distinguishes an application regression from expired data, who can repair the test, and who is authorized to hold the release? An unanswered question there is a purchasing requirement, not a testing detail.
Begin with one workflow whose failure has an identifiable consequence, and name an engineering owner and a business-side confirmer of expected behavior. Treat the tactical checklist Add test automation to legacy code without a rewrite QA playbook as a way to start that pilot, not as a purchasing decision, because implementation steps cannot establish whether your organization or a supplier will maintain the result. Put ownership in the pilot agreement before comparing product demonstrations.
Testing at stable boundaries avoids turning the pilot into a rewrite
Legacy internals make tempting targets because developers can see every branch they wish had a unit test. They are often the wrong first target: adding unit tests throughout tightly coupled code may require changing constructors, database access, and global state before a single release risk is reduced. Start at a boundary that already exists, such as an HTTP endpoint, a message consumer, or a user journey. The initial assertion should describe an externally observable result while leaving the underlying implementation alone.
For an HTTP boundary, use an OpenAPI 3.1 document to record the request and response shape that clients actually rely on. If a downstream service is controlled by another team, Pact v4 can make consumer expectations explicit; it is preferable to a hand-maintained mock when both teams will run contract verification, because the provider can then check the same expectations. WireMock is useful for isolating an unreliable dependency during a test, but its stub must be reviewed against real responses or it can faithfully confirm an invented behavior. These tools answer different questions; adding all of them to the first pilot would create maintenance work without clarifying the release decision.
Choose test data with the same care as assertions. A dedicated account and resettable records are preferable to a shared “test user,” because parallel runs can otherwise change each other’s state. Where a workflow writes to PostgreSQL, a disposable database seeded from a small fixture can make reruns repeatable. Do not copy a production database merely to make setup quick, because personal data, secrets, and unexplained historical states make both security review and failure diagnosis harder. If production-shaped data is indispensable, require a documented sanitization process and restrict who can access the resulting environment.
The risk-first memo How QA can add safety nets to legacy code without rewriting belongs in the kickoff alongside an inventory of dependencies, because a safety net is useful only when the team knows which external behavior it protects. For each proposed check, write down its boundary, fixture owner, expected result, and likely failure causes. That record gives a vendor a testable scope and gives an internal team a stopping point: the pilot is complete when it can identify a meaningful regression, not when every old module has been touched.
A narrow CI gate produces evidence without freezing every release
Make the first automated check executable from a clean checkout before connecting it to deployment. The following shell commands require Node.js and a machine that can run Chromium. They create a minimal Playwright smoke test against a public URL; replace the URL and assertion with an owned staging workflow before treating the result as release evidence.
npm init -y
npm install -D @playwright/test
npx playwright install chromium
mkdir -p tests
cat > tests/smoke.spec.js <<'EOF'
const { test, expect } = require('@playwright/test');
test('entry point responds', async ({ page }) => {
const response = await page.goto(process.env.APP_URL);
expect(response.status()).toBeLessThan(500);
});
EOF
APP_URL=https://example.com npx playwright test tests/smoke.spec.js --workers=1
The example deliberately checks little: a server can return a login page or an application error below HTTP 500 and still pass. Its purpose is to prove that a check can run independently of the legacy application’s internals. Next, assert a business result, such as a saved change appearing after a fresh login, and capture enough context to diagnose failure. A W3C Trace Context traceparent header can connect an HTTP test to OpenTelemetry traces where the application already propagates it; do not add tracing solely to compensate for an assertion that says too little.
In GitHub Actions, pin action versions such as actions/checkout@v4 and install dependencies from the committed lockfile with npm ci, because otherwise a changed dependency can alter the test runner without a code review. Keep retries disabled during diagnosis: Playwright’s –retries=0 exposes intermittent failures rather than making a green result depend on another attempt. Once the source of flakiness is understood, a retry may help distinguish a transient infrastructure problem, but it should not erase the original failure from reporting.
Set an initial runtime budget of 12 minutes for the blocking suite; this is a value to tune against your deployment cadence, not a universal benchmark. Make 3 critical journeys the pilot scope, a proposed limit that forces the team to choose consequences over easy-to-automate pages. Track the observed flake rate as runs that fail and then pass unchanged, rather than counting every red build as an application defect. Use a 1% provisional tolerance as a decision trigger for investigating or removing an unstable gate, not as permission to ignore failures. GitHub Actions run history can supply the counts, while Prometheus histograms can show whether test-environment latency is driving timeouts; neither metric tells you why a particular assertion failed.
Do not block deployment with the entire new suite on its first day, because unclassified failures make teams learn to bypass the gate. Run it as a visible, non-blocking check until the owner can explain failures and restore a broken fixture. Then promote only the dependable, high-consequence checks to required status. Record any override with its reason and follow-up owner, so a release exception remains an exception rather than an undocumented operating procedure.
A vendor pilot should earn renewal by reducing operational work
A procurement pilot needs an exit test as well as a product demonstration. Give the vendor and the internal option the same staging workflow, the same expected outcomes, and the same access restrictions. Over a 20-business-day evaluation window, a planning assumption rather than a vendor promise, record elapsed setup time, engineer hours spent repairing tests, failure triage time, and how many failures led to an actionable code or environment fix. If the bought option produces more tests but requires an engineer to interpret every ambiguous failure, its apparent capacity gain has not reduced the team’s operating burden.
Ask a vendor to show how tests are exported, how selectors and fixtures are versioned, and what remains usable if the subscription ends. These are cost questions because rewriting proprietary test definitions later may outweigh the time saved during initial setup. Request documented retention and access controls for screenshots, videos, logs, and credentials; test artifacts can contain customer information even when the test account itself is synthetic. A SOC 2 report may help assess a supplier’s controls, but it does not replace checking the permissions and data your team will actually send to the service.
Price the comparison as total ownership, not license versus salary. For the internal option, include browser updates, CI minutes, fixture repair, and on-call triage. For the managed option, include subscription terms, onboarding, integration work, test maintenance that remains yours, and the cost of leaving. Have the same engineer investigate failures from both pilots when possible, because differing triage skill can otherwise masquerade as a tool advantage. Document any missing capability as a concrete blocked workflow rather than accepting a broad claim that one platform is “more scalable.”
At the decision meeting, buy only if the vendor has demonstrated a lower burden for the particular checks you intend to operate, or provides capacity the team cannot reasonably staff. Build if the checks change alongside code and your engineers can keep the suite dependable within the agreed budget. A mixed result is acceptable: the team can own API contracts and fixtures while purchasing browser execution capacity. That division works when the contract names who fixes each failure, because a split stack without split responsibilities simply creates another handoff.
The first decision is which failure deserves a gate
Ask the release owner today for one recent failure they would have paid to catch before deployment. Write its observable outcome, test-data requirements, and failure owner on one page, then ask an engineer to make that check run against staging. Use its maintenance cost and diagnostic value to shape the buy-versus-build trial. Do not purchase a suite before you can specify that first gate.
