Crypto wallet QA and Playwright test automation for Chia Network

Before a single test could run, every machine needed a Chia node synced to testnet11, Chia's test network, and none of the three systems went quietly. Each one got stuck loading the keyring, a wallet could sync in a minute one time and take hours the next, and one release candidate's Windows build wouldn't open its window at all. Chia's team had the answers, run the full node rather than the wallet alone and start from the copy of the testnet database they publish, and with those all three machines were running. They sent test tokens and NFTs to my team's wallets, and my team minted more NFTs from the command line, tied to a decentralised identity, for the tests that needed them.

My team and I ran all three release candidates of 2.4.3 across the three systems, each within the three days Chia gave us, and logged every result in a testing sheet Chia's engineers could read. Anything that broke went onto Chia's GitHub, where Chia's engineers tagged it for their QA team and worked through it with my team, asking for logs and trying each one again on the next release candidate.

Some of what my team found sat outside the forty-two tests. On a Mac and on Linux, the identity picked to sign a message ran straight out of its box in the signing dialog. Chia's engineers had a fix up two days later, and it shipped in 2.4.4.

An NFT offer taken in one wallet still showed as waiting in the wallet that made it, and my team added its reproduction to the issue Chia already had open, which the 2.4.4 release lists as fixed. For that next release, Chia moved to a full-time tester from my team, who ran all three of its release candidates the same way and went through Chia's new website for broken links before it launched.

While the reference wallet's releases were under test, Chia asked my team to test its next wallet as well, the Cloud Wallet, which runs in the browser with no node to sync. Chia wanted more than people clicking through it. It wanted more tests, the testing automated, and a way of working that would hold as its QA team grew, and it chose Playwright to do it.

The Cloud Wallet has no passwords. People sign up, sign in and approve every send with a passkey, and the prompt that asks for one belongs to the browser and the operating system, outside the page, where Playwright can't reach. Behind the screens is a real blockchain, so creating a vault is a transaction that has to confirm, a send has to reach the mempool and settle, and the recovery phrase shows once and never again.

I built a Playwright suite that tested the Cloud Wallet's front end and back end, against its development build on testnet11, and my QA team assisted with the automation. Chia's product team wrote each feature as a story with acceptance criteria, and those became the tests, from the welcome message to the passkeys page.

The passkey was the key to all of it. I kept the test account's passkey in a password manager's browser extension and loaded the extension into the browser Playwright drives. When the wallet asks for a passkey, the extension answers inside the browser, where a test can click it, and from there every flow could run end to end. A test signs in, creates a vault, copies its recovery phrase and pastes it back to confirm it, renames the vault and deletes it. It sends test XCH with a fee, approves the send with the passkey, and follows the transaction from unsigned to the mempool to settled. Tokens, the address book and the settings each got a flow of their own.

The first full run passed, and Chia's product team asked for the suite to live in Chia's own GitHub. It moved there, on Chia's signed commits and commit conventions, and ran in the CI/CD pipeline on the main repository, and my QA team kept it current with each release of the Cloud Wallet.

Chia shipped 2.4.3 and 2.4.4 of its reference wallet with every release candidate tested on Windows, macOS and Linux, and with the bugs my team found filed on its GitHub, where its engineers worked through them. For the Cloud Wallet it had an end-to-end Playwright suite in its own GitHub that signed in with a real passkey and moved real test funds on testnet11.

When Chia ran its pre-release testing week for the Cloud Wallet, a QA engineer from my team was one of its nominated testers. Chia had its desktop wallet's releases tested on every system it targeted, and its new wallet tested end to end, passkeys and all, before it went out to the public.