Yet another Hacker News

Running Windows desktop tests from Linux with disposable VMs

94 points jadenlemmon 3 days ago 32 comments

jadenlemmon 3 days ago | parent

I work at QA Wolf and lead the development of automated testing for Windows desktop apps. I wanted to share what we did and how we did it.

This post covers how we added Windows to a test infrastructure built around Linux runners. The central problem was giving each test a clean, logged-in Windows desktop without waiting 2–3 minutes for a VM to boot.

We extended the device scheduler we already use for mobile testing. It maintains warm pools of Windows VMs on GCE. Each test claims a ready VM, uses it once, and releases it. The VM is then destroyed and replaced, rather than cleaned up for the next test.

The test code stays in a Linux pod. Playwright connects to browsers on the Windows machine over CDP, while WebdriverIO and Appium handle native apps through WinAppDriver. A small Go agent captures the desktop with FFmpeg and streams it over WebRTC for live viewing and recordings.

That separation lets a single test move between a browser and a native Windows app without requiring a separate Windows runner pipeline.

The trade-off is that we keep idle Windows VMs ready and install the app under test on every run. In exchange, tests don’t inherit another run’s files, registry changes, or saved sessions.

I’d love to answer any questions!

trollbridge 3 days ago | parent

Not to dismiss this, but when I was working on a problem on Windows recently my LLM (6.1 Sol / oh-my-pi) ended up doing the same thing, although it used Qemu and snapshots to accomplish that.

Eventually it got to a point it could iterate very quickly and was really only constrained by total system RAM which meant a dozen at most copies could be running simultaneously.

mcoliver 3 days ago | parent

Same. It's quite remarkable. Two other things Claude and I have done. I have a spare workstation running proxmox in my garage. Claude can now create, snapshot, delete, extend, start, and restart vms. It's been great. I also have skills that allow it to spin up spot instances when I need to test things that rely on hardware I don't have like certain Nvidia gpus or 100gb networking. What a time to be alive.

trollbridge 3 days ago | parent

To be fair, you never needed LLMs to be able to do all that.

rsolva 3 days ago | parent

To be fair, often you have good ideas, but not enough time. I have had a lot of fun having Qwen3.8 27b running at home, having it explore and implement ideas that was way to ambitious for the time I had available to tinker with Home Assistant and Proxmox. It is an enabler and can also be a good tutor, when used as such. I have learned a lot.

alasano 3 days ago | parent

"be able" implying capability rather than time constraints and opportunity cost which is what most people on here refer to when they mention something cool they're doing with LLMs that they weren't doing before.

mcoliver 3 days ago | parent

True but it was a whole bunch of fragile scripts/functions with lots of ifs. I'll give you a real world example of what's different. Agent spins up a VM to run a build on windows. During the build it realizes it's out of disk space. Agent resizes disk on demand. Build finishes and it launches the app. Runs some tests and realizes it's cpu and memory constrained. Adjust the cpu and memory and runs new tests. tears it down and sends the perf logs back to me. Or it looks at system resources and decides what other vms it can pause while it runs the tests and then turns them back on after.

There are so many unknowns and the ability for the agents to dynamically adjust and make decisions allows for some really powerful autonomous workflows.

c-hendricks 2 days ago | parent

Not sure the lower bounds of CPU/memory/disk space that your project needs should be considered "so many unknowns".

mcoliver 2 days ago | parent

Man. HN is brutal. Just one example. And yes, when you're looping agents and builds to identify performance optimizations and build features sometimes you don't know what those bounds are going to be and it's much faster to let the agents work it out, update their learnings, and keep building.

h4kunamata 2 days ago | parent

Not to take your credit away but there is something called automation, scripting, etc.

All those actions within my homelab are automated via Ansible and bash scripts (I only run Linux), I am only notified if something goes wrong like a backup failed.

I get it if you never worked in IT, that is fine, but you need zero AI to get the above fully automated.

Does Claude keep versioning of the changes?? I host my own repos via Forgejo, Ansible and the automation process consume the config files from there, one source of truth. If my whole setup burns with fire, I can bring everything back online from the offsite Ansible playbook located at Codeberg.

whalesalad 2 days ago | parent

Why not both? My entire home network (dns, zfs nas, subnets, kube, proxmox hosts, ipa domain, its overkill homeprod) is represented by a git repo. Docs, plans, scripts to bootstrap nodes to domain join, kube config, proxmox tooling, ipa as code, nas as code, cert management, my own ec2 as a service, etc. Making changes is done via a mix of one off tooling, ansible and terraform.

I use Claude or codex to manage the entire thing, interchangeably. They keep detailed daily journals. They keep a central TODO.md up to date. They refresh stale docs or stale plans when new plans invalidate old decisions.

I have so many layers of backups and fail safes that I work entirely in yolo mode. It’s really insane what I’ve been able to deploy myself on nights and weekends. Stuff I’ve wanted to tackle for years but never got around to it.

blahgeek 3 days ago | parent

Nothing new here. VM + WebRTC + warm pool to absorb boot time

joinjune 3 days ago | parent

We used several different LLM's and our own skills to build out some novel approaches to document creation with new workflows using QEMU in Linux after upgrading our desktops to Ubuntu from Windows 11.

ranger_danger 3 days ago | parent

Why not use snapshots/save states so the VMs start back up right away? This is what I do with QEMU.

ProblemExplorer 3 days ago | parent

Doesn't Windows do some drama if its not activated and has been running without a license for more than 48h or similar?

I don't use windows but have to run windows 11 VM to access special devices. I don't want to bother with licenses, although I have them, and creating a new incus QEMU VM from scratch everytime from an ISO takes ~5m, freeing me up from worrying about all the activation and license drama.

If the activation and license setup was easy/automated in QEMU, I would imagine snapshots would be way easier. I would prefer not needing to connect to the internet.

c-hendricks 3 days ago | parent

> Doesn't Windows do some drama if its not activated and has been running without a license for more than 48h or similar

Your info might be like decades out of date, since windows 10 it's just been an overlay and locking out some appearance customizations (you can still set them via other means though).

ProblemExplorer 2 days ago | parent

Ok, so you're saying I can run windows 11 without a license or activation for months on end?

c-hendricks 2 days ago | parent

Yes, since the release of Windows 10 in 2014.

1718627440 2 days ago | parent

Yes, you can also automate fetching a license and activating with https://massgrave.dev/ .

itissid 3 days ago | parent

Why is the comments from the OP marked as flagged or dead at the bottom?

realusername 3 days ago | parent

Looks like AI written so it got auto-flagged by a bot probably

defied 3 days ago | parent

We’ve been providing a similar approach with disposable VMs and emulators/simulators since 2014 at TestingBot. We run our own single-use QEMU VMs with Windows and Linux.

madduci 3 days ago | parent

Same here, we used Proxmox as host to spin up ready images built with Packer and starting them with Vagrant to provision the app on the system under test, the whole triggered by the CI build

Luker88 3 days ago | parent

what is the minimum license that I would need to do this legally, with just one or two VMs?

OneDeuxTriSeiGo 3 days ago | parent

It depends what you need to run. There exist a number of Windows images which can be run without any paid license requirements for test and validation IIRC.

https://learn.microsoft.com/en-us/windows-hardware/manufactu...

jadenlemmon 3 days ago | parent

it depends on where you are running it. If using a cloud provider, license is typically built into hourly price.

xlazom00 3 days ago | parent

This is super slow solution I did build similar solution based on dockur. And it was sooo slow. Just use windows docker containers on windows server. With some hacks you can run any gui application.

msla 2 days ago | parent

What's the licensing on Windows 11 these days? Can an individual still run an unregistered Windows 11 installation basically forever with only minor restrictions?

jadenlemmon 2 days ago | parent

For an individual, mostly yes. Unactivated will just keep running with a watermark & some minor restrictions.

msla 2 days ago | parent

Thank you.

fhn 2 days ago | parent

thanks for the advertisement

peter_d_sherman 1 day ago | parent

>"There’s a good reason why testing Windows desktop apps was one of the most requested features from customers: no one in the world has the infrastructure to parallelize thousands of runs per month..."

This article makes some excellent points!

If you've ever worked in, or alongside QA in commercial software environments, you know that being able to run a large amount of automated parallel Windows specific QA tasks is indeed a headache! And that's if an organization can accomplish that at all -- many organizations still use single human QA's and/or SDET's and still run many QA tasks manually... Of course, AI should/will help with some of this, but there still is the physical testing infrastructure to take into consideration... there are many barriers, not the least of which include such things as necessity for multiple Windows licences, setting up those Windows environments properly, instrumenting test harnesses properly, aggregating the results from multiple sources, etc., etc.

Phrased another way, there's a large market there -- if you can solve all or some of those problems, in whole or in part...

Anyway, an excellent article on this topic!