YHK - yet another hacker news clone

Launch HN: Momentic (YC W24) – Agents that catch bugs by using your app

57 points wuweiweiwu 1 hour ago 40 comments

Hey HN, we’re Wei-Wei and Jeff. We’re building Momentic, QA agents that autonomously plan, explore, and bug-bash controls and user flows in your web, iOS, Android, and (soon) desktop apps to catch bugs.

Here’s a demo: https://youtu.be/BFsM-Se1Fh8

We’re both software engineers that spent our careers on dev tooling and infra across companies like Robinhood, Retool, WeWork, and Qualtrics. Testing was always a pain in the ass (esp Playwright and Selenium), so we built agents to do it for us.

Modern day software has a ton of bugs and the change failure rate is only growing faster as coding agent adoption scales. LLMs are really good at generating code, but reinforce their own assumptions, write a ton of useless tests, and are bad at visual understanding. On DeepSWE v1.1, Sonnet 5.5 hit 66.2% pass rate when banned from writing tests, versus 65.3% when allowed.

OSS end-to-end testing frameworks like Playwright, Selenium, or Maestro force you to hardcode implementation details like XPath, text, or aria-* attributes in your tests. Even with agents, these take a nontrivial amount of time and tokens to build and maintain.

In 2023, we saw Nat Friedman drive a browser with GPT-3 via natbot. We were inspired to use this technology to automate the most painful and tedious parts of software engineering: testing. We started with using LLMs to generate and maintain Playwright, but quickly realized that those primitives were fundamentally misaligned with the agentic SDLC. We think the future of QA is agents testing themselves, not writing scripts.

Mo is a custom agent harness for catching bugs that builds on the past 3 years of Momentic and millions of test traces from complex, real-world applications. Mo is built for recall: it finds the most real bugs, though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin.

- Self-organizing agent graph. Mo builds a directed acyclic graph (DAG) as it explores. Large product areas automatically fork into parallel sub-agents, each with their own browser or device for concurrent execution.

- Specialized agents for domain specific tasks. Planner agents reason over screenshots and video while dedicated locator agents inspect page structure to resolve plain-English interaction targets. Others include image analysis, video understanding, audio generation, and more.

- Parallel agents coordinate through cooperative resource leases. Agents can announce that they’re “using” or “modifying” overlapping resources or fixtures. Multiple “using” claims can coexist, but only a single “modifying” claim can be active on any particular one.

- Scaled infrastructure. Mo ships with a tunnel that lets users expose private sites and local apps to our hosted agents and infra. A single bug bash can spin up 100s of concurrent browsers and devices.

We benchmarked Mo against coding agents on 4 popular open-source apps: Gitea (58.8k stars), Metabase, Twenty (YC), and Saleor, with 1,396 human-labeled bugs. Each agent got the same brief, a fresh E2B sandbox, and no access to source code. Codex and Claude Code used hosted Browserbase browsers, while Devin used its built-in browser. For evaluators, we implemented LLM as a judge (GPT-6.1 Sol) with a human reviewer.

Results from a time-boxed 60-minute run:

- Mo: 269 bugs, 19.3% recall, 84% precision, $2.67 per bug

- Codex (GPT-6.1 Sol + Playwright MCP): 86 bugs, 6.2% recall, 86% precision, $0.80 per bug

- Claude Code (Opus 5.5 + Playwright MCP): 59 bugs, 4.2% recall, 80% precision, $4.09 per bug

- Devin: 27 bugs, 1.9% recall, 84% precision, $3.89 per bug

Mo finds more bugs the more time it has. From 30 to 60 minutes, it went from 177 to 269 bugs (+52%), while Codex gained only 16%.

Results: https://momentic.ai/qa-bench

Write-up: https://momentic.ai/blog/qa-bench

Dataset: https://huggingface.co/datasets/momentic/qa-bench

Harness: https://github.com/momentic-ai/qa-bench

We just shipped PR testing for Vercel and Cloudflare deploy previews and we’re giving every new signup $250 in Mo spend. Paid plans have higher infra capacity and included usage along with options to top up.

We’d love for you to try it out and give us feedback. Feel free to comment here, we’ll be around all day!

aliasxneo 1 hour ago | parent

So is this yet another SaaS product that's going to attempt to charge me for token usage? Sorry to sound negative, but nowadays it's an instant turn off because it's like the current subscription crisis on steroids. BYOA should be a default.

To be more constructive: let's say I have 2 frontier subscriptions and I end up with 20% left on each at the end of the week. It feels awful knowing that I would _love_ to use that with Momentic but I can't because I have to pay for yet another set of proprietary non-transferable tokens. It's like dealing with airline rewards.

amunozo 1 hour ago | parent

What does BYOA mean?

aliasxneo 1 hour ago | parent

I literally just made it up: Bring your own Agent. It was a poor attempt at making fun of the current SaaS landscape. Although I would be surprised to be the first one to have ever used that acronym.

freedomben 1 hour ago | parent

Interesting, I assumed Bring Your Own API-key

nkozyra 1 hour ago | parent

Assuming agent/access tokens, in other words you're not going through a middleman for LLM access.

aduwah 1 hour ago | parent

I assume bring your own agent

jeffan 1 hour ago | parent

Hey I'm Jeff (one of the co-founders). That's fair feedback. We wrestled a lot with whether to support BYOA or open source the harness. Ultimately there was no configuration with off the shelf models that we were happy with performance-wise. A lot of the eng work that went into this came tuning custom agents from large datasets to do the "grunt work" like element grounding and visual understanding. Plugging in vanilla GPT6.1 Sol for every task into our harness just doesn't yield a pareto optimal result that makes sense as a standalone product (you are better off asking codex to test).

That doesn't mean we won't eventually support byok, especially as OSS models catch up! We have our finger on the trigger there.

kart23 1 hour ago | parent

the harness is the lock-in with these subscriptions. if you could just take your tokens to any app, there would be no point to stay on that companies ecosystem. I definitely agree that tokens and subscriptions should be portable and fungible, but any service offering that would be much more expensive than just getting a claude or codex sub.

MGabrielDenton 13 minutes ago | parent

It would be really nice to be able to plug in the subs from the labs. The industry seems to be moving that way even if its still super limited. Ex https://loginwith.ai

bortimus 1 hour ago | parent

So, what's the moat?

Aboutplants 1 hour ago | parent

The moat is about a week before the frontier models integrate the same capabilities. They aren’t snapping up projects like this for talent acquisitions anymore, they will just build it themselves.

0xy 1 hour ago | parent

I have something similar I made for my side project. You don't really need a platform for this.

Fordec 13 minutes ago | parent

Same, I just built mine in two months. Screenshots, agent spin up, and I'm cheaper than Codex because of how I built my own secret sauce.

This strikes me as a company that is late delivering a product for November 2025, when it is November 2026.

sercand 1 hour ago | parent

we also build similar product in-house for our apps.

jeffan 1 hour ago | parent

Hey there, I'm Jeff (one of the founders). To be honest I don't know if there is or will be any kind of moat in SaaS. But today:

1/ General purpose models are still not good at this task. I would personally love it if I could trust my agent when it says its done after a 24h build. But whether it's because of sycophancy, context degradation over time, or something else coding agents still struggle to verify their own work. Maybe with 1B context windows this will change.

2/ Infra is still annoying to setup. We wanted to provide a unified interface across web, mobile and soon desktop as well. So users don't have to worry about provisioning, concurrency, security, etc.

3/ As with any SaaS, even beyond the initial build, maintenance is a headache. I think most companies would rather not consider agentic QA as one of their core competencies and investments. It takes a ton of eng effort to keep the lights on, continually update the models used, train new ones, groom ground truth, onboard infra providers, etc.

Arkada 8 minutes ago | parent

When people say "just point Opus/Codex at localhost using Playwright MCP," they've obviously never attempted to do exploratory testing by leveraging parallel browser workers against stateful applications. A coding agent running in a loop spends almost all of its context window struggling with DOM selectors and doesn't even detect state persistence problems like the 2.5h->25h decimal inflation upon refreshing the page as demonstrated in the demo. Managing read/write resource leases for parallel sub-agents not to step on each other's test fixtures is already a non-trivial orchestration challenge by itself.

schainks 1 hour ago | parent

What does this do that UAT doesn't do? Sounds like you're combining UAT with gloves-on red teaming and I guess property testing APIs.

momentichenry 1 hour ago | parent

Hey! I'm Henry, one of the engineers working on Mo. The approach to testing itself is very similar to UAT, except that it's fully automated. This means you can speed up the feedback loop significantly. Imagine instead of having to wait for the next release and run a time-consuming manual bugbash in order to find out that your changes broke something, you can get feedback immediately on any PR you want, at a fraction of the cost.

staindk 1 hour ago | parent

Congrats on the launch!

Some questions are top of mind after having done POCs and trials with other companies that have products in this space (excuse me if they are answered somewhere obvious - I skimmed the site and your post before drafting this):

1. What is vendor lock-in like with Momentic? In my mind there's space for a mature QA platform that agentically generates and maintains playwright/similar tests, but perhaps I'm wrong.

1.b. will Momentic code or testing standard or some aspect be open enough that it is feasible to move away from it if needed, or continue using it if Momentic goes under/deploys major breaking changes?

2. How deterministic are Momentic tests?

3. Who is your target user? Product owner / developer / SDET / ?

----

Also - is this saying Codex is better and cheaper than Mo?

"... though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin."

jeffan 27 minutes ago | parent

Thanks! I'm Jeff, one of the co-founders.

1. Mo generates its own test cases in natural language, but you can ask it to write Playwright and maintain it in your repository if you really want. We actually do have a regression testing product as well where tests are just natural language in your repository. For that product yes there is some vendor lock-in since this is a proprietary format that only our CLI translates into browser actions. But ultimately since it's natural language it's pretty portable :)

2. Mo is focused on exploratory testing and so is inherently non-deterministic. But the regression testing product has a ton of caching built into it that we've written about in the past (not just a single css selector). https://momentic.ai/blog/teaching-browser-agents-user-intent This is a big part of reliability and latency optimization, otherwise your CI would just hang for hours on any reasonable suite.

3. I would say the regression testing product was very much SDET/developer focused since it required you to have a codebase and use Git. Mo is meant to be super easy for anyone to onboard onto and lower the barrier to entry, so we're hoping we'll see more product owners, PMs, QA and even leadership use the product. But devs are always going to be our bread and butter, we ultimately tried to build the product we wanted to use ourselves.

4. We're optimized for recall (# of bugs you find in a given amount of time). The bench shows we find ~3-5X more bugs compared to Codex and Claude Code, with approx. the same precision. The service costs more than the raw harnesses for sure, but that's the all-in cost including our infra, parallel agents, etc.

JulianPaul 1 hour ago | parent

Why is this better than running Opus 5.5 directly against my app with a custom Playwright-based harness for in my own subscription?

MGabrielDenton 53 minutes ago | parent

Hey I'm Gabe (one of the engineers @Momentic). In general Mo's advantage is in the total amount of bugs found. In comparison to claude code we find roughly 5x as many bugs at 35% less per bug (for the 60 min session).

cacio-e-pepe 36 minutes ago | parent

Thoughts on the bitter lesson? Eg what if models just keep getting better?

MGabrielDenton 20 minutes ago | parent

Good question, in short our harness & e2e tests act as compliments with the frontier rather than competitors. Architectural choices and data you provide really matter for domain specific tasks. Mo will be working with harness specific tooling like: previous momentic test results, our KB, and eventually creating new Momentic regression. Having smarter/more capable models will improve the efficiency of AI generated tests (coverage/runtime) and ideally do less repetition in exploratory testing.

mikert89 44 minutes ago | parent

It’s not

adwn 1 hour ago | parent

The last 3 years saw tons of "custom agent harnesses" come up, only for them to be superseded by the next generation of frontier models, which could do the same (and more) in their generic harness. So, what will you sell when Opus 6.0 swallows Mo's edge in 5 months from now? And probably cheaper, when using a Claude subscription instead of paying API prices.

jeffan 59 minutes ago | parent

Hiya, I'm one of the co-founders at Momentic. I would actually push back against that take, we work with frontier labs a ton and this year we've actually seen a lot of realization on their part that custom harnesses matter. E.g. you could argue Devin is just a frontier model wrapper and yet it does feel better and is super popular. Same with code review tooling -- afaik everyone on https://codereview.withmartian.com/ mostly uses off-the-shelf models and yet it's the architecture, durable context (knowledge base), and data they have that gives them an edge. IMO opus 6 would probably just unlock more capabilities for Mo rather than eroding the edge.

adwn 19 minutes ago | parent

Let's hope this is how it will play out. I wish you the best of luck!

podoman 1 hour ago | parent

> Mo is a custom agent harness for catching bugs that builds on the past 3 years of Momentic and millions of test traces from complex, real-world applications. Mo is built for recall: it finds the most real bugs, though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin.

is this good? so Mo costs more money per bug? I'm confused.

luckydata 1 hour ago | parent

it's right there in the writeup, it costs more but finds more bugs.

Smoofer 1 hour ago | parent

Right? Is the takeaway from that not just that Codex is better for cheaper?

momentichenry 51 minutes ago | parent

Hey! I'm Henry, one of the engineers working on Mo. You're looking at the right numbers, but the takeaway is slightly different from ours.

The main thing we're trying to show is that Mo finds over 3x as many correct bugs in the same amount of time as the next-best QA agent. This is the main thing we optimize for since the worst thing you can do in QA is miss a real bug and ship broken code to prod. Currently, the tradeoff you make for that better recall is higher cost. I would say the precision is more or less comparable, but marginally lower.

Obviously we'd love to be beating every other agent on all of these numbers, and we're actively working on it, but for now this is an honest and transparent look at how the tool is doing.

Feel free to take a look at the full writeup for more detail!

mlcruz 19 minutes ago | parent

I feel like your writeup could be a little stronger if you made it clear how important is hitting those recall targets.

I also do believe search quality is recall dominated, but the link between the metrics and results are not as clear as they could have been. I think you have a stronger sell here than it looks at first glance.

Congratulations to the team tho! Those are very nice results

momentichenry 10 minutes ago | parent

Thanks! That's fair feedback. We'll update the writeup to make this clearer. We're also hoping to release a v2 benchmark soon with more fixtures and hopefully an even more optimized Mo.

adwad 53 minutes ago | parent

Are you open to listing other agents on the benchmark? I’m building my own and would like to run it on the full suite, but want to check before putting in the effort.

jeffan 49 minutes ago | parent

Yes we are (it's MIT licensed)! Feel free to email me at jeff@momentic.ai or file a pull request directly on https://github.com/momentic-ai/qa-bench

Everything has to be reproducible and runnable by the general public

lejohnq 26 minutes ago | parent

congrats on the launch, agentic testing makes a lot of sense.

how does this fit in to fixing the bugs? how do we close the loop with the agents we have coding?

momentichenry 18 minutes ago | parent

Hey! I'm Henry, one of the engineers working on Mo. Currently, Mo is super focused on being the absolute best at finding bugs, but isn't meant to be a general purpose coding agent. For that we ship a CLI (https://github.com/momentic-ai/qa) and matching skill which you can use to integrate Mo with whatever coding agent you prefer. One pretty successful setup we've seen a lot of customers use is telling the agent to build a feature, then use Mo to test it, and only go back to the user for input once all the bugs are resolved.

sailorganymede 14 minutes ago | parent

In my company, we already have agents messing around on our internal systems interacting via Playwright to try break things

jon128 5 minutes ago | parent

Keep developing apps in Javascript and Typescript instead of bug-unfriendly ReScript. Momentum may eventually do an IPO so pump that shit up, homie.

  https://rescript-lang.org/