36 points qtqtqt 1 hour ago 16 comments

rafram 46 minutes ago | parent

> We have 13 tests at this time.

Does GIMP have a very restrictive AI policy or something? I get not wanting to ship a gigantic pile of UI tests without thorough review, but basically any model released in the last six months would be able to write UI tests covering every single control in an afternoon.

samuria 37 minutes ago | parent

It’s frustrating to create tests with models, not sure if it’s everyone’s experience, but every time I asked it to create tests, it makes the tests conform to the sometimes broken code. Rather than calling out what the logic/code is wrong, it will make the tests conform to whatever is already broken.

Waterluvian 34 minutes ago | parent

It can’t magically know expectations. If the code is well documented or there’s decent associated documentation, I’ve found it has no issue writing tests.

forgotmypw17 32 minutes ago | parent

I recommend iterating on your prompts.

Prod your agent first until it can explain back to you what's wrong with the tests.

Then ask your agent to write a better prompt for writing tests (and save it to an .md file)

Then reference that .md file and iterate on the tests again. It may take a couple of tries, but I've had good results with this strategy.

Ended up with a whole prompting framework.

deskglass 36 minutes ago | parent

locusofself 18 minutes ago | parent

Not using AI to at least writes tests in this day and age is crazy work. I started a desktop application 2 months ago and it has almost 3,000 tests now, MANY of them UI tests.

timcambrant 5 minutes ago | parent

AI is a bit too excellent at writing tests. The number of tests doesn't really convey any information on test coverage. At least I've found that agents that naively improve codebases will write tests for pretty much anything, including irrelevant behavior that has little value in ensuring stability, security or correct behavior. It would probably help to run a test review stage with a strong model, where agents attempt to correct and consolidate tests to guard behavior rather than ensuring that theoretical code paths are followed.

joegibbs 17 minutes ago | parent

They made a blog post to announce that they’d written thirteen unit tests? Sure, there’s value in handmade software and all that but GIMP is not really a particularly excellent piece of software - if anything was to benefit from unleashing the agents it would be GIMP.

Jolter 7 minutes ago | parent

I’d go so far as to say that in general, adding tests for legacy software before a refactor or rewrite is the perfect time to put the agent to work.

hulitu 3 minutes ago | parent

> if anything was to benefit from unleashing the agents

AI can write tests now ? For its own software ? Can it evaluate the results ? Correctly ? /s

dllu 7 minutes ago | parent

Funny how GIMP spawned the Gimp Toolkit, aka GTK, which is widely used in Linux, and yet GIMP's own UI seems to have long languished as a rather outdated design. Glad that they have tests, though, and hope they continue to improve it.