11 points tambourine_man 1 day ago 5 comments
StevenWaterman 47 minutes ago | parent
Models:
- Claude Sonnet
- Gemini 2.5 Flash
- Gemini 2.5 Pro
- GPT-5 Codex
- Llama 3.3 70B
It's inconvenient how quickly things are improving, makes potentially-interesting results like this dubiously relevant
plorkyeran 35 minutes ago | parent
> We study a common intermediate setting: a developer pastes one complete file into a chat assistant [5], which sees the file’s internal structure but not its callers or dependencies. We evaluate the first answer, not an agent’s result after iterating on errors.
Very unsurprising results given this methodology. The agentic approach where models are allowed to iterate on changes is a very big part of what has made LLM coding tools actually useful.
yodon 25 minutes ago | parent
> We study a common intermediate setting: a developer pastes one complete file into a chat assistant [5], which sees the file’s internal structure but not its callers or dependencies.
Ack! Do people actually try to develop software this way?
plorkyeran 2 minutes ago | parent
Not any more, but it was common before Claude Code came along (~18 months ago). Based on the model versions used it may have still been the most common thing when the paper was started.
AdrianB1 25 minutes ago | parent
One year ago I was very skeptical about such attempts. Today, it would be my first choice as LLMs evolved a lot and the result of carefully using a LLM for this task can be very, very good.