126 points rzk 9 hours ago 220 comments
https://www.linkedin.com/posts/ali-hamie_im-an-engineer-at-m...
> I'm an engineer at Meta and I never felt this uncomfortable doing my job. I'm writing code these days and it all feels like a black box. This new standard of shipping with AI is moving so fast that if we're going to spend our time checking each line of code, it's going to take forever. That means you can end up with systems you've built where you don't truly understand how all the code works. You understand the high level, but not every detail. And maybe this is becoming the new standard. It goes against every instinct of being a software engineer. I'm used to understanding what I'm building. Now I'm trying to figure out how much I need to understand before I'm comfortable moving forward. Are other engineers feeling this too?
How are you handling this?
spacedcowboy 9 hours ago | parent
That’s not to say I blindly trust an LLM - my project is an autovectorising compiler. I have it write a bazillion tests, and I have real, non-trivial programs that it compiled. I have it run a ‘gate’ before releases that takes literally hours on 5 different architectures. To say this code is tested is … masterful understatement.
I think what helps is a long planning session, going into details, working out corner cases, designing test-cases, you know work and effort. Once that’s done it’s easier to let it work through the night and do a major production feature.
I still review the code, but hand on heart I couldn’t say I’d reviewed everything, sometimes I miss something in a complicated check-in. I try to keep checkins small and contained, but sometimes that’s not easy to manage alongside the “don’t break stuff” rule as well.
So it comes down to: I’ve been working on this for 6 months or so, and so far this has worked out for me. You know, kind of like getting used to a colleague’s idiosyncrasies over a period of time.
I will say that in my experience the code quality is fairly high. Maybe that’s down to the effort put in ahead of time in the planning stages, I don’t know, but I don’t seem to suffer from the ‘slop’ that people complain of. YMMV.
yulaow 9 hours ago | parent
spacedcowboy 8 hours ago | parent
That doesn't mean I don't like progress mind :) This morning I got the preliminary results of a 'C' front-end to the compiler (it uses clang to compile to LLVM IR, then translates to our IR, then feeds that IR into the autovectoriser as if it had parsed the code itself). So for the exact same source code I now have:
┌─────────────┬────────────┬──────────────────────┬──────────┐
│ Benchmark │ clang -O2 │ xcc --par-auto (GPU) │ Speed-up │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ saxpy │ 1,546 µs │ 726 µs │ 2.1× │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ Mandelbrot │ 394,238 µs │ 2,110 µs │ 187× │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ nbody │ 37,090 µs │ 1,035 µs │ 36× │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ perlin │ 78,016 µs │ 1,850 µs │ 42× │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ mandel_deep │ 73.2 s │ 55.8 ms │ 1,312× │
└─────────────┴────────────┴──────────────────────┴──────────┘
Those are some pretty fine speedups, from naive C, with no alterations from the standard "C" code - it automatically finds places that can be made data-parallel, tracks the data-hazards to make sure its safe to "GPU" and writes all the kernels, boilerplate and data-stream-management if necessary. Obviously the checksums have to match, so I know it did the work correctly too :)Hacker News isn't interested - any post I make linking to the webpages is immediately blocked/dead so I'll wait until the C++ front-end is ready (some time next week) and announce it somewhere else. A compiler that takes standard C/C++ code and speeds it up by automatically finding data-parallel areas and scheduling them on the GPU with no programming overhead is pretty rare.
Oh, and it manages the local network as well, so you can run xcd on machines on your lan, or via the TB4/5 bridge between Macs and it'll schedule on other GPUs, just like it manages multiple GPUs on the local machine. It characterises both latency and compute-time to figure out which devices are worth scheduling on.
ch_sm 2 hours ago | parent
spacedcowboy 1 hour ago | parent
Bottom line: GPL3 for the compiler, standard library exclusions for derived works, so use it for anything but feed me back improvements to the compiler itself, and you don’t get to redistribute it as your own.
jazzyjackson 1 hour ago | parent
JMKH42 1 hour ago | parent
rpdillon 1 hour ago | parent
Silhouette 1 hour ago | parent
spacedcowboy 39 minutes ago | parent
Still working on some of the data-parallel recognition in LLVM IR space. In order to support both C and C++ with the same data-parallel section extraction, I moved the detection from the ‘xc source code’ phase to the IR lowering phase, meaning that everything could ‘feed in’ to the same routines. That’s caused a little friction with the slightly different ways the LLVM IR from clang and xc’s IR encode loops. The above results are all single-loop extractions and the double-loop ones are still ongoing.
The system works in 3 stages: 1) by being greedy about which loops could possibly vectorise and marking them, 2) by analysing the IR to reject loops which do pointer-aliasing, or object manipulation, or… and then 3) by measuring the fastest way these known-safe kernels could execute.
Which brings me onto one nice thing I haven’t mentioned: that the GPU choice(s) are made at runtime, not compile time, and the system automatically measures (and then persists so it doesn’t have to measure again) the fastest way to run a kernel on the current machine. It handles MSL on Apple devices, PTX on Nvidia cards, Spir-v on Windows, Linux, and Android, and WGSL in the browser. It’ll quite happily divide up the parallel task across a 3090 and a 5080 on my Windows test box for example, with the 5080 doing ~2/3 of the work, because the scheduler is opportunistic.
All without writing anything other than plain C with nary a mention of the GPU. Give me a week or two, and I’ll probably announce it on phoronix (which might then get a mention on HN if it goes down well there, and it’ll be a different url so hopefully won’t be insta-dead).
bee_rider 1 hour ago | parent
spacedcowboy 1 hour ago | parent
smokel 1 hour ago | parent
I wouldn't be too sure about that :)
spacedcowboy 1 hour ago | parent
I think it’s my own fault, I’ve posted the link to the compiler pages several times in what I thought were relevant posts, and I think HN thinks I’m spamming. So I’m not posting the home page for a while and we’ll see if a future submission “takes”
john_strinlai 41 minutes ago | parent
spacedcowboy 28 minutes ago | parent
chrisjj 2 hours ago | parent
A.k.a. "The concrete didn't crumble when I hit it with a hammer."
Process quality has been a thing for ages because it is essential.
spacedcowboy 21 minutes ago | parent
Except it’s about 1000 “hammers” (aka tests), x 9 different architectures, x 2 different (a reference and a production) variants, and x 3 different optimisation levels. Every time I find something that slipped through, it gets at least one test written to cover it in the future. My main time-sink is the hours it takes to run the ‘gate’ before a release across the test-farm. And that’s after I’ve been playing with it myself for a while, pushing corner-cases through whatever I’ve been working on.
jimbokun 1 hour ago | parent
phyzome 1 hour ago | parent
If your definition of "works" includes "doesn't have vulnerabilities", you're going to need to do more than just poke at it and add tests.
sixothree 58 minutes ago | parent
edwin0 8 hours ago | parent
ch_sm 6 hours ago | parent
seanw444 2 hours ago | parent
webarchitect 5 hours ago | parent
fractallyte 4 hours ago | parent
Occasionally, Claude may not be able to fix the issue – and it may turn out that Codex can, because it has a different approach.
Either way, at least two LLMs is the only way to get (mostly) reliable code.
mattbrewsbytes 7 hours ago | parent
My point is if you don't understand the fundamentals of the underlying code (SQL in the example above), you're more likely to have issues in the future. Or maybe not if that app doesn't reach a usage pattern or complexity where the nuances under the hood matter.
It really depends on what you're shipping. Meta ships social media apps where in the grand scheme of things if something is broken software wise it largely doesn't matter. What's the worst case? lost ad revenue? Users can't do things? If that were software for MRI machines or aerospace where software issues/bugs can result in lives lost then it matters much more in understanding what is going on at all levels of the software and potentially hardware.
billylo 7 hours ago | parent
ulfw 7 hours ago | parent
sdevonoes 6 hours ago | parent
The ones who get to keep their jobs will be rebranded as Maintenance engineers with a lower salary. VPs and Heads of will keep their cozy job (bc there are not so many of them), and they are not that dumb to fire themselves.
Code understanding doesn’t matter anymore (or won’t matter soon)
benfidge 6 hours ago | parent
lp92 5 hours ago | parent
What I do is ask Opus/Gemini to explain the parts I don't understand. And I do some of my own research to verify accuracy of the explanations.
In addition:
* Every new feature has a Feature Design document which I review, modify, and approve. And a corresponding Implementation Plan which I also review and approve.
* Every fix has a RCA document and fix Implementation Doc which I review, modify, and approve.
* There's a global high level Architecture and Design document with additional sub component design documents which are always kept in synch after a new feature or bug fix is done. Synching it is always a step in the Implementation plan.
This does adds some overhead but it allows me to have have check point gates. This has often allowed me to catch misunderstandings and align properly before any code gets edited by the AI agent.
Having the above docs also helps when it comes to referencing them as context to properly ground new features or bug fixes.
Alexsky2 2 hours ago | parent
bitbasher 5 hours ago | parent
Stop shipping code you don't understand.
Is there really any other solution? You can either ignore it and pretend like it's the new era of software like half the other commenters are, or you can sit down and understand the code and what you're shipping.
twelve40 2 hours ago | parent
ex-aws-dude 5 hours ago | parent
Is it a company culture thing or a domain specific thing?
webarchitect 4 hours ago | parent
So how do we build confidence anyhow when we can't know everything?
I think first, standard checks - does it compile? What do static analysis tools say - code quality, passes SAST analysis? Linters? Style checkers?
Do the tests run? I'm not comfortable with the tests I write let alone the AI.. by adding fault injection (fuzzing, chaos), and mutation testing, I have a better degree of confidence in the tests and code.
Spend even more time getting to know the product instead. Maybe I don't know the code as well as I used to, but I know the product backwards and forwards. That helps me have some new confidence that I didn't really have before. Knowing the greater whole.
chrisjj 1 hour ago | parent
Avoid known unreliable tools?
combyn8tor 1 hour ago | parent
p2detar 32 minutes ago | parent
namrog84 2 hours ago | parent
For me I dont look too closely at the implementation but I look at the api where if a human was using my framework how is the code they'd have to write and engage look at. Then I also do plenty of performance against a comparable target in my case slate (default unreal) and keep overhead minimal.
So good performance and reasonable public facing api are the 2 main pieces im focused on. Less with the internals.
Care about the seams and interaction and the things you can measure or count.
chrisjj 2 hours ago | parent
Recognise you aren't the one shipping it.
hackerbrother 2 hours ago | parent
fnoef 2 hours ago | parent
frumplestlatz 2 hours ago | parent
fnoef 2 hours ago | parent
akersten 2 hours ago | parent
This will not only not prevent CVEs, it will also make everything a giant pain in the ass
fnoef 1 hour ago | parent
frumplestlatz 1 hour ago | parent
Does the LLM understand it, in detail? Can you trust its interpretation of the information and instructions you provided? Did it independently make an adverse decision you haven't noticed, because you didn't read the code? Was the model missing any information it required -- and would you know if it was?
LLMs depend on humans and human oversight to ensure they have the information required to make the right decisions, to catch when they don't, and to steer them based on your much more complete understanding of the broader context and requirements. You can't rely on them to do that thinking for you (yet?).
fnoef 1 hour ago | parent
You said LLM need human oversight and it’s true, but it does not imply you have to read the code.
I think we are going towards a place where code no longer makes sense, you just describe what you want in English, and the LLM just builds directly in machine code. You then build different verification tests to make sure the program is doing what it’s suppose to do.
bigstrat2003 53 minutes ago | parent
LLMs cause errors at a noticeably higher rate than human devs do. If you lean on them without actually checking what they wrote, the error rate in your software is going to go up.
KaiserPro 47 minutes ago | parent
Yes, there are more CVEs discovered every day, but, if you don't patch, the hacker doesn't have to try.
4ndrewl 2 hours ago | parent
If you're an LLM prompter shipping products, you're outputs are more akin to a product managers, not an software engineer's.
techscruggs 2 hours ago | parent
mikeocool 2 hours ago | parent
It's hard to hold an LLM accountable for anything.
theK 1 hour ago | parent
i_love_retros 1 hour ago | parent
Software engineers are always responsible for their code except now they have to be responsible for Claude's code
4ndrewl 1 hour ago | parent
Silhouette 1 hour ago | parent
Which is - of course - absurd. Accepting responsibility without demanding the corresponding authority is a fool's game.
I have never been much of an advocate for unionisation in the software development industry. I didn't think it was necessary to protect the interests of the workers because for most of the industry's history the workers have largely been able to stand up for themselves anyway. Likewise I've never been a big believer in accreditation and licensing practitioners in our field - because who is qualified to judge someone else worthy in such a young industry with so little hard data about what works or doesn't work? I find my views on these subjects are evolving now.
It's not about protecting jobs if they truly aren't needed any more or protecting people working in the industry who refuse to move with the times. The Luddite philosophy isn't going to get us anywhere here. And besides - if AI is as effective as its advocates have been claiming then there should soon be more demand for capable developers to take advantage of the new opportunities. (The Jevons Paradox is relevant here.) But it is about protecting standards. If the tools aren't doing a good enough job then expecting professional developers to ship the code produced by those tools even though they know it's junk is something that needs to be challenged. And if developers are being expected to ship anything important without having the slightest idea whether it even works then that's even worse.
We had a lot of silliness going on when the very low interest rates and huge amounts of VC funding were propping up ideas and practices that would never have survived otherwise. Today we have a different kind of funny money distorting the industry but it's still propping up ideas and practices that we wouldn't normally support or perhaps even tolerate. And just like before no individual is going to be able to hold back the tide and it's going to have to be a collective effort to push back - hence my evolving views on topics like unionisation and accreditation. We've all seen the drop in quality across much of the industry and we all know why it's really happening because we see how the sausage is now being made.
p2detar 1 hour ago | parent
majorchord 2 hours ago | parent
Or at least, I don't think people should.
To me this completely goes against the whole point of engineering.
IMO If you don't understand the code, then you're just a manager now. Soon your salary may also reflect as such, assuming you get to keep your job.
orblivion 2 hours ago | parent
dboreham 2 hours ago | parent
pizzafeelsright 1 hour ago | parent
You trust the label. The person. The company. You hope someone did their due diligence.
Sometimes taste the shell. That is an acceptable risk.
michaelt 1 hour ago | parent
pizzafeelsright 2 minutes ago | parent
orblivion 35 minutes ago | parent
(made a small edit)
pizzafeelsright 14 minutes ago | parent
epihelix 1 hour ago | parent
orblivion 36 minutes ago | parent
mrbombastic 1 hour ago | parent
solatic 2 hours ago | parent
Time will tell whether we were right to move from a system where we trusted humans to understand the low-level system, to one where we trust fancy calculators. But so far, it's working.
akomtu 1 hour ago | parent
solatic 20 minutes ago | parent
Well, the fancy calculators do. Yes, they still need guidance to keep from going off the rails. But, at least from personal anecdotal experience, I need to go spelunking into the codebase less and less over time. The agents and models really did just start to become that good.
Silhouette 37 minutes ago | parent
There seems to be disturbingly little objective evidence of that. There's certainly a lot of code being written but how much actual value that code is adding is another question. With the scale of resources that have been pumped into AI-supported software development over the past few years where are all the new applications and the quantum leaps in quality and performance in the applications we already had? If anything I seem to be seeing more anecdotes going the opposite way so far.
solatic 24 minutes ago | parent
Disturbingly little evidence? I would hardly call that "disturbingly little"...
gavinray 2 hours ago | parent
The further away I got from the implementation details, the fuzzier the semantics became. To the point that now I could not in good faith say I understand the system I have built from the ground-up and worked on the last ~4-5 years.
The solution to this, in my mind, is not to revert back to reviewing implementation details. The economics of that from a business/productivity perspective don't play out well.
The only viable solution in my mind is adapting code/languages themselves to the shift by making programs formally verifiable. That way you can be sure that the code "does what it says on the tin", and provably so.
I feel so strongly about this I've written on it:
https://gavinray97.github.io/blog/design-by-contract-and-eff...
oblio 2 hours ago | parent
You weren't programming with LLMs in 2021 or 2022, be real. More like 2024 (that's already pushing it) or 2025.
gavinray 1 hour ago | parent
epihelix 1 hour ago | parent
The point is that it's easy to lose understanding of the finer details, even of a codebase you lovingly created, once this switch occurs and an LLM gets its hands on it.
pizzafeelsright 1 hour ago | parent
The bridge is built. How do we test the load? Drive bigger and bigger trucks until it fails, note the weight, print the sign and rebuild the bridge the same way.
torben-friis 2 hours ago | parent
"This new way of piling bricks is so fast that I can no longer afford to check the buildings integrity".
I just don't understand how the fuck we ended up with this narrative. Somehow the machine making things twice as fast forbids us from spending time ensuring quality? Surely if it does things faster you have more time and tools to check!
I'm half convinced this frenzy is the marketing work of AI companies at this point.
chrisjj 1 hour ago | parent
Worse. "AI" is a self-marketing contagious delusion, driven by an inexhaustible power supply. Human gullibility.
mooreds 2 hours ago | parent
You're going to have to understand it one way or another, either at writing or at debugging. Even if you delegate all debugging to AI, you'll still need to determine what is "right".
If you don't understand the code at writing or debugging, you'll pay the architectural price (where it degrades). That price may be worth paying, but that depends on the system you are building.
IAmGraydon 2 hours ago | parent
The problem here is that you're trying to do a job that is already being done for you. Maybe you don't yet trust it to do a good job, but that is quickly changing.
petesergeant 2 hours ago | parent
Barrin92 2 hours ago | parent
easy, don't write code with LLMs. If my job forced me to do so I would quit.
pizzafeelsright 1 hour ago | parent
Barrin92 1 hour ago | parent
scooblydoobly 28 minutes ago | parent
internet2000 2 hours ago | parent
platevoltage 45 minutes ago | parent
ryandrake 15 minutes ago | parent
To me, "It works" is pretty much the lowest possible quality bar for an acceptable creative work product. We should not aspire to "it works." We should not hold ourselves only to the "it works" standard. And, yes, I know that many traditional software companies fail to even meet "it works" for their shipping products.
I don't even know what to say anymore. This has become a deeply unserious industry putting out mountains of junk that barely even meets the "it works" bar, and LLMs are just accelerating this.
movsx 2 hours ago | parent
tcdent 1 hour ago | parent
Personally, with side projects and tooling, go nuts. There's no reason to read all of that code.
KronisLV 59 minutes ago | parent
I feel like an increasing amount of orgs out there have long abandoned this thinking.
vjvjvjvjghv 1 hour ago | parent
asimovDev 2 hours ago | parent
dboreham 2 hours ago | parent
ChrisMarshallNY 2 hours ago | parent
Lots and lots of testing. I usually interactively test with the LLM. Now that I'm spending less time, coding, I get to spend more time, testing.
In my experience, I test more, with an LLM, than I did, with my own code. I don't find coding errors, but I do find implementation errors.
On another note: I ran an LLM on a bunch of really mature code that I wrote, and could swear was "perfect," and it found a number of issues.
Nobody's perfect. Trust; but verify.
SimianSci 1 hour ago | parent
What have we gained aside from shifting where time and effort is spent in the design loop?
thevinter 1 hour ago | parent
theK 1 hour ago | parent
- code? up - build budget? Up - developer output? Up - developer oprrational costs? Waaay up!
Why is this Bad? Every one involved profits somehow.
alpha21io 1 hour ago | parent
onion2k 38 minutes ago | parent
The verification is what we always wanted but never had time for. For 30 years I wrote side project games without extensive testing or a fuzzer or anything because I wanted to spend my time writing the game. Now I can do that and still have the testing I always knew I should be building.
ChrisMarshallNY 19 minutes ago | parent
Right now, we're in a "honeymoon" stage, where non-engineers think that they can somehow, magically produce stuff that is comparable to what experienced and disciplined engineers can do.
That won't last long.
Despite all the caterwaulin', LLMs are just another engineering tool, and good engineers will figure out how to implement them, in ways that absolutely blow away the garbage produced by non-engineers.
It's like that rich, no-talent guy that buys a PRS guitar, and thinks he'll beat the experienced musician that still uses his beat-up old strat, but consistently makes hits.
michaelt 1 hour ago | parent
If someone finds it burdensome to review my 50-line AI-generated bug fix, adding 200 lines of tests only gives them more lines to review.
AnotherGoodName 1 hour ago | parent
I have no shame in accepting vast quantities of ai generated automated tests at review time without really paying attention to them. It’s better to spend more time on the core implementation.
Silhouette 1 hour ago | parent
I respectfully disagree. IME writing a good test suite is one of the areas where current LLMs perform worst. They certainly like to write plenty of tests but those tests aren't necessarily adding much value and they don't necessarily cover the important functionality.
Given how heavily LLMs and specifically agentic workflows still rely on producing something vaguely correct and then iteratively applying guard rails to reach something that actually works I humbly suggest that trusting a very substandard test suite to be one of those guard rails might be unwise.
flir 10 minutes ago | parent
I realise this is heresy, but my conclusion is that unit tests aren't important any more. E2E testing on the other hand...
(I'm agreeing with you about the quality of LLM tests, I'm just trying to make lemonade out of all these LLMons).
AstroBen 1 hour ago | parent
Black-box e2e testing isn't suitable, or practical, for a lot of applications. Unit and integration tests generally require you to understand the code you want tests for, to know what to actually assert
Are you just trusting the AI to do it all?
avgDev 56 minutes ago | parent
g-b-r 1 hour ago | parent
Even if your test suite tested every single possible behavior and code path, tests usually can't intercept all possible unintended behavior that an LLM might have added.
In all likelihood, most test suites don't intercept any unintented behavior (such as unintended internet connections).
That's not a big problem when a honest human has written or reviewed the code, because unintended code is usually easy to spot.
But an LLM might have dropped any whatsoever code: maybe some of your project's code resembles the one of a trojan in its training, so it completes it with the trojan.
Or maybe it found that if your server is unreachable, it can upload the client's data to a public Github repo; or that if there's no space left on the device, it can delete some of the user's folders, easy!
Would your clients be understanding if that happened?
ChrisMarshallNY 34 minutes ago | parent
Eventually, the tools become trustworthy, and we no longer need to worry about their products.
We're just about there, with frontier LLMs, but I don't think we're all the way there, yet.
AstroBen 8 minutes ago | parent
English isn't a good programming language if you want to trust that the output will be as you want it to be. High level code has no room for ambiguity - that's why we use it. It's a concise, very explicit language to tell the compiler exactly what to do.
Dealing with an LLM is a fundamentally different thing. The reason it's able to speed us up is because it makes assumptions for us and reads between the lines. The only world where we trust the output is one where it can literally read our minds. That's not coming any time soon
onion2k 40 minutes ago | parent
The goal is to get to a point where I can say with a good level of confidence that a change didn't break anything. It might be the wrong change, and it might be crap code, but I can ship it without anything blowing up. I've been using it for a while across a few game projects and so far it's worked quite well.
SimianSci 1 hour ago | parent
Models and harnesses have gotten really good, but they still are hard-capped by the operators that use them. The sycophancy of models is perhaps their greatest weakness, as when a user who might be experienced in one area attempts to use AI to build within an area they are less familiar with, the model does not take a step back and examine the "why" of what is being built from an architectural standpoint, it simply performs the expected task in a direct manner, ultimately creating sloppy work that may be easy to ship but difficult to maintain.
A good example of this was a recent interaction I had with a Dev who was very familiar with how to develop embedded and system level applications. He and every other dev are being asked for increasingly more due to AI, and he had to bring something he had built as a single-user desktop application into a distributed cloud environment. He ended up rsyncing a copy of his working directories including multiple credentials and sensitive internal project documents to a virtual machine binding it to a public ip address. Thankfully our tooling detected this before he could get any farther but it illustrates how the AI model isn't doing any real thinking or engineering at the end of the day. Unless the operator knew to instruct the model about proper cloud architecture and understood best practices around deployment of public systems, the model simply did what it was being told to do by a user who didn't know any better.
Another example of this is the sheer amount of unsophisticated users we've been detecting using Javascript injection into their web browser sessions as a form of automation for web services they may use. These users would never know to ask how to request an API token which may allow them to accomplish things in a much easier and approved way, and instead are relying on the frailest possible automations that expose them to risks of credential exposure.
We've got to be careful with how we engineer things with AI models. Just because we've made boilerplate easier to generate doesn't mean that we can avoid the actual engineering of what we do. And I can't help but think that firms which are fully committed to AI code generation, won't just be introducing an avalanche of slop into their product which will explode the costs of maintenance over time.
matltc 1 hour ago | parent
My dumb little utils in ~/.local/bin? Those get merged after a quick functional test without my having read a single line of it. Still, the skill I maintain for such a job directs agents to write such throwaway scripts in anything but shellscript because I might want to extend some day and fuck shellscript
YorickPeterse 1 hour ago | parent
The mental gymnastics developers are willing to go through just so they can avoid standing up for something is quite perplexing.
vjvjvjvjghv 1 hour ago | parent
It also took the industry quite a while to accept that server code needs to be secure even if that requires a lot of extra effort. AI will probably be the same.
netburst 1 hour ago | parent
ramesh31 1 hour ago | parent
archagon 1 hour ago | parent
By doing so, you debase your own profession.
skeledrew 1 hour ago | parent
gonzalohm 1 hour ago | parent
skeledrew 1 hour ago | parent
gonzalohm 7 minutes ago | parent
episteme 1 hour ago | parent
bsoles 1 hour ago | parent
skeledrew 1 hour ago | parent
AstroBen 1 hour ago | parent
Can you break down for me exactly how the unit and integration tests are made? Do you ask the AI what to assert?
skeledrew 55 minutes ago | parent
There should be enough detail in the spec that it's pretty obvious. And unlike - many - human devs, agents don't shy away from writing tests. Add a coverage guardrail and tell them to use red/green TDD and you're already a good way there. The main thing is to ensure shared understanding, so they also know business expectations and write appropriate tests. And... nothing here is really specific to an agent, as a human dev also needs sufficient guidance (tailored to their experience level).
AstroBen 42 minutes ago | parent
Here[0] we have benchmarks for AI writing tests for code that already has the correct behavior. Best case scenario is that they're at 67% with Fable, and that's with the highest quality specification we could possibly have (code). English is far more ambiguous
bluefirebrand 1 hour ago | parent
I have never in my career been afforded the time to make a complete spec.
skeledrew 47 minutes ago | parent
lijok 1 hour ago | parent
Noone understood the code to begin with. People understand the concepts, the primitives, the "theory" as Naur said, the basis upon which a project is built. Noone ever understood a codebase worth understanding to begin with.
bigwheels 1 hour ago | parent
cadamsdotcom 1 hour ago | parent
If the agent has written tests and crucially - witnessed them fail before starting the work to make them pass - you have a lot of confidence.
Review of your own work's easier too - test and production code tend to mirror each other, giving you a much more complete picture. Confused by the production code? Go read its tests for a bit, then come back.
It's also easier to understand the change if you were there for the build, watching the transcript, checking in every so often. Doesn't have to be your primary task - leave it on a second monitor, it'll catch your eye naturally.
And finally, on top of test-driven development, have the agent first (before the first test driven development cycle) build out an end-to-end test covering the work it'll do. Once that test is green (test driven word for "is passing") you're done with your work and can clean up at will.
Play with it before shipping, get it real nice. It's never been easier to clean the tech debt up and make the UI perfect.
Eventually you'll be ready to package it up into a PR!
Now's you can leverage agents and get creative. Make your colleagues froth to review your work. Consider making a Loom video to walk them through it. Consider reworking it into a commit series; it only takes a few second for your agent and you can tell a story for humans. Maybe you want to sequence it according to your struggles; maybe it's better to show the different layers emerge with each commit - it's up to you. Consider also that your agent can easily do one-off visualizations of the change for your colleagues if you think it'll help them, and you can attach those too - mermaid charts or even whimsical AI generated images. All this is because motivation is driven entirely by other humans' motivation and you want them frothing with excitement every time you drop a review. Make it fun, it takes seconds and gets your stuff merged fast while making your teammates smile.
This is a brand new job and we will work out how to do it together :)
SimianSci 1 hour ago | parent
There have been multiple times where my software dev team has gone back to prior implemented tests they had the model generate and found that after some time eventually the model got frustrated with the test no longer passing eventually modifying it to fit current needs around a feature push. You can probably anticipate how we came to discover this, as it ended up introducing a regression around multiple edge cases which the test was meant to control for.
The only answer is to inject actual engineering into the process as much as possible without giving it all up to the models to generate more slop in response to slop. tests are a great answer, but only if you treat them with serious intentionality.
cadamsdotcom 1 hour ago | parent
There are multiple places in my post where I explicitly describe the process by which you intimately:
- make the agent do verified work (the e2e tests and test driven thing)
- watch the agent work (the second monitor thing)
- read your own code (the tests/prod mirror thing
- play with the work and perfect it (the tech debt and UI perfect thing)
Plus a ton of other places in my post I very clearly describe being intimately involved in building and selling the work.
That to me is basic god damn competency. What your engineers did is a fad, and it won't last, and I'm sorry they abdicated their professional duty.
If you wish to screen for that sort of shit in future/current hires, show them my post and see if they turn up their noses or if they get excited and creative. You want only the latter.
jorgeleo 1 hour ago | parent
Recommend reading (All though a little self serving): https://www.linkedin.com/pulse/ai-code-quality-jorge-varas-a...
rubendev 1 hour ago | parent
gonzalohm 1 hour ago | parent
remywang 1 hour ago | parent
vjerancrnjak 1 hour ago | parent
AI still does not know how to even write code that makes impossible state unpresentable so testing won’t help.
09127398630 1 hour ago | parent
On the other hand, not even surprised anymore most people even try to provide serious answers to this. Welcome to the pure insanity.
trezm 1 hour ago | parent
spike021 1 hour ago | parent
I then make sure they write a lot of tests and that if functionality is changed or added/removed then tests are updated alongside.
Generally I try to review changes myself and ask it to explain the changes.
It's not perfect but I feel like the planning step being iterative isn't all too different from my old research -> think about implementation -> implement loop.
At that point, reviewing any code changes goes hand in hand with how I coded before LLMs.
The major difference is I usually have it make any changes for things I don't like.
bitwizeshift 1 hour ago | parent
By that I mean that, rather than let the LLM do the entire structure, I take the time to create the full hierarchy of abstractions and actors that I want my code to do. I decide, at the UML level, how abstractions will sit and interact — and then I make the LLM implement those gaps.
The key thing here is that, by doing this, I DO know how the code works. I may not know how each unit is implemented, but I know each unit’s responsibilities and how they interact with one another. I’ve found that LLMs are HORRIBLE at writing decent abstractions, regardless of how much prompting and skills I’ve given it. It will always get the job done, but it will often be unmaintainable and require bigger and heavier refactors as I introduce more and more requirement changes.
But by flexing my basic design skills, I don’t have those issues. Future refactoring fit into easy abstractions that I’ve designed. And by documenting it up front, I also have what becomes a living document on how this work will turn out.
It takes a few hours up front to plan ahead, but it makes the LLMs job easier and use less tokens, and it makes the code better long term. I don’t care if one unit is suboptimal, as long as the abstraction is right — because at worst I can change an implementation for a better one later.
forestry 1 hour ago | parent
mentos 58 minutes ago | parent
Of course I only arrived at the framework after trial and error with an LLM showed me the shape of the problem and the framework presented itself to me.
apinstein 52 minutes ago | parent
I have found that making the AI look the plantuml and the rendering and reviewing before presenting it to me results in much more legible designs. To me the peak of this would be a shared interactive whiteboard w the ai during design.
smashed 52 minutes ago | parent
It can keep patching everything left and right whenever a new requirement comes up so it does not "need" to build abstractions, and it will only do it to please the human reviewer.
j45 50 minutes ago | parent
Planning is fundamentally so critical to ensuring alignment and outcomes, that over time how one plans can be better trained into the process to help do more and more of it.
roadside_picnic 49 minutes ago | parent
Are you working on a team? I've found that for solo projects this works but, even when you own the design of your piece, integrating with the work of several people doing this ultimately leads to even the part you initially designed no longer working exactly as you remember it.
A good chunk of my time now is spend working with an LLM to understand how other parts of the code base work and how they relate to and interact with the product features I manage. But I'm still on a small enough team that this is possible. I'm not surprised at all that an engineer at an org the size of Meta would not even be able to perform this level of work in understanding what's happening.
eggplantemoji69 45 minutes ago | parent
fasterik 38 minutes ago | parent
swatcoder 33 minutes ago | parent
JohnMakin 9 minutes ago | parent
I'm old enough now to be a protected class, so I don't want people accusing me of ageism, because that's not where this is coming from - but there's a type of person that clearly is on their last job of their career and knows it and I try to avoid them like the plague, I've seen it too many times.
trio8453 31 minutes ago | parent
fasterik 30 minutes ago | parent
trio8453 20 minutes ago | parent
Sometimes, not enough to believe you that you can do it better. And even if you could, to believe that it's overall worth it to work in that way.
birdsongs 19 minutes ago | parent
Way to come off as incredibly insecure. There's no need to randomly put people down.
trio8453 18 minutes ago | parent
birdsongs 15 minutes ago | parent
That's it. Maybe you should go re-read?
fluoridation 13 minutes ago | parent
chasd00 11 minutes ago | parent
birdsongs 24 minutes ago | parent
I just approve PRs, because I have to. I don't know what else to do. I can barely keep an eye on all my code.
Actively looking, but job market is rough in Northern Europe. I quit caring about quality in any capacity here, the company has clearly chosen its direction, against my very vocal complaints.
My last interview, with what I thought was a great company, failed when I didn't understand what NLP was when they asked me about it.
> NLP? > Natural Language Programming, it's the future. Claude, GPT, etc.
I laughed, he didn't, looked offended and said it was a serious question. I had thought it was a joke, said "oh you mean LLMs". I tried to answer seriously, of course, but that was my third round technical interview and I didn't get a call back.
Just getting a paycheck and producing mediocre work until I get something new. (Senior engineer, 10 YOE, patents, 3 years at a unicorn).
Also considering becoming an electrician.
fasterik 17 minutes ago | parent
birdsongs 10 minutes ago | parent
No one cares at best, or I'm callously told "keep up or fall behind" at worst. When that's not even the problem, it just, sucks now. There's no joy in anything I do professionally.
Can I ask where you went? I'm seriously looking for an exit, I just don't know what yet. Maybe a trade, in Scandinavia the pay is actually comparable, I just don't want to go back to school.
fluoridation 15 minutes ago | parent
swatcoder 7 minutes ago | parent
Often, the lack of meaningfully challenging demands being put on you is punishing on its own. It can feel like a relief for a week or month or year, but if you catch a glimpse of that being your whole future, you can come to recognize yourself as being in a mind-eroding secular purgatory.
This anguish is especially common among on highly capable people, high achievers, and those used to some kind of ambition. And by accident of its history, there's a lot of those people in this industry and some of them are going nuts now.
d1algo12 4 minutes ago | parent
d0m 1 hour ago | parent
2) Ask agents to do more than just code.
They need to fully demonstrate and prove that their PRs are rock solid. Ask for screenshots, videos, code reviews, detailed architecture diagrams, etc.
It would be an insane amount of work for humans, but AI can do it easily for each PR.
Merging any PR from an agent should be a no-brainer if there's real proof that the feature is built correctly and working.
If there's still doubt, then the agent needs to do a better job of proving that the feature is built correctly, doesn't introduce regressions, etc.
alpha21io 1 hour ago | parent
PaulHoule 52 minutes ago | parent
It's totally fair to load up some open source project into an agentic IDE and instead of asking it to write something ask questions like "how does X happen when I use feature Y?" or "can you explain how client code could accomplish Z"
All the time I see an AI use a language feature or library feature that I never used before or I see some code I don't understand and it does much better than a lot of humans do at explaining what it did.
physicles 23 minutes ago | parent
I’ve been experimenting with writing a living spec — API endpoints, schemas, how things work at a low enough level that the LLM can fill in the details - and that’s been working well.
It also forces me to do the hammock-driven development necessary to build the right thing, and as a byproduct it keeps the design in my head longer.
platevoltage 50 minutes ago | parent
lijok 49 minutes ago | parent
The code was always a black box. Always.
Specs come in, passing tests come out. If you find an issue, your specs were shit.
Anyone that has ever had to outsource work to someone else knows this. If you've ever worked with another team whose codebase you didn't touch - you know this.
You write your specs, you launch an agent to implement them. You launch a 2nd agent to keep the first one honest. When done, you launch as many agents as you have budget to try and poke holes in the implementation. As it always was the case.
You can thrash around, you can fight it. Like it or not, the genie is out of the box, and the realization you were never really good at your job to begin with is gonna set in eventually.
chromak 46 minutes ago | parent
adamddev1 16 minutes ago | parent
hahn-kev 41 minutes ago | parent
Tadpole9181 15 minutes ago | parent
Each step in abstraction we've made had people who felt uncomfortable at the level of magic they were accepting. At one point in the 70s/80s, traditionalists were arguing that C itself was too high level. Then in the 90s and early 00s, that languages like Python and garbage collection were absurd: a "good" programmer doesn't need them and they give up too much control. Then as web development ramped up, people entered the "Javascript is slop" phase into "React is slop" phase.
Anytime the barrier to entry is reduced and some skill or profession becomes more focused on output than the ego of the professionals, you can bet a counter-culture of people will rise up to oppose it. It's a threat to their sense of "betterness" and/or the (perceived) value of their career.
netburst 2 minutes ago | parent
8note 38 minutes ago | parent
not understanding the detail is quite similar to a coworker writing the code and another coworker approving and merging the changes.
ge96 33 minutes ago | parent
No I only do that for unit tests so far
I went through like a mid-life crisis when the thing I learned/spent last 13 years on became meaningless.
AbrahamParangi 32 minutes ago | parent
TZubiri 25 minutes ago | parent
don’t wash your hands, or at least don’t expect us to wash them for you
keithnz 24 minutes ago | parent
willio58 23 minutes ago | parent
xinu2020 13 minutes ago | parent
In any case at the end you're the owner/accountable so you need to justify your and your agent choices.
westurner 22 minutes ago | parent
I have some work in Lean4 that I'm certainly unqualified to validate; Layer Coding QEC derivation, Positive Grassmanians and the Fresnel-Kerchhoff correction to the Parallel-Axis theorem, Huygens intensity as an alternate model for simulating photonic and electronic interference in a double slit experiment, for example.
It was a black box but I'm letting it tutor me about it so
Visual proofs, Tools for logic and formal methods in distributed systems
Things that compile logic!
> That means you can end up with systems you've built where you don't truly understand how all the code works
Though AI coding and system development can be overwhelming due to information overload and rapid growth in complexity (and concern for unwittingly embedded mistakes, the unsecurability of agent-based development, and number of LOC of code per day), not understanding the whole system sufficiently is normal and expected in engineering.
reaperducer 14 minutes ago | parent
Imagine the carnage if real engineers started vibe-designing bridges.
cheeze 12 minutes ago | parent
I think it's silly to make strong statements like this. _Most_ professional software engineers (women too, not just him!) are using LLMs and are moving some amount of faster as a result.
adamddev1 13 minutes ago | parent
quibono 11 minutes ago | parent
It then gives me a hint on the area or a typical code flow that would hit the bug - whichever gives me less info. I can obviously ask for more hints as I read along.
Obviously all honour-code based since I could just ask it to fix the bugs itself. I think this is something that can't be fixed systemically - either you're going to bother reading and thinking through the code or not.
Other than that I guess reading the tests - to make sure they fail without the code / fix, to make sure the setup makes sense etc.
jrockway 6 minutes ago | parent
I agree with the advice to design your system before implementing it. It is pretty easy to iterate on a design with your favorite coding assistant, and this is your chance to be ultra-picky and make sure you get exactly what you want. This was always where the "software engineering" was done, but in practice it's easy to have a vague idea and refine your (internal) list of requirements / tradeoffs as you type in the code. I find it harder to do that when the LLM is generating the code, just because one turn can change so much.
Keeping it under control requires discipline, just like before LLMs were a thing. You set a bunch of engineers loose on your codebase with no vision or supervision, you get spaghetti. You just get a little bit more control now. Agents don't feel shitty for the rest of the day when you tell them to do it some other way, so you are free to review as critically as you want without hurting anyone's feelings. I think it's a good thing.