54 points matt_d 5 hours ago 22 comments
troupo 2 hours ago | parent
It also doesn't help that LLMs tend to generate a lot of unnecessary Lean code to prove even the simplest statements.
mitxela 2 hours ago | parent
2/5
QuesnayJr 1 hour ago | parent
cobychapple 2 hours ago | parent
wodenokoto 1 hour ago | parent
Sure, the code works, but does it work at doing what you want?
Sure, the proof is valid, but does it prove what you set out to prove?
oulipo 1 hour ago | parent
j16sdiz 1 hour ago | parent
Fresh today, I asked llm to fix a unit test for me, it replaced the test with no-op.
bananaflag 1 hour ago | parent
fancyfredbot 51 minutes ago | parent
It's more like assuming that because your code is formally proven correct the client will be happy with it.
The code can be right but it may still do the wrong thing.
pasquinelli 24 minutes ago | parent
dgellow 32 minutes ago | parent
throwawayffffas 4 minutes ago | parent
That it's the equivalent of saying "It's full moon tonight because I wrote this program".
And he states that you have to check that:
1. The statement the author is making corresponds to the statement in the lean program.
2. The lean program actually compiles.
3. There is no shenanigans hidden in the lean program to disguise a trivial proof as the one stated.
4. The program does not exploit bugs in lean.
red_trumpet 1 hour ago | parent
As someone with a PhD in mathematics, but no experience with Lean, I found this pretty surprising. Shouldn't there be some kind of encapsulation, which insulates the statement from the proof?
UltraSane 56 minutes ago | parent
lickonographic 24 minutes ago | parent
The very meaning of interpretation in computer science is at odds with the use in philosophy of language and that is symptomatic.
We're still fundamentally trying to advance Hilbert's program. It's like Göedel's work hasn't sunk in to our collective reasoning about reasoning. It just doesn't seem possible to build a self contained correct and truthful system of symbolic logic. Truth is in the relationship of the system with other systems.
Automating mathematics, like automating software engineering, or automating automation, are all self contradictory propositions. Anything of what we build only ever means anything in how it interacts with us. I mean, people will go round and round in this forum discussing semantics and the meta implication of these unceasing discussions never seems to hit home. Namely, that there is no universal, truthful, correct, system of symbolic logic. It seems in fact impossible to have one. But the results that indicate this impossibility are built with a system of symbolic logic and the validity therein is contingent on people's understanding and, like all symbolic logic systems (including the ones that states the limits to universal symbolic logic systems) they are bootstrapped from axioms one must chose at the start or by default and they can never be fully proved by the very system built atop them.
CJefferson 20 minutes ago | parent
If 'Challenge.lean' states the theorem, the solution can't change the theorem. The only thing it can do is do a 'sorry', just skip proving some part, but that's printed out very clearly when it happens.
philipwhiuk 1 hour ago | parent
> It is probably worth stressing again: do not run random Lean code on your local machine unless you sandbox it first. Lean’s mathematics library is safe to run and build upon because this is human-curated and checked by human experts.
I was surprised by this - does Lean need the ability to run arbitrary code? It feels like limiting Lean to not being Turing complete would be an advantage.
yorwba 38 minutes ago | parent
DoctorOetker 1 hour ago | parent
96 cores and 500 GB RAM to ... verify FLT?
The huge size of the proof is also remarkable, since brute forcing a long sequence of steps is much harder than brute forcing a shorter sequence of steps.
I thought the whole "tactics" approach in Lean was supposed to permit shorter proofs? to skip the low level stuff?
If I formalize my short proof of FLT in MetaMath, it would just be a small file and any computer would be able to verify it.
This reflects so poorly on both OAI as well as the Lean project.
barrenko 1 hour ago | parent