113 points Brajeshwar 1 hour ago 96 comments
jmclnx 1 hour ago | parent
Story time :)
I remember when I was first starting out as a programmer. At QE (always occurred on a weekend) a slew of financial reports were created and printed for review the following Monday then I think used for summaries for Wall Street to look at.
A high level finance manager came to me and asked if the report could be changed, not knowing what he was asking I said sure. After a long discussion turned out he wanted the numbers on the report changed and re-printed. I said it was possible but the numbers would be all wrong.
So he said "do it". I went to my manager and told them what he wanted, then I refused to do it. I said I would quit first because I did not want the SEC to come after me. Luckily they all supported me and the finance manager was told to "FO".
FWIW, that company is long gone.
dofm 46 minutes ago | parent
I had to make this conditional promise once, back in the nineties, as a junior person in a related industry. Because in the dot-com era developers like me were often simultaneously a babe-in-arms on a professional career level and also the only person who knew how to do the work that was being sold. I decided that if it came to it, I would not be the person having a hard time in court. The client understood better than my bosses and they changed the product. Took them a few minutes at most, but I have reason to believe that my decision made some ripples.
Nowadays I don’t let those issues surprise me: I believe in establishing specific ethical standards at the beginning of projects, because if you don’t do that, you don’t have a basis to robustly point to actions that put them at risk.
ArcHound 1 hour ago | parent
Imagine: you start working at the company with a good will towards them. As you learn more and see more decisions, you slowly realize how bad it is. Now you've already contributed. Then you leave.
It might be possible to build everything with this goodwill and that's how we'd still end up with a torment nexus even if you immediately drop it.
What I'm saying is, at one point you have to break the company fully.
I'm not sure what more we need to wait for, if the claim "10% it ends humanity" is not enough.
sandeepkd 55 minutes ago | parent
Good people can be in tough position where even when they do not agree they may not have much of an option but to keep going.
ArcHound 40 minutes ago | parent
What I wanted to say is that usually people start with good intentions and only later they realize they are contributing to a torment nexus.
What you are saying definitely plays a role and it makes doing moral choices difficult.
godwinson__4-8 31 minutes ago | parent
seiferteric 21 minutes ago | parent
ArcHound 15 minutes ago | parent
Because I'm seeing mainly hacker attacks, job insecurity, environmental impacts and vague threats with smiling faces.
mofeien 9 minutes ago | parent
But the reality is that experts' probabilities for human extinctions vary wildly, with the lab leaders at 20%-30% and these researchers at >10% (he didn't state how much higher...), LeCun at <0.01% and the other two godfathers of AI, Bengio and Hinton at >10%.
And this decision to take our shot is not taken in a democratic way by humanity as a whole, but by a few companies racing as fast as possible.
So as long as no one can put an upper bound on the extinction risk, development must be globally shut down.
carterschonwald 1 hour ago | parent
phyzome 1 hour ago | parent
Yes, some members of the group don't have that freedom. But that doesn't excuse the ones who do.
calcifer 59 minutes ago | parent
> If the answer is the salary, say it louder, and do not force yourself to make excuses.
throwaway713 58 minutes ago | parent
rglullis 56 minutes ago | parent
skrtskrt 43 minutes ago | parent
pizzafeelsright 38 minutes ago | parent
moondistance 1 hour ago | parent
vvpan 57 minutes ago | parent
zaphar 13 minutes ago | parent
agumonkey 57 minutes ago | parent
rglover 42 minutes ago | parent
jyounker 35 minutes ago | parent
cluckindan 1 hour ago | parent
rochansinha 1 hour ago | parent
People do things.
Good or bad is relative to their own biases/experiences/ beliefs and moral compass.
Very few people are truly sociopaths who do evil stuff for personal gain.
Most do it because of the curse of the “banality of evil” (re: <https://en.wikipedia.org/wiki/Eichmann_in_Jerusalem?wprov=sf...>)
Doesn’t make it right, but like everything else in life the reality is much more complex than saying “good people refuse to do bad things”
Sha1rholder 54 minutes ago | parent
spencerflem 49 minutes ago | parent
My current read is that ~20-30% of people in America actively delight in hurting others (ICE supporters etc)
Another 30% do not seem to care about anything but their own well-being. (I don’t like to think I fall here but as a big tech peon it doesn’t look good).
The remaining percent seem to care and are trying their best but looking at how things are going it’s not enough
chadgpt8 23 minutes ago | parent
dgellow 1 hour ago | parent
api 54 minutes ago | parent
I studied biology in college. I actually did this because I was interested in AI, and already knew a ton of CS and programming. I was already coding in C and using Linux and writing fairly complex stuff. I figured studying the only actual example of a truly intelligent system (especially at the time, early 2000s) was the best way to understand intelligence.
Didn't end up going into AI as my day job, but I've kept up with it all and worked through the math and played with different types (genetic, neural nets, transformers/LLMs, CA / alife, etc.).
I think the root of my skepticism is in what I learned studying biology about how life, adaptation, learning, and evolution actually work in the real world.
I'll try to TL;DR it: I think the doom crowd (and also some of the AI singularity crowd) has a "mystical" view of intelligence. Intelligence is magic pixie dust, and intelligence beyond human is fully unknown and likely has magic superpowers.
The deeper underlying thing seems to be some kind of mathematical mysticism.
Real intelligence is ubiquitous in nature. Trees, grass, bugs, and even bacteria are intelligent. It's also domain specific and limited in capability to what its domain, available training data, and available compute allows.
It's not context-free limit-free magic. It's not special. It's an adaptation like any other.
spencerflem 40 minutes ago | parent
& ofc AI is making climate change worse, some of the Amazon datacenters are using fossil fuels to generate a country’s worth of power, but that’s never part of their calculations
mofeien 21 minutes ago | parent
Which one of these would it be? All of them in parallel? Or something we wouldn't think of. If I were to play chess against Magnus Carlsen, I wouldn't know with which piece he will checkmate me, but I know that I am going to lose.
petesergeant 38 minutes ago | parent
dgellow 30 minutes ago | parent
Both OpenAI and Anthropic are from that world, where Rationalist doesn’t mean you’re a rational human being, but instead that it is rational to believe that AGI will be created, thus that they should be the ones creating it. It’s a doomsday cult.
If you start looking at the links between prominent rationalists and effective altruists you see how that whole doom prophecy came about. It’s also how they got so much capital in the early days, SBF literally invested into Anthropic as part of his Effective Altruism effort foundation, and both Amodei siblings are deeply into that whole world
api 28 minutes ago | parent
I still can't believe anyone ever took Roko's Basilisk seriously, but people did... to the point of freaking out about it.
The groups themselves are a little cultish too, or a lot cultish depending on which one you're talking about. Some of them are full-on cults.
But what do I know. I'm probably low IQ.
Jtsummers 12 minutes ago | parent
If you've spent much time in US Christian circles you've probably encountered people suffering from apocalyptic thinking. The Roko's Basilisk folks seem like the atheist version of the same crowd. It's hard to take them seriously when they're always seeing portents of the end times, and their thinking is clearly clouded by anxiety and fear and not based in reality.
bena 4 minutes ago | parent
Once again, I've said it elsewhere here, I think we could have nipped this thing in the bud if when the first "effective altruist" started talking about the EV of saving billions of "future lives" vs dealing with today's real problems, someone looked at him, cocked their head to the side, and said "That's fucking ****".
All of those magical future people do not get to live if we bake the Earth today. The best way to take care of future people is to take care of people today. And if they were truly serious about taking care of future people, that's what they would be doing. But it's not, they're just collecting resources. I don't know how the fuck Sam Altman owning a Koenigsegg Regera today helps save 20 quintillion lives in 40,000AD. Deeply unserious people.
mofeien 37 minutes ago | parent
2. Humans put it in charge of everything, as fast as possible, because it's just so damn useful
3. Is it perfectly aligned to human values for all of future ????
4. Everybody dies as a byproduct of the ASI pursuing some project that humans don't even have a chance to understand, like an ant colony during construction of a hydroelectric dam.
bena 11 minutes ago | parent
If so, he's the person that's made me lament one loss the progressives took from us. I think there's one phrase that we should be allowed to say regardless of the fact that it is also used to refer to certain groups of people. Because the insult comes from the comparison, not the language itself. Every time we update the language, we just use the new word as the insult. And I don't think we're allowed to use the word here.
But sometimes, you just have to look at someone, cock your head to the side, and say "That's fucking ****".
It brooks no argument, it offers no sympathy, it just tells the other person that they should be ashamed for speaking those thoughts aloud.
oconnor663 58 minutes ago | parent
Good lord. You could apply the same logic to any taxpaying resident of any country whose government is doing something bad. There's some truth to it, it's worth bearing in mind, but it's also the very definition of an Isolated Demand for Rigor.
petesergeant 54 minutes ago | parent
That is a fantastic term for something that seems to come up a lot, and I had not heard before.
oconnor663 40 minutes ago | parent
wonnage 24 minutes ago | parent
The man is the definition of a useful idiot https://slatestarcodex.com/2013/08/30/military-strikes-are-a...
applfanboysbgon 47 minutes ago | parent
You can't, because those taxes are coerced from you at literal gunpoint if you refuse (or your property seized at gunpoint to the same effect). It's completely different than choosing to do that of your own free will.
When I was in the US, I did actively attempt to diminish my tax contribution as much as legally possible out of disgust at what it was used for, but nobody is morally tainted for being forced to contribute a small slice to something evil. Maybe there's a threshold where you should be actively laying down your life to resist, eg. if being forced to commit actual murder, but I don't think you can reasonably expect that of paying taxes at all.
---
Edit: rate-limited, attaching a reply to a below comment.
> The problem with avoiding taxes because of expenditures you think are bad is that you also reduce the pool available for things you think are good.
Indeed, but I also chose to contribute money to other good causes instead. In that way, the same sum could be used for 100% good instead of, say, 80% good 20% bad.
Now that I live in a country of my choosing, I am proud to pay taxes because of how they are used. 0% of my tax money goes to killing innocent people overseas, for instance.
gretch 40 minutes ago | parent
You should beware that you've rationalized everything you do as good enough, and everything that other people do is morally questionable.
Have you set the bar for what is 'reasonable' to be extremely convenient for your own circumstances?
applfanboysbgon 38 minutes ago | parent
gretch 16 minutes ago | parent
This is not what I was trying to say.
First let me state the assumption I made: You think that working at Anthropic/OpenAI is morally bad because that's the parent thread, and you made a comment about how somethings are good enough and other things are too bad to tolerate.
So yes, you hold yourself to the bar of I should not work at Anthropic/OpenAI - but the point is that this bar is convenient to hold yourself to.
Did you "took actions to diminish my contribution, and eventually left. I do not begrudge other Americans who did less than I did."
Or did you simply never work there, maybe never even had an offer to work there.
graemep 40 minutes ago | parent
Social security? Medicare?
The problem with avoiding taxes because of expenditures you think are bad is that you also reduce the pool available for things you think are good. Every country will do a mix of things you judge good, and things you judge bad.
> nobody is morally tainted for being forced to contribute a small slice to something evil
I agree.
rpdillon 33 minutes ago | parent
I like how this is being spun as virtuous. When multinational trillion-dollar companies do it, we say it's bad!
trumpdong 29 minutes ago | parent
autoexec 17 minutes ago | parent
petesergeant 55 minutes ago | parent
BatchJob 54 minutes ago | parent
The rule of all games played in capitalism is there's a sucker born every minute. you can work a few years or your whole life before you realize you've been had.
and most do....
bravoetch 54 minutes ago | parent
phoronixrly 52 minutes ago | parent
nativeit 42 minutes ago | parent
rpdillon 30 minutes ago | parent
trumpdong 27 minutes ago | parent
Aurornis 52 minutes ago | parent
It would be nice to see this survey repeated across professions. In my experience, most people have gripes about their profession because it’s what they know. Only half of respondents saying they would steer newcomers away feels a lot better than what my friends in medicine say all the time. Among my college professor and professor-track friends, this number probably reaches 100%.
bornfreddy 11 minutes ago | parent
nilslindemann 52 minutes ago | parent
cogman10 50 minutes ago | parent
Life isn't black and white with "good/bad". Further, there's no objective measure of what is good or bad. Everything is a shade of gray.
You certainly could argue that some actions are absolutely good or absolutely bad, but those aren't the majority of actions people take on a day to day basis. People aren't often faced with "should I kill this person" or "should I publish the cure for cancer". Instead it's "should I eat a hamburger or a salad" or "Should I finish this jira case or that jira case".
For example, you could argue that anyone working in an automotive plant is doing something bad because they are helping to produce vehicles which burn fuel which perpetuates climate change. But those plants also often produce parts for farm equipment or vehicles used to transport necessities like food.
People working at AI companies can make similar personal calculations. Like yeah there are bad things happening because of AI, but also they can believe that there are good things that can come as a result such as research and product developments.
graemep 46 minutes ago | parent
Balancing good against bad is not clear, I agree, but that makes it all the more important to think about it when making a big decision such as what job to take.
Its hard and people do not even agree about how to balance it - e.g. whether the ends justify the means.
ArcHound 38 minutes ago | parent
In the particular case of AI companies, I think a handbrake should be pulled when they start claiming things like "10% chance we're ending humanity."
chrisjj 36 minutes ago | parent
So, there's an ojective measure of shade of gray?
dofm 33 minutes ago | parent
But there are several workable contextual definitions of bad; it’s your job to know what they are.
For example:
- it would cause a harm to someone who was not made aware of the risk up front.
- it would cause someone to lose a job they didn’t know was under threat.
- the way the harm occurs precludes its only corrective/curative action or has no adequate compensation.
Bad things are almost always contextual but also almost always obvious in that context.
You can’t avoid bad things that you can’t foresee. If you do not attempt to avoid, avert or mitigate bad things that you can foresee, then you are one of the bad things.
Things may not always be black and white but it is wise not to seek the protection of the grey.
SaltyBackendGuy 27 minutes ago | parent
Under this guidance, wouldn't almost every technological advancement then qualify as bad? i.e. typewriter repair, etc.
dofm 13 minutes ago | parent
I mean everyone has to die someday but that doesn’t excuse murder.
wavemode 15 minutes ago | parent
arrowleaf 15 minutes ago | parent
Most of my life's work has been in areas of increasing productivity. Is that good or bad? On the good side of the spectrum, it reduces the time hundred of people need to spend on mundane tasks, or reducing the amount of mentally taxing complexity. On the bad side, based on this definition, there is less of a need to hire someone with specialized talent when the software can handle what their job function would be.
I've always felt kind of bad helping companies automate people out of jobs, that's the core of what most of us writing software are doing. Maybe I am a bad person.
dofm 7 minutes ago | parent
Is that necessarily so?
arrowleaf 4 minutes ago | parent
12887-19 31 minutes ago | parent
wonnage 29 minutes ago | parent
moolcool 16 minutes ago | parent
alexchantavy 12 minutes ago | parent
tetha 8 minutes ago | parent
I've been wondering what sectors I wouldn't maintain and develop infrastructure for, because I consider it a strength of mine that I don't entirely care about that. I'm not excited about your product, for good or bad.
Defense for example is on an edge there. Supporting the creation of death and suffering in other humans does not sit right with me. Being defenseless doesn't sit right either, as it can create suffering and death for people I know as well.
Online gambling is more on the nope side. Maybe with a decent regulated german company, but if you start working with some eastern-europan online video gambling sites with pretty girls as dealers, you are suddenly very close to human trafficking and many more things connected to it.
So I guess directly causing human suffering and pain is a boundary for me. And yes, if you push that argument to an extreme, if you search long enough, you realize that pure existence causes suffering to other people, animals and the planet and must be ceased.
But then there is that other side - in the current system (and arguably many before) you have to do a job to get money to live, and a lot of people like the person making cars have no choice about that. I guess we are contributing to that suffering as well by existing.
So, minimizing the direct suffering and pain I cause seems to be a sensible directive.
livvy 46 minutes ago | parent
He's still working in service of the AI companies though (same narrative) and no doubt will land a cushy job at so-callef "AI Safety" labs or make a living on the news/podcast circuit.
chilmers 44 minutes ago | parent
Here is one of his posts from 2022:
"Since a long time now I was interested in learning computer graphics and “do stuff” with computer graphics APIs. I began to work on a very simple Game engine in Swift / Metal last year but I began to switch to a Windows machine a few months ago, and Swift / Metal is not compatible with Windows (and will never be I think)."
https://carette.xyz/posts/opengl_and_cpp_on_m1_mac/
This is the same person who is apparently now writing:
"It is a transaction where you sell the part of you that was supposed to care. Do that often enough, and it does not come back. That is not burnout, but it is still a disease. Although, like every disease, it spreads: first to your team, then to your product, then to the people using it."
trumpdong 26 minutes ago | parent
tptacek 43 minutes ago | parent
dsign 19 minutes ago | parent
It's harder and harder to make the point that frontier objects are just hype objects though. And, even if they stay where they are today, that's more than enough to make them incredibly...destabilizing, to say the least. It's not only it, but frontier models act like knowledge mixers and percolators. Some people (not necessarily everybody), are adept at the discipline of learning, building things and executing projects, and that's all you need today with LLMs filling many of the gaps such person can have. Now, most of those people are actually engineers. So I'm going to go with the article and say that we must stop selling our souls for money.
moolcool 6 minutes ago | parent
fi-le 41 minutes ago | parent
armchairhacker 34 minutes ago | parent
skyberrys 33 minutes ago | parent
01100011 29 minutes ago | parent
That said, I don't buy this article. I'm AI neutral. It's going to break a lot of things and fix many others. Good for you if you think you know what the net outcome will be but I think it's delusional to believe you do. Also I don't think you can resign and stop it without ceding the world to bad people. Pandora's box has already been opened. We've discovered a new thing math can do and now it's yet another arms race between various factions seeking to exploit it. Help the faction you want to win. It's not the world I want it to be, but it is what it is.
magimas 27 minutes ago | parent
>So, why are you still working for them?
>If the answer is the salary, say it louder, and do not force yourself to make excuses.
>If the answer is “someone worse would replace me”, say it too, and louder too.
>If the answer is “I can change things from the inside”, then show me what you changed. Haugen tried the inside. Coxon tried the inside. At the end, unfortunately, the inside did not move.
>
Okay, but what if I think that both sides are LARPing their personal science fiction stories and reality will end up much more mundane (but still incredibly beneficial) than the utopias and dystopias everyone is cooking up?
This presupposing a moral dilemma that just doesn't exist in my opinion.
bzmrgonz 24 minutes ago | parent
atemerev 14 minutes ago | parent
stego-tech 5 minutes ago | parent
Obviously morality and ethics aren’t binary. OP isn’t saying that. What they’re saying in a clear and concise manner is that your actions have consequences far beyond your personal sphere, and good people acknowledge that and fight for what’s better despite the harm they personally may suffer as a result. That what makes someone “good” in their point of view isn’t a binary deed or outcome, but of the sacrifices they’ll make for themselves to build a better existence for others.
Planting a tree whose shade you’ll never enjoy, in other words.
So to reiterate OP’s challenge at the end, at least be honest about why you’re still in the game. Let others know where you stand so that we don’t waste our time on futility. If your voluntary position on something mutable risks making you a target, then the rational next step would be to evaluate how important that position is to you that you would weather said harm instead of changing it.
That’s all OP was getting at: promoting the mere notion of critically thinking about the decisions you make in regards to your own moral philosophy and acting more authentically, including tanking the damage trying to bring about change.