66 points rzk 3 hours ago 48 comments

cyanydeez 1 hour ago | parent

These people know they're doing anthropology right, sociology. LLMs are a raw dump of western civilization.

maxldn 59 minutes ago | parent

Not exactly raw because of the fine tuning they do to try and avoid bias (but in a biased way). And not exactly civilization so much as what is readily available online, which is in its self biased in many ways, e.g toward flame wars, via troll farms, etc.

I saw a paper on here a while back that said more or less the same thing, but didn’t frame it in a bro-friendly manner so it got flagged into oblivion

cyanydeez 50 minutes ago | parent

Your realize all those caveats exist in every type of anthropolgy class?

inglor_cz 3 minutes ago | parent

Now imagine how the LLMs would look like if they were a raw dump of the Islamic civilization instead.

I suspect we will soon know. This won't stay a Western/Chinese technology forever and there will eventually be some robotic fatwa issuer based in Tehran or Kabul.

jrrc_dev 1 hour ago | parent

Seems LLMs copy human behaviour pretty well

notenchange 1 hour ago | parent

Sincerely, did anyone that knows how LLMs are trained expected otherwise?

LLMs are reflexion of their training data, and their training data reflects mostly the current societal norms, with a bias (not sure how heavy) to the societal norms supported by the ones involved in training the LLMs.

What's interesting for me to see, is that Chinese models are as biased - if not more - than ChatGPT. This is probably because a big part of their training comes from distilation of US models.

It would be interesting to see Anthropic, Mistral and Grok in the comparison as well.

riedel 1 hour ago | parent

What one can theoretically do is to carefully remove gender signals from training data in a processing run. My guess is that you would not lose all to much knowledge, however you might lose the ability to recall facts about fictional and non-fictional persons. Wonder if anyone would go through that hassle though.

Planktonne 44 minutes ago | parent

Gender signals are baked deeply into our culture and language. It's not just the direct mentions, but the way people are described, the roles they take, the positioning of actors in a complex space.

"Remove gender signals carefully" is an incredibly difficult challenge.

turbo-outrun 34 minutes ago | parent

>"Remove gender signals carefully" is an incredibly difficult challenge.

Not to mention that, given that gender-signals have been around since the dawn of human history, you would remove them in a small test-region first; for safety reasons.

Assess the social consequences in a small region. Did it improve things?

Universal instantaneous-removal of any major socio-cognitive-structure is akin to vandalism because it might turn out to be, architecturally, vital.

Planktonne 20 minutes ago | parent

We already know what happens to some extent; think about gender-non-conforming people. That's a (relatively small) personal blurring of signals that theoretically doesn't affect much outside the person themselves.

Yet it's been a mainstay of our politics for the last few decades, and we've seen it used as a weapon for centuries. Our societies have gender embedded very, very deep.

graemep 8 minutes ago | parent

its political because it does affect much outside the person themselves. Most of the argument is about other people's use of language, about systems. Some of the most hotly debated topics here in the UK are around health (e.g. should everyone be asked whether they might be pregnant if women are, should I have got a letter asking whether I wanted a cervical smear) and government systems (what are the criteria for changing passport details? should a transwoman criminal go to a male or female prison?).

Gender is so deeply embedded that it is impossible for it to be just personal. It ceases to be political where there is a social consensus, which is true for most societies most of the time (not necessarily the same consensus), but not true for western society right now.

Planktonne 4 minutes ago | parent

That's what I'm saying; even a relatively small change (short hair on a woman, for example) can kick off a lot of responses, and ripples only spread further as people "deviate" further from the norms.

The idea that you can measure cultural bias purely by switching pronouns/nouns in some sentences is incredibly short-sighted; we know it doesn't work like that.

turbo-outrun 7 minutes ago | parent

> Yet it's been a mainstay of our politics for the last few decades...

The 1980s definitely felt a lot more fun to navigate, so I'm considering a roll-back on some more recent features...

riedel 17 minutes ago | parent

I would guess one could at least give adversarial signals like changing names in literature. Reading a lot of children books at the moment I feel it would be not so difficult disassociating social roles from gender. Wonder however also how much prompt rewriting could do: similarly to anonzmized CVs. Problem however is probably deeper: certain biographies will keep being discriminated.

Planktonne 11 minutes ago | parent

You could absolutely muddy the waters, but that's a first step and far more easily disregarded in context.

Modern children's books are much better about this, but consider all of the fairy tales; you need to swap not just names but descriptions and roles, and then you need to rework the way the characters respond to those roles before you have a hope of disguising the original genders.

catlifeonmars 23 minutes ago | parent

I said this elsewhere but what’s appropriate is going to vary quite a bit based on the cultural and personal values of the users of the model. Having a model with one particular set of standards is likely to leave a lot of people out. This isn’t something that you can just erase and assume that what you replace it with will be categorically better

throw9398922 12 minutes ago | parent

Let's keep gender roles, but we should respects people preferences.

Like I am total pussy, I identify as a woman! Do not expect me to sacrifice myself for some random people, just because I happen to have a penis!

rustyhancock 54 minutes ago | parent

I think the unstated part of your comment is important - increasingly acceptable especially in professional settings societal norms are female orientated.

I don't think this is particularly malign simply the net effect of complaint-adverse working cultures.

The net effect is that the accepted/promoted guidelines that get brought into the training data is a rejection of toxic masculinity and becomes "harsher" to men.

znpy 42 minutes ago | parent

> especially in professional settings societal norms are female orientated.

I’ve been noticing this for a few years now.

Particularly when it comes to managing issues, a lot more “work” is done on handling conflicts and feelings rather than addressing the actual issue at the root cause.

I know it sounds like a meme, but I’ve personally observed a bunch of colleagues and sometimes hr involved as well having meetings on meetings on how some system not working correctly and how that imposes a toll on work/life balance, mental health and stuff… rather than (trivially?) asking: “alright, why isn’t this working and how do we fix this?”.

rustyhancock 31 minutes ago | parent

I've seen this too, sometimes as a follow up meeting to address feelings about an issue that was resolved that few people go to.

Specifically the people who go are those most emotionally affected and unlikely to be men.

An effect of this is having meetings to address feelings scheduled before troubleshooting sessions! Opening with things like this is not a "fix it" meeting. Quite surreal!

boelboel 2 minutes ago | parent

Remember noticing in college and late high school that women were the better students (by a large margin when it came to group projects where form made up for +-50% of the grade). Whenever a sort of competition came up though men would rank at the top. Who's to say that the form portion of an assignment should account for that much of the grade.

RAWGOD 40 minutes ago | parent

>LLMs are reflexion of their training data

People don't understand machine learning. People seem to believe that with enough training data and parameters, a sudden switch happens that ignites a spark of pure intellect and turns a pile of matrices into a magic knowledge machine.

Unfortunately (or perhaps fortunately) our world is not magical, there are no Platonic Forms, and no Infinite Wisdom spells.

stymaar 30 minutes ago | parent

I don't think anyone expected otherwise, but it's good to have this kind of benchmarks since model makers can now correct these bias during post-training and measure how well they do.

Not unlike how the ARC-AGI benchmark family helped improve reasoning.

catlifeonmars 26 minutes ago | parent

> correct these bias

Are you assuming that there is some correct bias here? Wouldn’t that be highly dependent on the cultural and personal values of the users?

Lerc 23 minutes ago | parent

I did not expect otherwise based upon model pre-training.

Later training can be quite significant.

I'd like to see more benchmarks in this area, it is one thing to expect things to be like that, it's another to actually measure it.

Measuring it tells you something about the scale, and gives you a baseline for improvement.

gmbhgmbhgmbh 21 minutes ago | parent

I don't see how you can conclude that Chinese models as a whole are as biased or more biased than chatgpt from the results of the study. Some Chinese models were less biased than chatgpt, some models were more biased. The newer Chinese models in this study seemed to be amongst the less biased models in this study. The gemini models here seemed to be the more biased models in this study.

notenchange 13 minutes ago | parent

It;'s right there on the website with the code used for the paper: https://github.com/xufeiqiong/GAMA-Bench

In order of bias favorable to women:

1. Doubao-Seed-2.0-Pro

2. Qwen3

3. Gemini-2.5-Pro

4. Gemini-3-Pro

5. DeepSeek-V4-Pro

thenthenthen 17 minutes ago | parent

There is a meme that Deepseek tells woman to break up with their boy friends, and when asked the same to males, it tells them not to break up

Planktonne 10 minutes ago | parent

Who spreads the meme, and how did this paper remove their influence?

mft_ 10 minutes ago | parent

Trained on /r/relationship_advice then.

goodcake3963 1 hour ago | parent

One of the worst societal bias ever

zug_zug 50 minutes ago | parent

macic 48 minutes ago | parent

I'm not surprised, but I don't believe it's merely a result of the training data. I would expect the alignment phase to have a significant impact on it as well.

radiospiel 28 minutes ago | parent

Poor males.

Now, seriously: studies like this tend to ignore that every interaction between people is shaped by societal context. The same behaviour can mean something different depending on who does it. This is not double standard; what reads as patriarchal dominance from a man might reasonably read as assertiveness from a woman, because they're acting from different positions of power. Asking a model to grade isolated interactions at this small scale, stripped of that context, won't tell you anything useful.

Personally, I'm glad LLMs still account for the reality of male privilege.

(context: https://github.com/xufeiqiong/GAMA-Bench/blob/main/figures/o... from the linked repo)

BTW: I am not saying that the example response from the female in this example is acceptable, just saying that this "study" is basically pointless.

jadamson 17 minutes ago | parent

To be clear, you read your own link and think that a man locking an irresponsible woman out of a system is punitive, but a woman locking an irresponsible man out of a system isn't?

What is the actual reality, and why wouldn't you want LLMs to reflect it?

danielxt 11 minutes ago | parent

> basically pointless

I found it very interesting, not sure why you think it's pointless - I think it highlights something interesting in society

cat-snatcher 2 minutes ago | parent

They didn't like the implication of the study and want to keep believing that men are wrong and women are right even when they display the exact same behavior because of patriarchy.

stephbook 26 minutes ago | parent

Wasn't there a recent article on HN regarding race, gender, age and nationality bias in LLMs? I think I read it before it was flagged.

The results were similar.

zdc1 11 minutes ago | parent

I'd be curious about how the bias looks in the context of positive situations. Is the bias that LLMs hold men more directly accountable for any situation, and thus both punish negative and praise positive outcomes more strongly? Or is this bias only observable in negative situations?

To be clear: it's bad either way, since women should be seen as having the same levels of agency and accountability as men, but I am curious nonetheless...