32 points acossta 41 minutes ago 7 comments

rfgplk 37 minutes ago | parent

What's the perplexity of the average person?

jrflo 11 minutes ago | parent

Probably high because humans aren't trained on lowering cross entropy loss. Kind of funny to think that next token prediction is actually an emergent capability of the human brain...

Leary 12 minutes ago | parent

Looks like their latest internal model performs somewhat similarly to model 4 of anthropic.

amluto 3 minutes ago | parent

> The primary held-out datasets we’ve been using since GPT-3.5 are sourced from our internal codebase, and those are the numbers reported here (for a fixed snapshot).

Why would OpenAI expect any meaningful relationship between their models’ ability to predict the next token of supposedly held-out chunks of their internal codebase and the overall capabilities of the models? I expect both to improve, but that’s about the extent of it.

eulgro 3 minutes ago | parent

Didn't they hit a data wall roughly two years ago and had to rely on artifical datasets for pretraining i.e. LLM-generated?

I would expect the entropy of artificial datasets to be lower, thus easier to predict and scoring better on "perplexity"?

Retro_Dev 2 minutes ago | parent

Saying that perplexity is a metric of intelligence is misleading. Inkling-2-Mini was released recently, and I've measured its perplexity at 5.7 on a small varied text dataset. For reference, the graph OpenAI is glazing is plotting log(PPX), which would put this model at a 0.756 on the Y Axis... Which is apparently between GPT 6.2 Astra and their internal model. Perplexity can also give the same number for a model which understands text as to one that memorizes text... And as I'm pretty sure Inkling-2-Mini is smaller than GPT 6.2 Astra - and it's generally acknowledged that number of parameters equals memorized knowledge, this shows how weak this advertising stunt for OpenAI is.