I have been claiming that I don't think Chinese AI companies are benchmaxxing harder than American AI companies, which has gotten mixed reception: sometimes people agree, sometimes they disagree.
It seems I was wrong. American AI companies might actually be benchmaxxing harder.
gertlabs 5 hours ago [-]
We run an evaluation that is designed to be less vulnerable to benchmaxxing because there aren't correct solutions; agents are interacting in the same environment as other agents. And it's private, and our public benchmark is not well known enough for anyone to probably care to benchmax us yet. So I think it's pretty indicative of true relative aptitude.
All models have probably memorized significant swaths of solution sets for popular benchmarks at this point, either accidentally or intentionally, so it's all relative at this point. However, in our experience, Chinese models do benchmax harder. This is also consistent with interacting with Chinese labs soliciting data/environments, who literally asked us for datasets and tasks modeled around and formatted like popular benchmarks.
Opus 5 will be uploaded tomorrow, but we already have the tests locally and it is truly as capable as Fable, but at 81% of the real cost. (And from subjective usage, it has a very different personality)
Benchmaxxing via memorization is boring and doesn't fool anyone for too long. It works, but then new benchmarks test old models and the real results fall in line. Benchmaxxing by focusing on specific types of things that benchmarks test on, while still not improving intelligence or capability in the general case? Not only is it blatantly obvious that all AI labs do this, but it's not even obvious how you would go about it any other way.
Now I am not really specifically accusing Anthropic of anything here, I'm just saying their behavior is suspicious. Since you tested Fable, they wouldn't even have to lie to have optimized for your specific benchmarks, since they absolutely had permission to read your sessions if they wanted to. But obviously, that's only the situation if we take them at their word. Personally I would be a bit surprised if they just flat out were lying and secretly retaining data they say they are not, but not that surprised. The penalties for doing this are probably worth the rewards if it keeps them super far ahead in the benchmarks for years without anyone catching on.
(In actuality though, even if they really were trying to sneakily grab samples of benchmark tests via their Fable data retention rules, I don't really suspect there would've been very much time to optimize Opus 5 on it. So consider me bothered.)
throwa356262 10 hours ago [-]
"Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration."
I guess there is no way this can happen without benchmark being part of the training data??
pierrefermat1 10 hours ago [-]
What seems to implied is that some of his hold out testing suite includes simple/common tests that are out in the wild, and for those opus went straight to a memorised solution .
zamadatix 9 hours ago [-]
Simple/common tests is not an explanation for why only now Opus 5 is the only model encoding the answers like this. Something like the holdout test suite being leaked or Anthropic cheating (e.g. 'accidentally' including previous hold out run data in Opus 5 training) makes a much stronger fit.
modeless 5 hours ago [-]
They state the puzzle is "Witness-like" which I assume means that it follows the rules from the well-known puzzle game "The Witness" which Opus definitely knows.
stogot 10 hours ago [-]
It may read information about the benchmark, such as on blog post, or Twitter feeds (example OP) without active cheat
I was just thinking they need to mark each model per benchmark as "model released before the benchmark was released" and "model released after the benchmark was released".
root-parent 7 hours ago [-]
And being worst than previous model...
"...The traces tell the why:
(1) On our most classic Witness-style game, Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration. It already knows this genre.
(2) But on our most novel game (unusual mechanic combinations you can't pattern-match), Opus 5 regresses below Opus 4.8. Where rules must actually be discovered through interaction, the new model is worse than the old one..."
Zababa 7 hours ago [-]
>That decomposition (perfect on templates, regressed on novelty) is the signature of “scaffold-then-internalize” training on genre-specific data, not a general gain in interactive abstract reasoning.
They're smuggling a claim that benchmarks like ARC-AGI measure "interactive abstract reasoning" here, which is what is claimed by the people that make these benchmarks, and also not proven.
andrepd 10 hours ago [-]
I'm shocked, astonished even, that enterprises on which trillions of dollars are being poured would consider cheating on marketing benchmarks.
asdfologist 9 hours ago [-]
Meh, I doubt it was intentional. Deliberate benchmaxxing is incredibly damaging to credibility once it's discovered (see what happened to Meta with LlaMa 4).
It's more likely that the training data was contaminated with the benchmark data.
mupuff1234 9 hours ago [-]
You really think they saw the jump in arc-agi-3 (which they reported in their official card), and didn't even bother to check?
They maybe have not intentionally benchmaxxed, but they certainly know that's what happened .
goodmythical 3 hours ago [-]
Is it really all that different from how the models "know" anything else?
The best way to know that christmas trees are typically purchased in december isn't to access historical financial tables and run the entire analysis yourself but to just "know it" from the articles you read attesting to the fact.
If you've found data that is the answer to the question, be it a random user question or a benchmark, the softwares' goal is to produce the correct solution and it is cheaper to retrieve from storage than to compute.
It's the same with coding. The agent usually isn't really thinking about the problem from first principles, it's just giving you the answers it's already found in instances where someone else asked the same question.
It seems to me that if you're really testing for reasoning capability, just as if you were an instructor administering a test, you'll need to change the test from run to run in order to make sure the agent/student isn't just copying old tests.
johnfn 7 hours ago [-]
How do you propose they check for something like this? They can't exactly ctrl-f the model weights for "Arc-AGI".
vickychijwani 4 hours ago [-]
Anthropic expends tons of compute and effort on understanding internal model states [1]; this kind of thing is right up their alley.
> they can’t possibly know or find out what was in the training data
doesn’t appear to be a very strong argument
turing_curious 5 hours ago [-]
[dead]
codewiththiha 7 hours ago [-]
[dead]
dinp 14 hours ago [-]
The last time I checked, for the arc agi 3 leaderboard, the models are given a simple prompt and the game input and asked to play the game, no harness/tools. If harnesses were allowed, I would expect the benchmark to be saturated. There were a few harness attempts, but they could only be evaluated on the public set, so it's not an apples to apples comparison.
My guess is, the large score jump for Opus 5 is mainly because of getting the right RL envs for training.
It's becoming harder and more expensive to build and run meaningful benchmarks, it would be interesting to see what they do with arc agi 4, maybe just give it gameboy/steam games and see how they compare vs a human baseline? The latency requirements and very long horizons in games could be an interesting challenge for llms.
r0ze-at-hn 14 hours ago [-]
The exclusion of harness's feels really weird given that companies are recognizing the value of what harness's can do. By excluding them the benchmark is becoming less relevant.
usernametaken29 12 hours ago [-]
It’s because of inductive bias. Harnesses will massively skew results towards working solutions. You might think that’s a good thing but what it might mean that sometimes it becomes enough to run brute force search or a simple parameter search over the harness. Creating the harness is the actual work, because you’re selecting what are the levers to pull. There were some attempts of LLMs generating harnesses on the fly in ARC 2, but they were all mostly based on one handcrafted DSL that was copied over and over again. As it stands harnesses are not allowed because they’re simply not a meaningful measure. What you’d like is to measure how the model performs if it saw this benchmark for the very first time… but then again everyone knows the game is rigged, millions are at stake, and the AI companies fine tune and cheat on the benchmarks any way they can.
crazylogger 6 hours ago [-]
The goal is to test for AGI where G stands for general, that means ability to act in any environments, ideally solving novel tasks using novel tools we’ve never seen before in the world. If a specific prompt or tool design lifts a model’s score it’s a sign the model is overfitting to a particular modus operandi, therefore not general.
I think in this age where models are heavily RL-ed on acting in specific harnesses, this type of benchmark is more important than ever, to make sure they’re not in fact moving further away from general intelligence.
yladiz 13 hours ago [-]
You could argue that if you allowed a harness, and that harness was specific for ARC, then you don’t have AGI, you have something that is definitely not general.
Retr0id 13 hours ago [-]
What if the harness is developed by the same model in a prior session?
11 hours ago [-]
kypro 12 hours ago [-]
I think that's a strange way to look at it. The brain also has different regions with different functions, and part of what makes us humans intelligent is that we can use tools like pen and paper to keep notes and help us solve problems.
Similarly LLMs are not just massive uniform artificial neural nets, and now we also have harnesses, which I'd personally view more of an extension of the model itself. The harness is both the executive and also what allows it to keep notes, use a calculator, or maybe even create scripts to help it solve complex deterministic problems.
I think it's unfair to give a human a very complex maths problem and say that they're not intelligent if they can't solve it without pen and paper or a calculate. At least expecting a human to solve complex problems this way doesn't really measure anything useful in the real world.
ryoshu 11 hours ago [-]
iirc, a harness isn't allowed, but if the LLM wants to write its own tools to solve things that is allowed.
mzubairtahir 11 hours ago [-]
Harness can totally change capability of the model
slopinthebag 8 hours ago [-]
The value of a harness is more about developer workflows, I don’t think it really improves the model output.
14 hours ago [-]
throwaw12 16 hours ago [-]
Why Anthropic models are always leapfrogging these benchmarks, but in real life work I do feel like after 3 weeks I am back to Claude Opus 4.5? (regardless of the model I use, Fable was exception for 1 day when it was released)
OtherShrezzing 15 hours ago [-]
It could be that the set of your day-to-day workload which could feasibly be accelerated by AI just happens to be saturated around Opus4.5, but you can still see lots of “reasoning” which makes you think the model is more performant in the first days of use. That’d mean you couldn’t perceive any meaningful difference in more powerful models’ results, even though you can see a difference in the raw output due to the length of reasoning traces leading up to the result.
So for example, if your workload was literally just addition of sets of numbers, you’d never have noticed progress in the result beyond GPT3.x level models. But you would perceive a difference in the now-Tolstoyan length reasoning text accompanying the result.
SubiculumCode 14 hours ago [-]
5 seems incredibly smart to me in my conversations today about some pretty niche ideas in.longitudinal modeling. It.felt.like a big step up.from 4.8, to me
kranke155 10 hours ago [-]
5 felt both smarter than me and dumber in some ways - it gets stuck to its original ideas. I had never seen a model harder to talk into changing its initial opinions. it continuously hedges.
stared 15 hours ago [-]
It's called frog boiling.
We get used to the new level of intelligence so fast, any deviation feels like going back to the stone age.
If you don't believe me, create something complex with Opus 5 and then with Opus 4.5, and notice the difference.
vmg12 11 hours ago [-]
The actual term for this is hedonic adaptation.
ffsm8 8 hours ago [-]
esp. important to point that correct term because frog boiling is a urban myth.
frogs dont stay in a pot even if you slowly increase the heat. they leave. it has reportedly been attempted multiple times and they. always. leave.
yorwba 16 hours ago [-]
Well, what kinds of things do you see Opus 4.5 completely fail at? Maybe those are not the ones that newer models have improved on.
tudelo 15 hours ago [-]
I honestly just use GPT models nowadays, Claude models are too restrictive and more of a quitter and fable/whatever is just too expensive to be worth it.
submeta 9 hours ago [-]
Some 20 years ago, the telecommunications sector in Germany was liberalized. Many telephone card providers entered what had previously been a barely competitive market. They advertised their products with aggressive claims like: “Buy our €10 top-up card and get 660 minutes to destination X.”
For the first few weeks, they would actually provide those 660 minutes to establish trust in their cards. But after a while, they would quietly start reducing the number of minutes on subsequent top-ups—say, from 660 minutes down to only 300. They wouldn’t do this for every card, so it was difficult to prove. Instead, they relied on averages across their customer base to make the economics work.
Lately, I’ve found myself wondering whether something similar may be happening with frontier AI models. Companies launch with an exceptionally strong model and generous compute limits to build adoption. Once the model is established as a market leader, the incentives change, and users may start perceiving the service as becoming more constrained or less capable over time.
I don’t have evidence that this is what’s happening with Anthropic—or with any other AI company. It’s simply a pattern that the current situation reminds me of.
slopinthebag 8 hours ago [-]
Same, like I prefer 5.3 codex over the “stronger” models.
lwansbrough 16 hours ago [-]
Going to call it user error if you find Opus 4.5 better than 5, sorry.
rf15 15 hours ago [-]
[flagged]
IanCal 15 hours ago [-]
> I've worked with these systems for four years now and they have not meaningfully improved in that time frame.
Not meaningfully improved?! Four years ago was gpt *3.5*! ChatGPT hadn’t been released!
rf15 15 hours ago [-]
Yes! Impressive, isn't it? I see how it has improved for some minor points, that the big models can cover more finetuning ground, but my big gripes are still the same - you could do the same back then with multiple models and more targeted finetuning.
Marha01 14 hours ago [-]
> you could do the same back then with multiple models and more targeted finetuning
Definitely not, lol.
UltraSane 16 minutes ago [-]
The only thing impressive is how wrong you are. LLMs have improved by an absolutely incredible amount in the last 4 years.
Narciss 13 hours ago [-]
I have no idea how you could try and hold this argument without being facetious.
rf15 12 hours ago [-]
Am I missing something that my original points no longer hold for their products? Did it get meaningfully solved? Are your experiences flawless on that front?
Kiro 11 hours ago [-]
What do you hope to achieve here? Posting absurd and ridiculous things and then continuing like if someone would take you seriously after that. Keep going I guess.
thops-barrier 12 hours ago [-]
You're moving the goalposts. Initially it was "no meaningful improvement" and now suddenly it has morphed into "they're not flawless".
I'm pretty sure you're just baiting for engagement though so well done, ya got me.
yreg 14 hours ago [-]
This is obviously not true to all of us here…
virgildotcodes 14 hours ago [-]
> you could do the same back then with multiple models and more targeted finetuning
I mean, come on, this is just not true. You could not achieve anything like what you can with modern agentic coding with Fable / 5.6 Sol from any combination or configuration of GPT 3.5 era models.
It's like saying that a teenager isn't an intellectually meaningful improvement over a toddler.
Sure they're both still fundamentally flawed humans prone to cognitive error, but one is clearly more likely to hit the mark than the other when assigned a task.
IanCal 14 hours ago [-]
Utter nonsense.
There’s no way you could get models as smart by fine tuning. I couldn’t throw a problem like “build a pokemon database with UI to teach my son sql” and get a working system, nice ui, tests (which it iterated on) examples and explanations in one shot.
There weren’t thinking tokens. Maths is now dramatically better, making actual contributions when before they were mostly mocked for making extremely basic errors. Smearing is also something say is very rare in frontier models.
If you think they have barely changed you’ve either forgotten what they were like or not used them more recently, or you’re just being obtuse.
rf15 13 hours ago [-]
My company had such a system four years ago, for internal work, somewhat more limited in scope (one language). What you are seeing as the frontier is not necessarily the best you can have - just because people don't try to push it to market as a product doesn't mean it's not there.
Edit: we do have a system that uses LLM and fixes the above issues largely (tracking of state, calculations and objects, still flawed in finer details). No, we don't sell, it's experimental fun and not really ready in terms of setup/ux/etc.
It codes really well for our case though.
anonzzzies 15 hours ago [-]
> you could do the same back then with multiple models and more targeted finetuning
Are you one of those anonymous billionaires as if you did this a few years ago, you would've been famous and rich.
danielbln 14 hours ago [-]
OP is delusional or deliberately optuse. I work in the space and stare down these systems 12h/day, and saying the systems haven't meaningfully improved is ludicrous.
rf15 13 hours ago [-]
OP is largely pissed with what OAI/Antrophic are trying to sell as meaningful improvements and the market-bending money they ask for it. I work in the space and we trained LLM models on conceptual tokens, not language tokens, for example. See Symbolic AI and all the attempts at hybrid models.
Also, uh, fame and riches are not really my thing. Middle income is fine. My mistake was speaking up here because I got carelessly annoyed because I have skin in the game, research-wise. I'm sorry for that.
mdp2021 8 hours ago [-]
> are trying to sell as meaningful improvements
There may not be "core" improvements (structural reliability) but there are "emergent" improvements (apparent intelligence). Already the IQ tests from Maxim Lott ( trackingai.org ) show a progressive sliding towards the right side of the curve - which btw translates to a very much non-secondary decline in the user's frustration (and progresses with an increase of usability).
The more they work on it, the more probable the jump becomes - e.g. to achieve the Large Conceptual Models you say you worked on.
dash2 8 hours ago [-]
You're welcome to speak up, but saying models haven't meaningfully improved in three years, when the most recent models are solving top-level frontier maths problems, is indeed going to strike most people as weird. I still have no idea what you mean.
mdp2021 8 hours ago [-]
I think he means that, while results are there, they are mysteriously emergent, since an analysis of the process reveals it can be faulty.
fakwandi_priv 15 hours ago [-]
> - Math completely fails in longer contexts
Not sure what longer contexts we're talking about but didn't we have an old math problem optimized, which even the LLM itself was surprised about, just a week ago? Something which wasn't possible 6 months ago.
rf15 15 hours ago [-]
I mean calculations, not mathematical proofs
Barbing 15 hours ago [-]
If they use Python to fill the gap, and the end user doesn’t have to know or care, is it unfair to assess this as progress and attribute the progress to the _system_?
OK, the core technology that is the language model still can’t math as well as you’d hope, but how about the end result users see from the system when they interface with it?
“Did you know humans are better at flying today than they were a thousand years ago?” ‘No they’re not, they need planes.’ Technically correct in a way but isn’t it kind of annoying to be so stubbornly pedantic when the context is speed of reaching Point B from Point A?
rf15 13 hours ago [-]
You are correct, the frameworks around it have improved. In that regard, my assessment is unfair: I only judge the underlying technology and what is sold by the sota providers, with the premise of what it's like when you start fresh. You can achieve a lot by coding around the issues, but that's kinda against the point of 'AI', is it?
wahnfrieden 11 hours ago [-]
No, its capabilities with a harness are what we are interested in. Your assessment is only relevant to benchmarking, not practical value.
Barbing 8 hours ago [-]
orwin‘s response in a cousin comment helped me see your original valid point on harnessless LLMs!
>You can achieve a lot by coding around the issues, but that's kinda against the point of 'AI', is it?
Will think on that a bit more.
glimshe 12 hours ago [-]
Messages like this in the training data are how LLMs learn to say absurd things with total confidence.
azan_ 12 hours ago [-]
> I've worked with these systems for four years now and they have not meaningfully improved in that time frame.
That's absolutely insane. Is it some case of anti-AI psychosis?
Barbing 15 hours ago [-]
Like how toddlers’ skills don’t meaningfully improve on infants’, because either could wake up in a wet bed.
mdp2021 15 hours ago [-]
Let us be more clear: there is no structural jump, no architectural overcoming of the original fault.
(Edit: and on a similar point, structural properties such as having static ntetworks, as opposed to continuously learning and improving architectures (such as us), will reveal that there is still road ahead.)
Barbing 14 hours ago [-]
Maybe more fair then would be: “I've worked with these systems for four years now and while they _have_ meaningfully improved in that time frame, they’re not perfect and remain fundamentally flawed in various ways.”
You prompt less. You need not inject search results into the context window yourself, a window much larger than years ago. You get code that’s already been run successfully once instead of finding an obvious show stopping bug yourself.
The technology is not a brand new one that fixed everything wrong with the old one, no, but not sure I would’ve noticed your comment if it had been such a bland observation. I genuinely assume good faith here… will say am tempted to assume the standards of someone posting such a thing might be impossibly high. Glad to be having a fun conversation instead of getting your grades on my work product or something :)
orwin 12 hours ago [-]
If you go to a LLM without harness, GP original point in completely right.
LLMS by themselves are still shit at math, they still confuse weird correlation to causation every time (and sometimes in ways even a 9 year old would say "no, that's dumb"), and confuse original parameters very often.
I disagree with " "thinking" token generation being on the correct track just to 'no, wait' on an already correct conclusion", because i think that is an effect of the harness, not the LLMs.
80% off all the improvements since ChatGPT4 are in the harnesses, and the LLMs by themselves, while they improved in areas they already were good at (translation especially) did not fix any of they original issues (object permanence, calculusm correlation).
Just run old models in the playground and get them to play chess (maybe make a small custom harness if you feel like it), then replace it with a frontier model (i don't know if you still have API access without harness on US models, but if you don't try K3), you will see LLMs weaknesses were not at all fixed, even marginally. They're way better and not inducing bugs in the code, so that make them usable since Opus4.5 (anyone using them prior to that either had a greenfield project or like spending hours debugging).
user43928 7 hours ago [-]
> 80% off all the improvements since ChatGPT4 are in the harnesses
That seems easily falsifiable by putting an old model into the current harness and comparing it to 5.6 Sol or Fable.
rf15 11 hours ago [-]
I agree, harnesses is where everyone improved the most. Our internal experimental tool can now semi-reliably formulate small programs to assert the correctness of their theories, for example. Context length is still a weird factor that we haven't sensibly solved - if anything, the lesson learned was to keep the context as small as possible and do most of the true "thinking" in the harness and temporary generated code.
Barbing 8 hours ago [-]
Sounds very fair, thanks :)
sscaryterry 16 hours ago [-]
Enshittification.
94b45eb4 1 hours ago [-]
When we started talking about AGI a few years ago there seemed to be a relatively common consensus that LLM models could not be considered AGI because of how they work. Even if there was some changes to train the model on the fly, I still just don’t feel like this is AGI. It’s just a more convincing version of the existing “party trick” we have been doing all along. It convinces us it’s “AGI” the same way current models would convince someone 10 years ago that it was intelligent.
Nobody has any right to take anything I say seriously, because I’m just some random on the internet. But I don’t think true AGI is any closer than about 10-20 years away. That would be to create an actual analog for a human brain.
nozzlegear 37 minutes ago [-]
AGI and ASI are just a convenient myths that the American AI corps use to push for regulatory capture. The difference between the rhetoric from China surrounding artificial general intelligence, and the rhetoric from America, is pretty stark. The Chinese are a lot more grounded and realistic about the whole thing (they almost never talk about ASI, and only talk about AGI in practical terms), compared to the breathy "humanity is doomed but we're building this shit anyway" stuff coming out of Anthropic.
codedokode 10 hours ago [-]
Why is there no Kimi 3, and GLM5.2 didn't run the third benchmark? I am more interested in knowing the abilities of open weight models.
Reubend 2 hours ago [-]
ARC-AGI3 doesn't seem like a great benchmark to me in the first place. It assumes a lot of human like tendencies which an AI either shouldn't or wouldn't have. Particularly in the genre of "gameplay" where unspoken assumptions from prior games inform our understanding of rules.
tudelo 15 hours ago [-]
> Only systems which required less than $10,000 to run are shown. (Notes[1])
Am I lost or are their many models on this ranking (Opus 5 included) that clear this?
chmod775 15 hours ago [-]
Many models are much cheaper through their subscriptions' included usage. That could be what's happening here.
Claude gives you something like $5000 of tokens on a $200 plan.
versteegen 11 hours ago [-]
Isn't it ~$3000 per week? Extrapolating from the current limit on Pro plans.
chmod775 11 hours ago [-]
They don't exactly say. I was extrapolating from some sessions usage (my figure was per month, so weeks times ~4).
albatross79 12 hours ago [-]
ARC-AGI is a beauty contest for pigs where the pig's owners compete to see who can apply the lipstick most convincingly.
rad-b 12 hours ago [-]
Great comparison! We only have to take into account that applying lipstick well bears no consequences, but applying it poorly (i. e. new model tanking the benchmark) could amount to potentially losses of billions of dollars for the pig-breeders (AI labs).
I have a suspicion that they are just trained on puzzles by now
blovescoffee 16 hours ago [-]
There are private datasets, and 3rd party providers of these models. Fable doesn’t have a datapoint here because of its particular data retention policy. Even if you don’t trust AWS, do you think Opus on AWS is also sending the data to Anthropic? Do you have any evidence?
iLoveOncall 15 hours ago [-]
The fact that it says so in the licensing conditions on AWS?
MikeTheGreat 16 hours ago [-]
It's like we've come full circle:
First people practiced L33t3cod3 problems for interviews
Then people built AIs to build software
And now the AIs are studying L33t3cod3 problems
amarcheschi 10 hours ago [-]
Why the 3 rather than e?
mdp2021 8 hours ago [-]
("Why 'leet' or '1337' instead of 'elite'". Because restricted groups like to stress a difference.)
amarcheschi 1 hours ago [-]
Thank you
nfbdhdfbf 5 hours ago [-]
That’s the original form of the slang/jargon term that the site’s name derived from.
It's actually crazy to see the difference between opus 5 and the next best model on ARC AGI 3 when you actually look at the ARC AGI problems
zzleeper 16 hours ago [-]
How believable is this benchmark? EG maybe opus was training on this? (You can try to identify the IP of wherever previous ARC questions came from)
10xDev 16 hours ago [-]
That’s why you have a private dataset.
raincole 16 hours ago [-]
Which you have sent to Anthropic/OpenAI/Google's servers when you run the benchmarks for the previous models.
Jensson 16 hours ago [-]
Doesn't matter, people built harnesses that solves arc agi 3, so all you need is to train your model to work like that harness by default. That makes a model specialized at solving arc agi 3 without making it smarter in general.
It is very hard to make a benchmark you can't do that for, but it is very easy to make your own personal test that others can't do that for since now it isn't a benchmark they can target.
NitpickLawyer 16 hours ago [-]
> people built harnesses that solves arc agi 3,
They didn't. Kaggle is still running for a few more months, best result atm is ~2% with 9h runtime on one rtx6kPRO. Also note that these new results are on the semi-private set, not the public 25 games ones. Any announcement where you see "solved ARC3" is likely only dealing with the 25 public games. And that's highly questionable, until you get to see the code. (which, to my knowledge the team that claimed 99% hasn't yet published).
Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.
haldujai 14 hours ago [-]
No but the session traces are available. It passes the sniff test considering how AGI-3 is scored and how this wrapper works.
For example on bp35 it took fable 290M and >12k simulated turns for 566 real turns and finish more efficiently than a human.
Regardless of the true score I think the takeaway is the benchmark measures the wrapper rather than the model.
> # FRAMEWORK ARTEFACT: the run's very first transition is replayed WITHOUT advancing state
# (tools.py:954 and agent.py:468 both `continue` before `state = next_state`). So on the
# level that contains that step (level 0) our counters start exactly one action behind.
# That skipped step was action 1 with BOTH avatars moving, so seeding n=1, bumps=0 reproduces
# the framework's lagged state exactly.
# CAVEAT: this seed is only right while level 0 has never been RESET. If you ever RESET
# level 0, change the seed to n=0 (after a reset the rollout re-inits and no longer skips).
That tells me that there is some leakage between runs. The idea of ARC3 is that agents start working blind, on new tasks, via API. A RESET is counted as one action. Without seeing the actual code that produced these traces we have no way of knowing how many iterations it took, if the "framework" played the same level multiple times (comment hint above makes it likely) and so on. That's why I said that before we actually see the code / can replicate / ARC team confirms it on new envs, this should be taken with a grain of salt.
haldujai 8 hours ago [-]
The comment more likely means the harness source was read, not memory from a previous run and the first few turns of bp35 appear to be a cold start.
Sure none of this is certain without the source.
I do believe the authors that this schema significantly improves over the base, particularly given that it took 22x simulated turns over 14 hours, which is moving the trial and error to context rather than to game. I also don’t doubt there is some contamination.
Regardless, the approach is sound and I do believe it would significantly improve scores, even if that was +20-30 over baseline (49% in this case) it does imply the benchmark is measuring the harness more than the model.
Stevvo 15 hours ago [-]
If you stop and think about the problem it really is quite simple. Just need to build a graph of the game state and then run A* to get to the end.
NitpickLawyer 15 hours ago [-]
You really should play the 25 games before stating that it's "simple". The benchmark doesn't just track "completion", it also tracks the number of steps, and the score is based on the median steps took by human players. So in order to get 99% it would mean that the model solved every level of every game in less steps than the median humans. Which, having played the games and having setup harnesses for local models, I find hard to believe.
Also the models have to figure out what "end" means. And each game involves some kind of "gotchas" thrown in the harder levels. Some games are only solved by about 2/10 people trying them.
The 99% result most likely has some leakage somewhere, either in the preparation of the environments, or from session to session.
Seriously, play some of the games. They're fun.
16 hours ago [-]
Stevvo 15 hours ago [-]
Why? 30% is passing the first two problems only, which are really very simple.
lkbm 10 hours ago [-]
Huh. How do things end up with scores like 30.2% (and results between 0% and 1%) if it's that low resolution?
bob1029 15 hours ago [-]
I think it's way too easy to be deceptive with these benchmarks now. You don't even have to "train" the model on a new variant each time. The base models are powerful enough. All you need is a naughty little markdown document that provides explicit instructions regarding how to solve the new puzzle variant, and a willingness to be deceptive about the presence of that document.
If you want a know why the model providers are locking down and encrypting their reasoning process, this sort of workaround is potentially why. You can play this game of whack-a-mole indefinitely if the state of the system is concealed. They could have added something like:
> ### When solving arc-agi-3 puzzles: First convert the grid into a scene description. Identify connected components, colors, shapes, positions, symmetries, repeated structures, and relationships between objects. Do not reason directly from individual pixels... use this python script to help blah blah...
xiphias2 15 hours ago [-]
,,You can play this game of whack-a-mole indefinitely if the state of the system is concealed''
Not really as one of the main goals ofr ARC-AGI 3 was measuring task efficiency on unseen games.
I'm sure there are cheats everywhere but the most sensible thing is to just accept that the LLMs of today are much more intelligent in solving reasoning tasks than the ones from half year ago.
My own private benchmark shows the same thing.
brap 10 hours ago [-]
Are these typically the type of tasks that are genuinely worth tens of thousands of dollars?
NooneAtAll3 9 hours ago [-]
games are great (as for a human)
but I kinda wish I could select level... I accidentally pressed redirect button and when I came back I was once again shown level 1, all progress lost :(
nickvec 8 hours ago [-]
Why isn’t Fable 5 included on the leaderboard?
kypro 12 hours ago [-]
ARC-AGI-3 launched a few months ago which would suggest that prior models likely had no knowledge of ARC-AGI-3 or training on similar challenges.
I could be wrong, but given the large outsized jump solely in the ARC-AGI-3 score, it would suggest that the model didn't become significantly more intelligent overall, but significantly better at solving those specific problems.
This could mean one of two things (I think):
- Opus 5 was not benchmaxxed on ARC-AGI-3, but has benefited significantly from discussions about the various challenges and mechanisms deployed in ARC-AGI-3 such that it has far better heuristics to solve its challenges.
- Anthropic looking for buzz around their latest model picked a well regarded benchmark with significant room for improvement and focused some of Opus 5's training compute on ARC-AGI-3-style problems.
Or it could be some combination of both. Personally, given how much of an outlier the ARC-AGI-3 jump is I struggle to see it being the product of a significant improvement in general intelligence.
mdp2021 8 hours ago [-]
> it would suggest that the model didn't become significantly more intelligent overall, but significantly better at solving those specific problems
We would actually need a test that shows the ability of a model to export its skills to more problems ("interdisciplinarity" etc.).
fileyfood500 7 hours ago [-]
I also noticed that Opus 5 doesn't show a corresponding gap on the ARC-AGI-2 leaderboard. There is a significant increase in performance between Opus 5 and 4.8 on ARC-AGI-2 though.
MaskNinja 9 hours ago [-]
Not possible. I don't get how Opus 5 gets so high. Have they run it against the private and held-out games?
luciana1u 14 hours ago [-]
solving ARC-AGI and being useful turned out to be two different problems
nullbio 14 hours ago [-]
Because they're cheating.
spongebobstoes 10 hours ago [-]
this is not a good measure of current model capability. we need to test agents in harnesses, not models with a single prompt
test Codex, not Sol. test Claude code, not Opus
ChrisLTD 10 hours ago [-]
There are other benchmarks for that
dyauspitr 16 hours ago [-]
Why is Fable not on here? I wish Fable hadn’t come out because it’s taking the wind out of every release because that feels like the cap above which the US government will not let LLMs improve anymore and everything they’re releasing from this point has to be worse than that.
NitpickLawyer 16 hours ago [-]
> Why is Fable not on here?
Because the data retention policies didn't guarantee that the ARC team could run the semi-private set of problems without fear of them being trained on later on. They only run the semi-private set when they get assurances like ZDR.
3form 16 hours ago [-]
How do they handle these assurances? Personally I have zero trust in the AI companies not trying to use this data to get ahead in the game, and short of sharing the weights and harness so that the benchmarkers can run the models themselves, I don't see a satisfactory solution with this mindset.
villish 15 hours ago [-]
OpenAI's Zero Data Retention claim held up in court. They were unable to produce prompts and outputs because they were never retained.
I believe that is only available through Enterprise API for both Anthropic and OpenAI.
Barbing 15 hours ago [-]
Is there a distinction we can independently assess between never retained and deleted or hidden?
Interesting to place that level of trust in the providers, but I guess that’s the best you can do with closed models. Makes me wonder if Opus 5 could have been trained on data they promised they weren’t training on? One of the interesting things about LLMs is how opaque they are from the outside, even with open weights, it’s very difficult to know if a model incorporated benchmark data in their training.
claw-el 16 hours ago [-]
I think you could have accessed Opus on AWS then u don’t have to trust that the data will go to Anthropic?
Just like the hugging face incident, Opus 5 could have escaped and went to grab data for training it shouldn’t have been able to..
block_dagger 16 hours ago [-]
I don't know why exactly, but Fable has felt the most human LLM to arrive.
tpowell 15 hours ago [-]
I wrote this in June, and I'm honestly not sure I've felt the same magic since:
I was close to maxing out my $200 plan for the week, almost all Fable use [Claude CLI].
My observations:
Fable seemed to have bigger-picture thinking and completed tasks more thoroughly vs just focusing on executing the ask. It pieced together context and intent like an all-star employee would, vs one that just does what you say. Not overeager (important!), but if the above-and-beyond was warranted, it just did it. This was surprisingly delightful.
Coderabbit seemed to find ~1/3 or so as many issues when reviewing, too.
mscrivo 6 hours ago [-]
This is exactly my experience as well.
16 hours ago [-]
rurban 13 hours ago [-]
Deepseek V4 and Kimi 3 still missing, at least GLM is there.
saberience 7 hours ago [-]
ARC-AGI is a terrible benchmark for testing LLMs because LLMs are not made, trained, or tuned for playing games.
They are trained on text to respond well to text based questions and do tasks involving modifying text files.
They are not designed for playing games, looking at games, or visual puzzles. Also translating games into text input for the LLM skews the test completely.
Imagine trying to get a human to solve visual puzzle but they can’t look at the puzzle but it has to be explained to them in textual format, we would be terrible at it.
But yet we persist in wasting time on this benchmark. It doesn’t mean anything.
tonyhart7 15 hours ago [-]
cost 20k ???? man
those are like software engineer from third world country
https://x.com/quietnning/status/2080786711861407883
It seems I was wrong. American AI companies might actually be benchmaxxing harder.
All models have probably memorized significant swaths of solution sets for popular benchmarks at this point, either accidentally or intentionally, so it's all relative at this point. However, in our experience, Chinese models do benchmax harder. This is also consistent with interacting with Chinese labs soliciting data/environments, who literally asked us for datasets and tasks modeled around and formatted like popular benchmarks.
Opus 5 will be uploaded tomorrow, but we already have the tests locally and it is truly as capable as Fable, but at 81% of the real cost. (And from subjective usage, it has a very different personality)
Data at https://gertlabs.com/rankings
Now I am not really specifically accusing Anthropic of anything here, I'm just saying their behavior is suspicious. Since you tested Fable, they wouldn't even have to lie to have optimized for your specific benchmarks, since they absolutely had permission to read your sessions if they wanted to. But obviously, that's only the situation if we take them at their word. Personally I would be a bit surprised if they just flat out were lying and secretly retaining data they say they are not, but not that surprised. The penalties for doing this are probably worth the rewards if it keeps them super far ahead in the benchmarks for years without anyone catching on.
(In actuality though, even if they really were trying to sneakily grab samples of benchmark tests via their Fable data retention rules, I don't really suspect there would've been very much time to optimize Opus 5 on it. So consider me bothered.)
I guess there is no way this can happen without benchmark being part of the training data??
"...The traces tell the why: (1) On our most classic Witness-style game, Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration. It already knows this genre. (2) But on our most novel game (unusual mechanic combinations you can't pattern-match), Opus 5 regresses below Opus 4.8. Where rules must actually be discovered through interaction, the new model is worse than the old one..."
They're smuggling a claim that benchmarks like ARC-AGI measure "interactive abstract reasoning" here, which is what is claimed by the people that make these benchmarks, and also not proven.
It's more likely that the training data was contaminated with the benchmark data.
They maybe have not intentionally benchmaxxed, but they certainly know that's what happened .
The best way to know that christmas trees are typically purchased in december isn't to access historical financial tables and run the entire analysis yourself but to just "know it" from the articles you read attesting to the fact.
If you've found data that is the answer to the question, be it a random user question or a benchmark, the softwares' goal is to produce the correct solution and it is cheaper to retrieve from storage than to compute.
It's the same with coding. The agent usually isn't really thinking about the problem from first principles, it's just giving you the answers it's already found in instances where someone else asked the same question.
It seems to me that if you're really testing for reasoning capability, just as if you were an instructor administering a test, you'll need to change the test from run to run in order to make sure the agent/student isn't just copying old tests.
[1]: Recent example: https://www.anthropic.com/research/global-workspace
doesn’t appear to be a very strong argument
My guess is, the large score jump for Opus 5 is mainly because of getting the right RL envs for training.
It's becoming harder and more expensive to build and run meaningful benchmarks, it would be interesting to see what they do with arc agi 4, maybe just give it gameboy/steam games and see how they compare vs a human baseline? The latency requirements and very long horizons in games could be an interesting challenge for llms.
I think in this age where models are heavily RL-ed on acting in specific harnesses, this type of benchmark is more important than ever, to make sure they’re not in fact moving further away from general intelligence.
Similarly LLMs are not just massive uniform artificial neural nets, and now we also have harnesses, which I'd personally view more of an extension of the model itself. The harness is both the executive and also what allows it to keep notes, use a calculator, or maybe even create scripts to help it solve complex deterministic problems.
I think it's unfair to give a human a very complex maths problem and say that they're not intelligent if they can't solve it without pen and paper or a calculate. At least expecting a human to solve complex problems this way doesn't really measure anything useful in the real world.
So for example, if your workload was literally just addition of sets of numbers, you’d never have noticed progress in the result beyond GPT3.x level models. But you would perceive a difference in the now-Tolstoyan length reasoning text accompanying the result.
We get used to the new level of intelligence so fast, any deviation feels like going back to the stone age.
If you don't believe me, create something complex with Opus 5 and then with Opus 4.5, and notice the difference.
frogs dont stay in a pot even if you slowly increase the heat. they leave. it has reportedly been attempted multiple times and they. always. leave.
For the first few weeks, they would actually provide those 660 minutes to establish trust in their cards. But after a while, they would quietly start reducing the number of minutes on subsequent top-ups—say, from 660 minutes down to only 300. They wouldn’t do this for every card, so it was difficult to prove. Instead, they relied on averages across their customer base to make the economics work.
Lately, I’ve found myself wondering whether something similar may be happening with frontier AI models. Companies launch with an exceptionally strong model and generous compute limits to build adoption. Once the model is established as a market leader, the incentives change, and users may start perceiving the service as becoming more constrained or less capable over time.
I don’t have evidence that this is what’s happening with Anthropic—or with any other AI company. It’s simply a pattern that the current situation reminds me of.
Not meaningfully improved?! Four years ago was gpt *3.5*! ChatGPT hadn’t been released!
Definitely not, lol.
I'm pretty sure you're just baiting for engagement though so well done, ya got me.
I mean, come on, this is just not true. You could not achieve anything like what you can with modern agentic coding with Fable / 5.6 Sol from any combination or configuration of GPT 3.5 era models.
It's like saying that a teenager isn't an intellectually meaningful improvement over a toddler.
Sure they're both still fundamentally flawed humans prone to cognitive error, but one is clearly more likely to hit the mark than the other when assigned a task.
There’s no way you could get models as smart by fine tuning. I couldn’t throw a problem like “build a pokemon database with UI to teach my son sql” and get a working system, nice ui, tests (which it iterated on) examples and explanations in one shot.
There weren’t thinking tokens. Maths is now dramatically better, making actual contributions when before they were mostly mocked for making extremely basic errors. Smearing is also something say is very rare in frontier models.
If you think they have barely changed you’ve either forgotten what they were like or not used them more recently, or you’re just being obtuse.
Edit: we do have a system that uses LLM and fixes the above issues largely (tracking of state, calculations and objects, still flawed in finer details). No, we don't sell, it's experimental fun and not really ready in terms of setup/ux/etc.
It codes really well for our case though.
Are you one of those anonymous billionaires as if you did this a few years ago, you would've been famous and rich.
Also, uh, fame and riches are not really my thing. Middle income is fine. My mistake was speaking up here because I got carelessly annoyed because I have skin in the game, research-wise. I'm sorry for that.
There may not be "core" improvements (structural reliability) but there are "emergent" improvements (apparent intelligence). Already the IQ tests from Maxim Lott ( trackingai.org ) show a progressive sliding towards the right side of the curve - which btw translates to a very much non-secondary decline in the user's frustration (and progresses with an increase of usability).
The more they work on it, the more probable the jump becomes - e.g. to achieve the Large Conceptual Models you say you worked on.
Not sure what longer contexts we're talking about but didn't we have an old math problem optimized, which even the LLM itself was surprised about, just a week ago? Something which wasn't possible 6 months ago.
OK, the core technology that is the language model still can’t math as well as you’d hope, but how about the end result users see from the system when they interface with it?
“Did you know humans are better at flying today than they were a thousand years ago?” ‘No they’re not, they need planes.’ Technically correct in a way but isn’t it kind of annoying to be so stubbornly pedantic when the context is speed of reaching Point B from Point A?
>You can achieve a lot by coding around the issues, but that's kinda against the point of 'AI', is it?
Will think on that a bit more.
That's absolutely insane. Is it some case of anti-AI psychosis?
(Edit: and on a similar point, structural properties such as having static ntetworks, as opposed to continuously learning and improving architectures (such as us), will reveal that there is still road ahead.)
You prompt less. You need not inject search results into the context window yourself, a window much larger than years ago. You get code that’s already been run successfully once instead of finding an obvious show stopping bug yourself.
The technology is not a brand new one that fixed everything wrong with the old one, no, but not sure I would’ve noticed your comment if it had been such a bland observation. I genuinely assume good faith here… will say am tempted to assume the standards of someone posting such a thing might be impossibly high. Glad to be having a fun conversation instead of getting your grades on my work product or something :)
LLMS by themselves are still shit at math, they still confuse weird correlation to causation every time (and sometimes in ways even a 9 year old would say "no, that's dumb"), and confuse original parameters very often.
I disagree with " "thinking" token generation being on the correct track just to 'no, wait' on an already correct conclusion", because i think that is an effect of the harness, not the LLMs.
80% off all the improvements since ChatGPT4 are in the harnesses, and the LLMs by themselves, while they improved in areas they already were good at (translation especially) did not fix any of they original issues (object permanence, calculusm correlation).
Just run old models in the playground and get them to play chess (maybe make a small custom harness if you feel like it), then replace it with a frontier model (i don't know if you still have API access without harness on US models, but if you don't try K3), you will see LLMs weaknesses were not at all fixed, even marginally. They're way better and not inducing bugs in the code, so that make them usable since Opus4.5 (anyone using them prior to that either had a greenfield project or like spending hours debugging).
That seems easily falsifiable by putting an old model into the current harness and comparing it to 5.6 Sol or Fable.
Nobody has any right to take anything I say seriously, because I’m just some random on the internet. But I don’t think true AGI is any closer than about 10-20 years away. That would be to create an actual analog for a human brain.
Am I lost or are their many models on this ranking (Opus 5 included) that clear this?
Claude gives you something like $5000 of tokens on a $200 plan.
First people practiced L33t3cod3 problems for interviews
Then people built AIs to build software
And now the AIs are studying L33t3cod3 problems
https://en.wikipedia.org/wiki/Leet
It is very hard to make a benchmark you can't do that for, but it is very easy to make your own personal test that others can't do that for since now it isn't a benchmark they can target.
They didn't. Kaggle is still running for a few more months, best result atm is ~2% with 9h runtime on one rtx6kPRO. Also note that these new results are on the semi-private set, not the public 25 games ones. Any announcement where you see "solved ARC3" is likely only dealing with the 25 public games. And that's highly questionable, until you get to see the code. (which, to my knowledge the team that claimed 99% hasn't yet published).
https://schema-harness.github.io/
For example on bp35 it took fable 290M and >12k simulated turns for 566 real turns and finish more efficiently than a human.
Regardless of the true score I think the takeaway is the benchmark measures the wrapper rather than the model.
https://huggingface.co/schema-harness
> # FRAMEWORK ARTEFACT: the run's very first transition is replayed WITHOUT advancing state # (tools.py:954 and agent.py:468 both `continue` before `state = next_state`). So on the # level that contains that step (level 0) our counters start exactly one action behind. # That skipped step was action 1 with BOTH avatars moving, so seeding n=1, bumps=0 reproduces # the framework's lagged state exactly. # CAVEAT: this seed is only right while level 0 has never been RESET. If you ever RESET # level 0, change the seed to n=0 (after a reset the rollout re-inits and no longer skips).
from here - https://huggingface.co/datasets/schema-harness/arc-agi-3-sch...
That tells me that there is some leakage between runs. The idea of ARC3 is that agents start working blind, on new tasks, via API. A RESET is counted as one action. Without seeing the actual code that produced these traces we have no way of knowing how many iterations it took, if the "framework" played the same level multiple times (comment hint above makes it likely) and so on. That's why I said that before we actually see the code / can replicate / ARC team confirms it on new envs, this should be taken with a grain of salt.
Sure none of this is certain without the source.
I do believe the authors that this schema significantly improves over the base, particularly given that it took 22x simulated turns over 14 hours, which is moving the trial and error to context rather than to game. I also don’t doubt there is some contamination.
Regardless, the approach is sound and I do believe it would significantly improve scores, even if that was +20-30 over baseline (49% in this case) it does imply the benchmark is measuring the harness more than the model.
Also the models have to figure out what "end" means. And each game involves some kind of "gotchas" thrown in the harder levels. Some games are only solved by about 2/10 people trying them.
The 99% result most likely has some leakage somewhere, either in the preparation of the environments, or from session to session.
Seriously, play some of the games. They're fun.
If you want a know why the model providers are locking down and encrypting their reasoning process, this sort of workaround is potentially why. You can play this game of whack-a-mole indefinitely if the state of the system is concealed. They could have added something like:
> ### When solving arc-agi-3 puzzles: First convert the grid into a scene description. Identify connected components, colors, shapes, positions, symmetries, repeated structures, and relationships between objects. Do not reason directly from individual pixels... use this python script to help blah blah...
Not really as one of the main goals ofr ARC-AGI 3 was measuring task efficiency on unseen games.
I'm sure there are cheats everywhere but the most sensible thing is to just accept that the LLMs of today are much more intelligent in solving reasoning tasks than the ones from half year ago.
My own private benchmark shows the same thing.
but I kinda wish I could select level... I accidentally pressed redirect button and when I came back I was once again shown level 1, all progress lost :(
I could be wrong, but given the large outsized jump solely in the ARC-AGI-3 score, it would suggest that the model didn't become significantly more intelligent overall, but significantly better at solving those specific problems.
This could mean one of two things (I think):
- Opus 5 was not benchmaxxed on ARC-AGI-3, but has benefited significantly from discussions about the various challenges and mechanisms deployed in ARC-AGI-3 such that it has far better heuristics to solve its challenges.
- Anthropic looking for buzz around their latest model picked a well regarded benchmark with significant room for improvement and focused some of Opus 5's training compute on ARC-AGI-3-style problems.
Or it could be some combination of both. Personally, given how much of an outlier the ARC-AGI-3 jump is I struggle to see it being the product of a significant improvement in general intelligence.
We would actually need a test that shows the ability of a model to export its skills to more problems ("interdisciplinarity" etc.).
test Codex, not Sol. test Claude code, not Opus
Because the data retention policies didn't guarantee that the ARC team could run the semi-private set of problems without fear of them being trained on later on. They only run the semi-private set when they get assurances like ZDR.
I believe that is only available through Enterprise API for both Anthropic and OpenAI.
Asking especially given CEO’s track record https://news.ycombinator.com/item?id=47659135
Just like the hugging face incident, Opus 5 could have escaped and went to grab data for training it shouldn’t have been able to..
They are trained on text to respond well to text based questions and do tasks involving modifying text files.
They are not designed for playing games, looking at games, or visual puzzles. Also translating games into text input for the LLM skews the test completely.
Imagine trying to get a human to solve visual puzzle but they can’t look at the puzzle but it has to be explained to them in textual format, we would be terrible at it.
But yet we persist in wasting time on this benchmark. It doesn’t mean anything.
those are like software engineer from third world country