Rendered at 06:49:11 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
tristanj 1 hours ago [-]
It's pretty obvious that Gemini 4 Carbon is not token efficient. It also has relatively low tokens-per-second compared to other models. I can think of several reasons why this is the case:
1. Google has huge amounts of cash and compute. Their goal was to train a model that that has the highest quality & frontier performance, not create a model that's efficient to run and serve. They have other models for that, like Hydrogen.
2. Google created Carbon without caring for inference costs. DeepMind is subsidized by the rest of Google's empire. Selling inference is an insignificant part of Google's revenue. But for Anthropic and OpenAI, selling inference is their entire business model. Anthropic and OpenAI have enormous incentive to sell as much inference as they can in the most efficient manner. Google doesn't have this incentive.
3. They are one of the few companies that can be inefficient. Out of all the frontier labs, Google DeepMind is probably the least constrained by compute. Chinese labs have around 10 to 100 times less compute, so they must release highly optimized models.
zarzavat 7 minutes ago [-]
It's 1. Carbon was announced the day of Google's earnings call. It's a PR stunt.
fsh 21 minutes ago [-]
4. Google did a worse job than Anthropic and OpenAI.
dpe82 13 minutes ago [-]
Well sure, but parent is trying to reason about why.
teknologist 20 minutes ago [-]
Also, Google uses their own TPU architecture rather than GPU/Nvidia.
horacemorace 1 hours ago [-]
I was just using this one today for some coding tinkering and it was super slow. But the way it solves problems with a light touch while still driving toward sensible architecture is refreshing after claude’s bulldozer attitude. It was able to integrate repeated creeping scope nicely with what it had already done. Very different feel than sonnet/opus, and even gemini 3.8.
Lerc 35 minutes ago [-]
A Theo vid suggests there's a time to first token issue. When doing subagent actions it can compound to be massive from a bunch of internal short queries.
It really looks like something is broken more than just being inefficient.
96khz 6 minutes ago [-]
are we talking about token prefill here?
teknologist 18 minutes ago [-]
Theo seems to dislike anything that isn't Anthropic Claude. Suspicious, to say the least. On CursorBench, a good measure of real-world coding tasks, Gemini 4 Carbon comes out top: https://cursor.com/evals
lp92 1 hours ago [-]
>Carbon defended against cases the ticket's facts had already excluded, hardening the code against problems nobody had reported.
That's pretty interesting. Seems like it's over analyzing. Not the worst outcome though to have additional hardening.
With that said, I found Argon to be better for most coding tasks and Carbon for architecture/deep RCA analysis of complex issues.
user_7832 2 hours ago [-]
I am increasingly more convinced every day that every LLM company is just making their latest model to be the best at (coding) benchmarks, at the cost of pretty much everything else.
I use LLMs for non code purposes (probably not very common on HN) and Sonnet 3.5/3.6 still ranks as probably one of the best models I've used.
Can today's models put out tech demos and call more tools that make it way more impressive? Sure. But it's really hard to see any actual significant gains over the last few months, they're so mild (again, non coding).
wellcolormesurp 15 minutes ago [-]
They are cracking software and math, not finance, law or medicine or whatever it is you do.
Those two domains have leverage, the others do not. It makes perfect sense. If you and I had the goals they have we would start with code and math as well. The other dominos will fall automatically once you got the engineering automated. It makes no sense to stop focusing on that until you either run out of money or achieve your goal.
auyez 31 minutes ago [-]
At this point it might make sense to put all their resources into improving coding and math skills, because it might lead them to a state where they keep accelerating their capabilities. Even in the worst scenario where LLM approach would hit some wall, they would have technology to quickly explore all other approaches.
incognition 1 hours ago [-]
Oddly flash 3.8 for me is good. they really pushed carbon to some other extreme
SubiculumCode 2 hours ago [-]
That isn't a good look, and either this is unmeasured uncontrolled behavior, or intentional under some longer term plan...but the latter possibility seems improbable.
ahsillyme 1 hours ago [-]
Uhm, I'm not a programmer but why isn't this a good look? I would want my engineers to fix problems that they notice in passing. (Edit: the "unnecessary work" angle is what irked me in the article)
horacemorace 29 minutes ago [-]
Good SWEs get a feel for the minimum amount of changes to fix something. Fewer changes usually minimizes the chance for new problems to pop up.
Junior SWEs usually try to do too much.
jrflo 1 hours ago [-]
Is this due to Google wanting to catch up on benchmarks at all costs, and trains their models to be overly meticulous to do so?
1. Google has huge amounts of cash and compute. Their goal was to train a model that that has the highest quality & frontier performance, not create a model that's efficient to run and serve. They have other models for that, like Hydrogen.
2. Google created Carbon without caring for inference costs. DeepMind is subsidized by the rest of Google's empire. Selling inference is an insignificant part of Google's revenue. But for Anthropic and OpenAI, selling inference is their entire business model. Anthropic and OpenAI have enormous incentive to sell as much inference as they can in the most efficient manner. Google doesn't have this incentive.
3. They are one of the few companies that can be inefficient. Out of all the frontier labs, Google DeepMind is probably the least constrained by compute. Chinese labs have around 10 to 100 times less compute, so they must release highly optimized models.
It really looks like something is broken more than just being inefficient.
That's pretty interesting. Seems like it's over analyzing. Not the worst outcome though to have additional hardening.
With that said, I found Argon to be better for most coding tasks and Carbon for architecture/deep RCA analysis of complex issues.
I use LLMs for non code purposes (probably not very common on HN) and Sonnet 3.5/3.6 still ranks as probably one of the best models I've used.
Can today's models put out tech demos and call more tools that make it way more impressive? Sure. But it's really hard to see any actual significant gains over the last few months, they're so mild (again, non coding).
Those two domains have leverage, the others do not. It makes perfect sense. If you and I had the goals they have we would start with code and math as well. The other dominos will fall automatically once you got the engineering automated. It makes no sense to stop focusing on that until you either run out of money or achieve your goal.