ai-inference-cost - The LGTM

The LGTM

Sign in Subscribe

ai-inference-cost

A collection of 5 posts

DFlash Turns Speculative Decoding Into a Blackwell Utilization Story

DFlash Turns Speculative Decoding Into a Blackwell Utilization Story

Speculative decoding is one of those ideas that sounds like a serving hack until you remember the original problem is stranger: we run massively parallel accelerators and then ask large language models to emit text one token at a time. DFlash, the block-diffusion speculative decoding method NVIDIA is now highlighting

Tokens per Watt Is Becoming the AI Factory’s Real P&L Metric

Tokens per Watt Is Becoming the AI Factory’s Real P&L Metric

“Tokens per watt” sounds like a facilities metric until you have to run a real AI product under real capacity limits. Then it becomes the P&L. NVIDIA’s latest AI factory energy-efficiency post is easy to misread as green-computing content. It is really a margin story: every watt

NVIDIA and AWS Are Making GPU Retrieval Boring — Which Is Exactly the Point

NVIDIA and AWS Are Making GPU Retrieval Boring — Which Is Exactly the Point

GPU announcements usually arrive dressed as hardware news. This one is more useful if you read it as a retrieval infrastructure story. NVIDIA and AWS are not merely adding another accelerated EC2 shape; they are pushing GPU acceleration into the boring middle of production AI, where vector indexes get rebuilt,

NVIDIA’s TOP500 Lead Is an AI-Factory Check, Not a Trophy Case

NVIDIA’s TOP500 Lead Is an AI-Factory Check, Not a Trophy Case

NVIDIA’s latest TOP500 brag sheet is easy to dismiss as another vendor victory lap. Don’t. The useful signal is not that NVIDIA technology appears in 81% of the world’s 500 fastest supercomputers, or that it powers nearly 90% of the systems newly added to the list. The

Rubin’s 45°C Liquid Loop Is NVIDIA Turning Cooling Into Token Economics

Rubin’s 45°C Liquid Loop Is NVIDIA Turning Cooling Into Token Economics

NVIDIA’s new Rubin cooling story looks, at first glance, like facilities plumbing: warmer coolant, fewer chillers, less water, cleaner racks. That undersells it. The real announcement is that NVIDIA is trying to make the physical data center part of the AI runtime, because at agent scale the cost of