LLM Concepts

The biggest speedup in local inference is one you never turned on

Jul 1, 2026

•

2 min read

Optimization techniques

The biggest speedup in local inference is one you never turned on

To work through your prompt, attention compares every token with every other token. A thousand tokens means a thousand-by-thousand grid of comparisons. Flash attention just refuses to build the grid.

Yatharth Lakhera

Speculative decoding isn't free. It's a bet you can lose.

Jun 28, 2026

•

2 min read

Optimization techniques

Speculative decoding isn't free. It's a bet you can lose.

Best case, several tokens for the price of one. Worst case, slower than doing nothing. And what you're wagering on is how predictable your output is.

Yatharth Lakhera

The runtime is the new API. You already picked one.

Jun 27, 2026

•

2 min read

LLM Concepts

The runtime is the new API. You already picked one.

I went local to stop letting a company quietly decide things for me. My third command was ollama run. And just like that, something was quietly deciding everything again.

Yatharth Lakhera

Your bandwidth is fixed. Your speed isn't.

Jun 24, 2026

•

2 min read

Optimization techniques

Your bandwidth is fixed. Your speed isn't.

Pop-up wardrobes, tape-to-stream labs, and story prompts powering this week’s experiments.

Yatharth Lakhera

How models get smarter without getting slower

Jun 22, 2026

•

2 min read

Optimization techniques

How models get smarter without getting slower

Bigger means slower: more weights to drag out of memory for every token. Want more capability? Pay for it in speed. By now it feels like physics. But there is a hack!

Yatharth Lakhera

Jun 18, 2026

•

2 min read

LLM Concepts

Your model has two kinds of slow

When I quantized the cache and dropped the context, two numbers got better at once. The wait before the first word, and the speed of the words after it. I filed it under "faster" and moved on.

Yatharth Lakhera

I fixed every memory problem. It was still slow.

Jun 18, 2026

•

2 min read

LLM Concepts

I fixed every memory problem. It was still slow.

I did everything the last three posts told me to. Quantized the KV cache down to q8. Dropped the context from maxed-out to 8k. TTFT dropped, tokens per second climbed, the SSD stopped thrashing

Yatharth Lakhera

The model fit in memory. The conversation didn't.

Jun 18, 2026

•

2 min read

LLM Concepts

The model fit in memory. The conversation didn't.

The 9B model was 6.6GB. It sat inside the Mac Mini's 16GB of memory with room to spare. So what dragged "hello" down to one word a second?

Yatharth Lakhera

Jun 14, 2026

•

2 min read

LLM Concepts

What a layer in an LLM actually is

How do you put part of a model anywhere? You can't load half a brain. Turns out you can. Because a model isn't one object. It's a stack.

Yatharth Lakhera

For running LLMs, your RAM number is a lie. Look at the VRAM.

Jun 14, 2026

•

2 min read

LLM Concepts

For running LLMs, your RAM number is a lie. Look at the VRAM.

Few day's after I had the Mac Mini running models, I found the setting half the internet argues about: n_gpu_layers. On a gaming PC it's the make-or-break dial. I'd never once touched it.

Yatharth Lakhera

I bought a Mac Mini to run a 9B model. "Hello" took a minute.

Jun 14, 2026

•

2 min read

LLM Concepts

I bought a Mac Mini to run a 9B model. "Hello" took a minute.

I typed "hello" and watched the cursor blink. Sixty seconds later, one word came back.

Yatharth Lakhera

LLM Concepts

The biggest speedup in local inference is one you never turned on

Speculative decoding isn't free. It's a bet you can lose.

The runtime is the new API. You already picked one.

Your bandwidth is fixed. Your speed isn't.

How models get smarter without getting slower

Your model has two kinds of slow

I fixed every memory problem. It was still slow.

The model fit in memory. The conversation didn't.

What a layer in an LLM actually is

For running LLMs, your RAM number is a lie. Look at the VRAM.

I bought a Mac Mini to run a 9B model. "Hello" took a minute.

The Agent Engineer