Skip to content

Tag

#inference

Every story tagged inference, newest first.

Quantization Explained: What Q4, Q8, and FP16 Actually Do to a Local Model
Article · aiDeep read

Quantization Explained: What Q4, Q8, and FP16 Actually Do to a Local Model

Quantization shrinks AI model weights from 16-bit floats to 4- or 8-bit. Here is exactly what that trade-off costs you, and how to pick between Q4_K_M, Q8, and FP16 for the model you actually want to run, with an interactive memory calculator.

BitByteCore Silicon Desk · Aug 6, 2026 · 8 min read

Fresh
Mixture-of-Experts Models: How They Work and Why They Cut Inference Costs
Article · aiDeep read

Mixture-of-Experts Models: How They Work and Why They Cut Inference Costs

MoE models activate only a fraction of their parameters per token: DeepSeek-V3 fires 37B of 671B, delivering large-model quality at small-model compute cost. How routers, experts, and load balancing actually work, and where the savings are and are not real. With an interactive dense-vs-MoE view.

BitByteCore Silicon Desk · Aug 6, 2026 · 10 min read

How Much VRAM You Actually Need to Run a Local LLM
Article · aiDeep read

How Much VRAM You Actually Need to Run a Local LLM

VRAM is the hard constraint on running a local LLM. Here's the real math — parameters, precision, quantization, KV cache — what fits on 8GB, 24GB, 32GB, and unified-memory machines, plus where quality and speed actually break.

BitByteCore AI Desk · Aug 5, 2026 · 7 min read

The Real Privacy Audit: What Data Your AI Coding Assistant Sends Home
Article · securityDeep read

The Real Privacy Audit: What Data Your AI Coding Assistant Sends Home

Stop trusting privacy policies. Put your AI coding assistant behind a proxy and watch exactly what your code transmits — what fires on every keystroke, what "local mode" really hides, and how to shut the channel.

BitByteCore Security Desk · Aug 5, 2026 · 11 min read

The best local LLM runners in 2026: Ollama, LM Studio, vLLM, and more
Guide · aiDeep read

The best local LLM runners in 2026: Ollama, LM Studio, vLLM, and more

For most people the best local LLM runner in 2026 is still Ollama — free, cross-platform, out of your way. But LM Studio, vLLM, Apple MLX, llama.cpp, Jan, GPT4All, and Open WebUI each win a specific job. Here's which to pick — and what actually fits your GPU.

BitByteCore Research · Aug 4, 2026 · 9 min read

More stories