Tag
#quantization
Every story tagged quantization, newest first.
Quantization Explained: What Q4, Q8, and FP16 Actually Do to a Local Model
Quantization shrinks AI model weights from 16-bit floats to 4- or 8-bit. Here is exactly what that trade-off costs you, and how to pick between Q4_K_M, Q8, and FP16 for the model you actually want to run, with an interactive memory calculator.
BitByteCore Silicon Desk · Aug 6, 2026 · 8 min read
How Much VRAM You Actually Need to Run a Local LLM
VRAM is the hard constraint on running a local LLM. Here's the real math — parameters, precision, quantization, KV cache — what fits on 8GB, 24GB, 32GB, and unified-memory machines, plus where quality and speed actually break.
BitByteCore AI Desk · Aug 5, 2026 · 7 min read
How to Evaluate a Local LLM for a Real Task: A Repeatable Testing Framework
A leaderboard score is a population statistic, measured on a full-precision model you'll never run. Whether a local model can handle your invoice pipeline is a different question — here's the repeatable discipline for answering it on your own hardware.
BitByteCore AI Desk · Aug 5, 2026 · 11 min read
The best vector databases for RAG in 2026
Qdrant is still the safe default for new RAG projects in 2026, but the field moved on cost: Milvus 2.6 shipped 1-bit quantization, Pinecone launched Nexus to public preview, and object-storage-first newcomers Turbopuffer and LanceDB undercut the incumbents. An honest buyer's guide.
BitByteCore Research · Aug 1, 2026 · 12 min read
AI Inference on the Edge: How Embedded Chips in Cars, Cameras, and Appliances Actually Work
Edge inference is not cloud AI in a smaller box. It is a different silicon problem where every milliwatt and millisecond is a fixed constraint, not a knob. Here is how purpose-built NPUs, aggressive quantization, and three very different power budgets, in cars, cameras, and appliances, actually
BitByteCore Silicon Desk · Jul 29, 2026 · 12 min read
