Tag

#gguf

Every story tagged gguf, newest first.

Three large processor modules with metal heat sinks and three smaller circuit boards arranged on a gray concrete surface

Quantization Explained: What Q4, Q8, and FP16 Actually Do to a Local Model

Quantization shrinks AI model weights from 16-bit floats to 4- or 8-bit. Here is exactly what that trade-off costs you, and how to pick between Q4_K_M, Q8, and FP16 for the model you actually want to run, with an interactive memory calculator.

Ahmad J · Aug 6, 2026 · 8 min read

Two types of computer memory modules with gold connectors arranged on a gray surface, with a smaller chip positioned separately to the right

Tutorial · aiDeep read

How to choose the right quantization for a local LLM

Decode the Q4, Q5, and Q8 labels on model files, understand what bits-per-weight actually costs you, and pick a quantization that fits your RAM without wrecking quality.

Ahmad J · May 24, 2026 · 4 min read