Which model, which size, and what all those letters after the file name actually mean.
Sovereign Inference runs open models in the .gguf format. You point it at a folder, it finds what is in there, and you pick one. No account, no queue, no rate limit, and no company deciding one morning that the model you built a workflow around is now deprecated.
The awkward part is choosing. A single model can be published in twenty versions with names like Q4_K_M and IQ4_XS, and almost nobody explains what separates them. This page does.
Two numbers decide nearly everything: how many parameters a model has, and how heavily it has been compressed. Get those right and the rest is taste.
Every model on this page is a general one. If you need something that knows your products, your documents, your policies and the way your customers actually ask about them, that is a different job, and it is one we do.
Custom AI ModelsThe B in 7B means billion parameters. More parameters generally means a better answer and a slower one, and it always means a bigger file. The memory figures below are rough, for the common Q4_K_M compression, and they are the number to check before you download 40 GB of anything.
Fast, small, and limited. Good for tidying text, quick summaries, autocomplete and anything you want instantly on a laptop with no graphics card. It will not reason its way through a hard problem, and it should not be trusted with facts.
The sensible default. This is where a local model starts feeling genuinely useful: drafting, rewriting, everyday questions, light code. Runs comfortably on most modern machines and on nearly any dedicated graphics card.
Noticeably sharper. Better at following long instructions, holding a thread and writing code that runs. This is the range where a 16 GB card earns its money, and where most people settle once they have the hardware for it.
Serious work. Real reasoning, long documents, proper coding assistance. Needs a 24 GB card to be quick about it, or a lot of patience and a lot of system memory.
Workstation territory. Excellent, and demanding. Either multiple cards, or a machine with a very large pool of fast unified memory. Running one of these off system RAM works, but expect to read the answer as it types.
The exception to the rule. An MoE model may have a huge total parameter count but only wakes a fraction of it per token, so it can run far faster than its size suggests. It still has to fit in memory in full, so budget for the whole file, not the active part.
Models are trained at 16 bits per weight. Quantising rounds those weights down to fewer bits, which shrinks the file dramatically and costs some accuracy. The name on the file tells you exactly how it was done, once you know how to read it.
Four bits per weight · K style mixed precision · medium variant
It simply means the weights have been compressed below their original precision. Every file you download for local use will have it.
Roughly how many bits each weight is stored in. Lower is smaller and faster and less accurate. This is the single biggest lever on the file.
K quants do not compress everything equally. The layers that suffer most from rounding keep more bits, the tolerant ones get fewer. Better quality for the same size.
How generous that mixing is. S is small and leanest, M is medium and the usual pick, L is large and keeps the sensitive parts richer still.
_0, as in Q8_0, is the older uniform method that predates K quants. IQ is newer and uses an importance map to decide which weights deserve the bits.
There is no Q8_K_M or Q8_K_L in the standard set. At eight bits the loss is already so small that the clever mixing has nothing left to save, so Q8_0 is the practical maximum and is effectively indistinguishable from the original.
You will see names like Q4_K_L or Q6_K_L from certain repackagers. Those usually mean the standard quant with the embedding and output layers kept at higher precision. Useful, slightly larger, and not part of the core naming.
Whatever the letters say, the number of gigabytes tells you what you are actually getting. If two files claim different quality and weigh the same, they are the same trade wearing different names.
And remember the rule underneath it: a bigger model squeezed harder usually beats a smaller model kept pristine. A 13B at four bits will generally serve you better than a 7B at eight, for about the same memory.
The sweet spot, and the reason it is on almost every download page. Roughly a quarter of the original size for a quality drop most people never notice in normal use. If you are unsure, take this one.
Measurably closer to the original for a moderate size increase. Worth it when the model is doing something exacting, such as code or long structured output, and you have the memory spare.
Essentially the full model. Rarely worth double the memory of Q4_K_M for conversational use, genuinely worth it for small models where the file is tiny anyway.
Importance matrix quants squeeze harder for the same quality, so an IQ4 often matches a Q4_K_M in slightly less space. They can be a little slower on some hardware. Worth trying when you are a gigabyte short.
The damage starts becoming visible here. Instructions get missed, formatting slips, and long answers wander. Use it to get a model running at all, then move up as soon as you can.
At two bits a model is a shadow of itself. It still produces fluent sentences, which is exactly the problem, because the confidence survives the compression and the accuracy does not. A smaller model at Q4 is nearly always the better choice.
Every family below publishes open weights you can download and keep. We have deliberately not named specific versions or quoted benchmark scores, because both go stale within weeks and a page that pretends otherwise is lying to you. The character of each family holds far longer than its leaderboard position.
Rankings change faster than we could keep this page honest, and most published comparisons are measuring something other than whether the thing is useful to you. So we have told you how the sizes and the compressions actually work, and left the choosing to you. Download two, ask them both the same awkward question, and keep the one that answered it properly.