Running AI Models Locally: Why VRAM Decides What a Machine Can Do

A graphics card that plays anything at maximum settings is not automatically a graphics card that runs AI models well. The two jobs look similar from the outside and want different things from the hardware.

A game wants frames. A language model wants somewhere to sit. Before speed enters the conversation at all, the model has to fit into the memory on the card, and if it does not fit, nothing else about the card rescues it.

That is most of this article in two sentences. The rest is the arithmetic, and what it means for a machine that has to run in a room in Malta in August.

What Running It Locally Actually Means

A model is a file. It gets downloaded once, it sits on the drive, and a program on the machine loads it into memory and does the arithmetic there. No account, no request leaving the building, no queue.

People want that for four reasons, and usually only the first one is the real one.

• The material stays put. Client files, contracts, medical notes, unpublished source code, anything covered by an agreement that says it does not leave the premises. A local model reads it without it going anywhere.
• No running bill. The hardware is bought once instead of a monthly fee that grows with use.
• It works with no connection. Once the file is on the disk, the internet is optional.
• Nothing changes underneath. A hosted model gets updated, retired or reworded without warning. A file on a disk does exactly what it did last month.

All four are real. None of them help if the machine cannot hold the model, which is where this gets specific.

The One Number That Decides Everything

A model is a very large pile of numbers called parameters. A model described as 12B has roughly twelve billion of them. To produce one word of output the machine reads essentially all of them, and then it does it again for the next word, and again for the one after that.

So they have to live somewhere the graphics processor can reach quickly. That means the memory on the graphics card, usually written VRAM. Not the system memory. Certainly not the drive.

How much room they take depends on how precisely each one is stored. As released, a parameter is normally kept in sixteen bits, which is two bytes. Storing the same model with eight bits per parameter halves the file. Storing it with roughly four bits quarters it. That squeezing is called quantisation, and it is the single reason any of this runs on a home machine at all.

Read that chart the way a buyer reads it. A card with 8 GB holds a small model comfortably and a medium one not at all. A card with 12 GB reaches models in the twelve to fourteen billion range. Sixteen gigabytes is the first point where the useful middle of the field opens up. Twenty four and thirty two gigabytes reach the models in the twenty six to thirty one billion range, which is where a lot of people quietly decide the output is finally good enough.

The largest open models, the seventy billion parameter class and above, do not fit on any single consumer card even squeezed to four bits.

The trap in the name

Two things about names are worth knowing before any of this becomes a shopping list.

The first is that a family keeps its name while the parameter count underneath it moves. A model line that was small two years ago may not be small now. The word in the name is marketing. The parameter count and the published file size are the specification, and they are the only two things worth reading.

The second is subtler and catches technical buyers specifically. A lot of current models are built so that only a fraction of the model does the work for any given word. These get described with two numbers, something like 26B with 4B active, and it is easy to read the second number and conclude the model behaves like a small one.

It does not, as far as memory is concerned. Every part of the model still has to be sitting in memory, because which part gets used changes from word to word and there is no way to know in advance. The published download for that 26B model is 18 GB at four bits, which is exactly what a 26 billion parameter model should be, and nothing like a four billion parameter one.

That design buys speed. It never buys memory. Anyone who chooses a card on the strength of the active number will be short by a wide margin.

Where Squeezing Stops Being Free

Quantisation is not magic and it does cost something. The llama.cpp project, which is the engine underneath most local model software, publishes its own measurement of that cost. On a seven billion parameter model the file falls from 13 GB at full precision to 6.7 GB at eight bits and 3.8 GB at the common four bit setting. The measured quality penalty at that four bit setting is small. At roughly three bits it is several times larger. At two bits the model still runs and is noticeably worse at everything.

Which produces the rule that surprises people. A larger model squeezed to four bits is usually better than a smaller model kept at full precision, provided the larger one still fits. Fitting comes first, and quality is what gets traded to achieve it.

Four bits is the default in most of the software for this reason. It is not a compromise anybody should feel bad about.

The Conversation Needs Memory Too

The model file is not the whole requirement, and this is the part that catches people an hour into using the thing.

Everything the model has been told so far in the current conversation is held in memory as well, alongside the weights, in a structure that grows as the conversation grows. Current models advertise context windows of 128,000 tokens and more, and filling one costs real memory on top of the model itself.

The advertised window is a ceiling rather than an allocation, so a short exchange costs very little. A long document pasted in whole does not. A model that loaded happily with room to spare can slow down badly, or stop, once enough has been fed into it.

The practical version of this is simple. A model that exactly fills the card does not, in use, fit the card. Leave room.

What Happens When It Does Not Fit

Nothing dramatic, and that is the problem. The software does not refuse. It splits the model, keeps as many layers as it can on the graphics card and puts the rest in system memory for the processor to handle. Ollama reports the result plainly, as a line saying something like 48%/52% CPU/GPU.

Then the speed collapses, and the reason is bandwidth rather than computing power.

Generating each word means reading the whole model again, so the limit is how fast that memory can be read rather than how clever the processor is. The memory on a current high end card runs at roughly 1,800 GB/s. Ordinary dual channel DDR5 system memory in a desktop runs at somewhere around 90 GB/s. That is not a small difference and no amount of processor helps with it.

In practice a model that answers at comfortable reading pace with everything on the card can drop to a word every second or two once a meaningful share of it has been pushed into system memory. Nothing is broken. The graphics processor is sitting there waiting.

This is also why the honest advice is boring. Between two cards at similar money, the one with more memory normally wins for this work, even when the other one is faster on paper, because the faster one will spend its time waiting for system RAM.

Processor, Memory and Storage

They matter, in roughly that order of not very much, each with one exception that turns it into a real problem.

• The processor. If the whole model sits on the card, the processor mostly hands work over and waits. If part of the model does not fit, the processor is suddenly doing the arithmetic for that part, and then it matters enormously. A weak processor turns a bad situation into a much worse one.
• System memory. The model is read from disk into system memory before being handed to the card, so there has to be room for it there in the first place. It is also what a partial split spills into. Skimping here converts a fitting problem into a swapping problem, which is slower again by a wide margin.
• Storage. Model files are large and people collect them, so capacity is the requirement. A fast NVMe drive shortens loading, which happens once, and does nothing at all for generation speed.

None of these is the place to spend the budget. The card is.

Unified Memory, the Other Arrangement

Apple machines, and a small number of recent PC designs, do not have separate graphics memory at all. One pool of soldered memory serves both processor and graphics, and most of it can be handed to a model.

Apple's current top laptop part takes up to 128 GB in that pool, at a bandwidth of up to roughly 614 GB/s. That is a good fraction of a high end desktop card's read speed with four times the capacity, which is why these machines will load large models at moderate speed in situations where a PC with a 16 GB card cannot load them at all.

The trade is the obvious one. The memory is chosen once, at purchase, and cannot be changed afterwards. On a tower with a card in it, the card can be replaced later. That difference is worth thinking about before rather than after.

Power and Heat, Which Here Is Not a Footnote

This is the part that gets discovered after the machine is built, and in Malta it gets discovered sooner than elsewhere.

A game loads a card in bursts. Menus, loading screens, quiet stretches. A model loads it flat out from the moment the job starts until the moment it finishes, and a long answer means minutes, and a batch of documents means hours. The card runs at its ceiling and stays there.

The published ratings are not shy. NVIDIA rates its current top consumer card at 575 W for the card by itself and recommends a 1000 W supply for the system around it. A mid range card in the same generation is rated at 250 to 360 W.

Worth being accurate about what happens next, because the obvious guess is wrong. Measured, this work does not pin a card to its rating. Published measurements of language model serving on a current top card land somewhere between half and about six sevenths of the rated figure depending on the model and the load. The card is not at its limit.

It is there for a long time, though, and that is the part the cooling notices. Two hundred watts sustained for an hour puts more heat into a room than four hundred and fifty watts in thirty second bursts. Duration, not the peak, is what fills a small room.

Then Malta adds the part that appears in no review. Reviews are run in an air conditioned room at around twenty two degrees. The August daily maximum here sits around 32 °C, the humidity around three quarters, and the national record of 43.8 °C was set in August. This July the island came within half a degree of it. A case pulls in the air of the room it is standing in, and the room here in August is not the room in the review.

There is a detail here that is easy to miss. NVIDIA publishes no ambient temperature range at all for its consumer cards, only a maximum chip temperature. It does publish one for its own datacentre machines, and that range stops at thirty degrees. An unconditioned room in Malta in August is outside it.

The other half of the problem is that the nights do not help. The August minimum here sits in the low twenties, so a room that heated up during the day does not give it back overnight. A machine that runs long jobs starts the second day warmer than it started the first.

The consequences are ordinary and irritating rather than catastrophic. Fans that were inaudible in March become the loudest thing in the room in August. Clocks come down, and the same job takes longer for no visible reason. A power supply chosen with no margin ends up working near its limit in a hot room, which is the condition it likes least.

None of that makes a local machine a bad idea. It makes airflow, a supply with genuine headroom, and a case with somewhere for the heat to go part of the specification rather than an afterthought. In a laptop, where the manufacturer has already decided how much cooling the parts get, it is worth being realistic about what a long job will actually do.

The Software

The engine underneath most of this is llama.cpp, an open source project that runs models on ordinary hardware. Very few people use it directly. What they use is one of the things built on top.

• LM Studio. A proper application with a window. Browse models, download them, chat, adjust the settings, and see whether a model will fit on the card before committing to the download. The usual starting point for somebody who has not done this before.
• Ollama. Driven from the command line, installs in a minute, downloads and runs a model with one instruction. It also serves an API on the machine, which is what makes it the common choice when something else has to talk to the model.
• Open WebUI. A browser front end that sits on top of Ollama and looks like the hosted chat services people already know. Useful when more than one person in an office needs access to the same machine.
• llama.cpp itself. The engine, for anyone who wants every switch.

Most of them present an interface that imitates the commercial hosted ones, which is the part that matters beyond chatting. Existing software that expects a hosted service can often be pointed at the local machine instead by changing an address in a settings file.

It is not only text either. Image generation runs locally as well, and it is generally hungrier for card memory than text models of comparable reputation. Speech to text sits at the other end of the scale and runs on very modest hardware.

One warning belongs with the image side specifically, because it costs money rather than time. Open weights and open licence are not the same thing. Several of the best known open image models are free to download and free to experiment with, and their licence does not permit commercial use. Two versions of the same model, released on the same day, can differ on exactly that point. A business building a workflow on one of them needs to read the licence for the specific version being used, not for the family name.

There is also a question that comes up with every recent laptop, which is what the NPU does in all this. Almost always, nothing. The chip is there and it is genuinely capable, but the common programs for running models fall back to the graphics portion or to the processor unless a good deal of effort is put in. Support exists and it is improving. It is not what happens when the software is simply installed and pointed at a model, and it is not a reason to choose a machine.

Offline is achievable, not automatic

Since privacy is the usual reason for doing any of this, the promise deserves checking rather than assuming.

The model itself genuinely runs on the machine, and the text typed into it genuinely stays there. Around that model, though, several of the popular programs will still reach out on a fresh installation. Update checks. Anonymous usage counters that are switched on by default in some and off by default in others. In at least one widely used front end, a setting that points at a commercial hosted service unless it is deliberately turned off, which is not what anybody installing it for privacy reasons expects.

None of that is hidden and all of it can be switched off. It just has to be done on purpose. An installation that has genuinely been closed off is a different thing from an installation that merely runs the model locally, and for anybody whose reason for being here is confidentiality, the difference is the whole point.

One caution about lists like the one above. This field changes its favourites every few months, and tools that were unavoidable two years ago are now unmaintained. Whatever gets recommended, it is worth checking that the project has had a release recently. The same goes for model names, which is why the chart further up is labelled by size rather than by brand. The sizes have stayed put. The names have not.

When It Is Not Worth Doing

This is the section that tends to get left out, and it is the one that saves the most money.

• Occasional use. Somebody who needs a model twice a month should not buy a graphics card for it. A subscription to a hosted service costs a small fraction of the hardware, needs no maintenance, and does not have to be cooled. The same open models that run at home are also sold by the million words by hosting companies, at prices low enough that light use never repays a card.
• The very best answers. The largest hosted models are far larger than anything that fits in a home machine. A model that runs on one consumer card is a genuinely useful tool and it is not the same tool. Anyone whose actual requirement is the strongest answer available will be disappointed by the comparison, and that is better heard before the money is spent than after.
• A single heavy job. Renting is real and it is cheap. A machine with a current top card can be rented by the hour from the commodity providers for well under a euro. A few weekends of that is nowhere near the price of the card, let alone the machine around it.
• Nothing confidential involved. If the privacy argument does not apply and the volume is low, the sums simply do not work out.
• A machine meant to run continuously. Worth doing the electricity before committing, because the domestic tariff here is banded and the rate climbs as consumption does. A machine working around the clock is not billed at the headline rate. It is billed at whatever band it pushes the household into, and in a house that already uses a lot of air conditioning that is a steep band.

Local wins where those reverse. Material that cannot leave the building, steady daily use rather than occasional use, a fixed cost instead of a variable one, or work that has to carry on with no connection.

It is a question with an answer, and the answer depends on the work rather than on enthusiasm.

A Checklist Before Buying

• Decide what has to run first. The model size sets the memory, and the memory sets the card. Doing it the other way round is how the money gets wasted.
• Count the memory on the card, not the model number. More memory beats more speed for this work, nearly every time.
• Leave headroom above the model file for the conversation. A card filled exactly to the model size is a card that will spill.
• Size the power supply for sustained load with margin, not for the number on the card's box.
• Plan the airflow for an August afternoon, not for a review bench.
• Try the job on rented hardware first if the requirement is not yet clear. It costs very little to find out.

Working It Out Without Guessing

Everything above can be established at home with the published sizes of the models involved and the specification of the card being considered. The three answers needed are what has to run, how much memory that needs once the conversation is counted, and whether the machine can keep that hardware cool while it works.

Laptop-Malta is in St Julian's, and there are four separate things people come in for on this subject.

• Working out what is actually needed. Somebody arrives with a model they want to run, or a task they want done, and the useful answer is what that requires. Sometimes the useful answer is that it does not require a new machine.
• Building the machine. Specified, assembled, tested and handed over working.
• Upgrading a machine that already exists. Usually a card with more memory, often more system memory alongside it, and frequently the power supply that the new card needs.
• Setting up the software. Installing and configuring the models on the machine, so that it works rather than merely being installed.

The first of those is worth having before the other three. The expensive mistakes in this area are made at the shopping stage, by matching a budget to the fastest card rather than to the amount of memory the job actually needs. Our desktop machines are here, and the shop can be reached from the contact page.

If the answer turns out to be that the work does not justify the hardware, that is the answer given. It is a shorter conversation and it beats building a machine around a number on a box.

Related reading

On the memory that feeds all of this: what actually changed between DDR4 and DDR5. On the drive those model files land on, and why one can fit the slot and still not work: SATA, NVMe and what M.2 actually means.