Serving Many LoRA Adapters on One M5 Max: Hot Swapping Fine Tunes Without Reloading the Base Model (2026)

If you fine tune a model per client, per tenant, or per task, you very quickly own more fine tunes than a single machine can hold as separate models. Merge each LoRA adapter into its base and you get one full weight copy per fine tune, and a 70B model at 4 bit is roughly 40GB, so even a 128GB M5 Max holds two of them with no room to breathe. The point of LoRA is that you never had to merge. One frozen base can back dozens of adapters if the serving layer keeps that base resident and moves only the small adapter weights. Here is how to build that on Apple Silicon, and the numbers that decide how many tenants one box can carry.

Why Merged Serving Does Not Scale

The default path most teams take is to merge the adapter into the base at deploy time and serve the merged checkpoint like any other model. It is simple, it needs no special runtime, and the forward pass is a plain dense inference with zero adapter overhead. It also means every fine tune is a full copy of the base weights, and that is the wall you hit.

Consider a shop serving ten client specific fine tunes on an 8B base at 4 bit, roughly 5GB each merged. Ten merged copies is 50GB of weights that are ninety nine percent identical, differing only in the handful of low rank deltas the fine tune actually changed. You are spending 45GB of unified memory to store the same base ten times. Move to a 70B base and the arithmetic stops being wasteful and becomes impossible: two clients fill the machine. Merged serving treats an adapter as if it were a whole model, when it is really a small patch, and that category error is what caps your tenant count far below what the hardware could support.

Serving model	Memory per extra fine tune	Tenants on 128GB (70B Q4 base)	Switch cost
Merged per fine tune	Full base copy, tens of GB	2 to 3	None, but almost no tenants fit
Shared base, adapters resident	Adapter only, hundreds of MB	Dozens	Near zero if adapter is in memory
Shared base, adapters on disk	Zero resident	Effectively unlimited catalog	One disk load per cold adapter

What an Adapter Actually Costs

A LoRA adapter is two low rank matrices per targeted layer. The size is set by the rank, the number of layers you adapted, and the hidden dimension, not by the size of the base. For a common setup, rank 32 adapters on the attention projections of an 8B model land around 60MB to 100MB in fp16, and a 70B model with the same rank on the same projection set is a few hundred megabytes because it has more layers and a wider hidden size. Quantize the adapter or drop the rank and the number falls further.

That is the whole reason this works. If you keep the base resident once and hold adapters at a hundred megabytes each, a machine with 40GB spent on the base and 60GB free can keep a few hundred adapters in memory simultaneously. The base is the fixed cost you pay once; the adapters are the marginal cost per tenant, and the marginal cost is small. Our fine tuning walkthrough covers how rank and target modules drive that adapter size in the LoRA and QLoRA on MLX writeup, and the tradeoffs are worth understanding because rank is the main lever you have over both quality and resident footprint.

Three Ways to Hold the Adapters

Once the base is shared, the design question is where the adapters live and how a request reaches the right one. There are three regimes, and most real deployments blend them.

The first is all adapters resident. Every adapter you serve is loaded into unified memory alongside the base, and switching between them is a pointer change with effectively zero latency. This is the fastest path and the right default when your catalog of active fine tunes is small enough to fit. It is also the simplest to reason about because there is no cold path to worry about.

The second is adapters on disk with a resident cache. You keep the hot adapters in memory and leave the long tail on the SSD, loading an adapter on first request and evicting the least recently used one when memory pressure demands it. This is the pattern for a large catalog with a skewed access distribution, which is most multi tenant reality: a few clients are busy and the rest are occasional. The cost you pay is a cold load the first time a dormant adapter wakes up.

The third is a full reload of a merged model per request, which is the anti pattern this whole post exists to prevent. If your serving layer reloads base plus adapter on every switch, you have thrown away the shared base entirely and turned each request into a multi second model load. If you see per request latency in the seconds and disk reads the size of the base on every tenant switch, this is what is happening.

Measuring the Cold Path

The cold load is the number that decides whether the disk backed regime is viable, so measure it rather than guess. On an M5 Max reading from the internal SSD, pulling a hundred megabyte fp16 adapter and applying it to a resident base is fast, on the order of tens to low hundreds of milliseconds depending on adapter size and how much of the apply step you can overlap with the incoming request. Frame these as figures from a controlled test on one machine, not a guarantee; your rank, quantization, and layer coverage move them.

# Sketch: resident base, LRU adapter cache, cold load from disk on miss.
# Numbers are illustrative of the pattern, not a benchmark promise.
from collections import OrderedDict

class AdapterCache:
    def __init__(self, base_model, capacity):
        self.base = base_model            # loaded once, stays resident
        self.capacity = capacity          # how many adapters kept hot
        self.hot = OrderedDict()          # adapter_id -> in memory weights

    def get(self, adapter_id):
        if adapter_id in self.hot:
            self.hot.move_to_end(adapter_id)   # mark recently used
            return self.hot[adapter_id]        # warm: near zero latency
        weights = load_adapter_from_disk(adapter_id)   # cold: SSD read + apply
        self.hot[adapter_id] = weights
        if len(self.hot) > self.capacity:
            self.hot.popitem(last=False)       # evict least recently used
        return weights

The eviction policy matters more than it looks. A pure least recently used cache is fine for skewed traffic, but if a batch mixes many rarely used adapters you can thrash, evicting an adapter you are about to need again. Sizing the hot cache to cover your working set of concurrently active tenants, not your total catalog, is the practical rule.

Batching Requests Across Different Adapters

The subtle part is throughput. If you serialize requests by adapter, running all of tenant A's requests, then swapping to tenant B, you leave the machine idle during swaps and you cannot fill a batch. The technique that fixes this is batching heterogeneous adapters into one forward pass: the base computation is shared across the whole batch, and each request applies its own adapter delta at the adapted layers. This is the idea behind S-LoRA style serving, and it is what turns a multi tenant box from a switcher into a real server.

The implementation cost is that your kernels must apply per request adapter weights inside a batched matmul rather than assuming one adapter for the whole batch. Frameworks are catching up to this on Apple Silicon through MLX, and the shared base pattern generalizes the same admission and memory logic we used for running several distinct models in one pool, covered in the concurrent multi model serving writeup. The difference is that adapters share the base, so the memory story is far friendlier than serving unrelated models.

When This Applies to Your Stack

Multi adapter serving earns its complexity when you have many fine tunes that share a base and a machine that cannot hold them merged. That is the per client, per tenant, or per task shape. If you have one fine tune, merge it and move on; the batched adapter machinery is pure overhead for a single model, and the 70B fine tuning walkthrough covers the single model path. If you have twenty fine tunes and a 70B base, the shared base pattern is the difference between two tenants and dozens on the same hardware you already own.

If you are standing up local, private inference across many fine tuned models and want the serving layer designed so the base stays resident and adapters move cheaply, that is the kind of AI infrastructure work we do at Contra Collective. The gains are real but the failure modes, cache thrash and accidental full reloads, are easy to ship by mistake.

FAQ

Does serving an adapter slow down inference versus a merged model? Slightly. Applying the low rank delta at the adapted layers adds a small amount of compute per token compared to a fully merged dense pass. For most setups the overhead is minor and the memory savings that let you fit many tenants dwarf it. If you serve exactly one fine tune at high volume, merging is marginally faster.

How many adapters can one M5 Max realistically hold? It depends on base size and adapter rank. With a 70B base at 4 bit taking roughly 40GB and adapters at a few hundred megabytes, the free memory holds dozens of resident adapters. Smaller bases and lower ranks push that into the hundreds. Disk backing with a hot cache makes the total catalog effectively unbounded.

Is this only for MLX, or does llama.cpp support it too? Both ecosystems support LoRA against a shared base, with differing maturity on batched multi adapter forward passes. MLX is the more active path on Apple Silicon for the batched case. Test the specific batching support in your runtime version before assuming heterogeneous batches work end to end.