Self Hosting Open Source AI Models: Infrastructure, Costs, and the Trade Offs Nobody Talks About

Running your own AI models sounds like the ultimate cost optimization. The reality is more nuanced. Self hosting shifts costs from API bills to infrastructure and engineering time, and the break even point is further out than most teams expect. But when it makes sense, it makes a lot of sense: lower latency, full data control, and inference costs that drop to near zero at scale.

The question isn't whether self hosting is viable. It is. The question is whether your team, your workload, and your scale justify the operational overhead.

The Infrastructure Stack for Production Model Serving

Self hosting an open source model in production requires four layers, and each one has decisions that compound:

GPU Selection

This is the single biggest cost driver. The GPU market for inference has stratified into clear tiers:

NVIDIA A10G. The workhorse for models up to 13B parameters. Available on AWS (g5 instances), GCP, and most cloud providers. A single A10G with 24GB VRAM can serve a quantized Llama 4 Scout 17B model at production throughput. Cost: roughly $0.75 to $1.10 per hour on demand, $0.30 to $0.45 per hour with reserved instances or spot pricing.

NVIDIA L4. Google Cloud's inference optimized GPU. Better price performance than the A10G for pure inference workloads (not training). 24GB VRAM, Ada Lovelace architecture with efficient INT8/FP8 support. Cost: approximately $0.65 to $0.85 per hour.

NVIDIA A100 / H100. Overkill for most mid market inference workloads, but necessary if you're serving models above 30B parameters without aggressive quantization. The 80GB VRAM variants can host a full Llama 4 Maverick model. Cost: $2.50 to $4.50 per hour for A100, $5 to $8 per hour for H100.

AMD MI300X. The new contender. 192GB HBM3 memory means you can serve models that would require multi GPU setups on NVIDIA hardware, on a single card. Software ecosystem is maturing rapidly via ROCm. Cost competitive at roughly $3.50 to $5.00 per hour in cloud environments that offer it.

For most mid market ecommerce workloads, a pair of A10G or L4 GPUs handles everything you need. Don't overprovision.

Model Serving Framework

You need a serving layer between your model weights and your application. The three viable production options:

vLLM has become the default. It implements PagedAttention for efficient memory management, supports continuous batching, and handles concurrent requests without the throughput collapse you see in naive implementations. It supports most model architectures out of the box and integrates with OpenAI compatible API endpoints, so your application code doesn't need to change.

TensorRT LLM (NVIDIA) squeezes more performance from NVIDIA GPUs through aggressive kernel fusion and hardware specific optimizations. The trade off is tighter hardware coupling and a more complex build process. Worth it if you're running at high volume on NVIDIA hardware and need every millisecond of latency reduction.

Ollama is the right answer for development and prototyping. It's not production grade for high concurrency workloads, but it's the fastest path from "I want to test this model" to "it's running locally." Use it for evaluation, not for serving production traffic.

Quantization Strategy

Running a model at full precision (FP16/BF16) gives you the best output quality but requires roughly 2 bytes per parameter in VRAM. A 70B parameter model needs 140GB of VRAM at full precision, which means a multi GPU setup.

Quantization trades a small amount of output quality for dramatically lower resource requirements:

For ecommerce workloads (classification, extraction, summarization) INT8 quantization is the sweet spot. The quality loss is imperceptible for these task types, and you halve your GPU requirements. Only go to INT4 if you're running on constrained hardware and your use case tolerates the quality trade off.

AWQ (Activation aware Weight Quantization) currently produces the best results for inference focused quantization. Most popular models ship with pre quantized AWQ variants on Hugging Face, so you don't need to quantize them yourself.

Orchestration and Scaling

A single GPU instance handles testing and low traffic production. As volume grows, you need:

• Autoscaling. Scale GPU instances up and down based on request queue depth, not CPU metrics. Most cloud autoscalers don't understand GPU utilization patterns out of the box. Use custom metrics from your serving framework.
• Request routing. If you're serving multiple models (a small model for classification, a larger one for generation), route requests to the appropriate model endpoint. A simple API gateway with path based routing works.
• Health checks. GPU processes fail silently more often than CPU workloads. Implement inference level health checks that actually run a test prompt, not just TCP port checks.

The Real Cost Breakdown

Here's what self hosting actually costs for a mid market ecommerce workload (500,000 inference calls per month, mixed workload):

Infrastructure:

• 2x A10G instances (reserved, 1 year): $650/month
• Storage for model weights and logs: $50/month
• Networking and load balancing: $75/month
• Subtotal: $775/month

Engineering:

• Initial setup: 40 to 80 engineering hours (one time)
• Ongoing maintenance: 5 to 10 hours per month
• At $150 per hour fully loaded: $750 to $1,500 per month ongoing

Total monthly cost: $1,525 to $2,275 per month

Compare that to $4,500 to $8,000 per month in proprietary API costs for the same volume. The savings are real, but they don't show up until month 3 or 4 after you amortize the setup investment. And they assume your team has the skills to maintain GPU infrastructure.

The Trade Offs Nobody Mentions in the Blog Posts

Model updates are your problem now. When Meta releases Llama 4.1, you need to evaluate it, re quantize it, test it against your fine tuned benchmarks, and deploy it. This isn't hard, but it's ongoing work that never stops.

GPU availability is not guaranteed. Cloud GPU spot instances get reclaimed. Reserved capacity sells out in popular regions. If your inference workload is latency sensitive, you need a fallback strategy, usually a proprietary API endpoint that activates when your self hosted infrastructure is unavailable.

Debugging is harder. When a proprietary API returns bad output, you file a support ticket. When your self hosted model returns bad output, you're debugging quantization artifacts, tokenizer mismatches, context window overflow, and CUDA memory errors. Your team needs to be comfortable at this level.

Compliance can cut both ways. Self hosting gives you data sovereignty, meaning no customer data leaves your infrastructure. That's a clear win for GDPR and data residency requirements. But it also means you're now responsible for securing the model serving infrastructure, auditing access, and maintaining encryption at rest and in transit. The compliance benefit comes with compliance responsibility.

The Decision Framework

Self host when:

• Monthly API costs exceed $3,000 and are growing
• Latency requirements are below 200ms (hard to achieve consistently via external APIs)
• Data residency or privacy regulations require on premises or private cloud inference
• Your workload is well defined enough to fine tune smaller, specialized models
• You have at least one engineer comfortable with GPU infrastructure

Stay on APIs when:

• Monthly inference costs are below $2,000
• Your AI use cases are still evolving and you need the flexibility of general purpose models
• Your engineering team is fully committed to product work and can't absorb infrastructure responsibilities
• You need frontier model reasoning capabilities that open source hasn't matched yet

The Hybrid Architecture Most Teams Should Build

The pragmatic answer for most mid market brands is a routing layer that sends requests to the right backend:

Request → Router → [Self hosted model] (classification, extraction, search)
                 → [Proprietary API]    (complex reasoning, creative generation)
                 → [Proprietary API]    (fallback when self hosted is unavailable)

Build the router as a simple API gateway. Tag each request type with a model preference. Start by self hosting your highest volume, most well defined task. Expand from there as your team builds operational confidence.

How Contra Collective Helps

We design and implement hybrid AI architectures for mid market brands, covering everything from GPU sizing and model selection through production deployment and monitoring. We've seen what works and what turns into an operational burden. If you're evaluating whether self hosting makes sense for your workload, let's talk through the numbers.

Final Thoughts

Self hosting open source AI models is not a universal cost savings play. It's an infrastructure decision with real trade offs in engineering time, operational complexity, and organizational capability. For the right workloads at the right scale, the economics are compelling: lower costs, better latency, full data control. The key is being honest about whether your team and your workload are ready for the operational commitment, and building the hybrid fallback architecture that keeps you covered when self hosted infrastructure has a bad day.