Forward Deployed Engineer, Private LLM Inference
Stand up self hosted and private LLM inference inside client infrastructure, from model selection and serving stack to the agents that run on it, and hand it over running.
- Python
- vLLM
- MLX
- Kubernetes
- CUDA
- Terraform
Some clients cannot send data to a hosted model API. Regulated data, contractual restrictions, or plain cost at volume push them toward running open weight models on their own hardware or in their own cloud tenancy. This role goes on site, figuratively or literally, and makes that work.
You choose the model and quantization for the workload, stand up the serving layer (vLLM, SGLang, or MLX on Apple silicon for smaller deployments), wire it to the client agents and retrieval pipeline, and prove the latency, throughput, and quality numbers against their real traffic before anyone calls it done.
Stand up the serving layer
- Select and benchmark open weight models (Llama, Qwen, DeepSeek, Mistral, Gemma) against the actual client tasks, and document the quality trade offs of each quantization honestly
- Deploy inference on the client GPUs or cloud with vLLM, SGLang, TGI, or MLX, with KV cache sizing, batching, and prefill behavior tuned to the traffic shape
- Build the gateway layer: OpenAI compatible endpoints, auth, per tenant quotas, and routing between local and hosted models when a task needs a bigger model
Make the agents run on it
- Port the client prompts, tool schemas, and retrieval pipeline to the local model and close the quality gap with evals, not vibes
- Run structured output and tool calling reliably on smaller models, including the fallback paths for when the model fails to comply
- Measure time to first token, tokens per second, and cost per task under sustained load, and publish the numbers to the client
Hand it over
- Write the capacity plan, upgrade path, and model swap procedure the client platform team will follow
- Set up monitoring for the things that fail silently: quality drift, cache thrash, and GPU memory pressure
- Pair with the client engineers until they own the stack
Required
- Five or more years of production engineering with meaningful Python depth, plus enough systems knowledge to reason about GPU memory and network throughput
- You have deployed and operated at least one LLM serving stack (vLLM, SGLang, TGI, llama.cpp, or MLX) under real traffic
- Working knowledge of quantization, context length trade offs, and what actually changes when you drop from a frontier API to a 30B local model
- Comfortable in client infrastructure: Kubernetes, Terraform, and whichever cloud they already run
- You write benchmark reports that state the method, the caveats, and the numbers you did not like
Helpful, not required
- Experience with RAG pipelines, rerankers, and embedding models running locally
- Apple silicon inference with MLX and unified memory sizing for edge or workstation deployments
- Prior forward deployed, MLOps, or platform engineering role
Also hiring.
Engineering Lead and Client Partner
Remote Delivery
A player coach seat that combines technical lead, delivery owner, and second seat on sales.
Forward Deployed Engineer, Agentic Commerce
Remote Applied AI
Embed with commerce clients and ship agent driven discovery, catalog, and checkout systems into production on their stack, not ours.
Forward Deployed Engineer, Enterprise AI Integrations
Remote Applied AI
Wire LLM agents into the ERPs, CRMs, and ticketing systems enterprises already run, inside the client environment, and make the integrations survive production.
Show us something you built.
Send a resume or profile with a short note about work you have shipped. Links to code, systems, or launched projects tell us more than a cover letter ever will.
careers@contracollective.com