From the Lab Page 27 / 46

Insights, engineered.

Technical deep-dives, AI strategy, and engineering perspectives from the team building autonomous systems.

Jun 28, 2026 Headless Commerce

Salsify vs Akeneo vs Inriver for Shopify Plus Headless Catalog at 250K SKUs: TCO and Sync Architecture (2026)

A Shopify Plus headless storefront at 250K active SKUs cannot run on Shopify's native product admin alone; the catalog complexity, attribute governance, and multi channel distribution work moves to a dedicated PIM. Salsify, Akeneo, and Inriver are the three credible options for an enterprise headless build, and the right one depends on sync architecture, governance fit, and a 3 year TCO that the license sticker rarely captures.

Jun 27, 2026 Headless Commerce

Algolia vs Typesense vs Meilisearch for Shopify Plus Headless Search: TCO and Operational Ownership at 100K Plus SKUs (2026)

Shopify Plus brands at 100K SKUs and above face a search vendor decision where the headline price hides most of the cost. We modeled the 3 year total cost of ownership for Algolia, Typesense, and Meilisearch including engineering ownership, indexing pipeline maintenance, and the operational burden each path imposes on the team, and the procurement winner is rarely the cheapest license.

Jun 27, 2026 AI Infrastructure

BGE Reranker v2 vs Cohere Rerank 3 vs Qwen3 Reranker on M5 Max: Latency, Recall, and Cost for Local RAG (June 2026)

Rerankers are the second stage of any serious RAG pipeline and the stage that decides whether retrieval quality lifts the final answer or drags it down. We measured BGE reranker v2, Cohere Rerank 3, and the Qwen3 reranker on a 128GB M5 Max running MLX for latency per 100 candidates, NDCG at 10, recall lift over a cosine baseline, and the cost per million queries that decides whether reranking belongs on device or behind a hosted API.

Jun 27, 2026 AI Models

Claude Haiku 4.5 vs Fable 5 Mini vs Gemini 3.1 Flash on Terminal-Bench 2: Cheap Tier Agentic Coding Tested (June 2026)

Terminal-Bench 2 is the closest public benchmark to a working terminal agent because the tasks require multi step shell sessions, real file system state, and the model has to recover from its own mistakes. Haiku 4.5, Fable 5 Mini, and the newly released Gemini 3.1 Flash all compete for the cheap tier routing slot. The headline pass rate is one variable; tool call accuracy and cost per resolved task decide which one belongs at the front of a production agent fleet.

Jun 27, 2026 AI Models

Cursor vs Windsurf vs Zed on Aider Polyglot: Agentic IDE Benchmark for Multi-File Refactor Tasks (June 2026)

Aider Polyglot is the closest public benchmark to a working agentic IDE workflow because the tasks require whole file rewrites across six languages, not just patch generation. Cursor, Windsurf, and Zed all ship the same frontier models under the hood, and the gap that decides which one belongs on a production team is the edit format, the harness, and the cost per resolved task.

Jun 27, 2026 Headless Commerce

Klaviyo vs Iterable vs Customer.io for Shopify Plus Headless Lifecycle Messaging: TCO and Event Pipeline Ownership (2026)

Shopify Plus brands going headless face a lifecycle messaging vendor decision where the headline price hides most of the cost. We modeled the 3 year total cost of ownership for Klaviyo, Iterable, and Customer.io at 500K active profiles including event pipeline engineering, segmentation engine fit, and the operational burden each platform imposes, and the procurement winner is rarely the cheapest license.

Jun 27, 2026 AI Infrastructure

Fine-Tuning Llama 3.3 70B with LoRA on M5 Ultra Mac Studio: Throughput, VRAM, and Convergence Tested (June 2026)

Llama 3.3 70B is the largest open weight model that fine tunes cleanly on a single Mac Studio without leaving unified memory. We measured LoRA training throughput, peak VRAM, convergence behavior, and the wall clock per epoch on a 256GB M5 Ultra against a realistic 50K example instruction tuning dataset, and the result reshapes the build versus rent decision for teams that have been defaulting to cloud A100 boxes.

Jun 26, 2026 AI Models

Gemini 3.5 Pro vs Claude Opus 4.8 on Terminal-Bench 2: Agentic Coding Tested (June 2026)

Terminal-Bench 2 is the closest public benchmark to a real shell native coding agent because the tasks span environment setup, debugging, and multi file edits, all driven through a real terminal. Gemini 3.5 Pro and Opus 4.8 both clear the 60 percent resolve threshold, and the gap that decides which one belongs in a production terminal agent is not the headline pass rate.

Jun 26, 2026 AI Infrastructure

Qwen3-Coder 30B Local Agentic Loop on M5 Max: Prefill, Decode, and Tool-Call Throughput Tested (June 2026)

Qwen3-Coder 30B is the first open weight coder model that survives a real agentic loop on a single M5 Max without the latency collapsing on every tool call. We measured prefill, decode, and tool-call cycle time across MLX and llama.cpp with realistic agent transcripts, and the gap between a synthetic decode benchmark and a working agent loop is wider than most teams plan for.

Jun 26, 2026 Headless Commerce

Shopify OMS vs Manhattan Active Omni vs Fluent Commerce for Enterprise Headless Order Management (2026)

Order management is the layer where a headless storefront either delivers on its promise or breaks at the moment of fulfillment. Shopify OMS, Manhattan Active Omni, and Fluent Commerce sit at three very different points on the sourcing complexity, integration depth, and unit cost curve. The wrong pick produces either a fulfillment team locked out of the sourcing rules or an engineering team owning every store cluster change.

Jun 25, 2026 AI Infrastructure

Draft Model Selection for Speculative Decoding on Apple Silicon: Llama 3.2 1B vs Qwen3 0.5B vs Custom Distill on M5 Max (June 2026)

Speculative decoding on Apple Silicon collapses or doubles throughput depending entirely on the draft model. The wrong draft burns the prefill and decode savings on rejected tokens. We measured Llama 3.2 1B, Qwen3 0.5B, and a custom distilled draft against Llama 3.3 70B and Qwen3 32B on M5 Max to map acceptance rate, end to end speedup, and the engineering cost of each path.

Jun 25, 2026 AI Models

Claude Opus 4.8 vs Sonnet 4.6 on SWE-Bench Verified: Cost Per Resolved Issue Tested (June 2026)

SWE-Bench Verified is the closest public benchmark to a real coding agent workload because the issues come from production repos, the harness runs real test suites, and the resolve rate maps cleanly to engineering time saved. Opus 4.8 and Sonnet 4.6 both clear the 70 percent threshold, and the cost per resolved issue gap decides which one belongs in the production stack.