Choosing Your First AI Infra Stack: A Founder's Field Guide for 2026

In May 2026, a founder I worked with shipped a production agent on Claude Sonnet 4.7 with Postgres and pgvector, observability through Helicone, and a single Hetzner box. It cost $87 a month at launch and supported the first 4,000 users without flinching. Across town, another team had spent six months on a custom multi-cluster Kubernetes setup with Pinecone, Weaviate, LangGraph, and a homegrown evaluation harness. They had not yet shipped.

The difference was not budget or talent. It was that the first founder had treated AI infrastructure as a build-the-minimum-to-learn problem, and the second had treated it as an architecture problem before they had any users.

This guide is the opinionated version of the conversation I have with founders ten times a month: what to actually pick when you are starting out, what is genuinely worth the upgrade later, and which trendy pieces to skip until you have evidence you need them.

The Five Layers You Actually Need

A first AI stack has five layers. Anything else is premature.

1. Model access (the LLMs themselves)
2. A model gateway (rate limiting, fallback, key management)
3. Retrieval (vector storage and search)
4. Observability (traces, costs, evaluations)
5. Orchestration (the code that wires it together)

You can ignore agent frameworks, fine-tuning infrastructure, prompt management platforms, and synthetic data tooling for now. They are real categories. They are not your week-one problem.

Layer 1: Model Access

Pick three models and route between them by task. In May 2026 the pragmatic default is:

| Task class | First choice | Fallback | Why | |------------|--------------|----------|-----| | Hard reasoning, code, agents | Claude Opus 4.7 or Sonnet 4.7 | GPT-5 | Strongest tool-use reliability | | Cheap classification, extraction | GPT-5 mini | Gemini 2.5 Flash | Cost per 1M tokens under $0.60 | | Long-context summarization | Gemini 2.5 Flash | Claude Sonnet 4.7 | 2M context, low cost | | Embedding | OpenAI text-embedding-3-large | Voyage-3 | Quality is good enough; vendor lock is real but limited |

Do not start with open-source models unless you have a specific compliance or cost reason. The premium for hosted frontier models in 2026 is small relative to engineering time, and the quality gap on hard tasks is still real.

Layer 2: The Model Gateway

This is the single most underrated piece of the stack. A gateway gives you four things you will need within the first quarter: vendor fallback when a provider goes down, rate limiting per customer, caching, and a unified billing view.

Two clear choices:

• OpenRouter for the lowest friction (one API key, every model)
• Portkey for self-hosted control with the same interface

Both let you swap models without touching application code. Both expose prompt caching for Anthropic and OpenAI, which can cut your bill by 30-60% on workloads with stable system prompts. Either is worth the integration day.

Layer 3: Retrieval

The default 2026 advice is unchanged from 2024 and will probably be unchanged in 2027: start with Postgres and pgvector. It will get you to several million vectors, sub-100ms p99 query latency, and a single database for both your operational data and your embeddings. That last property is enormously underrated.

You move off it only when you hit one of these walls:

• Vector count above 50 million
• Query latency p99 above 200ms with appropriate indexes
• Hybrid search ergonomics (BM25 + vector) become painful

When that happens, the modern picks are Turbopuffer (cheap, S3-backed, surprisingly fast), Qdrant (mature, great filtering), or LanceDB (embedded, ideal if you control deployment). Pinecone is still a solid managed option but is no longer the obvious leader it was in 2023.

Layer 4: Observability

You cannot ship AI features in production without observability. The bug shapes are too weird and the costs too easy to spike. The 2026 baseline:

• Helicone or Langfuse for traces and cost dashboards
• A simple eval harness in your test suite (LLM-as-judge with a frozen reference model)
• Per-tenant spend caps wired to your gateway

If you skip evaluations, you will ship regressions silently. The discipline does not require Braintrust or PromptLayer or any vendor. It requires a directory of prompts, golden outputs, and a script that runs nightly. Build it on day one even if it only has ten test cases.

Layer 5: Orchestration

This is where founders most often overbuild. The honest path:

1. Start with plain TypeScript or Python functions calling the gateway
2. Add a state machine when you have a workflow with more than three steps
3. Adopt the Claude Agent SDK or LangGraph when you have a real multi-agent system

The Claude Agent SDK and Anthropic's Skills ecosystem changed the calculus in late 2025. For tool-use-heavy workflows, the SDK gives you durable execution, transparent tool routing, and a meaningfully shorter path to working agents than rolling your own. If the work fits the SDK's shape, it is now the default.

What to Defer

Things you do not need on day one, despite what conference talks suggest:

• A dedicated prompt-management platform (a folder of versioned files works)
• Fine-tuning infrastructure (you cannot fine-tune your way out of a bad retrieval setup)
• A synthetic-data pipeline (you do not have enough real data to know what is missing)
• Multi-region inference (latency matters less than your eval scores)
• Self-hosted models (the moment you own a GPU, you own a fleet)

A Concrete Reference Stack

For a founder shipping their first AI feature this month with under $300 in monthly fixed costs:

• Vercel or Hetzner for the application layer
• Neon Postgres with pgvector for data and retrieval
• OpenRouter for model access (Claude Sonnet 4.7, GPT-5 mini, Gemini 2.5 Flash)
• Helicone for observability
• A nightly GitHub Action running a 30-prompt eval suite
• All orchestration in a TypeScript module called ai/ in the application repo

This stack scales to roughly $50k MRR and 10,000 active users without rearchitecture. The companies you read about that "rebuilt their AI infrastructure" did so after that point, not before.

When to Upgrade

Three signals that it is time to invest more:

1. Your engineers spend more than a quarter of their time on AI plumbing rather than features
2. Your model bill exceeds your infra bill by 5x and is growing faster than revenue
3. A real customer asks for guarantees you cannot give on the current stack (data residency, SLAs, custom evals)

Until you hit one of those, the stack above is what you need. Adding more is fashion.

The Takeaway

The right first AI stack in 2026 looks boring. A model gateway. Postgres. Helicone. A handful of prompts in version control. The teams that win are not the ones with the most sophisticated infrastructure on day one — they are the ones who shipped something users wanted while their competitors were still picking a vector database.

A Walk-Through: Building the First Feature

To make this concrete, here is the exact build sequence I recommend for a founder shipping their first AI feature in May 2026 — start to first user in roughly two weeks.

Week 1: The Skeleton

Day one, sign up for OpenRouter and Helicone. Both onboarding flows take under thirty minutes combined. Wire OpenRouter as your model client and Helicone as a proxy in front of it. You now have model access, fallback, observability, and a unified billing dashboard before you have written any application code.

Day two, provision a Neon Postgres instance and enable the pgvector extension. The exact migration is two lines: CREATE EXTENSION vector; and a table with a vector(1536) column. Add an HNSW index when you cross roughly 100,000 rows; not before.

Day three through five, build the actual feature. Keep all AI logic in a single ai/ directory in your repository. Resist the urge to extract it into a service; the operational overhead does not pay off until you have multiple consuming surfaces. Functions like embed(text), retrieve(query, k), and generate(prompt, context) are enough to start.

Week 2: The Discipline

Day six, write your first ten evaluation cases. They do not need to be sophisticated. Pick ten queries that should return specific answers, run them on every commit, and fail the build when more than two regress. This single discipline is the difference between teams that ship reliably and teams that surprise themselves at 2 a.m.

Day seven through ten, polish. Add per-tenant rate limits in your gateway. Cache stable system prompts. Wire up cost alerts in Helicone. Confirm that your fallback path works by deliberately killing your primary model.

By day fourteen you should have a feature in front of users, a rough cost-per-active-user number, an evaluation harness running nightly, and a clear list of which prompts and which routing decisions are responsible for the bulk of your spend. That is the foundation. Everything you build on top of it is faster because the foundation is small.

Common Failure Patterns

Three specific traps I see repeatedly with first-time AI founders.

Optimizing the Wrong Layer

Founders frequently optimize their model choice when the problem is their retrieval. They spend a week comparing Claude Sonnet 4.7 against GPT-5 on their workload, find a 4% difference, and ship neither — when the actual fix was that their pgvector index was missing.

The diagnostic order should always be: data quality first, retrieval second, prompts third, model fourth. The model is the cheapest variable to swap and the one founders reach for first.

Treating Evals as a Phase Two Problem

A surprising number of teams ship without any evaluation harness, then spend the first quarter after launch debugging silent regressions. By the time you notice that quality dropped two weeks ago, you have two weeks of customer pain to recover from. Evals built on day one cost almost nothing and prevent this entire failure mode.

Multi-Cloud Before Single-Cloud Works

Almost no early-stage company actually needs multi-cloud AI infrastructure. The teams that build it from day one are signaling a level of operational maturity they have not yet achieved. The right time to think about multi-region or multi-cloud is when a real customer asks, with real money, for guarantees you cannot otherwise meet.

Related Reading

• agentic AI production lessons — What breaks first when agents hit real workloads.
• the cost curve behind AI agents — Token math and how it ruins margins if you ignore it.
• private inference and data boundaries — When your stack has to keep customer data off vendor servers.