Skip to content

Helix — Cost Optimization

Status: Draft v1 · Last updated: 2026-06-18 · Related: TSD §8 · ADR-006 · ADR-007

Goal: the default configuration costs $0 and ≥ 90% of active users never incur API spend (PRD §9), without crippling the product. Cost is treated as an architectural invariant, not a tuning afterthought.

1. Where cost would come from (and how we remove it)

Cost source Naive approach Helix approach Result
Embeddings (highest volume) Cloud embedding per memory Local fastembed bge-small $0, offline
Fact extraction LLM call per message Heuristic gate drops most; LLM only when needed ~0 calls for most turns
Which LLM, when used Premium model always Gemini 2.0 Flash free tier → gpt-4o-mini fallback free-tier-first
Repeated/identical work Re-call every time Response cache keyed by input hash pay once
Many small calls One call per turn Batch buffered turns into one call fewer, bigger, cheaper calls
Verbose prompts/outputs Free-form text Structured JSON + compact prompts minimal tokens
Runaway spend Unbounded Token budget guardrail (0 = no paid calls) hard ceiling

2. The pipeline (cost view)

 slice
   │
   ▼
 [Redaction]                         local, $0
   │
   ▼
 [Heuristic gate] ── "no durable fact" ─────────────► DROP   (most slices end here, $0)
   │  "maybe a fact"
   ▼
 [Cache lookup] ── hit ─────────────────────────────► reuse  ($0)
   │ miss
   ▼
 [Extractor]
   ├─ no key            → deterministic (rules+embeddings)    $0
   ├─ Ollama present    → local LLM                            $0
   ├─ Gemini key        → Gemini 2.0 Flash (free tier)         ~$0
   └─ OpenAI fallback   → gpt-4o-mini                          tiny $, only if needed
   │
   ▼
 [Embed] → local by default                                   $0

The two biggest levers are at the top: local embeddings make the highest-volume operation free, and the heuristic gate ensures the LLM is rarely invoked at all.

3. The heuristic gate (primary lever)

Before any model runs, a cheap local check estimates "is there a durable, novel fact here?" Signals:

  • Cues: "remember", "always/never", "I prefer", "we decided", "the convention is".
  • Structure: presence of entities, decisions, imperatives; slice length/type.
  • Novelty: embedding distance to the nearest existing memory — if it's basically a duplicate, there's nothing to learn.

If the gate isn't confident there's a fact (< HELIX_HEURISTIC_CONFIDENCE_CUTOFF), the slice is dropped with zero model calls. Most conversation turns don't contain durable new facts, so most are dropped here — which is what keeps default cost at ~$0 even with an LLM enabled.

4. Free-tier-first LLM router

Policy (LiteLLM under the hood, ADR-007):

  1. If HELIX_LLM_PROVIDER=none or no key → deterministic extractor (always $0).
  2. Else prefer Gemini 2.0 Flash (free tier) for extraction/consolidation.
  3. On rate-limit/unavailable, or if the user prefers it → gpt-4o-mini.
  4. On any failure → fall back to deterministic (degrade quality, never lose data or block).

Every call is cached (hash of prompt + inputs), batched where possible, and emitted as structured JSON to minimize tokens.

5. Guardrails

  • HELIX_MONTHLY_TOKEN_BUDGET — hard ceiling on paid tokens; 0 disables paid calls entirely (free tier + local only). When the ceiling is hit, the router transparently degrades to the deterministic path.
  • Cost telemetry (local): the dashboard shows calls, tokens, estimated spend, and gate-drop rate, so a user can see they're at $0.
  • No surprise upgrades: raising default cost requires an ADR (CLAUDE.md rule 3).

6. Quality ladder (you choose where to sit)

Tier Setup Cost Extraction quality
Floor nothing (default) $0, offline good — heuristics + rules + embeddings
Local LLM install Ollama $0, offline better — local model extraction
Free cloud Gemini free key ~$0 great — frontier-ish quality
Paid fallback OpenAI key tiny great — only when free tier is exhausted

The product is genuinely useful at the floor; keys only raise quality. Free should feel first-class, not like a teaser (PRD §8).

7. What we deliberately avoid

  • Cloud embeddings by default (the silent budget-killer).
  • An LLM call per message.
  • Storing/Re-embedding things that didn't change (decay + content hashing avoid churn).
  • Premium models for routine extraction.

8. Validation

Cost integrity is a tracked metric (PRD §9): ≥ 90% of active users on the $0 path. CI includes the no-key/offline configuration as a first-class test target so the free path can never silently regress.