LIVE

PostHog Implements Decision Models to Slash Compute Costs and Latency

🔌 Chips & Hardware 2 min read 👁 1

The developer platform PostHog has integrated specialized decision models to act as gatekeepers for expensive LLMs, reducing hardware demand and operational costs.

PostHog Implements Decision Models to Slash Compute Costs and Latency🔌Chips & Hardware

Photo: Hacker News Front Page

Key points

  • Decision models demonstrated speeds 193.6x faster than frontier LLMs.
  • PostHog reduced LLM calls by 98.7% in its PII scrubbing pipeline.
  • Small models are being used as 'routers' to select the cheapest hardware-efficient path for user queries.

PostHog has successfully deployed specialized decision models in its production environment to optimize how it utilizes high-end AI compute resources. By using these smaller, probability-based models to filter tasks before they reach frontier models like Gemini, the company has reported significant gains in efficiency. Data from their implementation shows that these decision models are up to 444.6 times cheaper and 193.6 times faster than traditional large language models (LLMs).

Optimizing Compute Pipelines

The primary role of these models is to serve as a high-speed filter for data processing. In one specific application, PostHog uses a decision model to scan session summaries for personal identifiable information (PII). By setting a low probability threshold of 0.4, the system identifies potential privacy risks and only escalates those specific cases to an LLM for rewriting. This architecture allows 98.7% of all sessions to bypass expensive LLM calls entirely, drastically reducing the load on data center resources.

Model Routing and Calibration

Beyond simple filtering, the company utilizes decision models as intelligent routers. When a user interacts with their Slack bot, a decision model evaluates the request and selects the most efficient model for the job—such as the lightweight GPT Luna for simple summaries—rather than defaulting to a more compute-heavy alternative. This ensures that hardware is only utilized at the level of intensity required by the specific task.

Maintaining this efficiency requires constant recalibration. PostHog engineers noted that probability thresholds do not remain consistent across different model versions. To manage this, they developed an internal AI gateway to test and calibrate models in a "shadow mode" before they are fully integrated into the production pipeline. This approach allows the team to balance the risk of false negatives against the cost of unnecessary compute cycles.

What to watch next: Whether the industry sees a broader shift toward "small model first" architectures to mitigate the high costs of GPU-intensive generative AI.

Source Hacker News Front Page — newsletter.posthog.com/p/decision-models ↗

Contact the newsroom

Got a tip, a proposal or a partnership question? Write to us and we will reply shortly.