Tag

Cost Optimization

2 posts

pankhllm: The LLM Gateway That Learns to Skip the LLM, Without Replacing the Stack You Already Run
Data Engineering10 min read

pankhllm: The LLM Gateway That Learns to Skip the LLM, Without Replacing the Stack You Already Run

Most of an agent's LLM calls are not writing anything. They are decisions: which tool, which skill, which parameters, made thousands of times a day by a model paid in seconds and tokens. pankhllm sits where your app already calls an LLM, learns those decisions from its own traffic, and starts making them in 0.2 ms on a CPU with a 262 KB model. What it is not sure about still goes to your LLM. On the same 14 questions: a 12B planner 1,743 ms, Laya 49 ms, pankhllm's own model 4 ms, all 14 correct. Here is what it is, what it is not, and where it stops.

59 views
Read
The Real Cost of Running LLMs in Production: A Breakdown
AI & Machine Learning10 min read

The Real Cost of Running LLMs in Production: A Breakdown

Token costs are just the tip of the iceberg. After running LLM workloads in production for a year, here's where the money actually goes — and how to cut costs without cutting quality.

119 views
Read