Pods and Prompts

Caching is one of the easiest ways to speed up applications and control cost. But LLM based systems don’t work well with traditional caching because users phrase the same idea in many different ways. This means most queries turn into cache misses. Still, caching is important. LLM agents can take time to run, and inference … Read more