LLM-on-Spark: Four Patterns That Actually Scale
"Just call the LLM in a loop." 9.6 years later, you finish. Here are the 4 patterns that actually scale to a billion rows: Spark UDFs, Ray+vLLM, warehouse-native SQL, or the Batch API. Code + costs.
Blog
Learning in the age of AI
AI already knows more than you ever will. That’s not the advantage anymore. Your edge is simple: ask better questions, get better answers.
How LLM applications learned to remember
We went from 4K token context windows to virtual memory filesystems in four years. Here's the engineering story of how LLM memory evolved - and what you should actually use today.
The hard part of AI engineering isn't the AI
I run a 19-node LangGraph pipeline serving 20000+ users. I've never written a PyTorch training loop for it. Here's what actually matters - and a 24-week roadmap built around it.
Your AI agent can't use your software. Here's how that's changing.
Tools gave agents hands. MCP standardized the wiring. CLIs were there all along. But none of them taught agents how to think about a task. The missing layer turned out to be a markdown file.
What Happens When You Let an AI Rewrite Its Own Instructions?
Most of us are stuck on the prompt treadmill - manually tweaking instructions that break every time the task shifts. This post lays out an architecture where the AI agent grades its own work, rewrites its own prompts, builds its own tools, and rolls back when things get worse. Every idea is backed by published research. No jargon, just the blueprint.
The Complete Guide to 17 Agentic Reasoning & Planning Algorithms
A practical deep-dive into the algorithms powering modern AI agents - from Chain-of-Thought to automated workflow discovery. Each algorithm is explained with flow diagrams, simple examples, and Python pseudocode
Coding Is a Commodity. Now What?
How the rise of AI-assisted development is redefining what it means to be a good developer.
What If Your AI Agents Could Find Each Other?
You're copying the same agents into every new workflow. There's a better way. A self-organizing architecture with RAG-based discovery, reputation scores, budget-aware planning, and dynamic composition that solves problems nobody anticipated.
What If Your AI Agents Could Find Each Other?
You're copying the same agents into every new workflow. There's a better way. A self-organizing architecture with RAG-based discovery, reputation scores, budget-aware planning, and dynamic composition that solves problems nobody anticipated.