Batch inference costs 50% less than real-time inference at Anthropic, OpenAI and Google, in exchange...
Comparing Batch and Real-Time LLM Inference Economics
Related
My personal agent is just a folder
How I use a plain-text folder and an off-the-shelf coding agent as my personal assistant, and what I learned along the way.
Ask More of Your UI
In my last post, I wrote about moving Forem’s performance and maintenance work into autonomous loops...
The Fastest LLM Call Is The One You Never Make
At Glean, we tested a model that does not write text. It only makes decisions. In our tests, it...