I’m trying to build a side project in the inference space. I’ve been talking to a few inference engineers and startups and I’ve been hearing how annoying it is to not have manual control over the KV cache at times and just constantly being subject to the black box caching methods of their infere...
Source: [Hacker News](https://news.ycombinator.com/item?id=49783040)