We've been thinking about this for a while now. Every couple months some LLM provider puts out a press release — "Now supporting 1 million tokens!" Then 2 million. We're pretty sure we'll see 5 million token context windows by end of year.

Let's start with a simple experiment: go to https://chatjimmy.ai/ and ask a random question, put the quality of the answers aside, let's observe how fast this website spits out answers. It feels as if they just spit out the entire paragraph in one-go compared