Memory in AI models is primarily an issue of determining what information is relevant and how to efficiently retrieve it during a task.

“The hard part is knowing like when to pull on what piece of information to make a prediction, to generate information, to reason. And so having this ability to be a lot more selective about the all the information you've seen in context is super important.”