Introducing LLMS-DB.txt

During the development of Petals, we've been battling the tradeoff between maximising context windows and building complex retrieval augmented retrieval (RAG) systems. We're observing that as frontier models continue to improve and become more intelligent, context windows are becoming wider and more consistent.

Recently, Nick Baumann at Cline published a blog post called 'Why Cline doesn't index your codebase and why that's a good thing' which attracted a lot of attention on social media, read the full post. Nick outlined how unlike Cursor, they opt to be more selective with their retrieval - utilising large context windows over indexing your codebase locally and performing RAG against it. Given their use case, this seems a reasonable decision - particularly considering their argument for codebases not being able to be chunked in a way that effectively maintains relevant context. There is of course a tradeoff with this method, being the increased cost and token throughput - though we can expect this to decrease with time.

While the 'RAG is dead' rhetoric is becoming increasingly popular, particularly following the release of Google's Gemini 2.5 Pro (with it's 1 million token context window) we at Petals strongly believe that RAG remains an important building block for scalable, context aware AI systems. However, the question still remains - What information does an LLM need to be given, and how, to effectively produce relevant, context aware responses?

The UX / Context Problem

I've long said that the ability to utilise LLMs for certain problems in your application, strongly hinges on your ability to guide users through providing relevant context around the problem you're trying to solve. This introduces UI/UX issues - some creativity is needed to effectively gather this additional context.

This has been an issue we've faced when designing features that require a high level of accuracy. You could one-shot an LLM and pray that it returns the response you're looking for, or you could design a system that ensures the LLM has the smallest of margins (if any) for inaccuracy. There are a few parts to designing this system:

System Prompt

Tool Provision

User Interface