Introducing Kojak the RAG Stack
Kojak is a open-source tool to show how RAG actually works. Using open-weight models and minimal infrastructure, we want to lower the barrier to understanding and entry to these tools.
We would like to present our teaching tool Kojak - a minimal, ultra-low on-cost RAG stack deployable on AWS infrastructure.
https://github.com/palefire-io/kojak
The usefulness of being able to search internal organisational documents using vector embedding and LLMs is clear. Vector embedding allows us to directly capture and then search on closeness in meaning rather than matching on words. Providing documents retrieved this way as context to prompted LLMs provides a really natural way of finding information. There seem to be three main blockers to widespread adoption:
- technical teams need to have a genuinely comprehensive grasp on how it works to advise the business well on privacy and guardrail concerns
- it hovers tantalisingly between something you should be able to do yourself and something to outsource
- parts of it seem genuinely bonkers without a deep dive into the maths
Sometimes it helps to be able to poke at something basic, tinker with it and break it to understand how it works better; hence this repository.
It's a functional but extremely minimal RAG implementation on AWS Bedrock + Google Gemma. It will deploy the following:
- Postgres vector database
- Python orchestration
- TypeScript UI
- two ingest workflows
- sample data
Spinning it up in your account will take about 20-30 minutes and cost you about £3 a day to idle (it needs a micro-sized Postgres instance, some Docker containers and some network resources). Gemma tokens are really cheap and cheerful; dozens to hundreds of times cheaper per token compared with the big models we are used to, see the Bedrock pricing page. It's simple to spin up and tear down as needed.
Please note, this is just a toy implementation with a minimal UI and orchestration, an ultra-low-cost model and, most importantly, no chunking (look out for that in a later release) and it will only ingest plain-text but it’s pretty usable and a helpful learning tool.
Install and deploy, run the ingest commands to load the sample data and either log into the UI, run the database connect command or peruse the source to see how the magic happens (search for the cosine similarity operator <=> to get straight to the heart of it).
The demo login passwords are shared and the UI is internet facing so don't load real data into it.
Have fun with it and feel free to raise questions or issues via GitHub