What We Learned from Piloting an AI Knowledge Tool with Our Engineering Team

As Mereka grew, so did the places we stored knowledge. Project tasks lived in ClickUp. Policies sat in Google Drive. Meeting notes were in Notion. Structured records were in Airtable. And important decisions were buried somewhere deep in a Slack thread from three months ago.

Getting a simple answer meant opening five different tabs. So we decided to build something better. We started with the engineering team and what we found during the pilot shaped everything about how we're thinking about knowledge management going forward.

The Challenge

Our engineering team is spread around geographically, working across multiple functions and tools. On paper, we had everything documented. In practice, finding that documentation was a job in itself.

The problems were clear:

• No single place to search
  ‍Every tool was its own island. There was no way to search across ClickUp, Google Drive, Airtable, Notion, and Slack at the same time.
• Time lost to searching
  ‍Team members spent 15 to 30 minutes per question just tracking down the right document or the right person.
• Repetitive questions
  ‍The same questions kept coming up, interrupting senior engineers who had more pressing work to do.
• Painful onboarding
  ‍New team members were overwhelmed. Getting up to speed meant pestering colleagues or hoping you stumbled across the right file.
• Decisions lost in Slack
  ‍Important discussions happened in threads and were rarely found again.

The Solution: Building Reka

We built Reka, an AI-powered knowledge assistant that lives inside Slack. Ask Reka a question and it searches across all of our documentation sources simultaneously, returning an answer in under 5 seconds with links to the original source documents.

What we built:

• One search across everything
  ‍Reka connects to ClickUp, Google Drive, Airtable, Notion, and Slack so team members never have to choose which tool to look in.
• Instant answers with source links
  Every response includes citations linking back to the original document, so you can verify and dig deeper if needed.
• Specialist AI agents working together
  Behind the scenes, Reka uses a team of AI specialists: one for finding documents, one for analysing data, and one for routing requests all coordinated by a director agent. Complex questions get broken down and handled properly.
• Budget controls built in:
  The system checks available budget before responding and tracks spending in real time.
• Access controls that respect existing permissions
  Users only see documents they are already allowed to access in the original tools. Reka does not expose anything that should not be exposed.
• Always up to date
  ‍When a document changes in Google Drive or ClickUp, Reka automatically syncs the update. Answers are based on current information, not yesterday's version.

How AI helped us build it faster:

AI was not just the end product, it was part of the build process too. We used Claude for architecture decisions and complex problem-solving, and Cursor as our AI-powered code editor for refactoring and exploration. The combination accelerated development significantly.

What the Engineering Pilot Showed

We started by rolling Reka out to our engineering team. The results from that pilot were clear:

Time savings from the pilot:

• Engineering team members collectively saved 20–30 hours per month previously spent searching for information.
• 85% of engineers reported meaningful time savings since Reka was deployed to the team.
• Senior engineers spent significantly less time fielding routine questions from colleagues.

What We Learned

The pilot surfaced insights we did not anticipate going in:

Junior engineers became more independent

Before Reka, junior engineers needed a senior colleague to answer basic questions. During the pilot, they started asking Reka first. Senior engineers got their focus back, and junior engineers got answers without feeling like a burden.

Onboarding got significantly faster

New engineering team members could learn on the job by asking questions directly in Slack, instead of spending weeks navigating documentation. The knowledge was always there, they just needed to ask.

The questions revealed where our documentation was weak

Because Reka tracks what is being asked, we could see exactly where knowledge gaps existed. When the same question came up repeatedly and Reka struggled to answer it well, that was a clear signal to improve our documentation. The pilot made our knowledge base better, not just more accessible.

Source citations matter more than we expected

Every answer Reka gives includes links to the original source documents. Engineers told us this was one of the most important features, they did not want to take Reka's word for it. Being able to click through and verify built trust in the tool quickly.

What's Next

The engineering pilot has given us enough confidence to think about what comes next. The roadmap has two tracks: expanding Reka's capabilities and expanding who uses it.

The big picture:

We want Reka to evolve from a knowledge assistant piloted with engineering into a complete Knowledge Management Platform used across the whole organisation, one that does not just answer questions, but proactively identifies gaps, flags outdated content, and keeps our organisational knowledge current and complete.

The Outcome

The engineering pilot proved the concept. Finding information no longer means opening five tabs and hoping for the best, it means asking a question in Slack and getting an answer in seconds, with a link to where it came from.

What started as a solution to a knowledge-access problem has become the foundation for how we want to manage information across all of Mereka. The pilot has shown us it works. Now we scale it.

This case study covers the initial engineering team pilot of Reka. A company-wide rollout is planned across all BBI teams over the next six months.

Key Takeaways

• Scattered information is a hidden productivity tax but consolidating search saves hours every week
• Piloting with one team first surfaces real problems before a wider rollout
• Source citations are not optional, they are what makes AI answers trustworthy
• Access controls must be designed from the start, not added later
• The questions your team asks reveal exactly where your documentation needs to improve
• Faster onboarding is one of the highest-leverage outcomes of good knowledge management