[UofT] Beyond Baseline RAG: Building a Reliable and Transparent Policy Chatbot for Government Guidance

Government procurement policies are often complex, lengthy, and distributed across multiple directives and guidance documents. Employees seeking clarification must manually search through these materials, which can be time-consuming and may lead to inconsistent interpretations. In this talk, we present the design of a retrieval-augmented policy chatbot that assists users in navigating procurement policies by answering questions directly from source documents while providing transparent citations.

Our system uses a Retrieval-Augmented Generation (RAG) architecture to ground responses in official policy text. Documents are segmented and embedded into a retrieval index, allowing the system to surface relevant policy excerpts in response to natural language queries. The language model then generates answers strictly from the retrieved evidence and provides citations so users can verify the source material.

Beyond a baseline RAG implementation, the project explores several mechanisms to improve reliability and transparency. These include structure-aware document chunking, detection of contradictory or overlapping policy statements, and a self-verification loop that checks generated answers for unsupported claims. The system also incorporates user feedback signals and an evaluation pipeline that measures retrieval accuracy, evidence grounding, and response quality.

Together, these components aim to demonstrate how AI assistants can support policy interpretation while maintaining transparency, accountability, and user trust.