AI RAG Application Development

Description

We have about 4,000 internal policy documents in SharePoint — PDFs, Word files, a few scanned and never OCR'd. Staff cannot find anything, so they ask the compliance team, and the compliance team answers the same questions repeatedly. We want retrieval-augmented generation over that corpus, and we are specific about one thing: every answer must cite the document and section it came from, and must refuse when the corpus does not cover the question. An unsourced answer about a compliance policy is worse than no answer, because someone will act on it. Expect real work on ingestion — the scanned files need OCR, the Word files have inconsistent heading structure, and document access permissions in SharePoint must be respected in retrieval. A user must not be able to retrieve a passage from a document they cannot open.

• Build the ingestion pipeline: SharePoint sync, OCR for scanned files, chunking that respects document structure
• Carry SharePoint access permissions through into retrieval so nobody sees passages from documents they cannot open
• Make every answer cite its source document and section, and refuse when retrieval returns nothing relevant
• Set up an evaluation set of real compliance questions and measure answer accuracy against it before launch
• Handle re-indexing when documents change, without a full rebuild each time

• RAG application with a search interface, deployed on our Azure tenant
• Ingestion pipeline with a scheduled SharePoint sync and OCR step
• Evaluation set and a written accuracy report against it
• Documentation of the chunking and retrieval design, so we can tune it later