Building an AI-Ready Knowledge System with RAG on Azure

• Multi-format data ingestion. The system takes in source materials of different types, including PDF, Word, Excel, and Markdown, and brings them into one unified knowledge flow.
• Cleaning and structuring. Raw content is cleaned of noise, duplicates, and formatting artifacts, then normalized into a consistent structure. This step turns documents written for humans into knowledge that AI can use reliably.
• Smart chunking and indexing. Content is split into meaningful chunks, converted into semantic vectors, and indexed in Azure AI Search. Hybrid search combines keyword precision with meaning-based retrieval, so the system finds the right information whether a question uses exact terms or everyday language.
• Self-updating knowledge base. When the team uploads new or updated documents, the pipeline processes and re-indexes them automatically. No retraining, no developers, no delay.

Most AI projects stop at “it seems to work.” We validated the knowledge system with the devPulse AI Assessment Framework, our in-house methodology for proving the quality of AI products before they reach users.

• Tests generated from the knowledge itself. A generation script reads the knowledge base and automatically creates test questions mapped to specific sections, so testing covers the full breadth of the content, not just a handful of easy questions.
• Four difficulty tiers. Questions range from direct ones with a single clear source, to casual paraphrases with typos and slang, to complex questions that require combining several sources, and finally adversarial and out-of-scope attempts designed to make the AI invent answers or reveal what it shouldn’t.
• Five quality dimensions. Every answer is scored automatically for accuracy, groundedness (no hallucinations), honest “I don’t know” behavior, completeness, and tone and clarity, using the original source as the reference.
• Multi-turn scenarios. Beyond single questions, the framework tests full conversations: whether the AI keeps context, accepts corrections, stays consistent, and resists gradual manipulation across several messages.
• Automation with human oversight. Scripts handle generation, execution, and scoring, while experts review every flagged answer. When the knowledge base changes, the affected sections are re-tested automatically, catching regressions before users do.

The results. Across 74 test scenarios, the system achieved 99% overall accuracy and zero critical fabrications. It answered 100% of direct questions correctly, resisted 100% of adversarial and prompt-injection attempts, and passed 97% of multi-turn conversation scenarios, with an average completeness and tone score of 4.94 out of 5. No critical or high-severity defects were found.

The first way to access the knowledge system is an AI chatbot on the client’s website, acting as an always-available product expert for prospects.

• Grounded, expert answers. GPT-5-mini generates clear answers based only on retrieved knowledge, and gives a fallback instead of guessing when the answer isn’t there.
• Smart escalation. The chatbot detects unanswered questions and signs of frustration, and checks in when a conversation goes quiet. When needed, it emails the request to the team with the full conversation history, delivering a sales-ready lead.
• Embeddable anywhere. A single lightweight, branded script renders in an isolated Shadow DOM and can be added to any page with a simple snippet.
• Flexible logging. This client chose a privacy-first setup with diagnostic logging available via a feature flag, while the same architecture supports full conversation analytics when needed.
• Cost and traffic control. Token limits, rate limiting, and bot protection in Azure API Management keep AI spending proportional to real traffic and within budget.

The knowledge system sits behind a shared infrastructure layer built on Azure Front Door and Azure API Management, designed to serve far more than a single channel.

• New access channels. The same knowledge can power internal assistants for sales and support teams, guided self-service flows for pricing, registration, subscriptions, and API keys, or direct API access for other applications.
• The client’s existing solutions. Current services can be connected to the same Front Door and APIM setup, gaining consistent security, monitoring, and access management.
• The future API platform. APIM natively provides a developer portal, subscription keys, usage plans, per-customer quotas, and analytics. When the client is ready to open its services to customers, the foundation is already in place.