Quality Monitoring and Tuning

At rapidflare.ai, we are committed to maintaining the highest standards of performance and accuracy in our enterprise AI Agent. Our comprehensive quality reports provide valuable insights, enabling both our internal teams and customers to track performance and ensure continuous improvement.

Types of Quality Reports

We offer two primary types of quality reports to assess and enhance the AI Agent's performance:

• Initial QA Report: Conducted during the onboarding phase by our customer success team, this report helps assess and fine-tune the AI Agent's responses before full deployment, ensuring it aligns with customer-specific needs.

• Ongoing QA Report: After deployment, this report captures, analyzes, and resolves customer feedback. It is essential for maintaining the AI Agent's accuracy and relevance over time. The ongoing QA report features a pie chart that highlights the percentage of issues across various "Issue RCA" (Root Cause Analysis) categories, including missing information, glossary gaps, analytical errors, and poor references.

Key Performance Metrics

Our quality reports measure several key performance indicators (KPIs) to ensure the AI Agent's effectiveness:

• Correctness of Answers: Evaluating the accuracy of responses across various query categories.
• Correctness of References: Ensuring the right sources and references are used for each query.
• Number of Conversations: Tracking the volume of interactions to monitor overall engagement.
• Customer Feedback: Collecting both positive and negative feedback to drive improvements.
• Issue RCA: Categorizing issues into root causes such as missing information, glossary mismatches, analytical errors, and poor references. This analysis allows for targeted improvements to the AI Agent's responses.

Each instance of negative feedback is recorded in the report, with a status field indicating the progress of issue resolution, ensuring transparency and accountability in addressing customer concerns.

Automated Answer Scoring

The reports above are prepared by our customer success team. Alongside them, every AI Agent answer is scored automatically, so regressions surface between reports rather than at the end of a review cycle.

Each answer is graded by a language model on three independent dimensions, each on a 0 to 100% scale:

• Faithfulness: are the factual claims in the answer grounded in the sources the agent retrieved, rather than invented.
• Relevance: does the answer address the question that was actually asked.
• Completeness: does the answer cover every aspect of the question, not just the easiest part.

The three are kept separate because they fail independently. An answer can be perfectly grounded yet miss half the question, and an answer can be thorough yet drift off topic. A single blended number would hide both cases, so an answer is flagged for review when any one dimension falls below 50%, not when the average does.

Two behaviours are deliberately excluded from being counted as failures:

• Clarifying questions: when the agent responds with a multiple-choice question instead of an answer, it is asking the user to narrow an ambiguous request. That is the correct action, not a missing answer, so those turns are recorded as answered. They are identified from the interaction itself rather than graded, so the classification is exact.
• Answers with no factual claims: a reply that makes no claims cannot be unfaithful, so a high faithfulness score there is meaningless. These are tracked separately instead of inflating the results.

Scores are computed per answer and retained alongside the conversation, which lets the Rapidflare team compare periods, isolate which dimension moved, and trace a drop back to the specific conversations behind it. Findings feed the ongoing QA report and the tuning work described above.

Report Frequency

We provide quality reports on a weekly and monthly basis, allowing customers to regularly monitor the AI Agent's performance. Additional reports can be delivered on demand, ensuring flexibility and real-time updates.

Report Format

Our reports are presented in Excel format, with organized data and visual elements. Pie charts are used to display key performance metrics, including the percentage of correct answers and a breakdown of issues based on Issue RCA categories. This visual representation offers customers a clear and quick overview of the AI Agent's performance trends.

Actionable Insights

Both the initial and ongoing QA reports include actionable insights, with comments and queries from our customer success team. Customers are encouraged to engage with these insights, helping us resolve issues such as missing documentation or incorrect references. This collaborative feedback loop ensures that the AI Agent continues to evolve and meet the dynamic needs of the business.