Quality Monitoring and Tuning

At rapidflare.ai, we are committed to maintaining the highest standards of performance and accuracy in our enterprise copilot. Our comprehensive quality reports provide valuable insights, enabling both our internal teams and customers to track performance and ensure continuous improvement.

Types of Quality Reports

We offer two primary types of quality reports to assess and enhance the copilot's performance:

Initial QA Report: Conducted during the onboarding phase by our customer success team, this report helps assess and fine-tune the copilot's responses before full deployment, ensuring it aligns with customer-specific needs.
Ongoing QA Report: After deployment, this report captures, analyzes, and resolves customer feedback. It is essential for maintaining the copilot's accuracy and relevance over time. The ongoing QA report features a pie chart that highlights the percentage of issues across various "Issue RCA" (Root Cause Analysis) categories, including missing information, glossary gaps, analytical errors, and poor references.

Key Performance Metrics

Our quality reports measure several key performance indicators (KPIs) to ensure the copilot's effectiveness:

Correctness of Answers: Evaluating the accuracy of responses across various query categories.
Correctness of References: Ensuring the right sources and references are used for each query.
Number of Conversations: Tracking the volume of interactions to monitor overall engagement.
Customer Feedback: Collecting both positive and negative feedback to drive improvements.
Issue RCA: Categorizing issues into root causes such as missing information, glossary mismatches, analytical errors, and poor references. This analysis allows for targeted improvements to the copilot's responses.

Each instance of negative feedback is recorded in the report, with a status field indicating the progress of issue resolution, ensuring transparency and accountability in addressing customer concerns.

Report Frequency

We provide quality reports on a weekly and monthly basis, allowing customers to regularly monitor the copilot's performance. Additional reports can be delivered on demand, ensuring flexibility and real-time updates.

Report Format

Our reports are presented in Excel format, with organized data and visual elements. Pie charts are used to display key performance metrics, including the percentage of correct answers and a breakdown of issues based on Issue RCA categories. This visual representation offers customers a clear and quick overview of the copilot's performance trends.

Actionable Insights

Both the initial and ongoing QA reports include actionable insights, with comments and queries from our customer success team. Customers are encouraged to engage with these insights, helping us resolve issues such as missing documentation or incorrect references. This collaborative feedback loop ensures that the copilot continues to evolve and meet the dynamic needs of the business.