AI Quality Assurance & Evaluation

The Nala QA Harness (nala.athomic.ai.qa) is a specialized module designed to evaluate non-deterministic AI systems (like RAG pipelines and Agents). Unlike traditional unit tests that rely on exact equality assertions, this module uses a flexible Strategy Pattern to assess quality dimensions such as Faithfulness, Answer Relevancy, and Context Recall.

Architectural Overview

The QA module is built upon three core pillars:

1. System Under Assessment (SUT): The AI component being tested. We use the Adapter Pattern to wrap different systems (RAG, Chatbots) into a uniform protocol (SystemUnderAssessmentProtocol).
2. Metric Strategies: Pluggable logic that calculates a score (0.0 to 1.0) for a given interaction. These can be deterministic (e.g., ExactMatch) or LLM-based (e.g., Faithfulness).
3. Orchestration Service: The QAService manages the execution flow, ensuring metrics are calculated, aggregated, and reported to the Observability stack (Prometheus).

Core Components

1. Quality Case

The fundamental unit of testing. It defines the input, the expected ground truth, and the context required for evaluation.

from nala.athomic.ai.schemas.qa import QualityCase

case = QualityCase(
    id="TC-RAG-001",
    input_data="What is the capital of France?",
    expected_output="Paris",
    expected_context_ids=["doc_france_wiki_01"],
    metadata={"difficulty": "easy", "domain": "geography"}
)

2. Adapters

To test a system, you must wrap it in an Adapter. For RAG systems, we provide the RAGAdapter.

from nala.athomic.ai.qa.adapters.rag_adapter import RAGAdapter
from nala.athomic.ai.rag.service import RAGService

# Assuming rag_service is your configured service instance
adapter = RAGAdapter(rag_service=rag_service)

3. Metric Strategies

Strategies define how success is measured.

• Deterministic (Fast/Free):
  • ExactMatchStrategy: String equality (case-insensitive options).
  • KeywordInclusionStrategy: Checks if specific keywords are present.
  • ContextHitStrategy: Verifies if expected document IDs were retrieved.
• LLM-based (Slow/Costly): (Requires LLM-as-a-Judge)
  • Faithfulness: Does the answer contradict the context?
  • AnswerRelevancy: Does the answer address the user query?

---

Usage Example

Here is how to wire everything together in a test or a continuous evaluation script:

import asyncio
from nala.athomic.ai.qa.service import QAService
from nala.athomic.ai.evaluation.strategies.basic import KeywordInclusionStrategy, ContextHitStrategy
from nala.athomic.config.schemas.ai.qa import QASettings

async def run_evaluation():
    # 1. Configuration
    settings = QASettings(enabled=True, model_alias="gpt-4-turbo")

    # 2. Define Strategies
    strategies = [
        KeywordInclusionStrategy(threshold=0.8),
        ContextHitStrategy(threshold=1.0)
    ]

    # 3. Initialize Service
    service = QAService(
        settings=settings,
        system_adapter=adapter,  # Your configured Adapter
        metric_strategies=strategies,
        service_name="nightly_rag_eval"
    )
    await service.connect()

    # 4. Evaluate
    result = await service.evaluate_case(case)

    # 5. Inspect Results
    print(f"Overall Pass: {result.all_passed}")
    print(f"Metrics: {result.metrics}")

if __name__ == "__main__":
    asyncio.run(run_evaluation())

Observability Integration

The QA Service automatically exports metrics to Prometheus, enabling dashboards for "AI Health" tracking.

Metric Name	Type	Description
rag_quality_last_score	Gauge	The most recent score for a specific metric/dataset.
rag_quality_score_distribution	Histogram	Historical distribution of scores (detects drift).
rag_quality_failures_total	Counter	Total number of test cases that failed the threshold.
rag_evaluation_duration_seconds	Histogram	Latency overhead of the evaluation logic itself.

Labels used: dataset, metric_name, model_used.