Skip to content

🏛️ Athomic AI: Adaptive Decision Engine (ADE) & RL Core Architecture

1. Module: src/nala/athomic/ai/rl/ (The Learning Engine)

Purpose: Pure mathematical and algorithmic core. It is responsible for the "Learning" part of Reinforcement Learning. It has zero knowledge of business rules, LLMs, or Routers.

📁 Sub-modules & Deliverables:

  • algorithms/:

    • Deliverable: Implementations of RLAlgorithmProtocol.
    • Responsibility: Logic for calculating probabilities (Exploration vs Exploitation) and updating internal weights based on scalar rewards.
    • Key Files: epsilon_greedy.py, contextual_bandit.py.

  • trainers/:

    • Deliverable: Orchestrators for the update loop.
    • Responsibility: OnlineTrainingStrategy updates weights immediately after a reward. BatchTrainingStrategy accumulates experiences to perform offline gradient-like updates.

  • memory/:

    • Deliverable: State persistence providers.
    • Responsibility: Storing Q-Values, weights, and historical $(S, A, R, S')$ tuples. Must support high-frequency reads/writes.
    • Key Files: service.py (facade over KV + VectorDB), protocol.py (RLMemoryProtocol), factory.py.

  • reward/:

    • Deliverable: Scalar feedback generators and the closed-loop reward pipeline.
    • Responsibility: Translating external events into a scalar float reward for the rl/ module. Manages buffering and aggregation before training.
    • Key Files: engine.py (RewardEngine), aggregation.py (RewardAggregationService), strategies/ (BinaryRewardStrategy, NumericScaleRewardStrategy, CustomFormulaRewardStrategy), types.py (RewardEvent, RewardSignal).

2. Module: src/nala/athomic/ai/decision/ (The Orchestration Layer)

Purpose: The "Brain" of the system. It manages the decision lifecycle, applying constraints, safety, and business logic. It uses the RL Core but can work with static rules.

📁 Sub-modules & Deliverables:

  • policies/:

    • Deliverable: DecisionPolicyProtocol and its providers.
    • Responsibility: The RLPolicyProvider acts as the Adapter. It receives a State from the ADE, calls the rl/ module to get a mathematical choice, and returns the action.

  • features/:

    • Deliverable: FeatureEncoderProtocol (protocol only — no concrete implementations yet).
    • Responsibility: Pre-processing. Converts raw data (like a JSON request or a Prompt string) into a numerical or structured "State" that the RL algorithm understands.

  • actions/:

    • Deliverable: ActionProviderProtocol.
    • Responsibility: Discovery. Identifies which actions are physically possible (e.g., "Which LLM connections are currently UP?").

  • governance/:

    • Deliverable: Pruning logic.
    • Responsibility: Removes actions from the list before the policy sees them (e.g., "User is Free Tier, remove GPT-4o from candidates").

  • safety/:

    • Deliverable: Validation logic.
    • Responsibility: Final check after the policy decides. If the policy chooses a high-risk action (due to exploration), the safety layer can override it.
    • Key Files: providers/pipeline_safety.py, providers/llm_safety.py, providers/ensemble_safety.py, providers/audit_only_safety.py; rules/strategies/ (action_schema_safety_rule.py, blocklist_safety_rule.py, range_guard_safety_rule.py).

  • audit/:

    • Deliverable: DecisionAuditService.
    • Responsibility: Persists a full record of every decision (state, action, policy, confidence) to enable delayed reward correlation and debugging.

  • graph/:

    • Deliverable: DecisionGraph and DecisionGraphManager.
    • Responsibility: DAG-based orchestration of complex, multi-step decision flows. Nodes (DecisionNode) are connected via typed routing rules (RoutingRuleProtocol) — threshold, probabilistic, state-based, action-based, error-based — enabling branching and chaining of decision policies.

3. The Contract Interface (The Bridge)

To ensure SOLID compliance, the AI must implement this bridge logic:

# src/nala/athomic/ai/decision/policies/providers/rl_policy.py

class RLPolicyProvider(DecisionPolicyProtocol):
    """
    This is the ONLY place where the 'decision' module
    talks to the 'rl' module.
    Includes graceful degradation: falls back to a fallback policy on RL failure.
    """
    def __init__(
        self,
        rl_service: ReinforcementLearningService,
        fallback_policy: DecisionPolicyProtocol,
    ):
        self._rl = rl_service
        self._fallback = fallback_policy

    async def select_action(self, context: DecisionContext, actions: List[Any]) -> DecisionResult:
        # 1. Encode business context into a numerical state vector
        # 2. Delegate to RL service; fall back to heuristic on any failure
        try:
            return await self._rl.get_action(context, actions)
        except Exception:
            return await self._fallback.select_action(context, actions)

4. Execution Workflow (Step-by-Step)

  1. Context Capture: DecisionEngineService receives a raw request.
  2. State Encoding: features/ module turns request into a "State Vector".
  3. Action Discovery: actions/ module finds all available "Target Nodes".
  4. Governance Filter: governance/ removes unauthorized nodes.
  5. Policy Selection: RLPolicyProvider (ADE) asks EpsilonGreedy (RL Core) for the best node.
  6. Safety Check: safety/ confirms the choice won't break the system.
  7. Execution: The system uses the selected node. DecisionAuditService persists the full decision record.
  8. Feedback: Later, an external event reaches rl/reward/RewardEngine, which translates it into a scalar reward and calls rl/trainers/ to update weights.