Feature Platform — Operational Runbook

The end-to-end operational story for the Feature Engineering Platform (nala.athomic.ai.features): define → materialize → serve online → monitor → shadow → canary → promote / rollback, plus training-set generation, multi-pod consensus, multi-tenancy, and the metrics you watch at each step.

This is the operations companion to the architecture deep-dive in CLAUDE.md (ai/features section) and the design docs under docs/backlog/todo/ (online-feature-serving, skew-free-training-generator, active-versioning, version-health-gating, multi-tenancy-features).

The three planes

FeaturePlatformService owns lifecycle + shared state and exposes behavior on three focused surfaces — every operation below is reached through one of them:

Plane	What it does	Hot path?
`service.serving`	`encode` / `encode_batch` / `get_online_features` / `get_dimension` / `trace_execution` (+ shadow + canary routing)	yes — inference
`service.control`	catalog, dependency graph, health, discovery, skew-free training sets	no — offline/inventory
`service.releases`	aliases, health-gated promote / rollback, canary rollout + auto-rollback, multi-pod refresh	no — control

1. Define & bootstrap

Feature vectors are declarative settings (FeatureSettings.vectors); the service AOT-compiles each into an immutable FeatureExecutionPlan at startup. A faulty vector is isolated (Error Kernel) so the rest stay live.

from nala.athomic.ai.features.service import FeaturePlatformService
from nala.athomic.ai.features.registry import FeatureVectorRegistry
from nala.athomic.config import get_settings

service = FeaturePlatformService(
    settings=get_settings().ai.features,
    registry=FeatureVectorRegistry(),
    # optional plugged capabilities (see later sections):
    # online_reader=..., pit_store=..., shadow_controller=...,
    # canary_router=..., release_repository=..., recomputer=...,
)
await service.connect()          # bootstraps + compiles all vectors
print(service.health_extra())    # per-vector status, plan_hash, warmup_ms

Health check after deploy: every configured vector should be healthy with a non-empty plan_hash. Zero healthy vectors (while some were configured) fails startup by design.

2. Materialize (precompute history & keep online fresh)

Two materialization paths feed the offline lake and the online store:

Batch (materialization/providers/batch.py) — writes time-partitioned Parquet for offline/training. Tenant-scoped automatically (§8).
Streaming — the windowing engine collects events and, on window close, either encodes per-event (FeatureStreamBridge) or computes windowed aggregations (WindowAggregationBridge: count/sum/mean/min/max/first/ last/std over tumbling/sliding/session windows) and sinks them.

Streaming aggregations are config-driven (FeatureStreamingSettings.aggregations) — empty preserves per-event encoding; non-empty enables the Stream Feature View.

3. Serve online (low-latency reads)

Two modes:

# On-demand: compute from raw state through the current plan (hot path).
resp = await service.serving.encode("user_risk", raw_state, routing_key="user-42")
matrix = resp.data            # FeatureTensor (numpy/torch behind the domain type)
rows = resp.to_list()         # list[list[float]] at the service boundary

# Precomputed read: low-latency lookup of materialized vectors + freshness.
online = await service.serving.get_online_features("user_risk", entity_ids=["u1", "u2"])
for rec in online.records:    # entity-aligned to the request
    rec.entity_id, rec.values, rec.plan_hash, rec.age_seconds, rec.status
print(online.fresh, online.stale, online.plan_mismatch, online.missing)

Each OnlineFeatureRecord carries freshness (age_seconds + a status of fresh/stale vs freshness_sla_seconds) and plan attestation (plan_hash the stored vector was computed under; plan_mismatch when it differs from the live plan). Treat stale/plan_mismatch/missing per your policy (refresh, fall back to encode, or flag).

4. Monitor

Watch these signals continuously:

Signal	Metric	What it tells you
Online freshness	`feature_online_age_seconds` (histogram) + `stale` count	materialization lag / SLA breaches
Output drift	drift baselines + `quality/` module	distribution shift vs baseline
Canary routing	`feature_canary_requests_total{route,version}`	per-route traffic + comparison
Authz decisions	`nala_authz_decisions_total{permission,result}`	who is denied (when RBAC enforced, §9)
Compile/warmup	`feature_graph_compile_memory_bytes`, `warmup_ms`	bootstrap cost / version drift

service.control.health_extra() is the cross-plane health snapshot for /health.

5. Training sets (skew-free + backfill)

service.control.get_historical_features(vector_id, spine, policy=...) builds a point-in-time-correct training matrix and attests every row against the live serving plan_hash — a row counts as skew_free only when the plan that materialized it is the plan serving online now.

from nala.athomic.ai.features.training import TrainingPolicy

result = await service.control.get_historical_features(
    "user_risk",
    spine=[("u1", t0), ("u1", t1), ("u2", t0)],   # (entity_id, as_of)
    policy=TrainingPolicy.ATTESTED_ONLY,           # only skew-free rows
)
print(result.report.coverage, result.report.stale_plan_hashes)

Policies:

ATTESTED_ONLY (default) — matrix holds only skew_free rows; the rest are reported (never silently mixed).
STRICT — raise TrainingSkewError if any row is stale/missing.
INCLUDE_ALL — exploratory: include stale rows too (tagged per-row).
RECOMPUTE_ON_MISMATCH — Model B / backfill: re-derive stale/missing rows from raw historical state through the current plan (needs a RowRecomputer wired with a raw-state PIT source). Backfilling a newly-added feature over history is the same path. Rows with no recoverable raw state stay missing.

result.to_arrow() exports the full spine (all statuses + feature columns) for label-join downstream.

6. Release lifecycle: shadow → canary → promote / rollback

Versions are registered as vector_id@version; aliases (current, previous, canary) are indirected pointers. The safe rollout sequence:

# 1. Shadow — run the candidate alongside current, compare divergence (no traffic).
#    (wire a ShadowServingController; divergence feeds version health)

# 2. Canary — route a fraction of real traffic to the candidate.
await service.releases.set_canary("user_risk", "2.0.0", fraction=0.05)
await service.releases.set_canary("user_risk", "2.0.0", fraction=0.25)  # ramp

# 3. Evaluate health (manual or background monitor) — auto-rolls back on breach.
aborted = await service.releases.evaluate_all_canaries()

# 4a. Promote — atomic swap of the serving plan (health-gated).
await service.releases.promote_version("user_risk", "2.0.0")

# 4b. …or abort / roll back.
await service.releases.clear_canary("user_risk")
await service.releases.rollback_version("user_risk")   # reverts to `previous`

The promotion gate (PromotionGateSettings) blocks promote_version when the candidate's health evidence (shadow divergence / canary error rate) breaches thresholds — force=True overrides. Automatic canary rollback (CanaryRollbackSettings, trigger="monitor") runs evaluate_all_canaries on a background loop and aborts unhealthy canaries. Both are off by default; enable via settings.

7. Multi-pod consensus

In a horizontally-scaled deployment, promotion/canary decisions must reach every replica. Inject a KV-backed RedisReleaseStateRepository (default is in-process MemoryReleaseStateRepository for single-pod):

Every promote / rollback / set_canary / clear_canary writes through the shared store.
A fresh replica reconciles at bootstrap (adopts the cluster's active version, skipping versions it has not compiled yet).
Long-running replicas converge live when ReleaseConsensusSettings(enabled=True) runs the periodic refresh loop; or call await service.releases.refresh() on demand.

Only the pointers are shared — compiled encoders stay per-pod.

8. Multi-tenancy

Tenant isolation is a per-deployment option, driven by the request ExecutionContext tenant + context.multi_tenancy_enabled:

KV-backed paths (online store, temporal L2 state) isolate transparently via the kvstore key-resolver — nothing to do.
Lake/parquet paths are tenant-scoped symmetrically on write and read via tenancy.tenant_path_segment() (a tenant=<id> partition segment); off when no tenant is in context, so single-tenant deployments are unchanged.
Catalog is deployment metadata (shared across tenants), not tenant data.

9. Authorization (RBAC) — enforced at the API edge

The features module imports no authorization logic. Sensitive operations are guarded at the API route with requires("features:<action>") (api/dependencies/security.py), which calls the central AuthorizationService:

Operation	Permission
`serving.encode*` / `get_online_features`	`features:read`
`control.get_historical_features`	`features:train`
`releases.promote_version` / `rollback` / `set_canary`	`features:promote`
`control.discovery.*`	`features:read`

Rollout is staged via AuthorizationSettings.mode: dry_run (default — meters would_deny without blocking), enforce (HTTP 403 on deny), disabled. Roles → permissions come from the static resolver (committed config) or the kv resolver (dynamic orgs).

10. Discovery & impact analysis

from nala.athomic.ai.features.discovery import FeatureQuery

service.control.discovery.search(FeatureQuery(owner="risk-team", status="healthy"))
service.control.discovery.describe("user_risk")
service.control.discovery.consumers_of("raw_txn_amount")  # impact: who breaks if I change this?

Quick operational checklist

Deploy → health_extra() shows all vectors healthy with a plan_hash.
Materialization lag → watch feature_online_age_seconds; investigate stale.
New version → register → shadow → canary ramp → watch feature_canary_requests_total + health → promote (gate) or rollback.
Multi-pod → confirm the shared release repo + (optional) refresh loop is on.
Training → get_historical_features; coverage should be ~1.0 for a healthy vector (use RECOMPUTE_ON_MISMATCH to backfill/repair stale history).
Multi-tenant → verify multi_tenancy_enabled + tenant present in context.