Test Conventions

Comprehensive reference for writing tests in this repository. Applies to all test types: unit, integration, benchmark, and e2e. Valid for both human engineers and AI assistants.

---

Table of Contents

1. Directory Layout
2. Test Settings — The Golden Rule
3. Settings Helpers Reference
4. Fixture Scope Guidelines
5. Markers
6. Async Tests
7. Mocking Conventions
8. Integration Test Patterns
9. Benchmark Tests
10. Running Tests
11. Checklist Before Committing a New Test File

---

Directory Layout

tests/
├── conftest.py              ← root conftest: shared fixtures + settings helpers
├── unit/                    ← fast, no I/O, no infrastructure
│   └── nala/athomic/...
├── integration/             ← require running infra (Docker Compose)
│   └── nala/athomic/...
├── benchmark/               ← pytest-benchmark performance suite
│   └── nala/athomic/...
├── e2e/                     ← end-to-end flows
└── settings/                ← TOML settings files (one per test file)
    ├── unit/...
    ├── integration/...      ← (same sub-tree as the test files)
    ├── benchmark/...
    └── ...

Every test file that needs settings has a companion TOML in tests/settings/ at the exact same relative path, with the .py extension replaced by .toml:

tests/integration/nala/athomic/ai/llm/test_int_ollama.py
  → tests/settings/integration/nala/athomic/ai/llm/test_int_ollama.toml

tests/benchmark/nala/athomic/ai/features/test_benchmark_feature_encoding.py
  → tests/settings/benchmark/nala/athomic/ai/features/test_benchmark_feature_encoding.toml

---

Test Settings — The Golden Rule

Never instantiate domain configuration objects inline in test code.

The following are forbidden in test files:

# ❌ WRONG — instantiating config objects inline
_VECTOR_SETTINGS = FeatureVectorSettings(
    vector_id="benchmark_v1",
    features=[
        FeatureDefinitionSettings(name="age_scaled", source_key="age", transformation="min_max", ...),
    ],
)
_ENGINE_SETTINGS = FeatureEngineConnectionSettings(backend="async_numpy", ...)

The correct approach is:

1. Define configuration in a TOML file at tests/settings/<same path>/<test-name>.toml
2. Load it via load_test_settings() / settings_loader fixture in a fixture
3. Read it with get_settings() inside the test or fixture that needs it

# ✅ CORRECT — settings come from the TOML companion file

# tests/settings/benchmark/nala/athomic/ai/features/test_benchmark_feature_encoding.toml
[default.ai.features.vectors.benchmark_v1]
vector_id = "benchmark_v1"
[[default.ai.features.vectors.benchmark_v1.features]]
name = "age_scaled"
source_key = "age"
transformation = "min_max"
params = { min_val = 0.0, max_val = 100.0 }

# test file
from tests.conftest import load_test_settings, clear_test_settings
from nala.athomic.config.settings import get_settings

def setup_module(module):
    load_test_settings("tests/settings/benchmark/.../test_benchmark_feature_encoding.toml")
    vector_settings = get_settings().ai.features.vectors["benchmark_v1"]
    ...

Why? Test TOML files are: - Version-controlled and reviewable alongside the code - Consistent with how production settings are loaded (same dynaconf stack) - Isolated per test — no shared mutable Python objects - Easy to diff when settings change

---

Settings Helpers Reference

All helpers are defined in tests/conftest.py and available everywhere in the test suite without import in most cases (fixtures). For non-fixture contexts (setup_module, etc.) import explicitly:

from tests.conftest import load_test_settings, clear_test_settings

load_test_settings(toml_path, **extra_env)

Sets NALA_SETTINGS_FILES, clears the get_settings LRU cache, and reloads the settings module. Call at the start of a fixture's setup phase.

@pytest.fixture(scope="module", autouse=True)
def setup_settings():
    load_test_settings("tests/settings/integration/.../test_foo.toml")
    yield
    clear_test_settings()

Pass additional env vars as keyword arguments:

load_test_settings("tests/settings/.../test_foo.toml", ENV_FOR_DYNACONF="testing")

clear_test_settings(*extra_keys)

Removes NALA_SETTINGS_FILES (and any extra keys) from os.environ and clears the settings cache. Call in the teardown phase (after yield).

def teardown_module(module):
    clear_test_settings()                          # removes NALA_SETTINGS_FILES
    clear_test_settings("ENV_FOR_DYNACONF")        # also removes extra vars

settings_loader fixture (function-scoped)

A pytest fixture that wraps load_test_settings with monkeypatch for automatic cleanup — no explicit teardown needed. Use this for function-scoped fixtures.

@pytest.fixture(autouse=True)
def setup_settings(settings_loader):
    settings_loader("tests/settings/unit/.../test_foo.toml")
    # no yield needed — monkeypatch restores env automatically

Rule: Use settings_loader fixture for function-scoped needs. Use load_test_settings / clear_test_settings for module-scoped and setup_module contexts (because monkeypatch is function-scoped by default).

---

Fixture Scope Guidelines

Scope	When to use	Notes
"function"	Default. Most unit tests.	monkeypatch works; full isolation per test
"module"	Integration tests — expensive setUp (LLM/DB/Kafka connects)	load_test_settings + yield + clear_test_settings
"session"	Config checks, environment probing (e.g. ollama_available)	Never mutate state; read-only

Module-scoped settings fixture template:

@pytest.fixture(scope="module", autouse=True)
def setup_settings():
    load_test_settings("tests/settings/integration/nala/athomic/.../test_foo.toml")
    yield
    clear_test_settings()

Function-scoped settings fixture template:

@pytest.fixture(autouse=True)
def setup_settings(settings_loader):
    settings_loader("tests/settings/unit/nala/athomic/.../test_foo.toml")

---

Markers

Register markers in pytest.ini. Current markers:

Marker	Description	Run with
unit	Pure unit test	make test (default)
integration	Requires infra	make test-integration RUN_INTEGRATION=true
e2e	End-to-end flow	make test-all
benchmark	pytest-benchmark perf suite	make benchmark
ai	Requires LLM / external AI call	--run-ai flag

Mark every file at module level:

pytestmark = [pytest.mark.unit]
# or
pytestmark = [pytest.mark.integration, pytest.mark.asyncio(loop_scope="module"), pytest.mark.ai]

---

Async Tests

• Async mode is auto (asyncio_mode = "auto" in pytest.ini) — no @pytest.mark.asyncio decorator needed on individual tests.
• For module-scoped async fixtures add loop_scope="module" to both the pytestmark and each async fixture:

pytestmark = [pytest.mark.asyncio(loop_scope="module")]

@pytest_asyncio.fixture(scope="module", loop_scope="module")
async def provider():
    ...
    yield
    ...

• For benchmarks, async code must be bridged to sync via an explicit event loop (asyncio.new_event_loop() + loop.run_until_complete()), because pytest-benchmark requires a sync callable. See Benchmark Tests for the pattern.

---

Mocking Conventions

• Prefer MagicMock(spec=SomeProtocol) — spec= ensures the mock only exposes attributes the real object has, catching typos at test time.
• Use AsyncMock for coroutines (async def methods, __aenter__, __aexit__).
• Patch at the usage site, not the definition site:

# The code under test imports from nala.athomic.config; patch there:
with patch("nala.athomic.config.get_settings") as mock_get_settings:
    ...

• Do not mock the database in unit tests that test DB interaction — use integration tests with a real database instead.

---

Integration Test Patterns

Checking service availability

@pytest_asyncio.fixture(scope="module", loop_scope="module")
async def ollama_available() -> bool:
    try:
        async with httpx.AsyncClient(timeout=2.0) as client:
            response = await client.get("http://localhost:11434")
            return response.status_code == 200
    except httpx.RequestError:
        return False

Skipping when unavailable

@pytest_asyncio.fixture(scope="module", loop_scope="module")
async def provider(ollama_available: bool):
    if not ollama_available:
        pytest.skip("Ollama not reachable — skipping.")
    ...

Full integration test file skeleton

# tests/integration/nala/athomic/some/test_int_foo.py
import pytest
import pytest_asyncio
from tests.conftest import clear_test_settings, load_test_settings
from nala.athomic.config.settings import get_settings

pytestmark = [
    pytest.mark.integration,
    pytest.mark.asyncio(loop_scope="module"),
]

@pytest.fixture(scope="module", autouse=True)
def setup_settings():
    load_test_settings("tests/settings/integration/nala/athomic/some/test_int_foo.toml")
    yield
    clear_test_settings()

@pytest_asyncio.fixture(scope="module", loop_scope="module")
async def service():
    svc = MyService(settings=get_settings().some)
    await svc.start()
    yield svc
    await svc.stop()

async def test_something(service):
    result = await service.do_thing()
    assert result is not None

---

Benchmark Tests

Benchmark tests live in tests/benchmark/ and require the pytest-benchmark plugin. They are excluded from make test and must be run explicitly.

Rules

1. Mark with pytestmark = [pytest.mark.benchmark].
2. Load settings from a companion TOML (same rules as all other tests).
3. Use setup_module / teardown_module for one-time engine setup (avoids benchmark overhead on every iteration).
4. For async engines: create ONE event loop at module level and use loop.run_until_complete() — do not use asyncio.run() per iteration (it creates a new loop and breaks asyncio primitives that bind to the first loop, such as asyncio.Lock / asyncio.Queue).
5. Build context columns as 2D (N, 1) arrays for scalar features when feeding AsyncNumpyEngine._execute_tensor_batch_impl directly — the engine's output matrix slice is (N, k) shaped.

Benchmark file skeleton

# tests/benchmark/nala/athomic/.../test_benchmark_foo.py
from __future__ import annotations
import asyncio
import pytest
from tests.conftest import clear_test_settings, load_test_settings
from nala.athomic.config.settings import get_settings

pytestmark = [pytest.mark.benchmark]

_SETTINGS_TOML = "tests/settings/benchmark/nala/athomic/.../test_benchmark_foo.toml"

_loop: asyncio.AbstractEventLoop
_engine: ...   # your engine

def setup_module(module: object) -> None:
    global _loop, _engine
    load_test_settings(_SETTINGS_TOML)
    settings = get_settings()
    _loop = asyncio.new_event_loop()
    _engine = MyEngine(settings=settings.something)
    _loop.run_until_complete(_engine.connect())

def teardown_module(module: object) -> None:
    _loop.run_until_complete(_engine.close())
    _loop.close()
    clear_test_settings()

@pytest.mark.parametrize("batch_size", [1, 32, 128, 512])
def test_foo_latency(benchmark, batch_size):
    def _run():
        return _loop.run_until_complete(_engine.do_work(batch_size))
    result = benchmark(_run)
    assert result is not None

Running benchmarks

make benchmark                      # run without saving
make benchmark-save NAME=baseline   # capture baseline JSON
make benchmark-compare              # compare against saved baseline

Results are stored under .tests_saves/benchmark/ in pytest-benchmark JSON format with p50/p95/p99 latency, throughput, and mean statistics.

---

Running Tests

make test                                    # unit tests only (default)
make test-integration RUN_INTEGRATION=true   # integration tests (requires infra)
make test-all RUN_INTEGRATION=true           # unit + integration
make coverage                                # unit tests + terminal coverage report
make benchmark                               # benchmark suite

# Single file
poetry run pytest tests/unit/path/to/test_file.py

# With verbose output + stdout
make td tests/unit/path/to/test_file.py      # -s -vv shortcut

# AI-marked tests (LLM calls)
make ta tests/unit/path/to/test_file.py

---

Checklist Before Committing a New Test File

• [ ] Settings are in a TOML companion file at tests/settings/<same path>/<test-name>.toml
• [ ] No inline instantiation of FeatureVectorSettings, FeatureDefinitionSettings, FeatureCacheSettings, or similar domain config objects in test code
• [ ] Uses load_test_settings / clear_test_settings (module-scoped) or settings_loader fixture (function-scoped) — never bare os.environ["NALA_SETTINGS_FILES"] = ...
• [ ] Appropriate marker set via pytestmark
• [ ] Integration tests check for service availability and skip gracefully
• [ ] Async tests use pytest_asyncio.fixture + loop_scope="module" where needed
• [ ] Benchmark tests use a single persistent event loop (asyncio.new_event_loop()), not asyncio.run() per iteration
• [ ] make test still passes (unit tests unaffected by new integration/benchmark files)