Build a Replay-Aware Telemetry Recorder

What we’re building

A headless recorder attempting telemetry.jsonl records for session boundaries, arena periods, player samples, time-warp markers, admission rejections and shutdown. Sampling requests a 0.5-second callback delay after work. Eight MiB is a post-batch rotation threshold with two managed predecessors; records and files have no hard byte cap.

Evidence: the earlier core-path runtime verification belongs to WG 2.3.1.3; rotation and rewind were not force-tested. The current review compares authored source and selected latest public client-source excerpts. The full current runtime/fault matrix remains unexecuted; the source below is preserved, including the documented limitations.

Final result

• Versioned envelopes with recorder-local attempt sequences.
• A 2048-entry producer queue and carried rejection counter.
• Up to 128 dequeued records per writer batch, approximate idle-aware flushing and threshold rotation.
• Separate wall, client, server and optional replay clock fields.
• Generation invalidation and one-slot sample/recovery scheduling, with failure limits.
• Recorded handler identities, staged cleanup and a requested two-second writer join.

What you’ll learn

• How envelope attempts, queue admission, encoding, writes and durable storage differ.
• How worker encoding and writes coexist with caller-side startup filesystem work.
• Why entry capacity and rejection counters do not measure every kind of loss.
• How replay discontinuities affect callback ownership and clock comparisons.
• How to preserve evidence limits when reviewing a reliability example.

Difficulty and prerequisites

Level 10 — Reliability and data engineering. Complete the Level 9 dashboard. You should understand generation tokens, event ownership, multiple time domains, schemas, bounded resources, and worker shutdown.

Requirements and constraints

Functional review targets

1. Start a writer before subscribing producers; thread start supplies no file-open or health acknowledgement.
2. Use put_nowait to avoid capacity waits. Queue mutex acquisition, clock reads and synchronous startup directory/stat/rotation work still have costs.
3. Attempt schema/session/generation/sequence/type/clock envelopes without treating admission as delivery.
4. Invalidate generations at warp boundaries and request deferred recovery; examine stale callbacks, failed cancellation and missing readiness.
5. Retain the session ID through a warp, without assuming the recovered avatar still has the same arena identity.
6. Attempt cleanup and drainage, while distinguishing the join request from a whole-shutdown deadline.

Privacy and scope

The selected producers collect local arena/session identity, lifecycle, period, position, clocks and rejection counts. They contain no keyboard/mouse, chat, account-descriptor collection or upload call. This is a source inventory, not a privacy audit: there is no privacy validator for arbitrary callers, encryption, anonymization or retention duration. Logs and tracebacks are separate diagnostic surfaces.

The helper derives the preferences directory; the recorder appends mods/replay_telemetry_recorder/telemetry.jsonl. That wrapper proves neither write permission nor exclusive ownership of the directory.

Acceptance criteria

These are review and separately authorized testing targets, not a passing report. The current source does not satisfy stronger guarantees of strictly nonblocking production, complete drop reporting, universal cleanup or exactly one sampler after every failure.

• Distinguish queue/batch entry limits from encoded bytes, rotation overshoot and actual elapsed shutdown.
• Verify admission rejection separately from serialization/write/flush loss.
• Check the primitive-data convention: the module imports BigWorld, and neither _emit nor the queue enforces primitive isolation.
• Interpret sequence as attempted production within one recorder lifetime, including rejected or failed attempts.
• Test pending rejection counts when construction fails, shutdown rejects admission, or no later record reaches disk.
• Exercise warp start/finish, stale callbacks, pending sample/recovery slots and replacement avatars.
• Recheck live operation with optional replay imports unavailable; other dependencies can still fail.
• Verify exact-handler removal, later cleanup reachability and timed-join outcomes under faults.

Architecture

GAME THREAD                               WRITER THREAD
context events ─┐
period events ──┼→ primitive envelope → Queue(max=2048) → batches → JSONL
sample callback ┘          │                    │              │
                           │ full               │ drain        ├→ flush
                           ▼                    │              └→ rotate 8 MiB
                      dropped counter ──────────┘

replay warp start → generation++ → cancel sampler → control event
replay warp finish → control event → callback(0) guarded recovery

Researching the implementation

Selected latest public default-branch excerpts currently match the local source used in this review. ReplayEvents creates zero-argument warp events as Event.Event: start sets warp/direction before dispatch; finish dispatches before clearing the warp flag. Event copies listeners and rethrows a handler exception, which can skip later listeners and that trailing clear. PlayerEvents instead inherits SafeEvent creation through ClientEventsContainer and EventsContainer; its copied-list dispatch logs and continues.

BattleReplay emits start before native beginTimeWarp; a false return still invokes cleanup and finish. Module isPlaying() includes warp-in-progress. A finish marker therefore proves neither successful native seeking nor restored avatar readiness. Its currentTime property projects a native time mark; this wrapper does not establish native timing guarantees.

Avatar fires entry before prerequisite/GUI loading and exit after arena teardown. ClientArena replaces its period tuple, then fires local and global period events. Inspect those lifecycle boundaries rather than equating entry, sampling readiness and GUI readiness. These are scoped source comparisons, not whole-module or runtime validation.

Source lookup

The recorder depends on replay discontinuities, callback invalidation, and queueing/persistence limits:

• Replay differences, time warp as a lifecycle boundary, and idempotent rewind handling.
• Replay seeks as separate transitions — why a seek invalidates transient work without creating a new battle session.
• Single pending callback and teardown rules — sampler and recovery ownership.
• Durable writes and path safety — limits of append, flush, rotation, and interrupted output.
• Version-sensitive source contracts — how to re-audit private replay events after a client update.

Search the target build for ReplayEvents.g_replayEvents, callOnTimeWarpStart, the time-warp finish producer, isLastWarpRewind, isTimeWarp, BattleReplay.isPlaying, and the current replay-time accessor. Treat them as investigation entry points and confirm registrations, event arguments, callbacks, and state mutation order at their firing sites.

Lifecycle and ownership

1. init: compute the preferences-based path, attempt caller-side directory/stat/rotation work and start the daemon before binding events. There is no writer-ready acknowledgement or current-avatar seed.
2. avatar entry: end any active session before validating the new avatar. A new generation and wall-seconds-based session ID do not deduplicate arena reentry or prevent ID collisions.
3. sample: clear the callback slot before checking generation/active state, read the stored avatar, attempt an envelope, then schedule after work. A stale callback can forget a newer handle.
4. warp start: advance generation, cancel sample/recovery, then emit a marker. An exception can interrupt these stages.
5. warp finish: advance generation and emit before replacing recovery. It does not cancel a pending sample. Recovery clears its slot before guards, adopts the current player and schedules only for a player avatar; there is no readiness retry or arena/session comparison.
6. avatar exit/disconnect: attempt end/cancellation before clearing active state. This is not an exactly-once terminal receipt or universal process-exit signal.
7. fini: attempt reverse unsubscription, forget the ledger and clear flags even after removal failures; then end, emit stop, cancel and detach/stop the writer. An earlier exception can prevent later stages.

State and data-authority model

Implementation walkthrough

Event-envelope convention

Schema v1 and event-specific data describe an authored convention. The boundary performs no schema/type/finite-number/size validation or deep copy; queued data retains references. Selected producers build primitive-valued dictionaries, but arbitrary callers are not constrained. Default encoding is not a strict finite-number JSON gate. Incompatible field-meaning changes need a new schema version; a new event type still requires reader compatibility review.

Queue admission and buffered output

put_nowait avoids waiting for capacity but still acquires the queue mutex in the inspected Python 2 queue implementation. Capacity is 2048 entries, excluding up to 128 dequeued records and buffered bytes. The stop-flag check and put are separate operations. Admission does not test worker health or promise encoding, writing, flushing or durability.

_emit allocates sequence and resets pending drops before clocks/envelope/admission. A false return carries the prior count plus one; later construction/admission exceptions can lose the pending count. There may be no next accepted report, and an accepted report may never reach disk. An empty queue can coexist with dirty output: the retained dirty flag keeps buffered bytes eligible for interval flushing during empty polls.

Rewind markers

Attempted start/finish markers and new generations describe discontinuities; they are not guaranteed persisted pairs or successful-seek receipts. Consumers must not derive motion across missing records, session/identity changes or rewind boundaries. The illustrative diagram below expresses design intent; its primitive/bounded/drain labels do not override these source limits.

Recorder contract review

Startup and installed flags are separate receipts. Binding cleanup is attempted after partial failure, but startup has no rollback transaction; post-install emission failure can leave subscriptions installed. The mod loader invokes entry points by name; module handlers log unexpected exceptions with active tracebacks and keep Error Reporter optional. The release/manifest version is 1.0.0, separate from envelope schema 1.

Samples resolve the stored avatar’s player-vehicle ID and require an entity, truthy isStarted and equal space IDs. Missing IDs can compare as None == None; there is no finite-position, fresh-avatar, arena-identity or atomic-vector-snapshot gate. Coordinates are read separately and rounded; direct arena-period access can fail. Caught sample errors log before the next scheduling attempt; scheduling itself is outside that catch.

The preferences helper supplies only the parent directory. The authored tree and manifest are references, not evidence of staged files or a package. No authored producer, event/queue/thread/file-fault fixture, replay or package was executed in this source review.

Complete source code

# -*- coding: utf-8 -*-
"""Record bounded local battle telemetry with replay discontinuity markers."""
from __future__ import absolute_import, unicode_literals

# Reader map:
#   1. _Writer owns bounded queue consumption, JSONL output, and rotation.
#   2. _Recorder owns game-thread producers and replay discontinuities.
#   3. Only primitive event dictionaries cross into the writer thread.

import json
import logging
import os
import threading
import time

# Queue was renamed in Python 3; the alias keeps test tooling portable.
try:
    import Queue
except ImportError:
    import queue as Queue

import BigWorld
from Avatar import PlayerAvatar
from PlayerEvents import g_playerEvents
from helpers import getPreferencesDirPath

# Replay support is optional so the same recorder still works live.
try:
    import BattleReplay
    from ReplayEvents import g_replayEvents
except ImportError:
    BattleReplay = g_replayEvents = None

__version__ = '1.0.0'
_SCHEMA_VERSION = 1
_LOGGER = logging.getLogger(__name__)
_MOD_NAME = 'replay_telemetry_recorder'
_OUTPUT_DIR = os.path.join(getPreferencesDirPath(), 'mods', _MOD_NAME)
_OUTPUT_PATH = os.path.join(_OUTPUT_DIR, 'telemetry.jsonl')
_QUEUE_CAPACITY = 2048
_BATCH_SIZE = 128
_MAX_FILE_BYTES = 8 * 1024 * 1024
_RETAINED_FILES = 2
_SAMPLE_SECONDS = 0.5
_JOIN_SECONDS = 2.0


class _Writer(object):
    """Drain a bounded queue to append-only JSONL away from game thread."""
    def __init__(self, path):
        self._path = path
        self._queue = Queue.Queue(_QUEUE_CAPACITY)
        self._stopping = threading.Event()
        self._thread = None
        self.error = None

    def start(self):
        """Rotate before opening and start one daemon consumer."""
        if not os.path.isdir(_OUTPUT_DIR):
            os.makedirs(_OUTPUT_DIR)
        self._rotate_if_needed()
        self._thread = threading.Thread(target=self._run,
                                        name='ReplayTelemetryWriter')
        self._thread.daemon = True
        self._thread.start()

    def put_nowait(self, event):
        """Never let disk pressure block the game-thread producer."""
        if self._stopping.is_set():
            return False
        try:
            self._queue.put_nowait(event)
            return True
        except Queue.Full:
            return False

    def stop(self):
        """Signal, drain, and join for a finite amount of time."""
        self._stopping.set()
        thread, self._thread = self._thread, None
        if thread is not None and thread is not threading.current_thread():
            thread.join(_JOIN_SECONDS)
            if thread.is_alive():
                _LOGGER.warning('%s: writer did not stop in time', _MOD_NAME)

    def _rotate_if_needed(self):
        """Retain a fixed number of predecessors once the size cap is hit."""
        if not os.path.isfile(self._path) or os.path.getsize(self._path) < _MAX_FILE_BYTES:
            return
        for index in xrange(_RETAINED_FILES, 0, -1):
            source = self._path if index == 1 else '%s.%d' % (self._path, index - 1)
            target = '%s.%d' % (self._path, index)
            if os.path.isfile(target):
                os.remove(target)
            if os.path.isfile(source):
                os.rename(source, target)

    @staticmethod
    def _encode(event):
        return (json.dumps(event, ensure_ascii=True, sort_keys=True,
                           separators=(',', ':')) + '\n').encode('ascii')

    def _run(self):
        """Batch encoding/writes while preserving accepted queue order."""
        stream = None
        last_flush = time.time()
        dirty = False
        try:
            stream = open(self._path, 'ab')
            while not self._stopping.is_set() or not self._queue.empty():
                batch = []
                try:
                    batch.append(self._queue.get(True, 0.25))
                except Queue.Empty:
                    pass
                while len(batch) < _BATCH_SIZE:
                    try:
                        batch.append(self._queue.get_nowait())
                    except Queue.Empty:
                        break
                for event in batch:
                    stream.write(self._encode(event))
                    dirty = True
                now = time.time()
                if dirty and now - last_flush >= 1.0:
                    stream.flush()
                    dirty = False
                    last_flush = now
                if stream.tell() >= _MAX_FILE_BYTES:
                    stream.flush()
                    stream.close()
                    stream = None
                    self._rotate_if_needed()
                    stream = open(self._path, 'ab')
                    dirty = False
                    last_flush = now
            stream.flush()
            os.fsync(stream.fileno())
        except Exception as error:
            self.error = '%s: %s' % (error.__class__.__name__, error)
            _LOGGER.exception('%s: writer failed', _MOD_NAME)
        finally:
            if stream is not None:
                try:
                    stream.close()
                except Exception:
                    _LOGGER.exception('%s: writer close failed', _MOD_NAME)


class _Recorder(object):
    """Own telemetry producers, sessions, clocks, and callback generations."""
    def __init__(self):
        self._installed = False
        self._writer = None
        self._active = False
        self._avatar = None
        self._session_id = None
        self._generation = 0
        self._sequence = 0
        self._dropped = 0
        self._sample_id = None
        self._recover_id = None
        self._replay_bound = False
        self._bindings = []
        self._enter_handler = self._on_avatar_enter
        self._exit_handler = self._on_avatar_exit
        self._disconnect_handler = self._on_disconnect
        self._period_handler = self._on_period
        self._warp_start_handler = self._on_warp_start
        self._warp_finish_handler = self._on_warp_finish

    def start(self):
        """Start the consumer before subscribing any event producer."""
        if self._installed:
            return
        writer = _Writer(_OUTPUT_PATH)
        try:
            writer.start()
        except Exception:
            _LOGGER.exception('%s: writer startup failed', _MOD_NAME)
            return
        self._writer = writer
        try:
            bindings = ((g_playerEvents.onAvatarBecomePlayer, self._enter_handler),
                        (g_playerEvents.onAvatarBecomeNonPlayer, self._exit_handler),
                        (g_playerEvents.onDisconnected, self._disconnect_handler),
                        (g_playerEvents.onArenaPeriodChange, self._period_handler))
            for event, handler in bindings:
                event += handler
                self._bindings.append((event, handler))
            if g_replayEvents is not None:
                replay_bindings = (
                    (g_replayEvents.onTimeWarpStart, self._warp_start_handler),
                    (g_replayEvents.onTimeWarpFinish, self._warp_finish_handler))
                for event, handler in replay_bindings:
                    event += handler
                    self._bindings.append((event, handler))
                self._replay_bound = True
            self._installed = True
        except Exception:
            _LOGGER.exception('%s: event binding failed', _MOD_NAME)
            self.stop()
            return
        self._emit('recorder_start', {'modVersion': __version__})
        _LOGGER.info('%s %s: initialized', _MOD_NAME, __version__)

    def stop(self):
        """Unbind producers before asking the writer to drain and stop."""
        for event, handler in reversed(self._bindings):
            try:
                event -= handler
            except Exception:
                _LOGGER.exception('%s: event unbinding failed', _MOD_NAME)
        self._bindings = []
        self._installed = False
        self._replay_bound = False
        self._end_session('client_shutdown')
        self._emit('recorder_stop', {})
        self._cancel_callbacks()
        writer, self._writer = self._writer, None
        if writer is not None:
            writer.stop()

    def _on_avatar_enter(self):
        try:
            self._end_session('superseded')
            avatar = BigWorld.player()
            if not isinstance(avatar, PlayerAvatar):
                return
            arena = getattr(avatar, 'arena', None)
            arena_id = getattr(avatar, 'arenaUniqueID', None)
            if arena_id is None:
                _LOGGER.warning('%s: arenaUniqueID unavailable', _MOD_NAME)
                return
            self._generation += 1
            self._active = True
            self._avatar = avatar
            self._session_id = '%s-%d' % (unicode(arena_id), int(time.time()))
            self._emit('session_start', {
                'arenaID': unicode(arena_id),
                'replay': bool(BattleReplay and BattleReplay.isPlaying())})
            self._schedule_sample(self._generation, 0.0)
        except Exception:
            _LOGGER.exception('%s: avatar-entry handling failed', _MOD_NAME)
            self._end_session('activation_failed')

    def _on_avatar_exit(self):
        try:
            self._end_session('observed_avatar_exit')
        except Exception:
            _LOGGER.exception('%s: avatar-exit handling failed', _MOD_NAME)

    def _on_disconnect(self):
        try:
            self._end_session('disconnected')
        except Exception:
            _LOGGER.exception('%s: disconnect handling failed', _MOD_NAME)

    def _on_period(self, period, end_time, length, _additional):
        try:
            if self._active:
                self._emit('period', {'period': int(period),
                    'endServerTime': float(end_time), 'length': float(length)})
        except Exception:
            _LOGGER.exception('%s: period handling failed', _MOD_NAME)

    def _on_warp_start(self):
        """Mark the discontinuity and invalidate the active sampler."""
        try:
            if not self._active:
                return
            self._generation += 1
            self._cancel_sample()
            if self._recover_id is not None:
                BigWorld.cancelCallback(self._recover_id)
                self._recover_id = None
            self._emit('time_warp_start', {
                'rewind': bool(g_replayEvents.isLastWarpRewind)})
        except Exception:
            _LOGGER.exception('%s: time-warp-start handling failed', _MOD_NAME)

    def _on_warp_finish(self):
        """Emit the finish marker before scheduling guarded recovery."""
        try:
            if not self._active:
                return
            self._generation += 1
            generation = self._generation
            self._emit('time_warp_finish', {})
            if self._recover_id is not None:
                BigWorld.cancelCallback(self._recover_id)
            self._recover_id = BigWorld.callback(
                0.0, lambda: self._recover_after_warp(generation))
        except Exception:
            _LOGGER.exception('%s: time-warp-finish handling failed', _MOD_NAME)

    def _recover_after_warp(self, generation):
        """Converge repeated seek finishes back to one sampler."""
        self._recover_id = None
        if (generation != self._generation or not self._active or
                not (BattleReplay and BattleReplay.isPlaying())):
            return
        self._avatar = BigWorld.player()
        if isinstance(self._avatar, PlayerAvatar):
            self._schedule_sample(generation, 0.0)

    def _schedule_sample(self, generation, delay):
        if self._sample_id is None and self._active:
            self._sample_id = BigWorld.callback(
                delay, lambda: self._sample(generation))

    def _sample(self, generation):
        self._sample_id = None
        if generation != self._generation or not self._active:
            return
        try:
            avatar = self._avatar
            vehicle_id = getattr(avatar, 'playerVehicleID', None)
            vehicle = BigWorld.entity(vehicle_id) if vehicle_id else None
            position = None
            if (vehicle is not None and getattr(vehicle, 'isStarted', False) and
                    getattr(vehicle, 'spaceID', None) == getattr(avatar, 'spaceID', None)):
                position = [round(float(vehicle.position.x), 3),
                            round(float(vehicle.position.y), 3),
                            round(float(vehicle.position.z), 3)]
            self._emit('sample', {'position': position,
                                  'period': int(getattr(avatar.arena, 'period', -1))})
        except Exception:
            _LOGGER.exception('%s: sample failed', _MOD_NAME)
        self._schedule_sample(generation, _SAMPLE_SECONDS)

    def _emit(self, event_type, data):
        """Create one ordered schema envelope and account for prior drops."""
        writer = self._writer
        if writer is None:
            return
        self._sequence += 1
        dropped, self._dropped = self._dropped, 0
        replay_time = None
        try:
            if BattleReplay and BattleReplay.isPlaying():
                replay_time = float(BattleReplay.g_replayCtrl.currentTime)
        except Exception:
            _LOGGER.exception('%s: replay-time read failed', _MOD_NAME)
        event = {'schemaVersion': _SCHEMA_VERSION,
                 'sessionID': self._session_id,
                 'generation': self._generation,
                 'sequence': self._sequence, 'type': event_type,
                 'wallTime': time.time(), 'clientTime': BigWorld.time(),
                 'serverTime': BigWorld.serverTime(),
                 'replayTime': replay_time, 'data': data}
        if dropped:
            event['droppedBefore'] = dropped
        if not writer.put_nowait(event):
            self._dropped = dropped + 1

    def _end_session(self, reason):
        """Emit one terminal event without inventing an official result."""
        if not self._active:
            return
        self._generation += 1
        self._cancel_callbacks()
        self._emit('session_end', {'reason': reason})
        self._active = False
        self._avatar = self._session_id = None

    def _cancel_sample(self):
        if self._sample_id is not None:
            BigWorld.cancelCallback(self._sample_id)
            self._sample_id = None

    def _cancel_callbacks(self):
        self._cancel_sample()
        if self._recover_id is not None:
            BigWorld.cancelCallback(self._recover_id)
            self._recover_id = None


_RECORDER = _Recorder()


def init():
    """World of Tanks mod-loader entry point."""
    try:
        _RECORDER.start()
    except Exception:
        _LOGGER.exception('%s: initialization failed', _MOD_NAME)


def fini():
    """World of Tanks mod-loader cleanup entry point."""
    try:
        _RECORDER.stop()
    except Exception:
        _LOGGER.exception('%s: finalization failed', _MOD_NAME)

<root><id>tutorial.replay_telemetry_recorder</id><version>1.0.0</version><name>Replay Telemetry Recorder</name><description>Records bounded local telemetry with replay discontinuity markers.</description></root>

[
  {"schemaVersion":1,"sessionID":"123-1787760000","generation":1,"sequence":2,"type":"session_start","wallTime":1787760000.2,"clientTime":41.5,"serverTime":1787760000.0,"replayTime":12.0,"data":{"arenaID":"123","replay":true}},
  {"schemaVersion":1,"sessionID":"123-1787760000","generation":2,"sequence":80,"type":"time_warp_start","wallTime":1787760040.1,"clientTime":81.4,"serverTime":1787760040.0,"replayTime":52.0,"data":{"rewind":true}},
  {"schemaVersion":1,"sessionID":"123-1787760000","generation":3,"sequence":81,"type":"time_warp_finish","wallTime":1787760040.4,"clientTime":81.7,"serverTime":1787760040.0,"replayTime":18.0,"data":{}}
]

Telemetry-envelope field guide: schemaVersion defines field meaning, sessionID correlates one observed battle/replay, generation marks discontinuities, sequence orders recorder-local envelope attempts, the clock fields retain their separate time domains, and data conventionally carries event-specific primitives without an enforced validator. The writer attempts one encoded object plus newline per JSONL record; the surrounding array is illustrative reference data, not captured output. Startup does not repair an old incomplete tail before appending.

Minimal project structure

replay-telemetry-recorder/
├── mod_replay_telemetry_recorder.py
└── stage/
    ├── meta.xml
    └── res/scripts/client/gui/mods/mod_replay_telemetry_recorder.pyc

preferences/mods/replay_telemetry_recorder/
├── telemetry.jsonl
├── telemetry.jsonl.1
└── telemetry.jsonl.2

Packaging and installation

1. Compile with the target-compatible Python 2.7 compiler.
2. Stage bytecode and aligned manifest only.
3. Create and inspect a stored-member replay_telemetry_recorder_1.0.0.wotmod.
4. Never package or delete runtime telemetry during upgrades.
5. Installation, live recording, replay seeking, and forced-exit behavior require user-side testing.

Testing and debugging

The following are unexecuted plans for separate authorized testing. An observed source limitation must not be recorded as a passed goal.

Deterministic test plans

• Force full/stopping admission rejection; inspect sequences and pending counts, including no later admission and exceptions after the counter reset.
• Validate output independently for schema, finite numbers, gaps and session/generation identity. The writer has no such validator.
• Inject delays, short writes, encoding/flush/rotation/close failures and stop with queued/dequeued/buffered work. Verify idle dirty flushing separately from clean polls.
• Exercise failed native seek, repeated/forward/backward seeks, pending recovery, stale callbacks, replaced avatars and exit during warp.
• Check cleanup stage reachability and retained external handlers after unsubscription/cancel/emission failure.
• Audit direct writer game calls separately from eager module imports and mutable-reference handoff.

Runtime matrix — goals to test

Failure modes and edge cases

• Writer dies: caught failures set an uncapped error string and log a traceback. Nothing reads that error or gates future admission on worker health. The initial worker clock read is outside the catch.
• Queue saturation/stopping: false admission increments a carried count; later producer or writer failures are not fully represented.
• Encoding, short write or disk failure: dequeued records have no retry/requeue, write-length check or readback. One failure can abandon the remaining batch.
• Rotation: remove-then-rename stages can lose predecessors without rollback/locking. The post-batch eight-MiB threshold can overshoot; unrelated or older files are not globally capped.
• Process kill/restart: buffered or partial tails may be lost. Complete valid lines can be parsed individually, but append does not repair an old incomplete tail and may join it to new data.
• Replay import absent: an optional ImportError disables both replay imports; other initialization or sampling failures are still possible.
• Finish without start/stale callback: slot guards do not prove one reachable sampler after every interleaving or failure.
• Shutdown: later stages may be skipped; the writer detaches its thread before timed join. Timeout neither cancels I/O nor retains a handle for repeated joining.

Performance considerations

Sampling requests a 0.5-second delay after work, rather than exactly two samples per wall second. Native reads, allocations, clock reads and queue locks have no measured latency bound here. Startup directory/stat/rotation work runs in the caller; the worker handles subsequent encoding/writes and batches up to 128 records. Entry capacity does not bound object or encoded byte size.

The dirty-output flush check uses loop progress and wall-clock time, including empty polls. Scheduling, slow operations and clock changes can delay the one-second condition. Final normal-loop flush/fsync/close is a separate path, not a crash/power-loss receipt. A two-second join request is not a deadline for the entire recorder cleanup and does not terminate a blocked worker.

Selected earlier authored writer-loop checks used an inert queue, clock and recording stream. They did not run threads, wait, write files or verify embedded Python 2 behavior. Historical WG 2.4.1.0 #977 clock/callback observations showed replay-pause stalls and backward client time on rewind; those observations are not recorder tests. Earlier client verification retains its original WG 2.3.1.3 scope.

Compatibility and version sensitivity

Why this architecture?

• JSONL: independently parseable complete valid lines are useful, but framing supplies no tail repair or schema/durability check.
• Fixed queue capacity: limits queued entries while allowing rejection; bytes and worker health need separate policies.
• Worker writing: separates normal producer callbacks from encoding/write work, while startup still performs synchronous filesystem preparation.
• Append and markers: preserve attempted observations within retained files; rotation deletes older history and markers can be lost.
• Rewind generations: invalidate stale work without intentionally creating a new session, while recovered arena/avatar ownership still needs verification.

Possible extensions

• Add a standalone offline validator that checks schema and sequence gaps.
• Add a checksum footer per rotated file.
• Add optional gzip compression after rotation in a separate bounded maintenance worker.
• Add a Gameface recorder-health panel fed only by copied counters.
• Define schema v2 only when a requirement cannot be represented by a new v1 event type.

Maintenance checklist

• Recheck current event factories, inherited behavior, registrations, signatures and firing order.
• Inspect playback/warp/current-time access and player/arena/vehicle readiness independently.
• Test callback counts, stale-slot clearing, identity replacement, cancellation and scheduling failure.
• Stress rejection accounting, worker death, oversized records, partial tails, rotation and timed shutdown.
• Validate actual JSONL schema/finite numbers and distinguish gaps from acknowledged loss.
• Audit fields and diagnostic surfaces for privacy; preserve Python 2.7 and optional Error Reporter attribution.
• Treat the offline validator, checksums, compression, health panel and schema revisions above as proposed extensions.

What you learned

Reliable telemetry requires distinct evidence for envelope attempts, admission, writing, flushing, recovery and cleanup. This example exposes useful mechanisms and concrete limits; its constants, counters and generations are not substitutes for fault testing or delivery receipts.