# CEM CRM POPUP_RING Cutover Readiness Runbook This runbook prepares a narrowly scoped `POPUP_RING` ownership pilot. It does not authorize production cutover. The legacy listener remains authoritative until every blocker in this document is resolved and a separately reviewed change approves the pilot. Related architecture: MPBX-002, MPBX-004, MPBX-006 and MPBX-CONF-001. ## Current readiness blockers Do not activate CEM ownership while any of these conditions remains: 1. **Queue-scope asymmetry (conditionally mitigated in Phase 9C2.2):** CEM now evaluates a queue-and-extension execution scope while legacy evaluates a separately declared extension suppression scope. Activation is permitted only with reviewed evidence that the extension is dedicated to that queue throughout the pilot. Without that operational proof, this remains a blocker. 2. **Runtime-health lease (resolved in Phase 9C2.1):** legacy suppression now requires a fresh, ready lease bound to the exact authority generation. A stopped, disabled, adapter-disabled, unhealthy or unready consumer therefore leaves ownership with legacy. 3. **In-flight generation race (resolved in Phase 9C2.3):** a short-lived atomic action pin binds legacy and CEM decisions for one ring to the same immutable authority generation. Rotation changes the next action, not the active action. Queue asymmetry is mitigated only under the explicit dedicated-extension constraint; without reviewed exclusivity it remains a mutual-exclusion defect. Monitoring alone does not prove exclusivity. ## Authority health lease Authority alone proves reviewed ownership intent; it does not prove that the CEM runtime can execute. The follow-mode consumer therefore refreshes an atomically replaced `0600` current-state lease every five seconds with a default 15-second validity. The lease is bound to the authority `generation_id`, tenant and environment. It is ready only when the consumer, general adapter, popup adapter, journal, caught-up checkpoint boundary and passive bridge checks are healthy and the authority interlock is active. The bridge check opens and closes a TCP connection without sending a frame. It proves local bridge reachability, not browser display. Dry-run, CRM preflight and authority export never write the lease. Controlled shutdown writes a non-ready lease; abrupt failure stops refresh, so legacy resumes no later than lease expiry. Missing, malformed, stale, integrity-failed, permission-unsafe or generation-mismatched health state always preserves legacy ownership. ## Dual-scope pilot constraint The reviewed authority rule must contain both scopes under one rule and generation. The execution scope includes the canonical queue and extension. The suppression scope repeats tenant, environment, `POPUP_RING` and extension and drops only the queue. Export must fail if the rule omits its `DEDICATED_EXTENSION_TO_EXECUTION_QUEUE` constraint or safe evidence reference. Before activation, PBX configuration and controlled evidence must establish that the extension cannot receive ring calls from another queue or non-queue path during the window. If that cannot be guaranteed, do not export the rule. The action-generation pin coordinates authority generation but does not prove queue eligibility. ## Phase 9C2.4 exclusivity evidence result The locally available controlled captures contain four explicitly classified rings to extension `3000`, all carrying authoritative queue `6500`, and three older rings classified `unknown` because those captures predate explicit `Queue` retention. No ring was explicitly classified as another queue or a direct/non-queue route. Absence of such a captured call is not exclusivity proof. No local reviewed UCM configuration export was available to prove all of the following negative conditions: no direct DID target, no ring-group membership, no other queue membership, no IVR direct-transfer route, no internal/direct popup-capable path, no follow-me/forwarding target, and no alternate routed path to extension `3000`. The legacy listener itself is not queue-restricted at its ring side-effect boundary: an initialized inbound call with a `DialBegin` to an eligible extension can produce `POPUP_RING`. The Phase 9C2.4 decision is therefore `EXCLUSIVITY_NOT_PROVEN`. Do not activate the dual-scope CEM rule until a same-window reviewed PBX configuration export and immediate pre-check establish the controlled-window constraint below. ## Controlled-window exclusivity gate Immediately before an approved pilot, an operator must produce a timestamped, sanitized review record confirming: 1. extension `3000` is a member of queue `6500` for the pilot; 2. it is absent from every other queue and ring group; 3. no inbound DID, IVR option, direct transfer rule or alternate inbound route targets it; 4. follow-me and forwarding are disabled; 5. internal/direct calls that could enter the legacy inbound-popup path are prohibited during the announced window; 6. recent AMI evidence contains explicit `Queue=6500` for every observed ring to `3000`, with zero `other`, `direct/non-queue` or `unknown` classifications; 7. the configuration remains frozen for the window and is checked again immediately after rollback. If any item is unknown, the gate fails and authority remains `LEGACY`. ## Ownership reference | Snapshot/result | Legacy behavior | CEM behavior | | --- | --- | --- | | `LEGACY` | Executes existing effect | Plans/audits; no effect | | `SHADOW` | Executes existing effect | Plans/audits; no effect | | `BLOCKED` | Suppresses effect | Suppresses effect | | `CEM`, interlock false | Executes existing effect | No effect | | `CEM`, interlock true, all CEM gates healthy | Suppresses only a symmetrically matched scope | May execute once | | Missing, unreadable, corrupt, expired or integrity-failed snapshot | Falls back to legacy | Resolves `BLOCKED` | CEM execution additionally requires consumer enabled, adapter enabled and idempotency/reconciliation permission. Those gates are not currently visible to the legacy reader. ## Pre-checks Before any future approved pilot: 1. Verify the reviewed baseline commit/tag and a clean worktree. 2. Run the complete regression suite. 3. Confirm the consumer is running and caught up to the Event Journal. 4. Confirm Vtiger read-only preflight succeeds. 5. Confirm the TCP/WebSocket bridge is listening and healthy without sending a production popup. 6. Confirm the popup adapter is configured and healthy using a synthetic local probe approved for the environment. 7. Confirm idempotency state contains no unresolved `EXECUTING` action. Any such action requires reconciliation; it must not be blindly retried. 8. Confirm authority default is `LEGACY`, every non-popup action remains `LEGACY`, and no global CEM rule exists. 9. Confirm a rollback configuration and command are ready in the same terminal/session. 10. Record the controlled-window exclusivity checklist above and require zero unknown items. 11. Verify the Phase 9C2.1 health lease is fresh, ready and generation-matched. 12. Verify the active authority generation/checksum and Phase 9C2.3 action-pin runtime permissions. 13. Confirm the bridge passive health check succeeds and the consumer checkpoint is caught up to the last complete journal record. 14. Without reviewed exclusivity evidence, stop before exporting CEM authority. ## Enable sequence This sequence is documentation for a future approved controlled pilot; it must not be run without reviewed dedicated-extension evidence. 1. Start the consumer in follow mode and verify it is healthy and caught up. 2. Verify the popup adapter and WebSocket bridge are healthy without sending a popup frame. 3. Keep activity, answered-popup and recording ownership `LEGACY`. 4. Prepare one exact rule only: tenant `dev-local`, environment `DEV`, action `POPUP_RING`, queue `6500`, extension `3000`, mode `CEM`. 5. Enable the separate cutover interlock only after the shared health/readiness lease is valid. 6. Export a fresh, atomically replaced authority snapshot. 7. Verify generation ID, checksum, restrictive permissions and expiry margin. 8. Reconfirm the exported authority generation matches the fresh health lease. 9. Make exactly one controlled inbound queue call. 10. Confirm one canonical ring with queue `6500` and extension `3000`. 11. Confirm one authority owner, one transport attempt, one browser popup and one `COMMITTED` action. 12. Confirm the legacy observation reports suppression for the same logical action and no legacy popup frame exists. 13. Export the prepared LEGACY/interlock-false rollback immediately after evidence collection. 14. Make a separate controlled call and confirm the legacy popup resumes while CEM executes nothing. 15. Stop and use emergency rollback immediately if any evidence is missing or contradictory. ## Validation sequence Required evidence for the pilot call: - one canonical ringing event with expected tenant, environment, queue and extension; - one planned and authorized `POPUP_RING` action; - one transport attempt and one transport success; - one `COMMITTED` idempotency record; - no `reconciliation_required` state; - one legacy suppression observation; - one bridge ring frame; - one browser popup; - no legacy popup frame for the same action; - replay/restart produces no second transport attempt. ## Rollback sequence Preferred rollback is an atomically exported safe snapshot from reviewed local configuration: ```text authority.default = LEGACY authority.rules = [] cutover_interlock_enabled = false php tools/cem_crm_consumer.php \ --config \ --export-authority-state ``` Verify the new generation is readable and that the next action resolves `LEGACY`. An already pinned action finishes under its immutable prior generation; the next action uses the rollback generation. No listener restart is required. ## Emergency rollback If a safe snapshot cannot be exported, remove the authority pathname from service using a reviewed local filesystem operation. Missing state restores legacy behavior and blocks CEM. Preserve the old artifact for audit where possible, and verify legacy ownership immediately. Do not delete idempotency or journal state. Emergency rollback changes the active generation for new actions. An already pinned action remains deterministic under its prior generation. Observe both runtimes and reconcile any action left `EXECUTING` before resuming a pilot. ## Post-checks - Confirm the snapshot is back to `LEGACY` with interlock false. - Confirm the consumer cannot execute the pilot scope. - Confirm the legacy popup resumes on a separately controlled call. - Preserve authority, action, idempotency, bridge and observation evidence. - Confirm no activity, recording or answered-popup ownership changed. - Record generation IDs and decision timestamps without caller or CRM personal data. ## Snapshot lifecycle recommendation The current validity is 300 seconds and must not be increased automatically. After the coordination blockers are resolved, use an export cadence of 60–90 seconds, warn when remaining validity falls below 120 seconds and treat less than 60 seconds as critical. Export through the existing temporary-file, flush/fsync and atomic-rename writer. Alert on generation stagnation, checksum failure, unsafe permissions, parse failure or expiry. Do not depend on expiry as the normal rollback mechanism. Expiry is a fail-safe. Generation rotation is coordinated by the short-lived action pin; uncertain adapter `EXECUTING` state still requires reconciliation. ## Monitoring and alert thresholds | Metric/evidence | Initial alert recommendation | | --- | --- | | `authority_execution_blocked` | Warn on any pilot-scope occurrence; critical if ownership is expected to be CEM | | `popup_transport_attempts` vs `popup_transport_success` | Critical on any unexplained difference | | `popup_transport_retryable_failures` | Warn on first; critical if unresolved after bounded retry | | `popup_transport_terminal_failures` | Critical on first | | `popup_duplicates_suppressed` | Observe every occurrence; alert if not attributable to deliberate replay | | `actions_committed` | Must equal expected controlled calls, never exceed them | | `reconciliation_required` | Critical on first; halt cutover | | `authority_state_unsafe` | Critical while pilot expected active; verify legacy fallback | | `health_lease_refresh_failures` | Warn on first; critical if a ready lease is expected | | health lease remaining validity | Warn below 10 seconds; critical below 5 seconds; legacy must retain ownership on expiry | | snapshot remaining validity | Warn below 120 seconds; critical below 60 seconds | | legacy suppression vs CEM commit | Critical if either side lacks matching evidence | External monitoring is not implemented by this phase. Operators must collect these local summaries and audit records during any future controlled validation. ## Known failure modes - dedicated-extension evidence is wrong or becomes stale, allowing a non-execution-queue call to enter the broader suppression scope; - missing or unready health lease restores legacy ownership, intentionally preventing suppression while CEM is unavailable; - missing/corrupt action-pin or immutable generation state restores legacy and blocks CEM; - bridge accepts a frame but browser does not display it; - interrupted `EXECUTING` action requires manual reconciliation; - expired/corrupt snapshot restores legacy but may interrupt an in-flight CEM decision; - journal/checkpoint unready state prevents a qualifying health lease and leaves ownership with legacy. Phase 9C2.1 resolves the consumer-health no-owner blocker. Phase 9C2.3 resolves mixed-generation decisions for one ring action. Phase 9C2.2 remains conditional: `CUTOVER_READY_FOR_CONTROLLED_PILOT` requires concrete reviewed evidence that extension `3000` receives pilot `POPUP_RING` calls only from queue `6500`; absent that evidence, broader readiness remains `CUTOVER_NOT_READY`.