# Production Migration Rollback Runbook

## Principle

Rollback restores service to the frozen Windows source; it is not a bidirectional synchronization procedure. The legacy listener remains the CRM-effect authority. Rollback must not leave both Windows and Linux listeners, schedulers or popup owners active.

Set the maximum rollback decision time before the window. Recommended initial bound is **60 minutes after traffic switch**, subject to business RTO approval and rehearsal measurements.

## Immediate rollback triggers

- login or core CRM access unavailable;
- failed database integrity/count checks or material charset corruption;
- missing attachments or unacceptable data loss;
- duplicate or absent production popups/call activities;
- PBXManager records missing, duplicated or materially wrong;
- recording proxy exposes paths/credentials or cannot serve required recordings;
- TLS, DNS, WebSocket or UCM connectivity cannot be corrected within the decision bound;
- unexplained Linux writes after a partial failure;
- security control, secret handling or audit failure;
- business acceptance owner rejects parity.

## Emergency containment

1. Declare rollback and record exact time/reason.
2. Re-enter maintenance mode; stop new Linux user writes.
3. Stop Linux legacy listener first, then CEM CRM consumer, CEM follower/journal if necessary, WebSocket bridge and write-producing scheduled/archive jobs.
4. Preserve Linux logs, DB and runtime evidence read-only; do not delete or overwrite it.
5. Confirm no Linux process can emit popup/CRM effects.
6. Restore routing/DNS/load-balancer target to Windows using the prepared reversal.
7. Restore the prior DNS TTL later through change control; account for resolver caches during validation.

## Database divergence decision

If Linux accepted no writes after the final dump, keep its database for evidence and reopen the frozen Windows system.

If Linux accepted writes, do **not** simply reopen the older Windows DB and discard the delta. Keep both sides frozen, quantify changes in users, contacts, tickets, PBXManager, settings and attachments, and obtain business/data-owner approval for one of:

- manually replaying a small audited delta into Windows;
- importing an application-supported, reviewed delta;
- extending the outage and repairing Linux instead of rolling back.

There is no proven automatic merge or reverse replication. PBXManager/call records created during the Linux interval require explicit duplicate-aware reconciliation.

## Restore Windows authority

1. Verify Windows DB/application/storage remained unchanged and consistent during the Linux interval, aside from an approved reconciled delta.
2. Start Windows Apache/Vtiger and required scheduled tasks.
3. Start the Windows WebSocket bridge.
4. Start exactly one Windows legacy AMI listener.
5. Keep all CEM authority default `LEGACY`, interlock false and CEM side effects disabled.
6. Confirm the Linux listener and popup bridge cannot compete before accepting calls.

## Recording continuity

Recordings remain on UCM/NAS, so application rollback does not move media bytes. Verify Windows resolver/playback configuration and network access. Do not write physical recording URLs or copy media as a rollback shortcut. Preserve any Linux recording audit as evidence without exposing filenames, paths or credentials.

## Rollback validation

1. Login and open representative CRM data/attachments.
2. Place one controlled known-contact call and verify one popup and one PBXManager result.
3. Place an unknown-contact call only if required; verify no false match.
4. Verify answered/missed behavior.
5. Validate MarvelPBX list/detail and one known recording, if the source supports it.
6. Confirm Windows jobs run once and Linux equivalents remain stopped.
7. Monitor DB errors, popup duplicates, listener connections and attachment access.

## After rollback

- retain final exports, Linux DB, manifests, hashes and logs under restricted evidence retention;
- document exact divergence and any reconciled records;
- restore normal DNS TTL after caches converge;
- rotate temporary credentials if exposure is possible;
- perform a root-cause review and another timed rehearsal before rescheduling;
- do not resume CEM cutover planning until platform migration failure is resolved.

## Rollback material checklist

- untouched/frozen Windows application and database;
- verified pre-window and final backup artifacts;
- source service/task start procedures;
- previous DNS/IP/load-balancer and TLS configuration;
- Linux stop commands and service dependency list;
- attachment manifest and final-sync report;
- named database/business/security decision owners.