# Production Migration Runbook

## Scope and authority

This runbook moves the Windows/XAMPP workload to Linux during an approved change window. It does not authorize the change by itself. The legacy listener remains the only production CRM-effect authority throughout migration. CEM CRM ownership, authority interlock activation and broad production side effects are explicitly out of scope.

Roles required: change lead, Windows/source operator, Linux operator, database operator, PBX/UCM operator, network/DNS operator, security/certificate owner and business acceptance owner. Every command, timestamp, artifact hash and decision belongs in the controlled change record; secret values do not.

## Rehearsal gate

Before scheduling production, restore a fresh sanitized or protected production-equivalent export to an isolated Linux clone. Measure dump, transfer, restore, attachment sync, cache rebuild and parity-test durations. Verify the rollback procedure. Resolve every blocker in `PRODUCTION_MIGRATION_READINESS` or record an approved exception.

## T-24 hours

1. Confirm approved window, provisional 2–4 hour outage, maximum rollback-decision time and named decision owners.
2. Lower DNS TTL only through the normal DNS change process if an address switch is needed; record previous value.
3. Confirm current backups, free space and restore rehearsal evidence.
4. Capture safe source inventory: versions, extensions, services/tasks, schema metadata, sizes, checksums and customization diff.
5. Stage the validated integration baseline and MarvelPBX package on Linux; verify hashes.
6. Create target DB/application/service accounts and certificates through approved secret channels.
7. Verify target-to-UCM AMI/HTTPS, DB, browser HTTPS, WebSocket, DNS, NTP and archive/NAS connectivity.
8. Keep authority default `LEGACY`, interlock false, CEM side effects disabled.

## T-2 hours

1. Confirm no unplanned source deployment or schema change.
2. Take a pre-window backup and verify it is readable; do not rely on creation success alone.
3. Perform a non-authoritative incremental attachment sync and record manifest/hash summary.
4. Load reviewed Linux config without displaying secrets; validate ownership and modes.
5. Run `production_readiness.php`, PHP syntax checks and package/regression tests.
6. Confirm Linux services are installed but the AMI listener and write-producing jobs are stopped.
7. Confirm rollback files, source service start commands and DNS reversal are immediately available.

## T-30 minutes

1. Open the change bridge and confirm all owners are present.
2. Announce impending maintenance and stop new user sessions through approved Vtiger maintenance controls.
3. Record source DB/storage sizes and replication status if any; this plan assumes none.
4. Verify no CEM authority snapshot can suppress legacy behavior.
5. Confirm target clock, disk, memory, certificate and ports.
6. Make explicit GO/NO-GO decision to enter freeze.

## Freeze and final export

1. Put Windows Vtiger into maintenance/read-only state.
2. Stop source scheduled tasks and background writers.
3. Stop the source legacy listener so no new PBXManager/popup CRM work is produced there; record time.
4. Verify no active DB-writing process remains.
5. Create the final consistent Vtiger DB dump including reviewed triggers/routines/events/views; hash it.
6. Perform final storage/attachment delta sync with a manifest. Exclude cache, sessions, temp, logs and runtime locks.
7. Preserve the Windows system intact and frozen for rollback.

## Restore and configuration

1. Restore the final DB into the clean target database using the rehearsed procedure.
2. Recreate portable view definers and least-privilege grants; never import source system accounts.
3. Deploy the reviewed Vtiger code/customization set and validated MarvelPBX package.
4. Sync application storage/attachments and set reviewed owner/group/modes.
5. Recreate runtime/cache/log directories empty; clear only documented Vtiger caches.
6. Recreate Vtiger, listener, bridge, recording and read-only integration settings through protected channels.
7. Preserve `application_unique_key` when needed to decrypt existing MarvelPBX settings.
8. Install the approved CA chain/certificate and verify hostname trust.
9. Do not import CEM checkpoints, journals, authority snapshots, leases, pins or PID files.

## Read-only validation before startup

1. Validate schema objects, engines, charset/collation exceptions and view definitions.
2. Reconcile safe row counts for users, contacts, accounts, tickets, PBXManager, `vtiger_crmentity` and attachment references.
3. Run Vtiger/Marvel preflight and PHP lint.
4. Verify recording resolver configuration without exposing credentials or paths.
5. Confirm all service configs point to the Linux target and approved UCM/DB endpoints.

## Service startup

Start in dependency order:

1. required archive/storage mount;
2. MariaDB if local;
3. Apache/PHP;
4. Vtiger scheduled jobs, initially held until interactive smoke tests pass;
5. WebSocket/TCP bridge;
6. legacy AMI listener with legacy authority;
7. CEM follower and Event Journal only in shadow mode;
8. CRM consumer disabled/non-side-effecting;
9. archive/recording workers only after their separate checks.

Never run Windows and Linux legacy listeners concurrently against the same production call flow.

## Smoke and business validation

1. Login and verify role/profile access.
2. Open representative contacts, tickets and attachments.
3. Connect a browser WebSocket client.
4. Place one controlled known-contact call: exactly one enriched ring popup, correct answered state and one PBXManager result.
5. Place one controlled unknown-contact call: one popup, `(Unknown)`, no invented CRM contact.
6. Validate one missed call and its PBXManager status.
7. Open MarvelPBX list/detail views and filters.
8. Play one known recording through the same-origin proxy and verify no path/credential exposure.
9. Verify one NOT_FOUND recording state and default download denial.
10. Confirm logs/audits contain no secrets or caller/contact data.

Queue `6500` / extension `3000` CEM ownership is not activated here. If those facts are checked, they are observation evidence only.

## Traffic switch and observation

If all pre-switch tests pass, switch DNS/IP/load-balancer routing through the approved mechanism. Confirm certificate and WebSocket origin behavior from an external client. Re-run login, popup and recording smoke tests.

Observe intensively for at least the agreed first hour and retain a 24-hour enhanced monitoring window. Track Apache/PHP/DB errors, listener connection, WebSocket clients, popup duplicates/misses, PBXManager creation, scheduled jobs, recording proxy outcomes, disk and queue backlogs.

## Go/no-go decision

The change lead and business owner decide GO only when all mandatory parity checks pass and no rollback trigger is active. Otherwise invoke `PRODUCTION_ROLLBACK_RUNBOOK` before the maximum decision time. Keep Windows frozen until GO acceptance; retire it only under a later approved change.

## Post-checks

- restore intended scheduled jobs without duplicates;
- verify backups of the Linux target and perform a restore/read test;
- restore DNS TTL through normal change control;
- record hashes, timings, validation results and exceptions;
- rotate temporary migration credentials;
- keep legacy CRM authority and schedule CEM pilot separately.