Production operational runbook for the GLOSS overnight batch processing chain: nominal sequence, failure diagnosis, restart rules, message queues, and SLA management.
- Pre-batch checks: exception queues empty, application locks released, next-day market calendar loaded.
- Ingestion of upstream feeds: trades and allocations from the OMS, securities master data, closing market prices.
- Generation of settlement instructions and dispatch of MT540-543 messages to custodians and CSDs.
- Integration of intraday confirmations and status messages received during the day (MT544-547, sese.024/025).
- Ingestion of custodian statements MT535/MT536.
- Depot, transaction, and cash reconciliations; generation of reconciliation breaks.
- Corporate Actions processing: calculation of entitlements, dispatch of MT564 messages.
- Scheduled reporting and data extractions (client statements, CSDR regulatory reporting, general ledger accounting extracts).
- Working table purges and business date rollover.
Target completion before 05:30 CET (SLA). The batch monitoring console sends a completion notice to the production distribution list.
- Identify the failed step in the batch monitor and review the step execution log (
/logs/gloss/batch/, retained for 90 days).
- Feed rejected during ingestion: check file format and business date; a security missing from master data blocks the entire batch → create the security record or quarantine the record line, then reprocess.
- Message queue latency or lag (inbound or outbound): inspect the SWIFT connector and identify poison-pill messages looping on error; quarantine the offending message before restarting the queue.
- Reconciliation step failure: almost always caused by a missing or partial MT535 statement — consult the reconciliation guide.
- Purge failure: check available disk space and database locks; never force a table purge during core business hours.
- Process stall without progress: check for database deadlocks or user sessions left open on data entry screens.
- Always restart from the failed step, never restart the entire batch chain from the beginning.
- Any feed replay must run in idempotent mode (with the "skip already processed" flag enabled): replaying feeds without this safeguard causes duplicate transaction entries in the stock record.
- Every restart requires prior verification that the target business date has been purged (specifically regarding CSDR penalty records).
- Log all manual restarts in the tracking ticket: step identifier, timestamp, operational reason, and outcome.
¶ SLAs and Drift
- Batch chain finishing after 05:30 CET on two consecutive nights → open a capacity problem ticket and analyze drift step by step (data volumes, database query execution plans).
- Client extractions execute following reconciliation: any batch delay defers client and regulatory reporting — notify Client Services by 06:00 CET if the batch chain has not completed.
- Never execute duplicate instances of the same step in parallel to "speed things up".
- Any modification to scheduling must follow formal change management, never direct edits in production.
- Following an application release, monitor the initial nightly batch execution in real time.
Batch chain not completed by 04:30 CET → escalate to L2 on-call engineering. Inability to dispatch MT54x messages before market opening → notify Settlements Lead and Management immediately. Suspected application bug → open an SS&C vendor ticket with attached step logs.