Inventory Ops Incident Response Playbook

Why You Need an Incident Response Playbook

Inventory operations incidents are not "if" events — they are "when" events. Sync pipelines fail. WMS systems crash. Suppliers ship wrong products. Marketplace APIs change without warning. The difference between a minor disruption and a major crisis is whether your team has a rehearsed response or is improvising under pressure.

A playbook does three things: it reduces response time (the team knows exactly what to do), it prevents secondary damage (containment happens before diagnosis), and it captures learning (post-incident reviews prevent recurrence).

Incident Classification Matrix

Severity	Definition	Response Time	Examples
Critical (P1)	Revenue-stopping; all orders or all channels affected	Respond within 15 minutes; all-hands until resolved	WMS completely down, data corruption pushed to all channels, OMS cannot process orders, complete sync failure across all platforms
High (P2)	Revenue-impacting; one major channel or system degraded	Respond within 1 hour; dedicated owner until resolved	Amazon sync failed for 2+ hours, one warehouse location cannot ship, supplier shipment arrived damaged (50%+ of units), single-channel inventory corruption
Medium (P3)	Operational degradation; no immediate revenue impact	Respond within 4 hours; resolve within 24 hours	Sync latency increased 3x, minor data quality issues on one channel, single SKU inventory discrepancy, reporting pipeline delayed

Incident Response Procedures

Procedure 1: Inventory Sync Pipeline Failure

Trigger: Sync monitoring detects no updates for > 30 min on any channel

CONTAIN (First 5 minutes):
  1. Identify which channel(s) are affected
  2. Check: is the source system (OMS/ERP) still generating events?
     YES → The pipeline between source and channel has failed
     NO  → The source system has an issue (escalate to engineering)
  3. For affected channels: if stock is low on any SKU, set those SKUs
     to zero inventory on the affected channel to prevent overselling
  4. Do NOT set all inventory to zero unless the outage will exceed 2 hours

DIAGNOSE (Minutes 5-30):
  1. Check event bus health (Kafka/RabbitMQ/EventBridge dashboard)
  2. Check channel adapter logs for errors
  3. Check platform status page for API outages
  4. Check authentication tokens (expired?)
  5. Check rate limit status (throttled?)

RESOLVE:
  If event bus failure → restart consumers, verify event replay
  If adapter error → fix error, redeploy adapter, trigger full reconciliation
  If platform API outage → wait for platform recovery; increase buffer on channel
  If auth token expired → refresh token, verify permissions
  If rate limited → reduce push frequency; queue updates for gradual push

VERIFY:
  1. Confirm sync is flowing again (freshness metric recovering)
  2. Run immediate full reconciliation for affected channel
  3. Check for overselling incidents during the outage window
  4. Monitor for 2 hours to confirm stability
      

Procedure 2: Inventory Data Corruption

Trigger: Distribution anomaly detected (mass inventory changes outside normal range)

CONTAIN (First 5 minutes):
  1. IMMEDIATELY pause all outbound inventory sync to all channels
  2. Identify the scope: how many SKUs are affected? Which system pushed the bad data?
  3. If bad data has already reached channels:
     → For channels with corrupted high values: set affected SKUs to zero
        (prevent overselling on phantom inventory)
     → For channels with corrupted low values: no immediate action needed
        (customers see out of stock, not overselling risk)

DIAGNOSE:
  1. Pull the lineage log: which system generated the corrupted data?
  2. Compare current values to last known good state (backup or last reconciliation)
  3. Identify the root cause:
     → Import/migration error (wrong file, wrong format)
     → API schema change (field name or type changed)
     → Manual bulk update gone wrong
     → Supplier feed corruption

RESTORE:
  1. Roll back affected SKUs to last known good values (from backup/snapshot)
  2. Push corrected values to all channels
  3. Run full reconciliation across all channels

VERIFY:
  1. Spot-check 20 affected SKUs: does system match physical?
  2. Verify all channels reflect corrected values
  3. Check for orders placed during the corruption window
  4. If overselling occurred: cancel affected orders, notify customers, initiate remediation
      

Procedure 3: Warehouse / WMS Outage

Trigger: WMS system unresponsive or warehouse cannot process orders

CONTAIN:
  1. Stop importing new orders from all channels (prevent unfulfillable queue)
  2. Determine outage scope:
     → Complete WMS failure → All orders affected
     → Partial (one location) → Reroute to other locations if available
  3. Notify fulfillment team and 3PL (if applicable)

TRIAGE:
  If single-location outage with multi-warehouse setup:
    → Reroute orders to functioning warehouses via DOM override
    → Accept higher shipping costs temporarily
    → Reduce channel inventory for affected location

  If complete WMS failure:
    → Estimate time to recovery
    → If < 4 hours: hold orders, extend handling time on channels
    → If > 4 hours: reduce channel inventory to slow incoming orders
    → If > 24 hours: consider temporarily pausing listings on highest-volume channels

COMMUNICATION:
  → Internal: Notify customer service team with talking points
  → Customers with pending orders: Proactive email if delay > 24 hours
  → Channels: Extend handling time if platform allows
      

Procedure 4: Supplier Shipment Failure

Trigger: Supplier shipment arrives damaged, short, or with wrong products

ASSESS:
  1. Document the issue (photos, count discrepancy, wrong items received)
  2. Calculate impact:
     → How many sellable units did you receive vs. expected?
     → How many days of stock does this represent?
     → Which channels are at risk of stockout?

RESPOND:
  If shortage is < 20% of PO:
    → Accept partial receipt
    → Adjust inventory for received quantity only
    → File claim with supplier for shorted units
    → Evaluate: do you need an emergency reorder?

  If shortage is > 20% of PO or wrong products:
    → Contact supplier immediately for resolution
    → Determine if supplier can ship replacement within 72 hours
    → If not: identify alternate supplier or adjust channel availability
    → Reduce safety stock trigger for affected SKUs (reorder earlier)

  If damage:
    → Quarantine damaged units (do not mix with sellable stock)
    → Document for insurance/supplier claim
    → Disposition: can any units be refurbished? Or total loss?
      

Communication Templates

Internal: Incident Notification

Subject: [P1/P2/P3] Inventory Incident — [Brief Description]

Severity:    [Critical/High/Medium]
Started:     [Timestamp]
Impact:      [Which channels/systems are affected]
Status:      [Investigating/Contained/Resolving/Resolved]

What happened:
  [2-3 sentences describing the incident]

Current actions:
  [What the team is doing right now]

Customer impact:
  [Orders affected, estimated delay, channels impacted]

Next update:
  [Time of next status update]

Incident owner: [Name]
      

Customer: Order Delay Notification

Subject: Update on your order #[ORDER_NUMBER]

Hi [CUSTOMER_NAME],

We're reaching out because your order for [PRODUCT_NAME] will be
delayed by approximately [X] business days beyond our original estimate.

We experienced a [brief, honest explanation — e.g., "warehouse system
issue" or "supply chain disruption"] that has temporarily affected
our fulfillment operations.

Your new estimated ship date is [DATE].

To make up for the wait, we'd like to offer you [REMEDY — e.g.,
"free expedited shipping" or "15% off your next order with code SORRY15"].

If you'd prefer to cancel your order, you can do so instantly by
replying to this email or clicking here. A full refund will be
processed within 24 hours.

We appreciate your patience and apologize for the inconvenience.

[TEAM NAME]
      

Post-Incident Review Framework

Post-Incident Review (Complete within 48 hours for P1/P2)

1. TIMELINE
   [Minute-by-minute reconstruction]
   - When did the incident start?
   - When was it detected?
   - When was it contained?
   - When was it resolved?
   - Detection gap: [time from start to detection]

2. ROOT CAUSE
   - What failed?
   - Why did it fail?
   - Was this a known risk or a new failure mode?

3. IMPACT
   - Orders affected: [count]
   - Revenue lost: [$]
   - Customers impacted: [count]
   - Channels affected: [list]
   - Duration: [total time from start to resolution]

4. WHAT WENT WELL
   - [What worked in the response]

5. WHAT COULD IMPROVE
   - [What was slow, confusing, or missing]

6. ACTION ITEMS (Preventive)
   - [Specific changes to monitoring, process, or systems]
   - Owner: [Name]
   - Due date: [Date]
      

Monitoring and Alert Configuration

Effective incident response starts with fast detection. Configure these alerts to catch incidents early:

• Sync freshness alert: No successful sync for any channel in >2x expected interval → P3 (auto-escalate to P2 after 1 hour)
• Volume anomaly alert: Inventory update volume drops >50% below 7-day average → P3
• Distribution anomaly alert: Any single update changes >20% of SKU counts by >50% → P2 (potential data corruption)
• Zero inventory spike: More than 10% of active SKUs go to zero in a single update cycle → P1 (likely corruption)
• Order rejection rate: More than 5% of orders unfulfillable in any 1-hour window → P2
• WMS health check: WMS API unresponsive for >5 minutes → P2 (auto-escalate to P1 after 15 minutes)

Common Mistakes

• Diagnosing before containing: The natural instinct is to figure out what went wrong before acting. In inventory incidents, every minute of investigation without containment is a minute where bad data can cause overselling or orders can queue up without fulfillment capability. Contain first.
• Not pausing sync during data corruption: If corrupted inventory data has been generated, pausing outbound sync prevents the corruption from spreading to channels. This is the single most important containment action for data quality incidents.
• Skipping post-incident reviews for "small" incidents: Today's P3 incident that nobody reviewed becomes next month's P1 incident. Review all High severity incidents and sample Medium incidents monthly.
• Blaming individuals in post-incident reviews: Blameless reviews produce honest analysis. Blame-driven reviews produce defensive teams that hide problems instead of surfacing them. Focus on systems and processes, not people.