Incident Response

[15 min] Confirmed issue affects file uploads. Database connection pool 
at max. Investigating cause.

[30 min] Root cause identified: Database connections exhausted. 
Restarting connection pool. ETA resolution: 15 minutes.

[45 min] Connection pool restarted. Testing uploads. Looks good so far.

[60 min] RESOLVED. Uploads are working again. All clear.

1. Check database server status
2. Restart database service if crashed
3. Check disk space (full disk would crash DB)
4. Check if under attack (unusual connections)
5. Restore from backup if needed
6. Verify data integrity after restore

1. Revert to last known good version
   git revert [bad commit]
   Deploy immediately
2. Do NOT continue debugging
3. Use post-incident review for deep dive
4. Once stable, deploy fixed version

1. Isolate affected systems immediately
   - Block suspicious API keys
   - Kill compromised sessions
   - Disable affected accounts
2. Preserve evidence (logs, files)
3. Notify security team and exec
4. Do NOT publicly admit breach yet
5. Let comms team handle announcements

1. Scale up infrastructure immediately
   - More database replicas
   - More app servers
   - Clear caches
2. Block non-essential features to reduce load
3. Scale down after incident resolves

INCIDENT: We're aware uploads are failing for some users. 
Our team is working on it. Updates every 15 min.

https://status.emberly.ca

Subject: Emberly Service Disruption - We're Fixing It

We're currently experiencing issues with file uploads. Our team is 
working on a fix. We expect to be back to normal within 1 hour.

Thank you for your patience.

Status: https://status.emberly.ca

Subject: [P0 INCIDENT] File uploads down

What's happening:
- Users cannot upload files
- Error message: "Upload failed"
- Started: [time]

What we're doing:
- Debugging database connection issues
- Working on fix
- Will update in 15 minutes

What you should do:
- Keep monitoring Slack #incidents
- Do NOT post to social media yet
- Comms team handles public messaging

# P0 Incident Report: Upload Service Down

## Timeline
- 14:32 - Users report uploads failing
- 14:35 - Confirmed database connection pool exhausted
- 14:42 - Root cause identified: connection leak in upload handler
- 14:55 - Fix deployed, uploads working again

## Root Cause
Connection pool set to 20 max connections. During bursty upload period,
connections accumulated faster than they were closed. No connection 
timeout was set, so stuck connections blocked new uploads.

## Why We Didn't Catch It
- No alerts on connection pool usage
- Load test didn't simulate sustained uploads
- Code review missed the connection leak

## Fixes
1. Add alert when connection pool > 80% (OWNER: [name])
2. Increase timeout on DB connections from infinite → 5min (OWNER: [name])
3. Add load test for sustained uploads (OWNER: [name])
4. Code review checklist should include connection management (OWNER: [name])

## Prevention
All of above by [date]. Follow up review [date].

# Check connection pool status
SELECT count(*) FROM pg_stat_activity;
# If number is close to max_connections setting in postgres.conf
 
# Restart connection pool
# In app code: reconnect() / reload()
 
# Or restart entire database service
sudo systemctl restart postgresql

Is a specific endpoint leaking connections?
Are there long-running queries blocking?
Is there a connection timeout?

# Check disk usage
df -h /data
# If 100%:
 
# Find large files
du -sh /data/* | sort -h | tail
 
# Delete old logs/temp files
rm -rf /data/logs/*
rm -rf /tmp/*
 
# Once space is freed, restart services
sudo systemctl restart postgres
sudo systemctl restart emberly-api

# Revert to previous version
git revert [bad commit] --no-edit
git push
npm run deploy
 
# Once stable, investigate offline
# Do NOT try to fix in production immediately

1. Revoke the leaked API key
   DELETE FROM api_keys WHERE id = [key]

2. Kill all sessions from that key
   DELETE FROM sessions WHERE api_key = [key]

3. Check what the attacker did
   SELECT * FROM audit_logs WHERE api_key = [key]

4. If they accessed data:
   - Note what was accessed
   - Notify affected users
   - Notify legal/security team

5. Rotate other keys as precaution

1. DO NOT write more data
2. Take database backup immediately
3. Isolate the problem
   - Is it one user or many?
   - Is it specific file type?
   - Is it correlated to specific event?

4. Investigate query logs
   SELECT * FROM query_log WHERE timestamp > [incident_time] LIMIT 100

5. Notify executive team immediately

Goal: Safely restart database without data loss

Prerequisites:
- Have SSH access to database server
- Know how to log into postgres
- Have backup of critical data

Steps:
1. Notify team in Slack "Restarting database in 5 min"
2. Wait for confirmations
3. Tell app to close connections
   App API: /admin/maintenance/pause-new-connections
4. Wait 30 seconds for existing connections to drain
5. SSH into database server
6. sudo systemctl stop postgresql
7. Wait 10 seconds
8. sudo systemctl start postgresql
9. Check it started: sudo systemctl status postgresql
10. Tell app to resume connections
    App API: /admin/maintenance/resume-connections
11. Test: Try uploading a file
12. Update Slack: "Database restart complete"

Rollback:
If database won't start:
1. Check logs: sudo journalctl -u postgresql | tail -100
2. If corrupted: Restore from backup
3. If full disk: Delete old logs, try again

Goal: Restart application without losing user data

Prerequisites:
- Multiple app servers running (for zero-downtime)
- Load balancer configured
- Enough replicas to replace one

Steps:
1. Remove one app server from load balancer
2. Wait for existing requests to finish (< 5 min)
3. sudo systemctl restart emberly-app
4. Check health: curl http://localhost:3000/health
5. Add back to load balancer
6. Monitor error rates for 5 min
7. Repeat for other servers one at a time

If something goes wrong:
1. Add server back to load balancer immediately
2. Other servers will handle traffic
3. Debug offline
4. Don't restart again until fixed

[P0 INCIDENT] Service disruption
Issue: [Brief description]
Started: [Time]
Impact: [Users affected]

Updates: https://status.emberly.ca
#incidents on Slack

Still investigating. Database team identified issue could be 
[A], [B], or [C]. Testing fix for [A]. ETA 30 min.

Root cause found: [explanation]. Deploying fix now. 
ETA resolution: 15 minutes.

RESOLVED - Service fully restored. All systems nominal.
Incident report will be published tomorrow.

INCIDENT SUMMARY

What: [Brief description]
Duration: [Start] to [End] = [X minutes]
Impact: [N] users affected
Root cause: [One sentence]
Fix: [What we did]

Post-incident review:
Date: [date]
Findings: Will share in engineering channel

Thanks for your patience!

INCIDENT DECLARED
├─ Slack alert posted
├─ Owner assigned
├─ Status page updated
└─ Updates every 15 min

INVESTIGATING
├─ Check recent changes
├─ Check infrastructure health
├─ Check database
└─ Check logs

FIX IDENTIFIED
├─ Test fix in staging
├─ Deploy (revert if broken)
└─ Verify with real traffic

RESOLVED
├─ Monitor for 30 min
├─ Update status page
├─ Post to Slack
└─ Schedule post-incident review