6/10/2025
What happened:
- A recent deployment inadvertently set our maintenance scheduler to run every 1 second instead of 1 hour, causing continuous background tasks to overload our database—particularly around the Announcements module.
Updated timeline (PDT):
- Monday ~5:00 PM: Performance began to degrade as the mis-scheduled tasks started stacking.
- Overnight: Database CPU fluctuates between 80% to ~99%, leading to slow-downs and occasional timeouts on database-intensive operations.
- Today 8:00 AM: Our engineers pinpointed the scheduler interval error and began working on a fix.
- Today 9:30 AM: The corrected scheduler (one-hour interval) was deployed, and monitoring confirmed full recovery.
Impact:
- Between 5:00 PM yesterday and 9:30 AM today, customers may have experienced intermittent delays or timeouts when using modules that issue complex database queries (notably Announcements).
- No data was lost; all operations completed successfully once the system was back to normal.
We apologize for the disruption and appreciate your patience. If you notice any lingering performance issues or have questions, please reach out to our support team.