This service interruption is now resolved. The interruption was caused by a bug in our database connection pooling which ultimately resulted in Watershed LRS becoming unresponsive. We have deployed initial mitigation, additional monitoring, and a potential fix. We will write a full postmortem with more details in the coming days. We apologize for the inconvenience.
Posted Sep 18, 2019 - 17:54 CDT
Update
Engineers are continuing to work on addressing unexpected load on our servers.
Posted Sep 18, 2019 - 11:37 CDT
Investigating
Engineers are working to address unexpected load on our servers.
Posted Sep 18, 2019 - 09:36 CDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Sep 18, 2019 - 09:15 CDT
Identified
The issue has been identified and a fix is being implemented.
Posted Sep 18, 2019 - 08:35 CDT
Investigating
Engineers are investigating an unexpected service interruption
Posted Sep 18, 2019 - 07:44 CDT
This incident affected: Web Servers and Analytics Database.