Emergency maintenance in USEast and Frankfurt.
Incident Report for MageMojo
Postmortem

The issue occurred due to a new build of php-fpm containers. This change was only slated for the UAT environment but was inadvertently released to production. The changes were pushed at approximately 05:30 AM EDT, but were not implemented unless a customer’s php-fpm container was restarted so it was not immediately evident there was any problem. This new container build included support for Datadog, which in addition to largely increasing CPU resources, needed also caused random SIGSEGV errors in php-fpm which resulted in 502 http status errors on sites. Once this issue was identified we rebuilt the affected containers and began restarting php-fpm containers that were contained the Datadog release. We will further investigate how these builds were inadvertently released into production instead of UAT.

Posted Oct 15, 2021 - 14:05 EDT

Resolved
This incident has been resolved.
Posted Oct 15, 2021 - 14:04 EDT
Investigating
We are currently investigating reported issues of 500 and 502s.
Posted Oct 15, 2021 - 10:52 EDT
This incident affected: Mojo Stratus - Northern Virginia and Mojo Stratus - Frankfurt.