One node in USEast under emergency maintenance

Incident Report for Webscale STRATUS

Postmortem

An investigation concluded that a comprehensive kernel bug hit the ZFS filesystem and caused the issue with one of the nodes in our fleet. The problem is identified as similar to the https://github.com/openzfs/zfs/issues/10642 bug already reported. We have captured kernel stack traces during this event, and a solution for prevention is under investigation.

Posted Jun 07, 2021 - 14:20 EDT

Resolved

This incident has been resolved.

Posted Jun 04, 2021 - 14:38 EDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jun 04, 2021 - 14:34 EDT

Identified

The issue has been identified and we are attempting to bring the node back online

Posted Jun 04, 2021 - 12:51 EDT

Investigating

We are currently investigating this issue.

Posted Jun 04, 2021 - 12:28 EDT

This incident affected: Webscale STRATUS - Northern Virginia.