We saw a node reboot twice due to a kernel panic, and we are working on reproducing the issue on our development build to confirm our fixes will work. This particular panic is fixed in a newer kernel version. The issue is not frequent and the root cause is a specific task load.

‌

We are also working with AWS to get fixes related to debugging software that does not work correctly on their instances (that is an AWS bug). ENA devices lack proper driver support from AWS needed to capture proper console and other logging output leading up to any panic, so data is lost. AWS has confirmed this is an issue and their support is working closely with our development team. Delays from AWS are why this is taking a long time to correct what would otherwise be straightforward to investigate.