UTHPC - GPU nodes on our Rocket cluster are currently unavailable – Incident details

All systems operational

GPU nodes on our Rocket cluster are currently unavailable

Resolved
Major outage
Started 3 days agoLasted 1 hour 14 minutes

Affected

rocket.hpc.ut.ee

Major outage from 12:09 PM to 12:44 PM, Partial outage from 12:44 PM to 1:24 PM

Services

Partial outage from 12:09 PM to 1:24 PM

Galaxy

Partial outage from 12:09 PM to 1:24 PM

Open OnDemand

Partial outage from 12:09 PM to 1:24 PM

Updates
  • Resolved
    UTC
    Resolved

    The emergency maintenance has been completed, and the GPU nodes on the Rocket cluster are available again.

    Affected jobs have been requeued and will run again as resources become available.

    Thank you for your patience and understanding.

  • Identified
    UTC
    Identified

    GPU nodes on our Rocket cluster are currently unavailable due to a recently disclosed vulnerability. All ongoing jobs on these nodes will be stopped as the nodes undergo emergency maintenance. Once GPU node availability is restored, the affected jobs will be returned to the job queue.

    We apologize for any inconvenience and will inform you when the nodes are available again.

  • Investigating
    UTC
    Investigating

    GPU nodes on our Rocket cluster are currently unavailable due to a recently disclosed vulnerability. All ongoing jobs on them will be killed as the nodes will undergo emergency maintenance. We apologize for any inconvenience and will inform you when the nodes are available for submission.