UTHPC - Network issue causing timeouts in HPC and Cloud – Incident details

All systems operational

Network issue causing timeouts in HPC and Cloud

Resolved
Partial outage
Started over 2 years agoLasted 2 hours 25 minutes

Affected

Cluster

Partial outage from 2:55 PM to 3:38 PM, Operational from 3:38 PM to 5:20 PM

rocket.hpc.ut.ee

Partial outage from 2:55 PM to 3:38 PM, Operational from 3:38 PM to 5:20 PM

ondemand.hpc.ut.ee

Partial outage from 2:55 PM to 3:38 PM, Operational from 3:38 PM to 5:20 PM

Services

Partial outage from 2:55 PM to 3:38 PM, Operational from 3:38 PM to 5:20 PM

galaxy.hpc.ut.ee

Partial outage from 2:55 PM to 3:38 PM, Operational from 3:38 PM to 5:20 PM

UT HPC webservices

Partial outage from 2:55 PM to 3:38 PM, Operational from 3:38 PM to 5:20 PM

Updates
  • Resolved
    UTC
    Resolved

    Marking this incident as resolved.

  • Identified
    UTC
    Identified
    A hardware issue has been found with one of the infiniband switches, systems seem to be returning to normal now. There might be some slowness due to previous timeouts playing themselves out. We will continue to monitor the situation.
  • Investigating
    UTC
    Investigating
    There's an infiniband network issue causing distributed filesystem operations to wait, which is causing cloud VMs and HPC not to answer properly.