UTHPC - Newly Disclosed Linux kernel Security Incident – Temporary Service Interruptions – Incident details

All systems operational

Newly Disclosed Linux kernel Security Incident – Temporary Service Interruptions

Resolved
Major outage
Started 1 day agoLasted about 24 hours

Affected

rocket.hpc.ut.ee

Major outage from 7:24 AM to 7:24 AM, Under maintenance from 7:24 AM to 11:13 AM, Major outage from 11:13 AM to 6:54 AM

UT HPC webservices

Degraded performance from 7:24 AM to 7:24 AM, Major outage from 7:24 AM to 11:13 AM, Operational from 11:13 AM to 6:54 AM

hpc.ut.ee

Degraded performance from 7:24 AM to 7:24 AM, Major outage from 7:24 AM to 11:13 AM, Operational from 11:13 AM to 6:54 AM

docs.hpc.ut.ee

Degraded performance from 7:24 AM to 7:24 AM, Major outage from 7:24 AM to 11:13 AM, Operational from 11:13 AM to 6:54 AM

support.hpc.ut.ee

Degraded performance from 7:24 AM to 7:24 AM, Major outage from 7:24 AM to 11:13 AM, Operational from 11:13 AM to 6:54 AM

registry.hpc.ut.ee

Degraded performance from 7:24 AM to 7:24 AM, Major outage from 7:24 AM to 11:13 AM, Operational from 11:13 AM to 6:54 AM

Updates
  • Resolved
    UTC
    Resolved

    All services previously affected by the Linux kernel security vulnerability impacting the XFS filesystem were restored by midnight on 23 July.

    The current service status is as follows:

    • All SAPU systems have been restored and are fully available.

    • Web servers are up.

    • Galaxy is available.

    • OpenOndemand and its services, RStudio and Jupyter, are available.

    • The Rocket Cluster is available for compute workloads.

      • A small number of older Rocket Cluster V100 GPU nodes remain temporarily isolated while they undergo the final upgrade. These nodes are expected to be returned to the queue later today.

    Thank you for your patience while we implemented the necessary mitigations to ensure the security and stability of our services.

  • Update
    UTC
    Update

    We have made progress in addressing the recently disclosed Linux kernel security vulnerability.

    The following services have now been restored and are available:

    • Cloud Service

    • Web Server (Website Hosting Service)

    Mitigation work is still ongoing for the following services:

    • Rocket Cluster and related services such as Galaxy, Open OnDemand

    • SAPU

    These services will remain unavailable while we complete the required patching and validation to ensure they can be restored safely.

    We will continue to provide updates as additional services become available.

  • Identified
    UTC
    Identified
    We are continuing to work on a fix for this incident.
  • Investigating
    UTC
    Investigating

    We are currently responding to a newly disclosed Linux kernel security vulnerability affecting the XFS filesystem. As a precaution, we are implementing mitigation measures across affected infrastructure.

    Affected services:

    • Web Servers

    • Rocket Cluster, along with OpenOndemand

    • SAPU

    • Cloud

    • Galaxy

    To reduce risk while mitigation work is underway, Cluster access has been temporarily disabled. You may experience temporary service interruptions across the affected services during this maintenance.

    Our teams are actively applying mitigations and validating service integrity. We will provide further updates as work progresses and restore normal access as soon as it is safe to do so.

    Thank you for your patience and understanding.