UTHPC - Kubernetes network connections – Incident details

All systems operational

Show current status

Kubernetes network connections

ResolvedDegraded performance1 hour 31 minutes

Affected

Services

Degraded performance from 2:15 PM to 3:21 PM, Operational from 3:21 PM to 3:46 PM

kubernetes.hpc.ut.ee

Degraded performance from 2:15 PM to 3:21 PM, Operational from 3:21 PM to 3:46 PM

registry.hpc.ut.ee

Degraded performance from 2:15 PM to 3:21 PM, Operational from 3:21 PM to 3:46 PM

ETAIS

Degraded performance from 2:15 PM to 3:21 PM, Operational from 3:21 PM to 3:46 PM

minu.etais.ee

Degraded performance from 2:15 PM to 3:21 PM, Operational from 3:21 PM to 3:46 PM

UT HPC webservices

Degraded performance from 2:15 PM to 3:21 PM, Operational from 3:21 PM to 3:46 PM

  • ResolvedUTC

    The issue was related to the amount of connections to/from Kubernetes, our firewalls were configured to not allow that high connection rate, but as we are adding new nodes to the cluster, the base rate is exceeding the limits. The limits have been revisited, and new monitoring is being added to not have the same issue in the future.

  • MonitoringUTC
    We implemented a fix and are currently monitoring the result.
  • InvestigatingUTC

    We're currently seeing higher request error and request latency rates to applications in Kubernetes. Working on trying to find a cause.