Jomablue - Platform Authentication and Registration Sites Failing to Load – Incident details

All systems operational

Platform Authentication and Registration Sites Failing to Load

Resolved
Major outage
Started 2 months agoLasted about 2 hours

Affected

Platform

Major outage from 5:55 AM to 7:37 AM, Operational from 7:37 AM to 7:50 AM

Portal

Major outage from 5:55 AM to 7:37 AM, Operational from 7:37 AM to 7:50 AM

Community

Major outage from 5:55 AM to 7:37 AM, Operational from 7:37 AM to 7:50 AM

Registration

Major outage from 5:55 AM to 7:37 AM, Operational from 7:37 AM to 7:50 AM

Live Streaming/Video

Major outage from 5:55 AM to 7:37 AM, Operational from 7:37 AM to 7:50 AM

Kiosk

Major outage from 5:55 AM to 7:37 AM, Operational from 7:37 AM to 7:50 AM

Updates
  • Postmortem
    Postmortem

    During an update to a subset of customer instances, a configuration error in the deployment process caused some environments to become temporarily unavailable. This resulted in affected instances being unable to load or operate as expected.

    Once the issue was identified, we halted the rollout and began restoring each impacted instance. Most environments were corrected quickly and all were fully restored.

    Our deployment process requires multiple automated and manual human verification steps to avoid situations like this, in this particular case - the third party interface we use to orchestrate deployments was misrepresenting the deployment configuration. We’ve since updated our deployment process to revoke and completely avoid this configuration issue from occurring again, and we’ve added additional monitoring that will alert us immediately if a similar condition is ever detected.

  • Resolved
    Resolved

    This incident has been resolved. A post mortem will be provided here in due course.

  • Monitoring
    Monitoring

    We have addressed the cause and are currently monitoring the result. More details regarding the incident will be published in due course.

  • Identified
    Identified

    We are continuing to resolve the incident, services are restoring.

  • Investigating
    Investigating
    We are currently investigating this incident.