Jomablue - Notice history

During an update to a subset of customer instances, a configuration error in the deployment process caused some environments to become temporarily unavailable. This resulted in affected instances being unable to load or operate as expected.

Once the issue was identified, we halted the rollout and began restoring each impacted instance. Most environments were corrected quickly and all were fully restored.

Our deployment process requires multiple automated and manual human verification steps to avoid situations like this, in this particular case - the third party interface we use to orchestrate deployments was misrepresenting the deployment configuration. We’ve since updated our deployment process to revoke and completely avoid this configuration issue from occurring again, and we’ve added additional monitoring that will alert us immediately if a similar condition is ever detected.

This incident has been resolved. A post mortem will be provided here in due course.

We have addressed the cause and are currently monitoring the result. More details regarding the incident will be published in due course.

We are continuing to resolve the incident, services are restoring.

Our upstream SMS provider temporarily paused outbound SMS delivery after detecting unusual traffic affecting a subset of US numbers. As a result, SMS messages sent from Jomablue were not processed during this period.

Once we became aware of the interruption, we rotated all SMS-related credentials as a precaution and worked with our provider to resume normal operation. SMS sending was unavailable while this rotation was in progress, but service has since been fully restored.

This incident has been resolved. A post mortem will be made available here in due course.

We had addressed the cause, SMS sending has resumed and we are currently monitoring the result. We will provide more information in due course.

We have identified the cause and are bringing SMS services back online.

We have implemented some changes in Capture app to better support Pixel devices and their permission management. Please ensure the latest version is downloaded from the Play or App stores.

We believe we have identified the cause and are implementing a workaround and additional handing for Capture App to improve the reliability of camera launching on Pixel devices.

This incident has been resolved. While there were three seperate issues, only a subset of users - in very specific scenarios that would have been affected by this issue.

There were a number of issues affecting adding the e-Tickets to Apple wallet for a subset of users. Firstly, there was an encoding issue with non-latin based characters and the handling of them within the Apple wallet pass - specifically around two byte characters. This has been resolved.

Secondly, adding e-Tickets to the Apple wallet from Slack's in-app web browser would fail. We found no specific evidence why link handling has changed within Slack's in-app browser but have evidence it has - we have resolved this by changing handling of the 'Adding to Wallet' functionality to continue to support in-app browsers.

Finally, we uncovered that for iOS users with hundreds of Safari tabs open - there is not resources for the device/browser to process the generation and adding of the pass to the wallet. This is some undocumented limitation, but from our testing, exceeding hundreds of tabs would cause the adding of the pass to the wallet to silently fail. Reducing the number of tabs in Safari would resolve the issue.

We have identified the cause and are working on a change. We will provide some more information in the next update.

This has been resolved, all pending emails were queued and now have been delivered. We are working with the upstream provider to produce a autopsy and ensure we can consider how Jomablue can be more resilient to such upstream provider issue.

If you have any further questions we can answer them, just reach out to support.

We have identified an issue with an upstream provider for specific email queues. We have escalated the issue and are working with the upstream provider to resolve.

Jomablue emails are split between Transitional and Broadcast/Marketing emails automatically ensuring high delivery rates across all your email communications. In this particular case, the Broadcast/Marketing emails are affected but transactional emails - such as for invoices, OTP's etc remain unaffected.

We will continue to update here as we learn more and work towards a resolution with our upstream provider.