No problems detected.

At 10:30pm AEST July 30, 2026, we experienced a service disruption that affected some users logging into Paytools.

What Happened

A routine update to a background software component (a "dependency") inadvertently removed a library required for processing images and documents. This caused the system to encounter errors whenever a user tried to view an attachment.

Technical Impact

These errors caused the web server processes to become "stuck" while trying to process the requests. Because the processes remained open but unresponsive, the system eventually ran out of available capacity to handle new requests, leading to intermittent "503 Service Unavailable" errors for some users.

Resolution

Our engineering team identified the missing library and restored the correct software version. We then cycled our servers to clear the stuck processes and restore full capacity.

Improvements

To prevent this from happening again, we are:

Enhancing Health Checks: Updating our automated monitoring to specifically test attachment rendering so we can detect similar issues faster.

Improving Timeouts: Adjusting how our servers handle long-running requests to ensure that a single error cannot exhaust the system's total capacity.

Refining Update Procedures: Adding additional verification steps for third-party software updates to ensure all required components remain present.