Summary

On 2026-10-07, between 19:05 and 21:35, some customers had slow responses and timeouts. The cause was unusually high traffic, above the levels we had covered in our load tests. It used up all the network bandwidth on our load balancers.

Impact

Requests to the affected customers were slow or timed out for about 2.5 hours.

Timeline

• 19:05 - Response times start to get worse for some customers.

• 19:58 - We start investigating.

• 20:20 - We find the cause: the load balancers have reached their maximum network bandwidth.

• 21:15 - We increase the load balancers' resources to raise the available bandwidth. This has no effect.

• 21:32 - We apply a temporary workaround that sends traffic around the overloaded load balancers.

• 21:35 - Response times are back to normal.

Root cause

Traffic was higher than the levels our load tests cover, and it reached the bandwidth limit on our load balancers. We are still working out what kind of traffic caused the increase.

We increased the load balancers' resources, which should have raised the available bandwidth, but it made no difference. We have opened a support case with Microsoft to find out why.

Resolution

To get service back, we set up a temporary workaround that sends traffic around the overloaded load balancers. It is still in place while we work on a permanent fix.

Next steps

• Analyse the traffic – find out what traffic caused the increase.

• Increase the bandwidth – work with Microsoft to make sure the load balancers actually get more bandwidth.

• Upgrade the load balancers – move the load balancer servers to a newer server generation.

• Remove the workaround – once the steps above are done, remove the temporary workaround.

• Update our load tests – raise our load tests to cover higher traffic before Black Month.

Response times are back to normal. The workaround stays in place while we work with our cloud provider on a permanent fix. We will publish a full incident report soon.

We have applied a workaround that sends traffic around the overloaded load balancers. We are monitoring the results.

We increased the load balancers' resources to raise the available bandwidth, but this did not fix the problem. We are trying another fix.

We have found the cause. Our load balancers have reached their maximum network bandwidth, which is slowing down traffic to some customers. We are working on a fix.We increased the load balancers' resources to raise the available bandwidth, but this did not fix the problem. We are trying another fix.