Post-mortem: Partial outage for the next-generation Viskan eCom engine

Date: October 22, 2026
Duration: 36 minutes (09:21 to 09:57 CEST)
Affected service: Viskan eCom engine (next generation)
Status: Resolved

Summary

On October 22, the next-generation Viskan eCom engine had a partial outage that lasted 33 minutes. During that time some requests to the platform failed or were slow. The cause was that one of the database services behind the platform didn't have enough capacity for the load it was receiving. We increased its capacity, and the service has been stable since.

Timeline (CEST)

09:24 - We saw degraded service and started investigating.
09:31 - We published the incident on our status page.
09:43 - We found that a database service didn't have enough capacity and started working on a fix.
09:57 - We increased the database capacity and service returned to normal.
10:19 - After a period of monitoring, we confirmed the service was stable and closed the incident.

Impact

Some customers had errors or slow responses on the next-generation eCom engine between 09:21 and 09:57 CEST.

Customers on the previous-generation Viskan eCom platform were not affected.

Root cause

One of the database services behind the eCom engine was on a capacity level that was too small. When load went up, it hit a resource limit, and requests to the platform failed or slowed down.

Resolution

We moved the database service to a much higher capacity level, with resources properly sized for our workloads. Performance went back to normal as soon as the change was in place.

What we're doing to prevent this

• Capacity review: We are checking the capacity of all production database services against their actual and expected load.

• Monitoring and alerts: We are improving our monitoring so that we get an alert well before a service runs short on resources.

• Provisioning standards: We are tightening our routines so that every production service starts with properly sized capacity.

We apologise for the disruption this caused you and your customers.

This incident has been resolved. A postmortem will follow.

Summary

On 2026-10-07, between 19:05 and 21:35, some customers had slow responses and timeouts. The cause was unusually high traffic, above the levels we had covered in our load tests. It used up all the network bandwidth on our load balancers.

Impact

Requests to the affected customers were slow or timed out for about 2.5 hours.

Timeline

• 19:05 - Response times start to get worse for some customers.

• 19:58 - We start investigating.

• 20:20 - We find the cause: the load balancers have reached their maximum network bandwidth.

• 21:15 - We increase the load balancers' resources to raise the available bandwidth. This has no effect.

• 21:32 - We apply a temporary workaround that sends traffic around the overloaded load balancers.

• 21:35 - Response times are back to normal.

Root cause

Traffic was higher than the levels our load tests cover, and it reached the bandwidth limit on our load balancers. We are still working out what kind of traffic caused the increase.

We increased the load balancers' resources, which should have raised the available bandwidth, but it made no difference. We have opened a support case with Microsoft to find out why.

Resolution

To get service back, we set up a temporary workaround that sends traffic around the overloaded load balancers. It is still in place while we work on a permanent fix.

Next steps

• Analyse the traffic – find out what traffic caused the increase.

• Increase the bandwidth – work with Microsoft to make sure the load balancers actually get more bandwidth.

• Upgrade the load balancers – move the load balancer servers to a newer server generation.

• Remove the workaround – once the steps above are done, remove the temporary workaround.

• Update our load tests – raise our load tests to cover higher traffic before Black Month.

Response times are back to normal. The workaround stays in place while we work with our cloud provider on a permanent fix. We will publish a full incident report soon.

We have applied a workaround that sends traffic around the overloaded load balancers. We are monitoring the results.

We increased the load balancers' resources to raise the available bandwidth, but this did not fix the problem. We are trying another fix.

We have found the cause. Our load balancers have reached their maximum network bandwidth, which is slowing down traffic to some customers. We are working on a fix.We increased the load balancers' resources to raise the available bandwidth, but this did not fix the problem. We are trying another fix.

Incident report: web store disruption on 6 October 2026

Summary

On 6 October 2026, web stores loaded slowly or failed to load for a period. A data service that the web stores rely on ran out of working memory and could not keep up with requests.

Once we identified the cause, we restored normal operation by restarting the affected service. We are now adding capacity and reorganising how data is stored in that service, so this does not happen again.

Timeline

All times are CEST on 6 October 2026.

Impact

During the incident, web store pages could load slowly or show errors.

Cause

The disruption was caused by a lack of memory in a data service we use (Elasticsearch).

The data in this service is divided into many small parts, and each part needs some memory regardless of how much data it holds. Over time, our data had been split into far more parts than needed. Together, they used up the available memory, and the service slowed down until it could no longer answer requests.

Resolution

We restored normal operation by restarting the affected service, which freed up its memory. We then removed unused data and restarted the related services.

What we are doing to prevent this

• More capacity, now: we are increasing the memory available to the service, to give it a safety margin.

• Leaner data structure, over the coming period: we are reorganising the data into far fewer parts. This removes the underlying cause and reduces memory use substantially.

• Cleanup of unused data: we have removed unused data that was no longer needed, and will make this part of our routine to keep unused data to a minimum

This incident has been resolved, after restarting affected services. We have a plan for a long term solution, and we will put this in place as soon as possible.

We have identified the root cause of the problem and are working on a fix. One service has been restarted, and some web stores are coming back to life.

Incident summary — Brief Backoffice unavailability

Period: 29 September 2026, 11:50–11:55 CEST
Affected services: Viskan Backoffice

Impact
For about five minutes, Backoffice and some related services were slow or unresponsive. No data was lost, and all services recovered fully by 11:55 CEST.

Cause
An unusually high volume of automated image requests from an external crawler caused the incident. Our media service, which resizes and optimizes images on demand, came under heavy load while processing these requests. This load affected the performance of other internal components, including Backoffice, until the requests subsided.

What we have done
We have set resource limits on the media service and guaranteed resource allocations for Backoffice and other critical components. A similar spike in image requests would now be contained within the media service without affecting the rest of the platform.

What we will do

• Introduce rate limits on image requests, so a high volume from a single source cannot overload the media service.

• Limit how many images the media service processes at once, so large bursts are handled gradually instead of all at once.

• Adjust our monitoring and alert thresholds so we detect this type of load pattern before it affects customers.

We have deployed a series of improvements, and performance is now at an acceptable level and has stayed stable since. We are therefore closing this incident. Improving performance remains a priority, and we will keep working on making the platform faster. Thank you for your patience. If you still have problems, please contact support.

The performance improvements deployed last week have had a clear positive effect, and overall platform performance is significantly better than at the start of this incident. We continue to monitor closely.

Some routines are still running slower than we expect. We have identified these areas and are preparing a further round of performance work targeting them. We will share another update as these improvements are rolled out.

We apologise for the inconvenience and thank you for your patience.

We have now deployed a series of performance improvements to the platform, and we are seeing measurable results. Updates are reaching web stores faster, and the larger routines are completing more quickly than they have in recent weeks.

We are not declaring the issue resolved yet. Our team will continue to monitor performance closely, and further improvements are already planned and underway.

We will post another update as the work continues, and of course when we consider the situation fully stable.

Thank you for your patience throughout this.

We have increased the capacity of the systems behind the platform. In short, the servers that handle data updates and heavier routines now have significantly more processing power and memory available to them.

We expect this to improve performance, but we want to be careful not to promise more than we can confirm at this stage. We will be monitoring the effects closely over the coming days, and we are continuing to investigate additional improvements.

You may still experience delays while we evaluate the situation. We will post another update as soon as we know more.

Thank you for your continued patience.