Web store outage
Affected
Major outage from 11:59 AM to 12:51 PM
- Postmortem · UTC
Incident report: web store disruption on 6 October 2026
Summary
On 6 October 2026, web stores loaded slowly or failed to load for a period. A data service that the web stores rely on ran out of working memory and could not keep up with requests.
Once we identified the cause, we restored normal operation by restarting the affected service. We are now adding capacity and reorganising how data is stored in that service, so this does not happen again.
Timeline
All times are CEST on 6 October 2026.
Time
Event
13:52
Web store pages are failing to load
13:56
We detect the problem and start investigating
13:59
Viskan Status page is updated
14:14
The affected service is restarted and the web stores recover
Impact
During the incident, web store pages could load slowly or show errors.
Cause
The disruption was caused by a lack of memory in a data service we use (Elasticsearch).
The data in this service is divided into many small parts, and each part needs some memory regardless of how much data it holds. Over time, our data had been split into far more parts than needed. Together, they used up the available memory, and the service slowed down until it could no longer answer requests.
Resolution
We restored normal operation by restarting the affected service, which freed up its memory. We then removed unused data and restarted the related services.
What we are doing to prevent this
More capacity, now: we are increasing the memory available to the service, to give it a safety margin.
Leaner data structure, over the coming period: we are reorganising the data into far fewer parts. This removes the underlying cause and reduces memory use substantially.
Cleanup of unused data: we have removed unused data that was no longer needed, and will make this part of our routine to keep unused data to a minimum
- Resolved · UTC
This incident has been resolved, after restarting affected services. We have a plan for a long term solution, and we will put this in place as soon as possible.
- Identified · UTC
We have identified the root cause of the problem and are working on a fix. One service has been restarted, and some web stores are coming back to life.
- Investigating · UTCWe are currently investigating this incident.