Yesterday, starting at 15:28, Lisbon Time, all Tenzix services became unavailable due to a nationwide power outage.
Even though the power outage started at 11:33, our backup energy systems picked up the workload and allowed us to continue operating normally.
Unfortunately, at 15:28 one of the generators powering the data center failed, making the energy systems unstable.
Afterward, a second unit started to fail as it was overheating from the extra workload, leading to another failure.
Our team was on-site and of site monitoring and updating all our customers and partners and trying to restore our normal operations.
At 17:45, our remote team rerouted our main website to a new server allowing us to better notify customers unaware of what was going on.
Even though the power grid started to come back online prior to the service restoration, we took precautions by checking all services to prevent damage.
Everything is now operational and all systems are restored to full capacity.
After this incident, we've decided to credit our users, even though we are below our annual SLA. We will be giving this credit to users who open a ticket and make that request to our sales team.
We're sorry for this incident that was beyond our control.
We are already working with our data center operator to check what can be done better next time on our side to prevent such a drastic downtime.
For starters, our website will now be routed through 2 different countries, to prevent communication issues like this. All our email services were down because we relied on a single location.
As so, we're moving some of our services to new geo-redundant solutions.
As for DNS services for customers, we will be adding another external location for redundancy.
Thanks for your continued trust and for making us want to improve for you.
