This Wednesday, March 28, all Qwant’s infrastructure broke down around 12.00 UTC, and the service could only be restablished hours later, around midnight. Since Qwant’s launch in 2012, it has been by far the longest service failure we’ve had. We are conscious that a search engine is an essential tool that needs to have maximum availability, and therefore we offer you our most sincere apologies, which must come with explainations — those of you who follow us on Twitter were able to read them throughout the day.
Long story short, Qwant was a ‘victim’ of its success, and famous Murphy’s law came by to make sure that anything that could make problems worse actually happened. Qwant was already on a steady growth, but recently the Cambridge Analytica scandal raised awareness about the importance of using online services that truly protect your privacy. We thus had an unexpected growth of users which required that we quickly install a new set of servers in our bays, and that we upgrade our infrastructure. That’s when the problems started.
First, you should know that Qwant is the only search engine of its kind in Europe that has its own indexing technologies and its own algorithms, which require important hosting and computing resources. We are also the only one who opted, from day one, to totally control our infrastructure instead of relying on third party hosting and computing services provided by Amazon, Google, Microsoft and others. We install our servers and equipements in our own bays, because we want to control all of the infrastructure to guarantee true protection of your data, and to have an independant, 100% European infrastructure. We believe it is essential for Europe to have its technological independance, and we accept the risks of failure that come with the building our own infrastructure.
So what happened?
In order to ensure the availability of our services in case of failure or overload, we have different sets of systems in place. In particular, we have three “redundancy” systems that ensure that servers can step in when an anomaly is detected. During the upgrade of our infrastructure, we had to deactivate one of the redundancy systems, to install everything that was necessary. Our services where then relying on two redundant systems. But for a reason still unknowned, a hardware failure caused the second system to shut down, then Murphy’s law did its magic by causing a failure on the third and last system. Our services were all down. Our technical teams inspected the hardware to determine the source of the failure, and made an express shipping order for spare parts. They were delivered a few hours later. Qwant came back to life after it was all installed, checked and rebooted.
Of course we will learn a lot from this incident, and take the measures we must take to ensure it can’t happen again. But most of all, in addition to our apologies, we want to thank you all for the incredibly kind and warm support you showed during this difficult time.
This is how we recognize a united community, and we feel very lucky and proud to have you as our users and supporters. It is thanks to you that Qwant will keep on protecting the privacy of more and more internet users, and we really hope this failure will remain the longest of our lifetime.