The minor maintenance turning to epic crash… and some data loss

If you saw SmartBots website offline today, we are really sorry for that! This was a result of an epic crash we’ve faced today. The techy details follow for those who interested or curios.

Once is a chance

We’ve decided to run a minor maintenance this morning. It was supposed to be a regular replacement of an SSD drive (these drives get worn out every 1.5 – 2 years). The drives (say, A and B) are joined into the RAID array, so it was enough to pull out one drive (drive B) and insert a fresh one.

The Data Center technician took out the broken drive B. The drives are installed quite close to each other, so the healthy drive A got slightly moved as well. The technician decided to adjust it, pushed with a finger… and, as we know now, killed BOTH drives with a static electricity. One should always wear an anti-static bracelet!

He did not realized that both A and B just died… so he proceed to the maintenance, and the affected drive A started writing random zeroes to the database, erasing the recently created bots and purchases.

Twice is the coincidence

In 15 minutes, we’ve discovered that the database is being zeroed. Another 15 minutes has been spend to understand that it is not a hacker attack, and 30 more minutes spent to find a healthy drive and put it to the system.

Once the reason was discovered, the backup guy has been sent to get the off-site backup – the one stored on an external server. He took a spare drive, connected it to the external backup and started downloading gigabytes of data. Unfortunately, the backup guy occasionally took the ALREADY BROKEN drive B (he was in a haste, yeah).

Triple is either a malicious intent or stupidity

The data brought back on this drive was also FULL OF UNEXPECTED ZEROES. We were rewriting a broken data with another broken data. It took two more hours to discover that… and unfortunately, during those hours the broken data were writing over the fresh backup. We’ve just forgot to stop the live data replication.

So, we’ve ended up with a completely ruined database, and a corrupted live backup. The previous full backup was few days old. We’ve spend hours restoring as much data as possible. 99% of bots are running smoothly now; however, something got lost.

How do we live with that now

In SmartBots account, you’ll see a notification asking to check your bots and Wallet. If you feel that something is missing, please contact us. We will check our records and add you the missing funds.

We really apologize for the worries and the downtime! Please give us few days to arrange the missing data, and then we’ll try to compensate the outage for all customers. Also, thank you for reading through this post – hope it was not too boring and revealed some interesting details to you.

Thank you for running your business with SmartBots!
Despite all the issues we sometimes hit together (ʘ‿ʘ)