{"id":2690,"date":"2019-08-17T01:18:00","date_gmt":"2019-08-17T06:18:00","guid":{"rendered":"https:\/\/www.mysmartbots.com\/blog\/?p=2690"},"modified":"2019-08-21T09:42:37","modified_gmt":"2019-08-21T14:42:37","slug":"the-minor-maintenance-turning-to-epic-crash-and-some-data-loss","status":"publish","type":"post","link":"https:\/\/www.mysmartbots.com\/blog\/2019\/08\/the-minor-maintenance-turning-to-epic-crash-and-some-data-loss\/","title":{"rendered":"The minor maintenance turning to epic crash… and some data loss"},"content":{"rendered":"

\"\"If you saw\u00a0SmartBots website offline today, we are really sorry for that! This was a result of an epic crash we’ve faced today. The techy details follow for those who interested or curios.<\/p>\n

Once is a chance<\/h3>\n

We’ve decided to run a minor maintenance this morning.\u00a0It was supposed to be a regular replacement of an SSD drive (these drives get worn out\u00a0every 1.5 – 2 years). The drives (say, A and B) are joined into the RAID array, so it was enough to pull out one drive (drive B) and insert a fresh one.<\/p>\n

The Data Center technician took out the broken drive B.\u00a0The drives are installed quite close to each other,\u00a0so\u00a0the\u00a0healthy drive\u00a0A got slightly moved as well. The technician decided to adjust it,\u00a0pushed with a finger… and, as we know now, killed\u00a0BOTH drives with a static electricity. One should always wear an anti-static bracelet!<\/p>\n

\"\"<\/p>\n

He did not\u00a0realized that\u00a0both A and B just died… so he\u00a0proceed to\u00a0the maintenance, and the\u00a0affected drive\u00a0A started writing random zeroes to the database, erasing the recently created bots and purchases.<\/p>\n

Twice is the\u00a0coincidence<\/h3>\n

In 15 minutes, we’ve discovered that the database is being\u00a0zeroed. Another 15 minutes has been spend to understand that it is not a hacker attack, and 30\u00a0more minutes spent to find a healthy drive and\u00a0put it to the system.<\/p>\n

Once the reason was discovered, the\u00a0backup guy has been sent\u00a0to get the off-site backup – the one stored on\u00a0an external server. He\u00a0took a spare drive, connected it to the external backup and started downloading gigabytes of data.\u00a0Unfortunately, the backup guy occasionally took the ALREADY BROKEN drive\u00a0B (he was in a haste, yeah).<\/p>\n

\"\"<\/p>\n

Triple\u00a0is either a malicious intent or stupidity<\/h3>\n

The data\u00a0brought back on\u00a0this drive was also FULL OF UNEXPECTED ZEROES. We were rewriting a broken data with another broken data. It took two more hours to discover that… and unfortunately,\u00a0during those hours the broken data were writing over the fresh backup. We’ve just forgot to stop the live data replication.<\/p>\n

So, we’ve ended up with a completely ruined database, and a corrupted live backup. The previous full backup was few days old. We’ve spend hours restoring as much data as possible. 99% of bots are running smoothly now; however, something got lost.<\/p>\n

\"\"<\/p>\n

How do we live with that now<\/h3>\n

In SmartBots account, you’ll see\u00a0a notification asking to check your bots and Wallet. If you feel that something is missing, please contact us. We will check our records and add you the missing funds.<\/p>\n

We really apologize for the worries and the downtime! Please give us few days to arrange the missing data, and then we’ll try to compensate the outage for all customers. Also, thank you for reading\u00a0through this post –\u00a0hope it was not\u00a0too boring and revealed some interesting details to you.<\/p>\n

Thank you for\u00a0running your business with SmartBots!
\nDespite all the issues we sometimes\u00a0hit together (\u0298\u203f\u0298)<\/p>\n","protected":false},"excerpt":{"rendered":"

If you saw\u00a0SmartBots website offline today, we are really sorry for that! This was a result of an epic crash we’ve faced today. The techy details follow for those who interested or curios. Once is a chance We’ve decided to run a minor maintenance this morning.\u00a0It was supposed to be a regular replacement of an […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[11,372,373],"tags":[88,520,366,521],"_links":{"self":[{"href":"https:\/\/www.mysmartbots.com\/blog\/wp-json\/wp\/v2\/posts\/2690"}],"collection":[{"href":"https:\/\/www.mysmartbots.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mysmartbots.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mysmartbots.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mysmartbots.com\/blog\/wp-json\/wp\/v2\/comments?post=2690"}],"version-history":[{"count":5,"href":"https:\/\/www.mysmartbots.com\/blog\/wp-json\/wp\/v2\/posts\/2690\/revisions"}],"predecessor-version":[{"id":2711,"href":"https:\/\/www.mysmartbots.com\/blog\/wp-json\/wp\/v2\/posts\/2690\/revisions\/2711"}],"wp:attachment":[{"href":"https:\/\/www.mysmartbots.com\/blog\/wp-json\/wp\/v2\/media?parent=2690"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mysmartbots.com\/blog\/wp-json\/wp\/v2\/categories?post=2690"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mysmartbots.com\/blog\/wp-json\/wp\/v2\/tags?post=2690"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}