"Help, my Linode VPS has died!" "Help, I've lost some data on my Linode VPS!" "Help, I stuffed up a configuration setting now my Linode VPS is bricked!" "WHAT DO I DO!?!" If you've ever uttered those words in desperation I have the answers you seek!

Having a server die, losing data or bricking your server through a wrong configuration setting are some of the most gut wrenching moments we can experience. Hopefully you've been taking nightly backups of your server at a bare minimum. Right? Right?!

Linode is a great online hosting company for cheap, reliable, Virtual Privatge Servers running linux. I have several setup for different systems I have developed and host.

Background information

The other day I found a bug in one of the SaaS systems I developed. It meant that the client data export functionality wasn't exporting dates correctly. I've no idea how this got past testing! Anyway it was a very simple fix (only had to change one character) and all was well again.

Around this time one of our client had exported their data, made some changes and wanted to import it again. With our product there are limits on how many records you can have in the system so in between exporting and importing they _purged_ their data. No biggie. Except they exported using the buggy client data export! YIKES!

Thankfully we had made the decision long ago to pony up the extra $5 a month for nightly, weekly and fortnightly backups. At $5 it was a no brainer. So much so that I didn't even really consider it again or what the process is for restoring backups.

Linode Backups to the Rescue

Whilst taking solace in our decision to pay for backups I logged into the Linode Manager and did the following:

  1. Selected the 'Backups' menu item for my linode
  2. Checked the backup history to determine which backup set would have the most up-to-date date before the client purged
  3. Found the backup and clicked the "Restore to..." link. It was at this point that I realised I'm using 100% disk capacity for my linode so couldn't do a restore.
  4. Thinking outside the square I simply spooled up a new linode box. At a pro-rata rate of $11 for the rest of the month it was a cheap cost to occur to restore the client's data.
  5. Repeated step 3 but selected the new linode I just spooled up.
  6. Grabbed some popcorn and watched some TV as I waited for the restore to finish. It took about an hour.
  7. Grabbed the IP address of the new linode and modified my Linode DNS Manager settings so I could load up a patch subdomain for my SaaS product
  8. Logged into the new linode server and modified the NGINX configuration file to listen on the new IP address and subdomain
  9. Applied the bug fix to the new linode server
  10. Logged into the SaaS web application using the patch subdomain
  11. Exported the clients pre-purge data
  12. Logged into the production SaaS web application and imported the client data
  13. Sat back and thanked my lucky starts as all the data was there!

Lessons Learnt

Firstly make sure your testing regime is thorough and covers all cases. I still haven't been able to figure out how this bug got through.

Secondly make sure you have backups in place and know how to restore them. The awesome thing with the Linode backup is that it takes a complete snapshot of the virtual machine. So when I restored it to the new machine it just booted up and was a fully functioning replica of production just with a data set that was a few days old. This made my job so much easier.

Finally keep a level, cool head and be prepared to think outside the square. What's the problem? What do I need to achieve? What are some options to get the desired outcome? The more options you have the less likely you are to get stuck.

--

In the end the whole process probably took about 2 hours, most of which was spent waiting for the new linode server to spool up and the restore to occur.

If you haven't already be sure to checkout Linode. I've been totally impressed with them over the 5 years I've been using them.

Onwards and upwards!



comments powered by Disqus