Black Monday Site Issues – w/c 17 September 2012

• Posted by R Fox September 23, 2012 #site  #black monday  #bigger better faster  2,793 views
  click to rate

  As promised, here is a blog about the issues last week, why they occurred, what was done about them, and plans for the future – I know it’s important you know as this is your site with your data.

   

  What Went Wrong – Oh, Bugger

   

  The site initially failed last Monday around 9:30. The servers were not accessible in anyway and neither was the hosting. Repeated attempts to contact the ISP by support tickets, email, IM and telephone were unanswered. The data centre where the servers were hosted was contacts who were unable to assists as the contract was with their customer, not with Transtastic. No details were available or given – but my assumption was due to their failure to pay a bill. I checked back-ups were available from the previous night (they are taken at 3AM GMT each morning) – they all looked sound (phew).

  
  

  Fixing It – No Going Back

   

  On Tuesday afternoon new servers were provisioned – these of course need configuring and data from the back-up restoring. Then no sooner had that been done, the old servers came back on line. Panic over. 

   

  Another back-up got completed on Tuesday evening and the plan to migrate to the new servers was being put in place for weekend to minimise down-time. Also there was still no response from the ISP.

   

  Wednesday morning, the servers went down again (same MO). This meant full scale let’s restore the site on the new server, regardless of if the old servers coming back on-line. (I need to point out I was away on business Tuesday and Wednesday with limit access to time and anything else). The new servers were configured on Wednesday evening ready for the restore.

   

  On Thursday morning the restore was kicked off – this is some 10GB of data files, and a further 500MB of database. Also the temporary website to keep you guys informed also went up – sorry this should have been up earlier, but I hope you found it useful (I enjoyed getting the comments and feedback, thank-you). The server configuration was also tested.

   

  The restore took over 43 hours – with a number of validations afterwards to ensure it was OK. Meanwhile, I was also panicking a little as I wasn’t too familiar with the new server (the old ISP helped me quite a lot – now I was on my own!).

   

  So Friday morning (remember I work full-time) so grabbed a few moments here and there to check progress, and Friday evening came and I started to restore the database which took longer than expected. All night, and it wasn’t right on Saturday morning when I checked.

   

  On Saturday morning the database was restored again – this time successfully, but it completed late on. The restored site wasn’t working as some additional server configuration was required – but at 1am on Saturday morning it was up and working in a fashion (a few things didn’t like Uploads, Updates etc.). I updated the domain name to re-point to the new site (this can sometimes take 24 hours to propagate around the internet) and updated the temporary site with some details that might get people on sooner (sorry they didn’t work for everyone).

   

   

  Sunday was then about seeing people back on the site (great feeling and thank-you for all the nice comments) and doing a few checks and trying to resolve some residual issues. Which, I hope are now all completed.

  
  

  What Next - Moving On, Moving Up

   

  So, we’re here. And I really do appreciate those comments. Transtastic is run on a ‘lowish-buget’ (i.e. I pay for it with some subsidy from the advertisers). It’s my time and the time of the moderators and the people that help with the radio and social media that keep Transtastic  going on the straight and narrow and if we had to pay for the resource it would be very expensive – likewise having more resilience on the site would too. The good news is, we didn’t really do anything wrong (of course it’s my responsibility 100%) – this was down to someone not being honest and keeping us informed. Thankfully, we had good back-ups and got a new site up pretty quick considering everything. 

   

  However,  the site was (and is) growing fast – which is great – and it is now (after such a sh*tty week) that I need ALL YOUR SUPPORT to make it grow more and more.

   

  The new servers are BIGGER, BETTER, FASTER than the old ones – so we have scope to grow. The ISP is also more reputable – and we’ll be taken more back-ups and replicating them to other servers to get us back online quicker if that happens again! I want to turn this Black Monday Week into something good, something better, something even more TRANSTASTIC.COM

   

  Are you with me? We want more photos and videos, more interactions and comments, more polls and forums – and I also want you go out there and share your love for Transtastic – tell people what we are about – and we are about YOU, YOUR FRIENDS and EVERYONE!