« Zooomr Service Update Video & Notes [UPDATEDx2]嗨, 來自台、港、澳的朋友 / Hello Taiwanese, H.K. & Macau Friends »

Back Online

by Thomas Hawk | 28 October 2006 | Announcements, System Notices | 32 Comments

Thanks to all of you for bearing with us over what has been a hellish past few days. On Thursday evening we had a massive hardware failure that took our photo storage system offline. As soon as we were aware of the failure Kristopher went down to our server farm where he has been living since.

Sometimes things are hard when you are small and with only two of us these past few days have been even harder. We are terribly sorry to all of you for having had Zooomr offline. Terribly, terribly sorry.

We are back online but we are working on our most recent backup from a few days before we went down.

We had multiple hard drives go down on one of our servers similtaneously because our RAID controller did not function as it should. Everything on Zooomr should be as it was the few days before we went down but photos uploaded in the past few days will still need to be extracted from our damaged drives. We are working with forensic specialists to try and get these drives recovered as soon as possible, but if you are not seeing photos from the immediate few days before we went down you may wish to just reupload these photos. We hopefully will have the shots from the few days prior to our failure up as quickly as we can pending the drive recoveries.

We purchased more drives and hardware over the weekend to do even more frequent backups of all of our data and again apologize and deeply regret the inconvenience that this has caused you our users. This has been the worst 72 hours of our life. More than anything we want Zooomr to be the best photosharing site in the world for you our users.

Thank you to all of you who left us supportive comments and remarks over the past few days, it has really meant alot.

Kristopher is pretty tired — both physically as well as drained emotionally from being up most of the past 72 hours working on this. He’ s going to get some sleep tonight and hopefully we will be able to give you more updates in the coming week. Should you have any specific concerns or questions feel free to email me directly at tom(at)bbridgetech.com.

32 Comments

  1. Steve said on 28 Oct 2006 at 9:09 pm:

    Congrats on getting the system back up, and thanks for keeping us all in the loop throughout. I hear we’re setting the clocks back an hour tonight to give you some extra sleep. You both certainly deserve it!

  2. Jordan said on 28 Oct 2006 at 9:19 pm:

    Wow, you guys are amazing!

    Really really appreciate all the hard work… I owe you one!

    Also, i noticed that next to my name it now says “pro 4 life”… is this for real?! That is too generous! I mean, much appreciated, but, I WANT to give you guys money! lol

    You guys are the best, really. Not only the best photo sharing site, but the best operators too!

  3. jpghouse said on 28 Oct 2006 at 9:20 pm:

    Thomas/Kristopher-

    There are no need for apologies. You guys put your heart and soul into this website/service and it shows. I greatly appreciate the updates you guys have provided during the downtime. Kris get some sleep man - you deserve it!

    Jpghouse

  4. cmiper said on 28 Oct 2006 at 9:20 pm:

    Good work guys!
    As I mentioned before, I have been in this situation (MS Exchange with total drive/controller failure including the mirrors) and it really does stink. You do the best you can with what you have, learn from the experience, and then you strengthen based on the experience. It’s a nasty feeling knowing that you are down and that people are trying (sometimes impatiently) to get to their stuff. But when all is said and done, you get through it and you get to breath that sigh of relief.

    Again, thanks, and Kristopher, please go get some sleep. ;-)

  5. fajro said on 28 Oct 2006 at 9:30 pm:

    We ♥ Zooomr
    :-D

  6. Jordan said on 28 Oct 2006 at 9:31 pm:

    I forgot to mention… I’m in love with you.

  7. nightfox said on 28 Oct 2006 at 9:32 pm:

    oh.. so that means the most recent photos i have in my blog (http://noel.feria.name/blog) will be gone forever?

  8. James said on 28 Oct 2006 at 11:11 pm:

    Thank you both for everything that you did to bring Zooomr back online, and thank you for staying in touch with us through all of this. I really appreciate it, and I’m sure that everyone else does too.

  9. gregoire said on 28 Oct 2006 at 11:57 pm:

    Thank you for all your hard work on Zooomr.
    You are making a very good website and it is normal if there is a failure sometime.
    I hope that Kristopher is sleeeeeeeeping because he had little eyes on the last video!
    Have a good week-end and thank you again.

  10. TravisP said on 29 Oct 2006 at 12:50 am:

    Great to see you guys back with us now, as well a great to see the community getting behind you for support… well some of them anyway. Thanks for the Pro 4 Life also. As above, this is very generous, but also understand the reasons behind it. Keep up the good work, I AM with you guys for life.

  11. rukasu said on 29 Oct 2006 at 1:30 am:

    Thanks for getting things back up, hope the retrievals are painless as well, and a big thank you for the ‘pro 4 life’ status, too generous man thanks a million

  12. Alexandru Savu said on 29 Oct 2006 at 1:33 am:

    Good work guys!

  13. tim said on 29 Oct 2006 at 3:22 am:

    Have a good rest, Kris…and take a day off. Thanks for putting in so much time, and thank you for the “Pro 4 Life” thingy. I lost a few photos, but this more than makes up for it.

  14. yodahome said on 29 Oct 2006 at 4:08 am:

    Wow, big thanks for the “pro 4 life”! Despite its actual meaning it simply looks and sounds cool. *g*
    Even more thanks for investing so much time and heart into this service, it’s surely not business as usual. Hope you won’t have such big problems for a long time. I didn’t actually realize until now but I understand you had a heck of a bad time.

  15. Oblomov said on 29 Oct 2006 at 4:21 am:

    Welcome back from Hell, good to see you alive and kicking again! But do get some sleep now :)

    Tonight in Europe we switched from DST, thereby gaining an hour of sleep … hope you can benefit from such a thing too, soon :)

  16. Raoul said on 29 Oct 2006 at 6:08 am:

    Wow, just woke up and Zooomr’s up, and it looks like I’m now a Pro 4 Life? This is so generous I’m speechless. I said it before, when you made me a Pro without letting me pay, and I’ll say it again: I’d like to support Zooomr. Can I at least donate some money?

    I really appreciate all of the hard work that went into bringing Zooomr back online. As if getting the service back up wasn’t hard enough, doing all of the integrity tests to ensure that the data was restored properly for every user and every photo must have been loads of “fun”. I’m missing a few photos, but so be it. When a catastrophic drive failure occurs, you can’t expect *everything* to come back up as if nothing happened. You’re going to have a few battle scars to remind you of the experience.

    Get some sleep, Kristopher! 72 hours with no sleep can be dangerous to your health. Go to bed, man. You did a fantastic job bringing the site back up, and things are working. Get some rest, and thank you, really, thank you!!!

  17. Kicou said on 29 Oct 2006 at 6:14 am:

    I, for one, welcome our new pic sharing site overlord ;)

    Get some rest now, you deserve it well.

    You guys are doing a hell of a good job, we love you.

  18. Thomas Hawk said on 29 Oct 2006 at 7:04 am:

    Thanks all for the understanding. As hard as the past few days have been one of the things that really kind of blew us away was that so many of you remained so supportive and so kind as we went through our few days of hell. That encouragement really helped as we worked through things.

    The Pro Account for Life thing is meant to be our way of saying both sorry and thanks for all that you’ve put up with over the course of the past few days. We hope we never have to go through something like that again and for those of you that stick with us we wanted some way to publicly share our gratitude. In some ways Pro for Life accounts will always be a reminder to us of the trials of building a new community.

    We could not do Zooomr without all of you and to those that stick with us through these trials we wanted to make sure that we could say thanks the best that we knew how.

  19. noeltykay said on 29 Oct 2006 at 7:59 am:

    Welcome Back! High Availability/Disaster Recovery solution should be on your short term wish list. What kind of servers are you using to host?

  20. Raoul said on 29 Oct 2006 at 11:16 am:

    There’s a nice silver lining for those who lost photos, provided they used jUploadr. It saves a cache of the set of photos last uploaded to Zooomr. In my case, all I needed to do was to open jUploadr, and re-upload the last set of photos. Easy as pie, no sweat, no headaches.

  21. Mike said on 29 Oct 2006 at 4:36 pm:

    Hi

    Great site

    Im a network geek so thought Id ask this.

    What is the hardware setup of Zooomr ?

    thanks

  22. Ben said on 29 Oct 2006 at 7:58 pm:

    Great work, fellas! Thanks for everything!

  23. Beau Liening said on 29 Oct 2006 at 9:59 pm:

    Thomas and Kristopher, you guys are doing an awesome job on Zooomr!

    Looking through my photos, it seems that some of the photos have lost their Square, Thumbnail, Small and Medium photo sizes, and sometimes the original size.

    So, seeing that, I have a feature request. I’d like to see an option of uploading a new revision of the photo instead of the old one. That way, the photo location/url stays the same, but the photo itself is updated.

    Let me know what you think.

  24. fajro said on 30 Oct 2006 at 12:13 am:

    I have a suggestion!

    The option “replace” or “update photo”.

    For example: I want to upload photos now and subsequently replace them with corrected versions (photoshoped versions :-P).

    And instead of using the Uh-Oh page have “empty pages” for the lost pictures with this option.

    What do you think?
    :-D

  25. Raoul said on 31 Oct 2006 at 6:53 am:

    Yeah, I second Beau Lining and fajro’s feature request. A “Replace Photo” feature would be a very nice addition to Zooomr. :-)

  26. Kor Shu Yen said on 26 Nov 2006 at 8:26 pm:

    I missed the Pro 4 Life bandwagon, can I get it now?
    Please? :D

  27. mgymfqosbf said on 18 Jun 2007 at 12:44 am:

    Hello! Good Site! Thanks you! regmrlzlpgkqgl

  28. bglafolyci said on 27 Jun 2007 at 11:48 am:

    Thanks for this site!
    hifue.info

  29. Hamlet Tori said on 4 Jul 2007 at 10:08 pm:

    You do it in the nigh. Hamlet Tori.

  30. social networks purchased said on 8 Jan 2008 at 12:34 am:

    social networks purchased…

    Websites with web 2.0 standards are not always the better ones….

  31. Payday Loan Advance said on 8 Jan 2008 at 7:20 am:

    Payday Loan Advance…

    PaydayEZ.com promotes immediate approvals for payday loans with no credit needed; ladies established in the wonderful state of Missouri might now apply via the Internet….

  32. Loans Missouri Payday said on 30 Mar 2008 at 4:16 pm:

    Loans Missouri Payday…

    Can it be that your server is infected with a virus - I get an Virus warning when I open your site with Firefox - Just for your Info….

Want to Comment? Continue the conversation: Visit Zooomr Groups!