No announcement yet.

Explanation and Time-line of RFO downtime the past few days

  • Filter
  • Time
  • Show
Clear All
new posts

    Explanation and Time-line of RFO downtime the past few days

    In the name of complete transparency and openness about what went down over the last few days, here's a time-line starting at the very beginning:

    12/7/2010 : RFO established. Forum "backups" are implemented using simple backup routine running on web-host server that performs daily backups of the forum mySQL database to files stored on the webhost server (the forum software is vBulletin, but is just a front-end to a mySQL database that contains all of our posts). Administrators manually download these backup files to local storage (a semi-daily to weekly basis).

    1/17/2011 : Admins start discussion/implementation of automating daily backups to additional remote location(s) for more dependable backup redundancy.

    1/19/2011 : Munkholm starts forum discussion/poll about site performance.

    1/25/2011 : It is determined that the "current" (at that time) web host is not up to the task of meeting RFO's requirements (backups, site growth, etc.). Admins start discussion/implementation of finding a new web host.

    1/26/2011 : RFO settles on GlobalGold webhosting. RFO expects to start migration in 7-10 days (giving us plenty of time to notify the forum of the upcoming switch).

    1/26/2011 : Just a couple hours later, by complete coincidence, but maybe also a foreshadowing, web host mySQL server melts down. Forum data is safe, but forum down for 16 hours (and with no way to communicate with membership).

    1/27/2011 : RFO facebook page is established (revitforumorg) as a way to communicate with membership in the event that the forum site is down. Please keep this in mind for future reference (or look it up on facebook and simply click the "Like" button). There's no plan to update it often, so don't worry about being flooded with posts. And you can click the "Unlike" button at any time.

    1/27-28/2011 : Maybe with a touch of panic setting in after the meltdown, RFO starts immediately implementation of switch to new web-host (GlobalGold). Forum is down for a couple of hours depending on your location (DNS servers take time to populate).

    RFO did not provide the amount of notice before this took place that we would have liked to have seen, so we do apologize for the inconvenience. It was decided that the immediate reliability of the forum took priorty over the notification window. However, personally, I was extremely pleased to see an email from RFO in my inbox with an explanation of what was taking place.

    The future : Admins hope to complete automated remote daily backup system on new web-host in the next week or two. More details of how that works will be posted once implementation is complete. Hopefully, there won't be any more web-host "excitement" or forum downtime for the foreseeable future.

    The admins at RFO appreciate that this kind of forum disturbance over the last couple days is a bit unsettling - while we all know that every web site goes down from time to time, we hope that the steps we're taking now will help mitigate potential issues in the future.

    I (and I think it's safe to say that I can speak on behalf of the other Administrators on this point) want to greatly thank Munkholm, who is the one who really did all the work. He's got an actual business and life to run outside of these forums, and he's volunteered a tremendous amount of time and a fair amount of money to getting this all worked out over the last week. Thank you Munkholm!

    Please feel free to post comments or questions.

    Two comments:
    As was stated several times before: where do we go to chip in?
    This is the way a meltdown like this should be handled. Rep for all admins. Great work guys!
    Martijn de Riet
    Professional Revit Consultant | Revit API Developer
    MdR Advies
    Planta1 Revit Online Consulting


      i tried to get them to put in a donations button... still none.
      -Alex Cunningham


        Just to add a little more insight to Iru´s very explanative post above. (Thanks Iru)

        When this forum was launched, I chose to use the same hosting account as my company website. At that time i HAD NO IDEA how much impact a forum like this have on the web servers (I do now).
        The old hosting account was not scalable in any way (could not be upgraded to a better "plan") and the Supporters clearly had little, or no knowledge at all about optimizing servers for running "busy" vB forums.

        Now, we´re hosted with GlobalGold (GG), who have dedicated hosting plans to vBulletin forums, and have optimized the web servers and mySQL database servers to handle the loads. Also GG have a VERY scalable hosting plan... actually they´re currently hosting forums with 13,000,000 posts and about 315,000 members... So we should have a nice "home" here with GG for quite a while now.

        I´m truly sorry for the downtime during the last couple of days, and would have liked to be able to plan the migration a little better But on the other hand, I could not just sit back and wait for the next 16 hour breakdown.
        Klaus Munkholm
        "Do. Or do not. There is no try."


          Re money donations:

          The other admins have asked Munkholm about it as well. At this point, he doesn't want to do it yet (Munkholm, feel free to jump in and say you've changed your mind ). He is leaving it open as an option in the future.

          Munkholm hasn't spoken to me directly about it (i.e. his reasoning), so I am NOT speaking on his behalf, but if I was to offer a guess to his resistance about a donation button at this time, I think he just doesn't want to complicate the expectations and accountability of/to membership with money.

          If he starts collecting money, there's a tendency for some people to feel additional expectations, e.g. "I donated money, so therefore, I expect a certain level of service."

          There's also the accountability, e.g. "how are you spending the money? are you profiting from the money?", etc.

          If I was Munkholm, I don't think I'd want the pressure of that either (right now), or feeling obligated because someone has donated money, when the site is still so early on.

          I think once the site is really firmly established, and everything is running smoothly for a good while, and we've all had a chance to really consider how donations might work in practicality, Munkholm might be more open to considering contributions to keep the site up and running.

          I do know he greatly appreciates that everyone is so ready and willing to help out with donations. But let's not bug him too much about it. I'm sure he'll bring it up when he's ready.


            I know, I know. I am just saying: there should / could be a way this can be set up. I'm thinking anonymous donations or something. A waver which adresses these facts. It just get's a bit frustrating having to watch you people investing a lot of time and money in this community without the possibility to return the favor.
            Martijn de Riet
            Professional Revit Consultant | Revit API Developer
            MdR Advies
            Planta1 Revit Online Consulting


              Guys (and Gals)

              Like I´ve said before, just don´t worry about it The money issue is not paramount, and at least until we hit more than 400 concurrent users, I think we´re better off if I just continue to cover the expenses.

              If I were in the same situation as you guys (and gals), I´d be offering my support too! But Iru is spot on (again) ! Donations would just complicate everything.

              Besides, if each member donated just $1 pr. year, I´d be making a profit - And THAT my friends, is NOT why I´m doing this

              But if you feel that you MUST return the favor, then help spread the word about the forum (Usergroup meetings, Facebook, Twitter, Blogs, Linked In, etc.), and go make some POSTS in the forum :beer:
              Klaus Munkholm
              "Do. Or do not. There is no try."


                also gotta re-add the RSS feeds to have the come thru again. just FYI.
                -Alex Cunningham


                  Don't need them anymore... I'm hanging out here way too often.
                  Martijn de Riet
                  Professional Revit Consultant | Revit API Developer
                  MdR Advies
                  Planta1 Revit Online Consulting


                    If he starts collecting money, there's a tendency for some people to feel additional expectations, e.g. "I donated money, so therefore, I expect a certain level of service.
                    I disagree, PAYING makes you wait a service or an object in return.
                    DONATION doesn't only deals with money, it's a disinterested act.
                    This way we share the effort you guys make to keep this alive and working.

                    If you're afraid of making a profit, just make once a year a game with tons of cookies for the winner!!:laugh:
                    And if donations are too high, it could be an Ipad.:thumbsup:
                    "Au royaume des aveugles, les borgnes sont mal vus!"
                    P. DAC
                    Follow me on Twitter @Jbenoit44 - Blog:


                    Related Topics


                    • Infrastructure / Transparency
                      As mentioned several times before, this forum is not MINE, it´s OURS :beer:

                      But since I´m behind the initial setup, there are things registered...
                      December 8, 2010, 07:36 PM
                    • 5 years ago today...
                      Five years ago today, on December 7th, 2010 (at least by my time zone), the RFO website officially launched. Since then, nearly 50,000 members have come...
                      December 7, 2015, 03:56 PM
                    • Important Announcement on Future Funding Of Revit Forum
                      Hi All,

                      Revit Forum is approaching our first birthday in a few months time.

                      When Klaus first set this up, I am sure he did...
                      Voluntary donations, NO restrictions / NO icons.
                      Voluntary donations, NO restrictions, WITH icons.
                      Voluntary donations, NO restrictions to the forum part, but with paid for content.
                      Paid membership
                      Advertisement (Google, Targeted, Geo, or similar)
                      Membership ads
                      None of the Above
                      August 19, 2011, 05:20 AM
                    • Core Tenets
                      REVITFORUM CORE TENETS

                      In the following, will be referred to as "RFO"

                      RFO Site Policy
                      • Open and transparent
                      May 7, 2011, 08:53 PM
                    • A new beginning
                      Hello, and welcome to this new forum, dedicated to all the Revit Users "out there"

                      This new Forum is not meant to substitute...
                      December 7, 2010, 10:18 AM