Server Currently Offline - Hardware Issues (RESOLVED)

Glitchless · 11-02-2010, 12:43 PM

We are currently working to restore the server which has had an unknown hardware failure. We will update you on our progress as we know more. Be advised that we duplicate player data daily on a second local hard drive as well as on a third remote hard drive so we are not expecting any problems more severe than some downtime and at most a 1 day rollback.

Glitchless · 11-02-2010, 05:24 PM

The problem appears to have been a partial hard drive failure. We are still waiting on word from the datacenter as to whether the hard drive information will be salvagable which will determine whether or not we have to do the 1 day roll back. If we do have to remotely restore files the downtime will be further extended, but we are hoping that is not the case. We will update you when we have more information.

Glitchless · 11-03-2010, 02:25 AM

We have not yet been able to begin the restoration process due to some problems resurrecting the original server. When that is complete we will likely be migrating Nodiatis to brand new hardware with more data redundancy so restoration in the case of a failed hard drive will only take a matter of hours instead of a matter of days as this is turning into. We assure you we are every bit as frustrated as the players.

We have still been unable to confirm whether or not a 1-day, 5 minute, or absolutely no rollback will be necessary, it will be determined by the state of the hard drives. If any rollback occurs all players currently with a standard or premium account will receive a free TC, but there is still a chance that no rollback will be necessary. We will not know until the original server is operational again.

Glitchless · 11-03-2010, 08:14 AM

We have regained access to the original server and are now in the process of assessing any damage to the database. We are tentatively optimistic that we will be able to restore without any rollback. We will still have some sort of compensation for players with standard or premium accounts due to the immense pain and suffering they no doubt have suffered.

Glitchless · 11-03-2010, 03:55 PM

We are still awaiting the new Nodiatis server to be provisioned, but after that it should be a matter of several hours of file transfers and several hours of configuration. We appreciate the incredible level of patience our playerbase has shown throughout this ordeal.

While waiting for the server to be provisioned we took the time to do some much needed forum maintenance. It appeared there were some scheduled scripts that were not being performed properly and were routinely causing the forums to lock-up for several minutes at a time. We still notice that IE is loading pages slower than Firefox, but that issue may be out of our hands short of writing our own forums and we'd rather focus on Nodiatis patches.

We do not expect the amount of extra work being put into this server restoration to impeded with our scheduled patches for this month including numerous class buffs mid month and dual classes at the end of the month.

Glitchless · 11-03-2010, 06:47 PM

We now have access to the new server and will begin uploading and configuring the Nodiatis database.

Glitchless · 11-03-2010, 09:09 PM

Most everything is now configured and ready for testing. We will not be able to fully test the functionality of the game, nor will users be able to properly access it, until the new DNS information (the new IP of the server) propagates. We did initiate the DNS propagation as soon as we knew the IP address of the server several hours ago, however, it is still somewhat unpredictable and can take many hours. We will update you when we notice propagation is complete.

Glitchless · 11-03-2010, 11:22 PM

Some lucky players who are in areas where the DNS info has propagated already should now be able to log in from the main page. It would be a good idea if you see the message saying we are repairing the hardware to restart your computer which may clear your DNS cache.

Glitchless · 11-03-2010, 11:30 PM

There was no rollback, your characters will be just as you previously left them. We have not coded any compensation yet, that will be announced tomorrow. Essentially players did not lose anything except about 18 hours of account status time if they had premium or standard subscriptions since downtime lasted under 42 hours and the entire day that the server was down did not countdown any account time. Nevertheless we do intend on some small form of compensation not yet determined.

Again we would like to thank the players for their incredible patience during this downtime. It was very frustrating for us as well having to simply wait and wait for hardware with nothing to do. Everything has worked out for the best as we now have a more powerful and reliable server for Nodiatis.

Glitchless · 11-04-2010, 07:37 AM

The server should be back up shortly, there are still some issues with the daily reboot configuration that need to be resolved that should be within the hour.

Glitchless · 11-04-2010, 08:19 AM

DNS has propagated to virtually all nodes now, however, some ISP's and home computers are still caching and there is a Cox node that has not yet updated. If you have already recently restarted your computer and you still get the message saying we are repairing hardware then you can manually set your DNS server to 69.28.203.75
This can be done by modifying the properties of your network cards TCP/IP settings and should only be attempted by people with network configuring experience and will have to be undone to resolve other domains. Otherwise you will have to wait it out because it is not possible unfortunately to properly access Nodiatis directly by it's new IP address.