Www24

[Joe]

We're addressing an issue on www24. It appears that the entire drive is in failure mode... not reading sectors, etc.

We're restoring all customers from backups done approximately 5am today, on a new server (xeon cluster) -

Anyone with nameservers on www24.hostpc.com
ns24a.hostpc.com
ns24b.hostpc.com

needs to update their DNS immediately to:

ns43a.hostpc.com
ns43b.hostpc.com

Contact us via the helpdesk if there are any questions.

[Fabio]

Since we are being moved to a new server will it have the spam solution installed already?

[Joe]

This server went down at 5:14PM ET ... we immediately implemented our disaster recovery plan put in place last year.

By 6:09 PM ET, all user accounts were restored to the new servers.

By 6:43PM ET, all users were restored to the new servers - and the DNS that we could update (those registered through Enom/RegistryRocket) were changed accordingly.

1:31 for a total system restore. We're working to cut that time even further.

If you haven't changed your DNS entries yet - please get that done to avoid any further downtime.

Thank you.

[mharvey]

Joe,

Thanks for getting us up and running again. It was as painless as could be expected... (for us anyway :) I am sure Joe suffered a bit!)

If anyone else is running Gallery (v1.xx) you may find that it does not work since we have been moved. There are two issues.

- You will have to submit a helpdesk ticket to get PHP Safe Mode turned off for your domain. This would have been turned off for you on www24 but it will have to be turned off again.

- You may get gallery errors about user files and album.lock files not being writable. This is because the file owner may have changed during the restore. Gallery creates files in the ALBUMS directory with the web server (Apache) as the owner. The web server has to be able to write to these files for Gallery to run. When my domain was restored the files were all owned by me rather than Apache. I was able to correct this by using the Direct Admin file manager to change the permissions to allow the web server to write to the Albums directory and its contents.

[Sean]

Originally posted by mharvey@Jan 23 2006, 11:29 AM
Joe,

Thanks for getting us up and running again. It was as painless as could be expected... (for us anyway :) I am sure Joe suffered a bit!)

If anyone else is running Gallery (v1.xx) you may find that it does not work since we have been moved. There are two issues.

- You will have to submit a helpdesk ticket to get PHP Safe Mode turned off for your domain. This would have been turned off for you on www24 but it will have to be turned off again.

- You may get gallery errors about user files and album.lock files not being writable. This is because the file owner may have changed during the restore. Gallery creates files in the ALBUMS directory with the web server (Apache) as the owner. The web server has to be able to write to these files for Gallery to run. When my domain was restored the files were all owned by me rather than Apache. I was able to correct this by using the Direct Admin file manager to change the permissions to allow the web server to write to the Albums directory and its contents.

Quoted post


Gallery creating files/directories as APACHE is one reason we dislike it. :angry:

Just lazy programming IMHO.

[Sean]

Originally posted by Joe@Jan 22 2006, 07:10 PM
This server went down at 5:14PM ET ... we immediately implemented our disaster recovery plan put in place last year.

By 6:09 PM ET, all user accounts were restored to the new servers.

By 6:43PM ET, all users were restored to the new servers - and the DNS that we could update (those registered through Enom/RegistryRocket) were changed accordingly.

1:31 for a total system restore. We're working to cut that time even further.

If you haven't changed your DNS entries yet - please get that done to avoid any further downtime.

Thank you.

Quoted post


One of the ways we are looking at reducing this time is the Enom nameservers updates. Right now the only way to update them is to manually go thru ever domain on the server and see if it's registered via us and update them accordingly. Enom has a "bulk edit" tool, but it's extremely unreliable and caused a ton of problems (updating domains it should not have, etc) when used in the past. Once we find a workaround for this, the process can be cut by at least 50%.