Boca Raton

[admin]

Our servers in Boca Raton appear to be offline - or experiencing a network outage in that Datacenter.

Traceroutes are showing massive packet loss into the Verio Network at Boca.

I'm waiting for resolution from the datacenter now - will post updates here as the situation progresses.

Joe

[admin]

There's a network outage in the Boca datacenter. Apparently a verio core router is down. Technicians from Verio / Cogent are working on a replacement now. We expect resolution soon.

Joe

[admin]

Our helpdesk ticketing system is in the Boca datacenter... therefore, our helpdesk is offline. All HostPC staff is onboard - we're monitoring the forums for questions and posting updates as they become available.

Joe

[admin]

The network is back online and stable at this point ... servers were never really offline, just not able to communicate with the network.

We'll be addressing this situation Wednesday morning (ET) with the owners/operators at the datacenter.

Looks like we took a 45-50 minute outage on this segment of our network due to this core router - where we previously had 100% uptime on all these servers for the past two months.

I'll post a followup (postmortem) after I've discussed the issue with the datacenter later this morning.

Joe

[tpietrzak]

I am having some problems attaching to my email with any sort of stability. Is this related to the problems from this morning?

Tom

[edmicman]

Are there still problems with this? I'm having issues connecting to my site which I think is on www37. If it responds at all, it is very slow to load, and then only comes up partially. Thanks.

[Marc]

Yep and yep. Servers all seem to be slow or sluggish for everyone. I suspect Joe's on the phone doing a little... uh.. reaming :nod

[Carbo]

Same here. My site and related emails have been sluggish all day.

[mwisk]

Same here. My site and related emails have been sluggish all day.

I'm on www32 and got a phone call because emails sent to me ~10am had bounced back :(

[dbmasters]

www5 is in boca, correct? It's still offline...or appears to be.

[stuartf]

all my sites have been extremely slow or inaccesible all day and are inaccessible at the moment

[Rich]

Seem to be down again. Server status reports show servers www0 through www47 as down right now as displayed on: http://www.hostpc.com/uptime

I came here to check if it was something I did (I was trying to FTP and was having problems).

I hate to say it, but I'm glad this isn't something I screwed up this time! :)

[mwisk]

I'm on www32 and got a phone call because emails sent to me ~10am had bounced back :(


Just discovered two more emails that never got to me. One at 7:30am (EST) and one at 9:30 (EST). Who knows how many others....

[twaern]

these servers listed below, are in the old datacentre.. currently affected by the outage.

Reseller Servers

www5, www12, www16, www22, and www36

Shared Servers

www26,www28,www29,www32,www33,www34,
www35,www37,www38,www40,www42,www44,www45,www47

[tnas]

I'd add www26 to that list.... trouble with email & site all morning.

[edmicman]

Just discovered two more emails that never got to me. One at 7:30am (EST) and one at 9:30 (EST). Who knows how many others....

You're telling me.....I recently set up a forwarder to my gmail account. When things are down, messages don't queue up, they just bounce back saying the address doesn't exist (which it doesn't, it's just a forward).

I got rid of my IMAP account and set up the forward because I didn't want messages to accumulate on the IMAP server; I just wanted everything to go to gmail. Now, I imagine without there being an IMAP account if the server is not responding then it doesn't appear the address is valid and just bounces back. Gah!

[huesmann2]

Anyone else find it ironic that HostPC's site is operational but our customer server space is FUBAR?

[mwisk]

Anyone else find it ironic that HostPC's site is operational but our customer server space is FUBAR?

To be fair, no. They have lots of different servers, different customer pricing levels etc and not everything is down. Plus, last night (~4am EST) not all of HostPC seemed to be up either.

[tnas]

Seem like my site (www26), and the helpdesk are back up now.... responding nicely.
:thumbUP

[huesmann2]

I can login to DA, but can't perform any actions, though. Anything I try to do (reports, manage, etc.) it takes me to the login screen where nothnig happens.

[twaern]

I can login to DA, but can't perform any actions, though. Anything I try to do (reports, manage, etc.) it takes me to the login screen where nothnig happens.

try submitting a helpdesk ticket

http://helpdesk.hostpc.com/index.php

or at least give us a domain name, userid, server you are on or something ..

[admin]

OK, here's the story as I know it... I've been on the phone with the datacenters all day, just been AFK

1. This morning at approximately 3am _COGENT_ took over the operations of the Boca Raton datacenter (they acquired physical ownership from Verio). This was UNANNOUNCED to me - we had ZERO notice. I personally went to bed around 1:30 - and was awoken at 3 with this incident where they changed out routers. I've been up ever since.

2. The routers that COGENT installed appeared, at first, to be fine once they were turned up. Problems quickly developed when they couldn't handle the load through the morning hours as people got back to business. Cogent and our datacenter operators have been onsite ever since reprogramming BGP and routing tables.

3. Our datcenter is running a new circuit from the Ft. Lauderdale datacenter (Level3, Quest and other carriers) to the operations center inside our datacenter at Boca Raton. Once this circuit is in place, (ETA Sometime Today) - all our traffic will run over that enhanced bandwidth pipe and away from the damn Cogent crap.

4. The datacenter owner, Jim Thomas, and his crew have been onsite since this occurred at 3am, and will continue to be onsite until it's resolved. This is an "all hands on" event.

To address some of your questions:


Just discovered two more emails that never got to me. One at 7:30am (EST) and one at 9:30 (EST). Who knows how many others....

While routes recoverge and settle down, there will be intermittent crap like this happening. Thats why people PLAN for events such as this - and HAD WE KNOWN it was going to happen, we certainely WOULD have planned for this issue by moving our servers OUT of BocaRaton prior to Cogent BW ever touching our servers.


You're telling me.....I recently set up a forwarder to my gmail account. When things are down, messages don't queue up, they just bounce back saying the address doesn't exist (which it doesn't, it's just a forward).

Two comments: 1) Dont change DNS too much, too often - it'll only confuse it more. 2) Open a helpdesk ticket to be removed from the spam filtering.


Anyone else find it ironic that HostPC's site is operational but our customer server space is FUBAR?

In this thread (http://www.hostpc.com/community/showthread.php?t=2802) I explained that for many reasons we're changing everything to the Ft. Lauderdale datacenter ... which does not have Cogent bandwidth, and in fact has 4 redundant carriers. This was a plan we scheduled for our January trip to Florida ... we may need to accelerate it. HostPC.com is hosted already in the new datacenter ... however our ordering system is still located in BocaRaton (affected). Trust me, if this was a major conspiracy to bring customers sites down, but leave hostpc.com online, I would have made sure the order system stayed up :) In fact, there are over 60 servers in the Ft. Lauderdale center - only about 15 left in Boca Raton.

Info on the new datacenter: http://www.hostpc.com/community/showthread.php?t=2801

So, what's the plan? First, we get that alternate circuit up and running asap ... Cisco and datacenter personell are onsite getting that working - properly - the first time. Next, we accelerate our plans to move servers (physically) out of the Boca datacenter and into Ft. Lauderdale. Since this is a physical relocation of servers, there will be downtime while in transit. Most likely this will not be completed until the January trip- but MAY be completed earlier depending on the status of that link.

Worst case scenario, we have the datacenter move the machines physically to the new center. I need a day or so to plan the IP migrations. This is a last resort.

What can YOU DO?

First option - ride it out for a few more hours, see what happens.

Second option - open a helpdesk ticket, ask for a new account to be setup in the FLL datacenter - include your domain name, username and OFFSITE EMAIL - NOT YAHOO!. We'll setup a new account - you can upload your backups (please dont ask us to transfer data - we're kinda busy with this issue currently). On the new servers, everything appears fine - stable mail, web, etc. We've got plenty of machines in FLL .... just need you to ask. You can be stable in a matter of a couple hours, and avoid any future downtime due to the move.

I dont like this situation - trust me, given the option of Cogent or a tin-can/string - I wudda chosen the tin can. I hate cogent - and never would have voluntarily put our customers on their network - we're better than that.

I'll post more information here as it comes in... Thank you for your patience and understanding.

Joe

PS: Our uptime guarantee does apply to this situation. This will likely be an un-reimbursed SLA issue with our datacenter, but I WILL honor any request made under our uptime guarantee policy. We've already lost 40+ minutes this month (which brings us below 99.9%) - even if it was at 3am ET. I'm gonna lose my shirt, but if you truly were inconvenienced today, please follow the terms of the SLA we offer and we'll issue one month of service credit for this outage. http://www.hostpc.com/community/showthread.php?t=2856

[edmicman]

Thanks for the update guys! I was recently planning on upgrading my account to one of your new hosting plans, and in turn moving my account to the FtL location; this just makes me want to speed up that process. I think I'll probably place the order tonight, hopefully by this weekend I can get the backup restored at the new place!

[ActuaryTm]

Thank you very much for the updates, Joe.

Any chance the helpdesk is also hosted in Boca? I've had a myriad of issues attempting to login and respond to a ticket all day long (in addition to the hosting related issue).



In this thread (http://www.hostpc.com/community/showthread.php?t=2802) I explained that for many reasons we're changing everything to the Ft. Lauderdale datacenter ...

I take it this move of all servers to the Ft. Lauderdale datacenter is definite then? If so (and only a question of when - now, or sometime later this month or January), I may opt to move now before adding additional sites to my account.

Edit: added a link to this response in the present ticket (rather than creating a new one).

[admin]

Thank you very much for the updates, Joe.

Any chance the helpdesk is also hosted in Boca? I've had a myriad of issues attempting to login and respond to a ticket all day long (in addition to the hosting related issue).


Yes, I forgot that - the helpdesk IS running in the Boca Facility... I'll be changing that out tonight.



I take it this move of all servers to the Ft. Lauderdale datacenter is definite then? If so (and only a question of when - now, or sometime later this month or January), I may opt to move now before adding additional sites to my account.

Edit: added a link to this response in the present ticket (rather than creating a new one).

Yes, it's been definate for a while... was taking our time as we've enjoyed consistantly near-perfect uptime in both centers. Now it's just a matter of if this fix tonight will work permanently - or at least till our scheduled trip in January - or if we need to effect the change now.

[ActuaryTm]

Now it's just a matter of if this fix tonight will work permanently - or at least till our scheduled trip in January - or if we need to effect the change now.

I believe I'll sit tight and wait for your further updates later tonight, then.

No sense creating additional workload for you when it may be unnecessary (at least for the time being).

Love that Denver to Boca hop:

http://img65.imageshack.us/img65/2989/hostpcwww80traceroute12tf3.jpg

[app-o-rama.com]

Anyone else find it ironic that HostPC's site is operational but our customer server space is FUBAR?
In this thread (http://www.hostpc.com/community/showthread.php?t=2802) I explained that for many reasons we're changing everything to the Ft. Lauderdale datacenter ... which does not have Cogent bandwidth, and in fact has 4 redundant carriers. This was a plan we scheduled for our January trip to Florida ... we may need to accelerate it. HostPC.com is hosted already in the new datacenter ... however our ordering system is still located in BocaRaton (affected). Trust me, if this was a major conspiracy to bring customers sites down, but leave hostpc.com online, I would have made sure the order system stayed up :) In fact, there are over 60 servers in the Ft. Lauderdale center - only about 15 left in Boca Raton.

I would suggest that it would be better for at least some of HostPC's customer support servers (e.g., the server that hosts this forum) to be housed a data center other than the data center where all of the customers are housed.

If a connectivity issue arose with the FLL data center, not only would all our sites be down, we would not be able to communicate with HostPC. Having a hosts support server on a completely separate network makes a lot of sense to me.

[admin]

We have a backup site specifically for emergencies if hostpc.com goes offline.

In the event of a total network outage in Florida, we can point http://www.hostpc.info to any server in the world that isn't affected to provide constant updates.

We used this once before, it worked out very well.

[randomcomputers]

aaah that explains the problem then. I woke up this morning and had numerous calls from customers complaining about their email not working this morning. thanks to Sean who alerted me to this thread in the help desk ticket I opened.

hope it was an isolated issue that doesnt happen again. keep up the good work guys!

[app-o-rama.com]

We have a backup site specifically for emergencies if hostpc.com goes offline.

In the event of a total network outage in Florida, we can point http://www.hostpc.info to any server in the world that isn't affected to provide constant updates.

We used this once before, it worked out very well.

Ah. I had forgotten about that.

[Rich]

OK, here's the story as I know it... I've been on the phone with the datacenters all day, just been AFK

. . .
PS: Our uptime guarantee does apply to this situation. This will likely be an un-reimbursed SLA issue with our datacenter, but I WILL honor any request made under our uptime guarantee policy. We've already lost 40+ minutes this month (which brings us below 99.9%) - even if it was at 3am ET. I'm gonna lose my shirt, but if you truly were inconvenienced today, please follow the terms of the SLA we offer and we'll issue one month of service credit for this outage. http://www.hostpc.com/community/showthread.php?t=2856

Joe,

Thanks so much for your update on this issue. You always do a great job of keeping us informed and of being more than fair with the situation.

Even though I'm eligible for some refunds with your guarantee with the few accounts I have on these servers, I'd rather not ask for any service credit, and encourage you to just keep doing a great job! :thumbUP

Thanks again!

Rich

[admin]

We're still working on this issue....

There apparently is still issues. I'm working on that backup plan to get servers moved asap.

Joe

[Rich]

Well, the servers are up, but either there is a heavy load on the servers or the connection to them is terrible.

I can barely do anything... I'm been trying to create and download a backup for some time now. I'm not having much luck.

[admin]

If anyone wants to get setup on the FLL servers today, please PM me here in the forums until the helpdesk is re-available.

I'll answer all pm's as quickly as possible.

[admin]

My personal AIM username is: jmack518

I can also be contacted through there temporarily

[admin]

How do you take your coffee? & don't say in a MUG.:thumbUP

Like my women, light and sweet :)

[admin]

I'm investigating options to go to FL today or tomorrow ... will update you as soon as possible.

Joe

[edmicman]

Doh, I just got the F L site set up last night, I was going to backup and restore today - I just need to be able to get into my site to do the backup! Here's crossing my fingers that there's a window I can get in, and many thanks to you guys in keeping us updated of what's going on! Keep up the great work!

[ahalikias]

Well, thanks for the updates but this is getting kinda ridiculous now.

It's day 2 of trouble, my domains are down completely, I am having customers complaining all around for bounced emails and the helpdesk is down - so I cannot even move my domains to the FL datacenter!

Although even if I could get to helpdesk, there is NO way to get in my domains to get a recent backup. Being Thursday, my downloaded backups are 5 days old.

Not your fault, but it is ridiculous that Cogent would go into their new datacenter and replace routers on the spot instead of migrating to their routers.

And what the heck kinda of moron down there cannot get the routers sorted out in 48 hours?? I've been involved in network installations for years and this is just obscene given the size of the datacenter. We are not talking about "issues", we are talking about bringing everything to a halt!

When you get to Boca Raton, land a square one right on the jaw of whoever *@!#%$ is responsible for this.

In the meantime, is there a game plan?? Is the Boca Raton datacenter completely down???

[edmicman]

Looks like things are "up", but sloooooooow. I'm doing my best to get a backup made and downloaded off!

[admin]

In the meantime, is there a game plan?? Is the Boca Raton datacenter completely down???

Game plan was outlined above, please read the post.

Yes, the entire facility was down. The facility houses major business including 3 major national newspapers, healthcare agencies and dozens of businesses and a tech shool - all went out I believe during this outage.

[admin]

I've ordered a slew of new servers to be turned up in the FLL facility ... most manpower is working on the outages and routing issues but I've been promised new servers online within a few hours to supplement the 60 or so that are already there.

Helpdesk is functional at this point, please open a ticket if you'd like to move manually ... and get the issue resolved now. Please - we're working VERY hard on provisioning new servers, dealing with customer inquiries, etc - we can NOT move your data at this time. All Others will be physically picked up and moved to the datacenter - but I dont have a schedule yet for when that might happen. I still need to coordinate a LOT of things.

Joe

[admin]

To lighten the mood, someone just sent me a link that made me LOL for a minute or two - hopefully it'll inject some humor into your day.

To save the guys bandwidth, he asked that I not link it here, so I copied the images to a different page on our server which (fortunately) can handle the load.

Enjoy: http://www.hostpc.com/kitties/kitties.htm

Joe

[dbmasters]

5 seems to be up, 37 is still down. I just helped a client release his web based business on 37 a week ago...this really isn't too cool, not that it's HostPC's fault or anything, I recognize that, but any update on the sites getting back online?

[huesmann2]

Fatal error: Call to undefined function: kbintro() in /home/helpdesk/domains/helpdesk.hostpc.com/public_html/index.php on line 154

[twaern]

Fatal error: Call to undefined function: kbintro() in /home/helpdesk/domains/helpdesk.hostpc.com/public_html/index.php on line 154

reload .. helpdesk is working, i am currently in their answering tickets ..

[admin]

Clear your cookies if this error happens.

Thanks



Fatal error: Call to undefined function: kbintro() in /home/helpdesk/domains/helpdesk.hostpc.com/public_html/index.php on line 154

[admin]

All servers are up and online at this point...

Joe


5 seems to be up, 37 is still down. I just helped a client release his web based business on 37 a week ago...this really isn't too cool, not that it's HostPC's fault or anything, I recognize that, but any update on the sites getting back online?

[dbmasters]

yep, it is now, thanks.

[admin]

I've just received a semi-postmortem from our datacenter ... I'm evaluating it and waiting for the "final report" before releasing it here - and after I get their permission to post a summary.

[dbmasters]

My rather wise father always told me that it's easier to beg forgiveness than ask permission...:nod

[tonydi]

My rather wise father always told me that it's easier to beg forgiveness than ask permission...:nod

I know for a fact that Joe likes the permission thing better. :D

[edmicman]

So I was finally able to make a backup, downloaded it, and am in the process of uploading it to the new server. I'm using Filezilla, and it starts off fine, and then grinds to a halt, then the client reinitializes the connection, it resumes the upload, goes for a little bit, then repeats. Eventually it says some error about "too many retries" and kicks me out, without the file having been uploaded. Is there a trick to this? My backup file is ~200MB....do I need to do anything special?

Also, if later I do a backup of just email, can I upload and restore that and have it just restore the emails? Ie, I want to have as current of a copy of my users emails as I can. Thanks for any info!

[admin]

No "permissions" necessary ... flames and kudos accepted 100%

[Rich]

Joe,

Alll I can say is YOU and you STAFF are amazing!! :thumbUP

I filled out a help desk to move my servers to the FLL datasite and I barely finished filling out the help desk tickets when the emails came in to let me know that these new sites were setup and ready for me!

Considering all that is going on, I find that AMAZING! :)

One tech question. On my previous server (www45), I had this in my .htaccess to allow standard html files and links to files with a .php suffix to be processed as a php script.


AddType application/x-httpd-php .php .php4 .php3 .html .phtml .htm

On the new server this doesn't work. Is there something different about these installs?

I had a bunch of links such as


<a href="myphpfile">link</a>


That I had to change like this to get them to work properly:


<a href="myphpfile.php">link</a>


Any clues would help!

[randomcomputers]

The phone calls and emails from disgruntled customers have finally settled now that the outage is done.

Can you give us any feedback on how this was resolved? And any assurances that this wont happen again after 2 consecutive mornings?

I'm not upset. I applaud Joe and the gang for all their hard work they do.

Frustrated uggg....

[admin]

Yes, a rather wordy summary will be coming very shortly, I'm adding the spit'n polish to it now :)

[admin]

Ok, I've seen, read and digested the datacenter "pre-postmortem" - or a summary of facts that led up to and through this incident. I'd like to give a short (ya right Joe) summary of what I've learned, what we'll do about it, and how we'll move forward.

1. The "cogent deal" wasn't supposed to happen until sometime in January. Switching it over early was really a FUBAR decision, that IMHO wasn't planned correctly, nor were all parties (including the datacenter managers/owners) notified of in advance. We all learned it was happening when the network monitoring alarms went off - at the same time.

2. Next Friday, a link between the two datacenters (dedicated fiber) was to be turned up to connect the Boca Raton facility with the Fort Lauderdale facility. This schedule was rapidly advanced yesterday - and we anticipate it being completed very soon - it's in the final testing stage now). This will allow the Boca Raton facility to retain it's IP addresses and at the same time give the servers in Boca Raton full access to the Ft Lauderdale (FLL) Triple Tier 1 providers for maximum redundancy. Conversely, I *BELIEVE* Cogent BW will be available as a failover should the need arise (god I hope not).

3. Because we're running these 18 servers on Cogent BW currently, our normal server loads of <1 have magnified trying to process all the spammers through the Cogent pipes. Our incoming TCP connections have quadrupled since this occurred on Wednesday. Most of the BR servers are handling it well, others firewalls are still adapting and learning the bad from the good. This WILL stabilize over the next 24 hours.

From the datacenters notes:

Here's what happened on Wednesday:

1. Late tuesday night we received notification of an emergency maintenance at 2am.
2. We call support to make sure they are not working on both of our links.
3. They also tell us they are just upgrading the supervisor cards in the routers.
4. They assure us no other changes will be made.
3. They assure us that only one router would be worked on at a time.
4. Wednesday 2am: one router down, then back up.
5. Second router down.
6. At this time Boca is down.



Where do we go from here.

I've made an "executive" decision to NOT force a move of all servers (physically) from the Boca Raton center to FLL. I know, this sounds crazy, but because we've had so FEW problems at BR, my confidence level is extremely high that they will get the private link established/tested/working ASAP, the first time, with no issues. We're also approaching a critical calendar time (Christmas/Hanukkah) where it's an extremely stressful time for e-tailers, site operators, etc - who may or may not be around to monitor the transitions, etc during a potentially difficult/stressful time of the year.

We'll re-address this situation daily - and it may be subject to change depending on the stability day by day. This decision could change at ANY time.

We're offering ANY / EVERY customer, including those dedicated servers in BR the option of moving to the FLL facility at YOUR CONVENIENCE. If you want to move now, manually, fine .. just open a helpdesk ticket and we'll get it addressed as soon as possible. If you're "ok" where you are, and have confidence in my/our decisions, then I'd encourage you to ride it out.

(latest update is that the dedicated link is nearly ready ... more details on that as they come in)

Some of our staff will be in FLL for the "Customer Appreciation Party" mid January - I'll plan on having everything done/ready during that trip.


If you have _ANY_ questions, please feel free to ask them here, or via email. I'll be happy to answer anything I can with the facts I have in front of me - and if I can't find the answer from my info, I'll find out and let you know.


Thank you for your continued patience and consideration.
Joe

[ActuaryTm]

To lighten the mood, someone just sent me a link that made me LOL for a minute or two - hopefully it'll inject some humor into your day.

To save the guys bandwidth, he asked that I not link it here, so I copied the images to a different page on our server which (fortunately) can handle the load.

Enjoy: http://www.hostpc.com/kitties/kitties.htm

This seems more apropos:

http://img243.imageshack.us/img243/5230/inurbocadatacenterzgr6.gif

[ActuaryTm]

Thanks so much for your update on this issue. You always do a great job of keeping us informed and of being more than fair with the situation.

Even though I'm eligible for some refunds with your guarantee with the few accounts I have on these servers, I'd rather not ask for any service credit, and encourage you to just keep doing a great job! :thumbUP

Thanks again!

I fully concur with and endorse this above statement, which worded my thoughts much more eloquently than I could have.

[admin]

Hahhhahahah - I"ll pass that on to the folks in BR/FLL ... thanks !

[ahalikias]

Yeah, I am with the rest on this too.

Although I am eligible for a few refunds, I am not going to claim them simply because I have a business too and I know that there is no point to be made by stressing financially the people that bust their humps to keep this working.

This does not mean that I am happy with what happened; it just means I don't blame Joe or his crew.

Now the question is this: Boca Raton or Fort Lauderdale ?

If the datacenters get the fiber link to each other, will this be a failover link or a private link between the Datacenters?

I would trust Joe and stay with Boca Raton but given this FUBAR, I am a little concerned that they are trying to establish this fiber link as we speak. Do they plan on turning it on over the weekend or as soon as possible??

I just don't want to get in a situation where for a 3rd day our domain is down because someone messed up the routing tables. Let's at least finish the business day today before they change anything again.

Obviously Joe since you are not the operator of either Datacenter there is not much you can do. I am just inquiring on the situation.

Thanks.

[admin]

Yeah, I am with the rest on this too.

Although I am eligible for a few refunds, I am not going to claim them simply because I have a business too and I know that there is no point to be made by stressing financially the people that bust their humps to keep this working.


Thank you!


This does not mean that I am happy with what happened; it just means I don't blame Joe or his crew.

Now the question is this: Boca Raton or Fort Lauderdale ?

This was in the Boca Raton facility



If the datacenters get the fiber link to each other, will this be a failover link or a private link between the Datacenters?

[/quote]

A private link - not controlled by Cogent - AT ALL



I would trust Joe and stay with Boca Raton but given this FUBAR, I am a little concerned that they are trying to establish this fiber link as we speak. Do they plan on turning it on over the weekend or as soon as possible??




I'm anticipating this link being active in 48-72 hours - maximum. It's in the final testing stages now.



I just don't want to get in a situation where for a 3rd day our domain is down because someone messed up the routing tables. Let's at least finish the business day today before they change anything again.



Absolutely ... at this point, we're up, and no other changes will be made (DC operators have been assured of that by Cogent - for what thats worth)



Obviously Joe since you are not the operator of either Datacenter there is not much you can do. I am just inquiring on the situation.

Thanks.

I'm not the operator, and no, this is NOT "our" datacenter ... we colo and lease servers in both facilities (same owner of both facilities, just diverse locations) However, with that said, we are the largest customer of the two datacenters - more dedicated servers than I believe anyone else. We've been with this company for several years now, and as such, we've established an EXCELLENT working relationship, and perhaps a few "IOU's" that allow me to call in favors for things that are within THEIR control. I've personally met the owners, had drinks/dinner with them, hell, during the hurricane I slept IN the datacenter (after that wonderful trip to FL the day after Wilma hit last year). I can't stress enough how nice, how professional and how accomodating these people are - AND they have the knowledge and expertise to get us through this. I wouldn't want anyone else.

After the hurricane last year, these folks promised to make it right - and they did - BIG time. They purchased their own facilities, with multi-honed links, exceptional services, security and infastructure (all developed within a year). That's a HUGE task to undertake - and with little sleep, Jim, Robin and the crew in Florida have done WONDERFUL things. I can't praise them enough. They've gotten my ass outta the fire more than once, and I'm pretty sure I saved their hineys a couple times too. We've just got a wonderful relationship.

So yes, I'm convinced this is the best course of action. Of course, if something changes, we'll adapt to the change, but always in the best interests of our customers. Customers put us where we are today ... and will move forward with us into Tomorrow! Without you guys, this success called HostPC would never have happened. For that, I'm very thankful.

Joe

[ahalikias]

Joe,

thanks for the answers. I'll stay with the Boca datacenter then.

Thank you and try to get some sleep over the weekend.

Andrew

[champion6]

I don't understand what changes have been made to correct the situation that occurred, but I'm on www38 and the response of my Joomla-based site is much quicker than it's been for the last few months.

It had gotten to the point that I would sometimes wait 5-8 seconds for my site to display a page. The little that I know about interpreting a trace route seemed to indicate the bottleneck was in Verio. Or I thought maybe my site shares a server with another very popular site.

Anyway, I'm much happier now :nod

[admin]

I believe, but have not confirmed yet - the fiber link to the new datacenter is in place ... I still need to verify this - that would explain your increased performance.

[champion6]

As a matter of fact, I don't think it's related to the fiber link. I noticed the improved performance right after I first read about the problems.

The fact is that I don't really know the cause for the slow response nor the reason it has improved. Generally speaking, how does the performance of and load on www38 look?

[admin]

www38:

16% drive utilization
13:03:36 up 12 days, 1:03, 2 users, load average: 0.38, 0.23, 0.18

load is VERY nice - just where we like to see it.

I was mistaken ... fiber transfer has NOT happened yet - I'll be posting an announcement soon.

Joe

[admin]

Update Here: http://www.hostpc.com/community/showthread.php?p=14746

[ActuaryTm]

Thank you for the continued updates and service, Joe.

[mwisk]

EDIT: Never mind, apparently the spam filter occasionally doesn't let emails through...


<mail@my domain.org>:
199.237.54.205 does not like recipient.
Remote host said: 550-Email blocked by ORDB - to unblock see
550 http://www.hostpc.com/emailnotdelivered.php (http://www.hostpc.com/community/../emailnotdelivered.php)
Giving up on 199.237.54.205.


Seems like I'm still having issues with email bouncing when it's sent to my account. I've gotten reports of one last night and one this morning.

Will moving to the FLL location solve this?

[admin]

Open a helpdesk ticket for mail issues

[Michelle]

EDIT: Never mind, apparently the spam filter occasionally doesn't let emails through...


<mail@my domain.org>:


199.237.54.205 does not like recipient.


Remote host said: 550-Email blocked by ORDB - to unblock see


550 http://www.hostpc.com/emailnotdelivered.php (http://www.hostpc.com/community/../emailnotdelivered.php)


Giving up on 199.237.54.205.

Seems like I'm still having issues with email bouncing when it's sent to my account. I've gotten reports of one last night and one this morning.

I'm having the exact same problem with couldntgetagoodname.com e-mail. I'll open a help desk ticket.

Michelle

[tnas]

I'm having trouble getting to my site on www26, and the helpdesk seems to be down too. I'm getting "Can't connect to the server." errors.

Any news?

[admin]

http://www.diratslabs.com/

Comes up fine for me... helpdesk as well

What's your IP address?

http://www.hostpc.com/whatsmyip.php

[tnas]

http://www.diratslabs.com/

Comes up fine for me... helpdesk as well

What's your IP address?

http://www.hostpc.com/whatsmyip.php

The domain with the problem is http://www.Agawamchurch.org (http://www.agawamchurch.org/)

I'm coming from ChoiceOne at IP: 216.153.219.122

Is it my provider? routing? I'll try a traceroute....

[jFree]

http://www.suneagleclan.com is also down this morning. Hosted on www32 I believe.

[tnas]

It may be a local routing issue. It's odd though, that I'm getting POP email from 2 other hostpc domains (www55, www51) just fine. Main HostPC page and forums just fine, but server error on the helpdesk and mail from www26.

I'll wait it out awhile...

[admin]

Might be a routing issue (again :( ) into Boca Raton - I'm waiting on the datacenter now to talk with Cogent (we have GOT to get off their pipes)

[admin]

This appears to be a Level3 routing issue - the NOC administrator is working on the situation and will keep us advised - likewise, we'll keep you updated

[jFree]

Thanks for the update. Access to my site (suneagleclan.com) has "improved" from can't find the server to server has timed out.

[tnas]

16:35 Eastern time, things look good right now. Site and helpdesk all available.

[admin]

We're experiencing a temporary routing issue in the Boca Raton center - thanks once again to everyone's favorite provider, COGENT!!

GRRRRRRRRRRRRRRRRRR

We're working on it folks.

[admin]

We'll have all servers out of Boca Raton by 1/21/07.

This is farkin ridiculous :(

Sorry folks, looks like we're back online.

Joe

[admin]

I'll be in Florida January 20 - and will, by then, have coordinted the move of servers to the Fort Lauderdale datacenter where this issue will not be present any longer. We're finalizing plans for this move, which involves a physical relocation of the servers left in Boca Raton and changing of IP addresses for each box.

This move will take about 4 hours overnight, so there will be additional downtime involved over that January weekend. I'll post a followup schedule as soon as it's cemented.

In the meantime, if anyone wants to move to the FLL datacenter NOW, we've got plenty of open space ready for clients on NEW servers there. Open a helpdesk ticket and we'll get you setup - which will negate any downtime you will experience either from future cogent issues, or the downtime when we move the boxes.

Include your main domain name, username, off-site email address and a copy of your billing receipt and we'll get it handled.

Joe

[tnas]

Is there a list of servers that are still in the facility that are affected by this? Not exactly sure where all my servers are located.

[admin]

Anything below www50 is in BocaRaton - check your nameservers for ns50a and below.

Joe

[admin]

Update received from the datacenter manager:



It is my very great pleasure to inform all of you that our link to cogent has been removed.

Yesterday, there was a brief outage caused by cogent. I want to keep you all informed, so here is the story of that outage.

Last month, we had informed cogent that we would be removing the link and they scheduled the removal to be the beginning of the year. However, as the day approached, they told us they were backlogged and they would contact us with a schedule for the removal. This was fine with us as then we would be able to schedule a maintenance and inform our customers.

Well, the next thing we know, it's yesterday at 6pm and they disconnect the link. No notification whatsoever.

At this point most of the boca facility is down and we are scrambling to figure out what they did wrong this time.

We finally determined that when they removed the link, they did not remove the routing configuration for our ips from their network. Also, because of manual, on the spot, fixes that they did the last two times we had problems with them, the internet thinks all the traffic should be routed to cogent.

Traffic to cogent.... Link down..... translates to boca being down.

all I can say is..... IDIOTS!

We will never, NEVER use cogent ANYWHERE, EVER!

Ok, sorry for the venting. After we get all the right people involved they finally get everything fixed.

Now, at this time, we are not doing anything with cogent. There is a possibility that they could still affect our routing if they add back our ip routes to their network. So, we are doing everything we can to make sure that all of our information is cleaned from all of their automated routing systems.

Once everything is clean, we are free from cogent.

I couldn't be more relieved, and I'm sure you guys feel the same way.

Again, my apologies for the interruptions.

[tnas]

Anything below www50 is in BocaRaton - check your nameservers for ns50a and below.

Joe

I have a 55, 51 and ding, ding ding, a 26!

I'll get moving on moving the 26. Thanks.

[app-o-rama.com]

With Cogent close to out of the picture, is the plan to still move out of the Boca Raton DC?

[admin]

Yes, absolutely - the new datacenter far exceeds the specs of the new center and has multiple carriers for bandwidth :)

[dbmasters]

new datacenter far exceeds the specs of the new center

eh? :confused:

[admin]

er, new far exceeds the old ...sorry

[champion6]

In the meantime, if anyone wants to move to the FLL datacenter NOW, we've got plenty of open space ready for clients on NEW servers there.
If I move now to a new server, will my AWStats historical data move with my web pages?

[champion6]

If I move now to a new server, will my AWStats historical data move with my web pages?
Bump for an answer

[Nick]

It is being worked on to make it possible. Which means it isn't at this time.
No ETA on when it will be possible.