If you’re one of the handful of people who actually read the drivel that I post here, you may have noticed that the site was down for a while. That’s because my previous Web host, Hostforweb, spectacularly shit the bed two weeks ago after many years of being one of the better hosting values in the business.
Information of an official sort being impossible to come by, all I can say with certainty is that HFW has been bought or merged with another company that is based overseas. My guess is that they’re in Belarus now based on the (unauthorized) root login to my last remaining server on their service this morning.
Before the takeover or merger, I’d been a customer of HFW for more than 15 years. In fact, until two weeks ago, I thought they were one of the best-kept secrets in Web hosting. Their servers were located at one of the better datacenters in North America (Equinix Chicago), the company had solid equipment and infrastructure, their support was better than average, and their prices were very decent.
All that ended two weeks ago.
Some time in late December, customers were sent an email advising us of “infrastructure upgrades” that would be performed over the Christmas weekend. Supposedly there would be little or no downtime. We were not advised that this was a migration (much less an overseas one), nor that IP addresses would change, which is a very critical thing. We were told the company was making “infrastructure upgrades,” as if they were talking about a new coffeemaker in the break room.
With sufficient notice and proper execution, a move to a new datacenter should be a yawner. The TTL (time-to-live) on the DNS entries on the old server are all shortened to something ridiculously short; the new servers are installed at the new DC; the data on the old servers is rsync’d to the new servers; and when that’s done, DNS on the old server is is pointed at the new ones. Then you make the nameserver changes, and life goes on with little or no downtime or lost data.
The way HFW did the migration turned out to be two weeks (and counting) of being dragged through the previously-unknown Tenth Circle of Hell. There were about 40 hours of complete downtime during which IP addresses were changed before the customers were notified, hostnames were changed, firewalls were disabled, and root passwords were changed — all without the prior knowledge and consent of the server’s lessees or owners.
To add insult to injury, there was a dearth of communication between HFW and their customers. There was nothing in their forum (which now appears to have been shut down), nothing on Twitter, nothing on Facebook, and nothing anywhere else that I know of. Ticket responses were ridiculously slow (when they were answered at all), and most of the responses seemed a lot like boilerplate to me.
Consequently, I spent the entire Christmas weekend in my office, hoping for some clue as to what was going on, and trying to think of how to respond to my clients’ complaints. I couldn’t answer them intelligently because HFW was keeping everyone in the dark.
I’ve been in this business for a while and I knew what was coming. I’ve seen it before. When good hosting companies go bad, they do it with gusto. They go down in a blaze of glory. Recognizing the signs from previous bad experiences, I set up servers at another company; and as soon as my servers on HFW came back online and were stable enough to move data, I started moving sites over to the new servers while the moving was good.
I had to do most of the transfers by using SCP to push the data from the old servers to the new ones and them semi-manually restoring on the receiving servers. All of the servers in question had cPanel installed, but HFW appears to have been blocking cPanel’s Transfer Tool. I can’t say that for sure, but I know they were refusing the cPanel connection and inbound SCP. But I was able to move the data using outbound SCP from the old servers.
One large site had to be restored from a backup I had on Amazon S3 because it had disappeared from the server on HFW. It makes all the money I’ve payed Amazon for backup space over the years worth it.
So that’s how I spent the Christmas holiday. But wait: There’s more.
At about 5:00 p.m. New York time on Friday of the New Year’s weekend, I was informed of other “critical” things I had to do to keep the remaining domains online. Yes, I said 5:00 p.m. on Friday. Among these were yet another batch of IP changes. They just selected a handful of IP addresses matching the number that the customer was paying for, and attached them randomly to various services as they saw fit, without thinking to inform the customers until after the assignments had been made — at 5:00 p.m. on Friday of the New Year’s weekend.
I spent a few hours fixing that mess on Friday afternoon, and spend most of Saturday assisting clients who were having problems. On Sunday I got to relax a bit, and I thought the worst was over. But then on Monday, New Year’s Day, all the mail on my one remaining HFW server stopped working. I have no idea why. It responded to ping and I’d made no changes to Exim or Dovecot. It just wasn’t processing mail.
That was the last straw for me. I provisioned yet another virtual server at the new provider and SCP’d all the sites over (because once again, cPanel’s Transfer Tool and incoming SCP appear to have been blocked). I SCP’d the backup files from the old server to the new one and restored them manually, restoring sites one at a time while the remaining sites were still transferring because the transfer rate was hovering around a pathetically slow (for a Web server) 3 – 4 mbps.
That’s how I spent New Year’s day.
Once all the clients’ sites were transferred, I set DNS on the old server to push traffic to the new one, and decided to leave it that way for a few days to avoid client-side caching problems. Then this morning I got an alert from the server of a successful root login from an unknown IP in Belarus. I logged in and found that the firewall had been disabled. Having no way to know whether it was a legitimate connection from HFW (because they don’t communicate with their customers anymore), I re-enabled the firewall, blocked the entire IP range, and restarted SSH.
I still don’t know whether the login was HFW or some random Belarusian hacker because, once again, no one told me anything in advance. Either way, I blocked them. At this point the server is just doing DNS duty, and I don’t want HFW touching it. If it’s still broken, I don’t care. It can do DNS, and that’s all I need it to do.
HFW can, of course change the root password again if they want to. But I can change it back in the Client Area. Or maybe not. I really don’t care anymore, to be honest. DNS resolution is much faster these days than it used to be. I probably could take the old server down now and no one would know the difference.
In case you’re wondering, the new company I chose is Turnkey Internet in Upstate New York. They’ve been like a breath of fresh air. Immediate provisioning, scary-fast ticket responses, and phone support by real, live, honest-to-goodness American technicians sitting in the company’s own datacenter. Moving to Turnkey is the only good thing that has come of this nightmare with HFW. I think I may have stumbled across the perfect hosting company.
In closing, my advice with regard to Hostforweb is to just stay away. If you’re already with them, then run — don’t walk — to another provider. Seriously. Do it now. This very second. Unless you’re a masochist, I can think of no reason to stay with the “new” Hostforweb.
Richard