How not to announce security issues
Hosting is a commodity and the differences between companies are often negligible. I happen to use HostGator because I bought it years ago and the price is ridiculously cheap. They give me plenty of storage for my sites and I’m able to adjust everything myself using cPanel. Usually HostGator is great, but recently they sent me a message about a mandatory password reset (full-text below). This is a great example in how not to address your customers about a possible security issue. I completely believe in transparency, and at HubSpot when we screw something up we come clean about it. That being said, you don’t want to alert all of your customers to all of your staffing and security issues in such a bombastic way. This message alone mentions:
- A former employee they’re suing
- Someone who took another job and decided to cause some damage before leaving
- A trustworthy employee with VPN access, using a trojaned computer
- Their early billing system stored plaintext passwords and may have been compromised at one point
I like their reasoning for not using the same password again, but they probably could have handled the packaging better.
The email you have received from hostgator with information regarding the forced password update is in fact real. You can confirm this by hovering over the url and seeing that it links to us at https://secure.hostgator.com/password_reset/
I have included more information in this post about why we did this and why YOU SHOULD NOT change your password back to what we had on file.We have over 150 employees currently and have had dozens and dozens more come and go over the years.We had one employee that is no longer with us from a few years ago that we are in the process of suing. He will be served in the next few days. He was operations manager of hostgator for a brief time period and could have very easily taken a username / pw list home from the billing system. We don’t have any evidence that he did this but at the same time we can’t say 100% that he didn’t. I don’t believe it’s worth the risk any longer especially knowing he’s most likely going to be pretty upset about being served.
We had another employee that got another job and decided before telling us that he was going to do some damage. He logged into our ticket system and closed all the tickets in que. While we don’t have any reason to believe he ever created a list of usernames / pw we can’t rule out this possibility. I just got word that this ex employee is in the process of being prosecuted by the DA for this malicious attack. Again it’s just not worth taking the risk knowing that there’s a small chance he could have a pw list.
We recently had to let a very trustworthy / hard working remote employee go. She worked for us back in Florida for years and wasn’t able to relocate with us to texas. We kept her on as remote employee since she was unable to relocate. Just recently we discovered that the computer she was using to login with had a trojan on it. We don’t believe her hacked machine ever gave out any customer usernames / pws, but again we can’t positively say it didn’t. Due to this security breach of her machine we gave her the choice of either moving to houston to work in house or let go.
Not to long ago we allowed many employees to login to the ticket system / billing system from home using a vpn. It’s very possible one of their computers could have been trojaned and someone was building a username / pw list. We have no evidence this ever happened but it’s very possible as slim as it is.
I could go on and on about different incidents that could have resulted in an intrusion that we never became aware of. It’s that unknown that keeps me up at night! The billing system we currently use just isn’t safe with passwords displayed.
I repeat DO NOT change it back to what it was!!!!! If you do and you get hacked don’t blame the gator!
The new billing system we are about to deploy will never display a customers full password to employees. This will help protect you from a hostgator computer ever getting hacked as well as any ex employees looking to get “even” with us.
Our systems have been locked down with only office ips being allowed access. We use to allow employees access from home back when we were smaller.
Modernbill had a major exploit years ago that would have allowed a hacker to view all usernames and passwords. We patched this the same day it came out so there’s no need to worry about this particular incident, but what if there was another 0 day exploit that hasn’t been discovered? It’s just not secure having passwords in plain text without encryption as modernbill does now.
I’m sorry for the lack of notice on this update but if someone out there did happen to have a list the last thing you would want to do is give them a warning. I also apologize about some of the confusion that resulted from customers on the first few servers being updated.
Thanks for reading all!
lame
time to switch to dreamhost
Way to be a positive contributor to the blogging community, Marcel.