Home / Drupal and the $6000 hosting bill, getting to the truth quickly

Drupal and the $6000 hosting bill, getting to the truth quickly

It's political season here in the United States, and that means it's rumor time. Research has shown that if we are exposed to information as little as eight times we are more likely to believe it. Logic, facts, reason don't necessary apply to our human brains. Political strategist know this, marketers know this and they can use it in influencing you. Journalists know this too.

On Monday morning, Drupal got caught up in a [Updated]conjecture that it was the source of a bug that led to a $6000 hosting bill. It turns out that the backup and migrate module was installed in a hosting environment with limited resources. The back-ups were running automatically but failed to complete, most likely due to a PHP timeout or CPU resource limitation on compressing the back-ups. The result was that the temporary files created as part of the back-up were not being cleaned up and were accumulating. That accumulation lead to 100 GB storage overage charge which resulted in a greater than $6000 hosting bill. In most hosting environments, the temporary directory would have filled up and that would have been the end of it.

The problem was posted in the Drupal forums and the community jumped in and tried to help, making suggestion after suggestion, with out a full list of modules. The site owner decided to publish an article (part 1, part 2) to a popular political site Huffington Post, and then it got picked up by the consumerist.com. The hosting company jumped on to the story and worked to resolve it. But by that time the story had gotten picked up by Digg.com and was on the front page.

Over the next 24 hours over 40000 people read that "Drupal, which, like a cancerous tumor, was unstoppably copying thousands of temporary files" was the technical source of this problem. The article was really a criticism of the hosting company but facts were not going to get in the way of a good juicy read.

At the time the hosting company was running well over 10000 instances of Drupal, which were not reporting the same problem. In fact, according to the Drupal.org project usage logs there were at least 1719 sites using the 5.x version and 2093 using the 6.x version of the backup and migrate module without these problems.

I contacted the editors of the article and they were kind enough to update the article to take the heat off Drupal. But they put it on the less defended third party module. They also updated the article to include the technical explanation I provided. I am glad they were responsive.

I managed to get the module maintainer, Ronan Dowling on the phone and he was very helpful. Here's what he had to say:

I'm glad this situation ended well for Adam. I hope I can figure out exactly what happened here so I can make sure other people don't run into this, and in the meantime I'm going to redo the module's docs to make the configuration and troubleshooting steps easier.

Some people believe that responding to these kinds of allegations are below us. I tend to agree. I just wish it was below our brains.

Comments

Posted on by jwhatcott Whatcott.

Good post, Kieran. Thanks for jumping on this situation and helping connect all the right dots to get the truth out there. Unfortunately, as you pointed out, the first impression was already set for many and "Drupal = risk of insane hosting company overage charges" will be sealed in the minds of tens of thousands of people who read the original errant story. A painful fact of life, I suppose.

In truth, the only way to really defeat insane situations like the above is to prevent adverse events from happening in the first place. Exactly how that might be achieved is a interesting engineering question that might be worth thinking about a little. Even if the solution was imperfect, but worked 80% of the time, it might be something worth considering.

If only there was a company out there dedicated to making life easier for operators of Drupal sites... ;-)

Posted on by Allen Varney (not verified).

"The back-ups were running automatically but failed to complete [...] The result was that the temporary files created as part of the back-up were not being cleaned up and were accumulating."

Regardless of PHP timeouts or CPU resource limitations of the host, this sounds like the backup and migrate module was indeed at fault. If the process fails to complete, the module should either delete the temporary files or at least wait for admin intervention before commencing another attempt.

The original report incorrectly implicated Drupal core, but it seems unrealistic to expect the reporter to understand the technicalities of the distinction. At the level of abstraction appropriate to HuffPo and The Consumerist, the story appears substantially accurate: The original poster was stuck with a $6K hosting bill because of problems related to (a module of) Drupal. Branding the story a "rumor" with the implication that we "shouldn't stoop to responding to such allegations" just makes us look finicky and defensive.

Posted on by Bios Element (not verified).

"Regardless of PHP timeouts or CPU resource limitations of the host, this sounds like the backup and migrate module was indeed at fault. If the process fails to complete, the module should either delete the temporary files or at least wait for admin intervention before commencing another attempt."

The script can't really tell if the PHP script timed out. I'm sure there's ways but not simple ones. It's too much to be expected.

The short summery of this entire situation is this. "It's your responsibility to watch your site and your modules. Just because something works for most people, Doesn't mean it will work 100% of the time. Despite peoples best efforts, Bugs still happen."

Posted on by EclipseGc (not verified).

Well, this is similar to your typical "edge case"... it's like image uploads, you don't upload a 100MB image and actually expect it to work inside of imagecache. And if it DOESN'T work you don't go blaming Drupal for its inability to handle images. PHP (like any language) has limitations and once you hit them... sorry. Furthermore those limitations vary based upon your host, so YMMV and you have to evaluate it.

Posted on by Kathleen Murtagh (not verified).

I'm extremely pleased that the Drupal community is organized enough to handle these types of PR issues. When news had first broken out, a few friends and I were concerned about how this huge open source community would be able to quickly come together and present corrections on the technical errors of the issue at hand. I started worrying about clients calling me or deciding to not invest in Drupal any longer.

It turns out we didn't have to worry at all! A plan was decided upon and executed by the end of the day, with the exact causes explained so I have something to tell my clients.

In general, I like the direction Drupal is going, and I'm happy that I picked it as my CMS of choice for my development speciality.

Posted on by Caleb G (not verified).

Kieran, thanks for sticking with this and contacting everyone involved in order to improve the quality of information being published about Drupal.

That said - maybe it is time for a really big drupal_set_message() or something to that affect on all contrib project pages and/or an alert when people download contrib modules, to the effect of "This module is not an official/supported part of Drupal core itself, it's been created and provided by private individuals and has no association with Drupal core. You use these files at your own risk and we recommend you monitor your results and hosting environment to ensure they are functioning properly."

Of course, even that won't guarantee that another incident like this can't occur, but if it helps educate users about the difference between core and contrib even a little bit and/or shows proof of the Drupal community's due diligence...

Posted on by frando (not verified).

Well, you also use Drupal core on your own risk. The message you propose might make believe the contrary, though. While Drupal core is a lot more stable and tested as most contributed modules, there still might be certain obscure server configuration or whatever in which Drupal might fail in unexpected ways. There is no one being responsible for Drupal core. You're not paying for it, after all - it's just provides "as it is".

Posted on by jmelnik (not verified).

I've been following the story from the get go and I keep hearing Drupal this or backup and migrate that-- with very little emphasis on a host that charges $6000 for 100 gb of disk quota overage. Never mind the absurdity of that amount of charges in relation to the cost of disk storage these days, what about a major commercial hosting company having a disk quota, with that kind of penalty, and no method of enforcing it or notifications at certain thresholds prior to exceeding the quota?

And let's not forget that regardless of what tools a host does or does not provide, it's ultimately the responsibility of a webmaster for monitoring his/her account parameters.

Drupal was the least responsible component of this problem yet received the most press coverage. It's a painful reminder that user impressions, regardless of facts, can have a lasting impact.

Posted on by kbahey (not verified).

Kieran

This is an unfortunate combination of:

a) the hosting environment being restrictive (killing processes that takes over a certain amount of resources), and

b) the module not using a unique prefix to its temp files (making them harder to diagnose as to who created them.

The solution in this case, short of changing hosts, is to fortify the module against such hosting situations.

I opened an issue #317887 for backup_migrate to make it use a uniquely identifiable prefix as well as cleanup the temporary files that are older than a certain number of hours.

Posted on by mjesales (not verified).

I had read that story before. its not all that surprising when some hosts have very lenient packages and crazy overage charges. I'm glad that everything worked out in the end.