Blog

Not All Bots Are Created Equal: How to Differentiate and Manage Bot Traffic

Bots can come from various sources. To ensure your site is secure and optimized, you need to know the different types and manage bot traffic.

Oftentimes when you think of bot traffic, you automatically think the worst. Although bots can be dangerous and be the cause of things like credential leaks, data theft and DDoS attacks, not all bot traffic is created equal. Bots come from multitudes of sources, from hackers or competitors to certain authorized parties or even Google and other search engines. Being able to differentiate between bots and use that information to your advantage could give you the edge you need.

How to Tell a Good Bot from a Bad Bot

Being able to tell the difference between bot traffic and regular traffic is not impossible, but is indeed a laborious process. Web engineers can look at network requests and analyze the traffic for certain indicators that point to bots. Things like abnormally high pageviews, higher bandwidth usage, poor website performance and increased server cost are all things to watch for when you expect bots are hitting your site. Once you know the bots are there, the challenge becomes how to easily categorize them. 

As we’ve established, not all bots are detrimental to your site’s operations or your business; some can actually be beneficial. The key questions to ask when differentiating between the two are:

  • Is it coming from a well-known source?
  • Are they transparent about the owner?

Once you’ve determined this, you’ll be able to sort them into several categories within the two types. Good bots include search engine bots, partner/vendor bots, aggregator bots and price comparison bots, while bad bots include web scraping bots, credential stuffing bots, spam bots, ad fraud bots, DDoS bots, credit card fraud bots and gift card fraud bots.

 

Why You Need a Bot Management Strategy

The importance of managing the bot traffic hitting your site goes beyond determining which bots you want to allow through your defenses. It gives you the competitive advantage needed to provide the best digital experiences that your users expect. With high bot traffic and no way to manage them, you end up sacrificing the performance of your site. You also make it that much more difficult to determine how your site is actually doing in terms of pageviews or bounce rates, conduct accurate A/B testing or optimize your conversion rates. But it doesn’t stop there. Without a proactive line of defense, your brand’s reputation could be at risk due to bots programmed to perform fraud or gain access to your users’ accounts. Even worse, a DDoS attack could slow down or crash your site entirely, not only costing you the effort to resurface your site but also the opportunity of people visiting your site during that time. Being able to manage the traffic that hits your site safeguards you from these issues caused by bad bots and affords you advantages from the beneficial bots.

Take Control of your Defense with Acquia Edge Bot Manager

Rather than paying for a team of engineers to constantly monitor your traffic when they could be putting their expertise to more beneficial uses, a bot management tool could provide a superior level of control and security. As an add-on to Acquia Edge Security, Acquia’s Edge Bot Manager is a new offering, available now, that gives you the ability to control the automated bots that interact with your site, whether good or bad. It strengthens your defense and empowers you to:

  • Leverage bot intelligence — access a directory of more than 1,400 known bots in 17 categories that commonly interact with other customers
  • Automate with machine learning — automatically update the characteristics and behaviors used to identify bots
  • Conduct bot-centric reporting and analysis — access real-time and historical reporting on bot traffic to better understand bot trends
  • Detect unknown bots — detect traffic from unknown bots using a variety of techniques, such as user behavior analysis, browser fingerprinting, automated browser detection and more.

With Bot Manager, your visibility into the characteristics of bot traffic increases, enabling you to automatically filter out the traffic you determine to be malicious without having to increase IT overhead to do so manually. Ultimately, reducing the malicious bots that are reaching your site not only protects customer relationships by preventing price and content scraping, but also reduces the load on your web infrastructure so that you’re not paying for unnecessary growth caused by non-human site interactions. 

 

If you’re interested in learning more, please reach out to your Acquia Account Manager to determine if it's the right fit for your organization.

Featured Resources

View More Resources