Home / Taxonomy term

Webinar

How to Scale MySQL in Support of Drupal [October 23, 2012]

Click to see video transcript

Speaker 1: Hi, everyone. Welcome to the Webinar Today. Today’s webinar is How to Scale MySQL in support of Drupal with Amrith Kumar, who is the CTO at ParElastic and Barry Jaspan, who is the Lead Architect here at Acquia.

Barry: Hi, I’m Barry Jaspan. I’m the Lead Architect for Acquia Cloud. We’ve been working to scale Drupal at Host It for several years now. Everyone likes to think that if you put something in the cloud you’d immediately get complete and automatic horizontally scalability of all components, and that would be terrific. It turns out that there are some parts of hosting Drupal that are easily scalable and there are some parts that are not. The database turns out to be one of those parts that has traditionally been very, very difficult. With MySQL, you can use a bigger and bigger server and eventually you kind of hit the wall, you can’t get bigger servers. Lots of people have come up with lots of possible solutions to this. Obviously, there’s database sharding, there’s various noSQL solutions. There are systems like Cassandra, and whatever that are fundamentally different kinds of databases that possibly horizontally scale better. Drupal doesn’t use those. Drupal is really designed to work with a relational database. The hunt test has been on for what is the best way to sort of move the pieces around among what’s possible while maintaining a relational database abstraction while also giving sort of the best combination of it works with existing systems and it gives us horizontal scalability, and there’s a lots of different players in this space.

When I was introduced to ParElastic, it looked to me like a pretty compelling option and so we’ve been working with them some. As a disclaimer: I’m actually on their technical advisory board. I have a very small amount of stock in the company but I’m doing that because it looked interesting to me, it looked like this might be the solutions that may or may not be the best for all possibly cases, but certainly for a system like Drupal, which really wants to just keep using a relational database and have it pretty much look like a normal single database install. It looked like a really good way to step a system up. So with that, I’m going to hand you off to Amrith and he’s going to talk about it.

Amrith: My name is Amrith. I’m the founder and CTO. For the next 40 minutes or so, we’re going to be talking about databases, database scalability, and how that impacts your Drupal site. I’d like to make this as interactive as possible so if you have very short questions, which you want to post, put them in the Q&A tab and we’ll try and get to them if they really topical immediately or at the end of the presentation.

One of the things which—thanks, Barry, for the intro. You all have probably experienced situations where your site becomes popular and it turns out the response time on your site is adversely affected by the performance of your database.

Drupal, as an application, is something which relies on database. We’re going to be talking about several ways to improve the scalability of your site, some solutions include reducing the load of the database, some include scaling the database. I want to talk to you also about the ParElastic solution, which helps improve your Drupal experience on the whole.

Drupal, as an application, is it’s very sophisticated web application and everything that you see on a web page in Drupal is in fact stored in a database somewhere; it’s a MySQL database, normally. When you request a page on Drupal, there’s some portions of it which potential may be coming out of the database. When people interact with your site, some of the things which they do on your site eventually do get stored in a database and that’s the MySQL database under the covers. Depending on how much you load your database, you may have some impact on your response time and a lot a very noble techniques have been used to improve the response of your database.

On the slides we look at some of those things, one of them is caching or CDN. When you have content which can be distributed on the network, then you put it out at various places—something like Akamai—and you make sure that when somebody wants to access that content they get it somewhere near to them. If the CDN doesn’t work, you may be able to use an application accelerator—something like varnish if that doesn’t work, you may use something like memcached, but eventually, the database when you look at the structure and you say, “The reason we’re doing some of these things, the varnish, the memcached above mySQL is largely because you want to reduce the load on the database. You have a trade off here. If you look at a typical Drupal site, an anonymous user tends to get most of their content either out of CDN or Varnish. If a person is an authenticated user, somebody who logs in to your site, the access may, in some cases, come from CDN or varnish; in some cases it may go all the way back to your database.
If you're logged in to a website, you expect personalized content and it’s not good enough if you get cached content, and that’s the reason why to improve the interactivity with your site, you do impact have to go, in some cases, all the way back to the database. Over time, we’ve noticed that the anonymous user is kind of no longer very popular. More and more people want to interact with your site, they expect that they’re going to be receiving content which is customized for them and, therefore, there’s a larger and larger load being placed on your database at all time. You’ll also notice a lot of people have been treating something like Facebook as a de facto single sign-on. They’re signed into Facebook and they expect that the content which they’re going to get is going to be customized for them. They have cookie on their system. They expect that the content is going to be customized to them so you tend to see a larger load on your underlying database and that is something which in a lot of cases can cost an issue for your site.
We talked about several alternatives to reduce the database mode. We talked about CDN. We talked about accelerators. We talked about caching. The shortcoming is many of these work only in cases where the user is anonymous. It’s not a good solution for an authenticated user. Also, it’s very, very good for cases where you're reading or you're just rendering a website, not very good if people are actually interacting. If they’re posting a comment, you still need to go all the way back to the database. At the end of the day, scaling your database is really the thing which you need to do.

There are several alternatives to scaling your database and we have a couple listed here on the slide:
Going from left to right, one option is get a bigger and bigger server. MySQL find a database and if you have really, really powerful hardware, MySQL can perform very well. One of the downsides with large hardware, especially if you happen to be running in the cloud is, your hardware is exponentially more expensive but the performance is exponentially more expensive. In other words, you can pay a lot more for server but you may not get proportionally more in terms of performance.

The other thing, as Barry mentioned earlier, there is final limit on how big of a server you can get. The largest server you can get on Amazon, for example, is sometimes not big enough for your site or for your database. What do you do then? There is an issue with just the scale up approach, if you will. Scale up is scalability solution where you just go for larger and larger hardware. The issue with scale up is there is cost and there is a limitation, you can’t go beyond a certain limit.

Some other people have come up with a solution called “sharding.” In the middle column in the slide here, you see what “sharding” looks like. Sharding, for those of you who are not familiar with it, is an application architecture where the application distributes its data into shards. Shards are independent databases and the application takes on the responsibility of saying, “if I want to get this data, which database do I need to go to?” The picture you see there looks awfully complicated. It’s not just to scare you, that is really what sharding looks like. The thing at the very top is your Drupal application and then there’s this whole infrastructure between your application and collection of databases which is, among other things, complex, fragile, very costly for you to write. Also, if you go down this route, it’s not something which is standard. What you're really going after the scaling the database, what you're ending up is a fragile solution, which may not in fact scale your database.

Also, as Barry suggested, there are some people who have attempted using solutions like Cassandra or other noSQL object data, noSQL databases. These require wholesale rewrites to Drupal. These are totally non-standard and in many cases, they’re totally unproven and risky. One of the downsides with this approach is—and I would be the first to admit that you can optimize a single code path with noSQL database and make it much, much faster than any relational database ever can. Any other code paths through the application becomes ridiculously slow.

I’ll give you just one example in the case of somebody who I spoke with, who tried to scale Drupal using a noSQL database. They were able to get very good response time on typical page loads. They were able to get modestly good time on comments. But, they had to totally disable the tag cloud because the tag cloud absolutely brought the server to its needs. That’s the fundamental problem with noSQL database is you optimize a primary code path but other code paths become totally sub-optimal. That’s also one of the good things with the relational database, which is you have a normalized data structure under the covers, which is the way in which Drupal stores data in the database and you can ask general queries in a standard query language and you're able to get good response time from most queries.

The implications to all of these three options, if you chose to go down this path, sharding and noSQL, for example, are totally custom-code. You don’t have standard module support for it. In the case of just getting bigger servers, you're limited by the size of the server and some cases that may be big enough, but in many cases, that is proved to be not big enough. There are well-known situations or well-documented situations where even largest servers available to you is not big enough if you're site really becomes popular. When that situation happens, you really need to look for another client solution.

Another kind of solution, which some people have talked about is to have database replication with masters and slaves. On the slide, you see a client on the very top talking to Drupal; and Drupal is sending all of its reads and all of its writes primarily to MySQL Master, which had been replicating to some collection of slaves; and then Drupal is directing some reads to one of the slaves. The solution works okay in some cases, but now Drupal has to realize when it’s doing the reads, does it know whether the writes has been replicated to the slave or not? You may not get the latest content on all the reads if that’s sufficient for your application. Again, in some cases this is acceptable; in many cases, this is not acceptable and therefore we need to look for other alternatives.
Just talking about the various alternatives you have for scalability that we talked about so far. You have replication, the one we talked about most recently, reads are targeted at slaves; writes are targeted at master. Standard MySQL replication between the master and slave and there’s challenges here because you don’t know in all cases whether you're getting the latest content.

There’s the option of sharding, which is an application architecture. It’s something where you have to make wholesale modifications to Drupal. And once you go the write or sharding, it’s not possible in all cases that you're going to be able to get a consistent and a holistic view of your data.

There’s the option of scaling up to a larger hardware that comes with the challenge that it’s ridiculously expensive in many cases and sometimes you can’t scale up large enough.

Finally, there’s the option of a noSQL or newSQL database where you have totally modified Drupal the application. It’s no longer a standard Drupal application, you’re on your own as far as maintaining it. To summarize these options: a combination of high operating costs—large server is very, very costly to operate; high maintenance costs—once you go and modified a code for Drupal all the way, you're on the hook to maintain every piece of it; it’s not standard—noSQL has no standard query language, it’s not like SQL which has a standard query language.; you need to understand the specific API’s for the noSQL implementation you choose to use; you dramatically increase your risk, not all of these solutions have had the same industry rigor, if you will, that MySQL has had. MySQL has a database that’s been around for several decades and that they’re very, very stable database for the most part.

All of these things have cost people who have gone down approaches to severely impact their time to market on their sites. Really what people are looking for is a solution which just works with Drupal or other web application, makes a database scale easily in the clouds.

Let me talk about ParElastic because that’s exactly what we do. Relational databases like mySQL are the thing which Drupal relies on. Drupal is coded to work with MySQL. The challenge is that MySQL is not always perfectly scalable in the cloud. What ParElastic does is, it takes a group of database servers and makes them work together as if there is single database server.

Let me say that again. If you take a group of servers and you distribute the load horizontally across them and all of these servers appear as if they are a single server. ParElastic is built for people who build interactive web apps, digital gaming, ecommerce, social networks… all of the kinds of things which people are using software like Drupal for. If you are using Drupal multi-site or you're building a multi-tenant and SaaS application, ParElastic is ideal for you. I will talk some of the reasons why that’s the case.

Finally, if you are wishing to offer database as a service, if you are wishing to offer some hosted application as a service, ParElastic, because it virtualizes the database, is ideally suited for you. The reason you need this is, database scalability is something which we all recognize as a serious impediment to the way in which you innovate and grow your own application and we want to mitigate many of the risks with other solutions. The risks with other solutions are: the cost; the complexity; and the lack of standardization and you want to get the overall of those things.

Let’s talk about how ParElastic does it. ParElastic is a solution, which allows you to scale your database on demand. Depending on the application load, depending on the Drupal load which replacing on your database, you can add storage, you can add process, and most importantly, because you can add and remove this on the fly, you only pay of what you use. Also, very important, we are not creating a brand new database; we’re just taking existing MySQL. If you’re running in the Amazon cloud, this could be either EC2 instance with mySQL or this could be an RDS instance or a collection of RDS instance. We pick collections of database instances and make them work together as if there is single database instance. This dramatically reduces your risk, reduces your disruption. It’s a database you know and love so all of the existing knowledge and skill about databases just continues. Since it mixes group of MySQL databases, if you're like a single MySQL database, the Drupal application just perceives it as a single MySQL database and it just works. That’s the simplicity of Drupal.

What Drupal application sees on the left hand side of the slide, a single MySQL database. Under the covers, it’s actually a collection of MySQL databases in which we’re all working together as if there is single database and that’s what ParElastic does. We virtualized the database for you so you just interact with what you think of the single database and we take your scalability behind the scenes.
How does all this work?

Let’s start with a couple of slides, which talk about your current configuration, you have Drupal, it talks PDO to a standard MySQL. This is what you have today. With ParElastic, make no change to Drupal, make no change to your PDO and you talk to ParElastic. Absolutely as simple as that.

Behind the scenes, there’s a collection of MySQL servers, these again, if you're running in the cloud, they could be in Amazon’s cloud, they could be EC2 instances with mySQL or they could be RDS instances. What ParElastic does is it distributes your data transparently partitions them across a collection of databases. All of these databases are now going to work together as if they are one single database. The application doesn’t see the difference that there’s multiple.

A very important aspect of ParElastic, something which is part of our intellectual property in the patent we have, is depending on work load on your system and depending on the amount of data you have on your system, you could add additional MySQL servers. Contrast this with something like sharding. If you have some number of shards in your application and you decide you want to have more shards, there is this very time consuming and risky operation called resharding which you have to do where you literally redistribute all of your data on the new set of servers. Part of our intellectual property, for which we have a patent, is ability for you to distribute data across a collection of servers and such that when you add new servers, you don’t need to do any redistribution.
That’s, I think, a very important aspect of our product, which makes it particularly applicable for a Drupal site as the site grows. Now overtime a Drupal site accumulates more and more data, you have more and more content which is published on your site, you have more comments, you have more data in your MySQL database, you can’t start off your system with the number of database servers you expect you're going to need in some number of years. You start with the number of servers you think you're going to need in the next six months, and overtime you grow to whatever you need. The important thing is, you can add additional servers and you don’t need to redistribute data–very important if you want a truly elastic architecture.

The other part of our system, also protected by another patent, is the ability for us to distribute processing on a collection of servers shown here on the left of your screen (in orange). When you have variable, when you have workloads which require processing, ParElastic has the ability to spin up additional databases, this is again off the shelf MySQL database instances where we can do this processing, there is no persistent data, there is no application involved, but you just use it in order to profound processing on the fly and that’s part of how ParElastic is elastic and able to deal with highly variable workloads which your application might face at any point in time.

Now, when you look at something like Drupal multi-site, you have multiple websites, multiple web properties, and in a traditional Drupal multi-site situation, each site has its own database, that database is sitting on a server and you might have some number of sites all sharing the single server. With ParElastic, it’s a little bit different, each site has wooded beliefs is a virtual database of its own and that virtual database is distributed on some ParElastic infrastructure, which is some combination of storage and processing site, which can grown and shrink based on the demand you place on your system at any point and time. With Drupal multi-site, we have some special optimizations, which are particularly good if you are an enterprise, you are using that you have multiple web properties on Drupal.

Finally, nobody wants to have an outage on their site. Those of you who are using Amazon probably knows that there were some outages in the last couple of days, high availability is really, really important for today’s website. ParElastic, as an application, is something which is highly available. You can have multiple instances of ParElastic. You can have this in multiple availability zones. You can distribute your load across all of them with some standard network load balancer.

Also, by the fact that we don’t make any changes to the underlying database, it’s standard MySQL under the covers, you can have MySQL replication going on your storage site as well. So you can have a primary copy in one of the availability zone, a set of mirror and another availability zone. You can have multiple ParElastic instances talking to these and therefore, if you at any point face a failure in your underlying infrastructure, your site is not compromised.

Again, the Drupal application running up top needs to know very, very little about all of these because with ParElastic, the entire database is virtualized and we deal with all of these for you.

Just to quickly summarize what are the details. We adapted the provision resources based on your demand. You don’t have to provision for peaks, you don’t have to provision storage for your demand two years down the road, you provision for what you need today. ParElastic automatically distributes data to all of these servers and we scale both reads and writes. Now, since we’ve distributed the data across the collection of servers, we can scale reads and writes for that. If you’re using Drupal multi-site or you're building a SaaS application, the ability to virtualize your database is really, really important. Finally, ParElastic makes all of your data no matter which server it happens to reside on. It makes it appear like one single database so you can think of the similar to cross-shard operations in sharded architecture.

In a typical sharded architecture, you can query of one shard or another shard and then it’s up to the application to consolidate those results. You don’t have any such thing with ParElastic. You just code your query, assuming you're talking to a single database, it’s up to ParElastic to provide you the single answer to that.

The important benefit to you, without ParElastic, you have to end up provisioning for peak demand. In many cases, that means that you're provisioning a lot of excess capacity. With ParElastic, you provision only for what you need and you dynamically change your provisioning based on the load on your system. This is really an impact on your bottom line.

Finally, unlike all of the other solutions which require you to make changes to your application, whether it’s in sharding or noSQL, with ParElastic, there is almost no change that you need to make your application. It’s a one-time change to get your data into ParElastic and then you application just works. Without ParElastic, you spend a lot of time building your underlying infrastructure. With ParElastic, you don’t have those issues, you just deal with your application and you innovate new application, which is really what you want to do anyway.

It’s always important, when we’re talking about things like this to discuss the performance of the system. So here are some results of standard benchmark. The red line on this benchmark is the performance of the system using standard mySQL (native), which was run on a m1.xlarge system on Amazon’s cloud. It had about a terabyte of data. I think it was about two billion loads worth of data and some of the large tables. The exact same data was loaded on 5x m1.xlarge systems and ParElastic was used against the 5x m1.xlarge system and both cases we used standard MySQL 5.5. The graph shows the response time for a standard database transaction with mySQL in red and with ParElastic in green. The Y axis in the response time in milliseconds so more is worse. You don’t want to be waiting three and a half seconds to get response to see your queries. With ParElastic, you wait barely half a second, 587 millisecond. The performance is about six times better. We have five servers. We pay less than twice as much as a single m1.xlarge so you have a dramatic improvement in performance here.

As the load on your system goes up, you don’t have this rapid decaying performance that you see with traditional MySQL, you have a gradual increase in response time but it’s something which is definitely manageable. What’s not shown on this graph is what happens with very small numbers of users. ParElastic as a product since between your application and the collection of databases, so it introduces a small overhead. In very small numbers of users, like if you have a single user or if you have queries which are very, very short, the overhead may actually dominate. So that’s a quick note on performance, but the important thing to note is you're distributing the load here on 5 m1.xlarge servers. You can equally well distribute it on much larger servers as well.

The databases, which we’re using in ParElastic are elastically scalable and depending on the load on your system, you could spin up more servers on demand. The mechanism for doing that is policy-based and you could monitor the load on your application and you could determine how many more servers you want to spin up or whether you want to spin down servers. From ParElastic’s point of view, when we process query, we’re going to look at what resources we need for those queries and we’re going to provision those resources and use them from within a pool that is provisioned by whoever is running the application. if you were running ParElastic on Amazon for example, you would be provisioning a set of servers and you would inform ParElastic of it and in the event that you're application requires more servers that you determine that to be the case, you would spin up more servers, you would inform ParElastic that they’re available for use and then we would continue to use them. But on the fly, we could provide you with the intelligence which says you actually do need some more servers. The numbers that were shown here are with Amazon EC2 with MySQL, the numbers with RDS are similar. The primary difference between Amazon EC2 and MySQL on the one hand and RDS on the other is— but with RDS, you get some of the managed options which Amazon provides you; with MySQL and EC2, you have to do that yourself. Performance on both cases are similar. In the numbers that we’ve shown here, we used standard IOPS or standard EBS volumes, even with provisioned IOPS you don’t notice a dramatic change in the performance here.

What are some of the benefits to somebody who uses ParElastic in application like Drupal?

The most important one is there are absolutely no lines of group of codes that need to be change. There is a one-time activity which you need to go through, which says, “Have do you migrate on to ParElastic?” “How do you make sure that all modules that you want to use are creating their tables and distributing their database correctly?” But once you do that and beyond that there is absolutely no lines of group of change and installing Drupal is literally the standard Drupal installer. You go back to focusing on the innovation in Drupal rather than trying to deal with how do you innovate in the data tier? Your user experience is dramatically better if you look at this performance chart again. Your user experience is not going to be very good if your response time for queries is dramatically going up to three and a half seconds rather if it’s a half a second, you're in much better territory. Your operation cost is so much lower because you provision and use only the resources that you need at any time. By virtue of the fact that we’re not a totally new database, we’re just standard MySQL under the covers, your risk is dramatically reduces. Since we deal with all of the issues of scalability for you, your time to market dramatically improves as well.

I’ve also attached here with this presentation some contact information for you and if there are any questions, now is your time to start putting them in.

Barry: There are two so far. One, maybe you covered already, which is what happened to a single node fails and you talked about how you used MySQL on occasion but maybe not about using ParElastic to do replication itself or to do duplication. The related question is ParElastic itself, how is ParElastic a single node of failure?

Amrith: OK. Thanks, Barry. Both these questions and I’ll try to get to them one at a time and it’s like don’t get both smack me. What happened to the single node of failure as they’ve built in redundancy? I switch back to the feature on high availability. A single node which fails could be one of couple of things. It could be a node running ParElastic and those are the notes on top. And those are not single nodes of failure because if one node fails, the query can be redirected to another node. Under the covers, a single node of failure could be a MySQL node. One of the storage nodes on the right hand lower part of the picture. Again, standard MySQL replication is used here so if a single node fails, that node has a copy somewhere else and at that point, it’s a matter of recognizing that node has failed and performing a fail of operation and then you direct all of your queries to that alternate node. Replication in this particular case is standard MySQL replication and we’re not at this point in the replication process itself and there are some things which we’re thinking about in that area which would make for product enhancement in the future. The second question was about…

Barry: ParElastic being single point of failure?

Amrith: The ParElastic servers at top were another single node of failure. It’s a very, very good question, which goes to the real heart of why ParElastic as product is innovative and cool. If you're building a traditional website, and you have a component in that site which is stateless. It’s very, very easy for you to horizontally scale that. For example, if I just need to spin up more web servers, I can spin them up because the web server fundamentally does not have state beyond the state of the current transaction which its process. you could have a load balancer and you can spin up and spin down web nodes on the fly. The issue becomes, how do you spin up and spin down databases because they are stateful and single ParElastic is the—the ParElastic engines itself is largely stateless. So we can have a large number of ParElastic engines as well and you can dynamically send traffic to one of many ParElastic instances and all the ParElastic instances are talking to the same underlying storage infrastructure so we’re not a single point of failure either. I hope that answers the questions. If not, just—

Barry: One way I can address this is that we have not deployed ParElastic if we have not been—we are working with it and experiencing with it. All of our masters cloud database clusters are high availability. We run two masters in an active passage configurations. All of the reads and writes are always going to one of the masters, but if that one fails, then all the reads and writes are going to the other master. One is always active; one is passive. What we are planning to do, I believe what we’re doing, I’m not directly involved in this, is we’re running the ParElastic engine on both of the database masters. So whichever one you're talking to, you're actually talking to ParElastic and then ParElastic may be talking to both of the database masters or maybe a collection of other storage nodes or whatever else you might be doing.

Effectively, ParElastic doesn’t really change the story of whether or not you have a single point of failure. If you’re running on one MySQL server, you have a single point of failure. If you're doing what we’re doing, and you have two, and your heartbeat or some other method of doing failure between your two database masters and you have active/passive replication set up, then you don’t have a single point of failure and ParElastic will not add a single point of failure. What ParElastic won’t do based on what Amir has explained so far is, it doesn’t magically take one database server and turn it into not single point of failure anymore. If you only got one underlying server, that’s your situation. Of course, you wouldn’t use ParElastic if one server was sufficient. That was sort of wasting your overhead under most circumstances. It’s actually the multi-tenant feature, it could actually be used anyway, but… all right.

Amrith: Thanks, Barry. Something you didn’t include, I don’t know, it’s probably too late to put in here is what Doug called the “marching app slides” where you showed how different kinds of transactions, different kinds of joins and queries played out across this kind of environment.

Barry: I did not have those slides on this presentation but I can certainly go over that with anybody who’s interested in that so I don’t know if we can do that here.

Amrith: OK. You might be able to—

Barry: Some of the questions actually asked for some of that, so I’m going to into that.

Amrith: I thing I’ll say is if you have time to put it into the 1:00 PM webinar, people could watch that they’re both going be online.

Barry: We’ll do that, OK.

Amrith: There’s a couple more questions here talking about where we store the data, what data is stored where, and so on and so forth. Let me talk about some of that. Where the data is physically stored? We’re not offering a service. What we’re offering is software which you run in your Amazon account, for example, or in your own data center, for example, we have people who are doing both things. Some who run it on the Amazon’s cloud; some were running from their own data center. As a matter of fact, we have some people who are running our software in other clouds as well. The data is physically stored in wherever you choose it to be. If you happen to be a person who was in the UK, who is using Amazon’s UK availability zone, that’s where your data is going to be. By the way, ParElastic is not a Drupal-only solution, it’s a generic database solution so if you have some other database app which requires, for example, the IO security on beta policy which says you can’t have identifying information If the person in the IO country moving out of the IO you're still fake there.

There were a couple of questions about what happens with failures and single point of failure and I think Barry addressed them. I’m going to talk about one of the questions, which is what data is stored in a persistent instance and what is stored in a dynamic instance? One of the things which any database does when it processes a query—let’s talk about simple MySQL, no ParElastic involved and you're attempting to join two tables then perform an aggregation and then sort the results and produce the results of the query. The query plan, which MySQL produces often generate these things calls temp tables.

What ParElastic logically does is it treats the processing nodes as the place where you store temp tables. Your persistent data, every piece of content which was in your website, is stored in the persistent storage part of ParElastic. When we’re processing queries, if we require— and this is important thing, if we require and not in all cases—if we require to join and some data happens to be on online MySQL server and some data happens to be on other MySQL server, this is what some people call “cross-shard” operations. Then and only then, we will use the process in sites, we will get the data from different places and perform the operation on the processing site. This is particularly useful because MySQL as a database is really, really fast if you are just using it for high performance index access to your data. MySQL performance gets a little worse when you start to do things like joins. It gets even worse when start to do things like aggregations and when you start to do something like sorts. What ParElastic does is it says, those operations can be done equally well on the processing those and the processing notes are truly a dynamic elastic resource. If you require that kind of processing, don’t bug down your storage servers for that, do it somewhere else.

There was a question about how you pay for ParElastic. ParElastic is something which we sell in license based on the number of servers on your management, the number of storage, persistent storage under management. Feel free to ping me, we can talk more about prices and things like that, but for the most part, I’d like to keep this to be a more technical presentation and stay out of marketing and sales.

Another thing which several people have asked in the past is how ParElastic compares, for example, with MySQL cluster? MySQL cluster is for the most part, the cluster option available would be NDB storage engine. Performance is really, really good if you can keep all of your data in memory. Performance drops out rapidly if you're data exceeds the capacity of memory. The place where MySQL cluster is particularly applicable is if you really high-end hardware, you're running it on your own data center and performance in the small number of milliseconds or reasonable number of micro second is really important to you.

That’s not really the use case if you're running a database in the cloud where you can’t guarantee, for example, all of your data is going to be in memory and as a matter of fact running MySQL cluster in Amazon produces some very, very peculiar performance results. There is a dramatic improvement that you can get with scale out, which you really can’t get the cost effective e way to scale up and that’s kind of the big picture difference between MySQL cluster and ParElastic. An entirely different level, ParElastic one with standard MySQL, therefore you don’t have to change the storage, ParElastic is not a storage engine. We sit between the app and the collection of MySQL servers. If you happen to be using NDB for some data, you continue to use an NDB for that data. If you want to use some of other high performance storage engines, you can go head and use that. Unlike the MySQL cluster option where you pretty much bound to use the NDB storage engine and you could be one of the brave pioneers doing that and helping them find the bugs out in that. But it’s not something which performs very well in the cloud. Other questions?

There are a couple of questions which I’m seeing about screenshots and administration consoles, that goes back to what Barry just mentioned. A prior arrangement of this presentation had a lot of that content in the interest of time I remove some of that. What I will do is put some of that content back. There is going to be a repeat of this webinar at 1:00 in this afternoon so the version of the slides which will be available online will include that information. Check back, I think it’s set in 48 hours that information will be on the slide or I have another option for you which is, my email address and phone number are right here. Drop me a note. Happy to talk with you about these and a lot more details, a couple of people who have asked about price list and things like that. Michael Aubin, whose phone number is here as well, he is the guy you should speak with about that. Other questions?

Speaker 1: All right. Thank you for the great presentation. Thank you everyone for attending. Again, slides and recording of webinar will be posted on the Acquia website and we’ll be emailed out to you. Have a great day!

Cloud Hosting for Government Agencies: Drupal Platform as a Service [October 18, 2012]

Eliminate Time Wasted Investigating Code and Configuration with Acquia Insight [October 16, 2012]

Making the Business Case for Open Source Drupal [October 10, 2012]

Click to see video transcript

Hannah: HI, everyone. Today’s webinar is Making the Business Case For Open Source Drupal with Kieran Lal, the technical director of enterprise sales here at Acquia. Kieran was employee number two of Acquia and recently celebrated his five-year anniversary. Congrats, Kieran. He has served on the Drupal Association board of directors where he focused on business development. We’re really happy to have him presenting today.

Kieran: Great. As Candace said, I’m Kieran Lal. I’m going to talk a little bit about the making the business case for Drupal and OP source. I think that the first thing that we’ve seen is just a huge amount of adoption worldwide with what we estimate to be around million sites or more than a million sites. More importantly, where Drupal’s really accelerating in its adoption which is with large organizations dealing with the more complex websites, and so we see lots of big media organizations that are standardizing and building Drupal as one of their key platforms, a lot of big media brands, universities, particularly in the United States, the adoption of Drupal and universities are sort of becoming the de facto standard. We’re now starting to see three, actually it’s moved from two to three of the global large music companies, so Sony, Warner Music, and Universal, now all starting to adopt Drupal as their platform of choice for their media brands and then about 130 nations using Drupal for government websites and now there’re number of people from the government, different government agencies here, so I’d encourage you to check out our wiki page of several thousand government websites, national websites, and regional websites. We’re also starting to see a significant adoption of Drupal in the global pharmaceuticals. As people are starting to build platforms and brands around their pharmaceuticals and they want to build a repeat features and functionality of their site quickly they’re starting to use Drupal.

One of the reasons that I mention this is because it’s good to understand why these organizations who have adopted Drupal has paved the pathway for you as you’re thinking about building the business case for Drupal. I actually come from a bit of an unusual background within the Drupal project. I started off as a programmer myself and ran some technology labs. Then, when I came over to the Drupal community, I found that there were tons and tons of programmers here but there were very few people who could really articulate the value in the business of the business advantages of Drupal. I sort of switched from being a classic sandal wearing, ripped jeans kind of guy to being the suit and was frequently known as one of the few guys, if not the only guy, in the Drupal community who wore a suit and wore it all the time. This actually became a bit of problem because some people were like, “The suits are here. This is terrible.” One of the things I did was when we had our Drupal cons to make people feel like the Drupal platform wasn’t being taken over by the suits. I had decided to change, to dress up a little bit and help out. Here’s a picture of me at Drupal Con London last year dressed up as a beefeater and then more recently at Drupal Con Denver dressed up as a cowboy going on the themes. Part of this was to really bridge that gap between how do we articulate the value of open source, grass roots programming model all the way up into the corporate office, the suit, having a little bit of fun with that motif here.

What I want to do is talk today a little bit about the business case for Drupal and really take you through four different things that you need to be considering. The costs and cost reductions. This is the one where a lot of people focus up front but when they actually get into it and you get further along in the decision making process the cost becomes less significant but they’re still important. Enabling of value in effectiveness of business processes, I’ll really walk through some concrete examples of organizations who use Drupal to help with their business processes. Flexibility, a key decision. A lot of people are coming off in existing CMS today or sets of HTML pages and they find themselves extremely limited by what they can do, and so they’re not abatable. Basically, their CMS did not adapt into the business. The last, and most importantly I think, is risk. The organizations, extreme large organizations are really first and foremost mitigating the risks that they’re willing to take in a CMS platform decision, and so being able to address that case is absolutely critical.

What I’m going to do now is I’m going to work through a little bit of the total costs of ownership angle. I think this is important that people have this down and really understand this, but I also want to put that caveat that it’s only part of the equation. We’re getting some feedback, at least from one person, that they’re having trouble seeing the slides. If you are having trouble seeing the slides, feel free to drop into the Q&A box and let Hannah know and we’ll make sure that the right slides are coming through. When you go to do your cost model there’re a few things that you need to take into consideration. Number one your license fees. If you’re buying a proprietary CMS there’s going to be a clear license fee. There’s also typically a maintenance and support fee that goes along with a proprietary product. In Drupal, the license fee for Drupal no matter how many instances you have is zero. It’s a free license. It’s an open source license under the GPL version 2+ or higher. However, there are realistic maintenance and support costs that go on with that and they’re comparable to what you can see with a CMS and you should be planning to have a budget for that. There’s an implementation cost often where the bulk of the capital costs that people are spending get put in, and so that’s often hiring or developing an internal web development team and having that team focus on the project anywhere from days to months to years.

Training costs are important. Often organizations have existing technical talent that are sophisticated web talent or sophisticated CMS talent, and it’s merely a matter of taking them through a training program so retool their skills onto this new web platform. Then, there’s also a content editor training challenge. Many organizations have dozens if not hundreds of content editors, some cases there are people using Drupal who have literally thousands and tens of thousands of content editors on their Drupal sites. There’s a training program that has to go on to help people, content editors, to be effective with the CMS. You can’t just put them in front of a CMS interface and expect them to succeed, particularly if the business demands some complex work flows where there’s a specific way of the business to CMS. They’re going to have to learn how to do those beyond just what-you-see-is-what-you-get editing.

There’s typically ongoing development. One of the things that we see with a lot of organizations is that once Drupal is deployed it deploys pretty quickly compared to other CMS solutions that they’ve had in the past. They really get excited about the rate of change that they can have with Drupal. The business starts asking for more and more features and what that means is that there’s typically an ongoing development cost that keeps on going, and then, of course, there’s an administration cost. From time to time, you have to think about what does it mean to administer my costs either managing content from external community and managing content internally but also administering the site in terms of applying security updates, patch and feature updates in the modules, taking advantage of new features that are coming out and refactoring parts of your sites, so it’s more maintainable, more flexible going forward. Those are the six buckets of costs that you need to think about. I won’t go into too much detail about that. I’ve gone about as far as I want to go in terms of detailed costs.

I do want to point you to a paper called the TCO for Open Source Social Publishing. There’s a bit.ly link on the top corner of this page that you can go in and get to it or you could search on that title in Google and it’ll show up. This is a fairly detailed whitepaper, eight to 10 pages long. It really goes into what are the costs typical for a proprietary CMS and the prices that you’ll pay for that versus what would the cost be for using open source Drupal. I think in a lot of cases you’ll see the costs are shift in some cases and in other cases the costs are really significantly lower in comparison, and particularly your flexibility and certain features are important to you. Also, a lot of people think about not of just having the CMS but does it actually take to operate the CMS to keep it up and running, keep it supported, keep it scalable, keep it flexible, make it adapt to you business. That’s a separate cost consideration, and we have a nice whitepaper that I worked on called Get a Grip on Your Hosting Costs for High Volume Website and where you can really go into detail, think about all the different costs that are involved, what does it take to hire talent and staff, and what do you pay for hardware, and what does it really take to turn and optimize and get great performance from folks. That’s a great whitepaper that really gets into the costs between cost of the CMS and the cost of the hosting.

Then, if you’re a slightly more independent view of the world these last two whitepapers are from Acquia. You can go and take a look at this paper from the London School of Economics. Again, you can get a copy off Acquia, but it was produced independently and we just happened to have distribution capabilities for it and it was produced by the United Kingdom Cabinet office on open source. This gives you a nice overview of factors to consider, cultural factors, cost factors, internal tooling, and skills factors when using open source and what are the advantages. This is a great paper to help build your business case.

What does this all really mean? I think at the end of the day you probably are thinking about you have to go build this business case and you have to take it to your CEO. What your CEO is really considering is not just the business case itself but what they’re really thinking about in terms of what are my capital costs and what are my operating costs, so how much initial outlay do I have to put out to get the site up and running and built, and then once it’s built and running what are the operating costs. That’s really what your CEO is asking you, not these esoteric important considerations but more of an academic or an analyst view of what the costs are. Your CEO is looking for clearer pragmatic dollar consideration. You have to take that.

What that means is that often when they’re thinking about those sort of immediate costs we’re really dealing with perceived costs, so software costs, free to use, implementation costs. We often see implementation costs around Drupal being 30% less than proprietary often because there’s not a big talent pool for the proprietary CMSs and there’s very large pools for open source because it’s so accessible, and also because a lot of proprietary CMSs are built off more classic enterprise languages which have a smaller pool and more expensive pool of talent. That could be an issue. Whereas, languages like PHP are more accessible, so there’re more people around the world that use them and because people are more likely to take a career path of setting up a hobby site or a site for their friend and really like using Drupal, start doing some consulting, then graduate into a full-time position. The size and the talent pool is very large around Drupal and that helps to keep costs down a little. Drupal’s very popular, and as a result there’s a huge percent of competition for talent in the market today. Then, maintenance and support, what we see is in our calculations often enterprise commercial support at certain levels is 20% of proprietary costs and where we see platform as a service which is a new trend in the industry around what does it take to have a dedicated application hosting environment called platform is a service. We’ll often see that many people in the community have been in the Drupal community for a long time have built up these custom hosting platforms internally. When they actually factor what it takes to maintain them, keep them up to date, keep all the tools, keep the staff and do all that, that they see as much as 40% savings when they switch over to having someone help as opposed to building the whole stack themselves.

From savings standpoint, one of the things that’s really key is that often when people do a CMS option they’ll find that that CMS is targeted at a specific niche. This CMS is meant to do the corporate CMS or the CMS is meant to be a microsite CMS or the CMS is meant to be a SAAS CMS. One of the things great things about Drupal is Drupal does all of those things really well and so it lends itself to consolidation. By consolidating or reducing the total number CMSs you can reduce the total number of technical teams that you have to build on the CMS and you can reduce the total footprint of operational costs, and so that consolidation is critical in terms of saving money. Many organizations, we’ll talk a little bit more, but a lot of universities or governments are seeing huge cost savings by consolidating onto a single open source CMS that’s really flexible, can meet the full range of needs in an organization. When it comes to vendor selection, if you buy a proprietary CMS often you’re tied to just one single vendor. Because it’s an open source license you can come to a vendor which is a vendor that targets corporations and enterprise primarily, but by no means are you down to just using Acquia. There’re huge, thousands of vendors out there who are providing different levels of implementation services around Drupal, or you can go and hire talent directly into your organization so you get a lot of freedom. Within that freedom, if your budget isn’t a large budget project then you might want to go with a small vendor that’s used to building small or lower budget projects and you’re also not tied to what could be a very small pool of national very large agencies that know how to do these large complex implementations. You can actually go and find a local vendor in your community who knows Drupal and can help you or use open source and that can help.

Last but no least, Drupal is also because it’s grown up as a grass roots worldwide platform from it’s very inception has always been focused on internationalization because it was started by Belgians because it was a team of people who spoke Dutch and English and Flemish and New French, and then had early developers who were eastern European languages or other European languages, and so Drupal had a very strong internationalization presence. A lot of businesses that are thinking about expanding internationally or supporting a customer base that is multilingual want to be able to have a Drupal site that can serve those many markets, so we see a common model where somebody stands up a Drupal site and as they go global and start to support more and more languages they’re quickly able to reuse that CMS to support those multiple languages. There’s significant cost savings from that.

I talked a little bit about Drupal as a platform standard. This is a great slide that really shows how Drupal can serve a wide variety of sites. We see community sites and product sites and corporate sites and marketing microsites. Because all these sites are available you just use Drupal as your standard platform and then build a wide variety of sites as opposed to having to go out and say we’re going to use this blogging service for this service, and then for this model we’re going to use as SAAS service, and over here we’re going to use Tumblr, and over here we’re going to use a big enterprise service, and then for this product campaign we’re going to buy an enterprise SAAS, and so you don’t end up with all these different costs and different teams. You just have one platform that works for everybody.

That’s a review of our costs starting off with initially looking at all the different things that are going to be initial costs, a list of six different costs that you have to address, but then really recognizing once you get through those and looking at some whitepapers that you’re going to have to have really take this up to your CEO or take it up to a stakeholders approved budget, and so they’re really focused on operating and capital costs. Then, thinking a little more a long term about what are the potential savings from that. All of those things may be factored into costs and cost reductions. This first part of our presentation is really been more of the classic academic view of sitting down and taking out a pencil and paper and really trying to figure out what it’s going to cost for a CMS.

Now we’re going to start getting into some of the things where a business really thinks about what is the CMS really do for my business and how does it actually make my business work better? We’re going to talk about the effectiveness of business processes. You’ll have to think about things like how do I update my website, how do I do content websites, what different services can I offer. Many organizations they’re in a situation where today they have a call center and they get a certain volume of calls into that call center and there are possibilities that they could put a nice little application that answers people’s questions like an FAQ or a little chat box or a forum or some kind of group where many of those questions could be answered quickly, and more easily, and convenient for customers rather than having everybody call in. Those kinds of additional customer services can be really effective because Drupal doesn’t just manage content, it’s also a full web application framework so you can build custom applications to service the needs of your business.

Because Drupal is open source and it’s freely available it has had a huge amount of contributions to it by thousands of developers around the world, and so it has a tremendous amount of features in it. it also has a very large pool of contributed modules, almost 19,000 contributed modules. That doesn’t mean that those contributed modules always perfectly or in combination always perfectly give your business exactly what you need, but it can give you a very significant leg up where you’re trying to do a common application that’s a well known architecture pattern for the web. By using this series of site building modules you can save a huge amount of custom development and have a lot of flexibility for the future if you need to change it because these are modules that are well architected and reusable, and so you get a tremendous amount of labor savings for your developers where they can click to do things, put together a site, in a very quick fashion.

You also can get some productivity savings. If people have traditionally been paying a webmaster to go edit HTML and now all of a sudden somebody’s in a content editing role on the marketing team or somebody who’s not very technical is able to push content live to the website which was previously very limited to technical world. There can be really significant savings there. Also, because the website is administerable through a user interface they don’t have to get down to the command line. They don’t have to be a very technical person to manage the website and so that can really help.

Then, one of the other things that we see certainly at Acquia, we spend a significant amount of our budget on outbound marketing and inbound marketing, and so some of the things that we do are search engine marketing. We’ll put ads so that if you go to Google.com and you type in enterprise Drupal you’re likely to see an ad from Acquia at some point or other. It’s really important for us to not only have a search engine marketing spend, but we also want to make sure our website is really well search engine optimized so that if there’s a chance for us to get a link on the front page of whatever query you put into Google that you’re likely to click through and get to us and that’s sort of free marketing for us, and so making sure our site is well optimized is really critical.

We have a number of people who registered for the call today and I’m not sure exactly, I haven’t checked and saw who’s on, but there was certainly on group of folks which was a number of people from the higher education base. We saw Babson College and Jefferson University Hospitals and the University of Washington, the University of Toronto, and the University of South Africa had registered for this conference. I think this a great example of organizations that are standardizing on Drupal, reusing it, trying to make it work across a wide variety of needs from the main university website to their alumni fundraising to professor’s websites, to student project websites.

What I want to do is spend a little bit of time talking about a college that’s closer to our headquarters there in Boston which is Harvard University. Harvard’s recently within the last year we launched their Harvard.edu on Drupal. They’ve built out a really great site that I think shows some great examples. In this particular case, they’re really four missions for higher education. They’re first recruiting, and so they want to be able to get people to come to school and pay tuition and be educated. In this case, you can see right there in their information architecture apply to Harvard, link for parents who are likely to be involved in paying for that school, and then getting admissions and aid. First mission, recruiting and you can see how this Drupal site clearly makes it easy for people to accomplish that recruiting goal.

The next is that higher education institutions really focus on teaching and learning, and so in this case we see that there’s a city lecture series so that people can visit and come to campus and be able to see a series. There’s a link directly to the faculty there. There’s a list of the courses, and then there’s also access to a few things in the library and academic research as well as other research that’s going on campus. That teach and learning component is really critical, and so the information architecture points people to that to help accomplish the mission.
Universities are being increasingly scrutinized for the huge rise in tuition fees. Nationally, at least in the United States, we’ve seen total tuition that’s managed by the Federal government or assisted by the Federal government hit almost a trillion dollars. There’s been a lot of question in about all of this money that’s being lent to students and whether or not it’s turned out to be a proper economic and educational investment. With that, comes a lot of scrutiny. In fact, some schools are being threatened that if they don’t improve the employment of their the students who are graduating and from what they’re being taught that they face some forms of sanctions or some forms of pullbacks in terms of their ability to get access to Federal dollars as well as potentially student loans for the students who want to come to the campus which is a real threat to their economic model. Employment is absolutely critical, and so obviously people graduating from Harvard want to see placement. We have really high unemployment rates for young people, particular between the ages of 20 and 25 as the economy’s slowed down.

Last but not least, we want to raise money. Building endowments, getting people to put gifts in their bereavements or to donate a certain percentage of their income back to their alumni, sometimes they do it because they love the sport team, sometimes they do it because they’re loyal or they have a great memory, they met their spouse on campus, they want their kids to go that particular campus, and so they believe that making donations that it’s a great way to stay close to the university over time. This is the fourth mission integrate university. If we look at them all together we see recruiting students, raising funds, teaching and learning, and job placements. This is a great example of how open source Drupal can be used to accomplish the mission of higher education to really focus on the business processes of a higher ed campus.

Just to summarize again, the first thing we looked at was the costs and different ways to model costs, and look at costs and savings. Then, in the second phase we focused on the effectiveness of business processes, so website and content updating, customer service, labor saving through developers, productivity savings, the amount of people using the website, and savings in sales and marketing from improved SEO can really give a concrete example of how a university was able to apply the business processes to the Drupal site.

The next thing that we want to take a look at is flexibility. Flexibility is really critical and we often hear training of evaluation phase that the existing CMS is being replaced because it just can’t do what the business needs it to do, or there’s a lot of tension or a lot of frustration between he vision that the marketing team has and capacity that the IT team has to deliver what they want with their current CMS. Flexibility is critical. Flexibility can mean a lot of things. At the very high level, flexibility means I can deploy Drupal as an Internet on private network inside my company, or it can mean that I put it into a cloud service and so I’m able to host my public website externally, or it could mean that I want to consume Drupal as a different deployment model as a SAAS model. It can also mean that you deploy Drupal in a multisite configuration so that you can have one code base but dozens or hundreds of sites potentially. There’re lots of different ways that Drupal can be deployed, and because of the way the license is free and because you have to access to source code there’s no real restrictions about what CPUs or where you deploy Drupal. That offers a lot of flexibility. We also see that Drupal as it grew up really focused on a number of things. When Dries was student in his dormitory he was working on parts of the Linux kernel and particularly documenting some parts of that. He believed that he should follow along and in that open nature that the Linux project could have, and so Drupal has always tried to adopt web standards and open stadards so there was interoperability. Because of that, at the heart of it, you’ll always see Drupal adopting those standards very early on. We also see that a large of number of organizations want to put a lot of data, particularly when they’re when they’re building complex web applications, and so we’ve always had open data formats and support for open data. That can extend to things like semantic formats. Some proof in the pudding, for example, there’s a been of number of large initiatives recently where governments have stood up open data sites and so we’ve seen organizations like data.gov.uk expose most of the national data through a Drupal site and data.gov here in the United States has exposed a lot of data through their site, and then recently just in the last few months I think India has exposed their open data site in Drupal as well. That’s a great example.

Then, last but not least, is open SAAS. Open SAAS is a really unique model. A lot of organizations are increasingly looking at their IT portfolio and saying, “We don’t want to manage this anymore. We really want it to be managed outside,” and so we see people turning to the cloud or turning to SAAS vendors and trying to get it open. There’s a good upside about that. It can be very effective. You don’t have to maintain a lot of technical talent. It just works as a service. It has great service level agreements behind it, and you pay a nice operating cost, and it all taken care of. That’s great. Probably one of the better known organizations for doing this is Sales Force who’s a very early adopter of cloud technology and software as a service. The caveat if that if you ever decide that you want to integrate more tightly those software as a service applications into your IT systems you may be in a steroid injection where you’re limited by the road map of the vendor. One of the nice things about Drupal is that you can have it in SAAS format through a product called Drupal Gardens and there’re a few others in the community. If at any time you want to take your site out of that SAAS environment you get your source code, you get your data, you get everything you, and then you take it and move away. That’s really a great option for a lot of organizations that are looking for I do manage low cost sites, lots of microsites and be able to roll them out.
The other thing around flexibility is talent. Many organizations if they either have a custom CMS or they have a proprietary CMS the talent pools are extremely small around those CMSs, and so working on an open source project gives you a lot of access to a very amount of talent and you can switch back and forth over time to look at different talent whether you go with internal talent, external talent, you go to an agency there’s a lot of Drupal talent in the world. We also see best of breed integration. Many organizations (I'll show a slide in a little bit) are really talking about how do I integrate not just my CMS, but I need to integrate it into my newsletter mailing systems, I need to integrate it into my CRM, I need to integrate into my enterprise marketing management systems, I need to integrate into my backend proprietary applications. Drupal’s built to integrate, and so if you need to tie it into some different business systems there’s a really high probability that if you go on drupal.org and look for a module there is a module to tie. If you haven’t, there’s probably one that very similar that can be based on it, and so your developer can build a module to tie into it.

Drupal by its very nature is agile. It’s developed in an agile methodology. The idea is that from architecture standpoint it starts off as a very lean core, and then as you need features over time you can add these small reusable modules that allow you to deploy more features in a short period of time as opposed to more of a proprietary CMS where it’s a gigantic proprietary one-time install and hopefully it has all of the features that your organization needs. Drupal’s really built around adding more things over time.

Then, last but not least, is the ability to keep up with innovation. I have a couple of slides where I’ll talk about some of that innovation. Certainly, the things that we’re seeing going on today is the huge shift to the mobile web, and so being able to tie your Drupal site into a mobile browsing experience or having a native mobile app or a native iPad app or a native tablet app is something that there’s a lot of stuff. If you go today to iTunes you can download the Drupal Gardens Drupal app, and then simply installing the mass module for Drupal 7 on your site you can have a native content offering experience with your Drupal site on a mobile application. Also, projects like Sparq which is a distribution to Drupal that’s hosted on Drupal.org you can have some really rich mobile editing experiences. We’re also seeing people want to tie Drupal into other products and other services. One of the things we’re seeing is a huge explosion of the social web. Marketers want to reach out where that audience is which on Facebook and Twitter and Google Plus and YouTube and things like that. They want to be able to tie their Drupal site in so that when they publish content on their Drupal site it also gets published out to Twitter, out to the social web. Then, people comment they’re actually able to interact on a Facebook app or a Twitter app that’s actually running in Drupal if just being exposed through that social media portal. That ability to have that integration is really key. The weight of the social web is incredible. It’s very difficult to keep up.

Earlier I talked a little about the best of breed tools. This is a great slide. It really talks about how you can line up your social media, your marketing automation, your CMS, your analytics tied into Drupal, and then on the other side of the equation where you have all these backend systems that are doing these measurements you’re then able to push forward and do all kinds of measurements of external applications like the primary website, your mobile app, your social and Facebook, Twitter, your campaigns, your video portal. It’s has that innovation, that integration is really critical.
I want to now start to get into some more case studies or some more concrete examples of things. This is a great picture from 2012 Olympics where in the women’s, I think, four by 100 relay the broke the world record. It’s one of those moments where Twitter is just all a buzz, a lot of people watching this, really, really exciting event that a lot of people wanted to experience in real time. One of the challenges that organizations like this had was how they do deal with this explosion integration of real time media. One organization that had to deal with this was NBC Sports. They had actually won the Olympic contracts in a previous round of bidding but most recently renewed in 2011. One of the challenges they got was how do I deliver content experience through television around the dinner hour when most of my viewers are watching yet at the same time this acknowledgement that most of the Olympics are being consumed by people in social media at the same time. They had received a lot of criticism around the Vancouver Olympics about this and there was even some estimates that they had lost hundreds of millions of dollars on the Vancouver Olympics. When the bidding came up for the 2011 Olympics there was a lot of focus on what is NBC going to do in order to be savvy around the social media, this more of a real time experience, how are they going to innovate this very old classic American TV company, how are they going to be able to innovate at the pace that the web is going. They went out and they did an evaluation of the CMS platform that they looked at it, and because there had been a lot of success within the broader NBC family, NBC Universal family with Drupal they had Drupal as a contender. I was brought in early on after I gave a talk not that similar to this one at NBC to help them with part of their evaluation to Drupal. In the end, they did go ahead and they did select Drupal. Part of the reason why they did they were in this situation where they were competitive, the bids had gotten very high, up to 4.4, almost 4.4 billion dollars to get the Olympics, the U.S. network for the Olympics through 2020, so a huge amount of money on the table and they had to be able to say we don’t want to get caught like we did in Vancouver again where we were stuck on a platform, we’re not able to innovate at the rate that Twitter and Facebook and all these things are happening, we need to be able to keep up in real time.

They chose Drupal because it was, at least in part because it was incredibly innovative platform that they had a huge community of talent, they had a lot of innovation, they had a lot of flexibility because who knows what’s going to happen in 2012 today. What’s the web going to look like in 2020 when you’re delivering that last game with that web experience? For that bid and for the London Olympics in 2012 they had to really focus on delivering that platform, so we worked with them to deliver their mobile site for NBC Sports and are now working with them to deliver a wide variety of other platforms due to the success of that platform for other properties within the parent organization Comcast.

Another thing that you want to be thinking about when it comes to innovation is it’s great to hear that there are innovative Drupal developers out there, but how do you really deal with getting that talent within your organization? One of the nice things that I encourage people to do is when they’re building a business case for Drupal and they’re thinking about the flexibility to have really being able to address the total amount of talent. One of the nice things about Drupal is that it has a very large talent pool. In fact, on LinkedIn if you go into LinkedIn/skills and you type in Drupal you’ll see that there’re 62,000 people on LinkedIn with Drupal skills. Obviously, that’s probably some expected distribution of the quality of talent over time, but certainly on Drupal.org there’s over a million people registered, and so not everybody has Drupal on their resume on LinkedIn. It’s a very large pool, and so there’re some really amazing people doing a lot of interesting work out there.

I’m going to point out just three people. I recently gave this talk in Texas. There three folks happen to be from Texas, but they’re indicative of the talent that’s available. Chris Rupple with Fourkitchens in Houston doing a lot of really innovative work around responsive teams, how do you have a team to automatically adapt to a desktop size to a tower size to a smart phone size. He’s been doing really great work in that space. That’s kind of become the de facto for the future of the web right now is that when build a Drupal site you build with a theme that takes the assumption that there will be a mobile experience if necessary right from the ground up and it’s built into the theme to be available at all three sizes. In the middle, is a partner is ours based in Dallas, guy named Tom McCracken. Tom McCracken’s been doing a lot of really interesting work around Drupal through an initiative called the App Store or Drupal Apps. The idea here is that we’ve always had all these site building and reusable components that have been really great from a technical site building perspective, but we wanted to create more of an experience for a site builder or an end user. He’s been really working on how do we combine the power and the flexibility of this great architecture and improve the figurability of Drupal to the point where you can click a button and say I want this feature on my site, I want that feature on my site. Tom has been doing some really amazing work with his team around Drupal Apps, in particular through a distribution that’s he's got called open enterprise. Last but not least, is a guy named Mark Sonnenbaum. Mark actually, also in the Dallas area, but he actually works for Acquia remotely and he’s our top performance engineer. Mark does a lot of really innovative stuff and he has very strong opinions like many engineers do. In particular, he focuses alot on performance. He’s been doing a lot of work on caching and scalability. If you want to laugh and if you know Mark or you want to see some of the personalities you go to a website called what would Mark Sonnenbaum do and you’ll see some anecdotes that a group of his friends have put together about some of his best known blurbs when it comes to scalability and performance of Drupal. Again, really amazing talent of people who aren’t necessarily working for you company and aren’t necessarily on your team, but if you’re using Drupal these are team Drupal for you. They are innovators who are doing amazing work for you and having access to these people who are just constantly day in day out doing great work that you’ll be able to benefit from using Drupal is really key to flexibility.

We’ve started to see more government sites use it. We have a number of people from different government sites. There are regional folks from British Columbia and Maine and some other regional governments. In particular, there’re folks at the national level who are on the call and I wanted to point out some of them. We have some folks from the SLAC which is the Stanford linear accelerator, part of the Department of Energy but located on the Stanford campus, or just off the Stanford campus, really interesting work there. We’ve also seen the National Institute of Standards and Technology announced that they’re likely to put Drupal and they’ve put out some bids and working through the bidding process right now. One of the things that’s really interesting is NIST is part of the Department of Commerce and Department of Commerce has been one of the earliest adopters of Drupal. You’ll see at doc.gov as well as many of the commerce sites. In fact, one of the things that was really interesting in terms of the communities that are using it is that the Department of Energy actually does these really interesting programs where they have Twitter parties where they get together and you can get on Twitter and you can chat with these people who are linear accelerator physicists and quantum physicists. In fact, one of the things that just recently happened was just yesterday the Nobel prize winners in physics were announced. One of the physicists at NIST was award winner and therefore a member of government Department of Commerce is actually a Nobel prize winner. That’s some really exciting times and I’m sure you’ll see some stuff posted to Twitter and through the websites as they work on their social media strategy around somebody in the Department of Commerce or NIST winning a Nobel prize. We also have a wide adoption of Drupal throughout the United Nations, and so we actually some folks helping to get the acronym right, but I think it’s the Economic Commission for Latin America. We’ve seen a large number of non-governmental organizations, like the United Nations, like the World Bank, adopt Drupal and in a really big way because they need flexibility.

A little closer to home, we started to see some organizations adopt Drupal. This is actually a website for a grocery chain in the United States called Whole Foods. Whole Foods does a lot of interesting stuff. They wanted to create really great experiences. One of the things they do is if you’ve spent any time in a Whole Foods they have these really beautiful store layouts, just lots of fresh fruits and vegetables. When you walk in, they always have these beautiful displays of food. They really spend a lot of time thinking about what is the experience when people go into our store. When it came to their web presence, they wanted a really great web presence. They started by launching a magazine on Drupal called Dark Rye that really showcases some recipes and talks about unconventional ideas and innovation and creative things around that. Finally, when that was project was a success they actually moved WholeFoods.com over to Drupal. The big focus here from an innovation standpoint was really on personalization. Their focus was they have about 250-300 stores throughout the United States. They’re constantly growing and adding. They wanted people not to just come to their store. It’s know it’s very hard to see, but if you see the second cow from the left on the top left it says welcome in some white lettering and it actually says Potrero Hills which where I’m broadcasting. Potrero Hills is a neighborhood in downtown San Francisco, and so about a mile from me is Whole Foods. I spend a lot of time there. One of the things they do is they give me the opportunity to log in and select my store for Whole Foods, and then at that store they actually have somebody whose job it is to go and put Potrero Hills specific content onto the website, so that when I go to the website either as a mobile experience or from my desktop experience I get personalized content that’s specific to my store and that I’m interested in. At the same time, they have a national strategy where the national brand can push down content and override what’s at the local store. If they have an anniversary that’s store specific because that store, just like me just had its fifth year anniversary, and so they had some specials on one day. Then, if there’s a national event they can override that content and decide what’s going to be on the home page. That ability to do customization and personalization is really critical to the mission of Whole Foods to create the kind of experience that they want to create.
We see this focus on innovation being really critical for everybody. Here’s a great example. Probably most of you have heard of Pinterest or Facebook just yes launched a competitive feature to Pinterest. Pinterest is the new hot social network. In February 2012, it was the first web property to reach 10 million unique visitors per month. Incredible rate of growth and you can see that nice little hockey stick style growth inside the Pinterest 10 million uniques logo there on the left-hand side. What happened if somebody got on Drupal.org and said Pinterest is now at 11 million users, could we add Pinterest capabilities to Drupal? A month later, sure enough, a developer in the Drupal community went out and created a Pinterest module and so there was Pinterest integration in 15 sites going live with Pinterest capabilities and integrated into the Drupal site.

If you think about that level of innovation what it does from just something big happens in the world, a few weeks later somebody gets on and says can I have this, and then a couple week later after that you’re actually people live in production with that feature. Take that and compare that to a proprietary CMS there’s no way they’re going to have that ability to put that into a road map. It just doesn’t work. They just don’t have that huge pool of talent of people around the world, tens of thousands of people around the world who are building custom sites and are contributing back components of what they’re delivering to their customers back to the big pool to make a huge difference in terms of the ability to innovate.

Again, just a quick summary of our third point. First point being cost and cost savings; our second point being adapting to business processes, getting value and value effectiveness out of business processes; and our third point being flexible. We talked about different deployment models, ways to push Drupal out in terms of SAAS and open and private Internets, and then we talked a lot about talent and innovators and people who can be effectively on your Drupal team to help you innovate and how is it agile. Then, we talked a little bit about a couple of different organizations both in government as well as a retail organization like Whole Foods that’s using Drupal to create great personalization experiences.

The thing I want to talk about is risk. Risks is probably the most important component here because really if the organization ultimately perceives that Drupal is too risky because they don’t understand it or they perceive open source to be too risky no matter all effective all the other things are, they just simply won’t pick it. Tackling the risk profile is really critical. There’re a lot of things that you have to think about in risk. The first is that it’s not just a matter of writing a check or getting an insurance policy. Managing risk and managing open source requires real change in the thinking of the organization if they haven’t adopted it yet. Most IT organizations are dabbling or have dabbled particularly around Linux source some kind of systems tools, but to be at the application level it may be a little less common. Thinking about open source applications requires real change. You also have to think about elasticity and some security. Procurement can be a real challenge. A lot of organizations are used to going through proprietary vendors license and saying we will accept this clause, we won’t accept that clause, and then all of a sudden they want to know who they can sue if the Drupal site doesn’t do exactly what they want. That’s a way a lot of enterprise procurement offices work. They want to put all kinds of clauses. They want to say that they own the rights to the security. They own the rights to the software. They want special clauses that if Drupal ever goes bankrupt that they’re guaranteed that they get access to the source code and kinds of things which they get out of the gate, and so they have trouble understanding this. Having to work through your procurement team to help them to understand what does it mean to work with an open source licensed product. Licenses, obviously, aren’t an issue. Champions are really critical. If you want to see Drupal succeed with their organization there’s almost always, always a champion, somebody who really understands the organization, understands the road map, who the stakeholders are, how the decisions are being made, what are the concerns of the organization and they’re able to translate those business concerns into advantages for Drupal, ways that they can explain the successes of Drupal to help it. Then, you deal with IT strategy, and then the fact that with a Drupal site because it’s going to be successful, because it’s going to offer a lot of business features you’re going to be probably doing a lot of development over time.

This is a picture I’m going to just tell you a quick anecdote about risk and risk management. This is a picture of Dries Buytaert the founder of the Drupal Project in I believe 2004, I think 2004 in December. This was his first presentation, Drupal 4.5 had just been released and he was flown out to Vancouver. In particular, he was flown out to work with a company out in Vancouver called Bright, one of the early evangelists and one of the teams of the really great top tier Drupal people who are contributing. It turns out that they had a really interesting customer. They had a customer that wanted to build a series of university styles sites that basically collaborated, but it was a very particular use case. Part of what they do is they have this organization called the National Defense University in I believe it’s in D.C. or Maryland, in that area. What they do is they teach advanced strategies, tactics, counterinsurgency, non-proliferation, all those kinds of things. We actually have people from our allies around the world come and take classes at the National Defense University. They may come take a class on counterinsurgency or border protection. These are advanced graduate level classes, so these are Ph.D., masters level classes, post doc level classes, very smart people, strong commitments. They’re coming, but they want to keep the dialog going after. Actually, as part of a larger initiative in the United States Department of Defense they have something called the Security Cooperation Agency. United States spends a fairly large amount of money, actually I think more money than the next 17 largest militaries in the world combined. Part of having a successful national strategy is having allies and strong relationships. Part of the way you keep those relationships is when people come and do training with you when they go back to their countries to their defense work they’re able to have a dialog. What the Department of Defense did through the Security Cooperation Agency was to create a series of 17 sites. They created the 17 community sites what actually based on Drupal Commons and some additional features on top of that. They wanted to be able to host these and manage them worldwide. Today, we’re able to host those and we host them in Acquia cloud. All these organizations that are really focused on security and how do we collaborate across borders. They maybe not state level department secret talks or not necessarily operational military mission level talks, but there are people talking important questions and the questions that not necessarily governments want publicly exposed, so security in the cloud and security with Drupal is really paramount. The Department of Defense when it comes to having models for security they have something called DIACAP which is their Department of Defense level strategy. These cloud sites have gone through the DIACAP. You can think of it as Drupal has gone through some of the most rigorous security standard issue of the Department of Defense and are even hosted in the cloud, and so this is a really important standard that Drupal has achieved in terms of security and risk.

One of the questions that people ask is can open source software be secure? If everybody has access to our source code does it mean that the hackers are going to figure out how to do it? I think what Dave said and my experience has been we’ve seen this code from the Department of NCIO continuous broad peer review enabled by publicly available source code supports software reliability and security efforts. One of the things that I’ve been doing in the Drupal project for a number of years is helping to coordinate the Drupal security team. We have a team of about 38 members right now. We put out regular security releases and people from the Department of Defense, people from agencies, people from newspapers, security experts around the world contact our team and basically say, “We found a problem with a security violation in Drupal,” so we have a process for working through with the security teams to validate that it is, in fact, a security issue. Then, we verify that there’s a fix that’s made, and then we push that out. We have a number of tiers in which tiers of modules, tiers of code, which we put our security. There’re some very rich processes that allow us to do hundreds, thousands of interactions with security people and developers around the world to manage the security profile. There’s actually in effect, because there’s a lot of discussion with security and because there’s a lot of openness about security that actually makes the product much more secure or at least as secure as proprietary products where there’s a only very small team and nobody really knows what’s going on in the code unless you’re a hacker.

Sometimes people have to think about how they secure borders. I was down in Texas recently and noted this really interesting article. One of the judges down there with the election coming up made a joke, he was serious, but he had said that he was afraid that if Obama was reelected that the United Nations would invade Lubbock, Texas, assuming from the Mexican border. Sometimes when people are thinking about security they’re thinking about really serious security and other times their not. Yes, the guy had made security to stop the United Nations from invading from Mexico, but it’s to train longhorn cattle to keep them out. This is a little joke I had going with our friends in Texas about security models.
There are some more serious security models. This is an article from the newspaper this morning from the Washington Post. One of the things is NATO has just come up this morning and said they need to be able to defend their ally Turkey if Syria continues to shell across the border. In the New York Times this morning we also saw an announcement that the United States is going send 150 troops to the border of Jordan to help primarily with relief logistics but also to prepare in case there were a complete collapse and certain weapons were available. These are the kinds of really serious discussions that allies can be having and can be talking about in these kinds of sites. These are Drupal sites. They’re secured in the cloud. They have to meet a standard. There’s a lot at risk when using Drupal in this particular manner, so it’s really important that security is critical.
In 2009, I had the opportunity to reach out to the White House that had started using Drupal for recovery.gov. This is a picture of me at the White House with Dries Buytaert the founder of the project. When we got three, we were asked a lot of questions and because I had been working on the security team for a number of years and thought a lot about enterprises issues having come from IBM we were able to convince them that Drupal was a viable model. Nobody really knows what the most hacked site in the world is, but certainly lots of people, lots of national governments would like to see a compromise of whitehouse.gov and we’ll see a hacker group say that they’re going to try to take it down. Whitehouse.gov is a site that launched Drupal and it has to meet extreme security standards.

We also see other organizations that have to deal with security. This is a screenshot of Tahrir Square in Egypt about a year and a half ago, the March time frame. What had happened was the protestors as part of the Arib Spring were rising up against Mubarak in Tahrir Square. They were being surrounded by paramilitary, militaries of the Egyptian government that were loyal to Mubarak. There were fears that he would be killed. One of the things that went on was that a lot of the protestors were very tech savvy, so they were on Facebook, they were on Twitter, but they were also on one of the most widely read news sources in the Middle East which was Al Jazeera, in particular Al Jazeera English blogs. They actually had huge massive spikes in traffic. People were blogging there. One of the things that the paramilitaries did was to try to take out the Al Jazeera site. What they did was they actually firebombed the offices of Al Jazeera in an effort to try to prevent them from publishing and being a voice pushing for freedom in Egypt. We actually got a call from some folks that we knew over at Al Jazeera and they were struggling with this massive, massive increase in scalability of traffic. We were actually able to help them and move them to Acquia cloud, and so today they’re a customer and help them with their scalability issues and things like that. That ability to look at risk profiles, the ability to say, “My site could really take off. I could have huge news. I could end up with the need to have elastic resources.” It’s that ability to be in a cloud source and to provision elastic resources on demand are really critical. This could help to really mitigate the risk of your site.

I think I wanted to just summarize with that to say risk is probably the most important, or if not the most important, consideration that your organization’s going to be making in the business case for Drupal. People like the White House and people like Department of Defense, people like Al Jazeera, really, critical times in history have evaluated Drupal, have used it, have taken through security models. Those are great use cases to help you show that this can be effectively managed around Drupal.

To review, we talked initially about costs and cost reductions. We talked about enabling the value, in particular that Harvard site showing how we helped to accomplish the missions. We talked about all the flexibility and the innovation all the great talent, how the people were doing really amazing and interesting things. Then, we talked about risks. We talked about organizations like the U.S. government and important organizations like Al Jazeera that are trying to get the word out and trying to get journalism and freedom out to across the world being under threat, and so they’ve had to deal with different security profiles.

That’s the summary of what I have. If you have any follow-up questions you can feel free to email me or reach me on my mobile. I’m sure Hannah has a long list of questions because I’m probably 10 minutes over. Let’s see what kind of questions. The first question I have is from Randy Seal and he says, “How do find Drupal developers who are willing to work with small 501c organizations that have sub six-figure development budgets? It seems that every Drupal you can find doesn’t want to work in that market. Actually, that’s a little surprising to me. I think, in general, we do find that there’s a huge amount of adoption of Drupal in non-profits. In fact, some of the recent surveys that had come out, listed non-profits have indicated that many if not more than 50% of all non-profits in the United States had switched over to Drupal. I think the challenge is many people who believe in openness, believe in sharing, believe in those kinds of things that are popular in open source are very interested in organizations that are serving a mission not just connecting economic agendas. There are a lot of Drupal talent in non-profits. The catch is that what’s happened is as we’ve seen Drupal take off in governments and all these industries, particularly in commercial industries, those people are making a lot of money with Drupal have then come and basically tried to pull talent out of those non-profits and try to get them to come to work. To some extent, they’ve had some success. Certainly, there are a lot of partners. I you go to the Acquia partner page you’ll find that there’s a lot of organizations that have very strong histories of delivering to non-profit sites, most of the early Drupal shops that I know that are five or six years old who are very focused on non-profits and servicing non-profits and universities. That’s possible. I think if you’re looking for talent, there’re always a couple of places to go. Groups.Drupal.org is a great place to post a job. I have a couple of web pages that I can send you. Let me see if I can switch over and show you a page. I think I may have been looking at your site earlier. You go to Drupal hire developer, hiring a site developer. If you go to Drupal.org and you go to hiring a site developer, and you get in here skills and essential backend web developer and skills of essential front end web developer.

What I would encourage you to consider doing is if you’re finding the market hard find Drupal developers since you’re competing against all these for-profit organizations, find somebody who’s just really believes in the mission of the non-profit who’s really trying to make a difference. Then, see if they have some basic web skills. If they have these basic web skills, then they can work on Drupal for you. You’ll also see a lot of people who jumped across. They started in non-profits and went to the enterprise. They found that they motivations in for-profits weren’t that good for them, and they’re actually looking to come back out. I certainly know of a number of developers who had recently chosen to transition out of that. Send me a follow-up email. I’m certainly happy to point you to a long list of Drupal shops that might deliver sub $50,000, sub $20,000 Drupal sites. There’s always sort of a part of the market that’s always going to try to make their way up the ladder, and so smaller budgets work and I’m happy to make some recommendations there.
I’m going to switch back to the Webex and see what other questions we might have. Are there any other chat questions?

Hannah: There was one more question. It said, “Could you teach upon the need to make the argument to your CEO about contributing code back to the project and budgeting for it?

Kieran: Sure. You can build, hire a developer, go out and build a module that does exactly what you need. Let’s say that costs 10 units of whatever. The argument that you want to make to a CEO is the effort that goes into that module and the maintenance and ownership of that module over time is not insignificant. Somebody has to look at that code, they have to go in and QA, they have to test it. There’s probably somebody in the business who has a non-proprietary need to do something else with that module of code does, and so you’re a lot better off having that module in Drupal.org’s repository where people with QA that module, other developers will come and maintain it for you where it can get external security reviews so people can see that code, review it. If there is a bug in it, it’ll be reported to the Drupal security team. If it’s a fully released module, security team will do a review. They will work with maintainers to get an appropriate patch. If there’s a broad problem in Drupal around security it could get code reviewed as part of a broad sweep of Drupal. Figure out the security costs, figure out the new feature costs, figure out the code maintenance costs, and then basically say, “Is it worth it to for us to keep this module solely proprietary to us?” There’re a lots of reasons why modules that are specific to businesses that aren’t proprietary. Oftentimes, a lot of modules are generic web architecture pieces that are plug and play. We want to contribute back.

The other thing that you need to consider is if your business is growing and you want to hire talent, when Drupal developers are looking around part of the way they find that companies are hiring Drupal developers and are looking where to work is they’re looking at these modules that are being shared and they’re saying, “You’re some really good work here. Would you like to do a part-time contract work with this module?” That can ultimately lead to full-time. It’s smart CEOs thinking about addressing talent, minimizing their costs, opening up to innovation, all those things will really take all those things into consideration. Any other questions?

Hannah: I don’t see any more. Thank you, Kieran, for the great presentation. Thank you everyone for joining. Again, slides and recording of the webinar will be posted to our site in 48 hours. Thanks, Kieran.

Kieran: Thanks, Hannah. Thanks, everybody.

L’Open Source Drupal et Acquia pour la Stratégie Digitale des Administrations Publiques [October 9, 2012]

Harmonious Development: Standardizing The Deployment Process via Vagrant and Puppet [October 4, 2012]

How to Use Drupal 7 to Meet Your Accessibility Goals [September 27, 2012]

Click to see video transcript

Hannah: Hi everyone. Thank you for joining the webinar today. Today’s webinar is “How to Use Drupal 7 to Meet Your Accessibility Goals with Mike Gifford from OpenConcept Consulting. I’m Hannah Corey. I’m a Marketing Specialist at Acquia.

Mike: My name is Mike Gifford as Hannah said and I’m a Drupal 7 and 8 Core Contributor and I’ve also been active in accessibility space having organized some accessibility end conferences or accessibility conference. I’ve also contributed a section to the definitive guide to Drupal 7 for accessibility so that was a nice thing to have the largest and most comprehensive Drupal book to also include the section on accessibility. I’m also the president of OpenConcept Consulting and I’ve been working with Drupal now for over seven years and I’m happy to be part of such vibrant and dynamic community.

I’d like to talk a bit more about, specifically, what are some of the goals that we are looking … What are we talking about when we’re talking about goals and alternative of trying to use Drupal 7 to go up and to accomplish that? It’s important to trying to find what that is. Of course, everyone would like to have a barrier free implementation for the website that is completely universally accessible to everyone that follows the principles of universal design and progressive enhancement and these are definitely worthwhile things to try for.

There is no complex website that is 100% accessible to everyone so there’s an element right now where because of assistive technology and the pace of change in the internet, there’s nothing that's perfectly accessible. There’s a lot that being done to move in that directions, standards and guidelines to help evaluate accessibility, but there’s nothing that’s completely universally accessible. One of the things that...one of the guidelines that’s available is the Web Content Accessibility Guidelines,the WCAG, and this is part of the worldwide web consortium.

W3C is involved in setting up all sorts of standards for the internet and it’s really critical institution for looking at and defining standards for the internet. They released, in December of 2010, they released WCAG 2.0, which is right now seen as the standard that people are looking at for accessibility internationally. There's also a number of draft standards that are being released that are worthwhile to consider and one of them is ATAG, or the Authoring Tool Accessibility Guidelines, and ATAG is still in draft but it’s...anytime you’re dealing with a content managing system, you have to worry not only about how the content is viewed from the user’s perspective but how it’s generated as an author and is it the tool that can, as an authoring tool, also go off and meet the same level of accessibility guidelines.

Finally, section 508, although Section 508 itself is one of the oldest internet standards for accessibility. It was initially published in 1998, and it is in a process of being refreshed and we’re very hopeful that in the next year or two, we’ll see a release of that that is able to keep up with and perhaps, we can exceed the standards set by the international community. Those are some ideas about goals and what is possible within web accessibility, and it’s important that this is something that we do work to establish this as more than just a lip service towards accessibility. That we need to look at what are the practical implementations and what are the practical approaches that we can do to achieve that. A lot of people in their publications and promotions talked about accessibility, but often, that’s more of a sale side than a practical on the ground. This is what has been done and tested and this is what we know worked for people with these disabilities.

I’m not sure what the status is of everyone who is in the call, but I wanted to enhance and talk a bit about the cost of the internet and the internet is constantly evolving, and this isn’t new for anyone here I’m sure, but I’m trying to understand both the clients that the users are using for whether it’s Firefox or Opera, or Safari or Internet Explorer. All of those tools are changing on a regular basis, so are the assistive technology devices that are being used.

If somebody is using a screen reader or is using a plugins tool to enhance the accessibility, these are tools, which change on a regular basis. Likewise, the demands of the internet and demands of what your website are, are constantly going to be changing, and there’s going to be new expectations for client deliverables and what matters as your site is being implemented. Even the software within that is changing on a regular basis as you’re keeping up with Drupal upgrades, and ensuring your site is secure. There may be changes within the code that is accumulated in Drupal, wherein the underlying infrastructure of jQuery or even how Apache is potentially dealing with requests that are changing.

You’re not dealing with the static implementation, and so there’s a need to constantly go through and look for barriers, and look for ways to find ways that your users are being limited too to access on the website. I would encourage people to look for feedback from the users and to elicit information and directions in terms of what's possible, and also to schedule a regular accessibility review. If you’ve done a full audit of your websites, then in six month’s time, it’s important to go up and look at a gate, and look at how, not necessarily doing a full audit, but to look at your website to find the ways to improve the process.

The best practices are changing and there's a lot of dialogue and discussion about what is the best way to approach things, especially when you’re trying to implement something like a responsive design, or you want to be able to even just add any feature like that, a slideer on the front page or something like that, that allows people to have a more interesting visual presentation for your front page. If you’re approaching and upgrading, you’re able to go up routine to look forward and to work with community, then there's always ways to improve and to constantly make your websites better if you’re working with the community of users and developers around the world.

It’s important to plan to go out and see that your website is accessible, and Drupal is a complicated system and accessibility itself is quite complicated. If you’re looking to implement an accessible implementation of Drupal, it’s important to try and leverage what expertise is out there and to where possible eliminate the unknowns. There’s a lot of modules that have some evaluation for accessibility and have some reviews or patches that improve the accessibility of them.

There's certainly places where there are problems that are known and also places where we know we’ve addressed the issue. If you can work with these known elements and read up on the board set that has been done to improve accessibility; that will certainly help a lot. It’s also important to try and contribute back to your changes. You’re going to find things with your users that are going to be ... that maybe something that others in international community have not yet identified. So if you you have an audit done on your website and the person doing it identifies that there's a problem with one particular widget It's useful to go up and to contribute, even those problems back, so that we can work through and try and resolve them as quickly as we can.

I wanted to also talk specifically about this in terms of WCAG. WCAG has a structure of their guidelines called POUR, that all of the information must be perceivable, the website must be operable. It’s important that it also be understandable and finally robust. There’s quite a lot of documentation available on this, but I wanted to make sure that people had a basic exposure to that as well.

Drupal seven has a lot of codified best practices built into it. We’ve done a considerable amount of work in about a three-year period of time to find issues and problems and to fix them in Drupal seven, and also to think beyond a single site implementation to try and say, "What can we do to see that we have a solution that works well for everyone?" I see that a lot of what we’ve done in Drupal seven has a lot of best practices, that if you just follow what is in core, you’re going to have a more accessible website, than not in many cases. When you’re working on developing Drupal that's of course an obvious example of where to look for, for example on how things should be done and how things should work.

One of the great things about core that Drupal Core has built with, there's a lot of information or APIs application processing interfaces, so that you can access the information and be able to organize its potential ways. That’s quite useful as well. The things are centralized and modular so that improvements in Core will propagate out to other pieces to your web setting.

There's also really excellent international communities that meets on a regular basis. There was a sprint earlier in March and there’s another one being organized in November. We do get together and meet, and then look at ways of improving Drupal for accessibility and there's also a very vibrant issue queue as well as the discussion group that we'll provide both of those at the end of the presentations.

Finally, one of the great things that we’ve accomplished in Drupal seven is the ability, not only to publish content that's successful but also to be able to administer the website in a way that is accessible. A blind user can install Drupal, can maintain Drupal, and can add content to Drupal, administer Drupal, and also read the content from Drupal. That operative is seen as a huge issue, but as websites become more interactive, the distinction between what is an administrative piece and what the user piece, they get kind of fuzzy. You may want to have a dashboard available to your users, you may want to be able have a discussion forum where they’re able to log in and have to have access to pieces of Drupal that are, within many cases would only be available for the admin users. It’s really useful to have the flexibility to know that the administrative pieces of the website has also been tested for accessibility and for the most part, had those barriers removed so that people can access the website.

I wanted to talk about some of these specific victories that we’ve brought into Drupal seven and ways that this can help you maintain your website in the future. The first one that you may notice if you’re needing to install Drupal is that there's a Skip to Main function that’s part of all Core themes. This is a great best practice for people who are keyboard only users or people who are blind and use a screen reader to navigate the website. What the Skip to Main section does, is it allows you to jump through all of the navigation that's generally part of a website and get right down to the content. It’s a great way to skip the navigation and get to the content and this is something that's also part of the admin themes as well that come with Core.

One of the issues in Drupal six that became a problem was the drag and drop, and drag and drop is a wonderful functionality that has a lot of – a much nicer user interface for people, but we had to go off and create a work-around so that we could more evenly and intuitively make it so that people who could not use a mouse were able to easily drag and drop the content and have the direct access with content, and to the administration of content rather. There’s also work to go off and improve the semantic markup in the breadcrumbs and the list, things like ... just trying to make sure that especially as we’re dealing with smarter assistive technology, the more semantics you can provide to the market, the better they ... to the assistive technology, the better they can interpret that and provide tools to help the ... or to assist the user in navigating the website.

We’ve also worked on improving the color contrast in the number of places and also added the area support. One of the tools or standards that W3C is building now is the web accessibility area, the Accessible Rich Internet Application Support. It’s nice way to build in more semantics, particularly when you’re dealing with dynamic content, so content that’s changing as you’re clicking on the website. More and more websites are becoming more like desktops than they were, the web pages of even two or three years ago, the type of functionality that users expect, and the type of dynamic responses are so much more complicated than what they were shooting back in 1998.

Being able to present the assistive technology with semantic information that describes what has happened and how to engage and how to go up and to make the information that’s been presented with the user in a way that is meaningful to the assistive technology. It’s a great thing that we’ve started, and there's certainly a lot more than we can and will be adding to Drupal eight, but there’s initial implementations of area for both the auto complete and ultimately, installation pages in Drupal seven. There will be considerably more available in Drupal eight.

Another great tool that we’ve added is a consistent way to go off and to deal with CSS displaying; none or visibility: hidden. This is a well-known problem, and one that we’re starting to have some good solutions for now and certainly the HTML5 boiler place has a solution that’s very similar to what we’ve implemented in Drupal seven. The issue is that assistive technology interprets display: none or visibility: hidden literally, so it will not display the information that's there to a blind user.

You need to actually find a different way to make that information, to hide it off screen and to ensure that people who are ... people do not see the information but it’s available for screen readers. We created a class called element-invisible that allows for screen readers to do this, or that works properly, we've tested it, and this is a larger challenge that one might think, because there are only a few lines of CSS.

Finding a solution that works well and is something that can be implemented across the board is a huge challenge. The solution that we had implemented prior to Snow Leopard, but Apple put out a release of its Operating Systems that destroyed the functionality that we’d implemented before, but simply even creating the class name .element-invisible, and standardizing on that was a huge move for Drupal seven, because it means that even if you want a different approach to how we implement this, you can just override that, but everyone who is ... all the modules and themes have a consistent way to name their information, to hide information off-screen and also to make sure that it’s visible on focus.

That’s like the Skip to Main link that shows up in the themes is one of the pieces that uses the .element-focusable, and there’s other elements where you would want to hide them unless a keyword only user is navigating to that particular element. Finally, there needs to be clear way to go up and hide it to make sure that it’s not presented to anyone, so that's the .element-hidden that’s available there. This will also be supported into Drupal eight, but we are changing the names a little bit. It'll have the same functionality but probably have a slightly different name space.

Another, we’ve done a considerable amount of work is with the Form Elements, and this is a huge challenge for a lot of sites that need to be accessible because so many web forms are not very accessible. The biggest way that they fall down is that there is not a direct link between the description of the input field and the input field itself. There’s no semantic tie between the two. We’ve added a title to ensure that it is with... We have a title setup so that there’s a direct link between the title and the label, so that
anytime you’re creating an input form that that information is setup and structured so that it's consistently convening over to the browser and that there's a semantic tie between the two. We’ve also built in control so that from a programmatic point of view, a developer can associate a label before, after, or make it invisible so that it isn't something that is going to necessarily ... that consistently to hide that information from the screen, but also make sure that it is all hidden from the screen, but it is available for users to be able to ... for an assistive technology to be able to read the information. This is all part of the Forms Application Programming Interface.

Essentially, all modules, and if they have a form associated with it, will use the Forms API to create them, so by addressing things in Core we’re able to go up and to address them in the application. There’s also work that we’ve done in the forms to ensure that there are more headers and greater semantic information associated with the form's interface, so it's easier for people to navigate, or especially for blind users to be able to navigate through the interface.

We didn't fix everything in Drupal seven and we won't fix everything in Drupal eight, as this is a continually evolving process and there is always going to be changes and new standards and new things that need to be addressed. If you're aware with those problems are, then you can look for it and design ... You can focus your efforts and attention on either avoiding them or focusing your programmerss on implementing solutions that will help get around that. In some cases, the problems that have been identified in Issue Queue already have patches for them, and they have a patch that you could apply back from Drupal eight to Drupal seven or that you could go to the modules that will allow you to take the ideas that are discussed in Issue Queue and bring into the application itself.

One of the problems is fieldsets, and complex forms. One of the issues with the complex form is you need to have something to bind them together, and the approach that is recommended is to tie them into a fieldset so that a phone number, if you've got an area code, a phone number and an extension, if those are in three different input forms, that those are bound together and that there's a label associated with the fieldset to explain what it is those different elements are, and that this is part of a group.

Likewise, dates are something that are often not dealt with properly, or if you've got ... if you have even an X and Y coordinate, you want to go up to express. If that X and Y coordinates are in two boxes to graph for accessibility, that should be wrapped in a fieldset. Unfortunately in Drupal, I'm not sure when it came in, but certainly since Drupal five, a lot of the collapsible elements have been done with fieldsets. Fieldsets have been used inappropriately in the past, and we’re hoping to go out to turn fieldsets into details in Drupal eight and address that particular issue. If you’re looking at moving your site to the HTML5, that is worthwhile to look forward to.

There's also some uses of HTML or other implementations like labels were used inappropriately in some cases and now we’re going through it and changing the HTML for that. We’ve addressed most of the issues where labels were used, but there still are some outstanding issues, and you can use the tool like WAVEY. Those will be identified and you’ll be able to see the user interface where label is used inappropriately. When form errors are being ... and they’re much better in Drupal seven than they were, but in terms of meeting AA compliance, and this is something -- this is a goal that the Drupal community has and we haven't been able to evaluate Drupal Core and verify that it does meet the AA standard.

There are some issues there, but we’ve made some dramatic improvements in terms of how error messages are displayed to users. So if somebody's filling out a web form and they hit submit ,a user, a blind user can navigate through that and can see that information, and be able to access the error messages, but they do need to be ... that does need to be improved. In order for it to be AA compliant, you want to ensure that there is a direct link into the issue or if somebody didn’t put in their name, that there's a direct link into or from the error message to the input form to allow them to add that information in.

Also, the publish status is something ... again, if you’re trying to go up, and to ensure that your website is set up for low vision users, there's ways to tweak Drupal seven so that the published status are in fact ... some Drupal themes do a better job than others in terms of marking up the published status, so that it's clearly visible what stuff is unpublished and draft versus what content is published, and the zen theme did quite a good job of that historically, and making sure that that was clearly visible but that’s not a hard thing to build into the theme.

There are also problems with the data tables.
If you’re looking to create data tables there in order to meet the guidelines, one of the things that is being proposed is that -- or one of the things that’s suggested is that there is a scope attribute added that clearly denotes when a header element is ... the header element has the scope of row scope colon. This is particularly important for complex tables.

For Drupal, most of the tables that are generated by Drupal are very simple and there’s very little accessibility advantage for the end user in terms of having this information in here. It's mostly a matter, trying to go up and make sure that the table data meets the guidelines so that it is following the best practices, and there’s a number of discussions on how to do this with both Drupal Core as well as the themes. That’s another area to look at. In terms of comments, if you are unable to comment, one of the challenges is in terms of threading comments, and what is the parent issue versus the child, and sort of establishing that semantic relationship.

We’ve also made some improvements in international innovation that have been brought forward. But for organizations that need to implement a multilingual implementation, you need to have a way to go up and tell the assistive technology what language you're viewing it in. If your main content is English, that’s great. If you have a link to translate to French it should say Francais, so that the screen reader does not trying to go out and pronounce Francais in English. These are useful pieces of things that are going to be added. In fact, some of them have already been added for Drupal six. It’s trying to standardized and build the best practices into Core, and there's certainly a number of issues where we’re looking and working on doing this for Drupal eight that can be used as examples or a great resource for people who are needing to go off and to setup multilingual websites.

One of the challenges that is not just the Drupal, but of websites in general is one of Modal Dialogs. This is essentially switching back and forth between and popping up an interface underneath it. In Drupal seven, the overlays module is a great example. If you’re logging in to the website, and you’ve not disabled overlays, then it’s very difficult for assistive technology to be able to know where the content is and be able to navigate with overlays enabled. One of the things we’ve worked on is a way of ensuring that a screen reader user can disable Modal Dialogs, but if you have people who are content editors who are using the assistive technology, then that will be something that you would want to disable for them because they may not necessarily know that this is a problem with overlays. In terms of the solutions for this, in many ways this is an issue that have been brought forward to the jQuery community, and jQuery is a much larger community of users.

JQuery is used in so many different content measure systems and also for proprietary or custom built web applications because it’s a really powerful JavaScript library. There’s work to go up on, and to be able to fix this you need the latest version of jQuery to bring that into Drupal eight, so that’s something that we can have addressed and have a common way to go up and address, switching the content and switching the focus of users between the pages of you now and to a particular element that they want to be able to access or edit.

I want to talk briefly about best practices now and to talk about things I've mentioned. Having regular reviews for accessibility is quite useful. and I really like the WebAIM’s WAVE Toolbar. You probably can’t see it very well here but there’s a link here for the main Acquia webpage, and I use this as an example of one that all of us will have probably seen having gotten here. If you use the WAVE Toolbar to evaluate a website, you can see areas where accessibility can be improved.

A red mark in many cases will indicate that there maybe there's a missing input form or maybe there's a label missing or maybe there's a heading or something that looks like a heading but isn’t marked up as a heading. It’s a great tool to go off and do a quick evaluation and grab the low hanging fruit. It doesn’t give you everything. None of these tools ... no automated tool will be able to effectively cover all of the accessibility challenges that you need to evaluate in order to make your website AA compliant. It does capture the low-hanging fruit and it’s really important to keep that in mind.

It’s also important to look at training your content owners. Your content owners are -- if they’re not writing good content that's well-structured and it has all text in it, it doesn’t matter how much you’ve spent on building a great framework, if it isn't implemented with the content owners keeping in mind accessibility and using tools like this, or there's another great one called -- a simple tool called CSS Home that basically use just CSS to alert the content owner if there is an accessibility, if there's alt tags missing, if there's use of an attribute tag, there's all kinds of things that can be done and highlighted simply with CSS if the content owners are educated about this, and there’s also growing tools like the Quail Library to do some basic accessibility auditing before a page gets launched.

Again, it requires training for the content office to go up and to have more awareness about accessibility issues. I definitely recommend getting an audit from a professional web accessibility individual, because there are things that you’re not going to be able to see that they are, so if you can look at it ... get somebody to give you feedback on your site from a professional perspective and give you places for improvement, it would be extremely useful. There are not enough websites out there that are actively going out and trying to reach out the users and to get the feedback from the user community, to see if there are any problems or barriers for the users.
I know that we haven’t done enough testing for people who use Dragon Naturally Speaking.
This is an area that if there are people on this call who have done testing or done work with people who have severe mobility issues and need to navigate by voice, the Drupal could probably use a lot of help because we don’t have access to enough people who are working to get access, to navigate the web in this way. Whenever you discover accessibility issues and whatever problems you run into, if you're able to contribute those back to the community, it will help a lot of us to build solutions in the future that address that. This is an ongoing issue, and if we can find the problems and address them in Core or contribute modules, will all be much further ahead.

There’s a lot of continuing work being done in Drupal, and those people who are using Drupal seven, there are 14,000 Drupal modules out there, and you're not using all of them but using a handful of Drupal modules and themes to go up and to build your websites. Many of those may have accessibility problems, and they would inherit things in Core, but they can always be overwritten by the module Maintainer. Almost all elements that we’ve worked to go up and to build into Core can be overwritten, and certainly if people are upgrading websites or modules from Drupal six to Drupal seven, they may not yet have updated their site to the new server or the accessibility standards that have been adopted in Core for Drupal seven. That’s an area that does need attention and you can look in the Issue Queue for any of the modules you use to see what are the accessibility issues that are there.

The ability to tag issues with the tag accessibility, so you can again look for modules that have been tagged with accessibility problems, or in Drupal Core there's also needs accessibility review. There's work done on Drupal eight as well, and in Drupal eight we're bringing the entire interface up to HTML5, and we’re also building Drupal so that it is responsive by design. All of the themes from Core will be responsive and flexible for mobile users. That’s again a huge improvement but it does require testing and evaluation to see how well it’s going to work for users in terms of ... too often people need to go off and added a new mobile theme.

They decided they want to add a mobile version of their website, and forget that it has to also be an accessible version of their mobile site, that in many cases the accessibility guidelines that are available for the main public website, also apply to the mobile website and it’s working with Drupal seven and in the future with Drupal eight, you'll have the confidence that a lot of that has gone through a consistent testing an evaluation to identify and address problems that have come up.

We'll need to have ... When ATAG 2.0 is released, it’s still in draft, and Section 508 are settled, we’re going to need people to review based on that as well, and to look at it and see if there’s ways that we can improve those, and if there’s a way that Drupal needs to be additionally configured in order to meet Section 508. Yes, that will be a challenge when it comes up, but we've made a solid ground work for this implementation so it should be easier for us to go up and to also address those issues. There’s a lot of work that’s being done on accessibility for CQ editor, BU editor and Aloha editor, editor have that fronts with a lot of work on addressing accessibility issues. Those are all part of -- the prominent parts of Drupal seven and eight.

Finally, I wanted to go off and to talk a little bit about the Drupal seven Web Experience Toolkit. This is something that we may be offering as a feature webinar. There's a limited organization that are working together and collaborating mostly governments, but not entirely. The Government of Canada, and University of Ottawa and the City of Ottawa are all collaborating together to go up and build a distribution of Drupal that has a lot of accessibility and usability components built into it. This is something that is going through quite a lot of testing and building on a framework that has been established by the Drupal community, but it’s something that we hope to involve more people who are looking to set up accessible websites, but also people are looking to build multi-lingual websites as well. With that, I'd be happy to go off and entertain questions, and to talk more about ways to get involved in the Drupal community and to address the accessibility issues in the future.

Hannah: Okay, great. Thank you. We have a couple of questions for you. The first is where is the best place to add area landmark in page pcl.php.Javascript or other? Can we attach an area landmark to a region?

Mike: Mostly it depends on whether you wanted to be validated by the ... the reason why people have been adding area in JavaScript is because the W3C validator has had problems with it historically, and I would suggest that the best place to add it is at this stage, in the page.pcl.php file. We weren't able to go up and add it historically into the templates for Drupal because we wanted to make sure that it was something that was going to validate when people went through the W3C. That was why it wasn't added.
There's also elements where the accessible helper module goes and adds them also in a block layer. That’s another space where it’s useful to add area landmarks, is with the blocks that are moving and migrating within the sites. If you can identify those programmatically -- or not programmatically. If you can identify those and expose that information, it helps out considerably. I would say, yes, those are the two places that I would look to add area landmarks.

Hannah: Okay, we have more questions. If I understood correctly the tool will become more linguistically diverse?

Mike: Drupal itself? The Drupal is already quite multilingual, and in Drupal six, the ability to have multilingual sites was brought into Core. It needs to be enhanced, so that you need to have additional modules to make that happen, in terms of a fully functional website, but the Core elements that were brought in Drupal six, they’re also part of Drupal seven. In terms of making it accessible, there’s extra elements that you need to do in order to be able to convey to the user that the language has changed.

To make sure that they have dedicated elements that show up in alternate language ... the most common instance for this is, if you have a website that is primarily in French, and somebody is trying to navigate that website in French, and there’s a string which has not been translated and it shows up in English. The screen reader will try to read that English screen in French, and if there isn't a default means to go up and to convey to the screen reader that the language has changed it won't know how to interpret that text. Did that answer the question?

Hannah: Yes, they didn’t specify anything else. There is one more question that just came in. What themes would you recommend for accessibility especially if you’re getting started with Drupal?

Mike: There’s a couple good themes I would recommend and certainly the Core themes are quite good in this, and has gone through some evaluation. Jeff Burns' is adaptive theme, is the one that we’ve settled on, and that has also brought its way into the web experience toolkit distribution. That’s another great base theme that we recommend. Jeff has been quite involved in the accessibility efforts around Drupal seven. The Zen theme is also a wonderful framework and John Elton has done some terrific work there as well. Particularly, in trying to look at simplifying and streamlining the CSS components, and we are really happy to go up and see the adoption. It’s definitely something that we’re very happy about, happy to see that there’s now a standard, consistent way to be able to programmatically code CSS, which historically is often a bit of a mess.That will have advantages for accessibility, and then also for performance as well. Those are the two main things that I would recommend for accessibility.

Hannah: Okay, great. I think that’s it for questions. Thank you Mike for the great presentation and for answering these questions for everyone. Thank you everyone for attending. Again, the slides and recording on this webinar, will be posted within the next 48 hours and sent to all attendees via email. Thank you.

Increase B2B Engagement Rates with Personalized Web Experiences in Drupal [September 20, 2012]

Pages