Home / Taxonomy term

Webinar

Creating Solid Search Experiences with Drupal [November 13, 2012]

Click to see video transcript

Speaker 1: Hi everyone. Thanks for joining the webinar today. Today’s webinar is Creating Solid Search Experiences with Drupal, with Chris Pliakas, who is the product owner of Acquia Search.

Chris Pliakas: Today’s webinar on creating a search experience with Drupal, I think we’ve done a lot of webinars in the past where we focused on Acquia Search, we focused on some of the basics. Today, I really wanted to focus on just Drupal in general, not having Acquia Search focus. Of course, all these techniques can be used with Acquia Search, but I just wanted to highlight some of the things that are in the community.

Also, based on our experiences hosting over 1,600 indexes, with 1,600 subscriptions, people who are experimenting with search pages and various UX talk about some of the trends that we’re seeing, and in order to get the best experience possible, we wanted to touch on some content strategy items that you can employ to make sure that your search is set up for success.

Then we’ll focus on the search page user interface, so we’ll do a live demo exploring some of the tools that are available to Drupal right now that can be used to create modern-day search user interfaces so that your users get the best experience possible out of the application and could find content that they’re looking for.

Also, we’re going to demo some things that are coming down the pike. I think it’s important to recognize that right now, enterprise search is at a crossroads, and I just want to distinguish for a minute what enterprise search means. When we talk about enterprise search, we’re talking about internal site search, and enterprise doesn’t necessarily mean large corporations. Enterprise simply means that that search is important to your business and important to you, so this isn’t just a big business thing. This is for searches of any size.

But we see some trends that are emerging with external searches, searches like Google, Bing, Yahoo!, that are now going to be expected by users of your internal site search. Trends that are emerging in the search community at large, really, there is going to be an expectation that your search experience matches what’s out there currently. It’s pretty advanced things, so we’ll talk about those trends and we’ll talk about what’s changing in this space specifically.

We talked about search is really evolving. Over the past 10 years or so, which is quite a long period of time, your internal site search really hasn’t been much more than the user entering keywords and then displaying results that are pretty basic. You have a title, you have a snippet, that sort of thing. But really, right now, search is starting to move into a different space where we have to identify what the user is actually looking for and then display relevant results. Relevant results don’t just mean keyword matching, meaning knowing things about your content, knowing things about your user to make some assumptions to present them with relevant results.

As we create more and more content on the Web today, it’s getting harder and harder to sift through that data and display meaningful data. One thing that we’ll start out with is just a simple example.

What I want to start out with is talking about Apple. How many people know Apple? All right, so I see some hands in the webinar. I guess I want to ask “How well do you know Apple?,” so a first question that I want to ask you is, “Is Apple growing?” I’ll let you answer in your heads. It’s not really a good forum for answering in public.

The second question that you should think about is does Apple have money? Then the third question, is Apple multilingual? Does Apple support multiple, does Apple have knowledge of multiple languages? Does Apple speak more than just English? Those are the three questions that I want you to answer in your head.

I’m just going to assume that you guys did a good job and you were able to answer that. Based on those three questions, I think there is no doubt that we’re talking about Apple Martin, who is the daughter of Gwyneth Paltrow and Coldplay lead singer Chris Martin. Apple Martin, like all kids, she is growing. Does she have money? Absolutely. I think her parents are doing pretty well; one is a rockstar; one is an actress. One useful tidbit is that she cannot watch TV in English, so she is getting raised as a multilingual speaker.

Is that the wrong Apple that you were thinking of? I’m assuming that it is. Tech audiences, let me say, Apple, usually think of the company Apple, and the problem really is about context. The first trend that I want to talk about is contextual computing. Right now, we start to see how Apple could mean different things. It could mean the fruit. It could mean the company. It could mean Apple Martin. It could mean Fiona Apple. It could mean a lot of different things.

When I talk about context, I mean the things surrounding it that expose the content for what it is. For example, if we are talking about Apple being a Fortune 5 company or a Fortune 1 company, whatever it is right now, then that context would expose Apple as being a company. If we were on a pop culture website, then it would be more likely that Apple is the daughter of Gwyneth Paltrow, like we mentioned.

Context and how it relates to your content is getting to be really important as we get more and more data. Sites aren’t just displaying one thing now. Sites are starting to display lots of different pieces of content, and we need to start recognizing that simple keyword searches aren’t going to serve our users. We really want our results to be relevant towards what people are actually looking for.

One way that we can do this is by search statistics. Search is a really unique tool in that it is a window to what your users are expecting on your site. By entering keywords and by clicking on various pieces of content, your users are actually telling you what they want from your website, and they are telling you what content they think is relevant.

There are things out there like voting or reputation metrics, but search is really the best tool to be able to extrapolate what people are trying to do with your website.

That also leads into structured data, which is another trend that we’re going to talk about. Structured data is a way to actually denote what type of content you have on your site. Whether, again, we’ll go back to the Apple example, is Apple the organization or Apple that’s something else? These are the three trends that search is really rallying around.

I want to talk about what Drupal is doing right now to address this and some of the things that are going to be coming down the pike within the next six months or so, because it’s important that as you start to build your search experience that you’re starting to recognize some of these trends so that when the Drupal tools emerge, you can make use of them effectively and provide the site search that your users are coming to expect.

Now I’m going to go to the live demo portion of the site just to set the stage here. I have a really basic Drupal install. It’s the standard Drupal blue that you see out of the box, and it has some prepopulated content. It has a couple of events, a couple of blogs. We’ll actually build out some of the search experiences and identify some of the trends that we talked about.

Now that we have the site up, right now, I’m connected to an Apache Solr backend. Again, if you’re connected to Acquia Search or you are connected to Apache Solr, I think there are demos soon that you can install Drupal. You can configure some of the basic modules. You can download, install the modules. We’re going to start with that assumption that that’s the level that we’re at.

If you do need some help or if you are unsure as to how to install modules, how to configure modules, I do recommend that after this webinar, there are some great resources on drupal.org and some great articles that Acquia provides as part of its forums, part of its library that can help ease that transition. But you can still get some value out of this webinar by following along and taking notes of which modules are being used and seeing how you can configure them once they’re installed.

First, what I want to do is I want to just execute a search. It’s the same whether you’re using core search or any other backend. But I’m going to search for DrupalCon, and we’ll start to analyze some of the results to see what the default behavior is that you get out of the box.

The default behavior we’ll see is somewhat useful but not really. But if I entered DrupalCon, it will give me the pieces of content that match that keyword. It will give me a highlighted results snippet, and it will show me a little bit of information in terms of who the user was that posted that content and what date that content was posted. Sometimes, that’s useful. Sometimes, that’s not. But again, this is a basic search interface that you get out of the box.

To be perfectly honest, this isn’t very useful. This isn’t what users expect. If you compare it to Google or Yahoo! or Bing or all the other major players out there, this is weak, and it doesn’t really give users the information that they need to effectively search the content of your site.

The first thing that I want to do is I want to explore something called Facets. And facets are filters that users can apply to help refine the search results, and it also gives some aggregate information such as the count or number of results matching that filter based on the keyword that you entered.

The first module that I want to explore is something called the Facet API module. I’m going to go to the project page here. This is a module that works with core search. It works with Apache Solr search integration. It works with Search API if you’re using that module. It’s a way to configure your search interface regardless of what search backend that you’re using.

If I expand the screenshot here, you’ll see that here are some examples of the types of facets that you can have. You can have facets by content type, by date. There are even some interesting contributed modules out there that allow you to display facets as graphs. You can really control the interface and display things in pretty interesting ways.

I’m just going to scroll down and show some of the things that you can … some of the add-ons that are available that you can make advantage of. Again, we have the graphs that we talked about. We have a slider, so if you have numeric facets, numeric content, you can say, “I want to show data between this range,” tag clouds, and also date facets, which we’ll actually explore and configure.

I’m not going to spend too much time. That’s just an overview to whet your appetite for what’s out there and what’s available in the Drupal community. But I do want to just go and start configuring this so you can see what this looks like and how this works.

The first thing that I want to do is I want to be able to filter this by the content type. I do have two content types here, blog and event, so I want people to say, “Okay, if I’m searching for DrupalCon, I want to filter by the blogs or I want to filter by the events that I want to see,” so that you can get the relevant information for you.

First thing I’m going to go do is configure the Apache Solr Search Integration Module. That’s the one that I’m using. I’m going to go to Apache Solr, going to go to Settings, and I am going to go to Facets. These are the lists of the facets that I have available to me. First thing I’m going to do is configure and enable the content type. I’m going to save this configuration.

Now that facet is saved, I actually have to position it on the page. The default facets are blocks. Blocks in Drupal are small pieces of content that you can position in various regions or various areas on a page. Once you enable a facet, there is a link up top that allows you to go directly to the block configuration page so that you can configure this immediately.

If I click on Blocks and scroll down … it’s actually enabled for me. I’m just going to reset this so that it is where you guys will see it when you start from scratch. But it will start down here in the disabled category. These are all the blocks that are disabled. We look for Facet API, the backend that we’re using, and then content type. This is the facet that we just enabled.

I’m going to position this in the first sidebar. It is recommended that you do position it in the first sidebar, so that will be on the left-hand side. The reason is because that’s where most of the major search engines position their facets, so in order to help people navigate your search page, we use expected patterns. That’s the best place to put it so that they don’t have to hunt around for it.

I’m going to save my block. Now I’m going to go back to my search page. I’m going to search for DrupalCon. Now I have a facet up in the upper left-hand corner that allows me to filter by events or by blogs. If I filter by blogs, it’s reporting that I have two results. If I click that, you’ll see that I do get my results filtered to the blog that I want. That’s pretty basic stuff, but it allows your users to actually target what they’re looking for.

The next thing that I want to discuss, this is very basic, facet configuration. The next thing that I want to discuss is a pattern called progressive disclosure. This is something that you’ll see on Amazon where if you go to Amazon’s search, you’ll see that you’ll be prompted to search for something that you’re interested in, whether it’s one of the products that they have. Then when you search on that product, you’ll be displayed different filters based on the different types of things that are returned. What it prompts a user to do is start out small, like selecting the department that they want to search in, and then based on that department, it will expose different filters or facets that are relevant just to that.

I do want to take a step back and talk about the events. The events that I have on this site have dates that are associated with them, so the date that the event actually starts, whereas the blogs have a different type of date. They have the date that the article was posted.

When you’re searching for events, you don’t really want to know the date that the event was posted. You want to know the time that the event is actually happening, so you’re going to have two different types of date facets, depending on the content that you’re targeting.

Instead of displaying all of that information, all the possible combinations of facets on the left-hand side, we want to only display the facets as we start to navigate down the content types that we’re interested in.

To highlight this, I’m going to go back to the configuration page, and I’m going to go to Apache Solr, and I’m going to go to Settings, configure my facets, and I’m going to scroll down. We’re going to see two types of date facets that I was referring to. One was the post date and one was the event date. I’m going to enable both of these. I’m going to go to my blocks, position them. I’m going to scroll down, and now I see that the new blocks are here and disabled, so I’m going to position them in the sidebar first, like the other. I’m going to make sure that they’re in the correct order that I expect.

I’m going to save these blocks. I’m going to go back to my search page, search for DrupalCon. You see that by default, now I have filter by post date, filter by event date. In order to configure this progressive disclosure pattern, what we’re going to do is leverage something in Facet API called Dependencies. Instead of just explaining, I’m just going to go for it and highlight by example.

When I mouse over the facet, I get a little gear in the upper right-hand corner. If I expand that, I have an option to configure facet dependencies. This is the date that the actual content was posted, so again, it makes more sense for the blog than it does for the event. The first option that I have here is bundles, which are synonymous with Drupal content types. I’m going to say at least one of the selected bundles must be active. I’m going to say I only want to show this for blogs. I’m going to save this and go back to the search page.

Now you see that that date facet is gone. If I click on blog, now it appears. Now filter by post date. Again, I’m only shown, I’m only displayed facets that are relevant to the content type that I’m looking for.

Again, I could do this filter by event date. Again, mouse over the gear. Click Configure facet dependencies, Bundles. At least one of the bundles is active, and I’m going to say Events.

Now I go back, and when I search for DrupalCon, I’m going to start off very small, limited options, kind of guiding your users to select something and refine their results. As I click on blog, we know that we’re in the blog context, so again, context meaning information that is used to determine what type of content you’re viewing. Now that I know that I’m viewing blogs, I see the post date, which is a little more interesting.

Whereas if I click on the events, now I get the filter by event date. I can say, “Show me events that start in August of 2012 or May of 2013.” It’s not going to really target the type of events that are relevant to me.

One thing too, I’m actually going to go back to the blog facets, you see here that for the blogs, we have this drilled down thing that starts … we have a couple of blogs that span a couple of years, and the default facet that’s coming out of the box, you have to actually drill down to 2011. Now I’m going to go in March. Now I’m going to go March 21st. It allows you to drill down by the specific date all the way down to the time. But that’s actually not what users expect when you’re dealing with types of displays that are blogs, that sort of thing.

I’m actually going to go to Google and search for Drupal blogs. If I click on Search Tools, we’ll see anytime they don’t have that type of drill-down. They actually have the ability to refine by a certain range. That’s usually what users expect, and that’s a use case that people commonly ask for that we’ve seen in our support requests.

The next module that I want to explore is called the Date Facets Module. Again, this is available on drupal.org, date_facets. This can be linked to by the Facet API project page. But again, if we look at the screenshot, we’ll see that it provides a nice little display widget that allows you to display your facet in the range selection. We’re going to assume that that module was downloaded.

Click on Modules. Once you download that module, you’re going to install it. I’m using the module Filter Module to provide this nice interface where I can make sense of my modules because anybody that builds Drupal sites know that you can get up to hundreds of modules, so you need to be able to filter them more easily in this Module Administration Page. All I have to do, I already enabled this, but if I select the check box, click Save configuration, that’s all I need to do to install the module.

Once the module is installed, actually, I’ll do this from the search page, again, filter by blog, you have an option with facets to configure the display. If I mouse over the gear and click it, same list of options that allow me to configure the facet dependencies that can configure the facet display.

After I’ve installed that module, I’m going to have a new display widget site. If I expand here, you can see up at the top there is a new date range widget. The type of display in Facet API is called the widget.

If I click on date range, I’m going to click Save and go back to search page, I’m actually going to get an arrow here, which I wanted to highlight purposely. It says the widget does not support the date query we typed. When you’re doing the date range, this is a common error that people report. You have to actually scroll down and select the different query type. This just tells Drupal that we’re not just doing the date filter. We’re doing the actual ranges.

I don’t want to get into the technical aspects of it, but behind the scenes, it actually changes the type of filter that the backend uses, so it’s important that we actually make this distinction.

Now if I save and go back to the search page, now you see that I get filters that are very similar to Google. I can refine things by the past week, which I have nothing, or past month, past year. It looks like I only have stuff within the past year. But it was able to refine that based on the time range of the content that you have, so it really allows people to narrow down the things that are more recent.

Those are a couple of the tips that I wanted to share regarding the fast configuration, but I want to stop and see if there any questions before proceeding. Do you have any questions? All right. We’ll move on from facets.

The next thing that I think is pretty interesting is that instead of having a unified search page which displays all the content across your entire site, sometimes it’s useful to actually have targeted search pages. These are things like, okay, I have a blog section on my site, which we have here. I only want to search across the blogs or I don’t want to make the user actually click on blog to refine the results. This can actually be done in the Apache Solr Search Integration Module, which we’re going to focus on.

I’m going to click Configuration then go to Apache Solr Search. One thing that I’m going to do to simplify this demonstration and something that I think is useful in Drupal in general is Drupal 7 provides this nice little shortcut functionality. You see here I have Apache Solr Search with a little plus sign. I can click this and it will now add this configuration page, a link to this configuration page in the toolbar so that I can navigate to it more quickly as opposed to having to go through the normal path. I’m going to do that for an easier demonstration. If you’re configuring your search pages, you might want to do that as well.

Some of the tabs here, we have one that’s geared towards pages and blocks. I’m just going to select pages and blocks. This is where we can actually manage search pages. I’m just going to go ahead and add a search page and we can see what this will do.

The goal here again is to create a search page that just narrows down your blogs. I’m going to say this is a blog search. I’m going to scroll down. I’m going to make sure that my correct environment is selected. In this case, I’m running Solr locally, but if you’re connected to Acquia Search, you’ll have an environment for Acquia Search. Environment is really named for the backend that you’re connecting to.

Again, in title, search blogs. That’s going to be the title of the page. The path, I’m going to put in search/blogs. The part that’s going to allow me to filter just by blog content is this part at the bottom, custom filter. It’s a little complex in terms of how you do it, but first, I’m going to select that custom filter check box to make sure that I’m using a filter. We’re going to read the description down here. It says, “A comma-separated list of lucene filter queries to apply by default.”

In English, what that means is lucene is a very low-level search engine that Solr is built on, but it’s a syntax that allows you to filter by specific things and do some pretty interesting stuff. The very basic part of lucene syntax is if you want to filter by field, it will be the field name, field and then colon value. We have this use case actually down here in the comments. We see here bundle:blog. Bundle is the actual name of the Solr field, and blog is the name, is the value that’s actually stored in the index.

If you want to see all the fields that are stored in Solr, you can actually click Reports and click on Apache Solr Search Index. These are all the different field names that you have at your disposal. It doesn’t show you the values, but in our case, we know that the bundle will index the machine-readable name as we specified when we created that content type. If I go to structure content types, we see here all the different machine names. Blog is just the machine name, with _blog.

Again, I’m going to match the Solr field to this machine name. I’m going to say bundle is the name of my Solr field, and then blog is the value that we want to filter by. I’m going to save this page. Now, I have a search page that’s dedicated just to blogs. I’m going to click on this. If I say DrupalCon, now we see that it only gives me two results because it’s only filtering by the blogs, not filtering by any of the events.

Sometimes, it is nice to have these targeted searches. For example, if you do have a blog section of your site, it is very nice so that you don’t have to actually set up a separate site for your blog. You can have your blog be a micro-site that is under the same Drupal installation but just has different configurations isolating that content so users can find what they are looking for.

I want to stop there and see if there are any questions on the search pages. No? We’re good? Okay. I’m actually, just to reduce the noise here, I’m going to disable … is there a question?

Speaker 1: Yes. Is there an autocomplete module?

Chris Pliakas: Yes, there is. Let’s see if I can find it. Yes. The module name is aptly named Apache Solr Autocomplete. The project name is Apache Solr_autocomplete. This will provide the type of autocomplete functionality that people are used to.

Now, it is important to note that, and this is one of the trends that we’re going to talk about, that this actually pulls off your index and does keyword matching. But as you have larger sites and more data, then sometimes, keyword matching isn’t necessarily the best option to guide people towards relevant results. There is a trend that’s going to match statistics as well so that you can actually autocomplete based on what people are searching for as opposed to just the keywords which theoretically will guide them towards more relevant results. As I talk about the Apache Solr Statistics stuff that we’re doing, we’ll relate that back to the autocomplete.

Speaker 1: We have a few more questions.

Chris Pliakas: Okay.

Speaker 1: Can that custom search be put in a block?

Chris Pliakas: Can this search be put in a block? Yes, I believe it can. Let me just search for a module. I believe there is a module that does this. I want to see if this is what it does. I might have to get back to you on that one. I believe there is actually a module that does allow you to expose your searches in a block, but I’m not 100 percent sure on that, and so I’ll take that as an action item and post that answer after the webinar is over.

Speaker 1: Okay. Also about the statistics stuff, is that available now?

Chris Pliakas: Yes. There is an Apache Solr Statistics module that does some very basic stuff, but it’s more geared towards administrators. It does things like the keywords, but it does so more or less how many times a search page is viewed, which isn’t really that useful to site builders. But there is a new extension to that module, a new branch, I guess I should say, that is available on the community. I’ll show you where it exists and I’ll give you a bit of timeline about when that is going to get merged back in, but that’s more geared towards site builders and talks about how people are actually using your search.

Speaker 1: We have a few more questions.

Chris Pliakas: All right. I’ll take it …

Speaker 1: All of this work with non-Drupal content if some other system populates parts of the Solr index?

Chris Pliakas: The answer to that is yes. The trick is getting that data into Drupal. There are some example code, which we’ll point to the links after the webinar, that allow for more easily getting content into Drupal. But once you get the content in, you can display facets and that sort of thing.

The display of the search result doesn’t really bias towards what type of content it is. Again, it’s more or less just getting that content into Apache Solr in a way that Drupal can recognize.

Speaker 1: We have one more. Where is the extension to have autocomplete?

Chris Pliakas: Again, that’s the … we’ll do it for Google. If you search “Drupal Apache Solr Autocomplete,” I’m going to venture that it is one of the first results. It’s on drupal.org. The URL is drupal.org/project/apachesolr, all one word, _autocomplete. It’s pretty easy to find on drupal.org and it’s available on this project page.

Okay. I’m just going to clear cache just to make sure that our stuff is gone. I’m actually going to go back to Google here.

If we look at Google, we see that the search results are displayed in a format that’s pretty familiar to us. Let’s go to Yahoo!, or let’s go to Bing. Search for Drupal.

Now pretty interesting, you’ll actually see that the search results are very similar. You have the title. You have the URL. You have the snippet, and you have some additional information about it. Third thing, go to Yahoo!, search for Drupal, and we’ll see that again, different results are returned because they have different algorithms that determine the relevancy, but the display is very, very similar. The reason is because there is actually a lot of standardization that was done in 2011 by Google, by Bing, and by Yahoo! What that is is something called schema.org.

Let’s go back to Google, and we’ll look at the search results. Let’s go to our blog. We see some interesting things here. We see that when we search for our schema.org blog, you scroll down, we see one of these results has an image. This is actually a great way to talk about schema.org in that it provides some structure around your data.

When we build content types and manage fields inside of Drupal, we’re actually just configuring the data model, so that’s the underlying buckets that we put data into, and it doesn’t really have any meaning beyond what we name it. Google doesn’t understand when you create a blog content type that that’s actually blog content. It’s only blog in name only. Or when you create an event content type, it’s only events in name only. That’s almost like Drupal provides you a leg up in that you don’t have to build your database but that you can do it through the UI. But I’ll actually go back to Drupal here, click Structure, click Content Types.

You see here that I have events. If I manage my fields and I added some extra data here, the date, the event date, an address, an image, if I wanted to add another field, what you do is you create your label and then you select the type of field that you want. We see we have date, file, we have text. This is all real basic stuff that again is just really low level and doesn’t actually expose what type of content that is.

Schema.org is the layer that sits upon that which says, okay, this text field is actually an address, or this image is the primary image of this piece of content, or this event date is the actual start date of an event. It will actually go up as well and say, okay, you can say this content type event is actually an event so that it can be recognized by some standard that’s out there that’s agreed upon by the major search engines.

This actually helps your Drupal site by not only when Google and Bing and stuff index your site, it will actually read this metadata, but there is actually some work that’s being done so that it can modify the display of your internal search so that users are presented with a familiar experience.

That’s probably the thing that people will recognize the most, but the module that I want to share with you is called the Rich Snippets module. We’ll actually just install it and see what it gives us out of the box. Again, Rich Snippets, rich_snippets. There is another module that’s similarly named, but it’s important to understand that this one is geared towards your internal site search.

This takes that schema.org metadata and actually will format your results accordingly. I’m just going to install this module and see what it gives us, and then we can break it down a little bit.

Again, I’m going to go to Modules. I’m going to go to Rich Snippets, enable this. I’m going to bring up a page here so that we can see what it looks like before. Again, very blah. Now, when I enable the Rich Snippets module, we go back to my search page. I’m going to refresh the page. Now you see that it displays the results very, very differently.

The goal of this module is to work fairly out of the box. With Solr, you might have to re-index your content. But as you can see, now the results are displayed in a way that’s much more friendly and much more in line with what users expect.

As a nice UI tip, this module is going to emerge as something that’s going to be a staple on sites with search. As you can see, for DrupalCon Portland, DrupalCon Munich, it displays a little image, and it also displays the start date.

Now, for the blogs, it displays who that blog is by and when it was posted. As you can see, based on the context or based on the schema that we’ve assigned to it, the search results are displayed very differently. This is really important when we’re displaying site-wide searches. There are tools in Drupal, such as Views, which people are starting to explore to build their search pages on, but that’s not really geared towards heterogeneous content.

When you have a mix of contents, then it’s really important that you’re able to display that effectively inside your search page. Whereas views, it gets really, really tricky to say, “Okay, for this content type, display it this way. For this content type or this schema, display it another way.”

That’s the first thing that the Rich Snippets module will give us, is a nice display. Now we’ll talk about how to actually say, okay, this is a date, this is the start date, that sort of thing.

There is a module called schema.org. It’s just schemaorg, one word. It’s a very simple module that doesn’t require a lot of configuration, but you effectively download it, install it, and it allows you to effectively tag your fields and your content with the type of schema that denotes what that content actually is.

If you download and install this module, what it does is pretty simple. If I go to my structure, go to my content types, edit my content type, it gives us this new vertical tab that says schema.org settings, and this allows us to actually specify what type of content this is.

If I said, okay, this is a blog, I could start typing, and it would give me the options that are available. All the options are on the schema.org website, and I’m not going to go over them in detail because there is a lot of them. Just to give you an idea of how much there are, you start off with your basic top level stuff like an event, organization, that sort of thing, and then inside of these have various properties that say, okay, for this event, this is the end date, these are the attendees, so a lot of structured information there.

Each one has a lot. Let’s see if I can get the documentation here. Okay. That’s not what I want to show. Full list. Again, this highlights why this is a great tool for this type of search results display because as we scroll down, this is the nested hierarchy of schema.org schema and properties, so you could see there is a ton of them. The module right now supports a subset of them but it’s going to support more.

As we’re building our content, it’s really important that you use this module and explain what your fields are. When I actually create a field, if I click Edit and I scroll down to the edit settings, you see here that I also have schema.org mapping so I can say the property. I could say this is the start date. Then what the Rich Snippets module will do is based on your schema and properties, it will display your content differently.

Because this is start date, if I go back to my DrupalCon settings, then it knows to display the actual start date up here because based on this result being an event, it’s probably what people are going to be interested in, so it gives them some context about the content that’s being returned so that they can see what’s going on without necessarily having to click on the piece of content itself.

I’m going to stop there and take a couple of questions for two minutes, and then we’re going to move on to statistics and then stop for general questions.

Speaker 1: Okay. We have two questions. Can the custom Solr search results page be used in panels? This might be from the last section.

Chris Pliakas: Yes, I believe it can be. The reason why I say that is because the Acquia Commons distribution is making heavy use of panels and is using Apache Solr for its search engine. I say with confidence that yes, it can be used with panels.

Speaker 1: There is one more. Where is the extension to add to Apache Solr autocomplete which allows for statistics to be involved and not just keywords?

Chris Pliakas: That’s one thing that’s not available just yet, but it’s on the road map for the statistics module that we’re going to display next. This is one of those items where I wanted to make people aware of the different trends that are emerging. This is one case where it hasn’t been implemented yet, but it’s going to be implemented. As you start to look forward in your search solutions the next three, six months, look for this as an option.

Speaker 1: We have one more question. Why do we need Acquia Search when everything seems doable from Drupal Search?

Chris Pliakas: Yes, and that’s a great question. The first thing is that Drupal Search won’t scale. The Drupal is built on relational database technology, and relational databases simply won’t scale for full text searching. They’re really geared towards saying, okay, find me all blogs or find me all users, that sort of thing. But when you start to enter keywords into the mix, it will take your entire site down pretty quickly because it will bog your stuff down.

Regarding Acquia Search, you can run Solr locally, and we’ve contributed a lot of these add-ons back to the community. However, the value add that Acquia provides right now is that we have Solr configured in a highly available cluster, so there is a master/slave replication so that if one server goes down that end users can continue to search. We also integrate the tools that allow for file attachment indexing. We also have a security mechanism that we’ve applied on top of Solr.

Solr actually doesn’t have security out of the box, so you can actually do a Google search and find a lot of Solr instances that are unprotected. You could delete that index. You could add content to that index. We’ve added a security on top of Solr that allows you to connect securely and make sure that you and only you have access to your server. Also, we manage it 24x7.

One of the things I do want to talk about going down as we talk about statistics and contextual computing, there are things that we’re experimenting with Acquia Search that will adjust relevancy based on user actions. This will be a set of tools that integrate with Drupal and integrate with various tools that will provide more relevant results to your users beyond just keywords. There is going to be a lot of value and a lot of focus on contextual computing with Acquia Search that’s really going to differentiate it from not only core search but from using Solr locally.

Speaker 1: There’s a few more, but we can get to them at the end.

Chris Pliakas: Yes, sure. What I’m going to do is just wrap up really quickly with the statistics. There is one point that I want to hit home, and I’ll try to stop by 1:55 to save some time for some questions afterwards.

There is an Apache Solr Statistics module. Let me clear out some of these tabs here. I think that’s it. Or maybe it’s Apache Solr Stats. It’s probably Apache Solr Stats. There we go.

There is an Apache Solr Statistics module that you can download, it works for Drupal 6 and Drupal 7, that gives you some information in terms of how many requests there are, what type of things people are searching for, but it’s more geared towards site administrators, not necessarily search page builders. The reason why I say that is because if I go to my search and I search DrupalCon, it’s going to count that as DrupalCon, the keyword being searched.

If I click on events, since the page reloaded and it actually queried Solr again, that statistics module is going to say, okay, DrupalCon was searched again. What this really does is it says show me content where people have to click around to find what they are looking for. It’s not necessarily indicative of what people are actually looking for on the site.

One of the branches that’s being worked on, it’s actually a sandbox project right now that will be merged into, back into the Apache Solr Statistics module by Q1 of next year … I can’t find it here … there is a sandbox that’s an Apache Solr Statistics fork that’s used to experiment with this stuff. That’s what I’m going to be showing you today. The important thing is that it’s more geared towards the search page builders, and it also tracks what people do after they search for something. It allows you to track what we call click-throughs.

If somebody searches for DrupalCon, we can see what pieces of content people are actually selecting, so we can make informed decisions about how to configure our search and how to modify the relevancy.

What I’m going to do is click on modules, search toolkit, and enable Apache Solr Statistics. When I click on Apache Solr Search, now I have a new tab that says statistics.

What I want to do is I want to enable the query log. This captures stuff about what searches are being executed. Also, I want to enable something called the Event Log. In order to enable this, you have to copy a file from the module to your Drupal Group so that it can capture the information as users are clicking on it.

We’re also going to capture user data. By default, that’s off, but you can capture data not only what people are searching for but who is searching for it. Based on your privacy policy, you can enable or disable that setting.

There is also what I’m about to explain, the law of retention policy and backend, by default, logged to the database, but for busier sites, again, there is going to be the availability to send that to different sources.

I’m going to say a configuration, and I’m going to execute another search. If I search for DrupalCon and now click on DrupalCon Portland, if I go to Reports, Apache Solr Index, Statistics, this gives me some interesting things. It gives me the top keywords, so it shows me what the top keywords are that people are actually searching for. Equally as important, top keywords with no results, so you can see what people are searching for and not getting any results for.

If people don’t find the content they’re looking for, they’re going to leave your site, so this is a really important metric. Also top keywords with no click-throughs, so if people are searching for things and they’re getting results but they’re not clicking on anything, then there is probably going to be some modifications to make sure that they’re getting displayed the correct results.

Here, we see the top keywords. We also have click-through reports. If I click on that, it will show me the pieces of content that people are selecting in the count. As you start to gain some more traffic on your site, this will give you some transparency in terms of what people are doing on your search page, and more importantly, what they’re doing after.

As we talked about the contextual computing, it’s really important that you monitor what people are looking for, and this is a great way to do it. Again, it’s what people are looking for in your site and what they are selecting, what they find relevant. The search page is a great tool to help you modify your experience and tailor it to your users.

We have a couple of slides to end up, but that’s really what I wanted to highlight, is that contextual computing is more the trend, that there are some tools that you can employ now that are going to be improved upon in the future to make sure that Drupal is the best solution available in search to serve relevant content to your users. Search is really becoming a big data problem, and search is also becoming a solution to that problem.

Big data is capturing a lot of information and then making sense of it, doing something with it. As your sites begin to amass a lot of data, search is a great tool to help your users sift through that data and find the relevant content that they’re looking for, and that’s really where the trend of computing is going over the next five years, so definitely pay attention to search as a tool to help make sure your site is keeping up with the latest trends and desires of your end users as they look for engaging experiences.

I went over but we’ll take some more questions.

Speaker 1: Okay, great. Would you recommend using these modules on a Drupal 6 site using domain access?

Chris Pliakas: Domain access is a little bit tricky especially with search. Some of these things are … let’s take a step back. The way a domain access works is that it builds upon the Drupal node access system, so that adds some challenges in terms of search. Not only does a search solution have to be domain-access-aware, but everything around your site has to be domain-access-aware.

Theoretically, you can use your Drupal 6 site with domain access. It’s just that it gets a little bit tricky because your index is logically separated as opposed to physically separated, so there always is the chance of your content either lagging behind in terms of getting that access information or accidentally getting exposed to other sites when it shouldn’t be, so it can be done, but there has to be a lot of thought and a lot of careful planning to make sure that it’s implemented properly.

Speaker 1: The next question is does the schema.org also expose the extra info to search engine spiders?

Chris Pliakas: Yes, it does. That’s actually what the module is geared towards. It’s geared towards the external use case, and it works, it provides that metadata that Google will pick up the images and the additional metadata. But what the Rich Snippets module does is it takes that information and uses it inwards. By default, it actually is geared more outwards, but the work that’s being done right now is taking that and also apply it to your site search, so it’s a win-win.

Speaker 1: The next question is what if the non-Drupal contents are dynamic pages, how do you import those contexts? If not, is there a federated search solution?

Chris Pliakas: I think it’s important to first say a federated search solution might not be exactly what’s being asked for. When we think offederated search solution, we think of things like Kayak or other engines that actually query out different data sources and compile them together.
There are tools in Drupal that allow you to query different sources simultaneously. However, that’s probably not what you’re looking for. You’re probably looking for a unified search solution that displays results instantly.

In order to do that, you can leverage tools such as crawlers, such as Nutch, which will integrate with Solr. The key is again getting that data into a format that Drupal can recognize. But the trick is using those tools to crawl or expose your external data to get them into Drupal.

There are also ways that you can programmatically connect a third party data store and index that into Drupal using the APIs. But again, it’s more of a developer task and something that has to be coded.

But with Acquia Search, definitely look for an offering sooner rather than later to index external content and bring it into your Acquia Search Index.

Speaker 1: All right. We’ll take one more. How can you make information more important based on the statistics? What ways to set this up are available?

Chris Pliakas: Can you repeat that question one more time? Sorry.

Speaker 1: How can you make information more important based on the statistics? What ways to set this up are available?

Chris Pliakas: Sure. I’ll give one example from Acquia.com. We have an offering called Dev Desktop, which is a local stack installer for Drupal. A long time ago, it used to be called DAMP, Drupal, Apache, MySQL, that sort of thing. What we actually have noticed is that based on our statistics, people still search for DAMP more than they do Dev Desktop. We noticed that trend, and the way that we modified our search results was to take advantage of some of the things that Apache Solr has, and when people search for DAMP, we add a synonym to Dev Desktop so that when they search for DAMP, they’re actually getting the content that’s relevant to Dev Desktop, which is what the products mean now.

This is what Google does. This is why Google results are very relevant. They have hundreds of full-time engineers analyzing their search and doing things like saying, “Okay. If you search for a FedEx tracking number, we’re going to show you the FedEx webpage.” Now it’s automated, but that was used by analyzing the statistics, and those are the types of techniques that you can employ on your site based on what your users are actually looking to do.

Speaker 1: Okay, great. I think we’re going to have to end it here. Thank you, Chris, for the great presentation, and thank you everyone for participating and asking all these wonderful questions. Again, the slides and recording of the webinar will be posted to the Acquia.com website in the next 48 hours. Thank you.

Visualizing and Solving Drupal Performance Issues [November 8, 2012]

Click to see video transcript

Jess Iandiorio: With that we will get to today’s content. Again, I'm Jess Iandiorio and I do product marketing for both Acquia Network and Acquia Cloud. I’d like to introduce Dan Kubrick, who is my co-presenter. Dan, if you could say hi?

Dan Kubrick: Hi, everybody. Thanks for joining us.

Jess Iandiorio: Great. I'm going to go through a little bit of information upfront about the Acquia Network, for those of you who aren’t aware, and then I’ll turn it over to Dan, who’s going to do the bulk of the presentation, as well as a great demonstration for you.

For those of you who received the invitation through us, you probably heard the heads up, that Trace View has joined the Acquia Network. We’re really excited about it. Those of you who don’t know what the Acquia Network is, it’s Acquia’s subscription service, where you can obtain support tools to help your Drupal site perform better, as well as access to our extensive knowledge library.

The library, we like to think of it, has the answers you need to all of your burning Drupal questions. There are about 800 articles, a thousand Frequently Asked Questions, a really extensive library of podcasts, webinars and videos. We do have a couple of partnerships with drupalize.me through Lullabots, as well as Build a Module for other training resources that you can get access to.

In terms of our support team, we have a 24/7 Safety Net and our support offering follows the sun, so wherever you’re located, you’ll have a local resource that can respond to your request. We also perform remote administration, which means for customers, we can go in and make Drupal module updates for you, as well as security patches. We have about 60 people across the world on our Drupal support team, so the best concentration of really high quality Drupal talent you can find, if you do happen to have Drupal supp
ort needs. We encourage you to learn more about that on our Web site.

The last area that Acquia Network provides is all the tools, and we refer to it as the Acquia Network Marketplace. Some of the tools we built ourselves, like Acquia Insight. If you’re not familiar, it’s about 130 tests we run proactively against your Drupal code and configuration to tell you how your site’s performing across security, overall performance, as well as Drupal best practice implementation. It’s a really great tool that customers use, probably on a daily basis, to get them a to-do list to figure out how they can enhance their site.

SEO Grader is a very similar tool that we built with our partner Volacci, and it has the same UI as Insight. You get a score, you get proactive alerts for tests that have tasked and failed, recommendations for fixing. It’s just looking at different criteria than Insight does. It’s looking at the things that help improve your site’s search engine optimization.

Acquia Search is our hosted version of Lucene Solr. That’s the webinar that we have next week. If you want to learn more about that, please feel free to sign up. On the right-hand side, we get to third-party tools that our great partners provide to the Acquia Network customer base. I mentioned drupalize.me and Build a Module already, and those are tools that are helping you learn more about Drupal.

When it comes to optimizing your site, we have a variety of other partnerships with Blitz and Blaze Meter for load testing, Yottaa for site speed optimization and Trace View and New Relic for looking at your application, and actually taking you through the stack and figuring out other areas for performance enhancement, and that’s what you’re going to hear about today from Trace View.

Lastly, we have partnerships that help you extend the value of your site. Oftentimes, these are most valuable to a marketing audience, but it could be to someone technical as well. Mollom, for instance, is spam blocking. The technical person would implement it, but at the end of the day, the marketing person typically cares about spam and how it could harm your site and your brand.

Visual Website Optimizer is A/B testing when you want to figure out whether one, promotion or call to action on your Web site performs better than another. Chartbeat is real-time analytics, trying to figure out where are your site visitors coming from and what are they engaging with on your site. Really great, easy-to-use tool, similar to Google Analytics, a little bit more of a focus on social activity and where people come and what their social behavior is.

Lingotek is translation/localization services, so you can work with Lingotek to bring your site into a new geography, localize the content and they have a variety of different ways that you can work with them. You can have a machine translate, you can tap into an extensive community of people who can help with translation or you can actually have a moderator that you can hire through Lingotek, to watch all of the translation process and ensure its success.

That’s a really quick overview of the Acquia Network. I’ll be on the webinar the whole time, monitoring Q&A and able to take questions at the end, but at this point I would love to turn it over to Dan for the remainder of the presentation. Dan …

Dan Kubrick: … that introduction, Jess. Again, I'm Dan Kubrick, one of the co-founders of Trace View, and what we provide is a SaaS-based performance management solution for Web applications. It’s part of being included in Acquia Network. We’re actually providing a free 60-day trial of our best day plan. You can sign up with no credit card required and try it out, but I'm going to talk a little bit more about what we actually do and show you a quick demo, and then you can see if you want to sign up after that. Without further ado … can you see my screen all right, Jess? Great.

Thanks again for tuning in. What is Trace View? As I just mentioned, we provide a SaaS-delivered performance insight service for PHP-based Web applications. We also support Ruby, Python and Java, but we’re really excited to work with the Acquia Network, and one of the reasons that they selected us to come onboard is because of our particularly strong Drupal insights.

That comes from a combination of our work, as well as community support in the form of a Drupal module that provides very deep insights. I’ll get into that in a minute. The types of users that really enjoy and take advantage of Trace View are developers, people in Ops for Web applications, support and also engineering management.

The goal of Trace View is to provide low overhead performance management and insights for production environments. What this means is, you have a Web application, I'm sure you’ve run into problems, as I have in the past, where there’s something that either due to production load, production hardware or production datasets, it’s very different performance-wise, in terms of throwing errors or whatever from development, and because users are really perceiving the performance of your production Web application, you need to be monitoring that all the time.

Trace View provides a very low overhead solution for providing extremely deep insights continuously in real time for your application. Our secret sauce that differentiates a little bit from other APM solutions is what we call full-stack application tracing. Basically, what this means, I’ll dig into it in a second, is that we’re watching the request from the moment it leaves your user's browser as it goes through the Web server, the application layer, out to the database and caches and ultimately return to HTML that then gets rendered and parsed in the user's browser. This provides the true end-user experience, as well as great diagnostic detail to get into what's actually going on in your application.

Finally, we take this data and put it into a slice-and-dice interface that’s really designed to provide the most actionable and clear insights for your performance data, and that means going beyond averages into very precise distributions, helping finding outliers, slow queries and ultimately, down to the line of code within request.

How does this all work? Let’s take a look at full stack application tracing for a minute. What we’re going to be getting in the browser is the network latency for communication between the browser and your Web servers, the time it takes to process the DOM, the elements in the HTML is returned, and finally to fully render the page and all of the things that go on with it, up until it’s document-ready.

On the server side of this, be that virtual or physical hardware, we can start looking at the request, starting at the load balancer or the Web server to see, are the requests getting queued up before they hit the application backend?

What’s our app layer performance like? What end points in the code are particular hotspots? Are we throwing exceptions, if so, from where? How are we using the database? What queries are being run? Are they taking a long time? Cache performance, how many cache requests are we making per page load? What’s our hit rate? Finally, down to the actual hardware underlying all of it, what's the I/O latency like? Do we have excessive CPU utilization or are we barely touching the cores at all? Taking all these data and providing not only visualizations and insights, but also proactive alerts based on it.

To make this a little bit more concrete, let’s look at an example of a Web application you might be familiar with. This is a simple LAMP stack, Apache, PHP and MySQL and I’ve also added in memcached and an external API, maybe you’re using Mollom, maybe it’s your payment gateway, whatever else. As a request comes into this system, it makes requests down to the different components of your application, calling out to memcache. Perhaps it’s a cache miss, so you go to the database and pull back some results out to the API and ultimately, you return HTML to a user.

After installing just a couple of debs or RPMs, which is the form of installation for Trace View, or actually a single click if you’re hosted by Acquia, we can put instrumentation at various points throughout your application, requiring no code modification, that reports data back to us in the cloud in real time. The cool thing about our instrumentation is how lightweight it is. A tunable fraction of request coming into the top level of your stack is selected for tracing.

At each point that’s interesting. We fire off a UBP packet non-blocking to a daemon running on local host. This daemon does the work of forwarding over a secure channel to us, and what this means is that throughout the request path of the request your application is serving, there’s actually no blocking calls, and so there’s no chance for the request to get held up. Additionally, the overhead is completely configurable. In production environments for our customers, we see one percent or less overhead from this tracing that’s, at the same time, providing very deep application insights.

On the quiet side, we gather data through a technique known as JavaScript injection. You might be familiar with this from stuff like Steve Souders’ Episodes or Yahoo’s Boomerang. Essentially, a small JavaScript snippet is inserted in the templates automatically, and this performs very similarly to the Google Analytics beacon. It’s fired up asynchronously from the user’s browser causing no overhead, and reports back statistics that we can use to figure out how the request performed from the browser’s point of view, how requests are performing in different parts of the world or the country, and which browsers are performing differently from others.

The final thing I should mention here is that our insight isn’t really limited to the interactions between components and your stack. Though we can start observing the browser and proceed through the load balancer and so on, there’s actually a great deal of Drupal internal insight that we’re able to provide, and this is largely thanks to a custom Drupal module that’s available on drupal.org. What you’re going to get from that is being able to break down your performance by menu item.

For instance, if you have many URLs that really map to the same underlying code path, you might want to look at particular URLs or you might want to look at that code path within your application. Being able to filter on menu item is pretty interesting. I’ll show all these in a minute.

The second interesting piece of functionality is the ability to partition your performance data by the type of user. Oftentimes, the same code path will exhibit different characteristics for authenticated versus anonymous users, depending on how many customizations there are on the page. There may be administrative pages that are slower or you don’t care about the performance of, and the module also picks up worked on by Drush, and so it’s nice to be able to filter out all of those populations separately in terms of performance data, so you can optimize what you really care about.

In terms of the Drupal internals, there’s two interesting things. The first one is time spent in Drupal hooks. You can see in hooking nits and watchdog and so on, really how your time is being spent throughout the stack as well as viewing individual node loads during the processing of requests. This module is very cool, and probably the best way to explain what's really going on here is to dive into a quick demo.

What we’re looking at right now is the performance overview for several different environments in one of our customers who’s been generous enough to share their data with us today. The company is All Players and they do product for groups to track and engage in common activities. We’re going to dive into the staging environment here and look at the data for the past 24 hours.

What we’re looking at is the performance of average requests, broken down by time spent in each layer of the stack over the past 24 hours. We can see that, on average, we’re spending a fair amount of time processing requesting PHPs as well as in the database through these two separate interfaces. Additionally, Apaches, our Web server and Nginx lowdowns are on top. Courtesy of the Drupal module, we’re also getting insight into the time spent in various different Drupal internals and we’ll dive into this a little bit more in a minute. We can see that on average, it looks like PHPs MySQL calls are contributing a lot to the latency of our application here.

In addition to just figuring out the time that’s spent on each layer of the stack felt, Trace View is also pulling out interesting features of the requests themselves. For instance, the domain and URLs are being requested. The menu items and the menu item parameters that are being requested, so they go pass through the application. Cache performance, in this case, because of the staging environment, we can see that our hit ratio is not very good. The traffic partitions that I was just mentioning, as well as the queries and RPT calls that may be expensive to make in your app.

Now, all of these tables are filters for our data, so if you wanted to see the performance of a particular endpoint here, in this case, it looks like our rest API, we can select that and we've now filtered the data, so we’re only looking at the performance here. We can see that for this particular click path, it looks like there’s a lot of query latency on average and in fact, here’s the top two queries that are usually coming out of here. It’s almost exclusively accessed by authenticated users as well.

Now, here’s all these data coming from? We’ve been looking at aggregate data, but I mentioned our unique data source the trace, so I'm going to switch over to the second tab here, which is like a view source, if you will, for the data we are just looking at and now we can see a list of traces. Traces are snapshots of individual requests in full detail as they go through your application.

Let’s take a look at a trace. For this particular trace, we’re looking at a request to this URL. We can see the time spent on average, or the time spent by this request, in each layer of our stack, and we can also see this utilization up here, which is the flow of control of the request through the application. I'm just going to zoom in a little bit here, because there’s a lot going on in this particular request.

The first thing that happens is the request enters Nginx which is our low balance here, and we can see that we've gathered some information about how the HTTP request came in and how it was proxied through to Apache, which is the second tier of stack here and finally into PHP underneath it. PHP starts to queue the Drupal bootstraps, so the first thing that happens here is we’re looking something up in memcache. We could see where it’s coming from in the application code, and a number of small queries start to fire.

As you proceed through this request, we can see, for instance, details about the different queries being executed, where they’re coming from within the code, how long each one took. This one only took one and a half milliseconds, and what exactly the code was trying to do.

Here’s the [boot hook 00:18:05], and so what we’re seeing is overall, this is taking about 85 milliseconds, and as part of it doing a number of sub-actions including hook a net here, which then triggers this query and so on. With the individual trait details here, you can drill down on what's going on, what sequence did the events happen for a particular request, what was the slow thing that really bugged it down. There’s some really interesting details down in here.

One of the cool things in PHP, that even though we instrument some other languages, you can't get is the sandbox notion of memory usage. We can actually see throughout a request here, we can see the memory use at the beginning and at the end of this particular query, the peak memory at any point in the request and so on, and this could be really useful for debugging problems where you’re hitting a memory limit for individual request. There’s a lot of great detail down here in individual traces, but let's actually go back up a level and come back and look at our aggregates.

In addition to being able to drill down on endpoints, they were interested in optimizing, we might also want to be able to view the data in a more précised manner. We’re looking at our averages here. I'm going to switch over to a different view of this exact same data which we probably keep mapped. I'm going to overlay the average on it again here.

What we’re looking at is like a scatter plot on the X axis, we still have time on the Y axis latency. The density of color in each root square indicates how many request had a particular latency at a certain time over the past 24 hours. We can see that while this red line indicating our average trapped this middle path, there’s actually really two distinct bands in the data here. There’s some faster ones, there’s some slower ones. The heat mass interactive, so I'm going to grab a little patch of these outliers from the second band and see what's going on exactly.

These are actually all requests for the same endpoint, then some resource here as a view stage made by anonymous users. It’s not surprising that they’re clustered like this. This is pretty interesting because a lot of times when you have numerous endpoints with distinct performance characteristics, an average really just blends them together. Similarly, we've got the request volume on this bar underneath, and you can see that there’s a large request volume around this time of the day. They’re actually requests for relatively fast pages to load which brought our average down. You can still see that it wasn’t that our application got faster overall, it was just that the distribution of requests that were made changed.

We can think about optimizing in a different way when we see that there’s this constant population of relatively slow requests here that are spending from 6 to 10 seconds on the server side. Heat map is a powerful tool for drilling down on these types of performance problems.

In addition to providing the interface for slicing-and-dicing this data and filtering down to what you’re really interested in optimizing, we also provide alerting based on this, so you don’t have to be watching your application 24 hours a day. It’s pretty easy to set up alerts. They’re based on latency, on the performance of different hosts in your application, or on the error rate. You can actually filter each of this down to particular layers of the stack you’re interested in, or even URLs or menu items.

For instance, it turns out that latency is actually a pretty good predictive alert, but maybe your latency for the application overall is kind of noisy and so instead, you decide to restrict to particular URL like your checkout page, and then you can get alerted if pages that are important to you start to perform outside of your standards.

The last thing I’ll mention on the server side is our host monitoring capabilities. Latency and the behavior of applications are obviously very important, but sometimes what's really underlying it, i.e. the hardware, is the long point attempt is something that you need to keep an eye on. We’re also gathering machine data in real-time without the performance of different hosts in your stack.

You can see there’s a view here where we can look at all the different machines that we’re monitoring, but actually, sometimes it’s useful to be able to correlate that performance data with the application’s performance itself. We can overlay the host metrics on our performance data here, so what I'm doing is we’re looking at the past date again, I'm pulling up the CPU utilization on our frontend node, and we can see that as our request volume spiked yesterday afternoon, so did our CPU usage.

The other thing that you can get out of Trace View is end-user monitoring. You may already be doing this with something like Webpage Test or even with Chrome Inspector, but it’s useful to be able to get not only the point of view of your test sessions, but of real users around the internet.

I'm switching over to a different customer kind here that runs some high traffic logs. We can see that they’ve actually done a pretty good job of optimizing the server site performance here at the average request taking about a quarter-second on the server site, yet the full page load is actually close to 11 seconds on average.

Let's drill down on the end-user performance data. We can see that on average, we’re spending a lot of time in down processing, so getting together all the elements of the page and also in doing the page render so getting the document ready. There’s a little blip here of network latency, but other than that, it’s behaving pretty well there.

In addition to getting a latency here again, we’re also associating it with the features of request. That includes geographically where the requests are being made from, the files are being used, the URLs requested, and the code path is within the application. If we wanted to figure out what our performance is like in the United States or maybe in British Columbia, we can filter down to data from this region.

We can see the URL is being requested and which ones are performing well or poorly as well as the browser is being used. We can get comparative browser usage and finally associate all of these down again to individual requests and individual browser sessions so that we can get into that performance data in a highly granular way.

That’s Trace View in a nutshell. I’d like to hand it back over to Jess and open it up for questions.
Jess Iandiorio: Thanks, Dan. Sorry, we’re on mute here. That was a great demo. We really appreciate it. The first question we have is, are you related to Stanley Kubrick?

Dan Kubrick: No, but thank you for asking.

Jess Iandiorio: Sure. We have one question. Would you mind reading it, and I can’t see them. Do you support Drupal six and seven?

Dan Kubrick: Yes. We support Drupal six and seven, and the community module does as well.

Jess Iandiorio: Okay. That person also asked about eight, but that’s not available yet, but I assume once that’s available next year, you guys will be supporting that as well.

Dan Kubrick: Definitely.

Jess Iandiorio: Do you support Linux distributions?

Dan Kubrick: Yes. Currently, we provide debs and RPMs for Red Hat’s CentOS to Debbie Anne and Amazon Linux-based environments. I should also mention, if I didn’t earlier, that it’s a one-click install for Acquia’s hosted members of the network.

I see there’s another question about the setup in general. After you register for a free trial, you actually get walked through the install process within the application. It’s basically just installing three components from most users: a package that has our base, a package that installs Web server instrumentation, say an Apache module, and a package that installs a PHP extension.

After that, as you install each component, you’ll get immediate visual feedback within the application which will prompt you to continue, then in the future, because we’re providing packages, it’s actually very easy to use, either Puppet or Chef to work this into your automated deploy.

Jess Iandiorio: All right. We’ve got about 10 questions in the queue here so hopefully we can get through all of these. The next is, do you support cached versus non-cached, CDN versus non-CDN analytics, can they break it out down at that granularity?

Dan Kubrick: We currently don’t have visibility into requests that are going to the CDN except for to the extent that they speed up your end-user performance. Getting more statistics on full-page caching behaviors is something that we’re interested in the future.
Jess Iandiorio: We have two questions on the difference between Trace View and New Relic. Could you speak to that at a high level?

Dan Kubrick: Sure. We get asked about this pretty frequently and there’s basically three main differences. The first one is our full-stack application tracing. The same technology that allows us to follow requests starting in Nginx or Apache or Lighty also allows us to cross the wire for subsequent RPC calls if we’re using backend services, maybe with restful APIs. We can actually piggyback the unique identifier across those, so you can associate the work being done in your frontend and backend services as well, which is pretty cool.

The second thing is our data analysis and visualization, in terms of the granularity of the individual request view, particularly those three-point internals as well as the analysis that you can do with the heat map, is pretty unique compared to New Relic.

The last thing is actually our pricing model. Instead of pricing per host, Trace View is priced per trace, and the number of traces that you send us is configurable via your sample rate. What this means is that you don’t have to worry about buying a number of licenses to cover your entire deployments or having auto scaled nodes not be covered by your application performance instrumentation. You can actually set it up and use consumption based pricing.

We offer plans that start at just $95 a month, but there’s a two-month free trial, so you can definitely get your feet wet without having to worry about it. A lot of people find that our pricing model can be interesting for their environment because of the lack of purpose pricing.

Jess Iandiorio: Great. Just for the folks on the phone who might not be Acquia Network customers, can they do a trial directly through you guys if they’re not an Acquia Network customer?

Dan Kubrick: Yes. Unfortunately, it’s not nearly as lengthy, but you can head to appneta.com/starttracing, or just go to appneta.com and follow the links to sign up for a free trial.

Jess Iandiorio: Okay. Does Trace View work with an environment where Varnish is used?

Dan Kubrick: Trace View does work with Varnish, but we don’t provide any Varnish specific insights.

Jess Iandiorio: Okay. We got a question on mobile. How can this be used to monitor the performance in tablet and other mobile devices?

Dan Kubrick: As far as applications on mobile devices, those should be monitored from the perspective of the API calls that they’re making to say a restful back end service, or actual browser page views on mobile devices that’s completely covered by our real-user monitoring instrumentation, and you’ll find that just looking at the real-user monitoring data we gather there’s some very long page that’s from mobile devices, which is pretty cool to be able to do to separate out there. Our instrumentation works on all mobile devices, but mobile applications are viewed from a server-side perspective.

Jess Iandiorio: Okay. Where is the performance data stored and how much storage do you need to store it? Any metrics that you can provide or …

Dan Kubrick: We actually take care of all the storage at a SaaS-based service, so you don’t have to worry about any storage or scaling the storage on your side maintaining our upgrades. What you do as a Trace View user is install the instrumentation that gathers the data and we’ll take care of the rest.

Jess Iandiorio: Great. This question is lengthy, asking about more information, do you collect pops or Sourcepoint breakdowns? What Geo has the slowest response time? I know you showed some Geo stats earlier.

Dan Kubrick: In terms of geography, what we’re basically doing is using the IP to look up the origin of the request. In terms of actual network conditions on the point between your servers and the end-user, Trace View doesn’t provide insights instead network connectivity, but AppNeta has other SaaS-delivered solutions that actually provide a great deal of insights into network performance, even from just a single side of the connection.

If you’re interested in that, feel free to shoot me an email afterwards to inquire or head to appneta.com. Trace View will tell you the latency and the fraction of it spent the network, but not a great detail about hop to hop performance.

Jess Iandiorio: Okay. Are there any HTTPS implications or loss of fidelity or metrics or details?

Dan Kubrick: No. We’re not putting any proxies in between. HTTPS works fine with Trace View.

Jess Iandiorio: Okay. You may have already answered this one. Is there a specific node under the lowdown’s identification, instrumentation, HTTP or MySQL daemons?

Dan Kubrick: Sorry. Can you repeat that question?

Jess Iandiorio: It’s there a specific node under LD’s identification or instrumentation, HTTP or MySQL daemons?

Dan Kubrick: I'm not sure I clearly understand the question, but in order to get installed, we actually observe many of the components from the application layer itself, and we live inside the Web server as far instrumentation goes, and the application layer so you don’t have to worry about modifying your database servers or anything else, if that’s what the question was.

Jess Iandiorio: Okay. If that person is not …

Dan Kubrick: If you’d ask that one again?

Jess Iandiorio: Okay, so there’s some clarity. Let's say there are five nodes under a node balancer wherein one node performs different than the others, can Trace View help identify the outlying node?

Dan Kubrick: Yes, because we’re gathering metrics on a purpose basis, especially if that’s showing up in terms of that node is thrashing or using more CPU, that’s something that you can identify using Trace View.

Jess Iandiorio: Okay. You’re doing a great job. We only have two more questions left on this first batch, so if anybody else has questions, please feel free to submit them now. The last two. First is, does this also monitor Solr search server performance?

Dan Kubrick: We watch connections mid Solr and view the performance there, and we also have Java instrumentation that can look into Solr’s internals to some degree, mostly CPU and load-wise, but a little bit inside job as well.

Jess Iandiorio: Okay. Are there any issues installing this on Amazon, classical balance servers with EC2 instances and RGS database?

Dan Kubrick: No. We have tons of customers running in EC2. The only caveat is, if you’re using RDS, you can't actually install our system agent on that RDS machine, and so we’ll just be observing the queries and query latency for RDS.

Jess Iandiorio: Okay. What about Windows and IIS support?

Dan Kubrick: Windows is on the roadmap for 2013, but today we only support Linux-based environments.

Jess Iandiorio: Okay. Does Trace View also track affiliate program performance codes on third-party sites?

Dan Kubrick: Not out of the box. You can add custom instrumentation that will allow you to track the metrics that you’re interested in for your application, but that’s not one of the things that we have automatic instrumentation for.

Jess Iandiorio: Okay. Someone heard Apache being mentioned a couple of times. Is Nginx supported as well?

Dan Kubrick: Yes. We provided a module for Nginx, as well as a number of package versions of Nginx that contain the module, so yes.

Jess Iandiorio: Okay. Great. We have a question about can we use New Relic and Trace View at the same time? I’ll answer from the Acquia Network customer perspective and then Dan may have something else to add. If you are an Acquia Network customer, and you’re currently using New Relic, you cannot run New Relic and Trace View at the same time.

You would need for us to turn off your New Relic agents in order to enable the Trace View ones, and then we would need to turn the New Relic ones back on for you after the Trace View trial, if that was your preference or you could move forward just with Trace View. That’s for Acquia Network customers right now, and I don’t know if that’s different for you, Dan, for other people who might want to work directly with you guys that aren’t Acquia Network customers.

Dan Kubrick: We can’t control what you do, but we don’t recommend it. Both of the New Relic extension and Trace View hook in to PHP’s internals, and so we can't always be on top of the releases that New Relic is putting out, and they’re not always keeping in stuff with us, and so we don’t advise customers to go down both rods at the same time. What we do have, especially during evaluations is often some customer will try New Relic on one or two machines and Trace View on one or two machines as well. That’s the ride I’d go.

Jess Iandiorio: Okay. Great. Well, that’s actually all of the questions we have. Nice work. That was a lightning round of questions. It’s really nice to see people engaged and asking lots of questions as someone who does two or three of these per week sometimes. We really appreciate all of the interest and attention and questions.

If anybody has any last questions, I'm just going to flip to the last slide here. It’s just contact information, if you’d like to get in touch with either Acquia or New Relic or Trace View. Any other questions? We’ll just hang out a couple of minutes here. Let’s see here. Is there a raw data extraction life cycle?

Dan Kubrick: Currently, we provide an API for exporting some of the data in the interface, namely the slowest queries, the slowest URLs, the slowest code path to the application. We don’t have a full read at API, but some of it is extractable.

Jess Iandiorio: Great. All right. Well, that was very productive. Everybody has 15 minutes back in their day. Thank you so much, Dan. Really appreciate your presentation, great overview and lots of good answers to the questions. You can get in touch with AppNeta and that’s the company that owns Trace View, if that wasn’t clear. It used Tracelytics, now the company name is Trace View and it’s owned by AppNeta, just to clarify that. You can get in touch there or you can get in touch with Acquia. Please feel free to send us any more questions you might have on Trace View and/or the Acquia Network.

Dan Kubrick: Great. Thanks so much, Jess, and thanks, everybody.

Implementing Continuous Integration Best Practices with Drupal [November 7, 2012]

Click to see video transcript

Speaker 1: With that, I am going to turn it over to Barry. We had about 30 minutes of content for you. So again, please ask questions along the way, and thanks for joining.

Speaker 2: Okay, thank you, Tess. Today I’m talking about continuous integration with Drupal, and what that means and how you get there. Creating really good websites, as anyone who has ever tried to do so, is hard. You have to do a lot of thought of user design and talk to your customers and figure out what you want and do the visual design and implement it. It’s a lot of work, and that’s what many of us do for a living.

Then you have to deploy it and get it up running on a site, on a server somewhere. Really, that shouldn’t be hard. That’s not ultimately the creative part. That’s not the business that most of us are in. The fact is, that it is kind of hard. We all know that getting your site up and scaled and running and deployed efficiently is kind of a pain in the neck, and there’s just no way to reduce any of that pain, right? Well, that’s sort of what we’re going to talk about.

There is a process, a development methodology, called continuous integration, that has become fairly popular along with the agile movement over the last maybe 10 years or so, that makes the process of developing most kinds of software, but particularly websites and dynamic products that can be changed very rapidly, unlike if you put software on a CD in a box and ship it, there’s a limit to how quickly you can iterate, but when you’re putting something up on a website you can iterate very, very quickly. Continuous integration is a process that really takes advantage of that.

Continuous integration has a number of principles. By working in very small batches, you’re able to make small changes, deploy them fairly quickly. You always know when you’re doing a deployment that you’re changing only a little bit of work that you did just yesterday or maybe the day before, so you haven’t forgotten what it was instead of work you did six weeks ago and you have no idea what the changes are going to be. By releasing very frequently, only work that you did yesterday or the day before, you’re never really surprised that oh my God, this thing that I did two weeks ago doesn’t work with the thing that I did four weeks ago, and now I’m kind of screwed.

Then as part of a continuous integration process, part of the methodology is to constantly test everything, so that when you do a deploy, you’ve already done that deploy 15 times before and you know it’s going to work. There are a number of basic principles of continuous integration, as I said. There are not principles that I or Acquia or the drupal project made up. These are, as I said, widespread within the software development industry. I have a link here to the Wikipedia article on continuous integration where I basically took these things, and we’re just going to sort of talk through them and describe what they mean and how it affects building sites with drupal and how you can take advantage of it.

I just managed to skip ahead many slides. Using a source code repository, this is sort of the minimum steps, step zero, for doing good software development. Most people that I’ve talked to within the drupal community are doing this, many are using GET, SVN, Exist. There are other systems that exist, I’m not going to talk about which one you should be using or why. Frankly, they all work. If you need one or the other, you’ll probably know it.

But you absolutely must be doing all of your development in a source code repository so that all of your developers, whether there’s one or multiple developers, are always committing their changes on a very regular basis, ideally every day, to a development branch. Then having that development branch automatically integrated with a testing environment so that as soon as the code is committed, each increment is a little step and pushed up to your main code repository, it gets deployed in a development environment that is a clone of your production environment, and then your tests get run on it. Every commit, every test is being run. You know immediately if something broke.

Making all versions easily accessible means that at any time, let’s say a test does break. You did some development, you commit the code, you run it in your test environment, and it blows up, and you’re like okay, what happened, what did I change, I don’t think my code broke it, maybe it did. You need an easy way to get back to any previous version that you had so that you can compare, did this really work before? Maybe it was broken before and my new change didn’t cause it. You need a way to get back to every previous version that you’ve released in order to be able to tell.

An audit trail, affectionately known as a blame list, helps you not just in the source control sense of who made this commit, but who deployed this commit? Yesterday, I see that someone pushed the latest version from testing into production. Who was that? Because maybe they knew, if something is breaking, or maybe if something is fixed, they probably know why they pushed it. You need to know who to go talk to in order to go figure out what was done and why.

Automating site deployment. You have an audit trail, you know who is doing what, but it still needs to be the case that in order to push your site live, you need some kind of automated process so it’s very quick and easy, right? The idea is you develop … you do a little bit of work in a small batch, you write a little change, you commit it, it gets tested, you push it into your testing environment, maybe your stakeholders or other people play with it, and then pushing that out to live needs to be a very easy process. If it’s a big chore to push out or release the production, then this whole process of trying to make small aggregated changes falls apart.

Measuring the results. Okay, so you pushed changes out. You’re making all these changes all the time. You need to know, are they really helping? Of course, helping depends on what you made the change for. Maybe it was a performance enhancement, in which case you need a way to measure is my site faster or is it now slower? Did my performance enhancement backfire on me? If it’s a usability improvement, are users able to use the new feature or make a conversion to a sale or whatever improvement you’re going for. Are you able to … you have to be able to measure the small change that you made and deployed rapidly is having the effects you want, because if it’s not, then you’re very quickly able to go back and iterate again and make another small change and test it and deploy it using a very effortless work flow.

This whole … all of these principles combined make it possible to iterate very rapidly. That all sounds great, right? Okay, terrific, I’ll just go turn that on and we’ll be all set. Well, how do you get there? How do you integrate this process into developing of a website?

Well, there is, roughly speaking, two choices. One is you can build it yourself. I’m going to take a few minutes to talk about what’s involved with building it yourself. There are a lot of people who have written a lot of different blog posts and articles and given presentations that drupal con-end every other technical conference out there, explaining how to do this. Many of those presentations are excellent and there’s a lot of good information out there, and there are companies that do build this themselves and it’s a perfectly reasonable thing to do if you wish to invest the time in it if you have a really good reason. Let’s talk about what’s involved.

The first thing, of course, the first principle is you have a version control system. Well, that means you’ve got to host your version control system somewhere. You need a GIF server or an FCN server or whatever other version control system you want. Maybe you could host your code at Get Hub. Okay, or there are hosted FCN services as well. Set up that account, start using that service, and now you have to integrate with that service.

The most common way, now that you’ve got all your code managed somewhere, the most common tool for building this yourself that people use is call Jenkins. It was formerly called Hudson. Jenkins is a fabulous tool. We use it internally, and what you end up doing is you write a lot of little jobs, a bunch of custom shell strips that do things for you. For example, I just made a commitment to my repository. I would like to deploy that into my development environment. Well, I can make a Jenkins job that listens for any commit on my version control, and then does a check-out, and maybe runs R-sync or some other copying tool to push that into my web server. Of course, if you’re running on multiple web servers, then you have to write your Jenkins job to know about that, push it to multiple servers. You have to decide do you do each of your web servers one at a time, in which case you have a problem because you’re deploying your code first to one server and then to the next and then to the next, and so maybe you’ll have a version skew for awhile, or you can try to write your script so that it copies it into a temporary area on each web server and then it simultaneously is possible, maybe throws a SIM link from one old directory to a new directory. Maybe you want to get really clever and try to do all those copies in parallel. As you can see, that script which starts out oh, I’ll just copy my code when I make a commit, it can get a little complicated and involved.

Anyway, now you’ve written that script. You’ve got your code deployed. Well, it’s frequently the case when you’re developing a site that you want to use a copy of the production database on your testing environment or your development environment. You want the latest data to test the latest user contributed whatever, to test your most recent changes on. You can write another Jenkins job. When you run this job, it will go connect to your production database server and copy that database into your test environment or to your development environment.

Frequently, when you do that, you want to operate on the database in some way by sanitizing it. Maybe you want to remove all of your customer e-mail addresses so that you don’t, in case of some bug, God forbid, you don’t spam all of your customers from your dev environment. Or maybe, this is the problem that I used to run into a lot, you have FSL turned on on your production site. Maybe for the whole site, maybe just for your user log-in pages or some other subset, and you copy your database into your development environment, and you go to try to log-in to your dev site and it redirects you to the FSL version and maybe you don’t have an FSL certificate on your development site, so now you’ve got to go manually turn off the module, it’s a pain in the neck. It’s another kind of scrubbing or sanitation process you want to perform on your database as you copy it from production back to staging.

So you write a script for that. Now, as I said, you want to test, when you deploy your code, either when you deploy it to your testing environment, maybe there are some tests you want to run when you deploy to your production environment. If you have your script that deploys codes on commit that you wrote a couple bullets ago on this talk, you add to it to say run drush simple test or run load test or some other process. Another script.

Now finally, your test has passed. You’d like to push your code into production. As I said, one of the principles is you need to be able to access all prior versions that you’ve released publicly so that you can always get back to every version you’ve released for testing or verification so when it’s time to push to production after your tests have passed in staging, you need to make some kind of symbolic tag in your version control system that says this is a version that I released today, and then push it out to your prod environment. You write a little script that uses the get tag command or the SDN tag command or whatever. The point is, you end up writing a lot of these little … they all seem simple in principle, and for the most part none of them are that complicated on their own. There’s just a lot of little steps that you end up having to do.

Before actually you can even do all of those scripts, of course you have to have a server close to run Jenkins on and to run your website on, and maybe you need multiple servers because you don’t want to be doing all of this deployment and testing and all that on your production box, so you’ve got production web servers and you’ve got your testing web servers, and unless you’re going to be running your … you probably don’t want your Jenkins service running on your development and test web servers because then your development and test web servers are not an exact clone of your production web servers, so when you push your code from development to production, you can’t be really sure it’s going to work because the server is not exactly the same. Therefore, you don’t want Jenkins installed on your development machine. You want it on yet a third server. Now you’ve got at least three machines to manage, assuming each of your … assuming your site is only running on one server.

Very quickly, you’re managing several different boxes. You need some way to build those servers, configure them, and it has to be an automated process. If you have a set-up where all right, when I need some new package on my web server, some PHP extension or whatever you might want, if you just log-in to the box’s route and install it, then when that server blows up, which it will, you’re not going to know how to rebuild your service, how to rebuild your server, so you have to automate the management of those servers.

You can do that. There are great tools out there to do that. There are tools called Chef and Puppet and CF Engine and lots of things you can go read about. You have to set all that up. Obviously, if you’re running a service, you have to manage security updates, bug fixes, stuff like that.

Then of course, when we were talking about the Jenkins jobs, and I was saying somewhat lively, oh, I need to copy it from the dev environment to my test environment. Well, you have to have a dev environment and a test environment. That means you’ve got, if you’re using Apache or Engine X, some other web server, you have to create multiple virtual hosts, one for each environment.

You have to configure the domain names and the FSL certificates and the PHP INI for each of those environments. Maybe they’re the same. Frequently, they’re subtly different. Maybe in your development environment, you want PHP to have the debug extension or the XP bug or XH prof extension installed, but you don’t want that in production. You have to manage a different PHP INI file or a different set of configs. Frequently, you’ll have FSL in production, but you won’t have it in your dev and test environment. You almost certainly want different domain names for each of your environments because when you visit mysite.com, mysite.com is going to have a DNF entry for one particular IP address and you can’t use that to access both your dev environment and your production environment, so you end up having configuration files you have to manage.

All right, well, you could edit those by hand and put them in a version control system as part of your server build. Again, not a hard thing to do, just another step. Multiple databases, right. Clearly, when you’re running your development and test environments, you don’t want them talking to your production database, because then they’re going to screw all kinds of things up off of your live site. That means that you actually need three databases, one for dev, one for test, one for prod. Each of those databases will have at least a different name, if not a different user name and password, so you have to manage the settings, your drupal settings at PHP files, to talk to each one of those databases in the correct place. So your dev code, your dev site talks to your dev database, your prod site talks to your prod database.

Then of course, if you’re not just running on one server, your site you want to have in a more scaleable or more high availability environment, you have to deal with all of that kind of configuration. Your database servers, if you set up dual master databases with fail over or if you want to use memcash or varnish or … each of these other tools, you have to configure those. You have to automate the configurations that you can get them back when they blow up. You have to be able to scale that service, so when the load on your database or on your solar server or whatever increases, you’re going to have to know how to scale that up.

This is not a talk about telling you how to do these things. As you can see, there’s a lot of stuff you have to do, it’s just to give you an idea of what all these things are. Oh, by the way, I haven’t yet talked about you need to build a back-up and restore all these things. You have monitor them 24 by 7 because by now, as you can see, you’ve got a lot of different pieces moving all the time, running your site. Any one of them can and will fail. Processes crash, servers crash, disks could blow up. You have to monitor it, and then of course you have to have someone to answer the monitoring. Get a page in the middle of the night and do something in response to it. Ultimately, you need Ibuprofen and beer and drugs to keep those people happy. It’s a chore. You can do this. I mean, I’m describing it as a lot of work. It is a lot of work. There are companies that do this themselves and are very happy with it. But just understand that you’re making a real investment in time and resources, so you have to have a really clear understanding of the reason that you’re building it yourself.

Okay.

Speaker 1: I just want to do … I want to interrupt you quickly, Barry, just to let everybody know that we’re actually running some polling questions during the webinar, so if you haven’t checked yet, we already asked one question, which is have you heard of continuous integration yourself. We had about 46 people respond. We’re going to launch another question in just a minute that’s going to ask if you’ve actually build a continuous integration functionality yourself. Thanks for your attention and participation there.

Speaker 2: Okay. Thanks. All right, so option one is build it yourself, of course, and as I said, you can do this. You really don’t want to. Maybe there are some companies that have a really good reason and for them, of course, it’s completely justified, but for the most part, this is not the business you’re in. You’re in the business of creating really killer websites, either for your own business or for your clients, and you’re not in the business of doing all of it infrastructure management.

As Jeff said at the beginning of the talk, if you actually like doing this stuff, please let us know because we will hire you.

Option two is use someone else’s system, and of course this is the part where I say we have a product that does all of this for you. Now I’m going to show you how it works. Just for my own little convenience, I’m going to have a cheat sheet where I pull back up the principles of continuous integration so that I can remember all the things I want to show.

For this, I am going to need to share my desktop, which I can do from here. Tess, can you confirm that my desktop is going through?

Speaker 1: Viewing your desktop.

Speaker 2: Okay. What I am looking at here is the main page of the Acquia Cloud area of the Acquia network, which is what I’m going to be talking about for the beginning today. The first principle of continuous integration … oh, this is interesting. I can’t actually see my cheat sheet because I’m in desktop sharing mode, so I’m going to have to remember it.

Speaker 1: I’ll write them up.

Speaker 2: Okay. It’s user version control system. Acquia Cloud supports both GET and FCN. We show you every time you create a site on Acquia Cloud you get a repository, the URL is displayed right up here at the top. If you prefer one or the other, you can switch from one version control system to another. It’s pretty easy to do. The way that Acquia Cloud manages this is that within either GET or FCN, you have what are called branches and tags. A branch is like the master branch or some feature branch, maybe you’re developing some new provident feature and you create a branch called provident. Frequently, everyone just develops on the branch called master, and then within each of your environments, development, staging, and production, you can choose which branch you want to be deploying. In this case, I’m deploying the master branch in dev. What that means is any commits that you make to the master branch is within a few seconds deployed into your dev environment.

Similarly, you can choose … I’m sort of skipping ahead of myself, but when you do a release … if I say, for example, all right, I’d like to deploy the code from my dev environment into staging, you can just drag and drop your code from here and it gives you a little confirm box and says are you sure you want to do this? When you say yes, what it does is it creates a symbolic tag. In this case, based on the date, 2012, March 7th, pointing at the tip of the branch that you dragged. So if I drag the master branch to staging, I’m going to make a tag. Of course, you get the tip of the master branch, and then deploy that in the staging environment. Now I have, each time I do that, a copy or a symbolic reference into the exact code that I’ve deployed on a certain date and time, and now that’s running in my stage environment. Then when I’m happy with this code I can deploy it over to my production environment, again with a drag and drop. That’s step zero, using a version control repository.

Automate testing. There is a bunch of things you might want to automate, as I said. There is running your system test, or running your drupal simple test, or running a load test, or any kind of other action that you might want to automate when you perform a task like deploying your code from dev stage. The way Acquia Cloud manages that is with something called a cloud hook. A cloud hook is a script that you put into your code repository that Acquia Cloud runs for you at the right time. I’m going to switch to a shell here. In here, I have a directory called back route, and that’s where my code lives. That’s my drupal site. I also have a directory called hooks. Within the hooks directory, I have a subdirectory for each environment. In this particularly one, I’ve only put a hook in for the staging environment, which is also called test. Within the hooks test directory, I have another directory that says what action I want a hook to run on.

This says in my test environment, whenever I do a code deploy, when the action is complete, I would like to run all of the scripts that are within this directory. At the moment, I happen to have one of them, which is called update DB, and what this one actually does, this … sorry, this doesn’t actually run tests, this runs … hold on … this actually runs the drupal update dot PHP functionality. It uses drush to do it. So here example is a hook script. It’s given its command line arguments, in this case, the target environment, in this case test, because this hook script is installed in a test directory, and it very simple runs drush update DB. It uses this, which is a drush alias. Some of you may or may not be familiar with drush. Drush is a command line and scripting tool that lets you perform various operations on your drupal site. These aliases, so @eebdotson.test is an alias that refers to my drupal site by name so that I don’t have to encode in my script what server it’s on, what directory it’s on, what usernames are run as, all that kind of stuff.

Simply by dropping this script into my hooks test post-code deploy directory, every time I do a code deploy, for example, dragging code from development to staging, then this script will run and drush update DB will run as well. If, for example, I do want to run my system test, then I can … I actually have a sample script for that. I think I know where it is. Sorry, I’m just going to reach into my development environment here, and see … drupal test, there we go. I have this sample script called drupal test, and I will install it into my hooks test post-code deploy. Great. Now let’s just look at what that test does, that script does.

Again, this one turns out to be just another invocation of drush with a little bit more wrapped around it, so here we get the site in the environment, we say which test do you want to run. This is a sample, of course, so it’s pretty verbose and commented.

Speaker 1: We have a quick question. Someone missed the first 25 minutes. Does continuous integration require hosting in the cloud?

Speaker 2: Continuous integration using Acquia Cloud requires using Acquia Cloud. The first 25 minutes talked about the things you could do to build continuous integration yourself if you don’t want to use a service like Acquia Cloud or some other hosted service. What I would recommend you do is … we’re repeating this webinar at 1:00 and you could come them.

Speaker 1: And can they do it on premise though? Could it be based out of a … does it have to be a cloud-hosting other website in general to do continuous integration?

Speaker 2: Let me get to this in the question section at the end. We’ll come back to this question. This hook script does some set-up, it makes sure the simple test module is enabled, but then it all boils down to this line right here, where again it happens to use drush to run tests, then it exits with success or failure. To automate your test, to automate update DB, to automate, as you can see, a database scrub or sanitation, you can put these hook scripts into your repo and then they run. I’ve shown you so far code deploys, but similarly, if you want to copy your files from production back to staging or a database from production back to staging or dev, then a similar directory full of hook scripts can run that will do whatever actions you want them to.

For this little demo environment, I actually have an example where a few minutes ago, just before the webinar started, a couple minutes after, I did a deploy from development to production. Now remember, in this particular code base that I showed you, in my hooks directory I don’t happen to have any production hooks, but what you can see is … so it says okay, I’m starting my deploy, I create a GET tag, I SSH into my servers and I do some stuff, then I deploy the code, then down here, we say … okay, so it’s all deployed now, then it says starting hook, post-code deploy. Of course, there are no scripts in this repo for the production environment, so then immediately it’s finished because there was nothing to do. But if there was anything to do, it would have run it there.

You might think wow, what a poor demo, he’s got a failure right here on the page. What’s going on with this is I’m just showing you how hooks can actually help you even when they fail. I set this demo up yesterday. I actually installed a distribution into dev and I deployed it to production. I deployed the code to staging, but I never installed the database on staging, so the site, on staging, if I visit it, it just gives you a white screen because there’s no site there. When I tried to deploy my code from development to staging, what happened was all right, I created the tag, I deployed my code, I did all this stuff, and then I started my hook post-code deploy, and it tried to run drush update DB, except there’s no database there, there’s no drupal site there. The code is present, but without a database, drush update DB can’t run, so drush update DB failed. What this is showing you is that when your hook script fails, here’s all the output. Drush is saying I tried my best and I blew up, and so it exited with a status code, and that gets surfaced via a failure in the UI. You can see that your hook scripts succeeded or failed based on whether the tack is done or failed.

That covers automating testing and using a source control test in a … okay right. So testing in a clone of your production environment. Obviously, we maintain each of these environments, dev, stage and prod, they might be on the same server, they might be on different servers, they might …your production environment might be on 12 web nodes and 2 databases or whatever, but we maintain the configuration of these environments to be identical. They always have the same PHP config, you have options you can turn certain extensions in certain places, but the environments are all … they all have … if FSM peg is installed, it’s everywhere. If ghost script or image magic or whatever, those things are always everywhere.

Making all versions easily accessible. I showed you you can very easily get back to any of the previous versions that you’ve deployed just by using the tags. You can see all the previous dates on which we’ve run a demo. Having an audit trail, well that’s the task log. It shows you at exactly what date and time every action was taken on your site. At the moment, one thing it doesn’t show you is who did it. We’re going to be fixing that very soon. We’re probably going to actually replace this task ID column, which customers don’t really have that much use for the internal ID number, with just a column that says who, and shows you what the name of the user was that did the deploy. That will be coming very soon. But you can already see what was done and when.

Automating site deployment. I sort of showed you that already. You can deploy code quite easily by dragging and dropping. If there’s any other actions that you need to take, for example, when every time you deploy if you want to clear your varnish cache or clear your drupal cache or, I know one of our customers, and I think he’s even on the phone at the moment, they store their user uploaded files not in the local file system but they store them in F3. If on deployment you need to synchronize new files into your F3 repository or whatever it is you might need to do, you just write a short little one-line script for your custom need and when you deploy your code, which we handle for you, those extra tasks will run.

Measuring the results, I’m going to actually put off for just a moment and show you at the end.

There’s one other thing, sort of you’ll be the first to see this publicly. A little bit of an internal feature that’s going to be released probably in beta next week, is that I’ve been showing you how to do this on the user interface. You can actually do all of this as well from a command line, and from an API. We have … actually, I don’t know if I’ve set up this environment to do it. I have not set up this environment as part of the private data. But what I can show you is that there are commands for … well, for example, I’ll just show … I actually renamed these yesterday.

There are drush commands to do all of these things as well. If you want to deploy code from one stage to another, you could run a command like drush on your site, on a testing environment, and I hit enter and that will do exactly the same thing that dragging my code from here to there will do. Similarly for files and for database and for a bunch of other things that work as a part of … as part of Acquia Cloud. As a quick little demo, as I said, this demo site is not yet set up, but if I do just my personal production site, I hope this won’t blow-up, I haven’t tried it yet today. Oh, there it is, okay. I just retrieved some information about my production site. It’s running GET. Here’s my repository. If I say I want to retrieve my production environment, I will say look, this is the tab I’m running. Again, this is the same information that you see on the UI, it’s just in a text form.

Very briefly, about measuring the results. There are a variety of ways you can measure the results. One I talked about is performance testing. Another is so you can test both sort of the external performance of your site, you can run a load test, have lots of fake concurrent users hitting your site. You can also do internal measurements of how much time is my site spending on database queries and MPHP and talking to mem cache and the file system, and at the very beginning, remember I said we are looking here at the cloud area of the Acquia network. There’s a lot more to the Acquia network than that. In particular, we offer a bunch of partner services that you have either free or reduced price access to. Generally, there’s always a free level that comes with the Acquia network that is generally more than you get for free if you don’t go to the Acquia network, that lets you do a lot of this performance measuring.

For example, we have a partnership with a company called Blitz-IO, where you can do a load test. You can have a little cloud hook script, one line script, that you drop into your hooks directory that every time you push code to testing, it runs a Blitz-IO load test and maybe 500 users simultaneously from around the world all hit your site. We have a partnership with New Relic, which is … let me see if … hold on a second … I’m going to go into … this is called winging the demo in real time. New Relic is an internal performance monitoring tool that runs on your site, and here I’m going to go look at a personal website. This is not a demo of New Relic, but very quickly, this shows you … you can see how much time you’re spending in PHP and in your database, and talking to external web services, and you can say oh, here’s my slow page, click on it, and they’ll give you information about what all your database queries are, you see clearly selecting from the access log is 95 percent of the use of my slow page.

You can see from this what’s causing your site to be slow, and a feature that actually, a really cool feature that they just released fairly recently is that New Relic now has an API where you can call them and say I just made a release, and they will simply draw a horizontal line on this graph that says you made a release right here, and then you can say … it’s … the tag for this release is, the person who did it, you can assign additional data to be shown on this vertical bar. We will very shortly be coming out with little sample scripts to put in your hooks directory so that every time you do a release, a vertical bar shows up here on your New Relic graphs and you can come and look and say oh, I pushed a release on day x, and/or five minutes ago, and now I can go look at my New Relic graphs and see the performance approved or it got worse, and you immediately have feedback about the nature of the change that you made.

There are a few other services. There’s AB testing, visual website optimizer lets you do AB testing where you can see if I moved my buy now button from the left side to the right side or whatever, does that affect user behavior? Again, you can start tests when you do new releases, see how the data … get the data very quickly. These services are not things Acquia has implemented, but they are things we’ve integrated from partners to let you do this very rapid iteration, like getting data tied to all of your releases.

That’s a quick demo. Let me go back now to the slides. Now, very quickly, I want to just talk about what the actual … what I’ve shown you is the technology of the Acquia Cloud platform. There are two different ways we … two different products we have based on Acquia Cloud. One is called dev cloud, the other is manage cloud, and I want to talk about what the differences there are.

Acquia Dev Cloud is the technology I’ve just … is primarily the technology I’ve just shown. It is Acquia’s continuous integration platform. It’s a developer-focused offering. It’s designed for people who are building sites, either for themselves or for clients. It provides all of the integration and automation and power tools that I’ve just been showing. It runs on … and drupal tune hosting. It’s the best hosting platform that we know how to build.

Managed Cloud is the enterprise version of this same technology, and what’s different about it is it has high availability, load balancing, you can have Managed Cloud, your site running, and U.S. East Coast and Europe and U.S. West Coast with automated failover, and we pair that with our enterprise level support contracts where 24 hours a day you can call us up. If it’s Saturday night at 3:00 a.m. and your site went down because of a bug in your custom developed module, we will help you fix that. Our support people will respond and do whatever is necessary to get your site back up.

From a technology point of view, in terms of the continuous integration platform, what this webinar is actually about, they both have the same capabilities. They all have the automated testing and the deploy from code repos and the hooks and the API and all that stuff. Managed Cloud is about white glove service, very high performance, very high-end support, and high availability and regional failover and that sort of stuff. Cloud Product Marketing perform, might want to give a more professional description of what those two products are, but that’s pretty much how I summarize it.

That’s the content that I had, and now we can talk about questions.

Speaker 1: We also have a poll open right now where we’re asking what do you use for a deploying code? We’d love for you guys to weigh-in there. But any other questions that you have right now, please submit them. We’ll stay on the line for the next couple of minutes, and thank you for your time. Maybe we can address the question about can you have continuous integration …

Speaker 2: Require hosting in the Cloud. Sure. I don’t know exactly what you mean by this question. There’s a lot of different ways to interpret it. Continuous integration, as a development methodology, you don’t need us at all. You don’t need the Cloud, you could run it on boxes in your closet with your own development team. It’s just a way of doing software development. I’m not sure if that’s what you’re asking me. Another thing you might be asking me is can we use Acquia Cloud’s continuous integration support? What I’ve been showing you with the automated deployment and the cloud hooks and the API, if you’re not hosting on our platform. The answer is yes, sort of, and the way that we have come up with doing that is that when … let’s suppose, for example, you want to do your development and your staging on our platform, but you want to deploy your production site in your own data center. If you’re a company that has an existing data center and an IT staff, and for whatever reason you want to keep using that, your code is going to live in our version control system, your databases … your development and staging environments can be on Acquia Cloud, but then you can write a post-code deploy script for a code that says if I drag my code from staging to production, Acquia will deploy it to what we think your production site is, which is to say on our platform, but then your post-code deploy hook script can then copy that code off via FSH or R-sync or whatever you want, to whatever server or environment you have in your own data center wherever it is.

Similarly, if you go in the UI and drag your production database back to staging, we will copy what we think your production database is, but then your hook script can say all right, actually, I’m going to forget what was just copied from Acquia’s concept of production. I’m going to go do an FSH my sequel dump or whatever, against my real production environment in my data center, and suck that data down and install it into the development environment. We actually do have customers who are doing this. They really like the user interface, the dev and staging environment, and the automation that we provide, but they need their own production data center, and so they’re using hook scripts to integrate with their own off-site … or their own on-site off-cloud environment.

Again, I’m still not sure if that’s what you’re asking, so if you need more information than that, why don’t you just send me an e-mail at the address that’s on the screen.

Speaker 1: A question about is there a gira plug-in for Jenkins?

Speaker 2: A gira plug-in for Jenkins? I believe that there is, but I’m not sure. What he’s talking about … so gira is a ticket tracking system. We actually use it internally here at Acquia, for where you track all the features and bugs and all that that you want to add to your software, and gira integration with Jenkins would be hey, every time someone posts a new fix to an issue in gira, go fire off a task that maybe will go run my system test. I suspect that exists. I know that gira and its code review system, crucible, has a API, so if it doesn’t exist you could write it. Again, that’s just work you’d have to set up yourself.

Speaker 1: You’ve shown dev stage and prod environments. Is there an issue if this is extended to dev tests, UAT and prod? So four environments, he said.

Speaker 2: Our system is very capable of supporting that. In Managed Cloud, we have customers running a dozen or so environments. In dev cloud, we do not yet have a UI for creating additional environments. If we see a lot of demand for that, we can. Again, it’s fully supported internally. Really, it’s just a question of do we add a UI to let you add and remove environments. If that’s something that you need, contact us and we will at least bump up the priority, because another person asked and/or get around to implementing it. Just a question of backlog.

Speaker 1: Great. That actually concludes the questions that we have coming in right now. Thank you everyone for your time. If you’d like to join us and learn a little bit more this afternoon, we’re going to host another session at 1:00 p.m. Eastern. One more question before we go.

Speaker 2: Do cloud hooks fire when commits are made to a branch, the master branch? That is coming soon. At the moment, we have cloud hooks for when you deploy code, either changing the branch or copying a branch or a tag from one environment to another, but not yet on commit. But we will have that probably within the next couple weeks. Just lots of things to do. We haven’t gotten to that one, but it is coming.

Another thing that is currently not … that we currently don’t do is … so as you can see here, when I dragged … oh, you can’t see, I’m not sharing my screen. I will share my screen very briefly so you can see what I’m talking about.

When I deployed my code from dev to prod, we created a task in the task list and here, I’ll do it again right now. What we show is that we show a little spinner, code is being deployed to prod and in the task environment … so the task list, you can see this task it was waiting, now it started, it’s going to be running, and when it finishes, when this task finishes … okay, it’s done now. You now know that that code deploy is finished. We don’t currently do the same for a commit, so if you make a commit to the master branch, we will deploy it within a few seconds, but nothing in the UI actually shows you we started and we’re done. Along with the hooks on commit, we’re going to be adding that into task environment so that you’ll know exactly when your commit is done deploying. Tess was just trying to get my attention.

Speaker 1: Another question coming in here. This question is an easy one. Does using continuous integration count as three sites on the network?

Speaker 2: No. What they’re asking about is when you have a subscription to the Acquia Cloud, you have a certain number of sites that you can create and we’re going to be moving to a pricing model where you can have as many as you want, you just pay per site. But each environment doesn’t count as its own site. A dev stage and prod environment is one site. Really, a site, to us, is a code base. You have a version control system, a repository, and within there is a doc group, and that’s your site. However many databases you want to create, if you want to do drupal multi-site, go to town, we over here on the databases UI you can create more databases and you can assign more domain names and using drupal’s multi-side feature you can run however many multi-sites you want. That’s still all one site, because it’s just one code repository, one code base.

Okay, did I stop … oh, here I thought I was pointing at the UI and you couldn’t see me. But anyway, the answer still stands.
Speaker 1: Great. I think that looks like … all the questions come at the end. Are there options like code sniffer, crack code, running code tests in dev cloud?

Speaker 2: Are there options like … I’m not sure what code sniffer or crack code are. Running code tests, you absolutely can run tests or anything else that can be scripted via the cloud hook that I showed. If there is some tool that you want to run, some script you want to run or some third party service, you write a little cloud hook that invokes it when you deploy your code or when you copy a database or when you copy your files or whatever it is you want to do. If that doesn’t answer your question, then send me an e-mail message.

Speaker 1: All right.

Speaker 2: Anything else?

Speaker 1: Going once, going twice. Let’s see here. Nope, that’s just some good feedback. People like your demo, Barry, good job.

Speaker 2: Terrific.

Speaker 1: Thanks so much for your time, and you will receive an e-mail with a recording in the next 48 hours. Thanks. Bye-bye.

The Hub, Bub: How Drupal acts as the Digital Media Hub for The Grammy Awards [November 6, 2012]

Is it Time to Consider Using Open Source WCM for Digital Experiences? [October 31, 2012]

Click to see video transcript

Hannah: Hi, everyone. Thank you for joining the webinar today. Today’s webinar Is It Time to Consider Using Open Source Web Content Management for Digital Experiences with Stephen Powers who is the vice president and research director at Forrester Research. Also, joining us is Anjali Yakkundi who is an analyst at Forrester Research, and to finish up the presentation who have John Carione is a senior director here at Acquia. We are very excited to have Stephen and Anjali speaking for us today and we hope you enjoy the presentation.

Stephen: Great, thank you. My name is Stephen Powers. I’m a vice president and principal research director with Forrester. I’m joined by my colleague, Anjali Yakkundi, who is an analyst with Forrester. Today, we’re going to talk to you on our portion of the presentation about whether or not it’s time to consider open source web content management for digital experience initiatives. Anjali and I have been working on some joint research with Acquia over the past few months researching organizations that are using open source web content management. This actually really has been a fun project for me. It’s something I’ve always wanted to do in my six years at Forrester. I actually did a project about four years ago, a paper, on open source WCM. One of the challenges back then was that it was very different to find examples of large enterprises, enterprises with a billion dollars a year or more revenue who were using open source WCM as their primary WCM for customer face experiences. That’s not to say that we couldn’t find plenty of people using open source WCM, but often it is was for part of their websites or Internets or for purely informal websites, some of the easier use cases.

That’s really changed over the past few years. As part of this study, we did find some great examples of organizations, large organizations, using open source as their primarily WCM. Those we did speak to within the course of this study were very careful to emphasize that there were some situational factors at play with their open source WCM success. The results of that is what we wanted to talk with you about today.

The agenda for our portion of today’s presentation is a fairly straightforward one. Number one, we’re going to talk a little bit about the state of the market. This has a been a rapidly changing market ever since I started here at Forrester six years ago, so it’s always been fun to cover. Anjali and I are going to talk to you a little bit about some of the trends in the market to give you some context of what’s going on overall. Next, we’re going to talk specifically about some of the results of what we found during our study about organizations who are exploring open source web content management for digital experiences or DX as we have it abbreviated in the agenda slide. Next, we’re going to talk about some specific things you should be thinking about in terms of evaluating open source WCM because there are some differences between that and proprietary WCM. We’re going to talk about that and what we saw from our observations during this project. Then, we’ll wrap up with some recommendations.

To start, basically, the WCM market returned to the wild, wild west. It is kind of a crazy time. Why? Organizations that we speak to as well as that we survey continued to have some problems with their existing web content management implementations. Only about 20% of those we surveyed really claimed to be very satisfied. Honestly, it’s not that surprising for a number of reasons. Number one, the use cases are getting a lot more difficult, aren’t they? They’re looking to use WCM solutions to support increasingly complex initiatives that rapidly change. We’re not just publishing informational websites right now. We’re not just publishing Internets. We’re publishing things like highly personalized websites that are also repurposed onto the mobile platform, and they may have a lot more rich media. They have may be connected to campaigns. The use cases are a lot more complex. Let’s face it, complexity brings more risks for dissatisfaction. Also, as I mentioned, there’s a lot more functionality that’s in play. There’re a lot of other technologies that go into digital experience. WCM is a big portion of it. Five years ago, when I started at Forrester, people were talking about web content management as part of web content manager and strategies. The difference now is when they ask us about web content management. It’s really within the context of a customer experience or a digital experience strategy.

What does that mean? That means that people are interested in how WCM is going to work with other components of the DX ecosystem such as search, such as e-commerce, marketing enablement technologies, digital asset management, CRM, commerce, testing and optimization, analytics. The list goes on and on and on. The integration issues are also adding to the complexity. Then, finally, there’re a lot of organizational issues. Those have been out there regardless of technology. There’s always misalignment between different groups. There’s that classic IT business/marketing conflict. Those are all adding to the complexity and frankly the dissatisfaction here. Definitely, a lot of factors going on, it is a wild west atmosphere right now.

Anjali: As Stephen said it, it is a complicated and it’s an evolving space. In order to make a little bit of sense about this we, Stephen and I, along with Acquia we conducted an online survey of 160 WCM decision makers. We interviewed about seven of those to get even deeper insights and a little bit more qualitative data. This was really the cornerstone of our research. This is going to be the cornerstone of all of the data and the findings that we’ll show you throughout the next 30 or 40 minutes.

A little bit about the survey. We surveyed organizations that were mostly medium to large sized businesses, all had about 1,000 or more employees. They came from a variety of different verticals, government, high tech, financial services, publishing, CPG, manufacturing, media and entertainment, a whole hosts of different verticals. What I found most interesting is these WCMs decision makers weren’t just from IT. Certainly, many of them were from central or distributed IT groups. We had almost 15% from line of business or corporate or online marketing groups. I think that’s definitely a trend that Stephen and I have seen as we do this research. WCM is going to be more of a joint decision between IT and marketing. We’re seeing marketing and these corporate and online marketing groups take more and more of a seat at the table in making these technology decisions. They might not be the only making the decisions, but they’re certainly taking a seat at the table.

Stephen: Anjali, that’s a great point. I think that it’s important to emphasize that this is going to be joint decision. When we talked to some of the larger enterprises, particularly with the ones who have had the most success, it is a joint decision. It’s not one or the other. I think there are some myth perceptions in the industry that everything is swinging away from IT and marketing in the business will make all the decisions here. That maybe the case in certain instances, especially with smaller initiatives, but the fact of the matter is if you truly want to have this universal multichannel experience WCM has to be plugged in to all of those other technologies, some of which I mentioned earlier, that contribute to supporting the digital experiences. Honestly, you can only do that with pretty heavy support from your technologist within your organization. I do think it’s going to be a joint decision. Frankly, within the context of this particular research, we found that to be true again. Those companies which had the most success really did take joint ownership of these projects.

What are some of the greater barriers here to WCM success in general? I think the ones that are most interesting here are probably the ones where we’re in the 20% or above and people were allowed to select more than one response. Corporate politics and culture at 43%, and lack of a company-wide online strategy 33% those are pretty closely related. It’s not unusual for us to talk with clients where they don’t have any kind of online strategy still. It’s still too much from the bottom up, still too much grass roots instead of top down, or they may have very much siloed digital experience initiative. Line of business number one may be trying to do one thing, line of business number two may be trying to do another thing, and line of business number three is trying to a third thing. There’s not a lot of agreement over what should be the priority. When that happens you can understand it’s very difficult to have successful technology when your customer experience needs aren’t prioritized.

Poor budget and resource allocation. That’s a huge problem here. I think that we can all agree that online experiences are getting more complex not less complex. With greater complexity comes, frankly, a greater need to devote resources to it. Sometimes we’ll talk to clients and they’re still thinking they’re going to be able to get away with the same amount of resources that they had when they were publishing pure informational websites. That’s a problem. As these experiences become hyper personalized and much more contextual, you do need some more resources to maintain that.

Next, difficulty integrating the product with other applications. We already spoke to this. WCM is an important part of the digital experience ecosystem, but it’s just one part of it. It has to integrate with all of these other applications like search and commerce and testing and analytics and things like that. When you have difficulty integrating with a product with other applications that’s going to contribute to your dissatisfaction and be a barrier to your success.

Limited flexibility of the product. This can a problem particularly with people who are on older WCM products and haven’t updated them in a while. They’re still feeling constrained by using out-of-the box functionality and they’re not feeling that the products are as flexible as they need them to be. Finally, 21%, it’s that old chestnut, lack of IT business alignment. I have to say, since I started this one, we solved some of the problems, the companies that we speak with they have solved some of those problems. It’s still a problem in some cases.

Anjali: We also asked these same decision makers to rate their satisfaction. I think if you see the data that’s a little bit of a mixed bag. Only 3% said they were very dissatisfied, 9% say they’re somewhat dissatisfied, a fair amount say they’re neutral, about one in five, and many are saying they are somewhat satisfied, and only one in five were very satisfied. I think, Stephen and I, when we talk we really see that very satisfied as what organizations should aspire to, the goal of what you want your WCM solution to be. We see only one in five have made that, actually believe that their solution is very satisfactory.

Another thing I find interesting about this data is when we cut this by the WCM decision makers who are from marketing or from the line of business versus WCM decision makers who are from IT, we see a big difference there. IT is much more likely to be satisfied by WCM solutions. I believe only 8% say they were very dissatisfied or somewhat dissatisfied. When we compare that to marketing or line of business decision makers, almost 30% say they were dissatisfied or very dissatisfied. There’s a little bit of a gap in perception and gap in satisfaction As we see marketing becoming more involved with the WCM decision making, I think we’ll see this satisfaction become a little bit more neutral or maybe even a little bit more satisfied.

I think it’s also interesting we cut the same data by solution type. Which of these 160 decision makers were using open source, proprietary, or home grown solutions? I think this one’s interesting. In a lot of places they’re very similar, but when we look at proprietary they hit a lot of highs. They’re very high in the somewhat satisfied, but there’s a huge drop off in the very satisfied. Only 8% of WCM decision makers who have a proprietary solution said that they were very satisfied. It’s a little different, though, when we look at open source. We see they are a little bit steadier. There’s 37% who say they are somewhat satisfied and there’s no drop off, about 37% also said that they were very satisfied. Home grown again is a little steady too.

Stephen: Great. Thanks, Anjali. I think we agree here that, number one, we have rapidly changing needs that need to be supported by WCM. Number two, there are a lot of issues contributing to WCM dissatisfaction that go beyond just plain technology. There’re a lot of people process issues that are going on as well, governance issues. I think it’s very interesting to see the fact that the open source respondants don’t have the same type of lows that the proprietary respondants do. I think that’s quite interesting. Then, finally, I think the other important point from that section is that there really is a need for integration. It’s a much biggest ecosystem that we’re dealing with than before.

Next, let’s talk about some specific examples of why organizations are exploring open source web content management for digital experiences. That’s what the DX stands for. Open source WCM is gaining traction as a viable enterprise option across verticals. This is something that even outside of this study we started to see a little bit. At Forrester, we talked to our clients, there’s bit more interest over the past 12 months in open source, a little bit more serious interest I guess I’d say. There’s a more serious interest in SAAS, another alternative delivery model, a SAAS WCM.

As I mentioned, five years ago it was difficult to find those larger organizations that had deployed open source web content management as their primary WCM solution. There were always plenty people who had deployed open source WCM, but they were supporting smaller microsites or smaller scale initiatives. For this particular research, pardon me, we spoke with several organizations, large scale organizations, that are using open source WCM for their customer experience. I think one of notable ones was a large supermarket chain here in North America. Another one was an international pharmaceutical company. Another one was an international electronics manufacturer. There were really fascinating examples here. The real question is what are these organizations with successful large scale WCMs what are they doing differently now? After speaking with those organizations we’ve boiled down the keys to their success to a couple of factors or a few factors. That’s what we’re going to speak to you about in this particular section.

Anjali: I think our data really proves the point that Stephen was just making, that here we asked what type of solution are you using for your primary web content management solutions. This is primary. We used to see open source WCM really being used as maybe a secondary solution or a microsite, as Stephen mentioned. Here in this question in our survey we asked what is your primary WCM solution. You can see about 27% say they’re using open source, 28% say they’re using home grown. I think this is a little bit of an anomaly. Like I mentioned earlier, we surveyed medium and larger sized businesses. I think many of the people who are using custom coded are home grown are much more of the medium sized businesses. The larger enterprises tend to go with proprietary or open source. You can see proprietary has about 45%. I think it’s also interesting to think about because even those who didn’t choose open source as their primary WCM, almost 60% did consider an open source solution. Even if they didn’t choose it, open source still had a seat at the table. They’re still under consideration. I think that’s an important point. As we see open source gaining more and more traction in the WCM market we can see this interest increasing.

Ant point I find interesting about this slide is when we asked all of these respondants why are they choosing an open source solution, it was interesting because more open source users chose open source WCM for a web redesign or web replatform effort more so than the proprietary or custom coded users. They had some other important point. We can see that open source is really gaining some traction in the market.
Stephen: I think the interesting thing there, though, I want to reiterate, I also felt that the home grown solution bucket was a little bit larger than I would’ve expected and certainly larger than I’ve seen through our customer inquiries I would say part of it might also be explained that it’s possible that some of those who are citing custom coded or home grown are using open source as a base. It’s definitely an interesting slide.

In a lot of ways the whole open source debate it’s kind of like Coke versus Pepsi. I’m probably dating myself, Anjali. I don’t know if you remember those. How many people remember the Pepsi Challenge back in the 80’s where they had people blind taste test the two colas. Frankly, if Coke versus Pepsi has always been one of those things were people are very religious and they’re calcified in their opinions. I am a Coke person The Pepsi Challenge showed a lot of people who claimed to be Coke people, but yet when they were blindfolded and they tasted the cola they actually preferred Pepsi.

In some ways, the open source debate is a bit like that in that proponents of either choice are often hardwired to believe what they believe. They like what they like often for valid and defensible reasons. Some organizations gravitate towards open source WCM based on factors side like in a corporate culture, they kind of like that exploratory type of project. Some of them have budgetary limitations and they’ve been mandated to look at open source because with the belief that it perhaps is going to save them money. There’re some organizations that simply have a greater comfort factor with open source due to previous experience with other open source applications such as operating systems as well as their familiarity with the community principles that underpin these solutions.

For other organizations, the opposite is true. They often lean towards proprietary solutions because they believe sometimes justified, sometimes not, that it’s the best way to mitigate risk, that they believe that they code is tested. It’s more secure. It’s more off-the-shelf. They’re not going to have to do as much customization. They’re used to dealing with the vendors. Some of the organizations we speak to they’ve dealt with some of these larger vendors for years. There’s a comfort level there and there it’s more off-the shelf. Again, its’ this whole idea of getting out of those calcified positions and really thinking about what the best product, what the best fit is, what the best product is for your particular needs. That’s where we found more satisfaction.

Organizations are definitely expressing interest in open source WCM for digital experience as we found in the survey. We asked them some of these responses why did you consider an open source WCM solution. I think the answers here is really interesting. Sixty-nine percent chose it because they believed that there was a lower cost. I’m not necessarily saying that’s true. In fact, when we did some of the qualitative interviews, our interviewees really encouraged people to look at more than just lower cost. I think we can probably all agree that there’s often a lower upfront cost because you’re not necessarily making a big license purchase. However, it’s really important to look at total cost of ownership. You may have a lower upfront cost, but depending on what your cost of customize and implement you could end up with higher costs at the end. I’m not necessarily saying you will it all depends on what product you select and how well it’s aligned with your needs. I think lower costs is one of those things that is more than just upfront license fees, and I encourage people who are thinking about this in terms of lower costs to really look at the bigger picture of total costs of ownership.

Another one here, easier to customize. That’s something that we definitely heard through our qualitative interviews. A lot of the companies that we spoke to were very happy. They had needs where they had to do a lot more customization. They felt that their web presence was really one of their main differentiators and they weren’t going to necessarily be satisfied with out-of-the box functionality from their WCM. They felt that they needed to customize in order to support some of the more differentiating functionality. People were definitely talking about that.

Reduced dependency on software vendors. What some of the respondees meant here was that they didn’t necessarily want it to be dependent on the software vendor for new features, didn’t want to be dependent on them for their release cycle, even bug fixes, and they were happy to do it themselves. Scalability of the solution. There were some of them who felt that they were going to have a better luck with scalability for their particular needs. Availability and viability of the developer community. That’s a huge one. That’s something we heard again and again in our interviews. One of the reasons why people chose open source WCM. Those were some of the interesting responses within why people were choosing the open source WCM solutions.

Organizations are definitely exploring. They’ve definitely had some success. What we did after this was we wanted to point out, we wanted to gain some insights from those we spoke to, those we interviewed on what they looked for, what helped them, what were the evaluation criteria they looked at that helped them have some open source WCM success.

Anjali: We started first by asking who didn’t consider, who didn’t buy open source what was the stumbling block, why didn’t they choose or consider an open source WCM solution. We though this could help frame the rest of our presentation about we need to think about some of these stumbling blocks and how to think about the myths a realities of some of these concerns.

You can see at 55% many people, we heard this often throughout our interviews, that in the beginning security of the software was a concern with open source. We’ll talk about a little later about some of the myths and realities and how organizations we’ve talked to who have had success have worked around that. Lack of ability for enterprise service and support. I think that’s at 31%, it’s the second most cited reason. That’s another issue that we heard come up again and again because open source is different. You don’t have the vendors you just call up. It’s more about coming up with a comprehensive plan for service and support and how are you going to work with open source in that way. It’s a little different that what many of you who have been using proprietary solutions. It might just be a little different than what you’re used to. We’ll talk a little bit about that later.

We have some other issues here like legal issues or inexperience with open source in general were stumbling blocks for some people. Lack of skilled resources. We’ll talk about that too later, about how people are coming up with support plans and coming up with a plan to actually support their open source solution. Product immaturity at 16% is another one that some people were concerned about, and overall complexity and difficulty of adoption.

We’ll start with security. What I thinks interesting is that was the number one cited stumbling block for people who didn’t choose or consider an open source solution. When we actually talked and surveyed people who are using open source they didn’t see to express that same sentiment. About 50% of organizations with open source actually said that open source had neither improved nor decreased their level of security. They said it was pretty much the same. One theme we heard over and over again as we actually did the qualitative interviews was that security concerns weren’t that different from those they’re faced when they’re implementing previous proprietary solutions. I think that’s an important point to consider. Security is going to be a problem or an issue no matter what type of solution you have. They might just have slightly different issues, but it’s still always going to be a concern.

I think the main thing that when we talk to organizations who had success with open source is that they came up with a plan for their security. They had a plan for perhaps one of the large supermarket chains we spoke to had a third party hosting their solution. That eased their fears a little bit about going with an open source WCM solution. Others we talked to thought that the security patches that were being released more quickly by the vibrant developer community they felt like that mitigated a lot of their concerns. Other organizations we talked to had independent security audits because, let’s face it, with an open source solution you have access to the entire code base. You don’t necessarily have that with a proprietary solution where you only have access to some but more likely you don’t have access of this any of this code base. They were able to mitigate their concerns by actually examining the code base and they felt that that mitigated a lot of their security concerns.

Stephen: The interesting thing there, Anjali, is that they didn’t dismiss the security issue just out of hand. They actually dug in and did some homework and did some comparisons on open source versus proprietary. Actually, we’re seeing this within SAAS as well. SAAS we always used to hear about I’m afraid about the security with SAAS, and people are now starting to think, “Why don’t I at least dig in to the model and see how it compares, and then I’ll make a more informed decision.”

Some of the other points to consider here, level of functionality. I think that when you compare some of the open source products to some of the higher end proprietary products you are going to see a different level of functionality. You’re not going to see that huge array of functionality with some of the really high end proprietary products who are, let’s face it, are in a functionality war. I think the real thing you want to question is how much of that are you going to use? For example, the proprietary vendors are really starting to focus on marketing enablement type technologies that allow you to multichannel campaigns and run really contextual websites and have a lot of personalization and things like that. You’re going to have to ask yourself are you going to use all that functionality if you’re considering open source product versus a proprietary one? Frankly, you could even use this argument if you’re comparing two different levels of proprietary products or two different levels of open source products. How are you really going to use? What you really want to do here is have somewhat of a five-year plan, one that’s a little bit solidified in the coming year and one that may be a little bit more vague as time goes on. You need to really understand what you’re really going to be using over that five-year period because five years is roughly the average lifespan of a WCM implementation.

If you are buying a product based on functionality that you say that’s really cool functionality, but you really don’t know when you’re going to use it that could be a warning sign that you may not ever use it. I think this is one of the big problems with enterprise software all the time is you’re buying functionality that you don’t need. Another one of our interviewees, one of the pieces of functionality that we’re looking at were the user interfaces, and they felt that the user interfaces of the high end proprietary product they were looking at that they were more polished than their open source counterparts. The open source was enough of a fit that they decided that it was worth having to do the extra configuration and customization of the open source UIs because of all the other benefits they were going to gain.

Integration, another huge point here. As I mentioned earlier, you’re going to be integrating WCM with all kinds of other products. What you really want to do is figure out what types of integrations you need, come up with a reference architecture and figure out where those dependencies are. Then, understand where there is integration functionality that you can buy off the market through prepackaged connectors and where you’re going to have to code by hand. That’s going to inform you whether or not you should go with an open source or a closed source product. If you can’t buy much of the integration off the shelf and you’re going to have to do some pretty complex implementations, pardon me, that’s an indication you may want to look closer at open source WCM.

Finally, then customization piece. That’s something that we talked about. It ties very closely in with level of functionality. You may get a little bit lower level of functionality and you may have to do more customization. On the other hand, you may require a lot of customization no matter what. This is the media entertainment story. For years, media and entertainment has used open source more heavily than some of the other industries that’s because they have some very specific use cases and they found open source was easier to customize than a proprietary product. You need to think about what level of customization you’re going to have. If you’re going to have to do a lot of customization no matter product you source, open source may be a better alternative for you.

Ultimately, integration is very important here. Just a little bit of data to emphasize that. We asked the respondees in the survey which of the following would you like to see included in the same stack as WCM. I think we all have seen some of the higher proprietary vendors are really building their digital experience and customer experience portfolios to not only include WCM but some of the adjacent technologies that we were talking about earlier. Ultimately, interesting result of this survey is that 35% said we’re not interested in having a whole lot of technology bundled with our WCM. We prefer best of breed. Why do they say that? Because of the fact of the matter is none of the people that we talked to or that I talked to as clients can really afford to do a complete rip and replace and replace all of their digital experience technologies with technologies from a single vendor. The fact is they already own some of these technologies, so they want to be able to do integration and they want to be able to integrate WCM with CRM and search and email campaign software and social network and platforms and web or social analytics. This digital experience story is going to be an integration play for the short term. It’s the short to medium term, frankly. When you think about it way it’s not too surprising that a lot of our respondees wanted a best of breed approach.

Anjali: It think the last point to consider is resource availability. I touched on that earlier, Stephen has mentioned it too. Working with open source is different. If you’re an organization who’s used to working with proprietary vendors, somebody you can up in the middle of the night, it’s not like that with open source. The organizations we’ve talked to who’ve had the most success with open source the first and foremost start it out with a plan of how they’re going to support their open source WCM solution.

One of the biggest workarounds that we’ve seen with the organizations who have had success is using a third party for-profit business who’s dedicated to providing open source support, whether it’s for implementation, hosting, or troubleshooting. That’s helped mitigate a lot of those concerns. That’s something that we heard over and over from interviewees. They were nervous about open source because there’s no enterprise support, but then the third party has helped them fill that gap and that’s how they’ve worked around those issues.

Lot of our other interviewees and things we’ve heard through inquiry and other things have really been positive about open source support because of the vibrant community. Of course, this depends on which open source solution you provide. Some have more vibrant communities than others. If you have the right community you can use them for not us trouble shooting or Q&A, but you can also use them for their community authored modules. That might be a really great way to add functionality very easily onto your platform without having to do too much work. I think these are some of the ways we’ve seen organizations work around their concerns over enterprises level support.

Stephen: Great, and a very important I think. It’s actually one of the most important ones of the factors that you should consider is that support issue. Let’s wrap up with some recommendations, wrap up our portion of today’s presentation.

Number one, definitely evaluate open source. I think it’s gotten to the point in its maturity where it’s worth looking at. I think one of the things you don’t want to get into is that calcified mentality of, my gosh, we’re not going to do open source. Frankly, the other way we are going to do open source. One of the interesting things that came out of this survey, one of the interesting observations that came out of the study was people said, “Don’t just evaluate something because it’s not open source.” Evaluate the products and evaluate them on their own merits and based on things like functionality as well as support and resource availability and make your decision that way. It’s must better than falling back into that religious mindset.

Number two, one of the really nice benefits of open source is that you can get it a community version of the product and that leaves you free to pilot it. You can actually pilot it and see who well it works. There’s also organizations we’ve spoken to who have installed open source alongside proprietary and they decided to see how it would go that way and they gradually expanded their use of open source after they found some success.
Third, engage with the community for information and validation. Any product that you are evaluating you should be able to see how active the community is, how responsive are they, are they willing to answer your questions, are they contributing a lot of community authored code, are there interesting references, are there people who like your company who are using it? Definitely engage with the community. I think the vibrancy of the community is a huge factor that you should be taking into account when you evaluate open source.

Fourth, explore those open source options carefully like Anjali said, particularly if you’re used to dealing with some of the commercial vendors. The support options is one of the big differences here, so you want to see that if you are planning to move away from the proprietary vendor support model you’re going to want to be very sure in how you’re going to replace that with some open source options.

Then finally, focus on more than just cost reduction. Open source is not necessarily free I think of the interviewees that we spoke with the ones who had the most success here looked at open source as more than just something that they could get for free. Of course, they looked at costs, but it was more within the scope of total cost of ownership rather than just that very narrow slice of upfront license fees. They looked at things like integration and maintenance costs and support and culture and some of the other factors that we spoke about today. If you keep those factors in mind rather than just very narrow slice of upfront costs, I think you’re going to be at a much better position to evaluate whether or not open source WCM is the right choice to support your publicly facing and digital experiences. That concludes our portion of the presentation. Now I’m going to turn things over to John.

John: Appreciate it. Fun to work on this solution and work on the survey with you guys. With that, I’d like to take some time to highlight a few more interesting points from the survey that we collected from our side and also talk a bit about our announcement last week around OpenWEM and finish off with a couple of case studies.

You’re probably wondering to yourself for those that chose to participate in the survey what were the list of platforms that we were talking about here. Here’s the full list and what’s interesting is that Drupal is highlighted in the top five and represents the majority of the respondants for those that offered this information up for open source. It’s important to understand that although it’s really talking about all the different technologies that are available from an open source perspective that Drupal came out on top and is in the top five of all platforms that are represented in the survey.

Other interesting statistic was the business drivers for new WCM solutions. Regardless of what you’re using there the top five for the main drivers for using a new WCM solution were limited flexibility, total cost of ownership, poor and inadequate functionality and the ability to quickly innovate. We found that was interesting because when we talked to customers those are most often cited for the reason they switched to open source. I think some of the top challenges folks and business drivers for folks moving are also reasons that folks are looking at Drupal specifically.

We also took a look at engagement by the solution type mapping proprietary and open source. One interesting point we found here is there have been some perceptions in the market that open source platforms like Drupal aren’t ready for the shift towards digital marketing. If you look at the data in the box below it’s actually statistically significant that it’s virtually no difference between folks that are using these platforms for interactive marketing on multichannels whether that’s proprietary or open source.

Barriers to success by solution types. If we split data again between proprietary and open source. I wanted to highlight a point at the bottom there in the orange box around limited flexibility of the product. What are the greatest barriers to success? For limited flexibilities, we think there’s extreme contrast there between the 30.6% around proprietary highlighting that versus only 9.3% for open source. Flexibility really leading the way with open source and Drupal.

A couple highlights around when we just took a look at the WCM decision makers who were using open source specifically with the majority being Drupal, we took a look at these how strongly you might agree with the following statements around open source. Ultimately, some of theses when we combined strongly agree and agree open source came on top for reduced dependency on software vendors at 77%, combined open source provided us more flexibility 73% combined, open source lowered our costs 70%, open source is easier to customized 70%, and open source scaled easily at 66% when you combined strong agree and agree. Then, a few others, open source made it easier to innovate quickly that came at 67% combined. Open sourced shortened our time to market 53% and viability of open source developer community was key to a successful implementation that was also at 53% combined. Some pretty strong statistics and a strong showing for those that chose open source platforms like Drupal.

I wanted to take another quick look at the state of the market, more specifically for the Drupal platform. We are seeing that inside many organizations the web is broken. There are lots of technologies out there that might be really strong in an Internet or an Extranet of if we’re doing wikis or doing your main corp.com site or an e-commerce site or a microsite. At the end of the day, these end up being stoked by technologies that really create islands of excellence, if you will, and it really costs a lot more to develop and create the extra piece needed to develop across these silos. They’re also disconnected from a reporting perspective if you think about a chief marketing officer that wants to know everything that’s happening in their hub of interaction on the web.

Where Drupal really shines is taking all these different siloed islands of excellence and really delivering integrated unified customer experiences out to your B to B to customers across web content management, social, business software, and e-commerce. Drupal really is a unified platform for content community and commerce which ultimately we think results in part of the reason you saw that open source gets the highest marks for folks that rated it very satisfied.

It also goes along quite nicely with some of the research that come out from Forrester this year. This is one of their recent reports which talks about defining your digital experience architecture, the need to manage, engage, and measure all of your customer interactions. If you look at engage I wanted to highlight three areas, content targeting, social, and transactions, we believe map fairly closely to content, community and commerce. If you are setting up a digital experience architecture for success we agree with Forrester that those capabilities need to be inside of a single platform to maximize those customer experiences.

Last week, we made an announcement around a new approach we’re calling OpenWEM. OpenWEM really is that ability to create these best of breed customer experiences with unlimited Internet freedom and flexibility across the full spectrum of the customer journey. As you can see here, we consider that customer journey from attracting and engaging folks on the web to associating and influencing them within communities, prospect to prospect, or prospect to company, and then the last mile of engagement in e-commerce being able to sell, retain, and resell to them online. Across these market segments Drupal is a unified platform for delivering that.

Again, some of the additional Forrester research from earlier this year that talks about the concept of the open web and the ties of the concepts of the open web to open source technologies. We fundamentally believe that to implement and take this OpenWEM approach that you really need to embrace the principles of the open web. Those are providing an open culture and community of thriving developers; Having open APIs to leading best of breed engagement tools so you can, as we saw in that statistic, offer up a best of breed solution; have accelerated innovation on the web that doesn’t rely on proprietary models for development and really relies on that open culture of contributing problem solving and code back to the community so it gets to customers faster for these experiences. Then, we fundamentally believe in responsive design mobile approaches and best practices as a very agile way to create those mobile experiences. Drupal really embraces the open web concept.

Overall, from an OpenWEM perspective of content community commerce, we believe it does it offer infinite freedom and flexibility to create these digital experiences online. There’s freedom to take your code, to take your content, and if you need to take it to another hosting provider if we’re not getting done. There’s no vendor lock in here with an open SAAS model. There is unmatched innovation to be able as new technology is brought to bear to be able to get that technology into your website as soon as you need it for content and campaigns. That’s through those 20,000 Drupal developers that are continually evolving the Drupal platform in real time.

We talked about the unified platform across the full customer journey. We think Drupal’s a great platform for best of breed experiences because it’s has a data driven modular architecture and content model that allows you really to very easily plug and play these best of breed marketing tools and input and export data as needed as Stephen talked about. Acquia in particular has been building a very mature cloud infrastructure and cloud business model over the last five years. As you’ll see, other proprietary vendors roll out B1 models this year and next year. We believe we’re absolutely ahead of the game for delivering digital experiences in the cloud if that’s your next route.

Then, everything we do is about simplifying digital experiences and simplification usually translates to a lower cost of ownership. We believe that developer community offers a better supply of highly skilled developers that are cheaper. When you think about the ability for them to ultimately develop across content, WCM, social, business software, and e-commerce on the same platform that that’s going to create better experiences. It’s also going to lower your total cost of ownership. This is where we think the Drupal platform really excels.

Ultimately, how is taking an OpenWEM approach different from proprietary approaches? Are you experiencing any of the symptoms such as limitations around road map or road map trade-offs by proprietary development teams? Do you issues around complex customizations even after you’ve already paid exorbitant license fees for the software? Do you have multiple platforms installed today for content community and commerce rather than building it on a unified platform? Do you consider your vendor lock in a problem from all-in-one solutions stacks that have control of that customer data and feel a bit trapped? Are you having issues around immature non-existent cloud business models or ongoing maintenance fees from perpetual license models that are your only option? Many of those issues if you are experiencing any of those issues this might be the platform and support model to consider.

Last week, in announcing the OpenWEM approaches and customer strategy we did talk about some tools and products that we launched in support of it. we have a new product called Acquia media cloud in the content or web content management market segment. That’s a new SAAS offering to manage rich media assets within the Drupal platform to do things like asset tagging and optimized transcoding for video across endpoints, even digital rights management.

On the community side, we announced a new version of Drupal Commons. Drupal Commons 3.0, very new with new responsive design templates for mobile collaboration, advanced content recommendations so folks in the community can get interesting content to themselves fast, and improved moderator productivity so that moderators can outsource SPAM blocking and community involvement to folks that they trust from an access control perspective. We continue to go build out ecosystem, so we announced Digitari as a new digital agency partner for us as well as Badgeville integration for doing gamification and reward systems within Drupal.

We think Drupal’s a great system for flexible agile marketing as well, like I talked about, the Pinterest example. In February, Pinterset hit 10 million unique users. As of March, folks believe that having Pinterest image pinning capabilities on their website was important for digital marketers, and by April 15 Drupal sites already went live with Pinterest integration, so that really is the definition of agile marketing on the web.

We also have integration with best of breed tools. Acquia is helping build out this ecosystem such that whether it’s CRM, analytics, marketing automation, campaigns and social, and whatever’s next, we’re going to offer APIs and continuous integration points to the latest versions of these technologies so you can use that Drupal module platform to plug these technologies in, and then Drupal if very powerful to be able to get that content out to your campaigns, out to your microsites, and across all the channels at the top here.

We have a very mature cloud model, as I mentioned, we’ve been building over the last five years. It’s based on Amazon web services. What I’m showing here is for developers there’s a whole host of developer tools to be able to create instances, develop them, and then publish, so take instances from development to staging to production and back and forth quickly and easily. We also have the Acquia network to be add additional technology such as Acquia search or SPAM blocking technologies, multibarrier testing, real time analytics. We have partnered with a lot of technologies to be able to get that quickly and easily from the Acquia network in a centralized location.

A couple quick examples across content, community, and commerce. I have the first example around content, specifically. Florida Hospital, one of the largest hospital chains in America, they were able to create mobile content experiences using responsive design techniques. In this case, what’s really interesting about their implementation was that they were able to allow their patients to understand exactly how far they were from a hospital or what the wait times were from their local hospital so they could pick the one that was most appropriate. They’re doing a lot of advanced use cases in health care along mobile.

For community, we have Daimler which set up a customer community to be able them build out their customer requirements for a particular model of their compact car line. They set up polls and interactive forums for prospects, to talk to prospects, prospects to talk to engineers inside of the company and really understand from the consumer themselves really a big data problem rather than doing focus groups to understand from thousands what they were looking in the next vehicle, so being able to set up those customer communities online and Drupal as well.

Cartier is using Drupal and the Drupal commerce distribution specifically for that to create these rich commerce and tranformative shopping experiences that embed content rich media, images, to really help folks in the last mile help them decide exactly what product is right for them. It’s a lot more engaging when you think about the high end customers that are shopping online for Cartier. It’s helped them increase their conversation rates by being able to use the rick content in Drupal.

I want to finish up right before we get to a few questions. We have some upcoming customer case study webinars. I went through those pretty quick. Customers that are initiating an OpenWEM approach for delivering digital experiences on the web. November 6th we have the Grammys, How Drupal Acts as a Media Hub for the Grammy Awards. November 25th, Humana, Fortune 200 company, How Haman is Using Drupal to Drive Repeat Visitors with Personalized, Multi-Channel Campaigns. You can go sign up for those now at the link. I think both of those will be very in depth reviews of customers embarking on an OpenWEM strategy.

Certainly, please download the full research paper. We went through a lot of it, but quite honestly, we probably only went through 50% or less of the research that’s in the paper that’s available today and that paper is titled Is It Time To Consider Open Source WCM For Digital Experience. You can go ahead to a microsite we created called OpenWEM.com and download that paper for free today. Please do that.
With that, I would like to open it up to some questions. I will who those questions are addressed to here. We have a few that have come in. For Stephen and Anjali, I’ll start off. This question says, “You mentioned some in person interviews around open source survey participants. Can you offer any specifics on the types of enterprise customers or verticals you spoke with and their specific reasons for choosing open source?”

Stephen: ... tended to be cite that reason. Traditionally, that’s been media entertainment. I think I mentioned that in the context of the presentation. We saw it in other areas as well. The supermarket chain, retail, they had some really specific needs around their UIs. They had a globalization, localization issue where they were going to have global content at the corporate level, but they wanted the local branches to be able to create their own content but also not override all of the global content. They did some customized user interfaces to take care of that. They also had to, I believe, had to integrate with some print-on-demand solutions in order to print out circulars. That is the type of integration they did. I think one of the other companies we spoke with they were in a regulated industry. They had the need for some fairly highly customized workflows. That was the type of thing that they were looking at. For some, they said they had a mandate. They were supposed to explore open source, and others they felt that they hadn’t gotten their money’s worth out of older proprietary platforms and they wanted to explore other options where the costs were done a little bit differently. As I mentioned earlier, I think the ones who were happiest didn’t necessarily choose open source or proprietary. They were open to both they just chose the best fit. They chose the product for the best fit.

John: Great. Another one came in. Cloud to planet models are growing in popularity for these types of solutions, but we didn’t have a lot of data or questions that focused on mature cloud models. Do you see that as a factor in some of the enterprise decisions?

Stephen: Eighteen months ago, I would’ve said no. Around that time we did a survey, an internal survey, of WCM decision makers. Most of them, honestly, didn’t care about cloud. They said we just want something that works. We’re interested in supposing our customer experience needs. That’s changed a bit over the last 18 months, John. Now people are starting to want to outsource some of this, but they’re still, I think, more concerned with getting it right and they’re also interested in the different flavors of the cloud, whether it’s a public cloud, a multitenant model, or whether it’s a private cloud. We’re getting more questions about it, but I still don’t think we’ve reached the level of maturity in WCM where they’re ready to wholesale outsource it. I also think that a lot of the organizations that we speak to understand that even if they should get a cloud WCM, it’s going to have to be integrated in with some of the enterprise applications that they have on prem.

John: Great. Here’s one. If I’m in the RFP stage for WCM and only have the resource to evaluate a couple solutions, what are the top three questions should I be able to answer to short list enterprise scale open source?

Stephen: That’s a good one. I’d say first of all it’s a functionality question. It always is. Is it going to meet my long-term functionality needs? As I mentioned in the presentation, answer that question less as does it have enough functionality to meet my needs, but also if you look at it does it have too much functionality to meet my needs. Do we really need all of this functionality? You may very well, but this is typical for enterprise software. Let’s face it, people tend to over buy functionality, so look at it from both angles. I’d say the second one for open source, we mentioned it in the presentation, it’s community. What is the vibrancy of the community? You don’t want to pick an open source product that the community isn’t helpful or there’s a smaller community and it’s not very vibrant and nobody’s creating community authored modules. Then, I’d say the final one is the support model. The support model is, let’s face it, it’s different from the typical support model when you’re getting support directly from the software vendor. Those are the three things that you should be asking yourself when you evaluate proprietary versus open source.

John: Great. One final question. I’ve seen some perceptions in the marketplace that open source WCM can’t scale. It’s not performance. Developers are needed inhouse or extensive customization is required. Would you say these blanket statements are accurate based on your research?

Stephen: No, I don’t think they are. This is the trouble with blanket statements. I don’t think you can generalize and say proprietary is more scalable than open source or vice versa. I think you have to look at the product itself. There are some high end proprietary products which have proven scalability. There are other proprietary products which have catered more to SMBs and they don’t have the necessarily proven scalability that their higher end counterparts have. It’s the same thing with open source. We talked to some companies or some organizations that had had a lot of success with certain open solutions, and then there are other source solutions which have more of a track record at the SMB level, again, the same thing with customization. As I mentioned when I was answering the first question, I encourage people not evaluate open source or proprietary as whole but instead evaluate the individual products.

John: Excellent. We’re a couple minutes over. I want to thank folks on the line for staying on and hope you enjoyed the presentation. Definitely, download the paper at OpenWEM.com. Also, I’d like to offer a special thanks to Forrester. It’s been fun working on this project, Stephen and Anjali. I think there’s some great work here and some great knowledge to be gained from the research and the reports. Thanks for joining us. With that, I’ll turn it back over Hannah.

Hannah: Thank you everyone for attending and thank you Stephen and Anjali for the great presentation and participating with us.

Cost-effective Development with Drupal Commerce [October 30, 2012]

Click to see video transcript

Speaker 1: Thank you for joining in on the webinar today. I'm going to go over a few slides and then pass it over to Simon. Today's webinar is: Cost-effective Development with Drupal Commerce, with Simon Bostock from Deeson Online.

Simon: Hello, everybody. This afternoon there will be a short webinar, hopefully followed by some questions around cost-effective e-commerce development. This is particularly with Drupal Commerce, and obviously with Acquia as well, and the idea of the talk really is to talk about how we worked on particular projects, but also about some other projects, and how we try to make sure that we are doing the right things and we are working … devoting our energy towards the right thing.

I work for Deeson Online; we are Britain's oldest Drupal Agency. That’s probably not meant to be doing Drupal longer than everybody else, but we've been around since 1959.

We are Acquia Enterprise Select Partner and we built that to more than 200 Drupal sites and we'll be doing e-commerce sites since 2002, on that picture there. I'm the one in the glasses at the bottom, and that’s the team, the team who worked across the Shepherd Neame Project, which we'll be talking about quite a lot.

The e-commerce work we do, we did the standard bricks and mortar style experiences, so shop replacements, but we also really specialize in doing membership sites as well, and also sites with a lot of digital content, and probably talk a bit about those as well. The project that I'm specifically talking about today is the Shepherd Neame Project. For those of you from outside the U.K., Shepherd Neame is Britain's oldest brewer that’s been going since … I can't remember when actually, but about 400 years, or something like that.

They have about 360 pubs and hotels across the Southeast of England, they also own a number of the biggest beer brands in the U.K. This particular project was a complete redesign and a rebuild of their old site and moving from some Legacy concept management system towards Drupal, and the key parts about it that made it interesting is it's got 360 micro sites contained within the main site. That’s all connected where each pub has its own micro site. The interesting bits which we'll talk a little bit actually are, obviously, the e-commerce part but there's also some geo location features, such that when you go to search for a pub, the website knows where you are. There's also an online job application tool.

That’s me, and the reason I give you my contact details is if anybody would like to ask any follow-up questions that you don’t think of today, then my email, and it's just there. There's also a blog post on our site, that address at the bottom, which will have the slides on immediately after the talk and then also a blog post about e-commerce in general.

We are talking really about Drupal Commerce. Drupal Commerce is a framework, it's a framework for e-commerce things in Drupal, and for the perspective that I'm talking about today then really this is a set of building blocks really, a set of materials that you can build a commerce build out of it. I should have said actually, earlier on, the talk is really, we have done webinars in the past, related to e-commerce which were much more aimed at designers and technical people. This one is aimed at people like me, people who are leading on projects, and I'm a producer, at Deeson Online, so my job is really to make sure that we come and spending money effectively and efficiently, and that clients are getting the best deal for it.

That’s where I'm coming from with Drupal Commerce. Drupal Commerce is very much a starting point for e-commerce development, not the end point. You get a number of tools that you can use to build e-commerce sites, but it doesn’t actually do anything much out of the box.

The three bits I want to talk about with why we use Drupal Commerce and why we chose to use it on Shepherd Neame projects, and almost certainly will choose to use it again in the future. It's one about my job, about using our energy wisely, and energy here means energy, but of course it means money as well. It's about getting the most value for the money that you're spending.

Will talk a bit about how Drupal Commerce integrates with the rest of, both the business, but also the main website. To finish, will talk about some really big-picture stuff about what's happening in Web experience management.

If you'd like to find out about the features, about Drupal Commerce then there's a link just there. Today I won't be talking too much about the features, I'll be talking about the benefits, and as I said, it's the first real thing here is about using energy wisely.
Drupal Commerce like Drupal is an Open Source thing, so it basically means that a lot of very hard work, a lot of the boring stuff has been done by Drupal Commerce, and when I think about Drupal Commerce, I really think that what it's done is exactly the same as Drupal.
It's taking all of that really boring stuff, put it in the background and all of the bits are the same on every e-commerce site across the planet and Drupal Commerce takes control of that and it handles that so that we can actually think about the really interesting things, which is about making good user experiences and making sure people can complete the tasks that they want to do.

As an example of this, the Shepherd Neame main site, what we've got there is a couple of the landing pages, and this is the Shepherd Neame Shop, and as you can see, the Shepherd Neame Shop really neatly inherits all of the design work and the styling that’s gone into the main sites.
Immediately every … we've made a huge site and so we've carried out a bit of branding and then the big redesign and the rebrand of the Shepherd Neame property and the fact that the shop inherits all of the styling from that, there's a huge cost saving for us.

In terms of the other stuff that’s out there, theres loads I think probably hundreds … more than a hundred there anyway, contributed modules to handle all the little bits and functionality that perhaps Drupal Commerce doesn’t do out of the box.

There's also a set of default functionality, and a bit quite active community as well. We found that when we were building this particular site, that some of the pieces of functionality we wanted didn’t exist, but they were in the issue queue, and by the end of the project, before we'd finished some of the … most of the functionality that we wanted had been contributed. It was quite a long project because it was a very big project. It was eight months in the end. At the beginning of the project we raised an issue with the Drupal Commerce community and said we would very much like to be able to do that.

A number of other people said, "Yes, we would also like to be able to do that," and by the end of the project those bits of functionality had been completed, not all of them, but most of them has been completed without any real extra effort from us.

Drupal Commerce is … if you know Drupal, you know Drupal Commerce, or if you are Drupal developers, you know what they're talking about, then they'll also be able to work with Drupal Commerce. As I said before, this is really important for us because it means that we can just concentrate on the really interesting things as opposed to the boring stuff.

If you are considering Drupal Commerce it's probably worth thinking about a few things, and there are some defaults on the Drupal Commerce thing, for example, there are no shipping or payment options, and there are contribution modules to do that, so there is bits of extra functionality that you can plug in, but out of the box from a standing start it doesn’t have any shipping or payment option functionality.

There are some odd things as well. For example, the product entity and product display, this means that when you want to display a project from the Drupal Commerce site you have to create the product and then you have to create the display of that and our clients have told us that this is slightly unusual, and there are very good reasons for this, it means that you can have different sizes of clouds, for example, and variance of products.
Then again, clients have said to us that it feels a little bit odd at first. I think in the end everybody agrees that it's probably the more powerful way of doing things, but worth considering in the beginning. The resource is something called Drupal Commerce Kickstart, and actually that shot there I'm showing you, Commerce Kickstart that’s what it looks like out of the box.

If you have a client that has a URL there, if you visit that URL there you'd have to play around with Commerce Kickstart. We've started using Commerce Kickstart quite a lot for prototyping and it means that when we are working on larger projects we are able to use Commerce Kickstart to knock up something quite quickly, and I'll show you an example in a moment, and that is something that’s better than using Photoshop visuals or something like that.

It means that we can go back to client say, "Have a play around with this. Does this work? How do you think it should work?"
That’s the bit where I think Drupal Commerce really demonstrates good value. The bit where I think it demonstrates power, and certainly how we found it to be very powerful, is in how it integrates with Drupal sites. The first thing to say here and I've put this at the beginning because we should take it lightly. It's more fund, and I'm not saying that as an e-commerce platform aren't as powerful, because they probably are, and it's just that we've found that the things that clients are asking us to do, Drupal Commerce, and actually Drupal e-commerce solutions n general have been more able to do that than, say, some of the other Open Source solutions.

It means we can … there was once … in the past we have used other e-commerce solutions, it meant that we … there was a site, and then have to … we built an e-commerce site and then we had to integrate that with the main site where Drupal Commerce is actually a big part of that. It means we can do all of the things that Drupal is really good at, and Drupal really stands out in some areas which I'll talk about in a second.
The fact that Drupal are good at that, that means that Drupal Commerce is also very, very good at that. There are some things which, for us, we've just not been able to do before and it's a lot of fun.

If you look at the first example of that, I'm not sure how much I like the phrase Content Marketing, but it's something that people are talking about an awful lot now. It means we can integrate the commerce functionality, the e-commerce part, or actually the name content of the site. If you go to a lot of shops online, and you can see that quite often I have a separate blog or a separate site somewhere; whereas for us it's been really useful to have this content linked directly to the shop for all of the benefits that can give.

It also means that we've got really powerful Drupal user roles and commissions, so we can show different versions of the shop or different parts of the shop to different people. This is a prototype that we are making, this is something we've done very little design work on and it is pretty much what commerce looks out of the box, and with that a little bit of styling. Raven is a TV star in the U.K., and when I say a star, she does the kind of Lifestyle TV shows about gardening and things like that, and this is a new shop, that we process happened for her and her company.
Basically what we've got at the top there; is a blog post that went to Salads from the Garden, and we can link that directly into the shop. We've got that, it's not styled very much there, but you can that the related products, and that product there, those orange flowers are linked directly to that blog post. We can do some interesting things like that and that’s a very simple example with blog. Maybe we could do it with any particular type across the Drupal site.

What that means is that basically, if you can describe it, you can pretty much do it, so for example, you could show a product to a person who has made more than 10 comments on the blog, and if you can imagine any situation where that … as I say you can do it, Drupal is fantastic at, not only manipulating the product data, but it can also manipulate the user data, so we can do really interesting things with that.
I'll show you another example now on the Shepherd Neame project. I think the key part about the cost-effective things for the Shepherd Neame is that they're not very … they're not a small company, and cost-effective things for the Shepherd Neame is that they're not very … they're not a small company and cost-effective here doesn’t necessarily mean it's a low-budget affair.

On the Shepherd Neame site they have fans basically, more than customers; they have fans, people who are really devoted to them. They also have shareholders who use their shop and I'll explain how they use them in a minute and how it's very linked to what Drupal does.

If you're a shareholder at Shepherd Neame you basically get a privilege card and a privilege card means that you're basically shown a different shop to everybody else so a different pricing structure anyway. They have a completely separate pricing structure to normal users like you or I. When we were commissioned to do this project; we were commissioned to do actually two projects, one to do the main site build, and two to do a special micro site for Shepherd Neame shareholders.

In fact, what we discovered was that it was fairly simple to just amalgamate those two projects, to put those two projects into one, and if fact, all of the budget that we had for the shareholders' micro site we were able to devote purely to doing the site, because in Drupal terms, the shareholders are just another user, and because they're just another user it's very easy to give them their own specific pieces of information across the site.

We didn’t do very much with this, but I think it's something that we have prototypes and have thought about in the future, but as I said before, the site has some geo-location features, so you can see on this screenshot here that when you go to the Pub Search page, so we have, like I said, 360 retail outlets and the site knows where you are, and we did this very cheaply, we did this … you can't use this on older browsers, this is really optimized for HTML 5 … the more modern browsers and smartphones.

As I said, the site knows where you are and I think we played around with this and this is not in production but you can fairly, simply build geo fences, and what that means is that the site can change according to where you are. If you're in a shop or near a shop or perhaps it would be a ticketed event and you are near the ticketed event, and the site can know that you're there, if you're within that geo-fence and can change show you different content, and I think that’s going to be something that you see more and more often, and it is becoming to be something that is quite cheap to build.

All of this really links in to the idea of what people are beginning to call I think, Web experience management. I think Web experience management is still one of those slightly-awkward phrases that is used by a lot of people who don’t really know what it means, and it's … it's really about building cross-channel services and if you look at the Shepherd Neame project, then what we've got there is we have a shop and we have a number of … so I've just come too quickly on that side … we have a shop and a number of retail services and if you're a customer of Shepherd Neame you're not interested in whether the shop is separate to the retail places.

As much as possible for a user this should be integrated, work together rather than being in different silos. I think what we have with Drupal Commerce is a way to build those things and integrate them into the site so that we really begin to think about the customer journeys and the user experience across all of the different places where they go to.

One of the tools that we've really found successful actually to have a look at really cheaply improve the things that we've built and the e-commerce site, there's something called CrazyEgg. CrazyEgg, which is all one word, it's CrazyEgg.Com is a tool which you … a bit like Google Analytics that you can plug into a site and it will basically show you where people are clicking. This is the CrazyEgg on our site, on the Deeson site. What it's told us very, very quickly is that one of the menu items in the top there is it has performed very, very badly and nobody has clicked on it, and nobody appears to know what it means.

We've got some really good information there that enables us to go back and make some changes. This is the same information for the Shepherd Neame site and when we design the Shepherd Neame site we chose to have a carousel on the homepage of the Shepherd Neame Shop. Carousels are often really poorly-performing items, and nobody really knows what to do with them, and often they are a design compromise which really is tantamount to admitting that you haven’t really made a decision about what to go there.

We can see from CrazyEgg here, actually this has been quite successful, and there's been one item there which is seasonal, on the left hand, which has not really worked, but we've got some really good data here that this says people are using this particular feature.

We can also see this is another thing that we get from CrazyEgg, which tells us: the lighter colors there are where most people are looking, so we can see that the eyeballs that the people are looking at this page, we are really focusing on where we want them to do, which is the area with all the product information on it, and this is the Shepherd Neame brands page and, again, very cheaply, we are going to have to find out what brands people are looking at, which brands have got the most recognition on the site.

I think the interesting thing for me here is that the brand in the top left, isn't necessarily the most popular brand there, but at top left pieces real estate has been really, really valuable. In addition to having something like Google Analytics which is the kind of standard thing, for doing a cost-effective Drupal Commerce build where it's quite complicated and complex user journeys through the site. We can put CrazyEgg on there, which is quite a cheap subscription, it's £15 or £20 a month, which is for what you get is quite a lot I think, and we get some really good information back about how thing are working, in a way that I think Google Analytics is not necessarily as good at.

Going back to the Cross-Channel Service idea and when you get any business, I think it's natural for any business to show different silos or different areas at the top that don’t necessarily talk to each other, and often you see that in the navigation part of the site, so we can see all of those different areas there. What Drupal Commerce is, is the ability is to link in to all of the other things that a site does, so you'll find sites where we've built that digital content, but it can also link into things like the knowledge base and the community functionality has been particularly powerful.

Those are the support, and all of things that you can do within a standard Drupal site, and I think even more complex in the future is that you're linking to these retail destinations, you're also linking through … or trying to join up services across campaigns, and also trying to join up things within your general data that you have, the marketing data and all of the data that you have on your users. I think … what I'm trying to say actually with Drupal Commerce is that a lot of the really boring things are in the background there, which means that the interesting things you have a lot more time to think about them. This is a Drupal Commerce which is an Israeli Drupal Commerce site actually, or at least built by the Israeli company.
It's basically an underwear site, but it's actually if you go and visit it, the address there, Under.Me, it's a really interesting site to use, or it is if you're somebody like who builds websites. It's actually really beautifully designed, and there are some beautiful touches on there. For example, when you add something into a basket the projects has a nice little animation and it's very clear about what you’ve done, and my theory is, you only have time to focus on these really delightful, interesting parts and you only have time to focus on the cross-channel experiences.

If you can somehow put in the background all of the boring stuff, I think you need a platform which does those boring things for you and then you can build beautiful and interesting experiences on top. That's certainly been our experience with Drupal Commerce, and I'd strongly recommend, have a look at this site, it's not one of ours, I wish it was one of ours, but it's a really interesting example of a very simple product but I'm actually a beautiful user experience.

Acquia has got some … has done some really interesting work actually with Web experience management. Web experience management, as I said before, is really that idea of going across channel and not multiple channels but across channels that are working in harmony together.

That’s really my overview of Drupal Commerce, and like I say, there will be something on the … there already is a blog post on the blog forum, it's a bit more detail and I'll put the slides up later on. Of course, anybody is always welcome to get in touch if they have any specific questions, but I know we've got some questions now.

Speaker 1: Thank you very much, Simon. We are just going to do the Q&A session now. If you have any questions just put them in the Q&A box at the bottom, so we have one question from Will Long at the moment.

He would like to know what you're hosting hardware configuration was for the Shepherd Neame project.

Simon: The hosting hardware, it's kind of outside of my area actually, but it's on Acquia-managed Cloud, and so for us most of the projects that we are working on of a certain size we would recommend then that the Acquia for. This is for exactly the same reason we would recommend Drupal, and exactly the same reason we would recommend Drupal Commerce, and Acquia do all that kind of boring stuff for us. It's a fairly standard Acquia managed Cloud installation.

Speaker 1: I have another question from Stephan. "Who have you seen as the fastest, most reliable payment processor for integrating into Drupal Commerce?

Simon: Yeah, that’s a really good question and actually we had two … I'm not going to say the name of the other one. We actually went through two payment providers here because the first one let us down, and very big one actually, let us down in a number of ways. Our preferred partner is PayPoint, and I think that’s largely … as much as anything, it's to do with history. We just have a lot of experience with PayPoint, and PayPoint has always worked really, really well for us and what we've found there is that I think the intelligent choice to make here is that you go with a payment gateway provider who specializes in Web things.

The reason we had a failure on this particular project actually is because the payment provider was focused on retail environment, so the point of sale, as well as the Web things, and things got a little bit confused, so our preferred one is PayPoint, but I have to say, if I'm really honest, I think that the preferred option is largely due to history as much as anything else, so I'm not sure it's a particularly strong recommendation, but it's worked very, very well for us.

Speaker 1: Randy would like to know, "How do you deal with large amounts of Legacy data from existing e-com sites?"

Simon: Yeah, another good question really. It really, really depends on where it's coming from for most of the time, this just gets imported into Drupal as you would any other content, so this works particularly anywhere with Drupal Migrate, and from my perspective it's someone who has to manage the budgets and work out where the value is. It's very rarely in our experience and it had been worse than anything too complex with the migration because it's actually been as cheap to start again with a fresh install.

With when we were doing the content things, however, that has been something that has always been about a custom migration pretty as much as I think it would be from any other standard site migration. That’s not what particular detailed answer, but using the Drupal Migrate module and importing it as you would any other particular content.
Speaker 1: Phil would like to know, "What has your experience been with PCI compliance when working with Drupal Commerce and your hosting environment?"

Simon: Again, that’s been much more outside of my area and the area for this. In general, it's not really been issue, obviously … in the same way that we go with Drupal Commerce to take that boring things there that usually that PCI compliant stuff would be handled, and very much by the painted gateway providers. Sorry, I haven’t really got a very good answer for that because it's … that’s the kind of stuff that we would leave to their solutions architect, and the people who are much more technical to deal with this. I've not really been there, and we've not really had any particular issues and problems, and it's been something that’s been handled by our third party partners.

Speaker 1: Are we able to ask about panels, because [Eggo 00:29:29], wanted to know, he said he found panels a good part of cost-effective sites, are you using panels and panels everywhere with your project?
(Laughter)

Simon: The reason I'm laughing is because we have a bit team of developers and I think every developer has their own favorite solution, and I know that panels is something that causes … you can go to any meeting at Deeson Online and bring up panels, and half of the people will think panels is the best thing ever, and then half of them will actually hate it. On this site there is quite heavy use of panels.

Having said that, we have we have the 360 micro sites on there and we, because the micro sites are very, very configurable, each micro site is able to have its own layout, so we actually use home box on that instead of panels, because we needed something which a non-trained person would be able to use. The main bit, the main front end panels is used quite heavily, and on the rest of the site using home box and with quite a lot of tweaking actually, and I wouldn’t necessarily recommend home box for a site like this, but yeah, home box with some other things.

Speaker 1: Michael would you to expand on the payment gateway question by asking, "When will there be a rock solid integration with an accounts package, so that single solution can be offered to clients with integration?"

Simon: Yeah.

Speaker 1: It's the integration.

Simon: Again, I just think that’s outside of my particular area of expertise actually. I think we've never had an e-commerce partner who didn’t have a Legacy payment system that they wanted to use, so I don’t if there will ever be a single rock solid integration. Sorry, Michael, I'm not very good at … I'm not the right person to answer that question, unfortunately.

Speaker 1: If you’ve got any further questions you can always email Deeson Online or Acquia for particularly technical questions that we haven’t answered today.
[Randy 00:31:58] would also like to know whether DC is open for membership management solution.

Simon: Yes, very much so. I think that the Deeson Group has a slightly bigger part than the Deeson Online, and what we do, as I said, we do the e-commerce sites of the placements of bricks and mortar experiences, but about probably 60 percent of business that we do is actually membership sites and we can do some really quite complex things with membership management things, that I don’t think we could do on any other platform.

I don’t know if there's any more detail on what you'd like me to say about that, but I can't anything more positive about it really and I mean for us it's been easily the most intelligent solutions because it can link in to all of the other … because it's so well integrated into the Drupal site, and if we take something like the Royal … I don’t know if you were in the U.K. or not, but the Royal Meteorological Society, is an international weather organization.
As an example of that membership site what we can link Drupal Commerce into is obviously the membership management things but also content management things and they have an enormously complex pricing structure for their annual conference because it includes one day, two-day, three-day passes, number of guests, dinners, hotel rooms and all of those kinds of things, and because we can integrate that into one particular place. I think it provides really a nice user experience we are able to do really quite interesting things and link that into their membership management system, and COM systems, and having something that’s integrated into Drupal has meant that those kind of things are really, really powerful for us.
We also do a number of trades with professional associations, so people like this, the College of Radiographers, which is an international organization for the people and to do medical imaging. Again, they have the same kind of thing which is linked up to a very, very large document library. We can link in those payment things to all of those things which are part and parcel of what Drupal does. Yeah, I think Drupal is outstandingly good dealing with people and use their own, so that makes them particularly suited for membership organizations.

Speaker 1: Thank you for that. Bryan would like to know why you recommended using Drupal with, say, [Ubercart 00:34:47] or another package rather than using an e-commerce platform like Magento or PrestaShop.

Simon: For two reasons, as I said before, we use Magento and in the end it was one of those things where it's not that Magento is more powerful than Drupal Commerce or Drupal Commerce is more power than Magento, it's just that the things that we were being asked to do with Magento, and we couldn’t do, not very easily, we couldn’t integrate with things like membership databases, and we couldn’t integrate with a normal Drupal site. It was very hard to integrate those particular things, and Magento, I think is really poor at managing actual content which is not e-commerce related, and I think Magento, especially, has got its place and particularly if you're doing an e-commerce business where you're turning over millions of pounds, then you'll probably do want to go with something which is geared towards doing that.

For us it wasn’t that it wasn’t powerful enough, it just wasn’t something that we could easily integrate into a content-heavy site, and most of the people that we work with have done e-commerce as part of their business as opposed to being their whole business. They’ve been largely focused either on members and users and content rather than selling particular things, less formally I should say, but we did ask our development team what we should do with Magento and they all, universally, said that they would prefer never to work with it again, because it was just so hard to make it do what we wanted it to do.

We have used Ubercart in the past, obviously we stopped using Ubercart when we've stopped doing Drupal 6, we still service a number of Ubecart builds, and in fact our previous webinar which was about working with future publishing and building beautiful e-commerce sites, that’s actually Ubecart end.

Speaker 1: Stephan would like to know, "Would you recommend any particularly well done site selling digital items?"

Simon: Not off the top of my head, I'm just trying to think actually. Not with the Drupal Commerce I think … the one we've built that shows digital and actually is Ubercart. No, but if anybody wants to … I think I've ducked a number of these questions, so if you want to email me I'd be able to get answers to all of those ones fairly simply actually. Particularly I'll go back to the payment ones or the PCI or compliance takes out that.

There are people here who can answer that much better than me. Certainly, I think, what I'll do is, I'll go away after this and add it to the blog post, examples of these two contents, and because I think you're right, it's a really interesting thing with this.

Speaker 1: It doesn’t look like we've got any more questions now, so I'd like to really thank Simon for presenting today. Thank you everyone for…

Introducing Drupal Commons 3.0: Driving Convergence of Content and Community on the Open Web [October 30, 2012]

Click to see video transcript

Bryan: Welcome everyone to today's Acquia Webinar, introducing Drupal Commons 3.0, Driving the Convergence of Content and Community on the Open Web.

My name is Bryan House, I'm Vice President of Product Marketing here at Acquia, I have the pleasure of being joined today by Ezra Gildesgame, Product Manager on our two-fold Distribution's team. Welcome, Ezra.

Ezra: Thank you, Bryan.

Bryan: We've got a great session in store for you today. We are very excited to talk about Drupal Commons and the D7- version of Commons that we are about to release, so we will get started here.

Without further ado let's jump into today's session. I want to talk a little bit about the state of the market and the concept of OpenWEM that we announced last week, and also talk about how Commons and community applications built in Drupal fits into that as a topic and as an area. We will walk through Commons 3.0 and give you a sneak peak at what the capabilities look like, and then Ezra is going to do a demonstration of the product as well, so you'll get to see it in action. Very exciting stuff, again, feel free to submit questions via Q&A at any time.

When we talk to organizations this is the conversation we are having a lot of and many of those organization the Web is broken. They’ve got lots of different sites, they’ve got lots of different business objectives with those sites, some of them are Web content management, or CMS style sites where it's one or few, to many publishing information out to audiences to drive leads, to publish information and distribute information to citizens, or various reasons.

The social software within the organization are: intranet, extranet, partner community, developer communities, and those are much more about engagement and connecting and having people communicate back and forth, and then e-commerce, people are increasingly trying to do transactions. Whether selling information, selling products, and so this is really complex; this is incredibly complex for IT organizations, how to manage different technologies and different systems, and we've talked to pharmaceutical companies, and have 600 websites and some 60 different CMSs.

We've talked to hospitals and they have 20 different CMSs, this is a significant challenge for organizations, but what they want, is they are looking for a way for these things to come together, to provide unified platform to address these needs and address them in from a single set of Web technologies, so that way they get scale and efficiency and their development side, in terms of managing the applications, managing the underlying infrastructure as well as building new capabilities and addressing the business requirements.

Scale on the user side, reduced their training burden, their training overhead and make it very easy for marketers, other folks in the organization, to contribute to those sites, and achieve their business objectives associated with those sites. We are also seeing this gap between the marketing and IT and looking … these two organizations are looking for solutions to help them better communicate and better achieve their joint objectives.

Where marketing is looking at personalization and translation, and harnessing social media and building ways to engage customers and prospects, in their community, on the Web. Acquia is looking at how to do performance, how to build scalable, reliable systems that meet the needs of the business and do that in a way that integrates well within their internal environment. Increasingly we are seeing these two groups come together, and become … working together towards joint objectives to exchanging ideas and how to do that as marketing becomes a much more tractable, measurable activity on the Web that aligns the interest of the CMO with the CIO, and so we are seeing this drive behavior in the market.

OpenWEM is really about addressing some of the issues that we've seen with proprietary Web experience management approaches, and so the question is as you're building websites and Web experience, whether those be mobile sites, community websites, delivering content to smartphone or tablet applications. Do you have closed-loop development, you have struggles where you are unable to add new capabilities or new features because the development process is so locked down, you are unable to innovate fast enough.

You are unable to bring new things to market fast enough because of the heavy development requirements for the new applications. This is a challenge we are hearing over and over again from large organizations. They are what take months in a proprietary technology because of all the customization and custom codes we are able to achieve in days or weeks with Drupal.

It begs the question, why are we spending money on licenses and maintenance contracts, and what have you for these proprietary systems which we can't use anyway because they don’t move at the pace of our business? They’ve locked in to one set of infrastructure and tools, are you struggling too when you talk … when one side of the organization is talking to the other side, we can't do … help you meet your objectives on the Web, because we are a Java Shop, or we are … we only use these tools, and so you as a marketer, you as a developer, community manager, are forced to go outside into this world of shadow IT because you're current organization can't meet your business needs.

Your budgetary pressure, and so you're not able to invest in large CapEx expenses, especially with licensed purchases and hosting purchases but need to now move into much more of an operating expense model, where subscriptions … and a subscription approach is a better way to meet your budget and capital requirements.

These are just some examples of the symptoms that these are OpenWEM strategy that we announced last week can help address, based on our experience in the market and our product offerings to address the needs of organizations looking to build Web experiences and do it on a unified platform as Drupal.

Really what we are trying to do, is offer the digital marketing organization, that includes both business folks on the marketing side as well as IT folks supporting those business people, is often in both freedom and flexibility, we give them a platform that allows them to create simple Drupal's digital experiences that have lower cost of ownership because they can deliver them faster to market to help them innovate the pace that marketing needs to do, and accelerate that process of new … meet new requirements and objectives on the Web as the mood changes.

By giving you a unified platform to create branded unforgettable customer experiences, so regardless of whether you need social plus content, or content plus commerce, you can do that all from a single platform, we just saw this from one of the leading Web content management vendors just as a big partnership with the leading Microsoft-based community provider, so in the top-two of the three bubbles in the report, and they announced a partnership. "Hey, we'll work together, if you want content build it in platform A, if you want community build it in platform B and we'll work those things together."

That just introduces complexity to the IT organization to end users which interface do I create and which access permissions model and security model am I going to use here? How does that replicate from one system to the next? I think all of us that work in Enterprise IT have lived that challenge and it's a problem. Having a unified platform doing it from the ground up and Drupal as a compelling alternative, and doing it in a way that not only do the unified platform for these three Cs that we've outlined here, but also it integrates nicely to the best of breed tools that you already have in place for Web analytics, for marketing automation, for CRM, and for other tools that you're now willing to replace.

From a sweet bed in there that wants to drop in all those technologies. All of this is build on an open source … in the open fast model so there's no lock-in, so if you want to make a change and you're a cloud provider, your underlying infrastructure, you can't do that with your infrastructure, so that’s a key piece of Acquia business models, to give you that freedom and flexibility as your requirements evolve. You're cloud platforms and many other pieces can evolve with you.

When we think about these unified experiences for digital consumers, it spans a couple different areas. It's about delivering content appropriately in campaigns across the right channels, and drive conversion rates and personalization or social capabilities build into those. Combine that with community capabilities from able groups, self-organized and connect, create things like support communities, developer communities, brand communities do that very easily.

Enable your consumers, your prospects to be the best representatives of your brand, there are much more transparency and credibility in the market than you do as a vendor and so enabling those people and giving them the tools to share their experiences with others is a great resource for your brand.

Then finally tying this, where appropriate, to commerce capabilities and this is a little bit forward-looking as we define our WEM strategy but clearly the market has shown us that content and social and community capabilities are critical for driving commerce experiences in the future about taking advantage of those tools to create immersive shopping experiences.

We think about optimizing each of these areas from a content perspective, this is about lead gen and social media, tying it into things like marketing animation and measurement for analytics. From a community perspective it's about creating social brand applicants that help do the selling for you, it's adding social capabilities to your intranet as well as to your extranet and to customer-facing, prospect-facing environment and then tracking all that, and a then managing and maximizing engagement with stakeholders, you're developers, your partners, your customers themselves to drive innovation through your product lines.

One of the things that we've been thinking about from this is particularly with regards to community capabilities, social; it's helping to evaluate where you are in this timeline and where you are from the evolution of your social capabilities and your social maturity, from your applications that you built.

We've identified five stages, if you will, of collaboration capability, so there are as simple as collaborating and documents, and those of us who … anyone who uses Microsoft Office somewhere in the organization probably a SharePoint for collaborating on documents, and they are step one in building social capabilities, all the way up to social as a service that’s embedded across all your applications and that’s part of how you work.

We are starting to define these different phases and helping organizations think through where they are today: are they doing just internal collaboration, are they doing external collaboration? Are their community experiences that they work with, people outside their firewall in getting … in driving that into the R&D organization or the customer support organization, or are they putting that across all of their Web experiences and properties, and so you'll see some of this happening already, some of this is still into the future, so it's early days, and sort of this enterprise social or community market. Ultimately we strong believe that social will be a service that is included in your Web experience or your Web contents stack and anything … applications or sites you build ultimately will have social in them, and so this is an important piece to think about: where am I in this lifecycle and where do we want to go as an organization?

Tools like we are going to talk about today with Commons can help you get there and provide a path to accelerate the process and get you to where you need to go either, and utilizing social as a service.

With that let's talk a little bit about the product, Drupal Commons 3.0, so I'm going to do a little bit about, a high level about the product and I'll pass it over to Ezra who is going to walk us through a demonstration. We'll do that.

Commons 3.0 is built on Drupal 7, and we are going to talk a little bit about what the product does, you'll see it's a different user experience for the product which we are very excited about and we are getting a lot of feedback and the first two major releases of Drupal Commons in terms of how we can make it more useful for you as developers that are building social applications in your organization, as well as for your users that are your community managers, participants in those communities making more intuitive and user-friendly for them as well, so we are very excited about that.

So we think about what makes Commons unique as compared to other solutions in the market or a do-it-yourself approach, and the nice thing is this is a community application that was built by a global community of Drupal developers, so at the very nature from a product perspective, Drupal is social and so you see this, we talk about the proprietary vendors where they need to come together to cobble together for social experiences that are native across the Web properties.

It's the same in the Drupal world, and this is one of the things that really separates Drupal as a technology, but also as a community, things like Drupal.Org groups like Drupal.Org, they're collaborative in nature, so the expectation of this has really changed in that that insight and experience really informs the products, in order to make it easier for you to utilize and build communities and roll them out faster, and test ideas, so that way they are productive and useful for your business.

It gives you the ability to update campaigns and content seamlessly with one tool so you're not jumping back and forth between multiple applications to launch a community site or community capabilities on your website. To give you the ability to join … create a next-generation social Internet with collaboration around knowledge sharing as well as marketing campaigns and giving the tools for your employees and stakeholders to collaborate very quickly on those environments.

A big piece that we think about is for project teams that are using Commons, how do you accelerate ideas to experience, how do you facilitate fast-cycle collaboration, so that way people have relevant information to make decisions to move projects forward to ensure you hit milestones and deliver ultimately what you're delivering. Whether you're delivering technology products, whether you're building cars, or other places that you might be doing collaboration applications.

Then maximizing engagement across the various different community types, be they customer prospects, customer support, developer communities, and utilizing things like Discovery tools that are built to document search to find great content, but also to locate relevant experts. People who are knowledgeable, who you may not know in sort of the first circle of your personal network but they were active and involved in the community in making those discoverability an important piece of that experience.

Let's look at a couple different pieces, so one of the things you can do in Commons is follow content, and get notified fast so you can see … you can follow trending content, create a personalized view that subscribe to content and time, so that way you can reduce the noise and improve usability, the application and focus your interests and creating a content stream that’s around your interest, so that way you're not digging through the community looking for something, but it's actually delivered to you and personalized.

An important piece of the Commons experience is this ability to create groups, to follow and post content in groups, and sort of create the right level of structure around groups, and so in some environments that may be ad hoc through creation and it's a relatively wide open environment for groups to self-organize and collaborate; other environments that may be a little more hierarchical top down where the groups are pre-defined and people join based on their departments in their organizations or their interest.

You can do this very, very easily. You can create group home pages, so it makes it easier to drill down the content and surface content for visitors to that group as well as members in there, so there's a lot of capabilities on the group side that you can do to facilitate that collaboration and make it easier to join in. I think one of the biggest things we've seen with the organizations and the communities, in general, is how do you lower that sort of participation barrier, and so a big piece of what we do in Commons is to make it easy to find content and drive that initial participation in a community.

One of the big areas that we are really excited about with Commons is Mobile Collaboration, so Commons 3.0 will have out-of-the-box responsive design templates, and so you can create and re-factor your Commons' presentation layer for smartphones, for tablets, for desktops because in reality we are all collaborating and interacting with information, on the road, on different devices. Sometimes during big storms on our phones when the power [is chopped 00:19:13]. This will be an out-of-the-box of 3.0, one of the biggest featured answers that we are very excited about.

Commons is has always had this concept of personalized experiences but we've made … we've cleaned up the user experience in Commons 3.0 to make it more useful and friendly, and usable for you as visitors, so you can create personal as well as group dashboards, things like social bookmarking. You can create content here and sort of have a view to subscribe to discussions and content, create a stream of activity that others can see the use of a particular user.

Again, this makes it an easy way to do expertise location, you can combine this with some of the gamification tools out there for people like Badgeville, and then reward badges to people that are participating or contributing above certain thresholds in the community. Again, the drive, participation and also reward that in the community where others can see.

One of the most important viable pieces of Commons is this content plus community in a single platform, so there's no need for a separate CMS, so out of the box it comes with preset configurable content types, content views, they get very easy to organize that information. You're not doing a lot of structural designs set up in this, but you're much more focused on the branded experience and how you're going to present this information, but you don’t lose the flexibility and the freedom to add different custom content types, tweak things, make it useful and configure this for your own needs within the organization.

We also gave you some tools to provide a sort of community-focused participation and engagement metrics, compliments that Web engagement … the Web analytic tools you already have in place that gives you some specific data to the Commons experience so there's a lot of: are you driving new members, are there members participating in those, provide the community manager a lot of tools to ensure that they're getting the most out of the community, and so you don’t suffer from the empty-restaurant syndrome, if you will, but actually people that are actively involved and engaged, and the community manager knows where to go do … and to drive and increase that engagement.

Then from a developer perspective, you know, developers have the freedom to innovate here, we've built this with Drupal best practices in mind, so it retains that openness and flexibility the platform has from Drupal perspective to add modules, to integrate with other systems, take advantage of things like Semantic Web document management is a big piece, particularly in Internet environments, so you can integrate it with tools like Alfresco to manage content. Document is part of your collaboration environment. Certainly single sign on is a critical piece in the enterprise and so there's lots of options there.

We've retained and built in some Drupal best practices to ensure that you get all the things that Drupal has to offer as a platform while taking advantage of a much more out-of-the-box experience with Commons.

I'm going to do a quick case study of Commons in action, and this was a customer that built Daimler who built a social community on Commons to replace their focus groups, so this is utilizing a community environment for social engagement around brand … advertising and sort of replace the focus groups that they traditionally use as marketers with their agency. This was a community for young sports car drivers, 18 to 45 years old that were … about what they felt about their brands, about Mercedes specifically, about some of their video and Web advertising.

This is a very cost-effective way for them to build this, to replace a costly cumbersome, time-consuming way, traditional focus-group approach, and do it with … do it in a new and innovative approach. They won awards for this, innovative use of social media for this community, and ultimately the reason they chose Commons was there were so many features out of the box that they didn’t spend their time building their community. They spent their time building the experience that was relevant to get the data they needed to share with their client, with Mercedes, around the marketing focus group.

It shifts the conversation from doing the heavy lifting on the technology side to where you're spending your time on the business result side, and changing the paradigm a little bit. This was just one example of how people are utilizing this platform today and we certainly have plenty more, if you're interested.

With that, Ezra, I'm going to pass the control over to you and let's take a look at a demonstration. Are you ready to go, Ezra?

Ezra: Absolutely. Let me just share my screen here. Can you see my desktop?

Bryan: I can see your desktop.

Ezra: Okay, great, so let's gets started. The first thing I want to show is by default , out of the box what Drupal Commons looks like with some content in it, if you are new to the community. At this point you haven’t signed up, maybe you're considering signing up, and you just want to learn what the website is about. We've looked at this design a lot and there's a lot that you can tell as the new community member to learn what this community is about.

At a glance you get this configurable text here on the front page, so right now it's talking about Commons itself, but of course when you're setting up your community you can configure this text, so right out of fresh install you’ve got a homepage that says, "At a glance what the community is about," and then as people participate, the best of that participation is shown on the front page.

We just take a moment to take a look at what we've got on this front page, you’ve got a nice visual lift, and active community members, we've got upcoming events related to the community, a list of … and activity stream from the community just to active groups, and then the list of featured content on the site.

This is one example of this whole content community commerce idea. This is a space for people to collaborate first and foremost, but it's also a space where you can create that content if you want to or, perhaps, make a site-wide announcement or share some community-wide views, and make sure that that reaches the different people in your community. You can see how blogging site combine with the community site would really suit well into that.

Of course once you're logged into the site and the screen is here to show you what that looks like, you'll get a little bit more of a personalized experience, so this is the default landing page for folks who are logged in to Drupal Commons. What you see here on the left is a list of content and you can see from the thoughts and focus on the top that it's sorted by the most active content for … so let me tell you about what that means. If you're a community manager, one challenge in creating the community is figuring out which content is interesting, and which content people should really see, and large communities with tens, or hundreds of thousands of active users, that can be its own fulltime job.

What we are including in Commons is just the idea of automatically active content, and what that is, is the system that benefits both community members and community managers, and what it does is it automatically takes information about content. The things like the number of comments that the content has, the number of people who are marking it as … that they like that content, and it takes that information and it produces an internal activity score. The benefit of that is that the score decays over time, so as a community member looking at this listing, I can see content that’s generated a lot of activity recently.

This is different from something like a system that just shows you most popular content of all time by votes or something like that. This really let's you get a snapshot of what's active on the site right now and that’s the reason that it's so time-specific is because that value decays over time, so maybe there's a really interesting or perhaps even controversial discussion going on in your site that will have a lot of comment. People will be liking it, people will be viewing it, that will all make that content register more active, but if that conversation blows over and something is more interesting today than that was yesterday, the new content will be more prominently featured. That’s just one way that it will be really focused on the goal of helping people find content that they want to view and content that they want to interact with.

Of course, you will really value personalization, so I can engage these filters up here, and I can change the way that this is sorted so I if I want to just to see the most recent content I can view that. You will notice that there's this idea of following, following is what you can do to subscribe, or like on Twitter as it sounds, follow people, content and even groups that you're interested in. I'll show that in a little more detail a little bit later.

I want to show the groups that are actually here, you could see we've got a really simple primary now and not overwhelming, easy to figure out where things around the site, and if you click on the groups' directory you can see there's a list of collaboration groups within the site and it's really attractive. You get this really nice, visual list of recent content in the group, and the type of content that it is and a list of intruders to that group, so that you get a sense, even just scanning the groups of what they're about.

Similarly you’ve got the gist of most active groups in that activity system that I mentioned for identifying interesting content that works at a group level as well, so the group has the sum of activity with all the content.
Let's take a look at a Drupal page, go to the Boston group here. You can see that this is a really sleek-looking group homepage and front and center is content that’s in the group. We call this the group content widget, and what's really nice about this widget is a couple things.

We've done a lot of usability testing on Drupal Commons, on Drupal Commons 2, the previous version, and that has really informed the designs that we have in this version of Drupal Commans. Here were are on the group's landing page, you can see all the different kinds of content that are in this group, they each have a tab and so let's say I'm interested more in the document in this group, or questions in this group, I can focus in on those items, and each of those kinds of content gets its own unique listing. Perhaps if I'm interested in the ideation and I can see that that’s presented a little differently, because it's a different content type.

The other great thing about the way that this is put together and we hope you'll think it's great too, is that the creation, the content creation controls are in the same spot, so if I want to go ahead and create a post I don’t have to go to some separate menu somewhere, I don’t go to the file menu, I go right here to create a post so that it reinforces the idea for folks, perhaps, folks who are savvy, but also maybe folks who are new, to participate in the online community, that the content that they're creating is going to go into that group, because that control is in the same place as the content that’s listed. Of course if I want to follow the group or vote it out, I can do that as well.

Of course, on the side, we've got other secondary and interesting information, so based on activity in group and some upcoming events in the group, which brings us to events. Whether you’ve got a virtual community or a community of people who are meeting in real life or a combination of both, that’s been supported by Drupal Commons, so if you do have events you can leave the events future-enabled, and you get this really nice, robust landing page for events, with these facets here that let you filter so that if you want to find an event that’s near you or an event that in your groups, it makes it very easy to do that.

That’s actually a lot of flexibility to this tool, because you can … if you want to elect people sign up you'll be able to do that as well, group site for that, you can collect registration. At the same time if you're maybe just linking to a separate conference site, perhaps you'll have the site that’s powered by conference organizing distribution for Drupal and you can link to that site for registration as well.

I want to show this following functionality in more detail, Bryan mentioned that earlier, so image I've gone through the site and I've followed a lot of different things, and I've followed groups, I've followed topics, I've followed people who I think are interesting, and then just individual threads because I thought those particular threads were interesting. I can go to this site there and interface here and control how I hear about that information. Let's say that everything that I follow up here is in the activity stream, but perhaps the New York City group generates a lot of activities, like right now there's a hurricane and I'm just getting a ton of emails from the New York City group about that hurricane.

I can choose to not receive email from that group so that I don’t have to un-follow them completely, I'll still see it when I see listings of information that I follow on the site, but at least I'm not getting an email. That holds true for topics and people as well, so you really get a lot of control to personalize what you see on the site, but we present that in a really simple interface, so that it's not daunting to effect that kind of control.

Bryan mentioned that the software is Mobile-out-of-the-box, and so I want to show you some examples of that. Here we've got the What's Going On page, and suppose I'm on a smaller device I'm just going to drag and resize my window here and you'll see how this screen automatically reorganizes to fit. If I'm on a phone, you'll see that this is still a coherent design, and it fits in a nice, narrow window there and I didn’t have to do anything as the site administrator to get that benefit. This is one of the features that we are most excited about that … Drupal Commons 3.0 or the previous version.

Let's show the group's directory. There's that group's director and I'll just make the window smaller and you'll see a nice breakpoint there, so that I still get all functionality of this group's directory, but it fits in a narrow, device-sized window.

I want to talk about how you deal with spam and inappropriate content, so as you know in any community there's always of size, at least, there's always a size at least, there's always at some point, somebody who submits content that’s inappropriate or that’s just out of place advertising.

Now the first line of defense is actually that Commons provides, is actually … so you almost will never see Commons integrates with the Mollom service, which is an intelligent service that analyses the content that people submit into the site, and also analyzes their reputation across … based on information from thousands of sites across the Internet, to automatically determine whether something is down.

Most of the time, spam will not even make it through the site to be submitted, because you’ve got that tool and of course Mollom is something that we included as part of the services in the Acquia network. However, sometimes content is going make it through there, right. There is so much spam on the Web and it's a great service but every now and then something will make it through.

Let's look at an example piece that’s in appropriate content, we've got a shoe sale and perhaps your site is not about shoe sale, and so for this site it's unsolicited advertising. As a community member I can go down and report this content that’s inappropriate. I get a confirmation message, and then I click Report as Inappropriate. Now I've reported this as inappropriate, and suppose some other folks have done this throughout … a lot of folks notice that, they clicked the links, and as a site administrator I can go to the main content with this and you'll see that we've added too, what you get with Drupal Core, you’ve got this Reported Content tab.

Now I've got a listing of all the content that’s been reported by the community, and you can see that this is organized in a very useful way. You can see where the content was last reported and you can also see the number of reports, so you can imagine that on a really active community you'd have a number of people would notice when something is clearly out of place, when something is clearly spam or an advertisement, and they'll flag it, and as a result you'll see this number getting higher.

Part of the challenge is not just identifying when comments are being inappropriate, but you’ve also got to get rid of it, and ideally you want to also make sure that the people who are posting it can't post content anymore. We we've integrated that all into one bulk form, so this lets you check all of the content that’s been reported or just the content that you think is inappropriate as the moderator and you can believe the content and block the office account.

One thing that’s core strength at Drupal is the way that roles and permissions are really powerful, so within Drupal you can, for example, have a role of content moderation assistance, and this is something that will be included in Commons 3.0, and you can empower them to access this moderation screen. If you have people on your site who you know are trusted, they’ve proven themselves to be trustworthy, you can empower those community members with the responsibility of actually deleting spam content, and you can do that, in a way I think that Drupal's granular permission system.

You can do that in a way that does not grant them permission to completely take over the site, so maybe you trust those folks but you don’t want to give them control to do everything, you can still grant the back customer screen.

Let's go ahead and get rid of this shoe sale content. I'll just click Delete Content and block off his account, I got a confirmation screen and I can confirm, and the systems is able to delete that for me. You can imagine I could do this in a manual way … pardon me, I've got upgraded to the latest development snapshot and I think I introduced a bug. In any case it wouldn’t be a demo without a bug, right; so we got an error message, but in fact the content is deleted.

Another thing to note about this interface is that when the content is deleted it's also reported from Mollom's, so part of the way that that Mollom service works, is by getting reports of content that are inappropriate, so that Mollom can improve using its machine learning and algorithms that the ability to identify when content or a particular poster of content is inappropriate. This, other than one click, you can delete the content, block the office account so they can't post new content, and report that content to Mollom so that they're … that spammer is less likely to break through your site, and also any other Mollom site in the future.

It takes a lot of these tasks that normally would be friction, folks or community members and community managers and it makes that really simple for both. For example, on one large community site that I'm a moderator on, when we have spam on the site where people have to send a message to the moderator, and then the moderator has to respond to the message, and the moderator has to go to each individual piece of content and delete and then unblock the user, it can really be cumbersome. Personally, this is one of the features that, as a site moderator that I'm most excited about using.

That’s our demo for today; I'd love to take some questions from the folks on the phone.

Bryan: Ezra, why don’t you pass it back to me, I've got a quick…

Ezra: Sure, let me…

Bryan: …then we'll move it over to Q&A and you can check out the prioritized questions, in the Q&A tab, they've already thrown some questions up there for you, and I will continue to queue them up for … Ezra, as they come in.

Ezra: Thanks, Bryan.

Bryan: So, just to quickly summarize before we move over to Q&A and let Ezra get into some of those questions. Really what this is about in Commons specifically, the product, and overall are OpenWEM strategy is about this combination of content and community today and in the very near future commerce, to create these content-rich social Web experiences. Experiences that are multi-channel in nature so they work well on your desktop as well as in your smartphone or tablet, that are easy and rapid to deploy. That way you are not spending time building infrastructure but spending time on value-added capabilities for your site.

That maybe the brand experience for your community, and if you look at some of the Commons sites that are out there, be they from the eBays of the world, or other … to think Mercedes Benz that we saw and plenty of others can give you tremendous flexibility in the brand size. Or, if it's integrating those communities, and those capabilities into an existing infrastructure within your organization, Commons and its open architecture, it gives you tremendous flexibility to leverage the investments you’ve already made on the IT side, be it those things like single sign on, document management, CRN and marketing automation, all of those things you could take advantage of.

Those are value-added places you can spend your time in deploying communities and building applications rather than trying to figure out how do we do profiles, or how do we do [trending 00:42:23] and like content, and some of the things that will come out of the box, so really it's about accelerating that.

Letting you focus on the value-add in that last mile rather than the basic building blocks of building a community application and doing it on a platform. Regardless of whether you're building a community experience a content experience or a commerce experience it's the same tools that … it's the same menus of training, and you get efficiencies over time as you launch more of those.

Really, it's a pretty powerful tool, we are very excited about it and most importantly it's built on Drupal 7.
I see lots of questions coming in, lots of questions in the chat, which is good. If you put them in the Q&A below that would be better. We'll get through those. Ezra, do you want to start talking through some of the questions that we have here?

Ezra: Sure. First question is, "Can partners get access to Commons and with some hands-on experience?"

The answer is absolutely. Commons is, through and through, it's an Open Source product, so we are doing all of the development on Drupal.Org, you can download a development snapshot from Drupal.Org to see where things are every night. At the same time we are currently … the version that’s up right now, is probably best used by folks who have a bit of Drupal's development expertise. We will have a more robust data out in November that would accessible and easy to use for a broader audience of people. You can always check the latest status on that Drupal.Org Project page.

The next question is, "What is the recommended server memory for Commons 3.0?"

What we are doing with Commons, and this goes for another question that we've got about performance scalability, and we are going through a phase of extensive performance testing, and it is absolutely core to the product that Commons work well and work in a fast way on sites with thousands and thousands of users and thousands of pieces of content.
We want to make a recommendation of the server memory based on those benchmarks, and that performance data, so as soon as that information … as soon as we figure out what the best number for that is, we'll make sure to release it and you can follow progress on that performance-tuning in the Drupal relationship queue.

Another question is, "A number of users are stuck on Windows XP currently and he would like to see a browser to access fact content. Does your solution work with IE-8?"

Yeah. Right now we are including Internet Explorer 8 as part of the supported browser. I don’t think that we are going to include Internet Explorer 7 but we are going with IE-8, and it's worth noting that Google Apps actually has announced that they'll be dropping support for Internet Explorer 8 in November. Our support there is basically … our current support commitment there is greater than even in Google's.

Next question is, "I have heard you comment on Drupal 6 a couple of times before, and it's great…" thank you "…however, I am not pretty much going to Drupal 7. When can I realistically expect you to use Commons 3.0 in a production environment?"

We will have production release of Drupal Commons in quarter four, so you can definitely expect to have that in quarter four. I would say, if you want to start building a site on one of the data that … depending on your level of expertise; that could be a good option as well. Folks who are familiar with the Drupal code might be comfortable building a site that starts out on the Commons data and gets updated as that data gets updated.

Another question was, "Responsive design on one hand with the Aloha on the other, an editor that does not currently work on Android 2.3?"

Yeah, Aloha is the WYSIWYG Editor in Drupal Commons, and Aloha actually is something that Commons inherits from our other Drupal distribution the Spark distribution of Drupal where we've made a significant investment, Acquia has the Drupal distribution to improve the content editor experience for Drupal. We've leveraged some of those benefits inside Drupal Commons, and I'm not familiar with this specific issue that you're referring to Android 2.3, but I know that the Spark Team is actively working to hammer out those issues and that I am … the Drupal Commons team is working with Spark to make sure that our editors works well in Commons as well.

If you're having a specific problem with Aloha and Android 2.3 I would encourage you to file an issue in the Drupal Commons issue queue and we'd be happy to look into that in greater depth. It's possible that investment would just be automatically fixed as part of the current and ongoing Spark development anyhow.

Another question; "If you want to use Commons' functionality in Drupal 7, would you start with the Commons 3 installation that has CMS capabilities? Or would you do it the other way around?"

This is a great question. I would say 90 percent of the time, the answer to that question is going to be that you can just install Drupal Commons; you know Drupal Commons certainly doesn’t take anything away from the Drupal Core functionality, and of course the strength of Drupal, historically, has always been as a content management system. I would say that you, in 90 percent of the case, you should be able to start with Drupal Commons and you're ready like Bryan said, we've got a range of different content types that are included in the product, or for doing content management.

Of course we have all the standard Drupal facilities that you might expect, so the used modules, the ability to create more custom content type, if your site needs that, and actually for developers we've made it really easy to add content types to that group's homepage, so if you member on that group homepage, there's that tab interface for each type of content. It's really easy to add a new tab for a new custom content-type; if that’s something that you’ve got on your site.

Another question is, "Can you password-protect each group?"

Commons does show the ability to make groups private, and so as part of that you can restrict either the content that’s in the group, so that people can only see the group if they joined, or you can even make it so that the group doesn’t show up unless a user has been explicitly invited. That the model for that is more of an invitation or a request to join a model that's a password model, and that tends to be robust because you can share the password but the access that’s on a per-user basis, and is controlled by either account.

However, if you wanted to restrict access to the group using a password, you could still do that and the thing to keep in mind, with this question, and basically any question about features and people commenting is that even if it's something that we don’t necessarily ship as part of the core Drupal Commons product, we've got those thousands and thousands of Drupal modules that you can use to expand or extent function that’s part of Commons. I would say this is a great example of that.

The next question is, "I have site analytics tools been updated since Commons 2, and is there an easy way to see which groups are the most popular out of the box with Commons 3?"

Let me answer that in two parts. The two main groups are the most popular, you’ve got that after group's listing, that was on a group's directory, and so that’s one great way to seeing which Drupals are the most popular. We are going to ship Drupal Commons with a similar set of analytics tools that are built in from the ones that were included in Commons 2, and we are also looking at it potentially adding third party analytics tools through the Acquia network.

Another question, "Can you add product content types and community details to this to create a community and corporate site?"

Yes. I think this is similar to the other question but a little more than e-commerce labor, so the apps that we could add additional content types to your Drupal Commons installation and you could, based on Drupal Commerce distribution, and you could also add functionality from Drupal Commerce to your Commons installation. For example, if you wanted to sell products or even, some sites sell thing like subscription, so I can imagine using Drupal Commerce, adding it into Commons to provide the ability to subscribe to pay, to subscribe to a particular group, there's a lot of possibilities there. A good question, thank you for that one.

"Can you limit events to a specific group?"

Yeah. With most of the content-type in Commons, virtually all the content-types are included are intended to be posted into a specific group, so if you have the New York City Group, and you want to post your event into that group, you can post it just into that specific group and it will be really associates with that group.

Another question is about internationalization; "Will it work well with [I-18 end 00:52:28] module which is a model that’s used to translate content within Drupal, with multiple language and so?"

Absolutely, Drupal Commons will have all of the strengths that Drupal Core does, and Drupal on the I-18 end module has. In addition, Acquia has partnered with another provider to provide community cloud-sourced translation. They are partners with Lingotek, and through the Acquia network you can user their service to … The way that Lingotek works is, suppose you have a piece of content on your site and the conventional translation system as an administrator you would have to manually translate that.

With Lingotek you can have the system send the content out to Lingotek, Lingotek will do an automated machine translation and save that to your site, and then it allows people on your site, community members to refine that translation. You'll get a head-start from best of breed automatic translation tools, and then humans can't go in and refine that. That’s a nice way to cloud-source translation on their Drupal site. That’s something that will work Commons but, of course, that’s part of operating network, so it has a broader applicability to Drupal sites, in general.

A lot of great questions today; "Is the administrative area also responsive? I could manage Commons from my phone."

Thank you for the enthusiasm there and, yes, in general it's responsive. The end interface is responsive; we've got a front and back end theme.

"Are you using a specific responsive template for Commons 3.0?"

Yeah. In Drupal we call it … we tend to call it simply, "theme" sometimes, and so Drupal is … Drupal Commons is powered by the adaptive themes which is one of the most popular and well-supported contributed group of fans, so the theme in Commons is called "Commons Origins" and that’s what gives that out-of-the-box look and feel that we just saw, and that’s Commons origin powered by an adapted theme.

One of the benefits of going with adapted theme is that, those landing pages that we showed in Commons, those are all customizable, so you can switch out the content on those pages, and they'll still retain their responsive ability, so it's the benefit of that … of the adaptive themes for, say, more template that you can have customized landing page that retains its responsive trait. It's not like you have to call up the Mobile Team or somebody like that to make something responsive once you’ve customized it.

"Can users interact bi-directionally with both Web post and email; in other words respond to an email notification of some kind of content; I'll reply?"

That’s a very popular question. In the initial release of Commons that’s not included, but that’s absolutely something that we've got on our radar, for inclusion in the following release. Having said that though there are a couple of great modules in the email comment module for Drupal that you could use if you want to add that functionality before Acquia adds that functionality, and that’s something I love to highlight.

Which is that, a lot of people are familiar with our experience using a vendor and they ask, "Well, there are such and such feature in the roadmap, I wonder if that could be included?" It's true that Acquia has a roadmap and things that Acquia works on; but the benefit of being based on Drupal and end up having Commons as an open-source product of its own, is that you don’t have to wait for us to implement a particular feature if that feature isn't in a particular release, so we would like to highlight that.
"How can integrate Commons functionality along with the full Drupal 7 site?"

The question here sounds like you’ve got an existing Drupal 7 site and you want to add one or more of the pieces of functionality from Drupal Commons into your site. In general, there are two ways to do this and the best way depends a little bit about how your site is built, but I'll describe those ways briefly.

The first is that each of the bits of functionality in Drupal Commons is broken out into what's called a feature module, so for example, if you like the Activity Streams in Drupal Commons but you don’t really need the other functionality, you can go to Drupal.Org and download the Commons Activity Streams module, and add that to your existing site. That’s one approach, where you take an existing site and just add one of the Drupal Commons modules to your site.

Another approach is in quoting the content from your existing site into a Drupal Commons site; I would say that different folks are … if there are different benefits to each of the approaches, and which one is right for you really depends on how your site is set up now.
Another question; "I'm not sure I see where this fits in with an organization that used this instead of SharePoint 0, with SharePoint."
The answer is really both, I think that if you have really basic document management capabilities, that you might to just use Commons and those documents. However, if you're accustomed to all of the robust functionality that we've got in SharePoint or that you're used to SharePoint, you can integrate Drupal to SharePoint in there, or the CMIS modules for Drupal provides that ability, so that you don’t have to choose one or the other. You can do a smooth integration of the best of three tools.
"Can you import Google Drive API for [doc 00:58:03] sharing?"

Yeah. That’s actually something we want Drupal Commons for Acquia's intranet that we use to collaborate for our whole company, and that’s actually a feature that we've added to our Internet and that’s been really popular, so that if you go to a site search, and of course Drupal Commons search is powered by Acquia search and Apache Solr, and you search for copy on a site, you'll also get results from Googld Docs, that might be interesting to you, and where that becomes really nice is when you’ve got … and intern that, it's like the connective tissue between all the different properties that you’ve got in your companies.

Maybe you're using a range of different systems, and ideally you’ve got those integrated but perhaps you do have one or two separate systems you could integrate them in that way. That’s something with Google Doc integration; it's something that is on my personal wish list, so I think that’s a great feature.

Another question, "Can you integrate Drupal Commons 3.0 into our native iOS app or only a Web app?"
Yeah, that’s a great question. Something that has been a strength of Drupal is its ability to speak to Drupal applications, or speak to Mobile applications, rather, and so one of the things that we are looking at for the long-term roadmap for Commons is to include a more robust services integration so that if you're somebody you like the responsive functionality that Commons provide but you want to provide a dedicated Mobile App, you could use that services functionality in Commons, you just build on top of that.
We've also got a fairly recently-released … related to Acquia recently released a Drupal Gardens app that lets you … if you left any site running that module to use post content to the site from an iOS app…

Another question here, "Interested in why you chose panels over display suite or context?"

That’s a great question, it's a technical one, and there's actually, I will refer you to the Commons issue too, there's an issuer, if you just search for Panels Context in the Commons Issue queue, you'll see that there is a public conversation about why we went with one approach over the other. In a project like Drupal where they stifle the module, sometimes you’ve got to release that … a module and choose between two, which is the best approach or the right approach for a particular use case, and so you can read about that particular conversation, with respect [PALS 01:00:46] and in case we've got any issues here.
"Is there any work being done to begin reporting to D8?"

Thank you for that question. We have not begun reporting Commons to D8 but the [we think 01:00:59] that that’s closer to the D8 release cycle, posting the D8 being available we will reporting Commons to the latest and greatest version of Drupal Core.
Another question, "Does Drupal Commons integrate a suite of modules for a partial implementation of phased approach?"
The answer there is, yes. One of the feedbacks we have received from developers from Drupal Commons 2, is that they wanted even more flexibility to just what you're talking about, to use some functionality without the others. We've gone to great lengths in Drupal Commons 3 to provide that, and so I just one example of that is that suppose you want to completely customize all the landing page as just [site holder 01:01:43], you can actually disable the landing pages and build a completely different set of landing pages. Powered by the Panels module, and you can run your site in that way so you're not actually overriding anything you're just disabling some things and building on the building blocks that Commons comes with, and a lot of…

Bryan: Hey, Ezra?

Ezra: Yeah.

Bryan: We are getting a few minutes past the hour, so I suggest that maybe we take the rest of questions offline, we could actually post them with the recording, you can post some brief answers there. Do you want to do maybe one more question and then we'll be respectful of folks' time and end there?

Ezra: Yeah, absolutely. The last question is another technical question, "Is there organic … user roles for Drupal Commons 3?"

The short answer there is, yes. That we do support that, that’s included in organic groups.
I would add, you can … if you have any other questions, about Drupal Commons, you can also … we'll make sure to answer the ones that are on the webinar, but you can also post in the Drupal Issue Queue for Drupal Commons.

Bryan: Actually, just build on that, you can see both Ezra and my own Twitter handle, so feel free to reach out to us, via Twitter, and we'll follow up with you directly. We'll download all the questions, and we can … and anyone we didn’t get to we'll post the answers and we'll push those up on the page with recording so you can get access to those as well.

I do want to thank everyone for joining us today; this is obviously an information-rich session, lots of great content and questions.
Ezra, thank you for a great demo, and for answering so many questions.

Ezra: Thank you.

Bryan: Hopefully we'll see you folks at a future Acquia webinar, lots of great stuff happening in Commons, so we look forward to seeing sites that you’ve built with that as well.
Thanks everyone.

Integrating Your Website With Google Gdrive or Google Gtalk, A Presentation by Babson College [October 24, 2012]

Click to see video transcript

Janet: Hi everyone. Thank you for joining the webinar today. Today’s webinar is integrating your website with Google Gdrive or Google Gtalk with James Barnett who is an architect IPSG from Babson College.

James: Hello everybody. We’ll be talking about Google and Drupal and integrating with Google drive and Google Talk. I'm going to run through my slides first and then I’ll show you some actual websites that in use. I’ll show you some code also that add on GitHub and we will take a little tour here then answer any questions.

We have Google Drive first which we’ve named Droogle; it’s the Droogle module on drupal.org. It got its name because there is Google module for Blackboard and I thought it’s fitting to have just a sillier name for the Drupal community. It was created for Babson College and it’s currently in production. It’s been in production for a few months now. It pulls in all of your documents and folders for display in a site-way context within Drupal or within inorganic group.

There also can be a block on the side of your page in an organic group that shows your group’s documents. We can talk about how to do it within inorganic group in a short while. Babson pulls in Google drive documents for each course. Documents are uploaded via Blackboard is the way we’ve chosen to do it. Then in Google drive they're existing and they're displayed in Blackboard and Drupal within a course context.

Now Babson has courses corresponding to organic groups in Drupal. The way that we do that in Google drive is we have a user corresponding to each organic group. If you have class number 001, you would have a user 001@babson, an email address that access with the Google with the same username as the course. We make sure it has a unique ID; each course has a unique ID through Babson. We use the unique Blackboard ID in fact. It just makes it possible to identify which Google drive user has the right documents for the course.

Droogle I’ll show in the settings page shortly can handle that in setting up inorganic group so that it can pull the right documents. There is some automation that happens behind the scenes and Droogle provides for that. There are reset of APIs that we’ve developed, myself and also another woman Ellen. These guys let you do almost anything in the Google universe. You can create users, you can submit users, you can create documents if you wanted to, you can delete documents, move them around in different folders, you can share them to different documents and folders.
We chose Google and Google drive in particular because their API is so complete. We found some of the other competitors out there maybe didn’t allow for sharing, things have changed in the last six months to a year. One of the competitors didn’t allow sharing of documents. That meant that when you created the organic group in a class through the API in the backend, it’s all automated when we have a new class. You couldn’t automatically through the API share all the documents, just an automated way. That meant it was not going to be useable for us.

Google has done a great job in opening up that API to do almost anything. It is true, just to be weary that the Google API is very dynamic and they’ve been adding the Google drive a lot. It changes a lot and you have to be a little patient and that the code needs to be updated sometimes to take those changes into account. I try to watch the SEQ very carefully and to use it as canary in the coal mine to see if there are any problems with Droogle. It’s great because then the community helps me find if I needed to change my code immediately or something.

Believe it or not they’ll change the XMLs start share once in a while that Drupal sets that and it will break things a little bit, maybe Excel files aren’t showing up correctly, that happened one time. I get the feeling they don’t need to change things so often, but once in a while there’s a snafu and I've seen it that the Excel file won’t show up one day, then they’ll change and then go back the way it was the day after. Usually it’s very stable and it’s going very very smoothly.

Google drive is an exciting thing and they’ve offered the ability to sync your desktop now and more offline support. We are very excited about it. Since we are very excited about it, let them have their documents wherever they go. If they go on a plane, they can have their documents offline. We have a lot of graduate students that are in industry and they're going say part time at night for their MBA and they're on the move. These are sometimes very important executives and they need their documents to go.

We love the Droogle and Gdrive combination. We are still integrating with Blackboard through that. Probably can be a different discussion set up by Blackboard. We pull the documents into Blackboard and the professors can upload the documents there. Droogle does have the ability to upload documents. We are not presently using that but the capability is there. We will go on to the next slide.

Here is Droogle in action in a site-wide context. It’s just slash Droogle once it’s set up and you can see all the documents there. It panels an infinite number of folders which was a bit tricky. Their XML believe it or not comes back without utilizing the next XML structure which is what it’s best at and it’s kind of flat list. I had to store all the memory and kind of sort it and handle that folder structure so it can handle infinite number of folders and documents within.

Here is the Droogle block in an organic group context. You can see that block in the right hand side and it pulls in all the documents into the block for the class. It’s got ease of use. They can log in through Google drive, they’ll have Google accounts, Google and education accounts are free. That takes that into account and we use it. They can do their documents through Google or they can have easy access and click on the link here in Drupal and you will be taken right to their documents.

It’s always about ease of use and helping them find their way to the information that they need in their busy lives; so they can succeed adoption and also continue to do their jobs and to their families all of that. We have another slide here; it’s just the Droogle settings page. You can define a title for the main Droogle page that's the site-wide context. You can set the username, password.

Droogle uses a CCK field for the organic group’s site. You can set up a CCK field and stick it on the setting stage here. You can name the CCK machine name, field, anything you would like. Then it will show up in your organic group page on the node when you click edit, it’s just regular CCK stuff and you can do the password that way as well. Or you can set a site-wide password which is useful to us. Let’s say we have 500 different organic groups or classes in our case. We want to just have one master password. We can just set it all in this page right here.

Maybe it will be questions about CCK afterwards but that’s maybe another seminar in itself. Basically nodes in Drupal you have the ability to dynamically add fields and you just add another field and it ends up on a forum. You can then add in the username and password on said form in the node. The organic group is essentially a node that then is related to other nodes. Basically all the nodes exist within the group and that’s the organic group module which is wonderful.

Now we have Gtalk which is Google’s XMPP chat. Most chats these days are using XMPP if it’s at this level, the Facebook chat, if you have the Google chat, AOL and some Messenger uses XMPP. Most of the chat protocols that are very very big and popular are using XMPP. The history of the word XMPP is that it used to be called Jabber and there are some confusion as to whether or not Cisco owns the name Jabber. This protocol is now called XMPP. It’s really synonymous the words XMPP and Jabber. I have the link here for the Jabber module and drupal.org.

You can read lots about XMPP online, lots of the specs on how to use it. I'm going to go over some of that too. I think it’s a pretty cool underutilized protocol. The Jabber module for Drupal uses a Punjab server which is a BOSH server. It integrates with any federated XMPP jabber server. Here at Babson, we created some DNS entries, they are SRD records. It maps us to the Google talk servers. Its set to like a DNS server so that chat node, that if you're going to send a chat message to jbarnett@babson.edu it allows it to know that it’s really talking to Gtalk and not a Babson hosted jabber server. You can host your own jabber server.

We just choose to integrate with Google talk in their jabber servers because then when they're in the Google universe, when they log in in Google, they can actually chat and talk to people that are in our Drupal application. We've chosen to use Google jabber servers and BOSH basically is the liaison. BOSH does a couple of things. Now there’s HTML and then there’s XMPP. They are different protocols. What happens is in HTML you’ll hit a HTML page on the BOSH server, on the Punjab server and we haven’t tested with other BOSH servers.

I do know what works with Punjab. You will do a long pull; it will talk to that BOSH server HTML page for a long time, in a long total. It means it talks to that page for a long time waiting for incoming messages, sending any messages and it hangs out there as long as it can. Then when it can’t hang out there any longer, it stops and does another get. It goes to the BOSH server again to the HTML page, waits for more incoming traffic or sends any traffic that needs to go and this is what is known as the long pull. You hit that HTML page in the BOSH server. The BOSH server then sends out the information it needs in the XMPP protocol and will send it to the jabber server at Google or if you have your own jabber servers.

That’s how the BOSH server works. The jabber module actually hasn’t been published on Droogle.org for that long. There’s 16 new right now, so has been out there a short while. It’s a little bit more complicated to set up than Droogle because you need this Punjab server. It’s not so hard to set up, but you have jump through a couple of more hoops. I’ll give you some resources for helping you jump through those hoops. On those resources of course you can always be in touch or preferably through the SQL is a great way to talk.

Jabber is working presently adopter now for a few months. Some of the things we run into if we were showing pictures in the roaster for every user. If you have 10 classes or however many classes and it also brings in your personal contacts, if you're using pictures, you're going to want to have them more dynamically not … you don’t want to have 100 pictures glued on a page if you can do without it because its glow sinks down. We started bringing in pictures dynamically when you open up your buddy list. We’ve done lots of trickery.

You can make Droogle API to add people automatically as buddies if you're in the same class. I don’t know if you're familiar with chat but you have to confirm being a buddy. Yes I want to this person’s buddy; they asked to be my friend. If you're at that time when we have it turned on, so it automatically confirms that intake. We've done some backend API calls to allow that to happen and that’s incorporated in the module as well. Some things you do have to do in the back end if you want to go that far like we have.

One thing that happens at jabber is it connects securely through PHP first using a small library I wrote called the PHP Speaks XMPP. The library I put on github and there’s the URL. The reason you want to Speaks XMPP and PHP first is you don’t want to pass the username and password to JavaScript. It would be insecure that way. What happens is if you grab the unique RID and SID, SID is Session ID and RID, I don’t remember off the top of my head what it stands for.

The unique combination of those two numbers, the RID and the SID is the equivalent of a password. What you can do is then pass the RID and SID to the JavaScript library we’re using which is strophe. Strophe is a module that’s been published in the web and it’s been tested for quite some time. Where everything is handled that needs to happen in XMPP, in chat. You attach the RID and SID to strophe. Then you end up taking over that session. In this way it’s completely secure.

What's nice about having the PHP Speaks, XMPP and JavaScript libraries is you have a large array of tools to do what you want to do. If you want to do some backend stuff with chat, I mean PHP can do that. If you want to utilize the JavaScript, you can do that. Now I’ll be showing off more about these technologies in a little bit. I think they are really cool and underutilized.

I'm going to point out a book that goes into a lot of depth about XMPP. XMPP, this book points out; I think it’s really interesting is you don’t have to just do chat with XMPP. You can have a game. You can have tic-tac-toe, you can have any sort of game that passes information in real time back and forth. A real time drawing app, you can mimic Gdrive itself and have dynamic authoring, people authoring a document at the same time, it’s really just passing information back and forth. It’s a really underutilized protocol that I've really come to admire.

There’s the URL for strophe here. Here’s the name of the book. I actually have a spelling error here. I’ll fix it when we post the slides to the web. It’s a Professional XMPP. That should be XMPP not XMLL. Professional XMPP Programming with JavaScript and Jquery by Jack Motiff. Jack is very active in Google Groups. That’s where he has an issue queue. He’s supporting strophe and he helps support Punjab. He was very very helpful along my journey in learning all these stuff. His book is really well done. Easy to read believe it or not because it’s a very complex subject. He did a great job with it. I really recommend his book. You can find him online.

Usually when he posted the Google Group’s queue, that’s an easily Googleable thing. You can find the XMPP, I see this strophe. If you Google strophe it says Google Groups, you will find his group very quickly. Jack will personally answer a lot of your questions and he’s really been great. Here’s a screenshot of Jabber. You see have apps and Drupal developing group. Usually that would be a class finance 101 or something. We have a class called Drupal Development, Apps and Drupal Development and we have Wikis in there that we use in our development shop. We have discussions there.

It’s also an organic group and as a class so we can also see what students are seeing, how the software is performing. Right here you see our groups here and these are all the group members as part of our team here. You can see its green next to someone who’s online and it does turn red if they choose to be busy and grey if they're just offline. Actually you see the little bubbles in the forum; there are little circles there too next to usernames. The bubbles will turn green and red anywhere that same username is used on the page.

If anywhere that a username appears on the page, we give their chat status and if you click on that little bubble, you can send a chat to that user right from that username. There’s another screen here. We are actually going to redesign this to look more like Google does chat, how Facebook does chat, we saw that’s how students are emphatically are used to seeing it. We are just going to give them that so there’s no confusion. This utilizes in strophe and jabber out of the box.

They use the tab structure where if you have a new chat, it uses Jquery UI tabs to open up another chat window and that’s what you’ve got with the jabber module. Whether or not we have an alternative theme public or drupal.org, I'm not sure we will see. Everything is in a theme conscious and if you choose to theme it differently, you have that ability with the jabber module. You can overwrite the look and feel.

Tips and tricks. The way that we do it so that we know if any username mentioned on the page is online or offline or busy is use the same class in the roaster as you do with the username. You use theme username. If you have the same class and say with CSS, that class dot username is green and online, then every instance of that class will show green as well. It’s a pretty neat trick and you can use theme username to overwrite what your username looks like on the page and you can consult the Drupal documentation for how to use theme username. It’s a great cool trick.

Breathing through my slides here. Here are some actually XMPP. I’ll show you, I'm going to share my screen and I’ll show you a little bit more of that. Just sharing my screen here, hold on. Here is a user online about @babson he's green. I have it in debug mode, so you can see a bit more. People are here. Let’s see, I must have gone offline there for a second. You can see down below these are the XMPP messages. It’s only because I commented something in here.

Here’s the actual code from strophe and I stuck a consult log here so that you can see the XMPP up. Usually this is fire bug here. I have it in debug mode. You can see some of the chats messages that I sent in raw XMPP. Actually the body tag is for the jabber server only. Jabber is actually from here until that end body tag. Here it is. This is actually XMPP right here that I have. I hope you can see the blue highlighted.

To send it through BOSH as a conduit you wrap it in this body tag. I can show you here’s the PHPC XMPP and I’ll run you through the library. I was very frustrated using other PSP libraries out there that you, XMPP they were very very complex. I didn’t see a good reason for it, for the complexity. I thought there are a few lines to code as possible and it’s very easy to read. We will just look at the functions. This function here, jabber send XML, that's what just sends the XMPP through the BOSH server. Jabber gets next RID. What happens is you get a unique SID, that doesn’t change, that’s actually an ID. Then each message, the RID number, it’s just a number. It increases by one.

You constantly have to keep incrementing that RID and if you don’t get the next RID, your chat session will just disconnect. Basically it needs that incrementing RID and SID to continue the session. We are sending the XMPP just using CURL, get jabber, get RID, SID. You're sending your password, you're authenticating, you're getting your first RID and also getting your session ID.

Right here, here are some things you're going to send and its some authentication stuff right here as I've just mentioned and here’s an actual message. This is what sending a message looks like. I'm going to send myself a message. I'm sending to one of my instant message accounts, my other Gmail account. This text has to load from my PHP XMPP API. That actually sends the text. That’s that, it’s a very simple little library I wrote so that you can get your SID and RID so you can securely authenticate and then passing that to strophe. I let strophe do most of the work because that’s what we do here adopting.

If things are a commodity, you don’t want to re-invent the wheel. It’s just silly to re-invent the wheel of course. Also strophe has already been tested. I heard lots of people say that XMPP modules can often times really hammer the server because there’s a lot of traffic. Basically I’ll bring this up gain. Constantly, let’s see if we can find it. Do you see this; its prowling here, this is the long pull. It’s talking to the BOSH server and hanging out there. When it’s done, it’s going to sign up another one of this.

There is a lot of traffic. If you’ve got say 100 users, you’ve got some more messages right there. Maybe you saw that at the bottom of the screen. You constantly have the traffic and if people leave their browsers open, that’s a lot of network traffic. I heard that with some other libraries or home grown things, it can handle things inefficiently and really take down a server. Like I said at the jabber module has been running successfully now for a couple of months at Babson.

We have some other performance things that we are working on but it’s not because of jabber. We’re trying to make things faster, fix our team up. Like I said we’re dealing with these pictures, the people who’d like to see a rich theme, like a lot of pictures we’re are dealing with a lot of sprites and making things faster. Luckily jabber has not bitten me from behind and it seems fairly fast.

Now that being said, if you’ve ever gone to Gmail and you know they have chat on their page, there is a pregnant pause as it connects to a chat, even in Google. I say it takes about an extra half second to a second and then you're on your way. It’s really no slower than what Gmail has. We do have it running on every page of our site. What we do is on every page currently we just get a new RID, SID and reattach the chat. I can show you how we keep track of chat history. Jabber does keep track of chat history.

Here is the jabber cable here. Here’s your current user JID. In chat JID is the unique handle that you … if you send me a chat at jbarnett@babson.edu, that’s my handle and my unique identifier, that's the JID. This is your current user JID and this is your conversation partner JID. We keep track of this chat messages. Jabber will on switching pages load up that chat history up to like, I think it’s like 15, 20 messages per person per chat partner. That’s just to keep the continuity between the pages. If you create pages, you're like, oh you know what? Let me go to this page. I want to tell you about this page.

Let me go look to see what it says. You switch pages and you can pick up right where you left off in your chat with that person and we’re hoping that’s a very good experience. Feedback from our student is they are very happy to be able to collaborate right from this tool. They may start a conversation in jabber, in Drupal but then they might say, “Hey let’s move to Google hangouts.” Let’s move to another tool or just go hang out in Google. Because they're in Drupal, they might be reading the post for their class, their forums for different classes.

They might be seeing it … you can go into Drupal and see if a professor is online. You can see if a classmate that you're collaborating with is online. We have breakout groups and people work in groups in the different classes and they can just quickly go online and see if they're online. Then if they want to go to Google, hang out, that’s fine. We don’t have a current plan to integrate video, but we will see what people are asking for.

Again we are not looking to re-invent the wheel. We are getting great feedback that people like to see who’s available and maybe just send the teacher or classmate a quick question. We’ve gotten some great feedback there. I spent a lot of time with jabber and I definitely didn’t talk about Drupal as much. The cool stuff with Drupal is not only pulling in the documents, but the APIs where you can … let’s see if I can get … this is where I upload the stuff to drupal.org. Here at Googletest.php, I keep a lot of these things commented out.

You can create a collection, they're commented out because they only do one test at a time and I comment out the unused test. You can create through the API, just through the command line a collection which is a folder basically in Google drive. This looks like one line here and there, change permissions on a folder; you can add users, retrieve user information. It’s got a very rich library that we've been able to contribute to the community. Babson has been very generous in letting us share.

Our hope is that not only will people consume this module, but maybe other people will have to contribute and I’ll be happy to work with you if you want to be a co-contributor. If you want to just add patches, I love if there are patches then we can make Droogle and jabber better with everybody’s help. I love hearing if there are any problems, if you have any bugs, it helps us. Sometimes the community finds the bugs before we do. We haven’t had any bug reports in a while.

There are some issues with Droogle with the version of PHP. I’ll be addressing that shortly. Possibly using Droogle as you say in Drupal Six. I'm a little slower in finding the Drupal Seven bugs but the Drupal Seven version, it is out there. Jabber we are using in Drupal Six and Drupal Seven. I think we are going to open things up to questions. I'm not sure Brian or Janet are listening, but …

Janet: Hi James. Yes I'm right here. We had a couple of questions coming in. I just want to announce that if anybody has any additional questions, please ask them in the Q&A tab of the WebEx that you're right now. The first question we had was a question on the Google docs’ module. Does the user must have a Google account? They are asking do you have to have a Google account for the Google docs’ module.

James: Yes, the answer is yes. You can have an individual account if you like and you can store your docs in your Google accounts. If you have a personal account and just have your own blog, yes you can do it that way. You’ll probably be using this site-wise context and not the organic group context. Yes you can definitely do that. Or if you have a Google for business account, or Google for education account, they're really the same thing. You can do it that way as well. Yes you can do either way but you need a Google account.

Janet: Okay, great. The next one is, is there a way to use the Google docs module with files depot module?

James: I'm not familiar with the file depot module, so I can’t really speak to that.

Janet: Okay. Great. That’s it for questions. If anybody has any other question, could you please ask them now? All right, I guess that's it. Thank you everyone for attending. Thank you James for the great presentation. Again files and recording of the webinar will be posted to the website (acquia.com) 48 hours and we will email them out to you as well. Thank you.

Pages