Home / Comment permalink

Battleplan for Search & Solr in Drupal 8


Contrib Search maintainers are committed to make Drupal 8 kick ass with Search API.

Search is a massively cool technology spectrum with loads of really tough problems such as language stemming, delivering search as a site scales and helping customers actually find what they want. However, solving these is not so easy. Let’s look at the history of Search and Solr in Drupal.

Drupal 6

Drupal 6 got a great Apache Solr Search module, thanks to PWolanin and RobertDouglass. There was a 1.x and even a 2.x branch early on. The 2.x branch was loaded with cool ideas but only saw most of its conversion in real working code in the Drupal 7 branch. The 2.x branch got deprecated and now we have 3.x. This is a straight backport from the Drupal 7 branch. At its most successful time the 6.x branch of the module was installed on around 7000 Drupal sites.

Drupal 7

Drupal 7 was the big start for me personally to get deeply involved in Search for Drupal. I had a 6 month internship at Acquia to improve the Apache Solr Search module and Acquia Search platform. The Drupal 7 module got a complete overhaul, fixing many bugs, improving the developer experience, and adding features like entity support and much more. The module is now used on more than 9100 sites.

At the same time we saw the Search API module emerging. For a while now both modules solved similar use cases with a very different architecture. Each module focussed on different use cases and were the right tool for the right job for many. However, some of the architectural decisions made it very hard for the Apache Solr Search project and the Search API project to combine efforts. Search API is now used by 24,000 websites, and 6300 of them use Apache Solr.

Not everyone understood the differences between the Apache Solr Search and Search API and this has led to confusion. Due to these architectural differences, some problems were solved twice by the community. For example, modules such as the facetapi_slider and searchapi_ranges tried to solve the same problem, which was to create a graphical slider for facets, twice. This duplication is lost effort in our collective goal of making Drupal the world class product for your search solution; we are basically slowing ourselves down with these two completely separate codebases.

Combined efforts

Druplicon Married Solr

During the last year combined efforts between the two projects took place, such as:

  • Facet API was created to become the common way to show facets blocks.
  • Presentations were given to show the difference and to encourage collaboration, for example the Solr Wizardry at Dev Days Barcelona 2012
  • common schema and solrconfig.xml were created with Acquia sponsorship . This effort made it much easier for users to switch between modules when trying them out on a shared hosting such as Acquia Search. It also offered many other benefits such as combined performance updates and more when we change something in solrconfig.xml. This change benefits more than just Acquia — it makes it easier for all hosting companies to support both modules.
  • Both maintainers worked together and migrated the connection classes from the Apache Solr Search module into Search API.

Drupal 8

With the upcoming release of Drupal 8 we are at an important point in time to decide if we want to step up and have an even bigger shared roadmap. At DrupalCon Portland, we started thinking but things got really interesting when we had meetings at DrupalCon Prague. In one of the meetings, we tried to identify common parts in the two systems and to see what we could share in Drupal 8. We were still going somewhat our separate ways but decided we could at least share the views query class and some other components.

At a second meeting at the DrupalCon Prague Sprint between drunkenmonkey and myself, we tried to identify why we couldn't make a unified solution. I explained some of the architectural problems some websites encounter with Search API and why it can't be used for the same use cases the Apache Solr Search module had in mind. This meeting was a big step in helping us  align visions and explain to each other what we wanted to achieve.

We then took some time to consider what we (as Acquia, but more importantly as a community) were going to do with the Apache Solr Search module for Drupal 8. Because we were able to see so much progress in sharing our vision and components, and to acknowledge and address some of the architectural challenges in the Search API, we decided to go fully on-board to make the Search API module a unified, kick-ass solution for search in Drupal 8.

All of the Apache Solr Search and Search API maintainers are aligned on this goal. There is a lot of work ahead of us if we want to make sure that all the good things of both worlds get combined and some bugs can be squashed. Naturally, we also have to get ready for the new way of working in Drupal 8, so it'll be a wild ride.

We have already started to fulfill this promise. We are working together with drunkenmonkey and the rest of the Search API crew to identify some of the current problems our customers have with Search API and to collaboratively solve those. If you want to see or help with some of these issues, I invite you to join the Search API issue queue, start reviewing, and chipping in. One of the issues that we are tackling is a better handling of the errors of indexing during cron runs. The solution that was proposed is to get rid of the Queue system, and while it sounds like a dramatic change, I can assure you it is not as dramatic as you'd think ;-).

There are some action items in order to move forward. I’m sure many more will be added in the course of the following months. There is already an issue in the Search API issue queue to list what is needed to upgrade Search API to Drupal 8. (https://drupal.org/node/2044421 and https://drupal.org/project/issues/search_api?version=8.x ). Here is a list of some of these issues that I think we need to have in Search API to make it a first-class search-solution.

If you have questions, comments, fireworks, please let us know below! And we hope to see you in the issue queue to make Drupal 8 kick ass with Search API!


Posted on by Jeff Geerling (not verified).

This is one of the best bits of news I've read this year! For many of my clients, I've had to help them decide which module would better fit their use case, and almost always, either module would work fine, just one would offer a few more 'icing on the cake' features.

Because of this, my familiarity with either module was always a little thin, and my ability to jump into the issue queue and help out was impeded.

Kudos to both groups of developers for getting together on making an already great search solution for Drupal way better for Drupal 8!

Posted on by drupaler (not verified).

Way to go! congrats!

Posted on by Ian Thomas (not verified).

Are you saying that you think that you can improve Search API to the point that a Drupal 8 version of apachesolr is unnecessary, or that you will be investing in both projects, but sharing as much code as possible.

Posted on by Nick Veenhof.

We think we can improve Search API to the point where every is happy and it's a stable and robust and fast solution for integration Solr in Drupal. There's a bunch of DX challenges to tackle so that changing solr queries stays easy and that, who want, can still just use Solr. But I believe we can do this. Apachesolr will become what search_api_solr is now, but with much less clutter.

Posted on by Kelly Lucas.

Super duper news!

With the performance improvements in Solr 4 (https://cwiki.apache.org/confluence/display/solr /Major+Changes+from+Solr...) I've heard that it is now a reasonable NoSQL option. Has there been any discussion about using Solr as a field/entity storage back-end ala the MongoDB module (https://drupal.org/project/m ongodb) and thus eliminating the need to "index" altogether?

Posted on by Pol (not verified).

I'm in ! :) Congrats!

Posted on by AeM (not verified).


it would be could for this api to be agnostic,
so that we could also use elasticsearch, the other lucene
search engine please !

so much easier to administer rather than solr :)

Posted on by Brian Altenhofel (not verified).

I maintain the search_api_elasticsearch module and I'm committed to following the D8 Search API development process as close as possible. The search_api_elasticsearch module is currently under active development with the current goal being a full-featured test suite. Because my hosting company offers Elasticsearch as a premium add-on to our service, this module is a very high priority for me.

Posted on by Jens Beltofte.

Some of my colleagues are developing and maintaining Elasticsearch Connector (https:// www.drupal.org/project/elasticsearch_connector). The D7 version is fully integrated with Search API, and the same is the goal for the D8 version in development.

Add new comment

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.

Filtered HTML

  • Use [acphone_sales], [acphone_sales_text], [acphone_support], [acphone_international], [acphone_devcloud], [acphone_extra1] and [acphone_extra2] as placeholders for Acquia phone numbers. Add class "acquia-phones-link" to wrapper element to make number a link.
  • To post pieces of code, surround them with <code>...</code> tags. For PHP code, you can use <?php ... ?>, which will also colour it based on syntax.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <h4> <h5> <h2> <img>
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.