Ruby: Count Articles by Category [Elastic Search]

Have you ever wanted to count how many articles will be left in a certain category after search? To give you an idea of what I mean, as an example let’s use the application that will be the outcome of this tutorial. Our database contains music albums which are categorized by the genre and subgenre.

Arek Poczobut

August 27, 2018 • 9 min read

After searching for, let’s say ‘Miles’ the album count in a category will change according to how many articles (albums in this case) meet the search requirement.

My solution utilizes Ruby on Rails elastic aggregations and Drapper. The best thing about it is that it doesn’t make additional requests to the database. Elastic makes the searching process much easier and faster. A solution on the basis of a database is much more time-consuming to develop and heavier on the server than mine.

The only downside I can think of is that you need to create an additional service.

Steps

I'll skip steps taken to create the app and to add some layout. I'll use controller articles with action index for listing and search.

Install prerequisites:

Elastic search or use docker image
Add these gems to the Gemfile:
- gem 'elasticsearch-model'
- gem 'elasticsearch-rails'
- gem 'draper'
Github: 'drapergem/draper'
Bundle install
Add elastic initializer to point elastic host

1 config = {
2    host: 'http://elasticsearch:9200',
3    transport_options: {
4      request: { timeout: 5 }
5    }
6  }
7  Elasticsearch::Model.client = Elasticsearch::Client.new(config)

tsx

Create models:

1   rails g model category name:string
2    rails g model author name:string
3    rails g model article name:string authors:references
4    rails g model article_category article:references category:references
5    rake db:migrate
6    rails generate draper:install

tsx

Create a decorator for the Category:

1rails generate decorator Category

tsx

Add attr_accessor :article_count to the decorator of the class.

Create collection decorator

Add an apply_counts method with a buckets_counts argument.

1   class CategoriesDecorator < Draper::CollectionDecorator
2      def apply_counts(buckets_counts)
3        each { |obj| obj.article_count = buckets_counts[obj.id] }
4      end
5    end

tsx

Later this will allow to set the article_count based on aggregations.

Add module `Searchable` to `concerns`

1 module Searchable
2      extend ActiveSupport::Concern
3
4      included do
5        include Elasticsearch::Model
6        include Elasticsearch::Model::Callbacks
7
8        def index_document
9          __elasticsearch__.index_document
10        end
11      end
12
13      module ClassMethods
14        def recreate_index!
15          __elasticsearch__.create_index! force: true
16          __elasticsearch__.refresh_index!
17        end
18      end
19    end

tsx

The Article model

Include Searchable:

Add associations:

1has_many :article_categories
2  has_many :categories, through: :article_categories

tsx

Delegate author_name:

1 delegate :name, to: :author, prefix: true

tsx

In the models folder create Articles::Index module to define the elastic index:

1module Articles
2    module Index
3    extend ActiveSupport::Concern
4      included do
5        index_name "article-#{Rails.env}"
6        settings index: { number_of_shards: 1 } do
7          mappings dynamic: 'false' do
8            indexes :name, type: :text, analyzer: 'english'
9            indexes :author_name, type: :text, analyzer: 'english'
10            indexes :category_names, type: :text, analyzer: 'english'
11            indexes :category_ids, type: :integer
12          end
13        end
14      end
15      def as_indexed_json(*)
16        {
17          name: name,
18          author_name: author_name,
19          category_names: category_names,
20          category_ids: category_ids
21        }
22      end
23      private
24      def category_names
25        categories.pluck(:name).compact.uniq
26      end
27      def category_ids
28        categories.pluck(:id).compact.uniq
29      end
30    end
31  end

tsx

type: :text is for thetext search.

indexes :category_ids and type: :integer will allow to aggregate the results by category.

Include the Articles::Index in Article model.

Seed data

I've prepared the seed with some jazz albums with assigned jazz sub-genres.

1jazz = Category.create(name: 'jazz')
2fusion = Category.create(name: 'jazz fusion')
3bebop = Category.create(name: 'bebop')
4cool = Category.create(name: 'cool jazz')
5
6author = Author.create(name: 'Miles Davis')
7['Bitches Brew', 'A Tribute to Jack Johnson', 'Miles In The Sky', 'Pangaea'].each do |title|
8  article = Article.create(name: title, authors: author )
9  ArticleCategory.create(category: jazz, article: article)
10  ArticleCategory.create(category: fusion, article: article)
11end
12
13['Kind of Blue', 'Sketches Of Spain', 'Birth of the Cool', 'Porgy And Bess'].each do |title|
14  article = Article.create(name: title, authors: author )
15  ArticleCategory.create(category: jazz, article: article)
16  ArticleCategory.create(category: cool, article: article)
17  ArticleCategory.create(category: bebop, article: article)
18end
19
20author = Author.create(name: 'Sonny Rollins')
21['Sonny Rollins With The Modern Jazz Quartet'].each do |title|
22  article = Article.create(name: title, authors: author )
23  ArticleCategory.create(category: jazz, article: article)
24  ArticleCategory.create(category: cool, article: article)
25end
26
27['Next Album', 'Easy Living', 'The Way I Feel ', "Don't Stop the Carnival"].each do |title|
28  article = Article.create(name: title, authors: author )
29  ArticleCategory.create(category: jazz, article: article)
30  ArticleCategory.create(category: fusion, article: article)
31end
32
33['Saxophone Colossus', 'Plus Three'].each do |title|
34  article = Article.create(name: title, authors: author )
35  ArticleCategory.create(category: jazz, article: article)
36  ArticleCategory.create(category: bebop, article: article)
37end
38
39author = Author.create(name: 'Chet Baker')
40['Chet', 'My Funny Valentine'].each do |title|
41  article = Article.create(name: title, authors: author )
42  ArticleCategory.create(category: jazz, article: article)
43  ArticleCategory.create(category: cool, article: article)
44end
45
46author = Author.create(name: 'Paul Desmond')
47['Feeling Blue', 'Bossa Antigua', "We're all together again"].each do |title|
48  article = Article.create(name: title, authors: author )
49  ArticleCategory.create(category: jazz, article: article)
50  ArticleCategory.create(category: cool, article: article)
51end
52
53author = Author.create(name: 'Dave Brubeck')
54['Concord on a Summer Night', 'Time Further Out', "Time Out"].each do |title|
55  article = Article.create(name: title, authors: author )
56  ArticleCategory.create(category: jazz, article: article)
57  ArticleCategory.create(category: cool, article: article)
58end
59
60author = Author.create(name: 'The Mahavishnu Orchestra')
61['Birds Of Fire', 'Between Nothingness & Eternity', 'The Inner Mounting Flame'].each do |title|
62  article = Article.create(name: title, authors: author )
63  ArticleCategory.create(category: jazz, article: article)
64  ArticleCategory.create(category: fusion, article: article)
65end
66
67Article.recreate_index!
68Article.import

tsx

The last two lines in the seed create the index in elastic, so:

1rake db:seed

tsx

Now we can take care of searching and aggregates.

Oh look! We're halfway through the post! Here's a picture of a cute kitten:

Add a simple class for the search form:

1class SearchForm
2  include ActiveModel::Model
3
4  attr_reader :search_text
5
6  def initialize(search_text)
7    @search_text = search_text
8  end
9end

tsx

Add search query object:

1class SearchQuery
2  def initialize(search_form)
3    @search_form = search_form
4  end
5
6  def call
7    Article.search(search_text).records.to_a
8  end
9
10  private
11
12  attr_reader :search_form
13
14  delegate :search_text, to: :search_form
15end

tsx

Search with elastic could be as simple as Article.search(search_text).records.to_a but we can ask elastic to count something for us in one go.

To do this we will need a little bit more complex query which we'll prepare using elastic DSL and put as an argument to the search method.

All methods beneath are private.

Search definition object will do almost the same thing as the search above.

1def query
2      {
3        size: 100,
4        from: 0,
5        query: simple_query
6      }
7    end
8
9    def match_all
10      { match_all: {} }
11    end
12
13    def simple_query
14      return match_all if search_text.blank?
15      {
16        query_string: {
17          query: add_wildcards(search_text)
18        }
19      }
20    end
21
22    def add_wildcards(text)
23      text.split(' ').map { |el| "*#{el}*" }.join(' ')
24    end

tsx

Attributes :size, :from are for paging, default elastic page size is 10.

How to add aggregations? Using elastic DSL allows us to define aggs criteria:

1     def aggs_categories
2        {
3          by_categories:{
4            terms:{
5              field: :category_ids
6            }
7          }
8        }
9      end

tsx

And add them to search definition object:

1     def query
2        {
3          size: 100,
4          from: 0,
5          query: simple_query,
6          aggs: aggs_categories
7        }
8      end

tsx

Result of our query object at the end should look like a.e.

1{
2    :size => 100,
3    :from => 0,
4    :query => {
5      :query_string => {
6        :query => "*miles*"
7        }
8      },
9    :aggs => {
10      :by_categories => {
11        :terms=> { :field =>: category_ids }
12      }
13    }
14  }

tsx

When we assign a search object to a search variable as search = Article.search(query) then we check search.response on search object which should look like this:

1    {
2        "took"=>62,
3        "timed_out"=>false,
4        "_shards"=>{"total"=>1, "successful"=>1, "skipped"=>0, "failed"=>0},
5        "hits"=> {
6          "total"=> 8,
7          "max_score" => 1.0,
8          "hits" => [
9            {"_index"=>"article-development", "_type"=>"article", "_id"=>"1", "_score"=>1.0, "_source"=>{"name"=>"Bitches Brew", "author_name"=>"Miles Davis", "category_names"=>["jazz", "jazz fusion"], "category_ids"=>[1, 2]}},
10            {"_index"=>"article-development", "_type"=>"article", "_id"=>"2", "_score"=>1.0, "_source"=>{"name"=>"A Tribute to Jack Johnson", "author_name"=>"Miles Davis", "category_names"=>["jazz", "jazz fusion"], "category_ids"=>[1, 2]}},
11            {"_index"=>"article-development", "_type"=>"article", "_id"=>"3", "_score"=>1.0, "_source"=>{"name"=>"Miles In The Sky", "author_name"=>"Miles Davis", "category_names"=>["jazz", "jazz fusion"], "category_ids"=>[1, 2]}},
12            {"_index"=>"article-development", "_type"=>"article", "_id"=>"4", "_score"=>1.0, "_source"=>{"name"=>"Pangaea", "author_name"=>"Miles Davis", "category_names"=>["jazz", "jazz fusion"], "category_ids"=>[1, 2]}},
13            {"_index"=>"article-development", "_type"=>"article", "_id"=>"5", "_score"=>1.0, "_source"=>{"name"=>"Kind of Blue", "author_name"=>"Miles Davis", "category_names"=>["jazz", "bebop", "cool jazz"], "category_ids"=>[1, 3, 4]}},
14            {"_index"=>"article-development", "_type"=>"article", "_id"=>"6", "_score"=>1.0, "_source"=>{"name"=>"Sketches Of Spain", "author_name"=>"Miles Davis", "category_names"=>["jazz", "bebop", "cool jazz"], "category_ids"=>[1, 3, 4]}},
15            {"_index"=>"article-development", "_type"=>"article", "_id"=>"7", "_score"=>1.0, "_source"=>{"name"=>"Birth of the Cool", "author_name"=>"Miles Davis", "category_names"=>["jazz", "bebop", "cool jazz"], "category_ids"=>[1, 3, 4]}},
16            {"_index"=>"article-development", "_type"=>"article", "_id"=>"8", "_score"=>1.0, "_source"=>{"name"=>"Porgy And Bess", "author_name"=>"Miles Davis", "category_names"=>["jazz", "bebop", "cool jazz"], "category_ids"=>[1, 3, 4]}}
17          ]
18        },
19        "aggregations" => {
20          "by_categories" => {
21            "doc_count_error_upper_bound"=>0,
22            "sum_other_doc_count"=>0,
23            "buckets"=>[
24              {"key"=>1, "doc_count"=>8},
25              {"key"=>2, "doc_count"=>4},
26              {"key"=>3, "doc_count"=>4},
27              {"key"=>4, "doc_count"=>4}
28            ]
29          }
30        }
31      }

tsx

In aggregations => by_categories we can find buckets and that's what we're interested in! Key buckets contain counts for category_ids.

Extract them:

1 def buckets_categories_counts
2    @categories_counts ||= search.response
3       .deep_symbolize_keys[:aggregations][:by_categories][:buckets]
4       .map{ |bucket| OpenStruct.new(bucket) }
5  end

tsx

Map categories_ids:

1 def buckets_categories_ids
2    buckets_categories_counts.map(&:key)
3  end

tsx

Prepare the bucket hash:

1 def buckets_hash
2    buckets_categories_counts.each_with_object({}) do |bucket, obj|
3      obj[bucket.key]= bucket.doc_count
4    end
5  end

tsx

Find categories, decorate the collection and apply counts:

1 def categories
2    ::CategoriesDecorator.decorate(
3      Category.where(id: buckets_categories_ids)
4    ).apply_counts(buckets_hash)
5  end

tsx

Update public method call:

1    def call
2       OpenStruct.new(
3         categories: categories,
4         articles: search.records.order('articles.name asc').includes(:author).to_a
5       )
6     end

tsx

We will return object with categories and articles.

Last step is to add some logic to `ArticlesController`

1  class ArticlesController < ApplicationController
2     def index
3       @search_form = SearchForm.new(search_text)
4       result = SearchQuery.new(@search_form).call
5       @articles = result.articles
6       @categories = result.categories
7     end
8
9     private
10
11     def search_text
12       params.dig(:search_form, :search_text)
13     end
14   end

tsx

Working app

That's all, you can check working example downloading repo:

Clone or download repo
Install docker if needed
Run

1docker-compose build 
2docker-compose run web bundle install 
3docker-compose run web rake db:create db:migrate db:seed

tsx

Web Development

Let’s Create a Great Website Together

We'll shape your web platform the way you win it!