Use batch loading & lazy relationships to eliminate N+1 queries in Ruby on Rails

4 min readJan 9, 2019

TLDR: Use ams_lazy_relationships gem ( https://github.com/Bajena/ams_lazy_relationships) to eliminate N+1 queries in ActiveModel::Serializers 🎉

Quick intro to batch loading

A few months ago I came across a post by exAspArk explaining how they built and used the BatchLoader gem to optimize the number of database queries in their apps.

The topic of batch loading was new to me back then, but it sounded like a perfect fit for our Rails app in Leadfeeder, because the recommended Rails way of using includes wasn’t flexible enough for many of our complex endpoints.

Quick explanation for people who just asked “what the hell is batch loading”:
Batch loading is a method for preventing unnecessary DB calls.
If we need to load a collection of records (e.g. for each blog post load its author), then instead of calling the DB to get every author we can collect author_ids, run one DB query and then assign the author to every blog post.
This example should give you a better idea of how it works.

Limitations of BatchLoader gem

After getting more familiar with the topic it turned out that there’s already a lot of work done in the GraphQL world, but unfortunately if you’re still an old-fashioned REST API guy like me you’ll quickly notice that there’s no Ruby gem that’d nicely wrap the batch loading functionality, so I started experimenting with it myself.

First I followed Usama Ashraf’s approach from this post and it worked quite well, however I stumbled upon two problems with this code:

It generates a lot of repeatable “boilerplate” code that pollutes the serializers a lot.
When requesting deeply nested relationships (e.g. blog_post.comments.author) ActiveModel::Serializers gem will still perform N+1 queries for the author relationship due to the way it serializes the records.

I decided to write my own gem based on BatchLoader to tackle these issues.

Lazy Relationships to the rescue!

The gem is called ams_lazy_relationships. It’s an extension for ActiveModel::Serializers gem based on the above-mentioned BatchLoader gem.

It introduces the concept of lazy relationships. Lazy relationships are methods (lazy_has_many/lazy_has_one/lazy_belongs_to) that wrap has_many/has_one/belongs_to relationship methods provided by ActiveModel::Serializers.

Lazy relationships are cool because:

They prevent N+1 queries when serializing complex object tress by using batch loading
They do not load excessive data (like Rails includes when improperly used).
They let you remove N+1s even when not all relationships are ActiveRecord models (e.g. some records are stored in a MySQL DB and other models are stored in Cassandra)

Probably this thing isn’t still 100% clear to you, so in the next paragraphs I’ll show you how to install the gem and explain the concept on a few examples.

Installation

The installation process is rather simple:

Include AmsLazyRelationships::Core module in your base serializer:

class BaseSerializer < ActiveModel::Serializer
  include AmsLazyRelationships::Core
end

2. As this gem uses BatchLoader heavily I highly recommend clearing the batch loader's cache between HTTP requests. To do so add a following middleware: config.middleware.use BatchLoader::Middleware to your app’s application.rb.

For more info about the middleware check out BatchLoader gem docs: exAspArk/batch-loader#caching

Example 1: Basic ActiveRecord relationships

If the relationships in your serializers are plain old ActiveRecord relationships you’re lucky, because ams_lazy_relationships by default assumes that the relationship is an ActiveRecord relationship, so you can use the simplest syntax.

Imagine you have an endpoint that renders a list of blog posts and includes their comments.

The N+1 prone way of defining the serializer would be:

class BlogPostSerializer < BaseSerializer
  has_many :comments
end

To prevent loading comments using a separate DB query for each post just change it to:

class BlogPostSerializer < BaseSerializer
  lazy_has_many :comments
end

Example 2: Modifying the relationship before rendering

Sometimes it may happen that you need to process the relationship before rendering, e.g. decorate the records. In this case the gem provides a special method (in our case lazy_comments) for each defined relationship. Check out the example — we’ll decorate every comment before serializing:

class BlogPostSerializer < BaseSerializer
  lazy_has_many :comments do
    lazy_comments.map(&:decorate)
   end
end

Example 3: Introducing loader classes

Under the hood ams_lazy_relationships uses special loader classes to batch load the relationships. By default the gem uses serializer class names and relationship names to instantiate correct loaders, but it may happen that e.g. your serializer’s class name doesn’t match the model name (e.g. your model’s name is BlogPost but the serializer’s name is PostSerializer).

In this case you can define the lazy relationship by passing a correct loader param:

class PostSerializer < BaseSerializer
  lazy_has_many :comments, serializer: CommentSerializer, 
    loader: AmsLazyRelationships::Loaders::Association.new(
              "BlogPost", :comments
            )
end

You can find all the available loader classes here.

Example 4: Non ActiveRecord -> ActiveRecord relationships

This one is interesting. It may happen that the root record is not an ActiveRecord model (e.g. a Cequel model), however its relationship is an AR model.

Imagine that BlogPost is not an AR model and Comment is a standard AR model. The lazy relationship would look like this:

class BlogPostSerializer < BaseSerializer
  lazy_has_many :comments, 
    loader: AmsLazyRelationships::Loaders::SimpleHasMany.new(
        "Comment", foreign_key: :blog_post_id
     )
end

Example 5: Use lazy relationship without rendering it

Sometimes you may just want to make use of lazy relationship without rendering the whole nested record.
For example imagine that your BlogPost serializer is supposed to render author_name attribute. You can define the lazy relationship and just use it in other attribute evaluator:

class BlogPostSerializer < BaseSerializer
  lazy_relationship :author
  
  attribute :author_name do
    lazy_author.name
  end
end

Summary

I hope my idea of lazy relationships appealed to you. If you managed to make use of the gem in your project or have any improvement ideas let me know in the comments or create an issue/PR here: https://github.com/Bajena/ams_lazy_relationships