Asynchronous processing with Sidekiq gem in Rails

Introduction to Sidekiq

Sidekiq gem is used to move long running jobs to background for asynchronous processing.

It is more efficient, with respect to memory usage, than delayed_job and Resque as it uses threads instead of forks.

Need of background process

For example, in a mailing application you need to send emails to a large list of recipients. It will take large time to process with the large list of recipients, which is too long for a User to wait for a response after clicking a link. So, it’s better to move the long running task as the background process by using Sidekiq gem.

Integration of Sidekiq in a Rails application

Steps to integrate Sidekiq gem in a Rails Application :

Step#1 – Add the gem to the Gemfile to use Sidekiq with Active Record

gem 'sidekiq'

You can also view the web interface to manage the background jobs. It isn’t included by default with Sidekiq. To use the web interface we need to add couple of gems to the Gemfile like ‘sinatra‘ and ‘slim‘

 gem 'sidekiq', '2.9.0'
 gem 'sinatra', '1.3.6', require: false
 
 gem 'slim', '1.3.6'

Run “bundle install” to install the sidekiq gem

Step#2 – Install Redis Server to manage its job queue

Redis is an open-source, advanced key-value pair store. It is often referred to as a data structure server.
Sidekiq uses Redis to manage its job queue.

If you’re running OS X follow the steps as below:
Logged in as a Admin user or add sudo prefix on terminal

$ brew install redis

After installation completed starts the Redis server up with the following command

$ redis-server /usr/local/etc/redis.conf

if you’re running CentOS 5.6 follow the steps as below:

yum install make gcc wget telnet
 wget http://redis.googlecode.com/files/redis-2.2.12.tar.gz
 tar -xf redis-2.2.12.tar.gz
 cd redis-2.2.12
 make && make install

Change the default redis.conf file to daemonize it.

mkdir /etc/redis /var/lib/redis
sed -e "s/^daemonize no$/daemonize yes/" -e "s/^dir \.\//dir \/var\/lib\/redis\//" -e     "s/^loglevel debug$/loglevel notice/" -e "s/^logfile stdout$/logfile     \/var\/log\/redis.log/" redis.conf > /etc/redis/redis.conf

To make management easy, use an init script by using sed to update it.

 Wget https://raw.github.com/gist/257849/9f1e627e0b7dbe68882fa2b7bdb1b2b263522004/redis-server
 sed -i "s/usr\/local\/sbin\/redis/usr\/local\/bin\/redis/" redis-server
 
 chmod u+x redis-server
 
 mv redis-server /etc/init.d
 
 /sbin/chkconfig --add redis-server
 
 /sbin/chkconfig --level 345 redis-server on
 
 /sbin/service redis-server start

Step#3 – Start up the jobs process

There are couple of ways to do this,

• If application is in development mode, we should use the below rake task file instead.

bundle exec sidekiq

If application is in production mode, then it is preferred to daemonizes the job process and allows multiple background processes to be spawned. To use this, follow the following step:
1. Add gem “daemon-spawn”, “0.4.2” to your Gemfile
2. Run bundle install
3. Run the below command to run it in background

bundle exec sidekiq -d -L log/delayed_job.log

4. You have to pass path of log file while using above command.

Step#4 – Add task to run in background

We need to add the background work in a separate class and store it in a new directory, let’s say app/background_jobs and ensures that it’s auto-loaded by the application.

We need to include the Sidekiq::Worker module.

class NotifyUsers
   include Sidekiq::Worker
   def perform
     User.find_each(is_subscribed: true) do |user|
       NewsMailer.newsletter_mail(user).deliver
     end
   end
end

This class must have a perform method. Add the code that need be to run in the background. Then call NotifyUsers.perform_async method inside the controller which will add the job to Redis and then it will call function perform asynchronously.

For Example:
Before background job in UsersController

class UsersController < ApplicationController
  def send_email
    User.find_each(is_subscribed: true) do |user|
      NewsMailer.newsletter_mail(user).deliver
      flash[:notice] = "Mail delivered"
      redirect_to root_path
    end
  end
end

After implementation of UsersController after adding to background job

class UsersController < ApplicationController
   def send_email
      NotifyUsers.perform_async
         flash[:notice] = "Mail is being delivered"
         redirect_to root_path
      end
   end
end

Conclusion

• No need to wait for a response after clicking a link to do a big stuff
• Sidekiq has client-side middleware which runs before a job is inserted into Redis
• It has server-side middleware which runs before the job is being processed. This middleware retry jobs, log them and handle exceptions

SEE ALSO: How to Install and Configure Redis-Server on Centos/Fedora Server

Sidekiq handles the processing of the jobs after they’re pulled from Redis and multithreaded behavior.

It is also providing a web interface there are a couple of gems that we need to add to the gem file too. To visit the web interface just go to the /sidekiq path

It tells us how many jobs have been processed, the number of failures, the currently-active workers and the number of jobs in queues.

Like this blog? I’d love to hear about your thoughts on this. Thanks for sharing your comments.