Tutorial - how to skip Sentry Slack notifications

We’ve been working on integrations of many different warehouse systems with the Shopify platform. All data exchange between them utilizes Sidekiq workers’ background jobs. Generally, we want to be notified about the first occurrence of an error. So most exceptions are caught by Raven and sent to Sentry. However, we faced some exceptions at remote systems, for example, connection issues. Luckily, after some worker retries the problems were solved without any additional actions. In such cases we wanted Sidekiq workers to have silent retries without spamming our Slack channel with Sentry messages.

The bad news is that Sidekiq doesn’t offer access to retry_count param from a worker. Fortunately, Sidekiq offers us developers middleware that allows us to add a functionality which has access to job attributes including retry_count. Raven allows to specify in config should_capture where we will add Proc, where we exclude custom error Sidekiq::SilentRetryError.

Let’s start with registering our retry middleware.

1Sidekiq.configure_server do |config|
2  config.server_middleware do |chain|
3    chain.add Middleware::Sidekiq::RetryMonitoring
4  end
5end

We want to delay some specific network/api errors, so let’s define an array that contains some of them.

1SILENT_RETRY_ERRORS = [
2  EOFError, Errno::ECONNRESET, Errno::EINVAL, Errno::ECONNREFUSED,
3  Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError,
4  Net::SSH::Exception, Timeout::Error, SocketError,
5  ActiveResource::ServerError, ActiveResource::TimeoutError
6]

In the next step we define a module which we include in our worker. This module will help us identify monitored worker and also allow us to define retry_count _for_sentry there.

1module Middleware::Sidekiq::RetryMonitoring::MonitoredWorker
2  extend ActiveSupport::Concern
3
4  included do
5    def retry_count_for_sentry
6      10
7    end
8  end
9end

Our custom error class:

1module Sidekiq
2  class SilentRetryError < StandardError; end
3end

In our middleware, we rescue errors for which we want to delay Sentry notifications. If worker’s retry_count is lower than our retry_count_for_sentry then we have to replace the original exception with a custom one and raise Sidekiq::SilentRetryError – otherwise we would have to reraise the original one. Without that, Sidekiq treats the job as completed.

1class Middleware::Sidekiq::RetryMonitoring
2  def call(worker, job, queue)
3    begin
4      yield
5    rescue *SILENT_RETRY_ERRORS => e
6      if silent_error?(worker, job)
7        raise Sidekiq::SilentRetryError.new([e.class, e.message].join(" "))
8      else
9        raise
10      end
11    end
12  end
13
14  private
15
16  def silent_error?(worker, job)
17   retry_count = job["retry_count"].present? ? job["retry_count"].to_i + 1 : 0
18   worker.is_a?(Middleware::Sidekiq::RetryMonitoring::MonitoredWorker) &&
19        retry_count < worker.threshold_retry_count_for_sentry
20  end
21end

The last thing to do is to define should_capture for Raven. We can define Proc which checks if the exception contains Sidekiq::SilentRetryError.

1Raven.configure do |config|
2  config.should_capture = Proc.new do |e|
3    e.to_s.exclude?("Sidekiq::SilentRetryError".freeze)
4  end
5end