Ruby on Rails Sidekiq Worker Fails to Complete Batch Processing Despite No Errors

Below is a list of possible causes and approaches you can take to debug and resolve the issue.

---

1. Mutation of the Query Scope During Iteration

Your worker’s query is

RUBYCopied
HeroProfile.analyze.where(analyzed_at: nil)

During iteration you update each record by calling:

RUBYCopied
hero_profile.update_column(:analyzed_at, Time.current)

Because your query condition is on analyzed_at: nil, updating a record removes it from the scope. When using ActiveRecord’s find_each, this dynamic change can have unexpected results (e.g., records shifting from one batch to another or the iteration “losing its way”). In some cases the internal batching logic might end up waiting for a “next” batch that never appears. If you intend to process all records, consider one of these approaches:

• Load IDs first: Retrieve all unsent IDs in one query into an array and then iterate over that list. That way you’re not altering the query as you update records.

• Process in batches without modifying the scope: Use find_in_batches to fetch a snapshot of records and then process the batch. For example:

  RUBYCopied
  HeroProfile.analyze.where(analyzed_at: nil).find_in_batches(batch_size: 10) do |batch|
    batch.each do |hero_profile|
      begin
        Rails.logger.info "Processing hero profile: #{hero_profile.id}"
        Hero::Profile::Reprove.call(hero_profile)
        hero_profile.update_column(:analyzed_at, Time.current)
      rescue StandardError => e
        Honeybadger.notify(error_class: e, error_message: 'Error processing hero profile')
        Rails.logger.error "Error processing hero profile #{hero_profile.id}: #{e.message}"
      end
    end
  end
  Rails.logger.info "Worker finished at: #{Time.current}"

This guarantees that what you iterate on does not change when you update analyzed_at.

---

2. A Single Record or Batch Hanging

Although no errors appear, one or more records could be taking unusually long in Hero::Profile::Reprove.call (for example, due to an external API call, a lock, or other I/O latency). When processing in a tight loop, a single record can stall the entire process. To check:

• Increase the logging around Hero::Profile::Reprove.call to measure timing.
• Consider adding a timeout around the call so that any hanging operation fails fast and is retried later.

Example using Ruby’s Timeout library:

RUBYCopied
require 'timeout'

begin
  Timeout.timeout(10) do  # maximum 10 seconds per call
    Hero::Profile::Reprove.call(hero_profile)
  end
rescue Timeout::Error
  Rails.logger.error "Timeout processing hero profile: #{hero_profile.id}"
  # Optionally, notify Honeybadger or handle the timeout accordingly.
end

---

3. Sidekiq or Environment Timeouts and Interruptions

Although you mentioned that no explicit error is logged, it’s also worth checking:

• Sidekiq Timeout Settings: Sidekiq has a default timeout for job shutdown. If your worker takes longer than that (e.g., during a graceful shutdown or deployment), it might be terminated before it logs “finished”. Ensure that your worker settings (and any custom middleware) are not interrupting long-running jobs.
• Server Process Environment: In production, if the server (or container) tends to kill long-running processes, you might not see an error within your Rails logs. Consider checking the hosting environment logs for any signals related to job termination.

---

4. Review Sidekiq Scheduling

Since you’re using sidekiq-scheduler, double-check if multiple scheduled jobs might overlap or interfere with each other. If a new execution is triggered while the previous one hasn’t finished, the concurrent execution might cause lock contention or unexpected behaviors.

• Disable overlapping runs if that’s not desired.
• Use a unique job identifier (via gems like sidekiq-unique-jobs) to ensure only one instance of that worker runs at a time.

---

Summary

• Dynamic query mutation: The most likely issue is that updating analyzed_at during iteration is affecting the result set of your find_each loop.
• Potential hanging call: Ensure no single record processing is stalling your loop.
• Sidekiq/environmental timeouts: Check if the job is terminated early by Sidekiq’s shutdown logic or your hosting environment.
• Job concurrency: Verify that overlapping runs are not interfering with processing.

By using a snapshot of IDs or processing via find_in_batches without modifying the original dataset during iteration, you should see more consistent job completions and a clear “finished” log.

Adjust your worker accordingly and monitor both the logs and any external timeouts to ensure completion.