Resque workers stop to queue jobs

Problem

Resque workers are failing to pick up and trigger new background jobs, loaded by Rails command resque.enqueue

Redis Database is up and running, workers have been restarted after latest deploy and from Redis monit they seem to be waiting for jobs. 

Explanation

Issue was identified as being Redis-specific. More specifically, Redis keys related to Resque jobs were not deleted although their TTL configuration of 360 seconds was over. As a result, new jobs for the stores that their jobs were stuck on this invalid state were unable to be queued, as Redis was returning a pre-existing key.

When a new Resque task arrived with the same ID before TTL expired, the following row was appearing on Redis :

1521713036.806469 "get" "resque:solo:queue:shortjobs:job:b7ad5683bc1342a31dc9a30f12247887"

Solution

After executing FLUSHALL on redis-cli to clean up the whole Redis Database, it started adding jobs again. To avoid problem to return, no new job for the same store can be queued before the TTL (of 360 seconds, in this specific case) expires, so an additional check must take place.

Comments

0 comments

Article is closed for comments.