【celery】任务重复执行

问题

celery定时任务里面启用延时任务，出现延时任务重复执行的问题。
如：
定时任务：

'project_schedule_task': {
            'task': 'apps.project.tasks.project_status_monitor',
            'schedule': crontab(minute=0, hour=2),
        }

project_status_monitor任务执行时，再生成若干个延迟任务：

send_company_project_daily_remind_msg.apply_async(
                (company.name,
                 date_str,
                 company.push_title,
                 company.push_content,
                 None if company.push_weather_enable is False else city_weather_map,
                 project_list,
                 exec_time),
                countdown=company.push_time*60)

这时如果countdown配置的时间太长，该任务回重复执行多次。

问题原因及解决方法

• 问题原因
  查看celery官网有段对redis的说明：https://docs.celeryproject.org/en/latest/getting-started/brokers/redis.html

Visibility timeout
If a task isn’t acknowledged within the Visibility Timeout the task will be redelivered to another worker and executed.

This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visibility timeout; in fact if that happens it will be executed again, and again in a loop.

So you have to increase the visibility timeout to match the time of the longest ETA you’re planning to use.

Note that Celery will redeliver messages at worker shutdown, so having a long visibility timeout will only delay the redelivery of ‘lost’ tasks in the event of a power failure or forcefully terminated workers.

Periodic tasks won’t be affected by the visibility timeout, as this is a concept separate from ETA/countdown.

You can increase this timeout by configuring a transport option with the same name:

app.conf.broker_transport_options = {'visibility_timeout': 43200}
The value must be an int describing the number of seconds.

原因是对于eta/countdown延迟任务，有超时时间，如果超过超时时间任务未被执行，会被丢到下一个worker去执行，造成循环执行。当我们设置一个ETA时间比visibility_timeout长的任务时，每过一次 visibility_timeout 时间，celery就会认为这个任务没被worker执行成功，重新分配给其它worker再执行

• 解决方法
  重设超时等待时间：

app.conf.broker_transport_options = {'visibility_timeout': 86400}

排查过程中用到的一些命令记录

• 查看celery eta任务队列

celery -A proj inspect scheduled
celery -A proj inspect scheduled |wc -l

• 关闭任务

celery -A proj control terminate 

• 关闭worker

pkill -9 -f 'celery worker'
ps auxww | awk '/celery worker/ {print $2}' | xargs kill -9

• 重启worker

celery multi start 1 -A proj -l info -c4 --pidfile=/var/run/celery/%n.pid
celery multi restart 1 --pidfile=/var/run/celery/%n.pid

参考：https://docs.celeryproject.org/en/stable/userguide/workers.html