airflow 2分布式集群中使用CeleryExecutor 在指定节点上启用task

一、环境

root@node1:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.5 LTS
Release:	22.04
Codename:	jammy
root@node1:~# airflow version
2.10.4

root@node3:~# celery --version
5.4.0 (opalescent)

二、配置dag和worker

启用celery worker时使用 queues参数，打上队列标签

# Worker 1 监听 queue1
airflow celery worker --queues=queue1

# Worker 2 监听 queue2
airflow celery worker --queues=queue2

在Dag的task中标上queue

    # [START basic_task]
    t1 = BashOperator(
        task_id="fping1",
        bash_command="fping -c 4 www.google.cn",
        queue='queue1',  # 指定任务运行在 queue1
    )

    t2 = BashOperator(
        task_id="fping2",
        bash_command="fping -c 5 www.google.cn",
        queue='queue2',  # 指定任务运行在 queue2