Airflow Xcom Exclusive [best] -

: XComs are automatically cleaned up when DAG runs complete, but consider custom cleanup logic for custom backends.

If you are using traditional operators, you can make XComs exclusive by using custom keys and specifying the task_ids during the pull.

Integration:

This approach is particularly useful with the TaskFlow API, where a task's return value is automatically pushed to XCom. By explicitly passing these references as arguments to downstream tasks, you create an intuitive, functional declaration of your pipeline. airflow xcom exclusive

| Feature | Use Case | Persistence | | :--- | :--- | :--- | | | Passing dynamic data between specific tasks within a DAG run. | Persists for the duration of the DAG run (usually cleaned up eventually). | | Variables | Storing static configuration or global settings (e.g., API keys, environment names). | Persists globally until manually deleted. | | External Storage | Moving large datasets (files, large DataFrames). | Persists until externally deleted. |

You can manually push data at any point within your task execution context using context['ti'].xcom_push(key='my_key', value=data) . The Downstream Pull

: Rely on XCom only for small, idempotent, non-critical data. For exclusive workflows, redesign your DAG or bring your own locking mechanism. : XComs are automatically cleaned up when DAG

This is the cardinal rule of XCom. It is designed for passing small pieces of state, such as file paths, status strings, or row counts. Avoid passing large DataFrames, entire query results, or large JSON payloads. If you need to pass large datasets, consider having your tasks write the data to a shared location (like your data lake) and then just pass the file path via XCom.

: Many operators (and all functions decorated with @task in the TaskFlow API) automatically push their return value to a key called return_value .

To ensure your Airflow pipelines remain stable, scalable, and easy to maintain, follow these foundational XCom best practices: By explicitly passing these references as arguments to

: Pandas dataframes, large JSON logs, images, and heavy files. "Exclusive" Strategies for Advanced XCom Architecture

Where is your (AWS, GCP, local, etc.)? What type of data are you trying to share between tasks? Share public link

The , introduced in Airflow 2.0, provides a more intuitive way to work with XComs without explicit push/pull operations. It automatically handles XCom creation and dependency resolution behind the scenes.