airflow-dag-patterns

Production-ready patterns for Apache Airflow DAGs, operators, sensors, testing, and deployment. Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies. - Covers DAG design principles (idempotent, atomic, incremental, observable) with task dependency patterns for linear, fan-out, fan-in, and complex workflows - Includes TaskFlow API decorators for cleaner code with automatic XCom passing, dynamic DAG generation from configs, and branching with conditional logic - Provides sensor patterns for S3 files, external task dependencies, and custom sensors; error handling with callbacks and trigger rules; and testing strategies with pytest fixtures - Best practices section covers idempotency, timeouts, worker slot management, and anti-patterns like hardcoded dates and stateful tasks - Creating data pipeline orchestration with Airflow - Designing DAG structures and dependencies - Implementing custom operators and sensors - Testing Airflow DAGs locally - Setting up Airflow in production - Debugging failed DAG runs - Use TaskFlow API - Cleaner code, automatic XCom - Set timeouts - Prevent zombie tasks - Use mode='reschedule' - For sensors, free up workers - Test DAGs - Unit tests and integration tests - Idempotent tasks - Safe to retry - Don't use depends_on_past=True - Creates bottlenecks - Don't hardcode dates - Use {{ ds...