ml-pipeline-workflow

Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment. This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring. Pipeline Architecture Data Preparation Model Training Model Validation Deployment Automation See the references/ directory for detailed guides: The assets/ directory contains: Data Preparation Phase Training Phase Validation Phase Deployment Phase Start with the basics and gradually add complexity: After setting up your pipeline: - Building new ML pipelines from scratch - Designing workflow orchestration for ML systems - Implementing data → model → deployment automation - Setting up reproducible training workflows - Creating DAG-based ML orchestration - Integrating ML components into production systems - Pipeline Architecture - End-to-end workflow design - DAG orchestration patterns (Airflow, Dagster, Kubeflow) - Component dependencies and data flow - Error handling and retry strategies - Data Preparation - Data validation and quality checks - Feature engineering pipelines - Data versioning and lineage - Train/validation/test splitting strategies - Model Training - Training job orchestration - Hyperparameter management - Experiment tracking integration - Distributed training patterns - Model...