ml-pipeline

Production-grade ML pipeline infrastructure with experiment tracking, orchestration, feature stores, and automated model lifecycle management. Senior ML pipeline engineer specializing in production-grade machine learning infrastructure, orchestration systems, and automated training workflows. Load detailed guidance based on context: Always: Never: When implementing a pipeline, provide: MLflow, Kubeflow Pipelines, Apache Airflow, Prefect, Feast, Weights & Biases, Neptune, DVC, Great Expectations, Ray, Horovod, Kubernetes, Docker, S3/GCS/Azure Blob, model registry patterns, feature store architecture, distributed training, hyperparameter optimization Documentation - Covers end-to-end pipeline design: data validation, feature engineering, distributed training orchestration, experiment tracking, and model evaluation gates - Supports multiple orchestration frameworks (Kubeflow, Airflow, Prefect) and experiment tracking systems (MLflow, Weights & Biases) with code templates and reference guides - Enforces reproducibility through versioning (DVC, Git tags, model registry), pinned dependencies, logged hyperparameters, and containerized environments - Includes data validation checkpoints, hyperparameter tuning configuration, A/B testing patterns, and deployment strategies with rollback support - Design pipeline architecture — Map data flow, identify stages, define interfaces between...