datanalysis-credit-risk

The data cleaning pipeline consists of the following 11 steps, each executed independently without deleting the original data: The generated Excel report contains the following sheets: - Get Data - Load and format raw data - Organization Sample Analysis - Statistics of sample count and bad sample rate for each organization - Separate OOS Data - Separate out-of-sample (OOS) samples from modeling samples - Filter Abnormal Months - Remove months with insufficient bad sample count or total sample count - Calculate Missing Rate - Calculate overall and organization-level missing rates for each feature - Drop High Missing Rate Features - Remove features with overall missing rate exceeding threshold - Drop Low IV Features - Remove features with overall IV too low or IV too low in too many organizations - Drop High PSI Features - Remove features with unstable PSI - Null Importance Denoising - Remove noise features using label permutation method - Drop High Correlation Features - Remove high correlation features based on original gain - Export Report - Generate Excel report containing details and statistics of all steps - DATA_PATH : Data file path (best are parquet format) - DATE_COL : Date column name - Y_COL : Label column name - ORG_COL : Organization column name - KEY_COLS : Primary key column name list - OOS_ORGS : Out-of-sample organization list - min_ym_bad_sample : Minimum bad...