Talk to Me
Healthcare Analytics

Cancer Risk Prediction Pipeline

End-to-end machine learning pipeline achieving 93% accuracy in predicting high-risk cancer patients using CMS Medicare claims data on Databricks

DatabricksPySparkDelta LakeMLflowscikit-learnPython
View on GitHubView Demo
93.15%
Accuracy
Model accuracy
100%
Recall
No missed cases
9,851
Cancer Cases
Identified
116K+
Patients
Processed

Project Overview

Built a production-grade machine learning pipeline to identify high-risk cancer patients at admission time, enabling healthcare providers to allocate resources effectively and improve patient outcomes through early intervention.

Delta Lake Architecture

Bronze Layer (Raw Data)

  • 116,352 patient records
  • 66,773 inpatient claims
  • CMS Medicare Synthetic Data

Silver Layer (Cleaned)

  • 9,851 cancer cases identified
  • ICD-9 code classification
  • Patient demographics joined

Gold Layer (Model-Ready)

  • 21 engineered features
  • No data leakage
  • Admission-time features only

Model Performance

MetricValueInterpretation
Accuracy93.15%Correctly identifies 93% of patients
Precision86.84%When model predicts high-risk, it's correct 87% of time
Recall100%Catches ALL high-risk patients (no false negatives)
F1 Score92.96%Strong balance of precision and recall
ROC AUC88.80%Excellent discrimination capability

Clinical Significance: 100% recall means no high-risk malignant cancer patients are missed by the model, which is critical for healthcare applications.

Problem Solving: Data Leakage

❌ Problem Identified

  • • Initial models: 100% accuracy
  • • Unrealistic performance
  • • Features included outcome variables:
  • - total_claim_amount
  • - length_of_stay_days
  • • Only known AFTER treatment

✅ Solution Implemented

  • • Removed all outcome variables
  • • Used only admission-time features
  • • Redefined target variable
  • • Achieved realistic 93% accuracy
  • • Production-ready model

Impact: This demonstrates strong ML fundamentals, production readiness awareness, and the ability to identify and resolve real-world ML problems.

Technical Implementation

Data Engineering

  • PySpark for big data processing
  • Delta Lake medallion architecture
  • Incremental data ingestion
  • Data quality validation

Machine Learning

  • scikit-learn models
  • MLflow experiment tracking
  • Hyperparameter logging
  • Model versioning

Features (No Data Leakage)

3 features

Demographics

  • • Age
  • • Gender
  • • Race
4 features

Clinical

  • • Cancer type
  • • Severity
  • • Comorbidities
3 features

Temporal

  • • Admission year
  • • Month
  • • Quarter
2 features

Patient History

  • • Prior claims
  • • Complexity

Project Structure

oncology-treatment-analysis/
├── 00_project_config/
│   └── project_config.py          # Configuration management
├── 01_data_ingestion/
│   └── ingest_cms_data.py         # Load raw data → Bronze layer
├── 02_data_processing/
│   └── data_cleaning_eda.py       # Clean & identify cancer cases → Silver
├── 03_feature_engineering/
│   └── feature_creation.py        # Engineer features → Gold layer
├── 04_model_training/
│   └── model_experiments.py       # Train & track models with MLflow
├── 05_model_evaluation/
│   └── model_evaluation.py        # Model comparison & selection
└── 06_deployment/
    └── model_deployment.py        # Model deployment

Key Takeaways

✅ Technical Excellence

  • •Production-grade ML pipeline
  • •Delta Lake architecture
  • •MLflow experiment tracking
  • •Data leakage resolution

✅ Healthcare Domain

  • •ICD-9 code processing
  • •Claims data analysis
  • •Clinical relevance
  • •100% recall priority

✅ ML Best Practices

  • •Identified data leakage
  • •Recognized overfitting
  • •Interpretable model
  • •Realistic validation

Interested in the Details?

Explore the complete codebase, notebooks, and documentation on GitHub

View Full ProjectGet in Touch