Urszula Urbaniak
MLOps Engineer — Model Quality · MLOps & Digital Engineering
Warsaw, Poland
MLOps Engineer specializing in model quality assurance and pipeline orchestration. Builds Kubeflow-based retraining pipelines, implements data quality validation gates using Great Expectations, and designs A/B testing frameworks for safe production model rollout. Focuses on making model quality measurable, automatic, and consistently enforced across all AI CoE production deployments.
Expertise
- ML model evaluation and testing frameworks
- Kubeflow pipeline orchestration
- model drift monitoring
- A/B testing for ML models
- data quality validation in ML pipelines
Technologies
Work History
2025-02
Model drift detection for production surrogate models — implemented statistical drift detection (Population Stability Index and Kolmogorov-Smirnov tests) on input feature distributions for 6 thermal and structural surrogate models.
Challenge: PSI-based drift detection was triggering false alarms for seasonal input distribution shifts — thermal analysis queries naturally shift toward higher ambient temperature inputs in summer. Required seasonally-adjusted baselines that compare current distribution to the same period in the previous year.
Learned: Drift detection for engineering analysis surrogates must account for seasonal input distribution cycles. Simple rolling-window baselines will falsely flag seasonal shifts as drift. Use year-on-year comparison (current month vs. same month last year) for aerospace simulation workloads with known seasonal usage patterns.
2024-06
A/B testing framework for production RAG model comparison — built a traffic splitting system that routes 10% of queries to a challenger model while logging results for statistical comparison against the control model.
Challenge: RAG model A/B testing required human evaluation of answer quality — LLM-as-judge metrics were insufficient for aerospace domain accuracy assessment. Built a review queue that samples 5% of traffic for human domain expert review, with results feeding into the statistical comparison.
Learned: A/B testing for aerospace domain RAG systems cannot rely solely on automated metrics — human domain expert evaluation is necessary for ground truth. Build the human review sampling and feedback loop into the A/B testing framework from the start, not as an afterthought.
2023-10
Data quality validation framework using Great Expectations — implemented data quality suites for 6 ML training datasets (sensor data, quality measurements, simulation results). Gates in Kubeflow pipelines block training if data quality checks fail.
Challenge: Great Expectations expects stable data schemas but aerospace simulation datasets often have variable column sets depending on analysis type. Built a schema-adaptive expectation suite that validates the core mandatory columns and separately validates optional columns when present.
Learned: Data quality frameworks for engineering simulation datasets must handle variable schemas gracefully. Great Expectations' default behavior on unexpected columns (warn or fail) is too rigid for simulation datasets where optional output columns vary by analysis configuration — build explicit schema adaptation.
2023-04
Kubeflow pipeline for automated RAG model retraining — built a Kubeflow pipeline that triggers on new document ingestion, re-embeds updated document set, updates Qdrant collection, and runs retrieval quality evaluation suite.
Challenge: Kubeflow pipeline step failures left Qdrant in an inconsistent state — partially updated collections with mixed old and new embeddings. Implemented a blue-green collection strategy: write to a new collection, validate, then atomic pointer swap from the serving collection to the new collection.
Learned: Vector database updates in Kubeflow pipelines must use atomic swap strategies. Partial updates to a live Qdrant collection are a data consistency risk — build blue-green deployment (new collection + atomic pointer swap) for any pipeline that modifies the production embedding store.