Tadeusz Tomaszewski
MLOps Engineer · MLOps & Digital Engineering
Warsaw, Poland
MLOps Engineer responsible for the infrastructure and processes that move ML models from research to production. Builds model registries, CI/CD pipelines for model validation and deployment, and monitoring systems for production model health. Works closely with AI Center of Excellence and Data & AI Platform teams to standardize the model lifecycle from experiment tracking through production monitoring.
Expertise
- ML model versioning and registry (MLflow)
- CI/CD pipelines for ML models
- Kubernetes-based ML deployment
- model monitoring and drift detection
- feature store design
Technologies
Work History
2025-01
Production model monitoring — deployed Prometheus + Grafana dashboards tracking inference latency, request rate, error rate, and prediction distribution drift for 8 production ML models. Set up automated alerting for anomalous drift.
Challenge: Defining appropriate drift thresholds for each model was non-trivial — threshold too tight generated alert noise, too loose allowed real model degradation to go undetected. Implemented a rolling baseline approach: drift is measured against the previous 30 days of prediction distribution rather than a static training distribution.
Learned: Static drift thresholds calibrated on training data become stale as the input distribution slowly shifts over time. Rolling baseline drift detection (comparing current distribution to recent production distribution rather than training distribution) is more robust to gradual concept drift without generating false alerts.
2024-05
Feature store design and implementation — built a lightweight feature store using PostgreSQL + Redis for storing pre-computed engineering features (document embeddings, quality metrics, sensor statistics) shared across multiple ML models.
Challenge: Feature versioning was the hardest design problem — features computed from different versions of source data needed to be versioned independently and linked to model versions. Built an explicit feature version manifest that tracks source data version, computation code version, and feature schema version.
Learned: Feature stores require a three-way versioning system: source data version, computation logic version, and feature schema version. A feature 'timestamp' alone is insufficient — two features computed at the same time from different source data versions are not equivalent.
2023-10
CI/CD pipeline for ML model deployment — built GitHub Actions workflow that runs automated model validation tests, accuracy regression checks, and performance benchmarks before promoting models to production on Kubernetes.
Challenge: Model validation tests for the FMEA surrogate model required a GPU-enabled test runner — GitHub Actions default runners are CPU-only. Configured self-hosted GitHub Actions runners on GPU-equipped Kubernetes nodes, with resource isolation to prevent test pipeline contention with production inference.
Learned: ML CI/CD pipelines with GPU validation tests require self-hosted runners — cloud CI services lack GPU availability at the required throughput. Design the runner infrastructure early: GPU runners need resource quotas to prevent test jobs from competing with production inference loads.
2023-04
MLflow model registry setup — deployed MLflow tracking server on Kubernetes with Azure Blob Storage backend. Established model versioning conventions, staging/production promotion workflow, and experiment tracking for all AI CoE models.
Challenge: MLflow server with concurrent writes from 12 AI team members on a single-pod deployment experienced database lock contention — some experiment runs were lost due to SQLite backend limitations. Migrated to PostgreSQL backend and horizontal pod scaling, resolving all concurrency issues.
Learned: MLflow with SQLite backend is not suitable for multi-user production environments. PostgreSQL backend is required from the start for any deployment with more than 3-4 concurrent users — SQLite locking limits become visible under normal team usage.