Bartosz Grabowski
Data Engineer · Data & AI Platform
Warsaw, Poland
Data Engineer responsible for the data infrastructure that feeds both ML models and business analytics. Builds and maintains pipelines from machine sensors, quality systems, and enterprise applications into analytical stores. Increasingly focused on data lineage and observability as the number of ML-consuming pipelines grows.
Expertise
- ELT pipeline design
- Apache Airflow orchestration
- dbt data modeling
- real-time data streaming
- data lineage and observability
Technologies
Work History
2025-01
Migrated Apache Airflow from version 2.5 to 2.9 — updated DAG syntax, replaced deprecated operators, and refactored 18 DAGs to use the TaskFlow API.
Challenge: Several DAGs used operators deprecated in 2.6 that were removed in 2.9. The deprecation warnings were present but ignored in the CI pipeline — added a CI check for deprecated operator usage as a post-migration improvement.
Learned: Airflow major version upgrades require reading the full migration guide, not just the changelog. Several breaking changes were not in the 'breaking changes' section but buried in component-specific notes.
2024-10
Optimized ClickHouse queries powering the SPC real-time dashboard. Query time reduced from 12 seconds to 0.3 seconds through materialized views, partition pruning, and index tuning.
Challenge: ClickHouse materialized views do not support JOINs — the view must be based on a single table or subquery. Had to pre-join dimension tables at ingestion time and store denormalized data, increasing storage by ~40% but enabling sub-second queries.
Learned: ClickHouse is optimized for append-only analytical queries on denormalized data. Trying to normalize data in ClickHouse the way you would in PostgreSQL leads to performance problems. Embrace denormalization.
2024-06
Implemented data lineage tracking using OpenLineage and Marquez. All Airflow DAGs and dbt models now emit lineage events, showing which source tables feed which ML model features.
Challenge: Getting the ML team to instrument their training scripts with OpenLineage emitters. The integration requires explicit annotation of input datasets — it is not automatic. Eventually created a wrapper utility that makes it a one-liner.
Learned: Data lineage only delivers value if it is complete. One uninstrumented pipeline creates a broken lineage graph. Providing the simplest possible API for emitting lineage events is essential for adoption.
2024-02
Implemented dbt models for quality data mart — SPC measurements, inspection results, PPAP status, and corrective action tracking from 3 different source systems.
Challenge: Quality data across 3 systems used different part number formats (with/without dashes, with/without revision suffix). Built a canonical part number mapping table based on fuzzy matching and domain rules, reviewed and approved by quality engineers.
Learned: Part number harmonization is a surprisingly complex domain problem in aerospace manufacturing. A single physical part can appear under different identifiers across design, manufacturing, and quality systems. The mapping table is now a first-class data asset.
2023-10
Built data pipeline for AI document ingestion — extracts documents from SharePoint (Word, PDF, PowerPoint), transforms to clean text, queues to Redis for downstream embedding service.
Challenge: SharePoint API rate limits (1200 requests per minute) caused throttling during initial bulk ingestion of 3000 documents. Implemented exponential backoff with jitter and a per-minute request counter.
Learned: SharePoint API throttling is aggressive and poorly documented. The error response body contains a Retry-After header that should be respected — ignoring it and using fixed delays often makes throttling worse.
2023-06
Set up Apache Airflow for orchestrating ML training and data processing pipelines. Migrated 14 cron jobs to Airflow DAGs with proper dependency management, retries, and alerting.
Challenge: Airflow scheduler process was memory-leaking — after 7-10 days of continuous operation it would OOM-kill. Root cause: LocalExecutor with long-running DAGs leaking memory per subprocess. Fixed by switching to CeleryExecutor with Redis.
Learned: Airflow's executor choice significantly impacts memory behavior. LocalExecutor is convenient for development but not suitable for production workloads with long-running or memory-intensive tasks.
2023-02
Built an ELT pipeline for machine sensor data — Kafka ingestion from 40 CNC machines, ClickHouse for analytical storage, dbt transformations for quality metrics mart.
Challenge: Schema evolution in Kafka caused consumers to break silently when upstream producers added new fields. Implemented Confluent Schema Registry with backward-compatible schema evolution enforcement.
Learned: Schema Registry is not optional for production Kafka pipelines. Unmanaged schema evolution will eventually break consumers, and the failure will be silent until someone notices missing data.