Rafal Szymanski

2025-01

Set up distributed tracing across all AI microservices using Jaeger and OpenTelemetry — instrumenting FastAPI backends, Celery workers, and Kafka consumers with trace context propagation.

Challenge: Propagating trace context through asynchronous Kafka messages required custom instrumentation — Kafka consumers don't automatically carry trace headers like HTTP services do. Implemented a header injection/extraction pattern in all producer and consumer code.

Learned: Distributed tracing in event-driven architectures requires explicit trace context propagation at every async boundary. There is no automatic magic — every queue, topic, and task queue must be instrumented manually.

2024-09

Optimized Kubernetes resource allocation for LLM inference pods using Vertical Pod Autoscaler (VPA) for right-sizing CPU and memory based on actual usage patterns.

Challenge: VPA and HPA conflict when both are enabled for the same deployment — VPA modifies resource requests while HPA scales replicas based on resource utilization, leading to oscillating behavior. Disabled HPA for VPA-managed deployments and used KEDA for event-driven scaling instead.

Learned: VPA and HPA are fundamentally incompatible for the same resource metric. For LLM serving where request queue length is a better scaling signal than CPU, KEDA with queue-based scaling is more appropriate than HPA.

2024-05

Implemented GitOps workflow with ArgoCD — all Kubernetes manifests in a dedicated GitOps repository, ArgoCD auto-syncs cluster state on merge to main.

Challenge: ArgoCD has no native secret management — secrets cannot be committed to Git. Implemented External Secrets Operator that pulls secrets from Azure Key Vault at deployment time and creates Kubernetes Secrets.

Learned: Secret management in GitOps is a first-class problem that must be solved before adopting GitOps, not after. External Secrets Operator with a secrets manager is the industry-standard solution.

2023-12

Implemented infrastructure as code for all AI infrastructure using Terraform — AKS cluster, Azure Container Registry, Key Vault for secrets, storage accounts, and networking.

Challenge: Terraform state locking conflicts when multiple team members ran terraform apply simultaneously — caused state corruption twice. Migrated to remote state in Azure Blob with state locking enabled.

Learned: Terraform remote state with locking is mandatory for any team with more than one person touching infrastructure. Local state is only appropriate for solo projects or experimentation.

2023-09

Deployed Qdrant in production as a 3-node distributed cluster with replication factor 2. Configured persistent volumes, health checks, and backup schedule.

Challenge: Initial Qdrant configuration used default ports (6333 for REST, 6334 for gRPC) which conflicted with existing services on the cluster. Reconfigured all services and updated 12 client configurations. Additionally, the IT security firewall exception request for cluster-internal traffic took 2 days longer than expected.

Learned: Always audit port conflicts before deploying a new stateful service to an existing cluster. Document all required firewall rules and submit them to IT security at least 2 weeks before planned deployment.

2023-05

Built CI/CD pipeline for ML model deployment — GitHub Actions workflow: run tests, build Docker image, push to Azure Container Registry, deploy to staging AKS namespace, run smoke tests, promote to production.

Challenge: Docker image sizes for PyTorch-based models were 8-12GB — build times of 20-30 minutes. Multi-stage builds separating dependencies from application code, combined with ACR layer caching, reduced build time to 6-8 minutes.

Learned: Large Python ML Docker images benefit enormously from layer caching — the dependencies layer (which rarely changes) should be built first and cached separately from application code.

2023-01

Set up Azure AKS cluster for AI workloads — GPU node pool (2x NVIDIA A100 80GB), CPU node pools with autoscaling, namespace isolation per team, and RBAC configuration.

Challenge: GPU driver version conflicts between PyTorch 2.0's CUDA requirements and AKS default driver installation. Pinning the NVIDIA driver version in the DaemonSet and disabling automatic driver updates resolved the incompatibility.

Learned: GPU driver management in Kubernetes requires explicit version pinning and a policy to prevent automatic updates. AKS node upgrades can silently change driver versions and break GPU workloads.