Tomasz Krawczyk

2025-01

Built LLM cost monitoring and budget control system: per-team token usage dashboards, monthly budget alerts, automatic request throttling when teams exceed 90% of budget.

Challenge: Mapping OpenAI API requests to internal cost centers requires tagging each request with a team identifier. Retrofitting this into existing integrations across 8 teams took longer than expected.

Learned: Cost attribution must be built in from day one. Adding it later requires touching every API integration point, which is painful in a large codebase.

2024-08

Led migration from manual Kubernetes deployments to GitOps with ArgoCD. All Kubernetes manifests moved to Git, ArgoCD syncs cluster state automatically on merge.

Challenge: Large model artifacts (>5GB) cannot be stored in Git. Implemented DVC for artifact versioning with Azure Blob as remote storage. ArgoCD manifests reference DVC-managed artifact versions.

Learned: GitOps for ML requires a hybrid approach: code and config in Git, artifacts in dedicated storage, with the manifest serving as the join key between them.

2024-04

Designed A/B testing framework for prompt changes — shadow deployment where both old and new prompts are executed, results compared offline before full rollout.

Challenge: Defining meaningful metrics without ground truth labels. Used a combination of: user thumbs-up/down feedback, LLM-as-judge evaluation, and embedding similarity to expected answers for automated scoring.

Learned: No single metric captures prompt quality. A dashboard combining 3-4 proxy metrics gives a more reliable signal than optimizing any single one.

2023-12

Implemented LLM observability stack: LangSmith for request/response tracing and chain visualization, Prometheus for latency and token usage metrics, Grafana dashboards for cost monitoring per team.

Challenge: LangSmith costs scaled with trace volume — tracing every request was adding 15% to monthly LLM costs. Implemented adaptive sampling: 100% sampling for errors and slow requests, 5% for normal requests.

Learned: Observability budget is a real concern for LLM systems. Stratified sampling by request type preserves visibility into failures while controlling cost.

2023-08

Designed and deployed model serving infrastructure on Kubernetes. Set up GPU node pool on Azure AKS with horizontal pod autoscaling for inference endpoints.

Challenge: GPU pod cold start time was 45-60 seconds when scaling from zero replicas. Implemented minimum replica count of 1 for critical models and scale-to-zero only for development endpoints.

Learned: GPU autoscaling has a fundamentally different cost/latency tradeoff than CPU autoscaling. Scale-to-zero is only practical for batch workloads, not for interactive inference.

2023-03

Set up MLflow for ML experiment tracking across all data science projects. Migrated from ad-hoc Excel logging and scattered Jupyter notebooks to a centralized experiment registry.

Challenge: Data scientists were resistant to adding logging boilerplate to their notebooks. Key insight: wrapping MLflow in a project-specific decorator with sensible defaults reduced the code change to one line per experiment.

Learned: Adoption of tracking tools requires making the default path the easy path. If logging is optional, it will not happen consistently.