Lukasz Wojcik

2025-03

Added OpenTelemetry distributed tracing to the AI API — tracking latency per pipeline stage: embedding, retrieval, LLM call, post-processing, response serialization.

Challenge: Propagating trace context through async Python with LangChain callbacks was not straightforward — LangChain's callback system uses a different context propagation mechanism than OpenTelemetry. Required a custom bridge.

Learned: Tracing async Python applications requires understanding how context variables propagate across coroutines. asyncio.Task copies context at creation time — fire-and-forget tasks break context propagation unless you explicitly copy the context.

2024-11

Implemented webhook-based document ingestion trigger: when a document is uploaded to SharePoint, a webhook fires, the backend queues a re-embedding job, and the vector store is updated automatically.

Challenge: SharePoint webhooks expire after 6 months and require a renewal call before expiry. Built an auto-renewal cron job that runs monthly and refreshes all active webhook subscriptions. Initial version missed the renewal window once — caused 3 days of silent ingestion failure.

Learned: External webhook subscriptions need monitoring. Added an alert that fires if no webhook events are received within a configurable window — catches silent failures from expired subscriptions.

2024-07

Built multi-tenant API layer — per-tenant configuration (LLM model selection, system prompt, Qdrant collection, token budget), request isolation, and audit logging.

Challenge: Qdrant's Python client is not thread-safe when sharing a single client across tenants in async context. Implemented a client-per-tenant pool with a maximum size limit to avoid connection exhaustion.

Learned: Connection pool sizing for async services requires profiling under realistic load. Default pool sizes are almost always too small for production AI workloads with multiple concurrent LLM calls.

2024-03

Integrated LangChain into the FastAPI async backend. Ran LangChain chains in async context to avoid blocking the event loop.

Challenge: LangChain's async support was incomplete in versions <0.1 — several retrievers and tools had only synchronous implementations. Wrapped synchronous calls with asyncio.run_in_executor to avoid blocking. This added complexity and a thread pool overhead.

Learned: When adopting an async framework for I/O-bound AI pipelines, verify that all dependencies genuinely support async. Sync wrappers work but introduce hidden thread pool bottlenecks under load.

2023-11

Built streaming response endpoint using FastAPI SSE (Server-Sent Events) — streaming OpenAI completion tokens to the browser in real time.

Challenge: nginx reverse proxy was buffering the SSE stream, causing the client to receive all tokens at once at the end instead of progressively. Spent half a day debugging before finding that proxy_buffering off was needed in nginx config.

Learned: SSE through reverse proxies requires explicit buffering disable. nginx, Caddy, and Azure API Management all buffer by default. Always test streaming end-to-end through the full proxy chain, not just against the FastAPI server directly.

2023-07

Integrated OpenAI API with production-grade reliability: exponential backoff retry for rate limit errors, automatic fallback from GPT-4 to GPT-3.5 when hitting per-minute limits, and circuit breaker for outage detection.

Challenge: OpenAI rate limits are enforced both per-minute and per-day. Per-minute limits caused burst failures; per-day limits caused slow degradation. Implemented a token bucket with Redis for smooth per-minute shaping.

Learned: Rate limit handling for LLM APIs is more complex than for typical REST APIs because requests have variable token costs. A request that sends 8K tokens consumes 8x more rate limit budget than a 1K token request.

2023-03

Built the FastAPI backend for the internal AI tools portal — authentication middleware, request routing to multiple AI services, per-user rate limiting, and request logging.

Challenge: JWT token refresh logic with concurrent requests caused race conditions — multiple requests would simultaneously detect an expired token and all attempt refresh, causing 429 errors from the auth server. Implemented a refresh lock with Redis.

Learned: Concurrent token refresh is a classic thundering herd problem. A distributed lock with a short TTL and a 'wait for refresh' path for competing requests is the correct solution.