Marek Wisniewski

2025-03

Defined the AI governance framework for the organization: model versioning policy, audit trail requirements, responsible AI checklist, incident escalation procedure for AI failures in production.

Challenge: Defining what constitutes a 'significant' model change requiring re-validation. Eventually settled on a risk-tiered classification: cosmetic prompt changes vs. model swaps vs. architecture changes.

Learned: AI governance frameworks need to be lightweight enough that teams actually follow them. A 40-page policy document will be ignored. One-page checklist with 5 mandatory sign-offs works much better.

2024-11

Ran structured benchmark comparing Claude 3 Opus vs GPT-4o for aerospace domain tasks: technical Q&A, document summarization, FMEA generation, structured data extraction from inspection reports.

Challenge: Defining fair evaluation criteria was harder than running the benchmark. LLM-as-judge metrics are model-biased. Used human annotations from domain experts for ground truth.

Learned: Claude 3 Opus showed better calibration for niche technical domains — fewer confident wrong answers. GPT-4o was faster and slightly better at structured JSON output.

2024-07

Led design of the multi-tenant RAG platform architecture. Defined data isolation model (per-tenant Qdrant collections), embedding namespace strategy, RBAC model, and audit logging requirements.

Challenge: Deciding between per-tenant collections vs. per-tenant namespaces within a shared collection. Collections won on isolation and filtering performance, at the cost of higher operational overhead.

Learned: Multi-tenancy architecture decisions made early are very hard to reverse once customers have data. Worth spending extra time on the design before any data migration.

2024-02

Architected an agentic system for automatic technical report generation. Integrated tool-use: code interpreter for data analysis, document retrieval for precedent lookup, structured output for report sections.

Challenge: End-to-end latency was 25-40 seconds — unacceptable for interactive use. Parallelized independent agent steps (retrieval and analysis run concurrently) and reduced to 12-18 seconds.

Learned: Async orchestration in LangGraph requires careful state management. Concurrent steps can produce race conditions if state is mutated in place.

2023-10

Designed a multi-agent architecture for automated defect analysis. Agents: retriever, analyzer, validator, report writer. Orchestrated with LangGraph.

Challenge: Hallucination rate in early iterations was too high — agents would fabricate part numbers and specifications. Explicit chain-of-thought prompting combined with output validation reduced error rate from 18% to 3%.

Learned: LLM agents need strict output schemas (Pydantic models) and a validation agent downstream. Free-form agent output is not production-ready.

2023-06

Led evaluation of vector databases for enterprise deployment — compared ChromaDB, Pinecone, Weaviate, and Qdrant on performance, operational overhead, and data residency requirements.

Challenge: GDPR and internal data governance policy ruled out most managed cloud vector DBs. Self-hosted Qdrant emerged as the only viable option that satisfied both security and performance requirements.

Learned: Qdrant's payload filtering and named vectors make it significantly more flexible than ChromaDB for multi-collection and multi-modal scenarios.

2023-02

Designed the architecture for an internal knowledge assistant using LangChain and GPT-3.5-turbo. Defined chunking strategy, retrieval logic, and prompt templates for engineering document Q&A.

Challenge: Convincing stakeholders that LLM inference costs are manageable at scale. Built a cost projection model showing that RAG reduces token usage by ~60% vs. sending full documents.

Learned: Chunking strategy has a larger impact on answer quality than model choice. Overlapping chunks with ~10% overlap significantly reduce context boundary artifacts.