Katarzyna Zielinska

2025-01

Built an automated test case generator for quality engineers: given a design specification, generate candidate failure scenarios for FMEA sessions as a starting point for review.

Challenge: The model generates physically plausible-sounding but sometimes impossible failure modes for aerospace components — e.g., suggesting a ceramic coating could 'melt' under conditions well below its melting point. Mandatory domain expert review step is non-negotiable.

Learned: AI-assisted FMEA generation is valuable as a brainstorming accelerator, not as a replacement for engineering judgment. Framing it as 'find the ones we haven't thought of' rather than 'generate the complete list' sets correct expectations.

2024-09

Worked on structured entity extraction from engineering drawings and inspection reports: part numbers, material specifications, geometric tolerances, and failure mode descriptions.

Challenge: Part number formats vary wildly across suppliers (alphanumeric, with/without dashes, with revision suffix). Providing regex pattern examples in the prompt for each supplier format dramatically improved extraction consistency.

Learned: Structured extraction with known format constraints should use those constraints explicitly in the prompt. The model is better at pattern matching than free-form extraction.

2024-04

Designed LLM evaluation suite: LLM-as-judge for fluency and relevance, embedding similarity to reference answers for factual consistency, and task-specific rubrics for structured outputs.

Challenge: LLM-as-judge systematically prefers longer, more verbose answers regardless of accuracy. Calibrated the judge with human-labeled examples and added an explicit instruction to penalize unnecessary verbosity.

Learned: LLM evaluation metrics need their own evaluation. Testing whether your automated judge agrees with human judgments is a necessary step before trusting judge scores.

2023-12

Implemented a versioned prompt library — all production prompt templates stored in Git with semantic versioning, changelog, and associated test cases. Built a regression test runner that evaluates all prompts on each merge.

Challenge: Prompt regressions are insidious — a change that improves summary quality can silently degrade extraction accuracy. Without automated evaluation, regressions went undetected for weeks.

Learned: Prompts are code. They need version control, tests, and CI just like application code. The tooling is immature but the discipline is the same.

2023-08

Developed prompt templates for engineering document summarization: Engineering Change Notices (ECNs), Technical Reports, and Supplier Non-Conformance Reports.

Challenge: GPT-3.5 would hallucinate specific part numbers, revision letters, and dates even when instructed to use only information from the document. Explicit instruction to 'only quote directly from the source for all specific identifiers' reduced hallucination rate from 22% to 4%.

Learned: Factual grounding in LLM summaries requires separate handling for different types of information. Narrative summaries can be paraphrased; specific identifiers must be quoted verbatim or omitted.

2023-04

Built a text classification pipeline to categorize engineering incident reports by type: mechanical failure, electrical fault, software bug, process deviation, and supplier quality issue.

Challenge: Severely imbalanced dataset — 'supplier quality' category had only 47 samples vs 1200 for 'mechanical failure'. Used class-weighted loss, SMOTE for oversampling, and data augmentation via back-translation.

Learned: For rare classes in engineering classification, human-in-the-loop active learning (model flags uncertain cases for expert review) is more effective than automated augmentation alone.