Building an LLM-Powered QA Document Validation System

QA Automation System: LLM-Based Quality Document Verification

Overview

We built a system that uses an LLM to automatically verify quality-related documents in an e-commerce QA (Quality Assurance) workflow. Over 28 commits from February 3–18, 2026, we constructed an AWS Lambda–based pipeline that ingests and adjudicates large volumes of quality documents.

Background

Selling products on an e-commerce platform requires a range of quality documents:

• Product specifications, certificates, test reports, contracts (OEM, etc.)
• Each document type has its own expiration rules, required fields, and pass/fail criteria
• Manual review is slow and produces inconsistent results

Core Features

1. Automatic Required-Document Checklist Generation

Analyzes product specifications and automatically generates the list of required documents for each product.

# 2026-02-13: feat: 상품기술서 분석 시 필수 서류 체크리스트 생성
def analyze_product_spec(product_spec: str) -> dict:
    """상품기술서에서 필수 서류 목록 추출"""
    required_docs = match_required_documents(product_spec)
    checklist = generate_checklist(required_docs)
    return checklist

2. Automatic Certificate Expiration Validation

Extracts expiration dates from certificates and determines whether they are still valid as of the current date.

# 2026-02-07: feat: 인증서 유효기간 검증 기능 추가
def validate_certificate_expiry(cert_data: dict) -> dict:
    """인증서 유효기간 검증"""
    expiry_date = extract_expiry_date(cert_data)
    is_valid = expiry_date > datetime.now()
    return {
        "certificate": cert_data["name"],
        "expiry_date": expiry_date,
        "is_valid": is_valid,
        "days_remaining": (expiry_date - datetime.now()).days
    }

3. High-Volume Batch Processing

To handle token-limit issues when processing 40+ files, we introduced dynamic batch construction.

# 2026-02-15: 고정 슬라이싱 → 크기 기반 동적 배치로 수정
def create_dynamic_batches(files: list, max_tokens: int = 120000):
    """파일 크기 기반 동적 배치 구성"""
    batches = []
    current_batch = []
    current_size = 0
    
    for file in files:
        file_tokens = estimate_tokens(file)
        if current_size + file_tokens > max_tokens:
            batches.append(current_batch)
            current_batch = [file]
            current_size = file_tokens
        else:
            current_batch.append(file)
            current_size += file_tokens
    
    if current_batch:
        batches.append(current_batch)
    return batches

4. Hardening LLM Response Reliability

We addressed several failure modes in LLM responses, including truncated JSON and corrupted fileNo/fileName values.

# 2026-02-07: feat: LLM 응답의 fileNo/fileName 강제 보정
def enforce_file_metadata(llm_response: dict, original_files: list) -> dict:
    """LLM이 임의로 변경한 파일 메타데이터를 원본으로 복원"""
    for item in llm_response["results"]:
        original = find_original_file(item["fileNo"], original_files)
        if original:
            item["fileNo"] = original["fileNo"]
            item["fileName"] = original["fileName"]
    return llm_response

# 2026-02-08: feat: LLM 응답 완결성 검증 및 재시도 로직
def verify_and_retry(response: str, max_retries: int = 3):
    """JSON 완결성 검증 후 실패 시 재시도"""
    for attempt in range(max_retries):
        try:
            parsed = json.loads(response)
            validate_schema(parsed)
            return parsed
        except (json.JSONDecodeError, ValidationError):
            response = retry_llm_call()
    raise ProcessingError("Max retries exceeded")

Bug Fix Log

The latter half of the project was focused on stabilization:

Prompt Architecture

FIRST_PROMPT
  └── Instructs the model to quote the original source text for each judgment criterion
  
BATCH_PROMPT
  └── Per-batch file analysis + rules for preserving file metadata

FINAL_PROMPT
  └── Final adjudication + missing document review + summary generation

Tech Stack

• AWS Lambda: Serverless execution environment
• OpenAI GPT-4: LLM for document analysis
• Python: Core business logic
• Pydantic: Output schema validation

Retrospective

This 28-commit project was a firsthand lesson in the practical challenges of deploying LLMs in production. LLM responses are never perfect, and we came to appreciate that defensive programming—forced correction, retry logic, schema validation—is not optional; it's essential.