Contextifier PDF Handler: Table Detection and Text Quality Analysis

The Challenge of PDF Processing

PDF is essentially a "format for drawing characters on screen." Because it contains no structured text information, extraction is notoriously difficult. Contextifier's PDF handler addresses this by breaking the problem into multiple modules.

PDF Handler Pipeline

flowchart LR
    A[PDF 파일] --> B[텍스트 추출]
    B --> C{품질 분석}
    C -->|고품질| D[직접 사용]
    C -->|저품질| E[Vector OCR]
    D --> F[레이아웃 감지]
    E --> F
    F --> G[테이블 검출]
    G --> H[테이블 검증]
    H --> I[청킹]
    I --> J[ChunkResult]

Text Quality Analysis

This module determines whether the text extracted from a PDF is actually usable. Scanned or image-based PDFs can produce meaningless character sequences when text extraction is attempted.

# 텍스트 품질 분석 로직 (간략화)
def analyze_text_quality(text: str) -> float:
    # 한글/영문 비율 체크
    # 유니코드 카테고리별 분석
    # 연속 공백/특수문자 비율
    return quality_score

Layout Block Detection

This module separates text blocks, table blocks, and image blocks within a PDF page. This prevents text inside tables from bleeding into the body text.

Table Validator

Validates whether an extracted table is actually a well-formed table:

• Minimum row/column count check
• Empty cell ratio check
• Header pattern analysis

Troubleshooting: Vector OCR Issues

Early on, OCR results were unstable when processing scanned PDFs. Garbled characters were especially common with Korean-language documents.

Solution: Multi-OCR engine support. In addition to the default Tesseract backend, we added an OCR option that leverages LLM Vision models (GPT-4o, Claude, etc.).

# OCR 엔진 선택
processor = DocumentProcessor(
    ocr_engine="llm_vision",  # or "tesseract"
    llm_model="gpt-4o"
)

Performance Comparison

The PDF handler alone required roughly 15 commits (v0.1.0 – v0.1.6).