Documents
Home>Documents>AI>Agent>Xgen

OCR with OpenAI and Anthropic Vision APIs

5 min readJun 1, 2025Feb 22, 2026

OCR Integration - Using OpenAI/Anthropic Vision APIs

While adding OCR (Optical Character Recognition) to the XGen platform, I opted for LLM Vision APIs instead of a traditional OCR engine. Here's a rundown of that approach.

Why Vision APIs?

Traditional OCR engines like Tesseract work well on structured documents but struggle with complex layouts and handwriting. GPT-4V and Claude Vision understand the context of an image, which makes for significantly more accurate text extraction.

Multi-Provider OCR Service

from abc import ABC, abstractmethod
import base64
from pathlib import Path

class OCRProvider(ABC):
    @abstractmethod
    async def extract_text(self, image_data: bytes, prompt: str = "") -> str:
        pass

class OpenAIOCR(OCRProvider):
    def __init__(self):
        self.client = AsyncOpenAI()

    async def extract_text(self, image_data: bytes, prompt: str = "") -> str:
        b64_image = base64.b64encode(image_data).decode("utf-8")
        default_prompt = "이 이미지에서 모든 텍스트를 추출해주세요. 원본 형식을 최대한 유지해주세요."

        response = await self.client.chat.completions.create(
            model="gpt-4o",
            messages=[{
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt or default_prompt},
                    {"type": "image_url", "image_url": {
                        "url": f"data:image/png;base64,{b64_image}",
                        "detail": "high",
                    }},
                ],
            }],
            max_tokens=4096,
        )
        return response.choices[0].message.content

class AnthropicOCR(OCRProvider):
    def __init__(self):
        self.client = AsyncAnthropic()

    async def extract_text(self, image_data: bytes, prompt: str = "") -> str:
        b64_image = base64.b64encode(image_data).decode("utf-8")
        default_prompt = "이 이미지에서 모든 텍스트를 추출해주세요."

        response = await self.client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=4096,
            messages=[{
                "role": "user",
                "content": [
                    {"type": "image", "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": b64_image,
                    }},
                    {"type": "text", "text": prompt or default_prompt},
                ],
            }],
        )
        return response.content[0].text

OCR Service Integration

class OCRService:
    def __init__(self):
        self.providers = {
            "openai": OpenAIOCR(),
            "anthropic": AnthropicOCR(),
        }

    async def process(
        self,
        image_data: bytes,
        provider: str = "openai",
        prompt: str = "",
    ) -> dict:
        ocr = self.providers.get(provider)
        if not ocr:
            raise ValueError(f"지원하지 않는 OCR 프로바이더: {provider}")

        text = await ocr.extract_text(image_data, prompt)
        return {
            "text": text,
            "provider": provider,
            "char_count": len(text),
        }

FastAPI Endpoint

@router.post("/api/ocr")
async def process_ocr(
    file: UploadFile = File(...),
    provider: str = "openai",
    prompt: str = "",
):
    image_data = await file.read()
    result = await ocr_service.process(image_data, provider, prompt)
    return result

Real-World Use Case

I built a pipeline that extracts text from PDF documents and indexes it into a RAG system. Because Vision APIs understand the content of tables, graphs, and diagrams — not just raw text — the extracted information is far richer than what plain OCR produces. The cost runs roughly $0.01–$0.03 per document, which is actually cheaper than high-accuracy OCR services (typically $0.05+).

Tags
OCRVisionAPIOpenAIAnthropicimage processing