Building BatchTester: Automated Testing for AI Workflows

How Do You Test a Workflow?

AI workflows are non-deterministic, which makes them hard to test — the same input can produce different outputs. To address this, we built BatchTester: a tool that automatically runs large numbers of test cases and analyzes the results.

BatchTester Architecture

interface BatchTestConfig {
  workflowId: string;
  testCases: TestCase[];
  concurrency: number;          // 동시 실행 수
  evaluationCriteria: EvalCriteria[];
  notifyOnComplete: boolean;
}

interface TestCase {
  id: string;
  input: Record<string, string>;
  expectedOutput?: string;       // 정답이 있는 경우
  metadata?: Record<string, unknown>;
}

interface EvalCriteria {
  type: 'contains' | 'similarity' | 'regex' | 'llm_judge';
  config: Record<string, unknown>;
}

Test Case Editor UI

Test cases can be edited in a spreadsheet-style table. CSV upload is also supported.

const TestCaseEditor: React.FC = () => {
  const [testCases, setTestCases] = useState<TestCase[]>([]);
  
  return (
    <div>
      <div className="flex justify-between mb-4">
        <h2 className="text-lg font-semibold">테스트 케이스</h2>
        <div className="flex gap-2">
          <Button variant="outline" onClick={addRow}>행 추가</Button>
          <Button variant="outline" onClick={importCSV}>CSV 가져오기</Button>
        </div>
      </div>
      <EditableTable
        columns={[
          { key: 'input', header: '입력', type: 'textarea' },
          { key: 'expectedOutput', header: '기대 출력', type: 'textarea' },
          { key: 'tags', header: '태그', type: 'tags' },
        ]}
        data={testCases}
        onChange={setTestCases}
      />
    </div>
  );
};

Batch Run Progress

Once a test run starts, results stream in alongside a progress bar in real time.

interface BatchTestProgress {
  total: number;
  completed: number;
  passed: number;
  failed: number;
  errors: number;
  currentTestId: string | null;
  estimatedTimeRemaining: number;
}

Results Analysis View

After a run completes, the results can be examined from several angles.

• Pass/fail ratio: visualized as a pie chart
• Response time distribution: displayed as a histogram
• Failed case details: input, expected output, and actual output shown side by side
• LLM Judge scoring: automated evaluation results using GPT-4

LLM Judge Evaluator

For tests without a clear ground truth, we use an LLM Judge. GPT-4 scores output quality on a 1–5 scale and provides a rationale alongside each score. This approach turned out to be surprisingly effective.

Using BatchTester for Regression Testing

BatchTester also serves as a regression testing harness — after modifying a workflow, you re-run the existing test suite to confirm that performance hasn't degraded. It has significantly increased confidence in workflow changes.