SSE Disconnection Handling - Bug Fix for Stable Streaming
After deploying SSE streaming to production, we ran into disconnection issues across various environments. Here's a record of how we diagnosed and fixed the problem.
Symptoms
- SSE stream drops unexpectedly during workflow execution
- Happens frequently with large LLM responses (1,000+ tokens)
- Only occurs behind an Nginx reverse proxy (direct connections work fine)
- Frequent
onerrorevents firing on the client
Root Cause Analysis
Cause 1: Nginx Proxy Buffering
Nginx was buffering responses by default. Since SSE is real-time streaming, buffering must be disabled.
# Before - default config
location /api/ {
proxy_pass http://backend:8000;
}
# After
location /api/ {
proxy_pass http://backend:8000;
proxy_buffering off;
proxy_cache off;
proxy_set_header Connection '';
proxy_http_version 1.1;
chunked_transfer_encoding off;
proxy_read_timeout 86400s; # 24시간 (긴 워크플로우 대응)
}
Cause 2: Proxy Timeout
Nginx's default proxy_read_timeout is 60 seconds. Any node execution running longer than 60 seconds would cause the connection to drop.
Cause 3: No Client-Side Reconnection Logic
The client had no error handling or reconnection logic for the EventSource.
Fixes
Server-Side: Heartbeat Implementation
Send a heartbeat message at regular intervals to keep the connection alive.
async def stream_with_heartbeat(
workflow_id: str,
) -> AsyncGenerator[str, None]:
heartbeat_interval = 15 # 15초마다 heartbeat
async def heartbeat():
while True:
await asyncio.sleep(heartbeat_interval)
yield f": heartbeat\n\n" # SSE 코멘트 (클라이언트에서 무시됨)
async def workflow_stream():
async for event in stream_workflow_execution(workflow_id):
yield event
# heartbeat와 실제 이벤트를 병합
async for event in merge_async_generators(heartbeat(), workflow_stream()):
yield event
Server-Side: Disconnection Detection
from starlette.requests import Request
@router.post("/api/workflow/{workflow_id}/execute/stream")
async def execute_stream(workflow_id: str, request: Request):
async def generate():
try:
async for event in stream_with_heartbeat(workflow_id):
if await request.is_disconnected():
# 클라이언트 연결 끊김 감지
break
yield event
except asyncio.CancelledError:
pass
finally:
# 리소스 정리
await cleanup_execution(workflow_id)
return StreamingResponse(generate(), media_type="text/event-stream")
Client-Side: Reconnection Logic
class SSEClient {
constructor(url, maxRetries = 3) {
this.url = url;
this.maxRetries = maxRetries;
this.retryCount = 0;
this.lastEventId = null;
}
connect() {
const headers = {};
if (this.lastEventId) {
headers['Last-Event-ID'] = this.lastEventId;
}
this.eventSource = new EventSource(this.url);
this.eventSource.onerror = () => {
this.eventSource.close();
if (this.retryCount < this.maxRetries) {
this.retryCount++;
const delay = Math.pow(2, this.retryCount) * 1000;
setTimeout(() => this.connect(), delay);
}
};
}
}
Result
These fixes resolved the SSE disconnection issues. The heartbeat naturally eliminated proxy timeout problems as well, and the client-side reconnection logic adds resilience against transient network failures. When using SSE in production, always account for all three of these: proxy configuration, heartbeats, and reconnection.