Long-Running Operations (Async APIs)
A pattern for handling API requests that take too long for synchronous response by returning a job reference and providing status polling or callback notification.
Description
Long-running operations handle API requests that can't complete within a reasonable HTTP timeout window (typically 30 seconds). Instead of blocking the connection, the server immediately returns 202 Accepted with a job resource containing an ID, status, and a URL to check progress. The client can then poll the status endpoint or register a webhook callback to be notified when the operation completes. Common use cases include report generation, data exports, video processing, bulk imports, and complex calculations.
The job resource should track granular status: queued, processing, completed, failed, and optionally a progress percentage or step description for long multi-phase operations. The polling endpoint (GET /api/v1/jobs/{jobId}) returns the current status and, when completed, includes the result or a link to download the result. To reduce polling overhead, the server can use the Retry-After header to suggest a polling interval, and clients should implement exponential backoff.
For a better developer experience, consider supporting multiple notification mechanisms: polling (simplest), webhooks (push notification to a registered URL), and Server-Sent Events (real-time streaming of status updates). The job should have a TTL after which its results are cleaned up, and the API should document how long results are retained. Failed jobs should include actionable error details, and idempotency keys should be supported so clients can safely retry the initial request without creating duplicate jobs.
Prompt Snippet
Return 202 Accepted with body { job_id: uuid, status: 'queued', poll_url: '/api/v1/jobs/{id}', estimated_duration_seconds: 120 } for operations exceeding 5 seconds. Implement GET /api/v1/jobs/{id} returning { status: 'processing' | 'completed' | 'failed', progress_pct: number, result_url?: string, error?: ErrorDetail, created_at: ISO8601 } with Retry-After header suggesting poll interval. Process jobs via BullMQ workers with configurable concurrency. Support webhook callbacks as an alternative to polling by accepting a callback_url in the initial request. Retain job results for 7 days before cleanup.Tags
Related Terms
Bulk Operations API Design
API endpoints that accept arrays of operations in a single request to reduce round trips and improve throughput for batch processing.
Webhook Design
A push-based integration pattern where the server sends HTTP POST requests to consumer-registered URLs when events occur.
Server-Sent Events (SSE)
A unidirectional HTTP streaming protocol where the server pushes real-time text-based events to the client over a persistent connection.
Idempotency Keys
Client-generated unique keys sent with mutating requests to ensure that retrying the same operation produces the same result without side effects.