Scraping webhooks

Configure one HTTP or HTTPS endpoint in Scraping jobs. For now, Magpie sends only get_pc_bulk.completed, after the compressed CSV or JSONL result and download URL are ready. Use HTTPS in production; HTTP exposes payloads and signatures in transit.

Magpie sends an HTTP POST with Content-Type: application/json and these headers:

X-Magpie-Webhook-Id: get_pc_bulk.completed:<task_id>
X-Magpie-Webhook-Timestamp: <unix-seconds>
X-Magpie-Webhook-Signature: v1=<hex-hmac-sha256>

{
  "id": "get_pc_bulk.completed:<task_id>",
  "type": "get_pc_bulk.completed",
  "created_at": "2026-07-24T02:00:00Z",
  "data": {
    "event": "get_pc_bulk.completed",
    "completed_at": "2026-07-24T02:00:00Z",
    "task_id": "2f0fd2e4-0a7c-4ec2-8c10-f6d734a73f1d",
    "total": 100,
    "success": 98,
    "failed": 2,
    "download_url": "https://storage.googleapis.com/..."
  }
}

Compute HMAC-SHA256 over timestamp + "." + raw request body. Use the exact bytes received, compare in constant time, and reject stale timestamps.

import hashlib, hmac

signed = timestamp.encode() + b"." + raw_request_body
expected = hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
valid = hmac.compare_digest(signature.removeprefix("v1="), expected)

• Any 2xx response succeeds; timeout is 10 seconds and redirects are not followed.
• Failures retry after 1 minute, 5 minutes, 15 minutes, and 1 hour, then fail after five attempts.
• Delivery is at least once. Deduplicate with the stable id.
• The download URL is time-limited; retrieve the task again if it expires.

The configuration card can send an unsigned sample POST with X-Magpie-Webhook-Test: true. The test passes only for a public HTTP(S) URL that accepts POST and returns 2xx. Test requests are not queued or retried.