Pipeline recipes
Production-shaped pipelines you can copy — every one runnable from the public repo.
Market share → daily price monitoring
Rank a category's SKUs by GMV with the Data API, then scrape exactly those product pages every day. The cheapest way to run daily price monitoring in Southeast Asia.
Read the recipe →Monthly category export to S3
Land a whole category's SKU rows in your S3 bucket every month as Parquet — estimate first (free), then submit, poll and download, on a GitHub Actions schedule.
Read the recipe →Daily price & stock monitoring
Bring a CSV of products you already care about, scrape them daily on Tokopedia, TikTok Shop, Blibli or Amazon, and get a price-change report against yesterday's snapshot.
Read the recipe →Airflow DAG for monthly exports
The monthly category export as a production Airflow DAG — idempotent, resumable via the export id, with the estimate step as a cost gate.
Read the recipe →Load into BigQuery / Snowflake
Export → object store → warehouse. Magpie exports deliver Parquet, which loads natively into BigQuery, Snowflake, Databricks and DuckDB with no schema wrangling.
Read the recipe →Share of search
Who owns the first page for your keywords? Scrape marketplace search results on a schedule and compute appearances, share of top-10 and best rank per brand.
Read the recipe →Amazon cross-market ranking
The same keyword ranked across ten Amazon storefronts (us, uk, de, fr, it, es, ca, au, jp, sg): top products per country with price and ASIN, from one keyword list.
Read the recipe →All recipes are maintained in the open-source repo — github.com/magpieiq/magpie-ecommerce-data-api — alongside the Python client (pip install magpie-data) and a live smoke test that keeps them honest.