Scrape a page with the newer JavaScript rendering engine | Prospect demo

14 body fields

JSON request containing the URL to scrape and optional browser rendering, request, proxy, retry, and extraction settings.

urlstringrequired

URL to scrape

waitForSelectorstringoptional

CSS selector to wait to appear in DOM tree before page is considered as loaded.

postWaitTimeintegeroptional

Wait for specified amount of seconds after page load (from 1 to 12s). Use this only if ScrapeNinja failed to wait for required page elements automatically.

headersarray<string>optional

Custom headers to send with the request. By default, regular Chrome browser headers are sent to the target URL.

retryNumintegeroptional

Amount of attempts.

geostringoptional

Geo location for basic proxy pools (you can purchase premium ScrapeNinja proxies for wider country selection and higher proxy quality). [Read more about ScrapeNinja proxy setup](https://scrapeninja.net/docs/proxy-setup/)

Default:us

proxystringoptional

Premium or your own proxy URL (overrides `geo` field). [Read more about ScrapeNinja proxy setup](https://scrapeninja.net/docs/proxy-setup/)

timeoutintegeroptional

Timeout per attempt, in seconds. Each retry will take [timeout] number of seconds.

Default:16

textNotExpectedarray<string>optional

Text which will trigger a retry from another proxy address.

statusNotExpectedarray<integer>optional

HTTP response statuses which will trigger a retry from another proxy address.

Default:[403,502]

blockImagesbooleanoptional

Block images from loading. This will speed up page loading and reduce bandwidth usage.

blockMediabooleanoptional

Block (CSS, fonts) from loading. This will speed up page loading and reduce bandwidth usage.

screenshotbooleanoptional

Take a screenshot of the page. Pass "false" to increase the speed of the request.

extractorstringoptional

Custom JS function to extract JSON values from scraped HTML. Write&test your own extractor on https://scrapeninja.net/cheerio-sandbox/

1 status code

200Returns the rendered HTML body and an `info` object containing response metadata, including the status code, final URL, headers, screenshot, and page cookies.

infoobjectoptional

bodystringoptional

HTML body of the rendered page.

Error handling

url is required and must identify the page to scrape. postWaitTime, when supplied, must be between 1 and 12 seconds; this endpoint is available only through APIRoad.

import { Client } from "scrapeninja-apiroad-—-unofficial-octri-demo";

const client = new Client({
  baseUrl: "https://scrapeninja.apiroad.net",
  auth: { apiKey: process.env.API_KEY },
});

const result = await client.scrapeJs.create({
  url: "https://api.example.com/v1",
  waitForSelector: "example",
  postWaitTime: "2024-01-15T09:30:00Z",
  headers: [],
  retryNum: 20,
});

{
  "info": {
    "statusCode": 200,
    "finalUrl": "https://example.com/url",
    "catchedAjax": {
      "url": "https://example.com/api/data.json",
      "method": "GET",
      "headers": [
        "content-type: xxx",
        "header2: val2"
      ],
      "body": "<html><body><h1>Hello World!</h1></body></html>",
      "bodyIframe": "<html><body><h1>Iframe content</h1></body></html>",
      "status": 200,
      "responseHeaders": {
        "content-type": "application/json"
      }
    },
    "headers": [
      "content-type: xxx",
      "header2: val2"
    ]
  },
  "body": "<html><body><h1>Hello World!</h1></body></html>"
}

Request

Response

Errors