Nodes: Scrapper

The scrapper data source allows to retrieve data from the client page using the link attribute from the input and

all data retrieved will be added and transformed to the mappings.

Scrapped data will added to the input data or will replace current attributes if using the same fields.

Link one input node to provide data to the scrapper

From your Scrapper node click on Config to access to the settings

Use CSS selectors to fetch elements from the product page. Adding a selector will create a field for your scrapper dataset. Use the right config depending on your field.
​

Output is the value returned by the scrapper after fetching through the selector:

Your scrapper datasource can be scheduled, using the cron scheduler

The scrapper will also be triggered when the input node completes. In order to disable this behavior, you must disable the auto execute option on the source link

The scrapper execution will run until all products in the input node are processed. The next nodes will then be executed except if auto execute option is disabled.

If you want to execute the scrapper in batches you can use the products limite parameter.

When the products limit is reached, the job will be snoozed until the next snooze schedule.

The same job will then resume until the limit is reached again or all products are processed.

By default a new request is sent as soon as the previous one completes

To reduce the request rate on the customer website, you can configure a delay to wait for some time between every single request.

When used with parallelism > 1 the same delay will be used across all parallel threads.
​

By default the scrapped uses a single thread to send requests sequentially

In order to increase the scrapper speed when the request take too long to complete, you can increase the parallelism to send multiple requests at the same time.

When used with delay > 0 all theads will be executed with the same delays settings

Set up your settings to affect the crawling of products by the bot Dataiads.

Special crawling fetching the product page after rendering. This mode can be used only there is no other way to scrap the page. Please note that the browser.

Always keep DataiadsBot by defaut

Headers to add to populate your request

To receive your data as JSON output, don't forget to add:​Accept: application/json

To add parameters to your endpoint URL

Once your setup finished you can test your products to check the scrapped data without saving the configuration.

Warning - input node must be connected in order to have input products for the test.

​
Optionally you can provide a product ID to test a specific product from your input feed. When using a product which is not from the feed. The response will be null.

Finally you can save your settings to commit your configuration

Mappings