The scrapper data source is an enhanced version of the current scrapper feature from the resources components.
As a data source, all data retrieved will be added and transformed to the mappings.
Create a Scrapper Data source
On the Data sources section click on CREATE
If you already created a Scrapper account choose Scrapper data source otherwise choose CREATE A NEW SCRAPPER ACCOUNT
Choose a Scrapper account
Create the Scrapper data source
Configure a Scrapper API Data source
From Scrapper source, click on configure to set up your dataset
Targeted products
Use a targeting to only sync your data with a batch of products
Selectors
Use CSS selectors to fetch elements from the product page. Adding a selector will create a field for your scrapper dataset. Use the right config depending on your field.
Output is the value returned by the scrapper after fetching through the selector:
html: returns the inner HTML
outerHtml: returns the outer HTML
attribute: returns the attribute value set on the input "Output attribute"
text: returns the inner text of all children elements
httpStatusCode: returns the status code of an element
Advanced settings
Set up your settings to affect the crawling of products by the bot Dataiads.
Browser mode
Special crawling fetching the product page after rendering. This mode can be used only there is no other way to scrap the page. Please note that the browser.
User Agent
Always keep DataiadsBot by defaut
Delay
The frequency between each crawling (in milliseconds)
Parallelism
The number of simultaneous products crawled
HTTP headers
Headers to add to populate your request
To receive your data as JSON output, don't forget to add:Accept: application/json
Query params
To add parameters to your endpoint URL
Testing your products
Once your setup finished you can test your products to check if any data if retrieved
You can save your settings and activate your source to begin the synchronization.
To learn about Data sources activation and synchronization please read this article
To learn about Mappings please read this article