large_logo-transparent-1024x1024.png

medium_logo-transparent-1024x1024.png

small_logo-transparent-1024x1024.png

thumbnail_logo-transparent-1024x1024.png

logo-transparent-1024x1024.png

The missing NoCode function library for any automation platform. Well tested and easy to use. Register for free and get started.

1001fx-1200x627.png

1001fx is the most complete and missing set of NoCode and LowCode functions for all automation platforms.

1001fx - NoCode Functions

Today we want to show you an example on how to scrape websites using the 1001fx scrapehtml function.

Hint: Use 1001fx Plus to get it right now ([https://1001fx.com/blog/1001fxplus](link))

In this example we will extract all blog posts from the [1001fx blog](https://1001fx.com/blog).

First we need the html selector of a card. Please see the following short video on how to get the selector.
<div style="position: relative; padding-top: 56.25%">
  <iframe
    src="https://iframe.mediadelivery.net/embed/86437/9763c204-8b43-4db2-87cc-deebadd2abfb?autoplay=true&&muted=true"
    loading="lazy"
    style="border: none; position: absolute; top: 0; height: 100%; width: 100%"
    allow="accelerometer;gyroscope;autoplay;encrypted-media;picture-in-picture;"
    allowfullscreen="true"
  ></iframe>
</div>


With this selector we can setup our call:

```
curl --location 'https://api.1001fx.com/data/scrapehtml' \
--header 'Content-Type: application/json' \
--header 'x-api-key: YOUR_API_KEY' \
--data '{
    "selectors": [
        {
            "selector": "#__next > div > div > main > div > div > div > a",
            "attribute": "href",
            "keyField": "url"
        }
    ],
    "url": "https://1001fx.com/blog"
}'
```

The result will be the following

```
{
    "meta": {
        // Page Meta
        ...
    },
    "result": [
        {
            "url": "/blog/1001fxplus"
        },
        {
            "url": "/blog/changelog"
        },
        ...
    ]
}
```

Nice, we got the URLs to all blog posts. We can now extend the selectors array in the call to furthermore get all titles and subtitles from the blog posts.

```
curl --location 'https://api.1001fx.com/data/scrapehtml' \
--header 'Content-Type: application/json' \
--header 'x-api-key: YOUR_API_KEY' \
--data '{
    "selectors": [
        {
            "selector": "#__next > div > div > main > div > div > div > a",
            "attribute": "href",
            "keyField": "url"
        },
        {
            "selector": "#__next > div.flex.min-h-screen.flex-col > div > main > div > div > div > a > div.flex.flex-1.flex-col.justify-between.bg-white.p-6 > div > div.mt-2.block > p.text-xl.font-semibold.text-gray-900",
            "attribute": "text",
            "keyField": "title"
        },
        {
            "selector": "#__next > div.flex.min-h-screen.flex-col > div > main > div > div > div > a > div.flex.flex-1.flex-col.justify-between.bg-white.p-6 > div > div.mt-2.block > p.mt-3.text-base.text-gray-500",
            "attribute": "text",
            "keyField": "subtitle"
        }
    ],
    "url": "https://1001fx.com/blog"
}'
```

I got the selectors for title and subtitle the same i got the selector for the card itself. Feel free to try it yourself.
Notice: By getting the selector you might notice something like a:nth-child(1). If you would use this selector or function would only access the first blogpost. Simply remove the nth-child option and you are good to go.

Finally our response looks like this

```
{
    "meta": {
        // Page Meta
        ...
    },
    "result": [
        {
            "url": "/blog/1001fxplus",
            "title": "1001fx Plus",
            "subtitle": "Access to the latest 1001fx functions"
        },
        {
            "url": "/blog/changelog",
            "title": "Changelog version 1.5.0",
            "subtitle": "This changelog lists all changes to our API."
        },
       ...
    ]
}
```

Now we have a list of all blog posts including the URL, title and subtitle. This call can also be extended to retrieve the linked preview image.
 

You can find more examples and all supported parameters here:
<fx group=data fx=scrapehtml showicon=true short=true></fx>

How to: Use our function "scrapehtml"

Example on how to use the scrapehtml function