Skip to main content
Basic website scraping. Works with most websites, but might fail with javascript heavy ones. This connector takes a given URL for a website and extracts the available text from it. Once extracted, the contents are outputted as an HTML.

​
Properties

Required properties:
  • url: URL for website
Available metadata
  • url: URL for website
Compatible loaders:
  • AutoLoader
  • HTMLLoader

​
Usage

from neumai.DataConnectors import WebsiteConnector
from neumai.Shared import Selector

website_connector =  WebsiteConnector(
    url = "https://www.neum.ai/post/retrieval-augmented-generation-at-scale"
    selector = Selector(
        to_metadata=['url']
    )
)

{
    "sources":[
        {
            "data_connector": {
                "connector_name":"WebsiteConnector",
                "connector_information":{
                    "url":"https://www.neum.ai/post/retrieval-augmented-generation-at-scale",
                    "selector": {
                        "to_metadata": [
                            "url"
                        ]
                    }
                }
            }
        }
    ]
}