Website Reader

WebsiteReader is a reader class that allows you to read data from websites.

Parameters

Parameter	Type	Default	Description
chunking_strategy	Optional[ChunkingStrategy]	FixedSizeChunking()	Strategy used to chunk documents
max_depth	int	3	Maximum depth level for crawling links
max_links	int	10	Maximum number of links to crawl
timeout	int	10	Request timeout in seconds
proxy	Optional[str]	None	Proxy URL used for requests
allowed_hosts	Optional[List[str]]	None	Hostnames the reader is allowed to fetch from. See Restricting URL Fetches.

WebsiteReader also accepts the base Reader constructor parameters.

Methods

read()

Crawls a website starting from the given URL and returns a list of documents.

Parameter	Type	Default	Description
url	str	Required	URL of the website to crawl and read
name	Optional[str]	None	Name override for the resulting documents

async_read()

Async variant of read(). Accepts the same parameters.