content as the building block of any piece of knowledge.
Content can be added to knowledge from different sources.
Knowledge content needs to be read and chunked before it can be passed to any vector database for embedding, storage and ultimately, retrieval.
When content is added to Knowledge, a default reader is selected. Readers are used to parse content from the origin and then chunk it into smaller
pieces that will then be embedded by the vector database.
Custom readers or an override to the default reader and/or its settings can be passed when adding the content. In the below example, an instance of the standard
PDFReader class is created
but we update the chunk_size. Similarly, we can update the chunking_strategy and other parameters that will influence how content is ingested and processed.
Next Steps
Search & Retrieval
How agents search and find information in your knowledge base
Readers
Explore content parsing and ingestion options in detail
Chunking Strategies
Optimize how content is broken down for better search results
Vector Databases
Choose the right storage solution for your knowledge base