WeaviateSink

The WeaviateSink class is designed to integrate with the Weaviate vector database, storing vectors produced from the Neum AI pipeline and retrieving them for semantic search operations.

​

Properties

Required properties:

• url: The URL of the Weaviate instance.
• api_key: The API key for authentication with the Weaviate service.
• class_name: The name of the class in Weaviate to store the data. Can be defined to any string you want.

Optional properties:

• num_workers: The number of workers used for batch processing.
• shard_count: The number of shards for the Weaviate class.
• batch_size: The number of vectors to store in a single batch.
• is_dynamic_batch: A flag indicating if batching should adapt based on the response time of the Weaviate instance.
• batch_connection_error_retries: The number of retries for batch connection errors.