from neumai.Pipelines.Pipeline import Pipelinepipeline = Pipeline( id = "Pipeline identifier", name = "Pipeline name", sources = [<SourceConnector>,...], embed = <EmbedConnector>, sink = <SinkConnector>)
If you have more than one source, ensure you design the metadata outputted by the source carefully. If the sources output different metadata properties depending on the sink this might lead to error or vectors in an index that don’t share metadata properties. This can be challening at retrieval time.
This will trigger the extraction of data from the data sources, transformation using the defined pre-processing steps and the loading of data into the vector store defined.