Quickstart

Set up enviornment

We will start by installing the required dependencies:

pip install neumai

Configure data source

For this guide, we will start with the Website source. This source will scrape the web contents of a site and return the HTML in the body.

To configure the Data Connector, we will specify the url property. The connector also supports a Selector to define what information from the connector should be used as content to embed and what content should be attached to the vector as metadata.

  from neumai.DataConnectors import WebsiteConnector
  from neumai.Shared import Selector
  website_connector =  WebsiteConnector(
      url = "https://www.neum.ai/post/retrieval-augmented-generation-at-scale",
      selector = Selector(
          to_metadata=['url']
      )
  )

We will then choose a loader and chunker to be used to pre-process the data extracted from the source. For the Website source, we will use an HTML Loader as we are extracting HTML code and will use the Recursive Chunker to split up the text. We will configure the Data Connector, Loader and Chunker into a SourceConnector.

from neumai.Loaders.HTMLLoader import HTMLLoader
from neumai.Chunkers.RecursiveChunker import RecursiveChunker
from neumai.Sources import SourceConnector

source = SourceConnector(
  data_connector = website_connector, 
  loader = HTMLLoader(), 
  chunker = RecursiveChunker()
)

Next we will configure the embedding service we will use to turn the chunks of text into vector embeddings.

Configure embed connector

We will use the OpenAIEmbed connector. This connector uses text-ada-002, one of the most popular embedding model in the market to generate vector embeddings.

Configure the connector with an OpenAI Key.

from neumai.EmbedConnectors import OpenAIEmbed

openai_embed = OpenAIEmbed(
    api_key = "<OPEN AI KEY>",
)

Next we will configure the vector storage service we will use to store the vector embedding we generated.

Configure sink connector

We will use the WeaviateSink connector. Weaviate is a popular open-source vector database.

Configure the Weaviate connector with the connection parameters including: url and api_key. Other parameters are available to further configure the connector. For example, we will use class_name to define a name for the index we are creating.

from neumai.SinkConnectors import WeaviateSink

weaviate_sink = WeaviateSink(
  url = "your-weaviate-url",
  api_key = "your-api-key",
  class_name = "your-class-name",
)

We now have all the parts of the pipeline configured, lets put it all together and run it.

Run the pipeline

To run the pipeline all together, we will first configure a Pipeline object and then use the built-in methods to run it.

from neumai.Pipelines import Pipeline

pipeline = Pipeline(
  sources=[source], 
  embed=openai_embed, 
  sink=weaviate_sink
)

Then we will run the pipeline locally with the provided built-in methods

print(f"Vectors stored: {pipeline.run()}")

The run method returns the number of vectors that were processed.

Search the pipeline

Finally, once it is done running, we can now query the information we just moved into vector storage.

results = pipeline.search(
  query="What are the challenges with scaling RAG?", 
  number_of_results=3
)

for result in results:
    print(result.metadata)

Deploy the pipeline

Once you have tested the pipeline locally, you can now take the configuration you created and deploy it to the Neum AI Cloud.