FME Workspace Description
Use case
This FME workspace is designed for processing, analyzing, and preparing data to feed a local ollama model with your own pdf. It is just a demo workspace to understand RAG concept and Ollama calls without python.
Requirements
Install Ollama
Load your embedding model plus your LLM model into it first.
Key Components
Chunk Processing:
Splits the input data into smaller chunks for parallel processing.
Utilizes transformers to manage chunk size and distribute data.
Embedding:
Embeds validation steps to identify and correct errors in data chunks.
Isolates problematic chunks for further analysis or correction.
Preprocessed Data Storage:
Outputs results in formats compatible with Parquet and other storage types.
Ensures streamlined data flow for further usage in external systems or analysis tools.
SQL Integration : Vector space:
Uses SQL queries to store and manage data in memory, enabling efficient data manipulation.
Queries include extension and table creation and selection for specific data extraction.
Result Generation:
Call to ollama local LLM
Would you like to know more? Click here to find out more details!
