FME Workspace Description

Use case

This FME workspace is designed for processing, analyzing, and preparing data to feed a local ollama model with your own pdf. It is just a demo workspace to understand RAG concept and Ollama calls without python.

Requirements

Install Ollama

Load your embedding model plus your LLM model into it first.

Key Components

Chunk Processing:

Splits the input data into smaller chunks for parallel processing.

Utilizes transformers to manage chunk size and distribute data.

Embedding:

Embeds validation steps to identify and correct errors in data chunks.

Isolates problematic chunks for further analysis or correction.

Preprocessed Data Storage:

Outputs results in formats compatible with Parquet and other storage types.

Ensures streamlined data flow for further usage in external systems or analysis tools.

SQL Integration : Vector space:

Uses SQL queries to store and manage data in memory, enabling efficient data manipulation.

Queries include extension and table creation and selection for specific data extraction.

Result Generation:

Call to ollama local LLM