File Custom Metadata Manager Transformer Description
This transformer allows you to manage custom metadata for files through three primary operations:
Set - Add or update metadata (either all metadata or a specific key-value pair)
Get - Retrieve metadata (either all metadata or a specific key)
Remove - Delete all custom metadata from a file
The transformer leverages Python to manipulate file streams with alternate data streams (ADS) in Windows file systems, storing metadata in a structured JSON format.
Use Cases
1. Data Lineage Tracking
Track data processing history by attaching metadata about transformations applied, processing dates, or source information to files without modifying their original content.
2. Workflow Configuration Management
Store configuration parameters as metadata directly with the data files, allowing workflows to read these settings during execution rather than hard-coding them.
3. Compliance and Auditing
Add regulatory or compliance information to files (e.g., data sensitivity classifications, retention periods, or responsible department information).
4. Enhanced Data Cataloging
Store additional descriptive information with files that might not be supported by the file format's native metadata capabilities.
5. Workflow State Management
Use metadata to track processing state (e.g., "validated", "transformed", "published") to support multi-stage workflows.
Would you like to know more? Click here to find out more details!
