Skip to main content

Data Virtualisation Transformers


alexbiz
Supporter
Forum|alt.badge.img+10

*Aka Data Virtualization Transformers, but with the french touch



Since I’ve been publishing a bunch of Data Virtualization custom transformers lately, I figured it was time to make a post about them—what they do and how to use them.

FME Flow’s recent update introduced a cool new feature: Data Virtualization.
https://community.safe.com/product-updates/introducing-fme-data-virtualization-38066?tid=38066&postid=86#post86

This feature lets you build your own APIs using FME workbenches.

To make that work, your features need to include certain attributes required by the Data Virtualization writer, such as:

  • response.status_code

  • response.body.content

  • response.body.content_type

  • response.body.content_file_path

  • response.headers{}.name

  • response.headers{}.value

Data Virtualization Writer

 

 


After manually setting these with an AttributeCreator a few times, I thought: why not create a tool to make this easier?

So I made the DataVirtualizationResponseSetter !


DataVirtualizationResponseSetter


This transformer takes care of setting all the required attributes, and gives you a list of predefined values for common fields like response code and content type.

 

 

While this helps format your features for FME, you still need to build your JSON responses manually—using AttributeCreators, JSONTemplaters, etc.

To help with that part, I’ve published three new transformers:

 

DataVirtualizationJSONList


It turns incoming features into a structured JSON array and sets the proper HTTP response attributes.

Example output:

{
  "data": [
    { "mail": "alice@example.com", "name": "Alice" },
    { "mail": "bob@example.com", "name": "Bob" },
    { "mail": "carol@example.com", "name": "Carol" }
  ]
}

 

 

DataVirtualizationJSONPaginate


This one handles pagination and includes metadata in the response, so it's great for building paginated API endpoints.
Example:

{
  "data": [
    { "id": 101, "name": "Item 101" },
    { "id": 102, "name": "Item 102" }
  ],
  "meta": {
    "offset": 100,
    "limit": 2,
    "total": 1000
  }
}

 

 


DataVirtualizationGeoJSON


This one generates valid GeoJSON from spatial features and wraps everything up nicely into a FeatureCollection.
Example:

{
  "type": "FeatureCollection",
  "name": "example",
  "features": [
    {
      "type": "Feature",
      "geometry": { ... },
      "properties": {
        "CODE_DEPT": "39",
        "NOM_DEPT": "JURA",
        ...
      }
    },
    ...
  ]
}


 

Hopefully these transformers help you build your own JSON REST APIs with FME Flow's Data Virtualization.

They're open, so feel free to explore or tweak them however you like. Feedback and suggestions are very welcome!


Check out my FME Hub publisher page to find out about my other tools

7 replies

hkingsbury
Celebrity
Forum|alt.badge.img+54
  • Celebrity
  • May 11, 2025

These are fantastic!!! That Pagination one is clever!

@todd_davis ​@rylanatsafe ​@NickAtSafe ​@mark2atsafe 


carlm
Contributor
Forum|alt.badge.img+12
  • Contributor
  • May 12, 2025

For a weird reason when I try to add them to my workspace it glitch, I don’t know if it’s related to the beta version (2025.1), the Hub, or the Transformers.

For example I add the JSONPagination, this on loads fine, but f i try to add the GeoJSON one it fails and load the ResponseSetter, same thing if i choose the JSONList, it adds the ResponseSetter instead.

If i choose to add ResponseSetter first I can’t load the other ones and it  keeps adding ResponseSetter instead of any of the 3 other Custom.


alexbiz
Supporter
Forum|alt.badge.img+10
  • Author
  • Supporter
  • May 12, 2025

Yes there might be a problem with both FME 2025.1 and FME Hub.

I think FME has an hard time loading CT using others CT inside of it. I’ve had this issue too.

As well as FME Hub glitching out, hiding some of my CT or duplicate others...
 

 


carlm
Contributor
Forum|alt.badge.img+12
  • Contributor
  • May 12, 2025

I was testing GeoJSON Custom and I get this message.

Data Virtualization Writer: Unrecognized content-type, please verify content-type: application/geo+json.

 

I can make it work if I change the content-type to application/json.

I was wondering if it’s working on anybody’s side.

 


alexbiz
Supporter
Forum|alt.badge.img+10
  • Author
  • Supporter
  • May 12, 2025

Hmpf I’ve done the application/geo+json content-type hoping FME will be ok with us using whatever we wanted. I guess we’re stuck using a list of preset content-type.
I’ll update the DataVirtualizationGeoJSON with content-type set to application/json.

I’ll keep it updated with the information we have, but we’ve got no documentation about that for now. 
We will have to wait a bit for the release to have a more stable version.


NickAtSafe
Safer
Forum|alt.badge.img+4
  • Safer
  • May 12, 2025
abizien wrote:

Yes there might be a problem with both FME 2025.1 and FME Hub.

I think FME has an hard time loading CT using others CT inside of it. I’ve had this issue too.

As well as FME Hub glitching out, hiding some of my CT or duplicate others...
 

 

@carlm ​@abizien - I have logged a note with the Form development team about the Quick Add having some challenges. They’re looking into it today and I’ll report back if there are any updates as they investigate. (FMEFORM-33784).


rylanatsafe
Safer
Forum|alt.badge.img+13

Thank you, ​@alexbiz , this is an excellent contribution to the FME Community! 

We have already received a lot of feedback about creating and configuring Data Virtualization APIs in FME and more flexible Content Type came up a few times.

A question for the community: Is it more important to support additional Content Types for the endpoint response, or for the request? We have our hunch, but it’d be great to learn what workflows you have in mind or are creating now. (Feel free to send a direct message through the FME Community platform here.)


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings