Introducing FME Data Virtualization!

Exciting!! This is going to allows us to build some really cool solutions for customers! Can’t wait for it to be out of beta

Great work ​@rylanatsafe and team

I predict this is going to do to the ‘webhook/streaming service’ what Automations did to the classic notification framework.

Thanks ​@rylanatsafe! I professionally love you. 

The ability for a user to create APIs without custom code is top Notch.

Thank you, ​@martinkoch and ​@hkingsbury ! A huge shout-out to our developers and design team for their hands-on involvement and genuine excitement from the beginning.

I am looking forward to seeing how you (and everyone else!) uses Data Virtualization to create new solutions.

Extremely keen to give this a whirl and explore it’s capabilities, nice job Safe!

For those eager to explore this new feature, I've developed a Custom Transformer to assist with HTTP response generation:
👉 DataVirtualizationResponseSetter on FME Hub

Thanks ​@abizien I woke up this morning with the idea: ‘Think what we can do, if we could stream back HTML’. Build kind of a web-catalog of our data (with download-buttons of the same data in different formats??)

That's a great idea — and funny you mention it! I'm currently working on a BetterReportGenerator group of Custom Transformers.It takes a different approach to generating web pages with FME than the native HTMLReportGenerator.

While it's not specifically aimed at building web catalogs, it could definitely be combined with the new data virtualization feature to achieve something similar — like dynamic pages with format-specific download links.

Still a bit early, so some patience is needed — but I’m looking forward to sharing more soon!

 

​This is great!

FYI ​@mark2atsafe 

Great idea. Does it set any headers as well? For example, it could set:

response.headers{0}.name = Date
response.headers{0}.value = @DateTimeNow()

I can’t remember what other headers exist, but Date is one that should always apply, I think.

Also, do you have a Sampler transformer? If not, it might be worth adding a Sampler to pass only 1 feature, in case someone accidentally sends 1,000 features through. I’m not sure what would happen, but it couldn’t be good!

I didn’t even think about returning HTML! Ooh. Now I have a number of areas opening up that I’d never considered!

I’m trying to think what other custom transformers might be of use.

I think DV workspaces need to always return something to the user, even if it’s an error, so trapping errors is going to be important. That includes parameter validation - make it case insensitive, check for valid data types and values, etc. 

In fact, some APIs return data in pages (e.g. fetch me page 6 of 100 records per page). A custom transformer could check if the page parameter values are valid, maybe generate a SQL statement like:

OFFSET @Value(ToSkip) ROWS FETCH NEXT @Value(query.pagesize) ROWS ONLY;

And maybe there could be a custom transformer that wraps up the JSONTemplater in a nicer way for beginners. For example, one of my templates starts out with:

{|
    if (fme:is-attribute-null("query.teamid"))
    then {"TeamID": fme:get-attribute("TeamID")}
    else (),
    {
    "PositionID": fme:get-attribute("PositionID"),
    "PositionLatitude": fme:get-attribute("PositionLatitude"),

Which is a lot of anyone to get their head around!

I’m not saying you - ​@abizien - should have to create these! I’m just throwing out some ideas.

As an extra bit of background, here’s a screenshot of one of my workspaces. You can see how I have a lot of validation going on, a number of response codes (that I can now replace with DataVirtualizationResponseSetters), a FeatureReader to get data from a database, and the JSONTemplater to create the response.

​@mark2atsafe 
“Does it set any headers as well?”

At the moment, no — headers aren’t directly handled. I was hoping to allow users to define a list of custom headers through the CT parameters, but FME doesn’t currently support editable table-type parameters that would be ideal for this use case. Maybe that’s something worth suggesting as a Workbench enhancement!

That said, I could definitely add some basic headers automatically — like Date — and I’d be happy to take suggestions on other useful defaults.

“Also, do you have a Sampler transformer?”
I don’t know what happen when you try to send more than one feature to a DV writer, but wouldn’t I choose to let the user experiment with that 👀
I guess I will try testing a bit more with the new Data Visualization feature before adding any user-proofness.


Thank you for your feedback and ideas for the BetterReportGenerator !
That’s an excellent use-case for those custom transformers.

The system I'm building is fairly modular: each report component (titles, tables, charts, etc.) is handled by a dedicated transformer, making it easy to expand or plug in new functionality. Here's a screenshot of the workspace that generated the sample report I showed earlier, if that make sense :

Note that an HTML Writer was used here, but we could use a DVResponseSetter + DV Writer to return the HTML report through DV API.

Given the modular design, adding a new transformer to fetch data using DV (as you suggested) would be totally feasible !

That said, it might require a well-structured DV API with standardized endpoints and schemas to really shine — so we’re probably talking about a full-on FME Flow project at that point!

I’ve just published two new Custom Transformers to simplify JSON response generations for DV !

👉 DataVirtualizationJSONList
👉 DataVirtualizationGeoJSON

 

Just released another Custom Transformer that might be useful for paginated responses in DV:
👉 DataVirtualizationJSONPaginate

Could be handy for your use case, ​@mark2atsafe ! It helps structure paged JSON outputs cleanly based on query parameters.