Convert feature to JSON

Hey ​@nic_ran - I was fussing with something very similar just yesterday. I ended up doing a really quick and simply python caller to take all the attributes and turn it into a json object (one per feature). It’s not very robust, doesn’t support lists or sub-objects, but it did help me out in the task I was doing.
 

import json
import fme
from fme import BaseTransformer
import fmeobjects


class FeatureProcessor(BaseTransformer):
    def __init__(self):
        pass

    def has_support_for(self, support_type: int):
        return support_type == fmeobjects.FME_SUPPORT_FEATURE_TABLE_SHIM

    def input(self, feature: fmeobjects.FMEFeature):
        # Extract all attributes from the feature
        attributes = feature.getAllAttributeNames()

        # Create a dictionary to hold the attribute-value pairs
        attribute_dict = {}
        for attr in attributes:
            attribute_dict[attr] = feature.getAttribute(attr)

        # Convert the dictionary to a JSON string
        json_string = json.dumps(attribute_dict)

        # Add the JSON string as a new attribute '_json'
        feature.setAttribute("_json", json_string)

        # Output the feature
        self.pyoutput(feature, output_tag="PYOUTPUT")

    def close(self):
        pass

    def process_group(self):
        pass

    def reject_feature(self, feature: fmeobjects.FMEFeature, code: str, message: str):
        feature.setAttribute("fme_rejection_code", code)
        feature.setAttribute("fme_rejection_message", message)
        self.pyoutput(feature, output_tag="<Rejected>")

Yeah I have the same wish - You can recursively flatten a json feature and it will use the dot notation for objects and sub objects and for arrays you get lists. It would be great to have something that will do the reverse.

Just making small changes to JSONObjects with the JSONUpdator is doable, however, it can be just a lot of configuration and it’s a bit sensitive. 

I work with JSON all the time and I don’t think the JSON writer has ever actually be helpful for me.   

Although it doesn't quite achieve what you want, have you looked at my custom DataVirtualizationJSON* transformers?

https://hub.safe.com/publishers/abizien/transformers/datavirtualizationjsonlist

https://hub.safe.com/publishers/abizien/transformers/datavirtualizationjsonpaginate

https://hub.safe.com/publishers/abizien/transformers/datavirtualizationgeojson

I thought about making a custom transformer that would do the equivalent of a json.dump(), but an FME feature often drags unexposed attributes you don't want to include in the result.

So you have to allow selection of the attributes you want to include in the JSON, and it is not necessarily simple to allow the selection and processing of list-type attributes.

I will take a deeper look at it though.

For more context on the issues with automatic conversion of FME entities to JSON, what should the JSON representation of this list be?

The FME data model combines the concepts of list and object

{
    "_creation_insntace": 0,
    "AttrList": [
        "Value1", 
		{
            "childAttr1": "Value3",
            "childAttr2": "Value5"
        },
        "Value1", 
		{
            "childAttr1": "Value4",
            "childAttr2": "Value6"
        }
    ]
}

This would be one option. Am I missing something there? 
 

Well, then the indexes no longer match (and it's a pretty ugly - but conforming - JSON list)

Maybe it’s not really an issue, but a list with two elements becoming a list with four elements doesn’t seem proper to me.

And trying to convert it back to FME feature (with a JSONFlattener) would give AttrList{0-3} instead of AttrList{0-1}. So it might be a lossy process.

Agreed - But I don’t really see any other option for it - In the end there is actually not way to create this from json in FME so either skip out the values directly in the list (in the case there is an object) or throw an error. The user needs to makes sure their data is compatible I guess

import json
import fme
from fme import BaseTransformer
import fmeobjects


class FeatureProcessor(BaseTransformer):
    def __init__(self):
        pass

    def has_support_for(self, support_type: int):
        return support_type == fmeobjects.FME_SUPPORT_FEATURE_TABLE_SHIM

    def input(self, feature: fmeobjects.FMEFeature):
        # Extract all attributes from the feature
        attributes = feature.getAllAttributeNames()

        # Create a dictionary to hold the attribute-value pairs
        attribute_dict = {}
        for attr in attributes:
            attribute_dict[attr] = feature.getAttribute(attr)

        # Convert the dictionary to a JSON string
        json_string = json.dumps(attribute_dict)

        # Add the JSON string as a new attribute '_json'
        feature.setAttribute("_json", json_string)

        # Output the feature
        self.pyoutput(feature, output_tag="PYOUTPUT")

    def close(self):
        pass

    def process_group(self):
        pass

    def reject_feature(self, feature: fmeobjects.FMEFeature, code: str, message: str):
        feature.setAttribute("fme_rejection_code", code)
        feature.setAttribute("fme_rejection_message", message)
        self.pyoutput(feature, output_tag="<Rejected>")

Thanks ​@NickAtSafe , I appreciate the reply and the source code. Definitely something could be done along those lines but it would be nice to see it as native FME functionality, including list support.

Thanks for your replies ​@alexbiz and ​@virtualcitymatt , I appreciate the input to the discussion and really appreciate the work you’ve already done alexbiz on your data virtualization JSON transformers.

Working with FME list structures is often a pain point and I can see there are definitely complexities in converting them to JSON. The example you’ve given with top level list attribute values and child values does pose a problem but, to be honest, I don’t think I’ve ever actually seen a list that looks like that.

I think whatever solution can be built it should make a best attempt at creating conforming JSON. The user needs to take some responsibility for ensuring the feature attributes (and lists) make sense before converting to JSON.

I just ran into this problem, but mistook it for a problem with encoding/decoding non-ASCII characters in DV, since it only erred when these were included.

I found a post from 5 years ago about it, and used the suggestion to manually build the JSON with other transformers, and this worked out just nicely. But it shouldn’t have to be this cumbersome.

I fully support serious work being done to remedy the shortcomings of the JSONTemplater. As it is now, it’s basically useless for anything but the simplest tasks. And yes, DV demands more.