Using $ref in FME Flow Data Virtualization APIs for Deduplication
 

This might be an imprecise post, but my question is whether FME Data Virtualization follows the exported API specification or the visual editor’s interpretation - and why. I need guidance on restructuring my specifications: Data Virtualization seems to accept arrays of schema references but silently drops them, replacing them with blank references in properties but losing the json-type array in endpoint responses.

This inconsistent handling of $ref makes it unclear how Data Virtualization processes schema references — especially for arrays, since the visual editor only supports $ref for objects. 

Thanks for taking the time in advance!

The Use Case of Arrays with $ref

Here is an example usage of Arrays of Referenced Objects from the OpenAPI spec. I use the schema /pet a lot for Deduplication. Also having Arrays of type pet. How would one achieve this in FME? 

4.7.8.3 Paths Object Example

{
  "/pets": {
    "get": {
      "description": "Returns all pets from the system that the user has access to",
      "responses": {
        "200": {         
          "description": "A list of pets.",
          "content": {
            "application/json": {
              "schema": {
                "type": "array",
                "items": {
                  "$ref": "#/components/schemas/pet"
                }
              }
            }
          }
        }
      }
    }
  }
}

/pets:
  get:
    description: Returns all pets from the system that the user has access to
    responses:
      '200':
        description: A list of pets.
        content:
          application/json:
            schema:
              type: array
              items:
                $ref: '#/components/schemas/pet'

If I understand this correctly, the schema used for the response is of type array with 1 property ‘items’ that references to a scheme ‘pet’ for pets.

So to break it down its array → items → $ref/pet 
the expected response would be somthing like this:

[
  {
    "name": "Doggo",
    "age": 2
  },
  {
    "name": "Kitty",
    "age": 3
  }
]

So to build this kind of response i would either use an array as response and reference the pet schema or create an schema petArray and use it in the Response Structure.

If you were to create an response of json type array, you cant select a schema. So you would have to create the pet item inline by defining the existing properties again leading to duplication. But, you would get the expected response.

[
  {
    "name": "Doggo",
    "age": 2
  }
]

If you chose object as the json type for your response you can’t create a property of type array with the array type of pet either as you can’t reference the type of the array to an existing schema. Object arrays are allowed in general but then there would be no documentation of the objects (pet) in it. 

If you chose object as the json type for your response and create a property, you could reference to an existing schema leading to my approach of petArray. But we have the same issue with schemas of type array again - no arrays of a referenced type allowed. 
So i chose array of type object, created a property of type schema referenced pet and named the property “ref’. In the spec  would normaly use something like items for arrays, but as this is not fitting and because a property name is required for $referenced schemas as well, i just chose “ref”.

Now i was curious what i would get as a response, because I thought  $ref works by replacing itself and everything on its level with the definition it is pointing at.
 
So i named this one “ref” again and selected petArray and get this response:

{
  "ref": [
    {
      "ref": {
        "name": "Doggo",
        "age": 2
      }
    }
  ]
}

So instead of a clean unnested response (the response i wanted to achieve) this now has additional brackets, and more problematic the references not replacing themself.

This is what i want:
 

[
  {
    "name": "Doggo",
    "age": 2
  },
  {
    "name": "Kitty",
    "age": 3
  }
]

How would one achieve this in FME?


I wondered, can you import such references as json-type response? Set up this fictious PetApi Spec and imported it.
 

{
  "openapi": "3.0.3",
  "info": {
    "title": "Pet API",
    "version": "1.0.0"
  },
  "paths": {
    "/pets": {
      "get": {
        "description": "Returns all pets from the system that the user has access to",
        "responses": {
          "200": {         
            "description": "A list of pets.",
            "content": {
              "application/json": {
                "schema": {
                  "type": "array",
                  "items": {
                    "$ref": "#/components/schemas/pet"
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "pet": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "age": { "type": "integer" }
        },
        "required": ["name", "age"]
      },
	  "PetArray": {
        "type": "object",
        "properties": {
          "pets": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/pet"
            }
          }
        },
        "required": [
          "pets"
        ]
      }
    }
  }
}

No errors, just a quiet warning that i found in the datavirtualization.log (not the petApi log itself).

WARN No request body to import for endpoint GET /pets.

If we look at the endpoint, pets just references to pet, the type “array” gets lost.
If we look at the schema petArray, the reference is also lost, leaving the type blank with <Choose type (optional)>.

So if the visual editor doesn’t show type array in the response i expected fme changed it in the OAS. I Exported the OAS immediately after importing without tampering within the editor. No changes:

{
  "openapi" : "3.1.0",
  "info" : {
    "title" : "Pet API",
    "description" : "",
    "version" : "1.0.0",
    "summary" : ""
  },
  "servers" : [ {
    "url" : "api/petApi"
  } ],
  "paths" : {
    "/pets" : {
      "get" : {
        "tags" : [ ],
        "summary" : "",
        "description" : "Returns all pets from the system that the user has access to",
        "operationId" : "getPets",
        "parameters" : [ ],
        "responses" : {
          "200" : {
            "description" : "A list of pets.",
            "content" : {
              "application/json" : {
                "schema" : {
                  "type" : "array",
                  "items" : {
                    "$ref" : "#/components/schemas/pet"
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "components" : {
    "schemas" : {
      "pet" : {
        "type" : "object",
        "description" : "",
        "properties" : {
          "name" : {
            "type" : "string"
          },
          "age" : {
            "type" : "integer",
            "format" : "int32"
          }
        },
        "required" : [ "name", "age" ]
      },
      "PetArray" : {
        "type" : "object",
        "description" : "",
        "properties" : {
          "pets" : {
            "type" : "array",
            "items" : {
              "$ref" : "#/components/schemas/pet"
            }
          }
        },
        "required" : [ "pets" ]
      }
    }
  }
}

So does FME Flow now stick to the exported Specification and if so do i now have an array of referenced schemas? If not, why doesn’t it change the Specification and therefore documentation accordingly?