Skip to main content

I am having some difficulty normalizing reverse geocodes for comparison from one Geocoding Service to another (in this case OpenStreetMap).  The _address that is returned can pretty easily split up into a list or separate attributes with most services as they are consistent.

 

However, OSM’s _address has extra information and parsing it to be consistent with anything else has been a challenge.  See below where instead of having an AddressNumber, a null, or a 0, I get Falmouth Public Schools & Moonlight Meadery in the 1st position.  For that matter, I sometimes get supplementary place names such as Seaport & South Boston when expecting a Town in a particular separation/position.  So it throws off looking in any specific list due to the inconsistent placement.

Some are normal, but many either don’t contain the expected info, or add info that is unnecessary for my needs.

What I want to make into attributes: AddressNumber  Street  Town  State  ZipCode

OpenStreetMap gives me this in a test...

 

Any ideas how I might achieve that without pulling out more hair like from the list above?  Am I missing something with Transformers that may be helpful?

Many thanks!

Not sure if I understand correct, can’t parse look through the list from right to left?


Left or right does not matter as much since the list changes with the position of the certain feature sets.  ListReverser does help to standardize the last 3 elements, but doesn’t quite get there in solving the rest.  I.E. there seems to be no real standard in that reverse geocode other than it will add extra attributes as it sees fit.  See why I have no hair!


Hi ​@bsw-gis,

 

I believe you can obtain those attributes by parsing through the full JSON response sent from the geocoding service. In the Geocoder transformer under Advanced, there is a parameter called “Include Full JSON Response”. If it’s set to “Yes”, the Geocoder will create an attribute called “_json_response” in the output.

If you connect a JSONExtractor after the Geocoder, you can name the target attributes you want to fill, then craft a JSON query to fill those attributes. For example, if we look at the structure of the JSON response, we can obtain the house number using following the JSON Query: 

json["address"]["house_number"]

However, this will require some knowledge and skill with JSON query in the JSONExtractor transformer. For reference, here is the JSONExtractor transformer’s documentation (JSON Query information located under Configuration > Parameters > Extract Queries > JSON Query).

 

Hope this helps, let me know if it works for you!


Hi ​@bsw-gis,

 

I believe you can obtain those attributes by parsing through the full JSON response sent from the geocoding service. In the Geocoder transformer under Advanced, there is a parameter called “Include Full JSON Response”. If it’s set to “Yes”, the Geocoder will create an attribute called “_json_response” in the output.

If you connect a JSONExtractor after the Geocoder, you can name the target attributes you want to fill, then craft a JSON query to fill those attributes. For example, if we look at the structure of the JSON response, we can obtain the house number using following the JSON Query: 

json["address"]["house_number"]

However, this will require some knowledge and skill with JSON query in the JSONExtractor transformer. For reference, here is the JSONExtractor transformer’s documentation (JSON Query information located under Configuration > Parameters > Extract Queries > JSON Query).

 

Hope this helps, let me know if it works for you!

Thanks Crystal!  I rarely work with JSON so it may take a little time to get ahold of it, but from what I have seen so far it is the most promising solution so far.

I am having some difficulty in the Query part in extracting the feature name after using the Geocoder...OpenStreetMap..etc etc….Full JSON Response. None of the below are pushing out anything in the Target Attribute.

 


Hi ​@bsw-gis,

 

I believe you can obtain those attributes by parsing through the full JSON response sent from the geocoding service. In the Geocoder transformer under Advanced, there is a parameter called “Include Full JSON Response”. If it’s set to “Yes”, the Geocoder will create an attribute called “_json_response” in the output.

If you connect a JSONExtractor after the Geocoder, you can name the target attributes you want to fill, then craft a JSON query to fill those attributes. For example, if we look at the structure of the JSON response, we can obtain the house number using following the JSON Query: 

json["address"]["house_number"]

However, this will require some knowledge and skill with JSON query in the JSONExtractor transformer. For reference, here is the JSONExtractor transformer’s documentation (JSON Query information located under Configuration > Parameters > Extract Queries > JSON Query).

 

Hope this helps, let me know if it works for you!

Thanks Crystal!  I rarely work with JSON so it may take a little time to get ahold of it, but from what I have seen so far it is the most promising solution so far.

I am having some difficulty in the Query part in extracting the feature name after using the Geocoder...OpenStreetMap..etc etc….Full JSON Response. None of the below are pushing out anything in the Target Attribute.

 

For the JSON Query column, you’re definitely on the right track! If we look at _json_response, the structure of the JSON has the attributes of “house_number”, “road”, “village”, etc. nested under “address”. Therefore, we need to add [“address”] as part of our JSON Query.

Snippet of the relevant portion of _json_response, note that attributes like “house_number” are nested within “address”:

{
    "address": {
        "ISO3166-2-lvl4": "US-NH",
        "country": "United States",
        "country_code": "us",
        "county": "Hillsborough County",
        "hamlet": "New Ipswich Center",
        "house_number": "690",
        "postcode": "03071",
        "road": "Turnpike Road",
        "state": "New Hampshire",
        "village": "New Ipswich"
    },

With that in mind, the JSON Query for “state” should be: json[“address”][“state”]. The same can be applied for the other attributes as well, for example we can retrieve “postcode” using json[“address”][“postcode”], and “village” using json[“address”][“village”].

If you would like to learn more, the JSONFragmenter transformer documentation has great information on constructing JSON Query expressions.

Hope this helps!


Hi ​@bsw-gis,

 

I believe you can obtain those attributes by parsing through the full JSON response sent from the geocoding service. In the Geocoder transformer under Advanced, there is a parameter called “Include Full JSON Response”. If it’s set to “Yes”, the Geocoder will create an attribute called “_json_response” in the output.

If you connect a JSONExtractor after the Geocoder, you can name the target attributes you want to fill, then craft a JSON query to fill those attributes. For example, if we look at the structure of the JSON response, we can obtain the house number using following the JSON Query: 

json["address"]["house_number"]

However, this will require some knowledge and skill with JSON query in the JSONExtractor transformer. For reference, here is the JSONExtractor transformer’s documentation (JSON Query information located under Configuration > Parameters > Extract Queries > JSON Query).

 

Hope this helps, let me know if it works for you!

Awesome Crystal!  This was the best solution. I didn’t immediately recognize the nesting under “address”.


For the JSON Query column, you’re definitely on the right track! If we look at _json_response, the structure of the JSON has the attributes of “house_number”, “road”, “village”, etc. nested under “address”. Therefore, we need to add [“address”] as part of our JSON Query.

Snippet of the relevant portion of _json_response, note that attributes like “house_number” are nested within “address”:

{
    "address": {
        "ISO3166-2-lvl4": "US-NH",
        "country": "United States",
        "country_code": "us",
        "county": "Hillsborough County",
        "hamlet": "New Ipswich Center",
        "house_number": "690",
        "postcode": "03071",
        "road": "Turnpike Road",
        "state": "New Hampshire",
        "village": "New Ipswich"
    },

With that in mind, the JSON Query for “state” should be: json[“address”][“state”]. The same can be applied for the other attributes as well, for example we can retrieve “postcode” using json[“address”][“postcode”], and “village” using json[“address”][“village”].

If you would like to learn more, the JSONFragmenter transformer documentation has great information on constructing JSON Query expressions.

Hope this helps!

 

One thing that seems to elude me is understanding and/or extracting the hierarchy.  When I utilize the same methodology that works with OSM Geocoding and apply it to HERE Geocoding, the hierarchy looks to be the same, yet I get blank results for HERE.  Any thoughts?

This returns nothing and using “items” before “address” also returns nothing:

 

This is one JSON response for the HERE request:

 

HERE JSON Response: {"items": [{"title": "700 Turnpike Rd, New Ipswich, NH 03048-3216, United States", "id": "here:af:streetsection:SffD8pAbQ3m1wkzMVaDlxA:CgcIBCCCh9", "resultType": "houseNumber", "houseNumberType": "PA", "address": {"label": "700 Turnpike Rd, New Ipswich, NH 03048-3216, United States", "countryCode": "USA", "countryName": "United States", "stateCode": "NH", "state": "New Hampshire", "county": "Hillsborough", "city": "New Ipswich", "street": "Turnpike Rd", "postalCode": "03048-3216", "houseNumber": "700"}, "position": {"lat": 42.75859, "lng": -71.86321}, "access": [{"lat": 42.75852, "lng": -71.86347}], "distance": 32, "mapView": {"west": -71.90508, "south": 42.75013, "east": -71.82752, "north": 42.78103}}]}

 

Thanks so much!


For the JSON Query column, you’re definitely on the right track! If we look at _json_response, the structure of the JSON has the attributes of “house_number”, “road”, “village”, etc. nested under “address”. Therefore, we need to add [“address”] as part of our JSON Query.

Snippet of the relevant portion of _json_response, note that attributes like “house_number” are nested within “address”:

{
    "address": {
        "ISO3166-2-lvl4": "US-NH",
        "country": "United States",
        "country_code": "us",
        "county": "Hillsborough County",
        "hamlet": "New Ipswich Center",
        "house_number": "690",
        "postcode": "03071",
        "road": "Turnpike Road",
        "state": "New Hampshire",
        "village": "New Ipswich"
    },

With that in mind, the JSON Query for “state” should be: json[“address”][“state”]. The same can be applied for the other attributes as well, for example we can retrieve “postcode” using json[“address”][“postcode”], and “village” using json[“address”][“village”].

If you would like to learn more, the JSONFragmenter transformer documentation has great information on constructing JSON Query expressions.

Hope this helps!

 

One thing that seems to elude me is understanding and/or extracting the hierarchy.  When I utilize the same methodology that works with OSM Geocoding and apply it to HERE Geocoding, the hierarchy looks to be the same, yet I get blank results for HERE.  Any thoughts?

This returns nothing and using “items” before “address” also returns nothing:

 

This is one JSON response for the HERE request:

 

HERE JSON Response: {"items": [{"title": "700 Turnpike Rd, New Ipswich, NH 03048-3216, United States", "id": "here:af:streetsection:SffD8pAbQ3m1wkzMVaDlxA:CgcIBCCCh9", "resultType": "houseNumber", "houseNumberType": "PA", "address": {"label": "700 Turnpike Rd, New Ipswich, NH 03048-3216, United States", "countryCode": "USA", "countryName": "United States", "stateCode": "NH", "state": "New Hampshire", "county": "Hillsborough", "city": "New Ipswich", "street": "Turnpike Rd", "postalCode": "03048-3216", "houseNumber": "700"}, "position": {"lat": 42.75859, "lng": -71.86321}, "access": [{"lat": 42.75852, "lng": -71.86347}], "distance": 32, "mapView": {"west": -71.90508, "south": 42.75013, "east": -71.82752, "north": 42.78103}}]}

 

Thanks so much!

It appears like the _json_response from HERE is a bit different from OSM, that’s why the JSON Query isn’t working. Taking a look at the HERE JSON response, a couple things to note here is that “address” is held inside “items”, which means that we need to reference “items” in our JSON Query as well.

However, an important note here is that “items” is a JSON array, so need to dive into JSON a bit more to work with this response. To extract “street” for example, we need to use the following JSON Query: json["items"][0]["address"]["street"]. This JSON Query can be easily translated for the other target attributes, just replace the [“street”] part with another attribute.

Summary: the _json_response from HERE holds objects such as “address” within an array called “items”. So, we need to reference “items”, specifically the first element, using 0 as the index. This can be a bit confusing, but HERE seems to wrap their results in an array, even if there’s only one result.


One thing that seems to elude me is understanding and/or extracting the hierarchy.  When I utilize the same methodology that works with OSM Geocoding and apply it to HERE Geocoding, the hierarchy looks to be the same, yet I get blank results for HERE.  Any thoughts?

This returns nothing and using “items” before “address” also returns nothing:

 

This is one JSON response for the HERE request:

 

HERE JSON Response: {"items": [{"title": "700 Turnpike Rd, New Ipswich, NH 03048-3216, United States", "id": "here:af:streetsection:SffD8pAbQ3m1wkzMVaDlxA:CgcIBCCCh9", "resultType": "houseNumber", "houseNumberType": "PA", "address": {"label": "700 Turnpike Rd, New Ipswich, NH 03048-3216, United States", "countryCode": "USA", "countryName": "United States", "stateCode": "NH", "state": "New Hampshire", "county": "Hillsborough", "city": "New Ipswich", "street": "Turnpike Rd", "postalCode": "03048-3216", "houseNumber": "700"}, "position": {"lat": 42.75859, "lng": -71.86321}, "access": [{"lat": 42.75852, "lng": -71.86347}], "distance": 32, "mapView": {"west": -71.90508, "south": 42.75013, "east": -71.82752, "north": 42.78103}}]}

 

Thanks so much!

It appears like the _json_response from HERE is a bit different from OSM, that’s why the JSON Query isn’t working. Taking a look at the HERE JSON response, a couple things to note here is that “address” is held inside “items”, which means that we need to reference “items” in our JSON Query as well.

However, an important note here is that “items” is a JSON array, so need to dive into JSON a bit more to work with this response. To extract “street” for example, we need to use the following JSON Query: json["items"][0]["address"]["street"]. This JSON Query can be easily translated for the other target attributes, just replace the [“street”] part with another attribute.

Summary: the _json_response from HERE holds objects such as “address” within an array called “items”. So, we need to reference “items”, specifically the first element, using 0 as the index. This can be a bit confusing, but HERE seems to wrap their results in an array, even if there’s only one result.

 

Ugh, probably would not have caught that!  Thank you so much for your help.


THIS THREAD IS WORTH ADDING for instruction, examples, Geocoding.  After learning how to effectively utilize this, it has been a game changer.


THIS THREAD IS WORTH ADDING for instruction, examples, Geocoding.  After learning how to effectively utilize this, it has been a game changer.

I’m glad I could help with your geocoding workflow!

Thank you for the suggestion as well, I have passed that message along to my team. It would definitely be useful to update our main geocoding article to cover working with raw JSON responses and point to resources for working with JSON.