Question

How to interpret `_response_body` from http call as content?

  • 3 May 2023
  • 5 replies
  • 31 views

Badge +8

The Reading JSON tutorial demonstrates using Feature Reader to fetch and parse data from a bare url. However I need to fetch from a url that requires a token in the http header. I've used HTTPCaller transformer to do this successfully and I have json data in `_response_body`. However when I feed response body to FeatureReader it treats that as a *path* and not *content*. (It took me hours to figure out this what the error message meant).

 

How do I read the `_response_body` from the http call as *content*?

 

https://community.safe.com/s/article/json-reader-configuration


5 replies

Userlevel 6
Badge +31

I suspect the response body is JSON? When you want to get the token from it you can just use a JSONExtractor.

Badge +8

I suspect the response body is JSON? When you want to get the token from it you can just use a JSONExtractor.

Thanks Niels.

I can choose csv or json. I did discover through this thread that for CSV you need to write the content to disk first. I'll go that route if needed but it's noise in the workspace I'd rather avoid (6 transformers to achieve a single function).

 

When I try JSONExtractor I do bypass the name vs content issue I noted in opening, but the result is a single output of the entire json content instead of features.

httpcaller to json extractor: only 1 out feature instead of many

Userlevel 6
Badge +31

Thanks Niels.

I can choose csv or json. I did discover through this thread that for CSV you need to write the content to disk first. I'll go that route if needed but it's noise in the workspace I'd rather avoid (6 transformers to achieve a single function).

 

When I try JSONExtractor I do bypass the name vs content issue I noted in opening, but the result is a single output of the entire json content instead of features.

httpcaller to json extractor: only 1 out feature instead of many

Hard to be sure what you want or need without sample data, but it sounds you want to learn how to parse JSON. If you have multiple features in the response you can “explode” these into features using a JSONFragmenter.

Badge +8

Thanks Niels.

I can choose csv or json. I did discover through this thread that for CSV you need to write the content to disk first. I'll go that route if needed but it's noise in the workspace I'd rather avoid (6 transformers to achieve a single function).

 

When I try JSONExtractor I do bypass the name vs content issue I noted in opening, but the result is a single output of the entire json content instead of features.

httpcaller to json extractor: only 1 out feature instead of many

Oh that's fun! Adding JSONFragmenter after JSONExtrator exploded a 4.2MB json into >100GB of temp .ffs features before running out of diskspace and crashing. I'll try to manually look over the json and see if there's anything to prevent me from sharing a sample.

 

The FFS file [...]\\AppData\\Local\\Temp\\wb-cache-fetch-assets-MxyZKs\\Main_JSONFragmenter -1 16 fo 0 FRAGMENTS 0 b6a49780b1913f42ae6f62418f64a71e24ce396e_68.ffs failed the CRC check. It is corrupt, and cannot be read

 

Badge +8

Thanks Niels.

I can choose csv or json. I did discover through this thread that for CSV you need to write the content to disk first. I'll go that route if needed but it's noise in the workspace I'd rather avoid (6 transformers to achieve a single function).

 

When I try JSONExtractor I do bypass the name vs content issue I noted in opening, but the result is a single output of the entire json content instead of features.

httpcaller to json extractor: only 1 out feature instead of many

ok, here's the 1st 4 lines of the _response_body after HTTPCaller and before JSONExtractor: https://gist.github.com/maphew/efb83fa3f0bf835ae8b3fcf8253c84d6

Reply