Skip to main content

Hi,

I use a CSV list with standard ArcGIS server queries + HTTP Caller to get geojsons. a CSV list contains also layer names etc. I use the File option when saving a response because I need a separate geojson file for every query. Everything goes great, however response files look like "http_download_153028689823_5840.json" which is hard to deal with.

Actually I can create a folder with a layer name but I would like to have an option to save a response file itself with a meaningful name and keep granular file structure.

Ideally I see it as follows: I run a query within my CSV then get some attributes from the same CSV and supply a response filename from HTTP Caller with these attributes on the fly.

There is a way to deal with a new Workspace and rename all the files using filecopy (upd - today I have finally managed with DatabaseJoiner for this purpose), but I am interested in a more integrated solution. Thank you in advance!

Evgeny

Hi @eshirinyan,

If you set the HTTPCaller to save the response body to an attribute, you can follow it with an AttributeFileWriter to write the contents of that attribute to a file. This transformer will allow you to set the output filename from an attribute, and will write every feature to its own file.


Thank you, @DaveAtSafe!

Just checked, it works fine! Hope there is no limitations for large feature sets too.


Hi @eshirinyan, I think you can set the Output Filename parameter to save the downloaded file into your desired path directly, as in:


@takashi Thanks a lot! This is the method I was searching for. FME is very flexible


@takashi 

 

I have a problem similar to one you've addressed previously.

I am retrieving text files/CSVs from a remote server of a consulting company using HTTPCaller. My goal is to create a copy of each file in my local database to provide access for other data engineers in my company and to process the data further for various purposes.

Currently, I achieve this by saving the files to my local disk, then using a FeatureReader/Writer to write them to my local database, and a FeatureDeleter to delete the downloaded files immediately after they are written to the database.

However, this method is not optimal and poses security risks since some of the data is sensitive. Therefore, I want to directly write the incoming text file data to my local database with minor attribute renaming.

How can I connect my HTTPCaller, which downloads to a file, directly to my writer?


As this is a 6 year old question, the preferred route is to create a new one.

One way to solve what you need is to use the TempPathnameCreator to create a temp folder where the files from the httpcaller can be stored and read. This takes away the need to clean up because the files are written to temp files.

Another way is to set the HTTPCaller to save response body to Attribute. You can then use that to send the _response_body to a Text File FeatureWriter. _response_body will nee to be renamed to text_line_data.


Reply