Solved

Download AGOL Photos to local JPG

  • 5 September 2019
  • 16 replies
  • 84 views
benvk
Rank
Badge +7

Hey All,

I am trialing FME currently to see if this software is going to be able to simplify my life, and it is looking good so far!

One thing I am hoping to do is a very simple "Download all attachments from AGOL feature layer".

I have seen dozens of posts on this site about various processes involving AGOL features and generating reports etc with the attachments, but they are a little advanced for me at the moment.

I am hoping someone can provide me with a simple workspace that I provide my AGOL feature layer, then provide an output folder location, and FME will download all attachments for me.

I have had a few cracks at this but I cannot work out the attachment part, the attributes are all I can get at this stage.

Can someone show me a screenshot or something showing the simplest solution to carry out this task?

I got to this point but it doesn't write anything to the output folder:

 

 

icon

Best answer by debbiatsafe 6 September 2019, 22:38

View original

16 replies

itay
Rank
Badge +16

Hi @benvk,

Don't have access to AGOL, so can you post a image of the attributes from the the reader as they are in the DataInspector?

It looks like you are pointing the AttributeFileWriter to fetch an image from a list element.

debbiatsafe
Rank
Userlevel 3
Badge +17

Hi @benvk

Please ensure the Download Attachments parameter is enabled within the AGOL feature type and the ArcGIS Online format attributes relevant for attachments are exposed. It appears like you may have already done so but never hurts to double-check.

The arcgisonline_attachment{}.data attribute contains a binary representation of attachment data (as mentioned in the documentation linked above). Since the attribute is a binary representation, try changing the Target File Encoding in the AttributeFileWriter to Binary (fme_binary).

I also suspect part of the issue in your case is the attribute used in the Target File Name parameter does not contain the full file path (file path + name + file extension) for your output file.

An alternative workflow instead of using the AttributeFileWriter is to convert the BLOB within arcgisonline_attachment{}.data attribute to raster geometry (eg. using a RasterReplacer) and then write using the JPEG writer (or other raster format writers).

Either method should work to create attachments as files locally.

benvk
Rank
Badge +7

Hi @benvk,

Don't have access to AGOL, so can you post a image of the attributes from the the reader as they are in the DataInspector?

It looks like you are pointing the AttributeFileWriter to fetch an image from a list element.

Is this the information you are looking for?

Cheers

benvk
Rank
Badge +7

Hi @benvk

Please ensure the Download Attachments parameter is enabled within the AGOL feature type and the ArcGIS Online format attributes relevant for attachments are exposed. It appears like you may have already done so but never hurts to double-check.

The arcgisonline_attachment{}.data attribute contains a binary representation of attachment data (as mentioned in the documentation linked above). Since the attribute is a binary representation, try changing the Target File Encoding in the AttributeFileWriter to Binary (fme_binary).

I also suspect part of the issue in your case is the attribute used in the Target File Name parameter does not contain the full file path (file path + name + file extension) for your output file.

An alternative workflow instead of using the AttributeFileWriter is to convert the BLOB within arcgisonline_attachment{}.data attribute to raster geometry (eg. using a RasterReplacer) and then write using the JPEG writer (or other raster format writers).

Either method should work to create attachments as files locally.

Hi, Thanks for the reply. I have just tried one of the two methods you have provided (changing my target file encoding to be FME Binary), but I am still getting an error with this. I have attached the log to this reply. I will now try the raster replacer method and see if I have luck that way. Thanks again. 201909061038.txt

 

benvk
Rank
Badge +7

Hi @benvk

Please ensure the Download Attachments parameter is enabled within the AGOL feature type and the ArcGIS Online format attributes relevant for attachments are exposed. It appears like you may have already done so but never hurts to double-check.

The arcgisonline_attachment{}.data attribute contains a binary representation of attachment data (as mentioned in the documentation linked above). Since the attribute is a binary representation, try changing the Target File Encoding in the AttributeFileWriter to Binary (fme_binary).

I also suspect part of the issue in your case is the attribute used in the Target File Name parameter does not contain the full file path (file path + name + file extension) for your output file.

An alternative workflow instead of using the AttributeFileWriter is to convert the BLOB within arcgisonline_attachment{}.data attribute to raster geometry (eg. using a RasterReplacer) and then write using the JPEG writer (or other raster format writers).

Either method should work to create attachments as files locally.

Hi, the RasterReplacer trick has worked for me in this instance. Its great.

I do have a follow up question, what are all the *.wld files it is writing to the output folder? I was not expecting/wanting anything extra, but it is creating a wld file for every jpg file it creates.

Any ideas how to stop these?

itay
Rank
Badge +16

Hi, the RasterReplacer trick has worked for me in this instance. Its great.

I do have a follow up question, what are all the *.wld files it is writing to the output folder? I was not expecting/wanting anything extra, but it is creating a wld file for every jpg file it creates.

Any ideas how to stop these?

Hi @benvk,

I would leave the world files if I were you. If you really want to delete them you can use the systemcaller transformer with a DOS command or filter the features before writing to files.

debbiatsafe
Rank
Userlevel 3
Badge +17

Hi, Thanks for the reply. I have just tried one of the two methods you have provided (changing my target file encoding to be FME Binary), but I am still getting an error with this. I have attached the log to this reply. I will now try the raster replacer method and see if I have luck that way. Thanks again. 201909061038.txt

 

The JPEG writer only accepts raster geometry. So you will need to create a raster geometry for each feature destined for the JPEG writer, which the RasterReplacer does. The AttributeFileWriter does not alter the geometry of the input feature and hence the workspace fails.

If you prefer to use the AttributeFileWriter method, you should not use the JPEG writer after it. You simply need to delete the JPEG writer after the AttributeFileWriter in order for your attachments to be written successfully (notice the AttributeFileWriter in the screenshot of my initial comment does not contain have another transformer or writer after it).
debbiatsafe
Rank
Userlevel 3
Badge +17

Hi, the RasterReplacer trick has worked for me in this instance. Its great.

I do have a follow up question, what are all the *.wld files it is writing to the output folder? I was not expecting/wanting anything extra, but it is creating a wld file for every jpg file it creates.

Any ideas how to stop these?

*.wld files are Esri world file associated with an image and contain georeferencing information in ASCII file. The JPEG writer has a parameter which controls whether to generate world files or not. By default, this parameter is enabled. If you do not wish to create these world files, simply select 'No' for the Generate World File parameter on the JPEG writer feature type.

jiro
Rank
Badge +6

Hi, the RasterReplacer trick has worked for me in this instance. Its great.

I do have a follow up question, what are all the *.wld files it is writing to the output folder? I was not expecting/wanting anything extra, but it is creating a wld file for every jpg file it creates.

Any ideas how to stop these?

Hi @benvk

You did not show how you set up your RasterReplacer to achieve and succeed your goal. I'm trying to replicate your workflow in FMEDesktop but i'm getting the following error message when running my workspace.

 

The only issue I suspect, the Blob Attribute seems to be required ans I cannot see it in my input file (feature layer).

Any ideas?

Kind regards,

 

workspace 

image imageRasterRepalcer 

Raster file output

debbiatsafe
Rank
Userlevel 3
Badge +17

Hi, the RasterReplacer trick has worked for me in this instance. Its great.

I do have a follow up question, what are all the *.wld files it is writing to the output folder? I was not expecting/wanting anything extra, but it is creating a wld file for every jpg file it creates.

Any ideas how to stop these?

Hello @jiro​ 

I would recommend creating your own Community question to raise visibility for your question in the future.

To answer your question, instead of the Blob Attribute parameter being set to _rasterBlob attribute which doesn't exist in your work, try setting this parameter value to the arcgisonline_attachment{0}.data attribute as it contains a binary representation of attachment data.

benvk
Rank
Badge +7

Hi, the RasterReplacer trick has worked for me in this instance. Its great.

I do have a follow up question, what are all the *.wld files it is writing to the output folder? I was not expecting/wanting anything extra, but it is creating a wld file for every jpg file it creates.

Any ideas how to stop these?

@jiro​ I have just taken a screenshot of the current settings, that work perfectly for me.

 

Very simple workspace, connect to AGOL feature layer, download the first image attached to each row in the table, and save to a defined local folder. If you work out how to download ALL images attached to each row, please do share with me! I sometimes have several photos against a single row, and those additional photos are not downloaded by this tool.

 

Reader Settings 

Raster Replacer Parameters 

Image Writer

jiro
Rank
Badge +6

workspace_agol_attachmentsraster_replacerattributefilewriterHi @debbiatsafe and @Benvk,

 

My apologies for the delay in responding to your email regarding the uploading of photos. I had a severe flu that kept me bedridden for 4 days, two weeks ago...

 

Back from work last Thursday, I had an urgent project to finalize, in short.

 

Thanks for the tip, I configured my workbench with the settings specified in your Workspace and it didn't work at first I still got the same error message. Besides, I attach the log file that I had saved.

 

But after several attempts, it finally worked, with the same parameters, I don't know by what magic!

 

For your question how to extract and download multiple photos per point, I tried it and I also, FME only downloads the first photo.

 

However, I know there is an ESRI procedure to batch extract photos stored in a hosted feature layer in ArcGIS Online. Here is the link to this resource here: https://support.esri.com/en/technical-article/000012232

 

Thank you very much!

 

jiro
Rank
Badge +6

Hello @debbiatsafe abd @Benkv,

 

My apologies for the delay in responding to your email regarding the uploading of photos. I had a severe flu that kept me bedridden for 4 days, two weeks ago...

 

Back from work last Thursday, I had an urgent project to finalize, in short.

 

Thanks for the tip, I configured my workbench with the settings specified in your Workspace and it didn't work at first I still got the same error message. Besides, I attach the log file that I had saved.

 

But after several attempts, it finally worked, with the same parameters, I don't know by what magic!

 

For your question how to extract and download multiple photos per point, I tried it and I also, FME only downloads the first photo.

 

However, I know there is an ESRI procedure to batch extract photos stored in a hosted feature layer in ArcGIS Online. Here is the link to this resource here: https://support.esri.com/en/technical-article/000012232

 

Thank you very much!

jiro
Rank
Badge +6

Here are some screen captures,

workspace_agol_attachmentsraster_replacerattributefilewriter

danminneyatsaf
Rank
Userlevel 2
Badge +10

Hello @debbiatsafe abd @Benkv,

 

My apologies for the delay in responding to your email regarding the uploading of photos. I had a severe flu that kept me bedridden for 4 days, two weeks ago...

 

Back from work last Thursday, I had an urgent project to finalize, in short.

 

Thanks for the tip, I configured my workbench with the settings specified in your Workspace and it didn't work at first I still got the same error message. Besides, I attach the log file that I had saved.

 

But after several attempts, it finally worked, with the same parameters, I don't know by what magic!

 

For your question how to extract and download multiple photos per point, I tried it and I also, FME only downloads the first photo.

 

However, I know there is an ESRI procedure to batch extract photos stored in a hosted feature layer in ArcGIS Online. Here is the link to this resource here: https://support.esri.com/en/technical-article/000012232

 

Thank you very much!

@jiro​ You could use the ListExploder before the RasterReplacer to explode those features that have two or more attachments into their own separate rows. You can configure the ListExploder as follows:

image 

Now instead of just having our arcgisonline_attachment{} attributes contained within one row, they have been split out into multiple rows (depending on how many elements there are in the list. The _element_index attribute indicates which element in the arcgisonline_attachment{} list each row comes from.

image 

You will also need to make sure that you provide a unique name for each of the features in the Dynamic JPEG Writer in your workspace so that each unique image gets output.

jiro
Rank
Badge +6

@jiro​ You could use the ListExploder before the RasterReplacer to explode those features that have two or more attachments into their own separate rows. You can configure the ListExploder as follows:

image 

Now instead of just having our arcgisonline_attachment{} attributes contained within one row, they have been split out into multiple rows (depending on how many elements there are in the list. The _element_index attribute indicates which element in the arcgisonline_attachment{} list each row comes from.

image 

You will also need to make sure that you provide a unique name for each of the features in the Dynamic JPEG Writer in your workspace so that each unique image gets output.

@danminneyatsafe (Safer)

Thanks for this tip. I will try as soon as I have a free moment.

Have a good day!

 

Regards,

Reply


Community Stats

28,669
Posts
109,359
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.