Skip to main content
  • Home
  • Forums
  • FME Form
  • Transformers
  • I’m trying to export a point dataset with attachments from an Esri (10.8.1) Portal to an Enterprise database. Using FME 2023.1
Solved

I’m trying to export a point dataset with attachments from an Esri (10.8.1) Portal to an Enterprise database. Using FME 2023.1

cstewartfortsas
Contributor
Forum|alt.badge.img+2

Since it almost exactly matches what I’m trying to do, I started with the workspace posted by @trantatsafe here https://community.safe.com/s/question/0D54Q000080hPh1SAE/agol-attachments-what-are-you-doing-with-the-attachments-after-youve-read-them

 

Problem 1)

There are 3693 attachments in the dataset.  When attempting to put them through this workspace, only a suspiciously even 2000 come out of the ListExploder. The others fail with the message MISSING_PARAMETER_LIST.

 

Since they exist (visible from Esri), they must have attributes, don't they? Or am I missing something here?

 

I hate to muddy the waters but there is also second issue:

 

Problem 2)

The 2000 records that do pass through do not link back to the points. I initially thought this was due to the reader bug https://community.safe.com/s/question/0D54Q00009PK7llSAD/read-arcgis-online-attachments-and-maintain-globalids-for-both-features-and-attachments?t=1703270433321 but I applied the GUID Generator solution without any success.

 

There are 200 records in the _ATTACH table, and as far as I can see, they are correct but Esri doesn't seem to think there is a link. What am I missing here?

 

Thanks

 

Best answer by churchfraser

Hello,

 

With regard to problem 1, I think it's probable that the max record count for the entire feature service is determining (and limiting) the number of attachments that are returned. When the ListExploder gets to the feature where attachment 2001 should be, the feature is rejected because FME doesn't find any sort of list, hence the MISSING_PARAMETER_LIST. That's also why there are no attributes found.

To get around this, you could the ArcGIS REST API queryAttachments with the point globalID to return the URL of the images.

 

Thanks

 

 

View original
Did this help you find an answer to your question?

4 replies

churchfraser
Contributor
Forum|alt.badge.img+5
  • churchfraserContributor
  • Contributor
  • Best Answer
  • January 9, 2024

Hello,

 

With regard to problem 1, I think it's probable that the max record count for the entire feature service is determining (and limiting) the number of attachments that are returned. When the ListExploder gets to the feature where attachment 2001 should be, the feature is rejected because FME doesn't find any sort of list, hence the MISSING_PARAMETER_LIST. That's also why there are no attributes found.

To get around this, you could the ArcGIS REST API queryAttachments with the point globalID to return the URL of the images.

 

Thanks

 

 

cstewartfortsas
Contributor
Forum|alt.badge.img+2
churchfraser wrote:

Hello,

 

With regard to problem 1, I think it's probable that the max record count for the entire feature service is determining (and limiting) the number of attachments that are returned. When the ListExploder gets to the feature where attachment 2001 should be, the feature is rejected because FME doesn't find any sort of list, hence the MISSING_PARAMETER_LIST. That's also why there are no attributes found.

To get around this, you could the ArcGIS REST API queryAttachments with the point globalID to return the URL of the images.

 

Thanks

 

 

Thanks @churchfraser​. I did a quick test on this by using two readers to query different records and you were right.

 

I'm not familiar with the the queryAttachments so I'll have to read up on that. Are there any particular resources you'd recommend?

churchfraser
Contributor
Forum|alt.badge.img+5
  • churchfraserContributor
  • Contributor
  • January 9, 2024
cstewartfortsas wrote:

Thanks @churchfraser​. I did a quick test on this by using two readers to query different records and you were right.

 

I'm not familiar with the the queryAttachments so I'll have to read up on that. Are there any particular resources you'd recommend?

You'll first need to read up on the generateToken page which is required to then queryAttachments.

  1. Use an HTTPCaller to generate a token, then extract the token with a JSONFlattener or JSONFragmenter.
  2. Merge the token onto the points
  3. Query the feature service REST endpoint (ending with a number), e.g. https://gisweb.fortsask.ca/server/SERVICE_NAME/PARK_AMENITY/0/queryAttachments, where objectIds = objectid (from point)
  4. JSONFragmenter to find the image URL
  5. ImageFetcher @Value(URL)?token=@Value(token) to download the image

 

Testing this made me guess that your problem 2 might be related to the fact that you haven't actually downloaded the images in your original workspace, you've just identified that there are attachments. So when those 2,000 "attachments" are processed further on into the relationship table, there isn't actually an image to attach. I think to fix your original workspace in that aspect, you'd need to create the image from the blob (in the data attribute).

Instead, using the ImageFetcher as described above should actually download the image which can then be written into the related table with the correct relationship.

imageIf I can upload the workspace, that should help.

 

cstewartfortsas
Contributor
Forum|alt.badge.img+2
cstewartfortsas wrote:

Thanks @churchfraser​. I did a quick test on this by using two readers to query different records and you were right.

 

I'm not familiar with the the queryAttachments so I'll have to read up on that. Are there any particular resources you'd recommend?

Well, thank you once again @churchfraser​. Fixing problem 1 and passing all the attachment records though somehow allowed Esri to recognize that attachments existed but it still wouldn't read them.

 

Since I don't have the queryAttachments working yet, I used a BinaryDecoder and RasterReplacer to convert it back to .jpg and now everything works.

 

I will continue following up the queryAttachments to get rid of the multiple reader kludge, but at least I now have a functional workspace.

Reply


Helpful Members This Week

Community Stats

30,942
Posts
117,257
Replies
38,742
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.