Question

Best way to retrieve raster values from ESRI REST Image Service?

  • 25 February 2020
  • 4 replies
  • 64 views
B
Badge +3

Best method I've come up with below? Looking for any other experienced REST Raster Image Service users to suggest any better solutions on how to get back 1,000s of raster values in an FME Workflow!

 

Use case it to query the Surface Levels of anywhere between 1,000 to 1,000,000 geometric points on a REST ESRI Web Image Service that is a 1 metre DEM raster.

The Server does not allow an HTTP EXPORT request of a JPG/PNG image more than 500x500 pixels, so that kind of rules out just pulling it down with HTTPCaller as a single Raster image and then interrogating that as a Workspace Feature.

I'm a total novice at using an ESRI REST Image Service, but the "best" method I can so far come up is using the IDENTIFY operation:

https://developers.arcgis.com/rest/services-reference/identify-image-service-.htm

Each Operation requires the request to have a single pair of X, Y values for a Point ie. Only one raster value at a time can be called for in an HTTPCaller (and then process the resulting JSON string to pull out the Raster Surface Level value).

So, say 1,000 points would need HTTPCaller to be triggered 1,000 times for the 1,000 sets of Point XYs.

I'm concerned, however that this will lead to QoS / Security measures on either end of the internet/LAN networks that might think my computer is trying to do some sort of Denial of Service attack by FME shooting rapid fire HTTP calls to the same internet domain.....but never having done this before, it might work fine, although chances are it will also not be a very high performance solution either.

Any other suggestions?

4 replies

gisbradokla
Rank
Badge +10

@bwn​ did you ever find a solution to this? You do realize that you have to use the (for export) version of the service right?

I found another question https://community.safe.com/s/question/0D54Q00008BzQlQSAV/how-to-pull-a-raster-from-an-esri-basemap-httpsservicesarcgisonlinecomarcgisrestservicesworldtopomap-to-use-within-fme-workbench

He has a sample workbench. but I cannot understand how he is stepping through all the bounding boxes.

B
Badge +3

@Brad Nesom​  Yes I did solve this. I ended up using the ESRI REST service to mosaic my own single LZW compressed GeoTiff since we do run a lot of DEM queries. It only takes ~an hour for the grid boxes that overlay our service territory to retrieve and mosaic the DEM images together. I have also written an FME workspace to alternately use the ESRI Rest "Identify" DEM value fetch command, but would post that solution separately if you are interested. This workflow gets used to cover off the situation where sometimes Points I am querying the elevation for may sit outside the boundaries of our built GeoTiff, so the fallback for those is to query their Z values directly from the online ESRI Rest service.

 

Basic process was:

1. Build a grid of 2,500 x 2,500 metre polygon boxes to cover the area needed for the DEM. Experimentation with the particular ESRI REST service using showed that it was configured in such a way that REST requests for images larger than this were denied, so instead had to break down the process to fetch images of no more than 2,500 x 2,500 pixels for the 1 metre DEM Grid that our State uses. I authored the grid boxes in ArcMap since our service area is not rectangular and so could tailor the mosaicked DEM image to be no larger than our service area.

 

2. In FME extract the Xmin, Xmax, Ymin and Ymax attributes of each of these Box features with BoundsExtractor and round these to the nearest 1 metre

 

3. With ImageFetcher , for each box feature, send a series of REST Image Fetch requests. The URL I used matches the ESRI REST API Image fetch format, and I had to consult with the online ESRI REST documentation to a large degree to figure out the correct parameters, but what worked for me was:

 

https://spatial-img.information.qld.gov.au/arcgis/rest/services/Elevation/QldDem/ImageServer/exportImage?bbox=@Value(_xmin)%2C@Value(_ymin)%2C@Value(_xmax)%2C@Value(_ymax)&bboxSR=28356&size=2500%2C2500&imageSR=28356&time=&format=tiff&pixelType=F32&noData=-9999&noDataInterpretation=esriNoDataMatchAny&interpolation=+RSP_BilinearInterpolation&compression=LZ77&compressionQuality=&bandIds=&mosaicRule=&renderingRule=&f=image

 

The "%2C" were this particular DEM host website's way of parameterising "," characters within the URL so this was a slight modification from the ESRI documentation on what URL format to use,

 

4. Mosaic them together in RasterMosaicker

 

5. Write this with a GeoTiff Writer. I set Create World File On, Compression Method = LZW, Create Tiled Files = Yes and set to 2,500 x 2,500 pixel tiles, and Generate Pyramids = On. The last two were because we not only use the DEM to process data, but as a background display in ArcMap and GIS.

 

capture 

gisbradokla
Rank
Badge +10

@Brad Nesom​  Yes I did solve this. I ended up using the ESRI REST service to mosaic my own single LZW compressed GeoTiff since we do run a lot of DEM queries. It only takes ~an hour for the grid boxes that overlay our service territory to retrieve and mosaic the DEM images together. I have also written an FME workspace to alternately use the ESRI Rest "Identify" DEM value fetch command, but would post that solution separately if you are interested. This workflow gets used to cover off the situation where sometimes Points I am querying the elevation for may sit outside the boundaries of our built GeoTiff, so the fallback for those is to query their Z values directly from the online ESRI Rest service.

 

Basic process was:

1. Build a grid of 2,500 x 2,500 metre polygon boxes to cover the area needed for the DEM. Experimentation with the particular ESRI REST service using showed that it was configured in such a way that REST requests for images larger than this were denied, so instead had to break down the process to fetch images of no more than 2,500 x 2,500 pixels for the 1 metre DEM Grid that our State uses. I authored the grid boxes in ArcMap since our service area is not rectangular and so could tailor the mosaicked DEM image to be no larger than our service area.

 

2. In FME extract the Xmin, Xmax, Ymin and Ymax attributes of each of these Box features with BoundsExtractor and round these to the nearest 1 metre

 

3. With ImageFetcher , for each box feature, send a series of REST Image Fetch requests. The URL I used matches the ESRI REST API Image fetch format, and I had to consult with the online ESRI REST documentation to a large degree to figure out the correct parameters, but what worked for me was:

 

https://spatial-img.information.qld.gov.au/arcgis/rest/services/Elevation/QldDem/ImageServer/exportImage?bbox=@Value(_xmin)%2C@Value(_ymin)%2C@Value(_xmax)%2C@Value(_ymax)&bboxSR=28356&size=2500%2C2500&imageSR=28356&time=&format=tiff&pixelType=F32&noData=-9999&noDataInterpretation=esriNoDataMatchAny&interpolation=+RSP_BilinearInterpolation&compression=LZ77&compressionQuality=&bandIds=&mosaicRule=&renderingRule=&f=image

 

The "%2C" were this particular DEM host website's way of parameterising "," characters within the URL so this was a slight modification from the ESRI documentation on what URL format to use,

 

4. Mosaic them together in RasterMosaicker

 

5. Write this with a GeoTiff Writer. I set Create World File On, Compression Method = LZW, Create Tiled Files = Yes and set to 2,500 x 2,500 pixel tiles, and Generate Pyramids = On. The last two were because we not only use the DEM to process data, but as a background display in ArcMap and GIS.

 

capture 

thanks @bwn​ I don't see where you are doing anything about authentication. All of the esri tilemapcached services require a token. I am really struggling to get the right parameters passed to get a return when doing that. Have you had any experience with that?

B
Badge +3

thanks @bwn​ I don't see where you are doing anything about authentication. All of the esri tilemapcached services require a token. I am really struggling to get the right parameters passed to get a return when doing that. Have you had any experience with that?

Not all ESRI REST services require authentication, and wasn't needed in my use case.

 

For example, if use the following URL to Fetch an Image in your browser, this will work with no authentication (and return a DEM 1 metre grid image 2,500 pixels x 2,500 pixels in TIF format). The parameters passed are as per the ESRI REST services help documentation being the bounding box coordinates (bbox=) , the bounding box coordinate system spatial reference (bboxSR=), the image size to return in pixels (size=), the coordinate system to project the image in (imageSR=), the image file format to generate (format=), the value to assign Pixels that have no DEM value (noData=), the TIF file compression (compression=) and whether or not to return the result as a formatted web page result or just the plain image file (f=)

https://spatial-img.information.qld.gov.au/arcgis/rest/services/Elevation/QldDem/ImageServer/exportImage?bbox=500000%2C7000000%2C502500%2C7002500&bboxSR=28356&size=2500%2C2500&imageSR=28356&time=&format=tiff&pixelType=F32&noData=-9999&noDataInterpretation=esriNoDataMatchAny&interpolation=+RSP_BilinearInterpolation&compression=LZ77&compressionQuality=&bandIds=&mosaicRule=&renderingRule=&f=image

 

A web browser was where I did most of my URL parameter testing before using this inside FME.

 

However, ImageFetcher does support different Authentication options: Basic, Digest, NTLM, Named Web Service, and Single Sign-On so I suspect this would cover most web service authentication types encountered?

Reply


Community Stats

28,731
Posts
109,530
Replies
36,565
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.