Can I make the WMTS Reader stream rather than download tiles?

Hello @rclarkenz​ , thanks for posting to the FME Community! I think it would be beneficial to know how you're asking for the WMTS. Would you be able to share a sample workspace and the WMTS service you're attempting to read from? Thanks, Kailin.

Hi @Kailinasafe

I'm just working with a blank workspace and trying to add a reader. Here is a screenshot of the reader parameters.  

This is the WMTS URL (i have retracted my token)

https://data.linz.govt.nz/services;key=<mykey>/wmts/1.0.0/layer/51769/WMTSCapabilities.xml

I have tried without a Selection Type, with a bounding box, and even a single tile. This is a sample of the log with no selection type, and a lower resolution:

Starting translation...
...
Current FME version: FME 2021.2.2.0 (20220106 - Build 21806 - WIN64)
...
FME Configuration: FME_BASE is 'no'
Operating System: Microsoft Windows 10 64-bit  (Build 19042)
FME Platform: WIN64
System Status: 88.70 GB of disk space available in the FME temporary folder (C:\FME_TEMP\wbrun_1649714250119_19564\fmetmp_5)
System Status: 31.85 GB of physical memory available
System Status: 127.40 GB of virtual memory available
START - ProcessID: 18524, peak process memory usage: 43912 kB, current process memory usage: 43912 kB
Creating reader for format: OGC WMTS (Web Map Tile Service)
Trying to find a DYNAMIC plugin for reader named `WMTS'
Loaded module 'WMTS' from file 'C:\Program Files\FME\plugins/WMTS.dll'
FME API version of module 'WMTS' matches current internal version (3.8 20210302)
<WMTS> Attempting to download a file from: 'https://data.linz.govt.nz/services;key=<mykey>/wmts/1.0.0/layer/51769/WMTSCapabilities.xml'
<WMTS> Selected Layer: 'layer-51769'
<WMTS> A default bounding box has been found on the selected layer with values Min X:'174.150364', Max X:'175.594043', Min Y:'-37.395588', Max Y:'-35.832361', and Coordinate System:'LL84'. This can be overridden by using any of the Tile Selection Options.
FME Configuration: Using FME Reprojection Engine
CS-MAP Reprojector: Transformation will be automatically selected for 'LL84' -> 'EPSG:3857', and is not guaranteed to remain the same in future releases of FME
Reprojector: Using transformation `WGS84_SPHERICAL_to_WGS84_FME,Inverse(Null)' when reprojecting from LL84 to EPSG:3857
<WMTS> Attempting to download a file from: 'https://tiles-a.koordinates.com/services;key=<mykey>/tiles/v4/layer=51769,style=auto/EPSG:3857/5/31/19.png'
[2]: Starting HTTP download from 'tiles-a.koordinates.com'. Expected transfer size: 75 bytes
[2]: HTTP Download: received 75 bytes
[2]: HTTP transfer summary (tiles-a.koordinates.com): status code: 403, download size: 75 bytes, DNS lookup time: 81 ms, total transfer time: 153 ms
<WMTS> WMTS attempted to download a tile that was not avaliable on the server, only valid tiles from the server will be read. If no features are read, try changing the tile matrix (resolution)
Merged 0 schema features read from 1 datasets into 0 resulting feature types
Opened mapping File C:\FME_TEMP\FME_1649725554842_19564.fmw for output
Mapping File Generation was SUCCESSFUL
Mapping File Generation was SUCCESSFUL
FME Session Duration: 3.2 seconds. (CPU: 0.3s user, 0.2s system)
END - ProcessID: 18524, peak process memory usage: 85292 kB, current process memory usage: 83920 kB

All my attempts have had a similar result and the reader goes into Navigator, but is unable to be dragged over into the workspace (I'm assuming because it hasn't been able to access any data).

I'm facing a similar issue when using WMTS as a background map in Visual Preview. ideally I'd want it to only download the tiles relevant for the area / zoom level currently visible, but I can't seem to get it to work. Any news on this @kailinatsafe​ ?

Hello @Hans van der Maarel​ , yes, this issue has been filed as FMEENGINE-72954. Apologies for not updating the thread! Would you be able to share the WMTS service you're experiencing issues with? If not, no worries! Best, Kailin.

HI @kailinatsafe​ ,

I am also experiencing time out issues with the following WMTS service:

https://geodata.nationaalgeoregister.nl/tiles/service/wmts?request=GetCapabilities&service=WMTS

Trying to minimise the request with the bounding box tile selection doesnt affect the number of request doen. Looks like FME is still trying to request all tiles.

Hello @itay​ , thanks for sharing the URL! May I ask what version of FME Workbench you're using? What resolution did you specify? I recieved a timeout while reading in 2021.0 (b21262), but reading in 2021.2 and 2022.1 was successful. These are my parameters:

The WMTS reader will download all the tiles of a specific resolution. If you do not want all the tiles, I suggest using the Tile Selection Options, which is what you're already doing!

Also, I noticed some of the icons in your screenshot are very large. If you want to get rid of the weird scaling issues, see this thread.

On a more general note... I believe the fix for FMEENGINE-72954 can now be found in FME 2022.1 b22588 and onwards (See Downloads Page).Unfortunately, we're still having issues reading @rclarkenz​ WMTS service, so we've now filed this seperately as: FMEENGINE-73550. Hope this helps! Kailin.

Hi @kailinatsafe​ ,

Thanks for the tip about the scaling issues!

I need to use the highest resolution (14) and also the client is using FME  Build 21812 and I am using FME build 21797.

I will test with FME build 22346 and see if the issue is resolved and if that is the case advise the client to upgrade.

Itay

Hello @itay​ , sorry just want to make sure you're testing in the correct build. The fix for FMEENGINE-72954 (which i think could help in this case) is found in the 2022.1 beta. So you'd likely want to test Build 22601, not 223346. Let me know if you're still having issues after that. Hope this helps, Kailin.

Hi @kailinatsafe​ ,

I have used the same workspace with FME 2022.1 build 22618 but unfortunatly this is not an improvement since the workspace still queries for all the tiles and not just the ones in the area of interest. I dont know if that was the fix in FMEENGINE-72954, but anyway I dont see much of an improvement.

So for a small area the workspace runs for 35 minutes while the tiles are read in the first 2 minutes. Seems like there is room for improvement when reading WMTS services....

Hello @itay​, hmm, when I try reading your WMTS with a bounding box, I see 323 tiles (resolution 14) read in 7.2 seconds. Perhaps my bbox is smaller than yours? Would you be able to share a workspace so I can try reproducing this on my end? If I can reperoduce, I can look into filing another ticket with development. If you're not able to share your workspace here, please submit a support case! Thank you for bringing this to our attention! Happy to help, Kailin.

Hi @kailinatsafe​ ,

I have had a good look at my workspace after your remarks and I have corrected an error that created this issue.

The workspace completes now in 31 sec and not 35 min.

Thank you for your assistence.

Itay

Hello @itay​, that's great news! Thank you for updating the thread! I did manage to look at your workspace, I saw the behaviour you initially described. For some reason, I couldn't properly run the .fmw, but when I recreated the use case in a fresh workspace, everything seemed to work OK.

I had luck with lowering the value set for 'Number of Concurrent Tiles to Download' and entering values in the 'Generating Output Ports' menu (which appears after closing FeatureReader parameter menu):

I chose to hardcode everything, opposed to using user_parameters. Not sure if this would change much however. Let me know if you start experiencing any other issues. Very glad you figured it out, Kailin!