Problem WFS query

Hi @andreasb,

The same query worked OK for me in FME 2019. That is a large dataset, so you may be getting a timeout during the transfer. Please try increasing the value for the WFS writer - Parameters - Transfer Timeout Length from the default 90 seconds, and see if that fixes the problem for you.

Hey @daveatsafe

I have changed those parameters (5000 sec) but it still gets stuck.

Can you maybe show me your other settings from the wfs reader? So I can check if there are any differences.

Also I'm using FME 2018 but I don't think that could be the problem.

Thanks already for running this writer.

Hi @andreasb,

I just used the default settings in FME 2018.1:

I tried the service from my local computer and from an AWS instance - in both cases it read all the features in about 15 seconds.

You can also try using a browser to read the data, using the link:

https://remotesensing.agiv.be/gis/geoserver/wfs?SERVICE=WFS&VERSION=1.1.0&REQUEST=GetFeature&TYPENAME=Agiv%3ALiDAR_DHMV_II_LAZtiles&NAMESPACE=xmlns%28Agiv%3Dhttp%3A%2F%2Fmobile-mapping.agiv.be%29&MAXFEATURES=30000

If the browser fails as well, then it may be an issue with your network. If it works, then try adjusting the Connection Timeout in the reader.

Hi @daveatsafe

if i open my browser to read the data, i got this error.

This is also a screenshot of my featurereader to download all the LAS-files from the dataset.

Maybe here lies the problem?

Or could it just be a server sided problem that lies with the remotesensing agiv itself?

I'm still looking deeper into this problem, thanks for the help already.

Andreas

I think you can not read a WFS with a LAS reader?

Hey Niels,

I'm not using a LAS reader. I'm using a WFS to get the data from the site and later on I'm using a featurereader to get the actual .las files from the server.

This has worked when I ran this module as a test for small areas. But for this large area I'm getting an error which I am trying to solve.

Here you can see the module I'm using to get pointclouds inside certain contours.

My bad, I did not understand correctly.

@andreasb

Hi,

You could read the lidar tiles with your search envelope and then trigger the FeatureReader with tile location string (which you got from the WFS reader).

Rather then trying to spatialy query the laz tile sets.

like this

i limited it to sampled 3 for testing.

17.7 sec for 3 laz. Pretty fast service it is.

And the namespaceprefix error is caused when you stringconcatenate in the feature reader. This is why i made the stringconcatenation outside the Feature Reader in the attribute creator.

Maybe because then the special characters are mutilated by fme(?).. just geussing here..

@andreasb,@daveatsafe

It still crashed, until i changed the WFS version on the wfs reader to 2.0.0

..and it still crashes this time at request

whatever i try iget this error eventualy at seemingly at random(?):

HTTP transfer error: 'Failure when receiving data from the peer'

XML Parser error: 'Error in input dataset:'https://remotesensing.agiv.be:443/gis/geoserver/wfs?SERVICE=WFS&VERSION=2.0.0&REQUEST=GetFeature&TYPENAMES=Agiv%3ALiDAR_DHMV_II_LAZtiles&NAMESPACES=xmlns%28Agiv%2Chttp%3A%2F%2Fmobile-mapping.agiv.be%29&BBOX=40050.00000000547%2C202950.0000002645%2C48385.37780000517%2C206209.4442002587%2Curn%3Aogc%3Adef%3Acrs%3AEPSG%3A%3A31370&OUTPUTFORMAT=application%2Fgml%2Bxml%3B%20version%3D3.2&COUNT=30000' line:1 column:589673 message:input ended before all started tags were ended; last tag started is 'gml:posList''

The XML Module halted on error, see the logfile for details

Hey @gio and @daveatsafe

It indeed works for certain areas, as I already tested it for some smaller areas. But when i run this on the big extent, I do indeed also get this error which seems to be something with the xml?

I'm curious if this can be solved by changing some of the settings in the featurereader? or is this something that is server sided?

Also the error doesn't seem to be at random but at a certain tile (if you click your link and mine they both look identically the same, well the first and the last tile id are the same) but i can't figure out which one gives the actual problem as my knowledge of xml is not yet sufficient.

Andreas

Hi @andreasb

It is the combination of the WFS reader and the FeatureReader that is causing the problem. It doesn't seem to matter what reader you are using inside the FeatureReader. I will create a problem report for this and notify you as soon as it is fixed.

In the meantime, using two FeatureReaders instead seems to work - use the first for the WFS and the second for the LAZ.

Hey @daveatsafe,

Thanks for the answer, I will try your solution! Is there a reason why I don't get the search envelop option when i'm using the feature reader? is there a workaround?

edit: Can you maybe show me a screenshot of the solution so I can replicate it?

Andreas

Hey @daveatsafe,

I did use your solution, and it did work for a certain project using the 2 featurereaders ! But now I'm using the same module for another project and I get the exact same error I used to get before. In the meantime i did upgrade to FME 2019. Can this be the result of updating? or is there still something wrong with the WFS module?

Kind regards,

Andreas

Hi @andreasb,

Would you mind sharing the latest copy of the workspace with me? Or at least part of it containing both FeatureReaders?

@daveatsafe Here is a view of my module. If you need any more information just let me know. Thanks for helping me with this problem.

Hi @andreasb,

Please send a log file from the attempted run as well - this should show me the error that is occurring.

@daveatsafe

2019-07-30 21:22:19| 158.0| 0.0|ERROR |HTTP transfer error: 'Failure when receiving data from the peer'

2019-07-30 21:22:19| 158.0| 0.0|ERROR |XML Parser error: 'Error in input dataset:'https://remotesensing.agiv.be:443/gis/geoserver/wfs?SERVICE=WFS&VERSION=2.0.0&REQUEST=GetFeature&TYPENAMES=Agiv%3ALiDAR_DHMV_II_LAZtiles&NAMESPACES=xmlns%28Agiv%2Chttp%3A%2F%2Fmobile-mapping.agiv.be%29&BBOX=152400%2C171400%2C157427.2496%2C178985.291%2Curn%3Aogc%3Adef%3Acrs%3AEPSG%3A%3A31370&OUTPUTFORMAT=text%2Fxml%3B%20subtype%3Dgml%2F3.2&COUNT=1000000' line:1 column:825298 message:input ended before all started tags were ended; last tag started is 'gml:posList''

2019-07-30 21:22:19| 158.0| 0.0|ERROR |The XML Module halted on error, see the logfile for details

2019-07-30 21:22:19| 158.0| 0.0|ERROR |<WFS> An error occurred while reading a GetFeature response, to ignore the error and avoid losing the previously read features, set the WFS "Ignore GetFeature Error" parameter. Total features downloaded: '0'

2019-07-30 21:22:19| 158.1| 0.0|STATS |Stored 2 feature(s) to FME feature store file `E:\\FME\\FME_TEMP\\wb-cache-LAS file WMS-VivRoZ\\Main_FeatureReader_2 -1 16 fo 0 _lt_SCHEMA_gt_ 0 2d3f75905d569e30fc1c53e0502347a493fb70b5.ffsupdating'

this is the error from the log

Hi @andreasb,

I was not able to reproduce the problem you are seeing. I am attaching a very simple workspace that seems to read both the WFS and the LAZ ok.

WFS_LAZ.fmw