+2hello
I'm using a HTTP caller with the request url https://opendata.cbs.nl/ODataApi/odata/84468NED/TypedDataSet
Unfortunalely there is al limit on download:
The given query is not allowed on this table. Please redefine your query so that it returns less than 10000 records.' from ...
How can I get the total data from this site?
And afterwards do I have to merge the data to a complete file?
Do you have an example workspace?
thanks for your support
You will have to look at the API documentation on how it implements paging (sometimes also called pagination), meaning request data in increment blocks.
Here's one way of implementing it in FME: https://knowledge.safe.com/questions/51270/loop-on-httpcaller.html
+18Hoi @perry,
The issue with pagination is that you need to know how many requests you should make and that it can overload the service.
Quering the service for the total number of features, using $count=true doenst work, unfortunately.
Luckily you can get the data using the ODatafeed and not the OdataApi, this way you can download the file and read it into the workspace or save it into an attribute.
Hope this helps,
Itay
+2Yes,
this works ok, thanks all.
@ all Is it possible to limit the URL with a parameter such as Municipaly so I don't select the full data?
@Itay: I suppose I have to use XML fragmenter to read the data.
The I filled Elements to match (select all), Options: enable flattening and attributes to Expose.
The output I get however I do not understand because it seems to have no common item between the separete records.
Can you give me a clue how to combine these records grouped by municipaly?
Do I have to make a list or do I have to use a different transformer?
Do you have an example preferable with this ODataFeed?
Manu thanks,
Perry
+18Yes,
this works ok, thanks all.
@ all Is it possible to limit the URL with a parameter such as Municipaly so I don't select the full data?
@Itay: I suppose I have to use XML fragmenter to read the data.
The I filled Elements to match (select all), Options: enable flattening and attributes to Expose.
The output I get however I do not understand because it seems to have no common item between the separete records.
Can you give me a clue how to combine these records grouped by municipaly?
Do I have to make a list or do I have to use a different transformer?
Do you have an example preferable with this ODataFeed?
Manu thanks,
Perry
Hi @perry,
You can use a FeatureReader (XML format) with the following elements to match: feed/entry/content/properties.
This will provide you with features that you can group by Gemeentenaam_1 attribute.
Unfortunately filtering of the request is allowed on a limited number of attributes, see for more information (https://www.cbs.nl/nl-nl/onze-diensten/open-data/website-open-data ) and the feed is limited for 10000 features per request So that means you probably need more that one request to get all the data.cbs-open-data-services.pdf
Succes!
Itay
