Question

HTTPCALLER: Getting all records from CSW or looping.

  • 19 February 2021
  • 2 replies
  • 15 views
J
Badge +1

Hi! I'm using the HTTPCaller to get records from CSW. 

I'm using POST as HTTP Method and posting:

Request URL: https://www.geonorge.no/geonetwork/srv/nor/csw-dataset

<?xml version="1.0"?>
<csw:GetRecords xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" service="CSW" version="2.0.2"
    resultType="results" outputSchema="csw:IsoRecord" startPosition="1" maxRecords="1000"><csw:Query typeNames="gmd:MD_Metadata"><!--
            <csw:ElementSetName>brief</csw:ElementSetName><csw:ElementSetName>summary</csw:ElementSetName><csw:ElementSetName>full</csw:ElementSetName>
        --><csw:Constraint version="1.1.0"><Filter xmlns="http://www.opengis.net/ogc" xmlns:gml="http://www.opengis.net/gml"/></csw:Constraint></csw:Query>
</csw:GetRecords>

The thing is that I want maxRecords to be 10000 to get all records. Max records in the database is between 7000-8000 but I want to build this for expansion.

When I'm setting maxRecords to over 2000 records, the translation fails. Is this due to connection timeout? How can I solve this?

 

As I see, it can be solved with one of these:

  • Editing parameters in FME to get all records
  • Using a loop to change start position and maxRecords

2 replies

nielsgerrits
Rank
Userlevel 6
Badge +32

You can change the timeout in the httpcaller. But if this doesn't work then indeed create smaller requests.

 

No WCS experience but some with WFS.

 

My best practice was not to loop, but to query for the total number of results, calculate the number of iterations, use a cloner to prepare this number of requests and run these to a second httpcaller.

ebygomm
Rank
Userlevel 1
Badge +21

Exactly as @nielsgerrits​  describes, my preference is to make a call if possible to find out how many records are in total and then make requests in chunks altering the start position rather than looping. Although if you have no way of finding out the total then looping is likely to be the way forward

Capture

Reply


Community Stats

28,688
Posts
109,427
Replies
36,543
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.