Skip to main content

Hi

I’m using oData web service url to do http request and extract data.

The request fetches every 5000 requests, I want to loop until ‘oData.nextlink’ attribute in the data is nil\empty.

Its frustratingly taking so long to resolve this issue !

FME doesn’t allow feature holder\sampler in loop, and there is no proper help\round about method.

And can’t see \ debug from custom transformer fmx file .. so, how to resolve ?

Attached are my ETLs, (main ETL & custom) (2023.0 version) for reference, can anyone please help identify issue and advise for fix Please ?

Thanks 

 

It’s a little tricky to debug without the data (the link returns a 404 for me).

Looping transformers are annoyingly tricky to work with.

Your best bet is probably to use a JSONExtractor to extract the next-link string from the response directly and then output the whole initial response from the looping transformer, then you can work with your features outside of the loop. 

You also need to be careful when testing for “odata_nextLink has a value” because this might still contain the string from the last loop. Something which works for me is to remove the attribute as soon as I don’t need it anymore - in your case it would be after the HTTPCaller_4

Another thing I noticed is that you have your odata_nextLink attribute getting looped back to the start but each time it’s using the “API_Query” attribute to make the call. You might want to think about reanaming odata_nextLink to API_Query or somethign like that to make sure your next link is actaully getting used to make another call to the service. 


Thanks 

but I have updated the odata_nextLink attribute in API_Query & set null if odata_nextLink is not available..but still the same issue.

At the end of the last request (remaining 5k records) the next link will not be available.

Pls see the attached fmx custom transformer.

 


It seems there is an issue with the uploaded file - I am unable to download it


 

can you please try the new zip file again ? lets know if any further issues


Reply