how do I use scroll api in the GET method in Httpcaller?

Hi @zatom

Please - share us the Log file.

Thanks,

Danilo

Depending on the api, your response may contain a next element which contains the link to the next set of records, and you can retrieve them iteratively.

@danilo_fme

Starting translation...

FME 2019.0.0.0 (20190328 - Build 19238 - WIN64)

FME_HOME is 'C:\\Program Files\\FME\\'

FME Desktop ESRI Edition (floating)

Temporary License: 125 days left.

Machine host name is: xyz6004

START - ProcessID: 14760, peak process memory usage: 31904 kB, current process memory usage: 31904 kB

FME Configuration: Command line arguments are `C:\\Program Files\\FME\\fme.exe' `C:/Users/USER/Downloads\\wb-xlate-155914265111_13680' `LOG_STANDARDOUT' `YES' `LogCountServerName' `{e35af8f7-36bb-4870-8b72-2e4121f3db41}'

FME Configuration: Connection Storage: 'C:\\Users\\USER\\AppData\\Roaming\\Safe Software\\FME\\'

Shared folders for formats are : C:\\Program Files\\FME\\datasources;C:\\Users\\USER\\Documents\\FME\\Formats

Shared folders for transformers are : C:\\Users\\USER\\AppData\\Roaming\\Safe Software\\FME\\Packages\\19238-win64\\transformers;C:\\Program Files\\FME\\transformers

Shared folders for coordinate systems are : C:\\Users\\USER\\Documents\\FME\\CoordinateSystems

Shared folders for coordinate system exceptions are : C:\\Users\\USER\\Documents\\FME\\CoordinateSystemExceptions

Shared folders for coordinate system grid overrides are : C:\\Users\\USER\\Documents\\FME\\CoordinateSystemGridOverrides

Shared folders for CS-MAP transformation exceptions are : C:\\Users\\USER\\Documents\\FME\\CsmapTransformationExceptions

Shared folders for transformer categories are : C:\\Users\\USER\\Documents\\FME\\TransformerCategories

FME Configuration: Reader Keyword is `MULTI_READER'

FME Configuration: Writer Keyword is `NULL_DEST'

FME Configuration: Writer Group Definition Keyword is `NULL_DEST_DEF'

FME Configuration: Reader type is `MULTI_READER'

FME Configuration: Writer type is `NULL'

FME Configuration: No destination coordinate system set

FME Configuration: Current working folder is `C:\\Users\\USER\\Downloads'

FME Configuration: Temporary folder is `C:\\Users\\USER~1.GUR\\AppData\\Local\\Temp', set from environment variable `TEMP'

FME Configuration: FME_HOME is `C:\\Program Files\\FME\\'

FME Configuration: FME_BASE is 'no'

FME Configuration: FME_MF_DIR is 'C:\\Users\\USER\\Downloads/'

FME Configuration: FME_MF_NAME is 'Time_range_filter.fmw'

FME Configuration: FME_PRODUCT_NAME is 'FME(R) 2019.0.0.0'

System Status: 248.80 GB of disk space available in the FME temporary folder (C:\\Users\\USER~1.GUR\\AppData\\Local\\Temp)

System Status: 128.00 TB of virtual memory available

Operating System: Microsoft Windows 10 64-bit (Build 17763)

FME Platform: WIN64

Locale: en_US

Code Page: 1252 (ANSI - Latin I)

FME Configuration: Process limit is 15.84 GB of physical memory

FME Configuration: Start freeing memory when process usage exceeds 47.53 GB of virtual memory

FME Configuration: Stop freeing memory when process usage is below 35.65 GB of virtual memory

Creating writer for format: NULL (Nothing)

Trying to find a DYNAMIC plugin for writer named `NULL'

Loaded module 'NULL' from file 'C:\\Program Files\\FME\\plugins/NULL.dll'

FME API version of module 'NULL' matches current internal version (3.8 20181030)

Creating reader for format:

MULTI_READER(MULTI_READER): Will fail with first member reader failure

Using Multi Reader with keyword `MULTI_READER' to read multiple datasets

FME API version of module 'NULL' matches current internal version (3.8 20181030)

FME API version of module 'NULL' matches current internal version (3.8 20181030)

Loaded module 'Geometry_func' from file 'C:\\Program Files\\FME\\plugins/Geometry_func.dll'

FME API version of module 'Geometry_func' matches current internal version (3.8 20181030)

Loaded module 'LogCount_func' from file 'C:\\Program Files\\FME\\plugins/LogCount_func.dll'

FME API version of module 'LogCount_func' matches current internal version (3.8 20181030)

Loaded module 'HTTPFactory' from file 'C:\\Program Files\\FME\\plugins/HTTPFactory.dll'

FME API version of module 'HTTPFactory' matches current internal version (3.8 20181030)

Loaded module 'JSONQueryFactory' from file 'C:\\Program Files\\FME\\plugins/JSONQueryFactory.dll'

FME API version of module 'JSONQueryFactory' matches current internal version (3.8 20181030)

Loaded module 'InspectorFactory' from file 'C:\\Program Files\\FME\\plugins/InspectorFactory.dll'

FME API version of module 'InspectorFactory' matches current internal version (3.8 20181030)

Emptying factory pipeline

HTTPCaller_2 (HTTPFactory): HTTP/FTP Transfer: Downloading resource 'http://localhost/index/_search?scroll=50000&q=FILING_DATE:gte%20now-10d&sort=FILING_DATE:desc&size=100' to internal memory buffer

HTTPCaller_2 (HTTPFactory): HTTP transfer summary - status code: 400, download size: '173 bytes', DNS lookup time: '1e-6 seconds', total transfer time: '0.422 seconds', url: 'http://localhost/index/_search?scroll=50000&q=FILING_DATE:gte%20now-10d&sort=FILING_DATE:desc&size=100'

HTTPCaller_2 (HTTPFactory): Received HTTP response header: 'HTTP/1.1 400 Bad Request' from 'http://localhost/index/_search?scroll=50000&q=FILING_DATE:gte%20now-10d&sort=FILING_DATE:desc&size=100'

Storing feature(s) to FME feature store file `C:\\Users\\USER~1.GUR\\AppData\\Local\\Temp\\wb-cache-Time_range_filter-qKvOXO\\Main_HTTPCaller_2 -1 3 fo 1 _lt_REJECTED_gt_ 0 d776f40359e236fde79f22273ec7f46bc9af9a37.ffsupdating'

Creator_2_XML_Creator (CreationFactory): Created 1 features

Creator_2_Cloner (TeeFactory): Cloned 1 input feature(s) into 1 output feature(s)

Creator_2_CREATED Feature Counter -1 6 (TeeFactory): Cloned 1 input feature(s) into 1 output feature(s)

Creator_2_CREATED Brancher -1 6 (BranchingFactory): Branched 1 input feature -- 1 feature routed to the target factory, and 0 features routed to the fallback factory.

_CREATOR_BRANCH_TARGET (TeeFactory): Cloned 1 input feature(s) into 1 output feature(s)

HTTPCaller_2_Output Feature Counter -1 7 (TeeFactory): Cloned 0 input feature(s) into 0 output feature(s)

HTTPCaller_2_<REJECTED> Feature Counter -1 2147680259 (TeeFactory): Cloned 1 input feature(s) into 0 output feature(s)

JSONFragmenter_2_FRAGMENTS Feature Counter -1 8 (TeeFactory): Cloned 0 input feature(s) into 0 output feature(s)

JSONFragmenter_2_<REJECTED> Feature Counter -1 2147680261 (TeeFactory): Cloned 0 input feature(s) into 0 output feature(s)

Stored 1 feature(s) to FME feature store file `C:\\Users\\USER~1.GUR\\AppData\\Local\\Temp\\wb-cache-Time_range_filter-qKvOXO\\Main_HTTPCaller_2 -1 3 fo 1 _lt_REJECTED_gt_ 0 d776f40359e236fde79f22273ec7f46bc9af9a37.ffsupdating'

Saving spatial index into file 'C:\\Users\\USER~1.GUR\\AppData\\Local\\Temp\\wb-cache-Time_range_filter-qKvOXO\\Main_HTTPCaller_2 -1 3 fo 1 _lt_REJECTED_gt_ 0 d776f40359e236fde79f22273ec7f46bc9af9a37.fsi'

Finished saving spatial index into file 'C:\\Users\\USER~1.GUR\\AppData\\Local\\Temp\\wb-cache-Time_range_filter-qKvOXO\\Main_HTTPCaller_2 -1 3 fo 1 _lt_REJECTED_gt_ 0 d776f40359e236fde79f22273ec7f46bc9af9a37.fsi'

Destination Feature Type Routing Correlator (RoutingFactory): Tested 0 input feature(s), wrote 0 output feature(s): 0 matched merge filters, 0 were routed to output, 0 could not be routed.

Final Output Nuker (TeeFactory): Cloned 0 input feature(s) into 0 output feature(s)

VisualizerProcessor (CreationFactory): Created 1 features

VisualizerCleaner (TeeFactory): Cloned 1 input feature(s) into 0 output feature(s)

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Features Read Summary

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

==============================================================================

Total Features Read 0

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Features Written Summary

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

==============================================================================

Total Features Written 0

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Translation was SUCCESSFUL with 0 warning(s) (0 feature(s) output)

FME Session Duration: 0.7 seconds. (CPU: 0.1s user, 0.1s system)

END - ProcessID: 14760, peak process memory usage: 47792 kB, current process memory usage: 47792 kB

Translation was SUCCESSFUL

I agree with @jdh, you should look into what's often called "paging" using a looping custom transformer. You can find more information about looping custom transformers in the training material: https://safe-software.gitbooks.io/fme-desktop-advanced-training-2018/content/DesktopAdvanced5CustomTransformers/5.11.CustomTransformerLoops.html

Also have a look at the following post, in particular the answer from @jdh:

https://knowledge.safe.com/questions/51270/loop-on-httpcaller.html