http://localhost/index_name/_search?q=FILING_DATE:gte now-1d&size=100 goes into the Request URL in Httpcaller parameter and it works and only gives me 100 results. But, size has the limitation of only 10000. So, I want to use scroll api to get all the results. How do I do that? I tried putting scroll=500000 after search , at the end of the query. Still doesnot work
Question
how do I use scroll api in the GET method in Httpcaller?
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+38- Contributor
Depending on the api, your response may contain a next element which contains the link to the next set of records, and you can retrieve them iteratively.
Most apis also have a start parameter.
@danilo_fme
Starting translation...
FME 2019.0.0.0 (20190328 - Build 19238 - WIN64)
FME_HOME is 'C:\\Program Files\\FME\\'
FME Desktop ESRI Edition (floating)
Temporary License: 125 days left.
Machine host name is: xyz6004
START - ProcessID: 14760, peak process memory usage: 31904 kB, current process memory usage: 31904 kB
FME Configuration: Command line arguments are `C:\\Program Files\\FME\\fme.exe' `C:/Users/USER/Downloads\\wb-xlate-155914265111_13680' `LOG_STANDARDOUT' `YES' `LogCountServerName' `{e35af8f7-36bb-4870-8b72-2e4121f3db41}'
FME Configuration: Connection Storage: 'C:\\Users\\USER\\AppData\\Roaming\\Safe Software\\FME\\'
Shared folders for formats are : C:\\Program Files\\FME\\datasources;C:\\Users\\USER\\Documents\\FME\\Formats
Shared folders for transformers are : C:\\Users\\USER\\AppData\\Roaming\\Safe Software\\FME\\Packages\\19238-win64\\transformers;C:\\Program Files\\FME\\transformers
Shared folders for coordinate systems are : C:\\Users\\USER\\Documents\\FME\\CoordinateSystems
Shared folders for coordinate system exceptions are : C:\\Users\\USER\\Documents\\FME\\CoordinateSystemExceptions
Shared folders for coordinate system grid overrides are : C:\\Users\\USER\\Documents\\FME\\CoordinateSystemGridOverrides
Shared folders for CS-MAP transformation exceptions are : C:\\Users\\USER\\Documents\\FME\\CsmapTransformationExceptions
Shared folders for transformer categories are : C:\\Users\\USER\\Documents\\FME\\TransformerCategories
FME Configuration: Reader Keyword is `MULTI_READER'
FME Configuration: Writer Keyword is `NULL_DEST'
FME Configuration: Writer Group Definition Keyword is `NULL_DEST_DEF'
FME Configuration: Reader type is `MULTI_READER'
FME Configuration: Writer type is `NULL'
FME Configuration: No destination coordinate system set
FME Configuration: Current working folder is `C:\\Users\\USER\\Downloads'
FME Configuration: Temporary folder is `C:\\Users\\USER~1.GUR\\AppData\\Local\\Temp', set from environment variable `TEMP'
FME Configuration: FME_HOME is `C:\\Program Files\\FME\\'
FME Configuration: FME_BASE is 'no'
FME Configuration: FME_MF_DIR is 'C:\\Users\\USER\\Downloads/'
FME Configuration: FME_MF_NAME is 'Time_range_filter.fmw'
FME Configuration: FME_PRODUCT_NAME is 'FME(R) 2019.0.0.0'
System Status: 248.80 GB of disk space available in the FME temporary folder (C:\\Users\\USER~1.GUR\\AppData\\Local\\Temp)
System Status: 128.00 TB of virtual memory available
Operating System: Microsoft Windows 10 64-bit (Build 17763)
FME Platform: WIN64
Locale: en_US
Code Page: 1252 (ANSI - Latin I)
FME Configuration: Process limit is 15.84 GB of physical memory
FME Configuration: Start freeing memory when process usage exceeds 47.53 GB of virtual memory
FME Configuration: Stop freeing memory when process usage is below 35.65 GB of virtual memory
Creating writer for format: NULL (Nothing)
Trying to find a DYNAMIC plugin for writer named `NULL'
Loaded module 'NULL' from file 'C:\\Program Files\\FME\\plugins/NULL.dll'
FME API version of module 'NULL' matches current internal version (3.8 20181030)
Creating reader for format:
MULTI_READER(MULTI_READER): Will fail with first member reader failure
Using Multi Reader with keyword `MULTI_READER' to read multiple datasets
FME API version of module 'NULL' matches current internal version (3.8 20181030)
FME API version of module 'NULL' matches current internal version (3.8 20181030)
Loaded module 'Geometry_func' from file 'C:\\Program Files\\FME\\plugins/Geometry_func.dll'
FME API version of module 'Geometry_func' matches current internal version (3.8 20181030)
Loaded module 'LogCount_func' from file 'C:\\Program Files\\FME\\plugins/LogCount_func.dll'
FME API version of module 'LogCount_func' matches current internal version (3.8 20181030)
Loaded module 'HTTPFactory' from file 'C:\\Program Files\\FME\\plugins/HTTPFactory.dll'
FME API version of module 'HTTPFactory' matches current internal version (3.8 20181030)
Loaded module 'JSONQueryFactory' from file 'C:\\Program Files\\FME\\plugins/JSONQueryFactory.dll'
FME API version of module 'JSONQueryFactory' matches current internal version (3.8 20181030)
Loaded module 'InspectorFactory' from file 'C:\\Program Files\\FME\\plugins/InspectorFactory.dll'
FME API version of module 'InspectorFactory' matches current internal version (3.8 20181030)
Emptying factory pipeline
HTTPCaller_2 (HTTPFactory): HTTP/FTP Transfer: Downloading resource 'http://localhost/index/_search?scroll=50000&q=FILING_DATE:gte%20now-10d&sort=FILING_DATE:desc&size=100' to internal memory buffer
HTTPCaller_2 (HTTPFactory): HTTP transfer summary - status code: 400, download size: '173 bytes', DNS lookup time: '1e-6 seconds', total transfer time: '0.422 seconds', url: 'http://localhost/index/_search?scroll=50000&q=FILING_DATE:gte%20now-10d&sort=FILING_DATE:desc&size=100'
HTTPCaller_2 (HTTPFactory): Received HTTP response header: 'HTTP/1.1 400 Bad Request' from 'http://localhost/index/_search?scroll=50000&q=FILING_DATE:gte%20now-10d&sort=FILING_DATE:desc&size=100'
Storing feature(s) to FME feature store file `C:\\Users\\USER~1.GUR\\AppData\\Local\\Temp\\wb-cache-Time_range_filter-qKvOXO\\Main_HTTPCaller_2 -1 3 fo 1 _lt_REJECTED_gt_ 0 d776f40359e236fde79f22273ec7f46bc9af9a37.ffsupdating'
Creator_2_XML_Creator (CreationFactory): Created 1 features
Creator_2_Cloner (TeeFactory): Cloned 1 input feature(s) into 1 output feature(s)
Creator_2_CREATED Feature Counter -1 6 (TeeFactory): Cloned 1 input feature(s) into 1 output feature(s)
Creator_2_CREATED Brancher -1 6 (BranchingFactory): Branched 1 input feature -- 1 feature routed to the target factory, and 0 features routed to the fallback factory.
_CREATOR_BRANCH_TARGET (TeeFactory): Cloned 1 input feature(s) into 1 output feature(s)
HTTPCaller_2_Output Feature Counter -1 7 (TeeFactory): Cloned 0 input feature(s) into 0 output feature(s)
HTTPCaller_2_<REJECTED> Feature Counter -1 2147680259 (TeeFactory): Cloned 1 input feature(s) into 0 output feature(s)
JSONFragmenter_2_FRAGMENTS Feature Counter -1 8 (TeeFactory): Cloned 0 input feature(s) into 0 output feature(s)
JSONFragmenter_2_<REJECTED> Feature Counter -1 2147680261 (TeeFactory): Cloned 0 input feature(s) into 0 output feature(s)
Stored 1 feature(s) to FME feature store file `C:\\Users\\USER~1.GUR\\AppData\\Local\\Temp\\wb-cache-Time_range_filter-qKvOXO\\Main_HTTPCaller_2 -1 3 fo 1 _lt_REJECTED_gt_ 0 d776f40359e236fde79f22273ec7f46bc9af9a37.ffsupdating'
Saving spatial index into file 'C:\\Users\\USER~1.GUR\\AppData\\Local\\Temp\\wb-cache-Time_range_filter-qKvOXO\\Main_HTTPCaller_2 -1 3 fo 1 _lt_REJECTED_gt_ 0 d776f40359e236fde79f22273ec7f46bc9af9a37.fsi'
Finished saving spatial index into file 'C:\\Users\\USER~1.GUR\\AppData\\Local\\Temp\\wb-cache-Time_range_filter-qKvOXO\\Main_HTTPCaller_2 -1 3 fo 1 _lt_REJECTED_gt_ 0 d776f40359e236fde79f22273ec7f46bc9af9a37.fsi'
Destination Feature Type Routing Correlator (RoutingFactory): Tested 0 input feature(s), wrote 0 output feature(s): 0 matched merge filters, 0 were routed to output, 0 could not be routed.
Final Output Nuker (TeeFactory): Cloned 0 input feature(s) into 0 output feature(s)
VisualizerProcessor (CreationFactory): Created 1 features
VisualizerCleaner (TeeFactory): Cloned 1 input feature(s) into 0 output feature(s)
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Features Read Summary
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
==============================================================================
Total Features Read 0
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Features Written Summary
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
==============================================================================
Total Features Written 0
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Translation was SUCCESSFUL with 0 warning(s) (0 feature(s) output)
FME Session Duration: 0.7 seconds. (CPU: 0.1s user, 0.1s system)
END - ProcessID: 14760, peak process memory usage: 47792 kB, current process memory usage: 47792 kB
Translation was SUCCESSFUL
I agree with @jdh, you should look into what's often called "paging" using a looping custom transformer. You can find more information about looping custom transformers in the training material: https://safe-software.gitbooks.io/fme-desktop-advanced-training-2018/content/DesktopAdvanced5CustomTransformers/5.11.CustomTransformerLoops.html
Also have a look at the following post, in particular the answer from @jdh:
https://knowledge.safe.com/questions/51270/loop-on-httpcaller.html