Solved

Use Variable on Sampler

  • 4 October 2018
  • 3 replies
  • 8 views
C

I have a need to sample 10% of a data set. I use the Counter and then the Statistics Calculator to get the total record count. From there, the Expression Evaluator is used to multiply that total count by 10% (setting to a whole number). The Variable Setter then sets that value to a variable. The variable is used for the N input on the Sampler. I do not get an error, but no records are output from Sampler. I have confirmed there is a value in that variable (using eh Variable Retriever). Enclosed are the details. Any suggestions?

icon

Best answer by takashi 5 October 2018, 00:52

View original
1 Attachment

3 replies

daveatsafe
Rank
Userlevel 2
Badge +17

Hi @chris_wright,

I'm sorry, but the variable value can only be accessed by the VariableRetriever transformer, and is not available as an parameter input on any other transformer.

If you need to get the first 10% of the features, you can use a Tester to check whether the count attribute is less than (total count*0.1). This will run just as fast as using a Sampler.

takashi
Rank
Userlevel 2
Badge +17

Hi @chris_wright, if the sampling rate is always 10% (i.e. extract one-tenth of the features), the Sampler with these parameters does that.

  • Sampling Rate (N): 10
  • Sampling Type: Every Nth Feature
  • Randomize Sampling: <set Yes if you need to perform random sampling>
C

Hi @chris_wright,

I'm sorry, but the variable value can only be accessed by the VariableRetriever transformer, and is not available as an parameter input on any other transformer.

If you need to get the first 10% of the features, you can use a Tester to check whether the count attribute is less than (total count*0.1). This will run just as fast as using a Sampler.

Thanks Dave, good to know. With the tester, it only takes the first 10%...any suggestions on randomizing that? Maybe use the Random Number Generator and then sort on that before the tester.

 

 

Reply


Community Stats

28,684
Posts
109,412
Replies
36,542
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.