Skip to main content
Solved

Looping with parameter based on result and blocking transformers - HowTo?

jkr_wrk
Influencer
Forum|alt.badge.img+28

I got my Looping Transformers working but it feels wrong to do it the way I solve it now. How would others solve my problem?

 

I do a webrequest and in my result I recieve a value I need for my next request. At this moment I use a CustomTransformer with a published parameter set to AttributeName. Then I use an AttributeCreator to move the value to the AttributeName I use in my customTransformer. Right before the looping port I write the Value back to the Published parameter.

 

For information about the Api:

https://docs.ndw.nu/api/trafficsigns/nl/index.html

 

My goal is to download the dataset from a specific date to now.

 

Start

 

End

 

 

 

Best answer by ebygomm

jkr_da wrote:

In my case this way is no option. Unfortunately.

 

Every request contains 100 features, and i need the max value of an attribute of these features to request the next 100 features.

 

In this case this is the worst API you could imagine.

 

So after my request i have to proces all features, use a summerizer to get the max value (because it is not the last feature that has the max value), this is a blocking transformer. And then use this attribute to go back to the HTTPCaller and ask for the next 100 features.

 

But for a regular API where the first request has the max dataset value and the request size the solution can work. Then after the first request you can use the attribute to clone your features and use the clone attribute to calculate the offset.

 

 

Could you use Adjacent Feature Handling in an AttributeCreator to get the max so avoiding a blocking transformer in your loop?

 

e.g. set the max from the first value received, then if the next value is greater use that as the max otherwise remain unchanged, then a sampler to get a single feature with that max value for the next request

View original
💡 Did you know... The FeatureWriter is more intuitive. 😏
Did this help you find an answer to your question?

8 replies

fmelizard
Contributor
Forum|alt.badge.img+17
  • fmelizardContributor
  • Contributor
  • August 22, 2020

Not completely sure about the total use case, but in general, I agree with @mark2atsafe​ that if possible, it is better to avoid loops if you can figure out how to frame the problem in an FME friendly way. @Ryan Cragg​ and I have done thought experiments over lunch (when we were obviously out of things to talk about) over the years, and concluded many looping cases can be rephrased into parameterized Cloners which generate some calculated number of copies. Then you can do math to get you the "index" you might otherwise want.

 

See below screenshot and attached workspace for a starting point idea.

Note that this technique is particularly well suited to a situation where you know the number of calls or copies ahead of time (which this question seems to be). What if you have no idea? Well, you might make a super wild guess at an outrageous maximum. Way way way more than you'd ever imagine you'd need. 100,000? Sure. 1 million. Why not? Clone that many. Go into a VariableRetriever -- that will tell you if you need to stop or not. It is a voice from the future. Use a Tester to see if it says "STOP" (you choose your variable name and your sentinel value -- note that if a variable has never been set, it is blank, so no need to set an initial value). Anyway, you are are to STOP, DO NOT send that feature anywhere (so all those extra clones that you didn't need will just whip by super duper fast and go straight to the feature boneyard). If it does NOT tell you to stop, send the feature to whatever is going to do your work. And then after you did that work, if you can tell that you're done, be a good sport and use a VariableSetter to set your variable to "STOP". That will send that message back to the "past", back there where the clones are being made and tested, and turn off the tap. Wild. But true. Elegant. Not really. Fast. You bet.

 

NoLooper

jkr_wrk
Influencer
Forum|alt.badge.img+28
  • jkr_wrkInfluencer
  • Author
  • August 26, 2020
fmelizard wrote:

Not completely sure about the total use case, but in general, I agree with @mark2atsafe​ that if possible, it is better to avoid loops if you can figure out how to frame the problem in an FME friendly way. @Ryan Cragg​ and I have done thought experiments over lunch (when we were obviously out of things to talk about) over the years, and concluded many looping cases can be rephrased into parameterized Cloners which generate some calculated number of copies. Then you can do math to get you the "index" you might otherwise want.

 

See below screenshot and attached workspace for a starting point idea.

Note that this technique is particularly well suited to a situation where you know the number of calls or copies ahead of time (which this question seems to be). What if you have no idea? Well, you might make a super wild guess at an outrageous maximum. Way way way more than you'd ever imagine you'd need. 100,000? Sure. 1 million. Why not? Clone that many. Go into a VariableRetriever -- that will tell you if you need to stop or not. It is a voice from the future. Use a Tester to see if it says "STOP" (you choose your variable name and your sentinel value -- note that if a variable has never been set, it is blank, so no need to set an initial value). Anyway, you are are to STOP, DO NOT send that feature anywhere (so all those extra clones that you didn't need will just whip by super duper fast and go straight to the feature boneyard). If it does NOT tell you to stop, send the feature to whatever is going to do your work. And then after you did that work, if you can tell that you're done, be a good sport and use a VariableSetter to set your variable to "STOP". That will send that message back to the "past", back there where the clones are being made and tested, and turn off the tap. Wild. But true. Elegant. Not really. Fast. You bet.

 

NoLooper

In my case this way is no option. Unfortunately.

 

Every request contains 100 features, and i need the max value of an attribute of these features to request the next 100 features.

 

In this case this is the worst API you could imagine.

 

So after my request i have to proces all features, use a summerizer to get the max value (because it is not the last feature that has the max value), this is a blocking transformer. And then use this attribute to go back to the HTTPCaller and ask for the next 100 features.

 

But for a regular API where the first request has the max dataset value and the request size the solution can work. Then after the first request you can use the attribute to clone your features and use the clone attribute to calculate the offset.

 

 

💡 Did you know... The FeatureWriter is more intuitive. 😏
ebygomm
Influencer
Forum|alt.badge.img+31
  • ebygommInfluencer
  • Influencer
  • Best Answer
  • August 26, 2020
jkr_da wrote:

In my case this way is no option. Unfortunately.

 

Every request contains 100 features, and i need the max value of an attribute of these features to request the next 100 features.

 

In this case this is the worst API you could imagine.

 

So after my request i have to proces all features, use a summerizer to get the max value (because it is not the last feature that has the max value), this is a blocking transformer. And then use this attribute to go back to the HTTPCaller and ask for the next 100 features.

 

But for a regular API where the first request has the max dataset value and the request size the solution can work. Then after the first request you can use the attribute to clone your features and use the clone attribute to calculate the offset.

 

 

Could you use Adjacent Feature Handling in an AttributeCreator to get the max so avoiding a blocking transformer in your loop?

 

e.g. set the max from the first value received, then if the next value is greater use that as the max otherwise remain unchanged, then a sampler to get a single feature with that max value for the next request

nielsgerrits
VIP
Forum|alt.badge.img+53
fmelizard wrote:

Not completely sure about the total use case, but in general, I agree with @mark2atsafe​ that if possible, it is better to avoid loops if you can figure out how to frame the problem in an FME friendly way. @Ryan Cragg​ and I have done thought experiments over lunch (when we were obviously out of things to talk about) over the years, and concluded many looping cases can be rephrased into parameterized Cloners which generate some calculated number of copies. Then you can do math to get you the "index" you might otherwise want.

 

See below screenshot and attached workspace for a starting point idea.

Note that this technique is particularly well suited to a situation where you know the number of calls or copies ahead of time (which this question seems to be). What if you have no idea? Well, you might make a super wild guess at an outrageous maximum. Way way way more than you'd ever imagine you'd need. 100,000? Sure. 1 million. Why not? Clone that many. Go into a VariableRetriever -- that will tell you if you need to stop or not. It is a voice from the future. Use a Tester to see if it says "STOP" (you choose your variable name and your sentinel value -- note that if a variable has never been set, it is blank, so no need to set an initial value). Anyway, you are are to STOP, DO NOT send that feature anywhere (so all those extra clones that you didn't need will just whip by super duper fast and go straight to the feature boneyard). If it does NOT tell you to stop, send the feature to whatever is going to do your work. And then after you did that work, if you can tell that you're done, be a good sport and use a VariableSetter to set your variable to "STOP". That will send that message back to the "past", back there where the clones are being made and tested, and turn off the tap. Wild. But true. Elegant. Not really. Fast. You bet.

 

NoLooper

Thanks for your post @Dale Lutz​. I never realized the VariableRetriever can get an answer from a VariableSetter which is downstream. Learned something new today!

VariableRetrieverVariableSetter

jkr_wrk
Influencer
Forum|alt.badge.img+28
  • jkr_wrkInfluencer
  • Author
  • August 26, 2020

 

Use of Retrievers for stopping workbench

 

This is another sample for a NoLooper. This works when the HTTPCaller gives an answer that should stop the workbench from requesting new features from the HTTPCaller. The Cloner should be set to the absolute max expected requests, but still resonable value. And the Setter will stop the request.

 

💡 Did you know... The FeatureWriter is more intuitive. 😏
ebygomm
Influencer
Forum|alt.badge.img+31
  • ebygommInfluencer
  • Influencer
  • August 26, 2020
nielsgerrits wrote:

Thanks for your post @Dale Lutz​. I never realized the VariableRetriever can get an answer from a VariableSetter which is downstream. Learned something new today!

VariableRetrieverVariableSetter

Be aware that if you have transformers set to Preserve Feature Order: Per Output Port, variable setters and retrievers may not work as expected

 

jkr_wrk
Influencer
Forum|alt.badge.img+28
  • jkr_wrkInfluencer
  • Author
  • August 26, 2020
jkr_da wrote:

In my case this way is no option. Unfortunately.

 

Every request contains 100 features, and i need the max value of an attribute of these features to request the next 100 features.

 

In this case this is the worst API you could imagine.

 

So after my request i have to proces all features, use a summerizer to get the max value (because it is not the last feature that has the max value), this is a blocking transformer. And then use this attribute to go back to the HTTPCaller and ask for the next 100 features.

 

But for a regular API where the first request has the max dataset value and the request size the solution can work. Then after the first request you can use the attribute to clone your features and use the clone attribute to calculate the offset.

 

 

This is the solution to get the max value without using a blocking transformer. In combination with a VariableSetter and VariableRetriever this may be the complete solution for my problem.

 

Thanks for thinking with me here.

 

It is not the answer to my original question about the best way to work with customTransformers but a good solution to my problem.

💡 Did you know... The FeatureWriter is more intuitive. 😏
jkr_wrk
Influencer
Forum|alt.badge.img+28
  • jkr_wrkInfluencer
  • Author
  • August 26, 2020
nielsgerrits wrote:

Thanks for your post @Dale Lutz​. I never realized the VariableRetriever can get an answer from a VariableSetter which is downstream. Learned something new today!

VariableRetrieverVariableSetter

It was a surprise this works and indeed it can have unexpected results in a larger workbenches. I wouldn't trust my variableRetrievers if I didn't test the results first.

 

💡 Did you know... The FeatureWriter is more intuitive. 😏

Reply


Helpful Members This Week

Community Stats

30,985
Posts
117,436
Replies
38,765
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.