Not completely sure about the total use case, but in general, I agree with @mark2atsafe that if possible, it is better to avoid loops if you can figure out how to frame the problem in an FME friendly way. @Ryan Cragg and I have done thought experiments over lunch (when we were obviously out of things to talk about) over the years, and concluded many looping cases can be rephrased into parameterized Cloners which generate some calculated number of copies. Then you can do math to get you the "index" you might otherwise want.
See below screenshot and attached workspace for a starting point idea.
Note that this technique is particularly well suited to a situation where you know the number of calls or copies ahead of time (which this question seems to be). What if you have no idea? Well, you might make a super wild guess at an outrageous maximum. Way way way more than you'd ever imagine you'd need. 100,000? Sure. 1 million. Why not? Clone that many. Go into a VariableRetriever -- that will tell you if you need to stop or not. It is a voice from the future. Use a Tester to see if it says "STOP" (you choose your variable name and your sentinel value -- note that if a variable has never been set, it is blank, so no need to set an initial value). Anyway, you are are to STOP, DO NOT send that feature anywhere (so all those extra clones that you didn't need will just whip by super duper fast and go straight to the feature boneyard). If it does NOT tell you to stop, send the feature to whatever is going to do your work. And then after you did that work, if you can tell that you're done, be a good sport and use a VariableSetter to set your variable to "STOP". That will send that message back to the "past", back there where the clones are being made and tested, and turn off the tap. Wild. But true. Elegant. Not really. Fast. You bet.
Not completely sure about the total use case, but in general, I agree with @mark2atsafe that if possible, it is better to avoid loops if you can figure out how to frame the problem in an FME friendly way. @Ryan Cragg and I have done thought experiments over lunch (when we were obviously out of things to talk about) over the years, and concluded many looping cases can be rephrased into parameterized Cloners which generate some calculated number of copies. Then you can do math to get you the "index" you might otherwise want.
See below screenshot and attached workspace for a starting point idea.
Note that this technique is particularly well suited to a situation where you know the number of calls or copies ahead of time (which this question seems to be). What if you have no idea? Well, you might make a super wild guess at an outrageous maximum. Way way way more than you'd ever imagine you'd need. 100,000? Sure. 1 million. Why not? Clone that many. Go into a VariableRetriever -- that will tell you if you need to stop or not. It is a voice from the future. Use a Tester to see if it says "STOP" (you choose your variable name and your sentinel value -- note that if a variable has never been set, it is blank, so no need to set an initial value). Anyway, you are are to STOP, DO NOT send that feature anywhere (so all those extra clones that you didn't need will just whip by super duper fast and go straight to the feature boneyard). If it does NOT tell you to stop, send the feature to whatever is going to do your work. And then after you did that work, if you can tell that you're done, be a good sport and use a VariableSetter to set your variable to "STOP". That will send that message back to the "past", back there where the clones are being made and tested, and turn off the tap. Wild. But true. Elegant. Not really. Fast. You bet.
In my case this way is no option. Unfortunately.
Every request contains 100 features, and i need the max value of an attribute of these features to request the next 100 features.
In this case this is the worst API you could imagine.
So after my request i have to proces all features, use a summerizer to get the max value (because it is not the last feature that has the max value), this is a blocking transformer. And then use this attribute to go back to the HTTPCaller and ask for the next 100 features.
But for a regular API where the first request has the max dataset value and the request size the solution can work. Then after the first request you can use the attribute to clone your features and use the clone attribute to calculate the offset.
In my case this way is no option. Unfortunately.
Every request contains 100 features, and i need the max value of an attribute of these features to request the next 100 features.
In this case this is the worst API you could imagine.
So after my request i have to proces all features, use a summerizer to get the max value (because it is not the last feature that has the max value), this is a blocking transformer. And then use this attribute to go back to the HTTPCaller and ask for the next 100 features.
But for a regular API where the first request has the max dataset value and the request size the solution can work. Then after the first request you can use the attribute to clone your features and use the clone attribute to calculate the offset.
Could you use Adjacent Feature Handling in an AttributeCreator to get the max so avoiding a blocking transformer in your loop?
e.g. set the max from the first value received, then if the next value is greater use that as the max otherwise remain unchanged, then a sampler to get a single feature with that max value for the next request
Not completely sure about the total use case, but in general, I agree with @mark2atsafe that if possible, it is better to avoid loops if you can figure out how to frame the problem in an FME friendly way. @Ryan Cragg and I have done thought experiments over lunch (when we were obviously out of things to talk about) over the years, and concluded many looping cases can be rephrased into parameterized Cloners which generate some calculated number of copies. Then you can do math to get you the "index" you might otherwise want.
See below screenshot and attached workspace for a starting point idea.
Note that this technique is particularly well suited to a situation where you know the number of calls or copies ahead of time (which this question seems to be). What if you have no idea? Well, you might make a super wild guess at an outrageous maximum. Way way way more than you'd ever imagine you'd need. 100,000? Sure. 1 million. Why not? Clone that many. Go into a VariableRetriever -- that will tell you if you need to stop or not. It is a voice from the future. Use a Tester to see if it says "STOP" (you choose your variable name and your sentinel value -- note that if a variable has never been set, it is blank, so no need to set an initial value). Anyway, you are are to STOP, DO NOT send that feature anywhere (so all those extra clones that you didn't need will just whip by super duper fast and go straight to the feature boneyard). If it does NOT tell you to stop, send the feature to whatever is going to do your work. And then after you did that work, if you can tell that you're done, be a good sport and use a VariableSetter to set your variable to "STOP". That will send that message back to the "past", back there where the clones are being made and tested, and turn off the tap. Wild. But true. Elegant. Not really. Fast. You bet.
Thanks for your post @Dale Lutz. I never realized the VariableRetriever can get an answer from a VariableSetter which is downstream. Learned something new today!
This is another sample for a NoLooper. This works when the HTTPCaller gives an answer that should stop the workbench from requesting new features from the HTTPCaller. The Cloner should be set to the absolute max expected requests, but still resonable value. And the Setter will stop the request.
Thanks for your post @Dale Lutz. I never realized the VariableRetriever can get an answer from a VariableSetter which is downstream. Learned something new today!
Be aware that if you have transformers set to Preserve Feature Order: Per Output Port, variable setters and retrievers may not work as expected
In my case this way is no option. Unfortunately.
Every request contains 100 features, and i need the max value of an attribute of these features to request the next 100 features.
In this case this is the worst API you could imagine.
So after my request i have to proces all features, use a summerizer to get the max value (because it is not the last feature that has the max value), this is a blocking transformer. And then use this attribute to go back to the HTTPCaller and ask for the next 100 features.
But for a regular API where the first request has the max dataset value and the request size the solution can work. Then after the first request you can use the attribute to clone your features and use the clone attribute to calculate the offset.
This is the solution to get the max value without using a blocking transformer. In combination with a VariableSetter and VariableRetriever this may be the complete solution for my problem.
Thanks for thinking with me here.
It is not the answer to my original question about the best way to work with customTransformers but a good solution to my problem.
Thanks for your post @Dale Lutz. I never realized the VariableRetriever can get an answer from a VariableSetter which is downstream. Learned something new today!
It was a surprise this works and indeed it can have unexpected results in a larger workbenches. I wouldn't trust my variableRetrievers if I didn't test the results first.
