Skip to main content
Solved

Process a table with a loop

dataman
Contributor
Forum|alt.badge.img+5

Hi there,

We have an excel file with an easy table. Just 2 columns: Position and Id. The file has about 100 lines.

We want to read the file and process each line. We have tried with a custom transformer but we are not able to process the lines individually. If we read the Excelfile we get the 100 lines and all lines will be process at the same time. If we convert the lines into a List, we get the error "Transformer/Factory test_loop_2_ListBuilder is blocking and cannot be used in a loop, but was found between 'Loop to test_loop_2_Looper1697190238 Input Splitter' and 'test_loop_2_Looper1697190238 Input Splitter'".

 

Has anyone an idea to make it?

I have found this link (https://docs.safe.com/fme/html/FME-Form-Documentation/FME-Form/Workbench/transformers_custom_looping.htm) but it helps me only with the loop. Not how I can process the ids separatelly.

 

Many thanks in advance!

Best answer by nielsgerrits

nielsgerrits wrote:

Most transformers have a Group Processing option which will isolate features which have a common ID. See the Clipper for example, if you check Group Processing and select "GroupId" it will only compare clippers and clippees which have the same GroupId.

 

In the custom transformer, in the Navigator window, Transformer Parameters, Group By, right click and choose Create User Parameter. Click OK. In the main workspace, you can now choose an attribute to run the custom transformer grouped by.

 

If needed, you can also create user parameters for Group By Mode and Parallel Processing.

Can I read the table and after that just put a HTTPCaller or would it be too much for the performance?

 

Yes, this is how FME works. Just concat the url from the attributes in the HTTPCaller. Not hard on performance.

View original
Did this help you find an answer to your question?

7 replies

nielsgerrits
VIP
Forum|alt.badge.img+54

When you assign unique ids to the lines, you can configure the custom transformer to do a group by and process the lines individually. I would not use a loop for this.

geomancer
Evangelist
Forum|alt.badge.img+47
  • geomancerEvangelist
  • Evangelist
  • October 13, 2023

I'm wondering what process would make you use a loop for processing this simple dataset. FME will already process each line (each feature) apart.

nielsgerrits
VIP
Forum|alt.badge.img+54
geomancer wrote:

I'm wondering what process would make you use a loop for processing this simple dataset. FME will already process each line (each feature) apart.

@dataman​ is probably not aware of the default Group By functionality in FME. Or is missing it in certain transformers. A custom transformer can than be a possible solution. I remember a specific case where I needed it in a FeatureReader.

dataman
Contributor
Forum|alt.badge.img+5
  • datamanContributor
  • Author
  • Contributor
  • October 16, 2023

Thanks for the answers!

Yes, I am a freshman in FME and I have no experience using Group By... I do have one column with unique IDs. Where can I configure the custom transformer to do a group by? I do not see any option to do that (I mean clicking on the green Input transformers). Or should I check the "Group by" check box of the normal transformers within the custom transformer?

nielsgerrits
VIP
Forum|alt.badge.img+54
dataman wrote:

Thanks for the answers!

Yes, I am a freshman in FME and I have no experience using Group By... I do have one column with unique IDs. Where can I configure the custom transformer to do a group by? I do not see any option to do that (I mean clicking on the green Input transformers). Or should I check the "Group by" check box of the normal transformers within the custom transformer?

Most transformers have a Group Processing option which will isolate features which have a common ID. See the Clipper for example, if you check Group Processing and select "GroupId" it will only compare clippers and clippees which have the same GroupId.

 

In the custom transformer, in the Navigator window, Transformer Parameters, Group By, right click and choose Create User Parameter. Click OK. In the main workspace, you can now choose an attribute to run the custom transformer grouped by.

 

If needed, you can also create user parameters for Group By Mode and Parallel Processing.

dataman
Contributor
Forum|alt.badge.img+5
  • datamanContributor
  • Author
  • Contributor
  • October 16, 2023
nielsgerrits wrote:

Most transformers have a Group Processing option which will isolate features which have a common ID. See the Clipper for example, if you check Group Processing and select "GroupId" it will only compare clippers and clippees which have the same GroupId.

 

In the custom transformer, in the Navigator window, Transformer Parameters, Group By, right click and choose Create User Parameter. Click OK. In the main workspace, you can now choose an attribute to run the custom transformer grouped by.

 

If needed, you can also create user parameters for Group By Mode and Parallel Processing.

I have to use the Values of one column for the URL of a HTTPCaller. I have checked the parameters of the HTTPCaller and there I have no GroupBy Option.

Can I read the table and after that just put a HTTPCaller or would it be too much for the performance? The Workbench must run about 100 URLs.

 

Maybe I explain what we want to do in order to a better understanding:

  1. We read an Excel file with 2 Columns: Location | ID
  2. for each ID we have to get the values from a CSV File (HTTPCaller)
  3. Process the data: if we have no data of this ID or the values are always the same -> this device has a problem and we write the ID (and Location if possible) in a output Excel file.

 

Because of this iteration (100 times) I had a loop in mind. But if there is a better option, it is wellcome :-)

nielsgerrits
VIP
Forum|alt.badge.img+54
  • nielsgerritsVIP
  • Best Answer
  • October 16, 2023
nielsgerrits wrote:

Most transformers have a Group Processing option which will isolate features which have a common ID. See the Clipper for example, if you check Group Processing and select "GroupId" it will only compare clippers and clippees which have the same GroupId.

 

In the custom transformer, in the Navigator window, Transformer Parameters, Group By, right click and choose Create User Parameter. Click OK. In the main workspace, you can now choose an attribute to run the custom transformer grouped by.

 

If needed, you can also create user parameters for Group By Mode and Parallel Processing.

Can I read the table and after that just put a HTTPCaller or would it be too much for the performance?

 

Yes, this is how FME works. Just concat the url from the attributes in the HTTPCaller. Not hard on performance.

Reply


Helpful Members This Week

Community Stats

31,183
Posts
118,189
Replies
38,791
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.