Question

Custom transformer issue

  • 16 May 2023
  • 8 replies
  • 14 views
B

I want to loop number of times (22), so, created Custom transformer. Tried embedded, didn't work, so now Linked Always with fmx.

1. Unable to work out whats the issue, it works if just run for only one iteration (test purpose), so, seems no issue in both main & fmx (custom).

 

2.But an issue to loop, or going to next iteration

 

3.Could not change CustomTransfomer>Transformer Pramaters>Advanced > Maximum number of iterations or Attribute to holdl iteration count !?

 

4.However, I have used 'LoopRecs' for iteration from main to loop transformer and added 'MyCounter' for loop test.

 

5.Please note, added sampler, just to run 3 times for testing, but actually wanted to loop based on 'LoopRecs' variable, which is passed thru main to custom.

 

Whats issue here ? I have spent a lot of time figuring out . . is this a bug ? tried to get help from our FME tech support here , no response..

 

Can anyone help PLEASE , getting really frustrated to figure out the issue !

 

Attached both FMW & FMX for your review.

Many many Thanks

CustomTrans_Advanced 

8 replies

caracadrian
Rank
Badge +20

We all struggled with this the first time.

You basically start by creating an iteration number attribute equal to 1, do something to process the data and test fot itNo<=22 (any number). If itNo<=22 send to loop, increase by 1 and do the same something again until itNo=22.

Iterate in custom transformerI attached a sample workspace that does just that.

B

Thanks for your response, really appreciate.

It works if one or two loops and no exploded records within the custom transformer.

 

My issue is: in the loop, I'm exploding a list, which returns 100's values, which I'm writing into writer then setting counter and loop back at the last record from the list.. this is where its causing issue ! blocking or something else..

 

As per FME doc help for custom loops, recommended to link Advanced Parameters; but I can't link to Transformer Properties (as shown in my screenshot earlier) .. should we ? I'm using FME 2022.2.3 .. am I missing something here ?

 

caracadrian
Rank
Badge +20

Thanks for your response, really appreciate.

It works if one or two loops and no exploded records within the custom transformer.

 

My issue is: in the loop, I'm exploding a list, which returns 100's values, which I'm writing into writer then setting counter and loop back at the last record from the list.. this is where its causing issue ! blocking or something else..

 

As per FME doc help for custom loops, recommended to link Advanced Parameters; but I can't link to Transformer Properties (as shown in my screenshot earlier) .. should we ? I'm using FME 2022.2.3 .. am I missing something here ?

 

If you are paging through an API response and know how many records you are getting each time you can use that as a test. With each response you get some total number of features which you will use in the Tester and you increase the start feature number or the page number by the number of features you are receiving.

You can put any blocking transformers outside the custom transformer or you put the loop inside a "level 2" custom transformer inside the one with the blocking part.

Take a look at eterra.ancpi.roApplicationList | FME Hub (safe.com) custom transformer that goes to a level 3 of embedding to avoid any blocking (the deepest one contains just an aggregator).

B

If you are paging through an API response and know how many records you are getting each time you can use that as a test. With each response you get some total number of features which you will use in the Tester and you increase the start feature number or the page number by the number of features you are receiving.

You can put any blocking transformers outside the custom transformer or you put the loop inside a "level 2" custom transformer inside the one with the blocking part.

Take a look at eterra.ancpi.roApplicationList | FME Hub (safe.com) custom transformer that goes to a level 3 of embedding to avoid any blocking (the deepest one contains just an aggregator).

Thanks for your reply, I did that test (each response), first record works out perfect.

The problem is the second record onwards, when it passes thru the list exploder, fetches attributes, then at the sampler to the loop.

This is still says its blocks and stops at the loop.. I tried several times with both embedded and exported to fmx (linked).

It doesn't work.

I can reupload both fme & fmx if you wud like to have a look, really appreciate your help

 

Thanks

 

caracadrian
Rank
Badge +20

If you are paging through an API response and know how many records you are getting each time you can use that as a test. With each response you get some total number of features which you will use in the Tester and you increase the start feature number or the page number by the number of features you are receiving.

You can put any blocking transformers outside the custom transformer or you put the loop inside a "level 2" custom transformer inside the one with the blocking part.

Take a look at eterra.ancpi.roApplicationList | FME Hub (safe.com) custom transformer that goes to a level 3 of embedding to avoid any blocking (the deepest one contains just an aggregator).

You should test if you are on the last page before exploding the list. Use JSONExtractor (or something equivalent for your data type) to get current page and total pages.

Else you could try putting Sampler (just the Sampler) inside a custom transformer but you would beed a Group By attribute on it for iteration number.

B

If you are paging through an API response and know how many records you are getting each time you can use that as a test. With each response you get some total number of features which you will use in the Tester and you increase the start feature number or the page number by the number of features you are receiving.

You can put any blocking transformers outside the custom transformer or you put the loop inside a "level 2" custom transformer inside the one with the blocking part.

Take a look at eterra.ancpi.roApplicationList | FME Hub (safe.com) custom transformer that goes to a level 3 of embedding to avoid any blocking (the deepest one contains just an aggregator).

> Yes, I have JSONExtractor before exploder, don't need after the ListExploder. Pls refer screenshot

JasonFormatter 

> Yes, indeed I have put my sampler in the custom transformer , thats the one is blocking (as per error message), please refer screenshot

 

Sampler> I have also, attached full workbench again, you could run to trial, both way tried, custom transforer embeded & linked (external fmx).

 

B

If you are paging through an API response and know how many records you are getting each time you can use that as a test. With each response you get some total number of features which you will use in the Tester and you increase the start feature number or the page number by the number of features you are receiving.

You can put any blocking transformers outside the custom transformer or you put the loop inside a "level 2" custom transformer inside the one with the blocking part.

Take a look at eterra.ancpi.roApplicationList | FME Hub (safe.com) custom transformer that goes to a level 3 of embedding to avoid any blocking (the deepest one contains just an aggregator).

Please note, both of these files (FMW & FMX) are in 2023 edition

B

If you are paging through an API response and know how many records you are getting each time you can use that as a test. With each response you get some total number of features which you will use in the Tester and you increase the start feature number or the page number by the number of features you are receiving.

You can put any blocking transformers outside the custom transformer or you put the loop inside a "level 2" custom transformer inside the one with the blocking part.

Take a look at eterra.ancpi.roApplicationList | FME Hub (safe.com) custom transformer that goes to a level 3 of embedding to avoid any blocking (the deepest one contains just an aggregator).

Another update, I went thourgh the FME online doc, and had to click 'yes' for Enable Blocked Looping, which adds two attribute requirements.

as shown below..and linked to user parameters.

Block_Yes 

But now the trouble is 'custom transformer' is not allowing to pass the attribute value from main workspace ?

Then how do I pass my dynamic loop numbers ?

shown as below :

 

NoUserParameter 

FME doc help is bit clunky I thought..or I'm missing something..

Tried to save, close, open, refresh etc.. to ensure changes went thru correct..

 

 

Reply


Community Stats

28,723
Posts
109,510
Replies
36,559
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.