• EN
Question

Combined loop + attribute assignment question

  • 15 November 2012
  • 8 replies
  • 9 views
H
Badge
I have a large number of roads that I need to treat on a per community basis. My programming paradigm is heavily influenced by the tools I used before (a mix of matlab, python and perl), and so I'm pretty quick to see for-each-loops as the logical way to proceed. 

 

 

The problem: I'm doing some simple geometric operations to extract some metadata about the roads of each community. Those operations include aggregating, finding a common midpoint and so on - basic stuff I've implemented and tested - but I've yet to find a dynamical way to add an attribute with the ID for that particular community. 

 

 

For me, the next step is to iterate over each community, reading and treating data separately for each community polygon. I see I need to create custom transformers to do that. I'm happy with getting a little more hands-on with custom transformers - but I would appreciate a little guidelines on the way. 

 

 

In my head the solution would look like this: 

 

 

1) Read all polygons that define the community borders

 

2.0) For each polygon 

 

2.1 - Use FeatureReader with that polygon as initiator

 

2.2 - Do my simple geometry stuff

 

2.3 - Assign the community ID of the initiator to all features of this community

 

3) - save to appropriate format 

 

 

It irritates the heck out of me that step 2.3 seems to be beyond my grasp... using attribute(s) from  the initiator of featureReader as basis to  create new attributes on each feature being read seems to me to be a very intuitive thing to do. I guess I need to publish that relevant initiator-attribute as a parameter and use attributeCreator to grab that parameter and copy it. 

 

 

Step 2.0 is certainly doable, but involves some steps I'm neither familar nor comforable with (yet). I use "blocking transformers", so my custom transformer needs to be both external and  linked. And then there is the "black art" of defining the loop for my custom transformer. This was probably described extensively way back in 2006 when it was introduced, but today I have trouble finding a step-by-step guide describing what I should do and the pitfals to avoid. I'm not looking for recursive algorithms or fractal arts - I want to take one feature, do XYZ, then proceed to the next. 

 

 

Of course I know that my problem is solvable without iteration. FME impresses me with the ability to read a huge bunch of roads, process them and spit it out. But my solution - where you only treat a subset of roads at a time - will be hugely more efficient. 

 

 

Any hints on how I best should proceed? Links to looping-for-dummies? Or is there a non-looping solution that I should explore? 

8 replies

david_r
Rank
Userlevel 5
While it is perfectly possible to create custom transformers with loops in FME, it always feels a bit alien to me. However, I feel your pain, as I too come from a developers paradigm where loops and iterations are common.

 

 

For your task, here is my take on how to approach it. Remember, though, that I do not have all the details of your particular project, so not all of it may apply, but hopefully it can give you some ideas:

 

  • Read all road features
  • Read all the community polygons, containing community ID
  • Use a SpatialFilter or SpatialRelator (read the doc for pros and cons) to assign the community ID to all road objects based on their spatial relationship (usually INTERSECT)
  • From there on, put the community ID in the "Group By" clause of all the following transformers to group the treatment of the road objects by their community ID
  • If you need to use a Transformer that does not include a Group By, consider using either a ListBuilder/ListExploder og Aggregator/Deaggregator before and after to perform operations on the group of features as a whole.

 

Good luck!

 

 

David
jdh
Rank
Badge +22
Like David, I don't have the full details of your situation, but do you actually need to loop at all?  In FME terms For each loops are dealt with by Group By options, it's only Do while loops that need esoteric loop knowledge.

 

 

Assuming the only looping is for each polygon, I would do the following:

 

 

In the main workspace add your community polygons.

 

 

create a custom transformer, set the parallel processing to minimal, the newly activated parallel procees by paramter should be set to the attribute containing community ID.

 

 

In the custom transformer you should have

 

 

FeatureReader -> "SimpleGeometryStuff"->ParameterFetcher (to get the community ID)

 

 

back in the mainworkspace you would have your writer, presumably with some sort of fanout.

 

 

Joanna
davideagle
Rank
Badge +14
The best I can offer is a bit of help on how to construct a custom Transformer that contains a loop to give you a bit of insight. I have created you a short "How to" movie and its here for you.

 

 

http://www.youtube.com/watch?v=fD6jeHZlPc8

 

 

Best wishes, Dave

 

H
Badge
Thanks all,  you rock! 

 

 

@david - your way seems to be the standard FME way, and is pretty similar to what I will try out if I can't figure out the looping thing. 

 

 

@Joanna - very interesting twist with the "process by parameter" option - I am stunned :) 

 

 

@Dave: Thanks a lot for the video, exactly what I need to make looping less of "black art" 

 

 

And NO, I don't actually NEED looping. FME is perfectly capable of eating all the roads, do  stuff on it those roads and make me happy - in a not too unreasonable period of time. But on this particular subject it seems such a waste of computing power to ingest ALL the road at once when absolutely any operation is done on a per-community basis....

 

 

... or it  would at least be more efficient be if my data source (old arc SDE server) would have supported the kind of geometric queries it recieves from FeatureReader... (it doesn't, which means featureReader will have to read in a larger chunk of data and then do the spatialFilter equivalent - not very efficient). Bummer! :( 

 

 

Still, I feel very handicapped until I've added looping to my FME arsenal. Thanks all! 
H
Badge
@SpatialDave, 

 

 

Thanks a lot for the video, repeating those steps was very educating. This article now makes perfect sense: http://fmepedia.safe.com/articles/Samples_and_Demos/Looping-within-a-Custom-Transformer

 

and I feel pretty confident about defining recursive loops.

 

 

But what I want to do is to create a non-recursive loop: Take the first instance of feature A, do a bunch of interesting stuff, append to the output buffer, then take the next instance of feature A and do interesting stuff on this instance. I am banging my head against a wall here, failing to see how your recursive iteration may be reused for a non-recursive example. 

 

 

Or will custom transformers process one feature at a time automagically? 

 

 

jdh
Rank
Badge +22
For vector features FME tries to push each individual feature though the process as far as possible before going back to fetch the next feature.

 

 

As long as "all the interesting stuff" is independent (no blocking transformers), then the first instance of feature A get's processed, then the second, then the 3rd...   No need for loops at all.

 

 

If you do need to use a blocking transformer then you can use the group by parameter.  The counter can give you a unique attribute to group by if your data doesn't already contain a suitable attribute.
H
Badge
@JDH - there is one caveat: The "group by" option does NOT reduce the number of comparison that  my blocking tranformers needs to do (aggregator, LineJoiner). Unless I misunderstand completely those transformers are of order O(N^2). 

 

 

Logically, you are of course perfectly right - you should be able to solve my problem without loops in FME. But hardware will impose physical restrictions on what you can do. If my problem was significantly smaller - or my hardware way, way, way more powerful - then the solution is straightforward. So simple, fact, that I've already implemented it  - and while it works like a charm for three communities it quickly hits the capacity roof rather quickly when I increase the number of communities. I have 429 communities, and I'm guessing my capacity roof will be somewhere between 10-50 on my machine.... 

 

 

Some problems are way more elegant and quicker to solve if you can iterate on them, stuffing small chunck of a huge data set into a custom transformer that has some O(N^2) operation. So, yes - there IS a need for looping when , even with FME. My problem is  an excellent example.  

 

 

 

H
Badge
I finally made it work using workSpaceRunner: Not quite as elegant as a linked custom transformer, but this was essentially the first thing I could make work - finally being able to deliver the data. 

 

 

A chain of events eventually  pushed me toward WorkSpaceRunner: 

 

 - Some non-FME related connectivity issues with my geodatabase made me save a temporary data set as a FFS-file

 

 - I can't use featureReader to actually read FFS files (weird? YES!). So the obvious solution of a custom transformer with featureReader doesn't really work that well. 

 

 - There are some fine points to making and using custom transformers that takes a bit of training before you truly master them. WorkSpaceRunner is litterary a walk in the park in comparison. 

 

 

 

 

As a part of my ongoing FME education I now have a set of workspaces to make more elegant and improve, reading fresh data from the database and if I can make it work: Use custom transformers... 

 

Reply


Community Stats

28,869
Posts
109,984
Replies
36,672
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.