Solved

Cumulative totals

  • 25 November 2012
  • 5 replies
  • 7 views

How do you take a specific attribute from a grouped/sorted set of features, and cumulate the sum feature by feature and store in separate field?
icon

Best answer by fmelizard 25 November 2012, 13:23

View original

5 replies

Userlevel 3
Badge +13
As far as I know, there is currently no single transformer that can acomplish this. As this requires saving some state between features, global variables get involved (unless you want to do some Python or Tcl scripting).

 

 

In the workspace at https://dl.dropbox.com/u/2275900/fmepedia/accumlate.fmw, I set up a small network of transformers that use a global variable named CumulativeCount to accumulate the values in the _creation_instance attribute for a series of features. For each feature, it performs the following sequence of operations:

 

 

1. Get the current value of CumulativeCount into the feature attribute _accumulated

 

2. Create the feature attribute _summed with a value of _accumulated + _creation_instance

 

3. Update the variable CumulativeCount with the value of _summed

 

4. Remove the temporary feature attribute _accumulated

 

 

This effectively keeps an accumulator in the global variable CumulativeCount, which it applies to the _summed attribute in every feature that passes through. 
Userlevel 3
Badge +13
I just realised that my previous solution does not fully answer your question: it does not take grouping/sorting into account.

 

 

I'll have to put some thought into this one. It is pretty straight forward to accomplish with parallel processing or Python/Tcl scripting, but a "pure" solution is probably a better approach.
Userlevel 3
Badge +13

I gave it some thought, and here's what I've come up with.

 

If the volume of features is not too great, we can use a Sorter, a Tester and another VariableSetter or two and a VariableRetriever to accomplish the goal. I'm going to throw in a Counter as well, to ensure that the ordering of features within each group is maintained.

 

The process will be:

 

1) Use the counter to assign a unique serial # to the incoming features

2) Sort the features by the group and serial# attributes

3) Retrieve the previous feature's group from a global variable (LastGroup)

4) Use a Tester to see if the group changed

4a) If so, reset the CumulativeCount variable to 0, and write the group name to the global variable LastGroup.

5) Process the feature as before

6) Clean up the temporary attributes used to store the serial number and LastGroup value.

 

I have encapsulated this approach in the workspace at https://dl.dropbox.com/u/2275900/fmepedia/accumlate2.fmw .

 

The first thing I did was to move the processing into a custom transformer ("Accumulator") to make separate it from the rest of the workspace. This just cleans things up a bit. After that I duplicated the Creator, directing two streams of features to my Accumulator. I give each feature an attribute named _groupnum to specify which stream they come from.

 

In the custom transformer, I added a Counter ("SerialCounter"), which adds an increasing attribute _serialnum to the features to maintain relative ordering of features within each group. This is followed by a Sorter, which sorts by both _groupnum and _serialnum. The output from this sorter is arranged by group, and the ordering within each group is maintained by the sorting of the _serialnum attribute.

 

The second line of my custom transformer looks at the group of the previous feature processed (stored in the LastGroup variable). If it has changed (detected by the Tester), we reset our CumulativeCount variable to 0, and update the LastGroup variable to reflect the new feature.

 

The rest of the transformer provides the same processing as before, with the addition of removing the new temporary attributes _serialnum and _lastgroup.

I like to use a FeatureMerger to merge an attribute such as a maximum count, sum or other attribute onto a group of features.  Mainly because it responds to group by and can be made to apply one features attribute onto a group of features.

 

For your question, I would copy off a stream of features into Aggregator which can create the cumulative sum including group by, then FeatureMerge the Aggregator result back onto the main stream of features using group by again.  Trick is to make the Join attribute something in common so that they all merge properly and to remove the suppliers unnecesary attributes.

 

True to say, not one transformer, four or so, tho' might accomplish the task.
Thanks all.  I was able to solve the problem using combination of answers. 

Reply