+1I'm calling 6 separate StatisticsCalculator transformers, and they each take a (relatively) long time to run. All I'm calculating each time is a min, max and count attribute so this seems like overkill.
Is there an easy alternative I'm missing? Is there a way to use a math function against the entire dataset for example?
+28Depending on your data, it might also be quicker to sort on your attribute and sample the first and/or last feature for the min/max.
The quickest way I've found to get a total count attribute is to use the Aggregator (with List) with a count attribute, followed by a deaggregator to restore the original features.
If your data source is a SQL database, I think it would be hard to beat the performance of a SQLExecutor with something like:
select
max(my_value) as my_max,
min(my_value) as my_min,
sum(my_value) as my_sum
from
my_table
where
...If your data source is a SQL database, I think it would be hard to beat the performance of a SQLExecutor with something like:
select
max(my_value) as my_max,
min(my_value) as my_min,
sum(my_value) as my_sum
from
my_table
where
...
+28Depending on your data, it might also be quicker to sort on your attribute and sample the first and/or last feature for the min/max.
The quickest way I've found to get a total count attribute is to use the Aggregator (with List) with a count attribute, followed by a deaggregator to restore the original features.
class FeatureProcessor(object):
def __init__(self):
self.Total = 0
self.Features=[]
def input(self,feature):
self.Total+=1
self.Features.append(feature)
def close(self):
for feature in self.Features:
feature.setAttribute("TotalCount", self.Total)
self.pyoutput(feature)
+17
Hi @ld, before thinking of an alternative, I'd like to know why you need to use 6 StatisticsCalculators. If you intend to perform statistics calculation simultaneously on 6 attributes in the same feature type, just a single StatisticsCalculator would be enough.
+1Hi @ld, before thinking of an alternative, I'd like to know why you need to use 6 StatisticsCalculators. If you intend to perform statistics calculation simultaneously on 6 attributes in the same feature type, just a single StatisticsCalculator would be enough.
+1If your data source is a SQL database, I think it would be hard to beat the performance of a SQLExecutor with something like:
select
max(my_value) as my_max,
min(my_value) as my_min,
sum(my_value) as my_sum
from
my_table
where
...
+1
class FeatureProcessor(object):
def __init__(self):
self.Total = 0
self.Features=[]
def input(self,feature):
self.Total+=1
self.Features.append(feature)
def close(self):
for feature in self.Features:
feature.setAttribute("TotalCount", self.Total)
self.pyoutput(feature)
Thank you, why didn't I think of Python? I will try this.
Hi @ld, before thinking of an alternative, I'd like to know why you need to use 6 StatisticsCalculators. If you intend to perform statistics calculation simultaneously on 6 attributes in the same feature type, just a single StatisticsCalculator would be enough.
I provide another way. If you are using FME 2018.0, you can use the Sampler (sampling the last 1 feature) effectively to do that. This custom transformer implementation might be useful.
quick-statisticscalculator.fmw (FME 2018.0.0.3)
Hi @ld, before thinking of an alternative, I'd like to know why you need to use 6 StatisticsCalculators. If you intend to perform statistics calculation simultaneously on 6 attributes in the same feature type, just a single StatisticsCalculator would be enough.
quick-statisticscalculator-2.fmw (FME 2018.0.0.3)
+1I provide another way. If you are using FME 2018.0, you can use the Sampler (sampling the last 1 feature) effectively to do that. This custom transformer implementation might be useful.
quick-statisticscalculator.fmw (FME 2018.0.0.3)
+28Thank you, why didn't I think of Python? I will try this.
The above code assumes that you want the values added to every feature, if you just want a summary feature instead, then you don't need to build the feature list, and can just output a single feature in the close method. This will be a lot more efficient, since all the features do not need to be stored in memory.
+28I provide another way. If you are using FME 2018.0, you can use the Sampler (sampling the last 1 feature) effectively to do that. This custom transformer implementation might be useful.
quick-statisticscalculator.fmw (FME 2018.0.0.3)
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
