I just created what is going to be our department's first production FME workspace. Before moving from test to production, however, I would like to know if there is anything I can do to reduce the resources and time it takes to run this simple workspace.
This is the workspace:
I would perhaps replace the three first transformers with clipper transformers to start with
Hi @FMELizard
I can think of a few improvements to make here:
1) Use the AttributeManager to remove unwanted attributes earlier in the workspace, before carrying out overlays. Otherwise you are overlaying data with attributes that aren't necessary.
2) Experiment with different transformers for the overlay. It looks like you are using the PointOnAreaOverlayer. Using the SpatialFilter or Clipper transformers should give you the same results, but may be quicker and more memory efficient.
3) Use the Areas First (PointOnAreaOverlayer), Filters First (SpatialFilter) or Clippers First (Clipper) parameters to speed up processing. You'll need to ensure those features are first - probably by ordering the readers in the Navigator window
4) If it's possible, pre-filter source data using a Where clause. eg if there are some addresses that you know will never overlap a ward, then there is no point reading them. Performance is defined as the rate of useful work carried out. If the data is not useful, then it is impeding performance!
5) Check out this article on performance. Particularly examine your log file to see if there are any clues as to what might be a bottleneck.
The PointOnAreaOverlayer is capable of parallel processing, you may want to check out our article: Parallel Processing.
Looks (from the annotation) like you mixed up the zone and ward polygons. You claim to test ward first, but are using the zones first.
I would perhaps replace the three first transformers with clipper transformers to start with
Hi @FMELizard
I can think of a few improvements to make here:
1) Use the AttributeManager to remove unwanted attributes earlier in the workspace, before carrying out overlays. Otherwise you are overlaying data with attributes that aren't necessary.
2) Experiment with different transformers for the overlay. It looks like you are using the PointOnAreaOverlayer. Using the SpatialFilter or Clipper transformers should give you the same results, but may be quicker and more memory efficient.
3) Use the Areas First (PointOnAreaOverlayer), Filters First (SpatialFilter) or Clippers First (Clipper) parameters to speed up processing. You'll need to ensure those features are first - probably by ordering the readers in the Navigator window
4) If it's possible, pre-filter source data using a Where clause. eg if there are some addresses that you know will never overlap a ward, then there is no point reading them. Performance is defined as the rate of useful work carried out. If the data is not useful, then it is impeding performance!
5) Check out this article on performance. Particularly examine your log file to see if there are any clues as to what might be a bottleneck.
I tried SpatialFilter - filters first with attribute managers added before the filter - this took 5 m 27.6s. I could not get the clipper to work with both zone and ward features - the result only included Zones, no wards. Any tips on using the clipper transformer with two clipper sources?
I only have one reader for all three feature classes because they all come from the same sde. Do I need to add a new reader for adcounty if I want to set a where clause for addresses that are only ZIPCITY = 'TUCSON'?
I experimented that Clipper take less time on FME Desktop 2016 1.2 x64 than in version 2016 1.3 and 2017.
I tried SpatialFilter - filters first with attribute managers added before the filter - this took 5 m 27.6s. I could not get the clipper to work with both zone and ward features - the result only included Zones, no wards. Any tips on using the clipper transformer with two clipper sources?
I only have one reader for all three feature classes because they all come from the same sde. Do I need to add a new reader for adcounty if I want to set a where clause for addresses that are only ZIPCITY = 'TUCSON'?
I don't see that the Clipper shouldn't work. Just make sure that the parameters are right in the second Clipper. ie Merge Attributes is on (sorry, but it's often the obvious things that we overlook) and Clipper Type is not Single Clipper. It's also a scenario where I would try to debug with Inspector transformers to check the input going into the second Clipper, although that's really only confirming where the problem is, not always what it is.
I don't think you need another reader. Check the feature type (brown object on the canvas) for a where clause. If there is a where clause there, then you can use that and it won't affect the other tables. But if the where clause is on the reader in the Navigator window, then it will affect all tables.
Hope this helps.
Mark
Since you are only interested in the address features that intersect with the WARD or the ZONE feature class, I could also image that you use a FeatureReader to do so. Within the FeatureReader you can set a SpatialFilter and that way you already read only the features that are interesting for you and it might save you the reading of a lot of features you are not interested in.
And I am not sure how the WARD and ZONE classes compare to each other, but I can also imagine that you intersect the both first, where I assume that the number of features of both aren't that large. The new areas contain the full attributes you are interested in per zone/ward combination. You can clean them up and finally use those as the SpatialFilter in the FeatureReader.
The biggest advantage in this scenario is that the largest number of features (the addresses) will be read just before they are sent to the Writer.
Hope this helps you getting a better performance.