Using AttributeManager to update 86,000 (existing) records using APN for key. Is this process optimized by FME? It takes a long time, and I assume it is N*N time. (Does the Reader iterate through each record, one at a time, then searching through all the records in the Writer to find the matching key?)
Can you post a screenshot of your workspace? Specifically, how are you doing the matching?
Here is screen shots. Parameters on the Writer:
Feature Operation: Update
Table Handling: Use Existing
Update Spatial Column(s): No
Match Columns: parcel
or
WHERE Clause: "parcel"='@Value(parcel)'
Here is screen shots. Parameters on the Writer:
Feature Operation: Update
Table Handling: Use Existing
Update Spatial Column(s): No
Match Columns: parcel
or
WHERE Clause: "parcel"='@Value(parcel)'
I suspect that since parcel is unique for every feature you're essentially sending 86000 queries to that geodatabase. Have you tried the match columns option and if so, does that go faster?
I suspect that since parcel is unique for every feature you're essentially sending 86000 queries to that geodatabase. Have you tried the match columns option and if so, does that go faster?
Will try the match column option on file geodatabase, and report back here when done.
I suspect that since parcel is unique for every feature you're essentially sending 86000 queries to that geodatabase. Have you tried the match columns option and if so, does that go faster?
Here is result, using a file geodatabase on a network path, e.g. \\\\server1\\GISData\\FMEData.gdb
Translation was SUCCESSFUL with 12 warning(s) (86652 feature(s) output)
FME Session Duration: 6 hours 55 minutes 2 seconds. (CPU: 7387.8s user, 15737.0s system)
END - ProcessID: 25168, peak process memory usage: 206772 kB, current process memory usage: 109080 kB
Translation was SUCCESSFUL
I suspect that since parcel is unique for every feature you're essentially sending 86000 queries to that geodatabase. Have you tried the match columns option and if so, does that go faster?
Here are the warnings. Maybe I need to adjust for this? (Feature Caching is ON?)
Feature Caching is ON
The workspace may run slower because features are being recorded on all output ports.
No rows matched UPDATE query to table/feature class 'OwnerInfo_TEST' where parcel = N'231232020'
No rows matched UPDATE query to table/feature class 'OwnerInfo_TEST' where parcel = N'151380055'
No rows matched UPDATE query to table/feature class 'OwnerInfo_TEST' where parcel = N'172390006'
No rows matched UPDATE query to table/feature class 'OwnerInfo_TEST' where parcel = N'215231016'
No rows matched UPDATE query to table/feature class 'OwnerInfo_TEST' where parcel = N'146141078'
-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-
-~ ~-
-~ Feature caches have been recorded at every stage of the translation. ~-
-~ To inspect the recorded features, ~-
-~ click the feature cache icons next to the ports. ~-
-~ ~-
-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-
I suspect that since parcel is unique for every feature you're essentially sending 86000 queries to that geodatabase. Have you tried the match columns option and if so, does that go faster?
I suspect that since parcel is unique for every feature you're essentially sending 86000 queries to that geodatabase. Have you tried the match columns option and if so, does that go faster?
There were only those five mismatches, and expected. I will be able to try without feature caching later today. Thank you.
I suspect that since parcel is unique for every feature you're essentially sending 86000 queries to that geodatabase. Have you tried the match columns option and if so, does that go faster?
Yes. I will use a local Windows drive (using a SSD).
I suspect that since parcel is unique for every feature you're essentially sending 86000 queries to that geodatabase. Have you tried the match columns option and if so, does that go faster?
Here is results: a little less overall time, but still a lot of system time. I'm curious enough to try on C:\\ instead of D:\\ and will post results on Monday. (Those 5 warnings are the expected non-match records).
Translation was SUCCESSFUL with 5 warning(s) (86652 feature(s) output)
FME Session Duration: 5 hours 20 minutes 56.8 seconds. (CPU: 7000.0s user, 12215.5s system)
END - ProcessID: 17056, peak process memory usage: 256780 kB, current process memory usage: 109360 kB
Translation was SUCCESSFUL
I suspect that since parcel is unique for every feature you're essentially sending 86000 queries to that geodatabase. Have you tried the match columns option and if so, does that go faster?
Here are more results using local C:\\ drive, slightly slower than D:\\ (but I was also working on other things at same time).
Translation was SUCCESSFUL with 5 warning(s) (86652 feature(s) output)
FME Session Duration: 6 hours 8 minutes 29.9 seconds. (CPU: 7508.1s user, 14629.1s system)
END - ProcessID: 19268, peak process memory usage: 256484 kB, current process memory usage: 109056 kB
Translation was SUCCESSFUL