Skip to main content
  • EN

I'm using the feature caching Run option.

I'm reading a postgresql table (no geometry) not a postGIS table. Unfortunately it is rather lengthy. For each record, I'm getting Adding to Spatial Index (paraphrasing) and after the last record a rather long pause before the processing progresses.

This might be fine for small record sizes but it is crippling for larger ones.

The bottom line is shouldn't there be no spatial index created as long as the reader has no geometry in any case - feature caching or not?

Here are my details:

Edition: FME Database Edition (node locked-crc)

 

Version: FME(R) 2019.1.0.0 (20190704 - Build 19604 - WIN64)

 

Locale: en_US

 

Codepage: 1252 (ANSI - Latin I)

 

Registration Key: 1-542-353-546

 

Customer Number:

 

Home Folder: C:\\Program Files\\FME\\Beta\\

 

Operating System: Microsoft Windows 10 64-bit (Build 17763)

 

Thanks for any assistance or bug fixing.

Just to confirm, you are using the Postgres writer, not the PostGIS one, right? Is it also creating a geometry column in the output?

Just to confirm, you are using the Postgres writer, not the PostGIS one, right? Is it also creating a geometry column in the output?

Yes, Postgres (no geom column), not PostGIS.

Could you post the log messages -- I'm wondering if this is coming from our internal feature caching storage and is unrelated to your source. Having the full log or at least the section in question could help. I agree there shouldn't be a spatial index being created, but seeing what is going on will help alot to figure out what is actually happening.

Test_noFeatureCache.log.txtTest-withFeatureCache.log.txtHere are the log files. It looks like it only happens when enabling Feature Caching. Postgres table attributes are being joined with Shapefile but Postgres rows are also being spatially indexed (how?). When a lot (>13M) of rows are being read from the Postgres table, then this takes quite some time...

Test_noFeatureCache.log.txtTest-withFeatureCache.log.txtHere are the log files. It looks like it only happens when enabling Feature Caching. Postgres table attributes are being joined with Shapefile but Postgres rows are also being spatially indexed (how?). When a lot (>13M) of rows are being read from the Postgres table, then this takes quite some time...

Here's a full run including 15M+ postgres rows.

Excerpt from the log:

Attached is the "Mega" log.Test-fullRunWithFeatureCache.log.txt

Please let me know if I'm not grasping something here. A.) there is a substantial lag at the end of adding features to a spatial index. And of course, the spatial index is invalid for a non geometric table.

Here's a full run including 15M+ postgres rows.

Excerpt from the log:

Attached is the "Mega" log.Test-fullRunWithFeatureCache.log.txt

Please let me know if I'm not grasping something here. A.) there is a substantial lag at the end of adding features to a spatial index. And of course, the spatial index is invalid for a non geometric table.

Thanks for the logs. That clears it up a lot. I can recreate the issue now. I'll get in touch with the developers and let you know what they say,

Regards,

Mark

Here's a full run including 15M+ postgres rows.

Excerpt from the log:

Attached is the "Mega" log.Test-fullRunWithFeatureCache.log.txt

Please let me know if I'm not grasping something here. A.) there is a substantial lag at the end of adding features to a spatial index. And of course, the spatial index is invalid for a non geometric table.

For your reference this is filed with the developers under the ID FMEENGINE-60953.

Here's a full run including 15M+ postgres rows.

Excerpt from the log:

Attached is the "Mega" log.Test-fullRunWithFeatureCache.log.txt

Please let me know if I'm not grasping something here. A.) there is a substantial lag at the end of adding features to a spatial index. And of course, the spatial index is invalid for a non geometric table.

Is there any way you can share this workspace with us at Safe? I can replicate the messages easily enough, but I can't create a workspace where the time taken under those messages is as long as you experience. I'm using the same number of features, but I just can't get the same results. I think the best way is for us to try and duplicate your exact scenario, here at Safe. Safe.com/support would be the best place to create a case, and submit your workspace. I'm really hoping that we don't need the data as well. I think if we can inspect the workspace then that will be enough to put us on the right track. Thanks for your patience.

Is there any way you can share this workspace with us at Safe? I can replicate the messages easily enough, but I can't create a workspace where the time taken under those messages is as long as you experience. I'm using the same number of features, but I just can't get the same results. I think the best way is for us to try and duplicate your exact scenario, here at Safe. Safe.com/support would be the best place to create a case, and submit your workspace. I'm really hoping that we don't need the data as well. I think if we can inspect the workspace then that will be enough to put us on the right track. Thanks for your patience.

Done. The case number is C148442 . The workspace is not complex. I attached it to the case. We deal with a huge amount of records on almost a daily basis, there is always a need for shorten processing time (like I said, there were >13M rows read and then a significant number of time was spent creating a Spatial Index where none was necessary). Let me know if I can help some more...

@mark2atsafe​ any updates on this? I have a feature joiner taking in tabular data that is resulting in a message 'Spatial Index: Adding 19530 features'. Is this to be expected. Using FME 2023

@mark2atsafe​ any updates on this? I have a feature joiner taking in tabular data that is resulting in a message 'Spatial Index: Adding 19530 features'. Is this to be expected. Using FME 2023

Yes, that's absolutely fine. I think the message is badly worded because it includes tabular as well as spatial data in the index, but it's definitely OK behaviour. You should only need to worry if that message was seen to be taking a substantial amount of time to process. If the log shows it was just a fraction of a second of processing time, then all should be fine.

Yes, that's absolutely fine. I think the message is badly worded because it includes tabular as well as spatial data in the index, but it's definitely OK behaviour. You should only need to worry if that message was seen to be taking a substantial amount of time to process. If the log shows it was just a fraction of a second of processing time, then all should be fine.

I see. My workspace was failing after the message I included, but I've been having some processing issues that I think are mostly tied to my PC's memory capacity. Closing programs and disabling unnecessary sections of the workspace tends to help when I need to run it with feature caching enabled.

 

I installed 2023.1.1 yesterday and the workspace is running great, so in the end, all is good for now. Thanks for your response.

Reply


Community Stats

30,610
Posts
115,984
Replies
38,247
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.