Question

Anyone notice that spatial index is being built on a non-geometric feature class using 2019.1?

P
Badge

I'm using the feature caching Run option.

I'm reading a postgresql table (no geometry) not a postGIS table. Unfortunately it is rather lengthy. For each record, I'm getting Adding to Spatial Index (paraphrasing) and after the last record a rather long pause before the processing progresses.

This might be fine for small record sizes but it is crippling for larger ones.

The bottom line is shouldn't there be no spatial index created as long as the reader has no geometry in any case - feature caching or not?

Here are my details:

Edition: FME Database Edition (node locked-crc)

 

Version: FME(R) 2019.1.0.0 (20190704 - Build 19604 - WIN64)

 

Locale: en_US

 

Codepage: 1252 (ANSI - Latin I)

 

Registration Key: 1-542-353-546

 

Customer Number:

 

Home Folder: C:\\Program Files\\FME\\Beta\\

 

Operating System: Microsoft Windows 10 64-bit (Build 17763)

 

Thanks for any assistance or bug fixing.

12 replies

mark2atsafe
Rank
Userlevel 4
Badge +25

Just to confirm, you are using the Postgres writer, not the PostGIS one, right? Is it also creating a geometry column in the output?

P
Badge

Just to confirm, you are using the Postgres writer, not the PostGIS one, right? Is it also creating a geometry column in the output?

Yes, Postgres (no geom column), not PostGIS.

fmelizard
Rank
Userlevel 4
Badge +13

Could you post the log messages -- I'm wondering if this is coming from our internal feature caching storage and is unrelated to your source. Having the full log or at least the section in question could help. I agree there shouldn't be a spatial index being created, but seeing what is going on will help alot to figure out what is actually happening.

P
Badge

Test_noFeatureCache.log.txtTest-withFeatureCache.log.txtHere are the log files. It looks like it only happens when enabling Feature Caching. Postgres table attributes are being joined with Shapefile but Postgres rows are also being spatially indexed (how?). When a lot (>13M) of rows are being read from the Postgres table, then this takes quite some time...

P
Badge

Test_noFeatureCache.log.txtTest-withFeatureCache.log.txtHere are the log files. It looks like it only happens when enabling Feature Caching. Postgres table attributes are being joined with Shapefile but Postgres rows are also being spatially indexed (how?). When a lot (>13M) of rows are being read from the Postgres table, then this takes quite some time...

Here's a full run including 15M+ postgres rows.

Excerpt from the log:

Attached is the "Mega" log.Test-fullRunWithFeatureCache.log.txt

Please let me know if I'm not grasping something here. A.) there is a substantial lag at the end of adding features to a spatial index. And of course, the spatial index is invalid for a non geometric table.

mark2atsafe
Rank
Userlevel 4
Badge +25

Here's a full run including 15M+ postgres rows.

Excerpt from the log:

Attached is the "Mega" log.Test-fullRunWithFeatureCache.log.txt

Please let me know if I'm not grasping something here. A.) there is a substantial lag at the end of adding features to a spatial index. And of course, the spatial index is invalid for a non geometric table.

Thanks for the logs. That clears it up a lot. I can recreate the issue now. I'll get in touch with the developers and let you know what they say,

Regards,

Mark

mark2atsafe
Rank
Userlevel 4
Badge +25

Here's a full run including 15M+ postgres rows.

Excerpt from the log:

Attached is the "Mega" log.Test-fullRunWithFeatureCache.log.txt

Please let me know if I'm not grasping something here. A.) there is a substantial lag at the end of adding features to a spatial index. And of course, the spatial index is invalid for a non geometric table.

For your reference this is filed with the developers under the ID FMEENGINE-60953.

mark2atsafe
Rank
Userlevel 4
Badge +25

Here's a full run including 15M+ postgres rows.

Excerpt from the log:

Attached is the "Mega" log.Test-fullRunWithFeatureCache.log.txt

Please let me know if I'm not grasping something here. A.) there is a substantial lag at the end of adding features to a spatial index. And of course, the spatial index is invalid for a non geometric table.

Is there any way you can share this workspace with us at Safe? I can replicate the messages easily enough, but I can't create a workspace where the time taken under those messages is as long as you experience. I'm using the same number of features, but I just can't get the same results. I think the best way is for us to try and duplicate your exact scenario, here at Safe. Safe.com/support would be the best place to create a case, and submit your workspace. I'm really hoping that we don't need the data as well. I think if we can inspect the workspace then that will be enough to put us on the right track. Thanks for your patience.

P
Badge

Is there any way you can share this workspace with us at Safe? I can replicate the messages easily enough, but I can't create a workspace where the time taken under those messages is as long as you experience. I'm using the same number of features, but I just can't get the same results. I think the best way is for us to try and duplicate your exact scenario, here at Safe. Safe.com/support would be the best place to create a case, and submit your workspace. I'm really hoping that we don't need the data as well. I think if we can inspect the workspace then that will be enough to put us on the right track. Thanks for your patience.

Done. The case number is C148442 . The workspace is not complex. I attached it to the case. We deal with a huge amount of records on almost a daily basis, there is always a need for shorten processing time (like I said, there were >13M rows read and then a significant number of time was spent creating a Spatial Index where none was necessary). Let me know if I can help some more...

carmijo
Rank
Badge +10

@mark2atsafe​ any updates on this? I have a feature joiner taking in tabular data that is resulting in a message 'Spatial Index: Adding 19530 features'. Is this to be expected. Using FME 2023

mark2atsafe
Rank
Userlevel 4
Badge +25

@mark2atsafe​ any updates on this? I have a feature joiner taking in tabular data that is resulting in a message 'Spatial Index: Adding 19530 features'. Is this to be expected. Using FME 2023

Yes, that's absolutely fine. I think the message is badly worded because it includes tabular as well as spatial data in the index, but it's definitely OK behaviour. You should only need to worry if that message was seen to be taking a substantial amount of time to process. If the log shows it was just a fraction of a second of processing time, then all should be fine.

carmijo
Rank
Badge +10

Yes, that's absolutely fine. I think the message is badly worded because it includes tabular as well as spatial data in the index, but it's definitely OK behaviour. You should only need to worry if that message was seen to be taking a substantial amount of time to process. If the log shows it was just a fraction of a second of processing time, then all should be fine.

I see. My workspace was failing after the message I included, but I've been having some processing issues that I think are mostly tied to my PC's memory capacity. Closing programs and disabling unnecessary sections of the workspace tends to help when I need to run it with feature caching enabled.

 

I installed 2023.1.1 yesterday and the workspace is running great, so in the end, all is good for now. Thanks for your response.

Reply


Community Stats

28,662
Posts
109,327
Replies
36,529
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.