Hey,
Since a day or two I can no longer read my data. I always get the same error, even when I try it in another workspace. Does anyone know what it could be?
I use FME 2021.0 and i'm trying to read a postgis table.
Thx
Nathalie
|ERROR |Error executing SQL command ('FETCH 1000 IN "centraal_v_streng_export_flat_crsr"'): 'Error connecting to database' |A fatal error has occurred. Check the logfile above for deta
Userlevel 4
+26
- Influencer
- 1636 replies
-
16 June 2021
The message suggests a connection issue. You should check if you can still connect to the database using a different tool (like PGAdmin)from the same computer you are running FME from. If you can connect with PGAdmin then you should check the connection details in FME to make sure they are correct.
If you can't connect to the database with PGAdmin then you will need to contact your IT
Userlevel 1
+5
- Contributor
- 69 replies
-
16 June 2021
Thx but i think it isn't a connection issue. I can connect to the database, i have tested that and i'm using the same connection already for years, i have no problems reading other features in the same "centraal" database.
Userlevel 1
+5
- Contributor
- 69 replies
-
16 June 2021
2021-06-16 12:57:09| 24.3| 4.1|ERROR |Error executing SQL command ('FETCH 1000 IN "centraal_v_streng_export_flat_crsr"'): 'Error connecting to database'
2021-06-16 12:57:10| 24.4| 0.0|ERROR |A fatal error has occurred. Check the logfile above for details
Error executing SQL command ('FETCH 1000 IN "centraal_v_streng_export_flat_crsr"'): 'ERROR: canceling statement due to conflict with recovery
DETAIL: User query might have needed to see row versions that must be removed.
'
A fatal error has occurred. Check the logfile above for details
A fatal error has occurred. Check the logfile above for details
Failed to specify the feature index as constraints on the reader
Error executing SQL command ('FETCH 1000 IN "centraal_v_streng_export_flat_crsr"'): 'ERROR: canceling statement due to conflict with recovery
DETAIL: User query might have needed to see row versions that must be removed.
'
A fatal error has occurred. Check the logfile above for details
A fatal error has occurred. Check the logfile above for details
Failed to specify the feature index as constraints on the reader
Userlevel 4
+26
- Influencer
- 1636 replies
-
16 June 2021
Hmmm
ERROR: canceling statement due to conflict with recovery
DETAIL: User query might have needed to see row versions that must be removed.
This is the error. Searching on google it looks complicated. Sorry I'm not much more help here. Are you able to run the same 'FETCH' query from PGAdmin or with another tool?
Userlevel 1
+5
- Contributor
- 69 replies
-
16 June 2021
Yes, with PGAdmin it works... i can also read the table in any GIS application. I really dont know what the problem is. It must be an FME issue.
Userlevel 4
+26
- Influencer
- 1636 replies
-
16 June 2021
Yes, with PGAdmin it works... i can also read the table in any GIS application. I really dont know what the problem is. It must be an FME issue.
Totally strange. Perhaps @steveatsafe is able to help with this one?
+2
@nath I would recommend that you file a support case with your local FME provider. They can then walk through the problem with you and try and get to the bottom of this. If you do find a solution that way - please report it back here if possible.
You mentioned that: "Since a day or two I can no longer read my data" : does this mean that this FETCH used to work in FME but no longer works? If that's the case then you have to determine what has changed on your environment.
The FETCH retrieves rows from a query using a cursor. I suspect cursors are not supported by FME and I'll investigate that a bit further.
+11
- Safer
- 373 replies
-
18 June 2021
@virtualcitymatt Thanks for the ping.
In addition to @Mark Stoakes suggestions... I'm assuming you are using the PostGIS Reader? (Assuming the PostgreSQL Reader produces the same
Could you test with the SQLCreator (PostGIS)... what's the outcome?
Could you test with the JDBC Format?
You can download the driver from https://jdbc.postgresql.org/download.html and copy to C:/<user>/<MyDocuments>/FME/Plugins/Java/ and restart FME Workbench.
Create a new JDBC Database Connection and point to the Postgres database.
Try the JDBC Format Reader... what's the outcome?
In my own opinion... depending on how long the workspace runs for it could be dropping the connection. Is there a timeout or a connection limit that has been added or created recently?
After that, is there some data that FME is not handling? Can you try to duplicate the table to a new table (PGAdmin), bring the first 1/4 of the data into the new table... can you read it with FME? Then work on the next chunk of records data and so on... perhaps there's a recently updated record that FME is failing on.
Userlevel 1
+5
- Contributor
- 69 replies
-
28 June 2021
Hi,
i am still having memory issues and have yet to try your tips as our IT is currently setting up new vpc to run FME on. I will come back to it later. At the moment I can run small datasets and models.
+10
- Contributor
- 42 replies
-
17 August 2021
Thx but i think it isn't a connection issue. I can connect to the database, i have tested that and i'm using the same connection already for years, i have no problems reading other features in the same "centraal" database.
I got the same error here, using a FeatureReader on a Postgis database. Here is the setup:
- Using Postgis Reader, read a list of tables to process
- Use for each feature a FeatureReader to read all features from the table name in the list
- Transform it using SchemaMapper (on a Postgis table...)
- Write the features read using a postgis writer in dynamics mode
It seems to run fine for about 10 out of 23 tables, then crashes, complaining (at line 39):
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_ontwerp_deelname_clusters_crsr" cursor for select "naam","clusterid","primarykey","deelname_scenario","toelichting_deelname","tips_omtrek","input_bd","snelmt","peakshaving_percentage","cityplan","scenario","datum_versie","mgt_checksum","mgt_insert_date","mgt_job_id","mgt_last_update_date","mgt_tijdstempel","mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."ontwerp_deelname_clusters"'
Reading POSTGIS table: 'lab_cityplannen.ontwerp_deelname_clusters'...
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_21
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Merged 1 schema features read from 1 datasets into 1 resulting feature types
SchemaMapper_Mapper (SchemaMappingFactory): Splitting bulk mode features into features
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_22
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Performing query against POSTGIS dataset `geodata_dev_svc_fme'
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_23
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_output_clustersmetalles_crsr" cursor for select "clusterid","deelname_scenario",...,"mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."output_clustersmetalles"'
Reading POSTGIS table: 'lab_cityplannen.output_clustersmetalles'...
Error executing SQL command ('FETCH 10000 IN "lab_cityplannen_output_clustersmetalles_crsr"'): 'Error connecting to database'
A fatal error has occurred. Check the logfile above for details
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Storing feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
The below feature caused the translation to be terminated
Storing feature(s) to FME feature store file `C:\Users\503898\Documents\Klus\User Story 53842 Herbruikbare FME template workspace om (historie)tabel Snowflake in de spoelen\2_execute_copy_to_pg_log.ffs'
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `FEATURE_TYPE'
Attribute(string) : `_reader_error' has value `A fatal error has occurred. Check the logfile above for details'
Attribute(string) : `fme_feature_type' has value `stg_open.mgt_tables_to_load'
Attribute(string) : `fme_geometry' has value `fme_undefined'
Attribute(entangled: string): `fme_type' has value `fme_no_geom'
entangled to postgis_type
Attribute(encoded: UTF-8) : `mgt_active' has value `T'
Attribute(encoded: UTF-8) : `mgt_category' has value `CITYPLAN'
Attribute(encoded: UTF-16LE): `mgt_insert_date' is <null>
Attribute(encoded: UTF-16LE): `mgt_last_update_date' is <null>
Attribute(encoded: UTF-8) : `mgt_process_delta' has value `F'
Attribute(encoded: UTF-8) : `mgt_type_verlading' has value `pg_to_pg'
Attribute(32 bit integer) : `multi_reader_full_id' has value `0'
Attribute(32 bit integer) : `multi_reader_id' has value `0'
Attribute(string) : `multi_reader_keyword' has value `POSTGIS_1'
Attribute(string) : `multi_reader_type' has value `POSTGIS'
Attribute(string) : `postgis_type' has value `postgis_none'
Attribute(encoded: UTF-8) : `src_table' has value `lab_cityplannen.output_clustersmetalles'
Coordinate System: `'
Geometry Type: IFMENull
===========================================================================
FeatureReader_<Rejected> (TeeFactory): FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader (QueryFactory): A fatal error has occurred. Check the logfile above for details
Stored 6425 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector_2.ffs'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Stored 1 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
A fatal error has occurred. Check the logfile above for details
+10
- Contributor
- 42 replies
-
17 August 2021
I got the same error here, using a FeatureReader on a Postgis database. Here is the setup:
- Using Postgis Reader, read a list of tables to process
- Use for each feature a FeatureReader to read all features from the table name in the list
- Transform it using SchemaMapper (on a Postgis table...)
- Write the features read using a postgis writer in dynamics mode
It seems to run fine for about 10 out of 23 tables, then crashes, complaining (at line 39):
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_ontwerp_deelname_clusters_crsr" cursor for select "naam","clusterid","primarykey","deelname_scenario","toelichting_deelname","tips_omtrek","input_bd","snelmt","peakshaving_percentage","cityplan","scenario","datum_versie","mgt_checksum","mgt_insert_date","mgt_job_id","mgt_last_update_date","mgt_tijdstempel","mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."ontwerp_deelname_clusters"'
Reading POSTGIS table: 'lab_cityplannen.ontwerp_deelname_clusters'...
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_21
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Merged 1 schema features read from 1 datasets into 1 resulting feature types
SchemaMapper_Mapper (SchemaMappingFactory): Splitting bulk mode features into features
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_22
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Performing query against POSTGIS dataset `geodata_dev_svc_fme'
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_23
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_output_clustersmetalles_crsr" cursor for select "clusterid","deelname_scenario",...,"mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."output_clustersmetalles"'
Reading POSTGIS table: 'lab_cityplannen.output_clustersmetalles'...
Error executing SQL command ('FETCH 10000 IN "lab_cityplannen_output_clustersmetalles_crsr"'): 'Error connecting to database'
A fatal error has occurred. Check the logfile above for details
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Storing feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
The below feature caused the translation to be terminated
Storing feature(s) to FME feature store file `C:\Users\503898\Documents\Klus\User Story 53842 Herbruikbare FME template workspace om (historie)tabel Snowflake in de spoelen\2_execute_copy_to_pg_log.ffs'
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `FEATURE_TYPE'
Attribute(string) : `_reader_error' has value `A fatal error has occurred. Check the logfile above for details'
Attribute(string) : `fme_feature_type' has value `stg_open.mgt_tables_to_load'
Attribute(string) : `fme_geometry' has value `fme_undefined'
Attribute(entangled: string): `fme_type' has value `fme_no_geom'
entangled to postgis_type
Attribute(encoded: UTF-8) : `mgt_active' has value `T'
Attribute(encoded: UTF-8) : `mgt_category' has value `CITYPLAN'
Attribute(encoded: UTF-16LE): `mgt_insert_date' is <null>
Attribute(encoded: UTF-16LE): `mgt_last_update_date' is <null>
Attribute(encoded: UTF-8) : `mgt_process_delta' has value `F'
Attribute(encoded: UTF-8) : `mgt_type_verlading' has value `pg_to_pg'
Attribute(32 bit integer) : `multi_reader_full_id' has value `0'
Attribute(32 bit integer) : `multi_reader_id' has value `0'
Attribute(string) : `multi_reader_keyword' has value `POSTGIS_1'
Attribute(string) : `multi_reader_type' has value `POSTGIS'
Attribute(string) : `postgis_type' has value `postgis_none'
Attribute(encoded: UTF-8) : `src_table' has value `lab_cityplannen.output_clustersmetalles'
Coordinate System: `'
Geometry Type: IFMENull
===========================================================================
FeatureReader_<Rejected> (TeeFactory): FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader (QueryFactory): A fatal error has occurred. Check the logfile above for details
Stored 6425 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector_2.ffs'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Stored 1 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
A fatal error has occurred. Check the logfile above for details
Well, did some more testing. Situation above was running into FME Data Inspector. After changing the SchemaMapper source from postgis into Excel, it ran fine. After that running again into the postgis database (dynamics mode writer), it stopped at table #6 (no more action on the Postgis reader, it seemed). Removing all result tables and running again, it crashes at table #9, saying:
Committed transaction at feature '142000'
SchemaMapper_Mapper (SchemaMappingFactory): Processed 26123 of 100000 features
Committed transaction at feature '143000'
Committed transaction at feature '144000'
Committed transaction at feature '145000'
SchemaMapper_Mapper (SchemaMappingFactory): Processed 30001 of 100000 features
Committed transaction at feature '146000'
Bulk copy failed on table 'dl_open.da_cityplannen_output_aansluitingen' using delimiter ':'. Error was 'server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
'
POSTGIS writer: A fatal error has occurred. Check the logfile above for details
... Last line repeated 2 times ...
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
FeatureReader (QueryFactory): A fatal error has occurred. Check the logfile above for details
Bulk copy failed on table 'dl_open.da_cityplannen_output_aansluitingen' using delimiter ':'. Error was 'SSL error: bad length
'
POSTGIS writer: A fatal error has occurred. Check the logfile above for details
POSTGIS writer: A fatal error has occurred. Check the logfile above for details
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
A fatal error has occurred. Check the logfile above for details
Translation FAILED with 9 error(s) and 3 warning(s) (146134 feature(s) output)
FME Session Duration: 3 minutes 14.5 seconds. (CPU: 167.7s user, 7.8s system)
END - ProcessID: 10988, peak process memory usage: 1095352 kB, current process memory usage: 113288 kB
A fatal error has occurred. Check the logfile above for details
Program Terminating
Translation FAILED.What struck me is that by default postgis reader uses 10000 features/blockread, and the postgis writer uses only 1000 features/blockwrite. Pumping up all blocksizes for the three postgis readers/writers to 100.000 gives the same error after processing 9 tables. Shrinking them down to 100 gives me the "Error executing SQL command ('FETCH 10000 IN" again.
+10
- Contributor
- 42 replies
-
17 August 2021
I got the same error here, using a FeatureReader on a Postgis database. Here is the setup:
- Using Postgis Reader, read a list of tables to process
- Use for each feature a FeatureReader to read all features from the table name in the list
- Transform it using SchemaMapper (on a Postgis table...)
- Write the features read using a postgis writer in dynamics mode
It seems to run fine for about 10 out of 23 tables, then crashes, complaining (at line 39):
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_ontwerp_deelname_clusters_crsr" cursor for select "naam","clusterid","primarykey","deelname_scenario","toelichting_deelname","tips_omtrek","input_bd","snelmt","peakshaving_percentage","cityplan","scenario","datum_versie","mgt_checksum","mgt_insert_date","mgt_job_id","mgt_last_update_date","mgt_tijdstempel","mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."ontwerp_deelname_clusters"'
Reading POSTGIS table: 'lab_cityplannen.ontwerp_deelname_clusters'...
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_21
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Merged 1 schema features read from 1 datasets into 1 resulting feature types
SchemaMapper_Mapper (SchemaMappingFactory): Splitting bulk mode features into features
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_22
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Performing query against POSTGIS dataset `geodata_dev_svc_fme'
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_23
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_output_clustersmetalles_crsr" cursor for select "clusterid","deelname_scenario",...,"mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."output_clustersmetalles"'
Reading POSTGIS table: 'lab_cityplannen.output_clustersmetalles'...
Error executing SQL command ('FETCH 10000 IN "lab_cityplannen_output_clustersmetalles_crsr"'): 'Error connecting to database'
A fatal error has occurred. Check the logfile above for details
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Storing feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
The below feature caused the translation to be terminated
Storing feature(s) to FME feature store file `C:\Users\503898\Documents\Klus\User Story 53842 Herbruikbare FME template workspace om (historie)tabel Snowflake in de spoelen\2_execute_copy_to_pg_log.ffs'
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `FEATURE_TYPE'
Attribute(string) : `_reader_error' has value `A fatal error has occurred. Check the logfile above for details'
Attribute(string) : `fme_feature_type' has value `stg_open.mgt_tables_to_load'
Attribute(string) : `fme_geometry' has value `fme_undefined'
Attribute(entangled: string): `fme_type' has value `fme_no_geom'
entangled to postgis_type
Attribute(encoded: UTF-8) : `mgt_active' has value `T'
Attribute(encoded: UTF-8) : `mgt_category' has value `CITYPLAN'
Attribute(encoded: UTF-16LE): `mgt_insert_date' is <null>
Attribute(encoded: UTF-16LE): `mgt_last_update_date' is <null>
Attribute(encoded: UTF-8) : `mgt_process_delta' has value `F'
Attribute(encoded: UTF-8) : `mgt_type_verlading' has value `pg_to_pg'
Attribute(32 bit integer) : `multi_reader_full_id' has value `0'
Attribute(32 bit integer) : `multi_reader_id' has value `0'
Attribute(string) : `multi_reader_keyword' has value `POSTGIS_1'
Attribute(string) : `multi_reader_type' has value `POSTGIS'
Attribute(string) : `postgis_type' has value `postgis_none'
Attribute(encoded: UTF-8) : `src_table' has value `lab_cityplannen.output_clustersmetalles'
Coordinate System: `'
Geometry Type: IFMENull
===========================================================================
FeatureReader_<Rejected> (TeeFactory): FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader (QueryFactory): A fatal error has occurred. Check the logfile above for details
Stored 6425 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector_2.ffs'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Stored 1 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
A fatal error has occurred. Check the logfile above for details
Also enabling the cache timeout on the FeatureReader doesn't seem to change anything. May be I'm tired. :-)
+10
- Contributor
- 42 replies
-
17 August 2021
I got the same error here, using a FeatureReader on a Postgis database. Here is the setup:
- Using Postgis Reader, read a list of tables to process
- Use for each feature a FeatureReader to read all features from the table name in the list
- Transform it using SchemaMapper (on a Postgis table...)
- Write the features read using a postgis writer in dynamics mode
It seems to run fine for about 10 out of 23 tables, then crashes, complaining (at line 39):
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_ontwerp_deelname_clusters_crsr" cursor for select "naam","clusterid","primarykey","deelname_scenario","toelichting_deelname","tips_omtrek","input_bd","snelmt","peakshaving_percentage","cityplan","scenario","datum_versie","mgt_checksum","mgt_insert_date","mgt_job_id","mgt_last_update_date","mgt_tijdstempel","mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."ontwerp_deelname_clusters"'
Reading POSTGIS table: 'lab_cityplannen.ontwerp_deelname_clusters'...
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_21
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Merged 1 schema features read from 1 datasets into 1 resulting feature types
SchemaMapper_Mapper (SchemaMappingFactory): Splitting bulk mode features into features
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_22
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Performing query against POSTGIS dataset `geodata_dev_svc_fme'
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_23
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_output_clustersmetalles_crsr" cursor for select "clusterid","deelname_scenario",...,"mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."output_clustersmetalles"'
Reading POSTGIS table: 'lab_cityplannen.output_clustersmetalles'...
Error executing SQL command ('FETCH 10000 IN "lab_cityplannen_output_clustersmetalles_crsr"'): 'Error connecting to database'
A fatal error has occurred. Check the logfile above for details
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Storing feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
The below feature caused the translation to be terminated
Storing feature(s) to FME feature store file `C:\Users\503898\Documents\Klus\User Story 53842 Herbruikbare FME template workspace om (historie)tabel Snowflake in de spoelen\2_execute_copy_to_pg_log.ffs'
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `FEATURE_TYPE'
Attribute(string) : `_reader_error' has value `A fatal error has occurred. Check the logfile above for details'
Attribute(string) : `fme_feature_type' has value `stg_open.mgt_tables_to_load'
Attribute(string) : `fme_geometry' has value `fme_undefined'
Attribute(entangled: string): `fme_type' has value `fme_no_geom'
entangled to postgis_type
Attribute(encoded: UTF-8) : `mgt_active' has value `T'
Attribute(encoded: UTF-8) : `mgt_category' has value `CITYPLAN'
Attribute(encoded: UTF-16LE): `mgt_insert_date' is <null>
Attribute(encoded: UTF-16LE): `mgt_last_update_date' is <null>
Attribute(encoded: UTF-8) : `mgt_process_delta' has value `F'
Attribute(encoded: UTF-8) : `mgt_type_verlading' has value `pg_to_pg'
Attribute(32 bit integer) : `multi_reader_full_id' has value `0'
Attribute(32 bit integer) : `multi_reader_id' has value `0'
Attribute(string) : `multi_reader_keyword' has value `POSTGIS_1'
Attribute(string) : `multi_reader_type' has value `POSTGIS'
Attribute(string) : `postgis_type' has value `postgis_none'
Attribute(encoded: UTF-8) : `src_table' has value `lab_cityplannen.output_clustersmetalles'
Coordinate System: `'
Geometry Type: IFMENull
===========================================================================
FeatureReader_<Rejected> (TeeFactory): FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader (QueryFactory): A fatal error has occurred. Check the logfile above for details
Stored 6425 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector_2.ffs'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Stored 1 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
A fatal error has occurred. Check the logfile above for details
Still trying, something strange is going on here... I have to remove the created tables before I start a new run, using DBeaver for that. It seems to take an endless time to remove the tables now, and running all my manual table deletes as a background task. It seems DBeaver has some trouble connecting to the database also. Might it be that the postgis database is running out of space or otherwise is too busy handling requests from my FME Desktop?
Userlevel 2
+10
- Safer
- 458 replies
-
20 August 2021
I got the same error here, using a FeatureReader on a Postgis database. Here is the setup:
- Using Postgis Reader, read a list of tables to process
- Use for each feature a FeatureReader to read all features from the table name in the list
- Transform it using SchemaMapper (on a Postgis table...)
- Write the features read using a postgis writer in dynamics mode
It seems to run fine for about 10 out of 23 tables, then crashes, complaining (at line 39):
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_ontwerp_deelname_clusters_crsr" cursor for select "naam","clusterid","primarykey","deelname_scenario","toelichting_deelname","tips_omtrek","input_bd","snelmt","peakshaving_percentage","cityplan","scenario","datum_versie","mgt_checksum","mgt_insert_date","mgt_job_id","mgt_last_update_date","mgt_tijdstempel","mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."ontwerp_deelname_clusters"'
Reading POSTGIS table: 'lab_cityplannen.ontwerp_deelname_clusters'...
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_21
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Merged 1 schema features read from 1 datasets into 1 resulting feature types
SchemaMapper_Mapper (SchemaMappingFactory): Splitting bulk mode features into features
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_22
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Performing query against POSTGIS dataset `geodata_dev_svc_fme'
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_23
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_output_clustersmetalles_crsr" cursor for select "clusterid","deelname_scenario",...,"mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."output_clustersmetalles"'
Reading POSTGIS table: 'lab_cityplannen.output_clustersmetalles'...
Error executing SQL command ('FETCH 10000 IN "lab_cityplannen_output_clustersmetalles_crsr"'): 'Error connecting to database'
A fatal error has occurred. Check the logfile above for details
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Storing feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
The below feature caused the translation to be terminated
Storing feature(s) to FME feature store file `C:\Users\503898\Documents\Klus\User Story 53842 Herbruikbare FME template workspace om (historie)tabel Snowflake in de spoelen\2_execute_copy_to_pg_log.ffs'
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `FEATURE_TYPE'
Attribute(string) : `_reader_error' has value `A fatal error has occurred. Check the logfile above for details'
Attribute(string) : `fme_feature_type' has value `stg_open.mgt_tables_to_load'
Attribute(string) : `fme_geometry' has value `fme_undefined'
Attribute(entangled: string): `fme_type' has value `fme_no_geom'
entangled to postgis_type
Attribute(encoded: UTF-8) : `mgt_active' has value `T'
Attribute(encoded: UTF-8) : `mgt_category' has value `CITYPLAN'
Attribute(encoded: UTF-16LE): `mgt_insert_date' is <null>
Attribute(encoded: UTF-16LE): `mgt_last_update_date' is <null>
Attribute(encoded: UTF-8) : `mgt_process_delta' has value `F'
Attribute(encoded: UTF-8) : `mgt_type_verlading' has value `pg_to_pg'
Attribute(32 bit integer) : `multi_reader_full_id' has value `0'
Attribute(32 bit integer) : `multi_reader_id' has value `0'
Attribute(string) : `multi_reader_keyword' has value `POSTGIS_1'
Attribute(string) : `multi_reader_type' has value `POSTGIS'
Attribute(string) : `postgis_type' has value `postgis_none'
Attribute(encoded: UTF-8) : `src_table' has value `lab_cityplannen.output_clustersmetalles'
Coordinate System: `'
Geometry Type: IFMENull
===========================================================================
FeatureReader_<Rejected> (TeeFactory): FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader (QueryFactory): A fatal error has occurred. Check the logfile above for details
Stored 6425 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector_2.ffs'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Stored 1 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
A fatal error has occurred. Check the logfile above for details
Hi @Helmoet de Zwijger. Chiming in here with some suggestions on how to troubleshoot this.
It might be a good idea to check with your Database Admin to see if anything shows up in the PostgreSQL logs just to ensure that something on the database side is not causing the issue. Additionally you could try to break down the workflow into a Parent and Child workspace (i.e. send a single feature name to the child workspace, it will run and finish) and see if this completes.
In the meantime I would also suggest creating a new Question on the FME Community! Creating a new Question will help bring your issue to the attention of many other FME users who may have experience something similar and would be able to help! In your new Question I would mention this thread.
Userlevel 1
+5
- Contributor
- 69 replies
-
25 August 2021
I got the same error here, using a FeatureReader on a Postgis database. Here is the setup:
- Using Postgis Reader, read a list of tables to process
- Use for each feature a FeatureReader to read all features from the table name in the list
- Transform it using SchemaMapper (on a Postgis table...)
- Write the features read using a postgis writer in dynamics mode
It seems to run fine for about 10 out of 23 tables, then crashes, complaining (at line 39):
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_ontwerp_deelname_clusters_crsr" cursor for select "naam","clusterid","primarykey","deelname_scenario","toelichting_deelname","tips_omtrek","input_bd","snelmt","peakshaving_percentage","cityplan","scenario","datum_versie","mgt_checksum","mgt_insert_date","mgt_job_id","mgt_last_update_date","mgt_tijdstempel","mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."ontwerp_deelname_clusters"'
Reading POSTGIS table: 'lab_cityplannen.ontwerp_deelname_clusters'...
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_21
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Merged 1 schema features read from 1 datasets into 1 resulting feature types
SchemaMapper_Mapper (SchemaMappingFactory): Splitting bulk mode features into features
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_22
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Performing query against POSTGIS dataset `geodata_dev_svc_fme'
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_23
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_output_clustersmetalles_crsr" cursor for select "clusterid","deelname_scenario",...,"mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."output_clustersmetalles"'
Reading POSTGIS table: 'lab_cityplannen.output_clustersmetalles'...
Error executing SQL command ('FETCH 10000 IN "lab_cityplannen_output_clustersmetalles_crsr"'): 'Error connecting to database'
A fatal error has occurred. Check the logfile above for details
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Storing feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
The below feature caused the translation to be terminated
Storing feature(s) to FME feature store file `C:\Users\503898\Documents\Klus\User Story 53842 Herbruikbare FME template workspace om (historie)tabel Snowflake in de spoelen\2_execute_copy_to_pg_log.ffs'
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `FEATURE_TYPE'
Attribute(string) : `_reader_error' has value `A fatal error has occurred. Check the logfile above for details'
Attribute(string) : `fme_feature_type' has value `stg_open.mgt_tables_to_load'
Attribute(string) : `fme_geometry' has value `fme_undefined'
Attribute(entangled: string): `fme_type' has value `fme_no_geom'
entangled to postgis_type
Attribute(encoded: UTF-8) : `mgt_active' has value `T'
Attribute(encoded: UTF-8) : `mgt_category' has value `CITYPLAN'
Attribute(encoded: UTF-16LE): `mgt_insert_date' is <null>
Attribute(encoded: UTF-16LE): `mgt_last_update_date' is <null>
Attribute(encoded: UTF-8) : `mgt_process_delta' has value `F'
Attribute(encoded: UTF-8) : `mgt_type_verlading' has value `pg_to_pg'
Attribute(32 bit integer) : `multi_reader_full_id' has value `0'
Attribute(32 bit integer) : `multi_reader_id' has value `0'
Attribute(string) : `multi_reader_keyword' has value `POSTGIS_1'
Attribute(string) : `multi_reader_type' has value `POSTGIS'
Attribute(string) : `postgis_type' has value `postgis_none'
Attribute(encoded: UTF-8) : `src_table' has value `lab_cityplannen.output_clustersmetalles'
Coordinate System: `'
Geometry Type: IFMENull
===========================================================================
FeatureReader_<Rejected> (TeeFactory): FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader (QueryFactory): A fatal error has occurred. Check the logfile above for details
Stored 6425 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector_2.ffs'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Stored 1 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
A fatal error has occurred. Check the logfile above for details
I am also still struggling with my FME. Have already sat down with our IT department and it is not due to the memory or the post gis itself. I am also still struggling with my FME. Have already sat down with our IT department and it is not due to the memory or the post gis itself. Dividing into a child and parent is not an option because they are very long processes that then have to run at night one after the other. Until a few weeks ago it usually worked, it's something new. I admit it is a very large FME model, but it should be able to run. It doesn't work with or without the cache option. I sometimes also have it with smaller models, sometimes even in the first readers.
Userlevel 2
+10
- Safer
- 458 replies
-
26 August 2021
I got the same error here, using a FeatureReader on a Postgis database. Here is the setup:
- Using Postgis Reader, read a list of tables to process
- Use for each feature a FeatureReader to read all features from the table name in the list
- Transform it using SchemaMapper (on a Postgis table...)
- Write the features read using a postgis writer in dynamics mode
It seems to run fine for about 10 out of 23 tables, then crashes, complaining (at line 39):
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_ontwerp_deelname_clusters_crsr" cursor for select "naam","clusterid","primarykey","deelname_scenario","toelichting_deelname","tips_omtrek","input_bd","snelmt","peakshaving_percentage","cityplan","scenario","datum_versie","mgt_checksum","mgt_insert_date","mgt_job_id","mgt_last_update_date","mgt_tijdstempel","mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."ontwerp_deelname_clusters"'
Reading POSTGIS table: 'lab_cityplannen.ontwerp_deelname_clusters'...
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_21
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Merged 1 schema features read from 1 datasets into 1 resulting feature types
SchemaMapper_Mapper (SchemaMappingFactory): Splitting bulk mode features into features
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_22
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Performing query against POSTGIS dataset `geodata_dev_svc_fme'
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_23
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_output_clustersmetalles_crsr" cursor for select "clusterid","deelname_scenario",...,"mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."output_clustersmetalles"'
Reading POSTGIS table: 'lab_cityplannen.output_clustersmetalles'...
Error executing SQL command ('FETCH 10000 IN "lab_cityplannen_output_clustersmetalles_crsr"'): 'Error connecting to database'
A fatal error has occurred. Check the logfile above for details
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Storing feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
The below feature caused the translation to be terminated
Storing feature(s) to FME feature store file `C:\Users\503898\Documents\Klus\User Story 53842 Herbruikbare FME template workspace om (historie)tabel Snowflake in de spoelen\2_execute_copy_to_pg_log.ffs'
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `FEATURE_TYPE'
Attribute(string) : `_reader_error' has value `A fatal error has occurred. Check the logfile above for details'
Attribute(string) : `fme_feature_type' has value `stg_open.mgt_tables_to_load'
Attribute(string) : `fme_geometry' has value `fme_undefined'
Attribute(entangled: string): `fme_type' has value `fme_no_geom'
entangled to postgis_type
Attribute(encoded: UTF-8) : `mgt_active' has value `T'
Attribute(encoded: UTF-8) : `mgt_category' has value `CITYPLAN'
Attribute(encoded: UTF-16LE): `mgt_insert_date' is <null>
Attribute(encoded: UTF-16LE): `mgt_last_update_date' is <null>
Attribute(encoded: UTF-8) : `mgt_process_delta' has value `F'
Attribute(encoded: UTF-8) : `mgt_type_verlading' has value `pg_to_pg'
Attribute(32 bit integer) : `multi_reader_full_id' has value `0'
Attribute(32 bit integer) : `multi_reader_id' has value `0'
Attribute(string) : `multi_reader_keyword' has value `POSTGIS_1'
Attribute(string) : `multi_reader_type' has value `POSTGIS'
Attribute(string) : `postgis_type' has value `postgis_none'
Attribute(encoded: UTF-8) : `src_table' has value `lab_cityplannen.output_clustersmetalles'
Coordinate System: `'
Geometry Type: IFMENull
===========================================================================
FeatureReader_<Rejected> (TeeFactory): FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader (QueryFactory): A fatal error has occurred. Check the logfile above for details
Stored 6425 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector_2.ffs'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Stored 1 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
A fatal error has occurred. Check the logfile above for details
Hi @nath & @Helmoet de Zwijger. If you’re comfortable with sharing your workspace and log files here, we can try investigating and reproducing the issue on our end to figure out what's causing the issue. Alternatively, if you’re not comfortable sharing your files over the community, you could submit a support case and attach the files to the case: https://community.safe.com/s/submit-case
- Contributor
- 8 replies
-
22 September 2021
I had the same problem and after talking with the DBA's, we found out that the following Postgres setting was configurated
idle_in_transaction_session_timeout = '5s'
(see: idle_in_transaction_session_timeout - pgPedia - a PostgreSQL Encyclopedia)
This will cause a connection that stays open for longer that 5s to timeout.
In my case I had a query (launched by an SQL Executor) that returned 2700 records but do to the fact that during the fetch there was a warning logged for each record (Coordinate system of reader identified by keyword `R_1' has changed from `EPSG:31370' to `'), the connection stayed open for more than 5s so I got the following error: Error executing SQL command ('FETCH 10000 IN "public_custom_query_crsr"'): 'Error connecting to database'.
By setting the coordinate system on the reader I could avoid the extra time for writing the warnings thus avoiding the timeout.
The problem still remains for queries taking longer that 5 seconds to complete.
To resolve this I did some tests with an SQL Executor trying to set the idle parameter value for 1 specific query :
FME_SQL_DELIMITER /
SET idle_in_transaction_session_timeout = '20s'/
SELECT *
FROM
<table>
Using this setup I was able to work my way around this problem.
A better solution would be setting up roles on the DB side that have a specific idle value for 'long' queries. Assign this role to a user and query with this specific user
+10
- Contributor
- 42 replies
-
2 October 2021
Hey @bbnj356 thanks for your remark, I'll have this checked.
Userlevel 4
+26
- Influencer
- 1636 replies
-
23 December 2021
I got the same error here, using a FeatureReader on a Postgis database. Here is the setup:
- Using Postgis Reader, read a list of tables to process
- Use for each feature a FeatureReader to read all features from the table name in the list
- Transform it using SchemaMapper (on a Postgis table...)
- Write the features read using a postgis writer in dynamics mode
It seems to run fine for about 10 out of 23 tables, then crashes, complaining (at line 39):
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_ontwerp_deelname_clusters_crsr" cursor for select "naam","clusterid","primarykey","deelname_scenario","toelichting_deelname","tips_omtrek","input_bd","snelmt","peakshaving_percentage","cityplan","scenario","datum_versie","mgt_checksum","mgt_insert_date","mgt_job_id","mgt_last_update_date","mgt_tijdstempel","mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."ontwerp_deelname_clusters"'
Reading POSTGIS table: 'lab_cityplannen.ontwerp_deelname_clusters'...
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_21
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Merged 1 schema features read from 1 datasets into 1 resulting feature types
SchemaMapper_Mapper (SchemaMappingFactory): Splitting bulk mode features into features
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_22
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Performing query against POSTGIS dataset `geodata_dev_svc_fme'
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_23
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_output_clustersmetalles_crsr" cursor for select "clusterid","deelname_scenario",...,"mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."output_clustersmetalles"'
Reading POSTGIS table: 'lab_cityplannen.output_clustersmetalles'...
Error executing SQL command ('FETCH 10000 IN "lab_cityplannen_output_clustersmetalles_crsr"'): 'Error connecting to database'
A fatal error has occurred. Check the logfile above for details
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Storing feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
The below feature caused the translation to be terminated
Storing feature(s) to FME feature store file `C:\Users\503898\Documents\Klus\User Story 53842 Herbruikbare FME template workspace om (historie)tabel Snowflake in de spoelen\2_execute_copy_to_pg_log.ffs'
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `FEATURE_TYPE'
Attribute(string) : `_reader_error' has value `A fatal error has occurred. Check the logfile above for details'
Attribute(string) : `fme_feature_type' has value `stg_open.mgt_tables_to_load'
Attribute(string) : `fme_geometry' has value `fme_undefined'
Attribute(entangled: string): `fme_type' has value `fme_no_geom'
entangled to postgis_type
Attribute(encoded: UTF-8) : `mgt_active' has value `T'
Attribute(encoded: UTF-8) : `mgt_category' has value `CITYPLAN'
Attribute(encoded: UTF-16LE): `mgt_insert_date' is <null>
Attribute(encoded: UTF-16LE): `mgt_last_update_date' is <null>
Attribute(encoded: UTF-8) : `mgt_process_delta' has value `F'
Attribute(encoded: UTF-8) : `mgt_type_verlading' has value `pg_to_pg'
Attribute(32 bit integer) : `multi_reader_full_id' has value `0'
Attribute(32 bit integer) : `multi_reader_id' has value `0'
Attribute(string) : `multi_reader_keyword' has value `POSTGIS_1'
Attribute(string) : `multi_reader_type' has value `POSTGIS'
Attribute(string) : `postgis_type' has value `postgis_none'
Attribute(encoded: UTF-8) : `src_table' has value `lab_cityplannen.output_clustersmetalles'
Coordinate System: `'
Geometry Type: IFMENull
===========================================================================
FeatureReader_<Rejected> (TeeFactory): FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader (QueryFactory): A fatal error has occurred. Check the logfile above for details
Stored 6425 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector_2.ffs'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Stored 1 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
A fatal error has occurred. Check the logfile above for details
Just if anyone is interested I got this message :
Error executing SQL command ('FETCH 10000 IN "<schemaName>_custom_query_crsr"'): 'Error connecting to database' When losing connection to the database (the database server was restarted during the process).
Could it be that your connections are somehow getting killed?
Were you either or you (@nath or @Helmoet de Zwijger) able to figure out the issue?
+10
- Contributor
- 42 replies
-
23 December 2021
I got the same error here, using a FeatureReader on a Postgis database. Here is the setup:
- Using Postgis Reader, read a list of tables to process
- Use for each feature a FeatureReader to read all features from the table name in the list
- Transform it using SchemaMapper (on a Postgis table...)
- Write the features read using a postgis writer in dynamics mode
It seems to run fine for about 10 out of 23 tables, then crashes, complaining (at line 39):
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_ontwerp_deelname_clusters_crsr" cursor for select "naam","clusterid","primarykey","deelname_scenario","toelichting_deelname","tips_omtrek","input_bd","snelmt","peakshaving_percentage","cityplan","scenario","datum_versie","mgt_checksum","mgt_insert_date","mgt_job_id","mgt_last_update_date","mgt_tijdstempel","mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."ontwerp_deelname_clusters"'
Reading POSTGIS table: 'lab_cityplannen.ontwerp_deelname_clusters'...
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_21
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Merged 1 schema features read from 1 datasets into 1 resulting feature types
SchemaMapper_Mapper (SchemaMappingFactory): Splitting bulk mode features into features
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_22
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Performing query against POSTGIS dataset `geodata_dev_svc_fme'
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_23
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_output_clustersmetalles_crsr" cursor for select "clusterid","deelname_scenario",...,"mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."output_clustersmetalles"'
Reading POSTGIS table: 'lab_cityplannen.output_clustersmetalles'...
Error executing SQL command ('FETCH 10000 IN "lab_cityplannen_output_clustersmetalles_crsr"'): 'Error connecting to database'
A fatal error has occurred. Check the logfile above for details
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Storing feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
The below feature caused the translation to be terminated
Storing feature(s) to FME feature store file `C:\Users\503898\Documents\Klus\User Story 53842 Herbruikbare FME template workspace om (historie)tabel Snowflake in de spoelen\2_execute_copy_to_pg_log.ffs'
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `FEATURE_TYPE'
Attribute(string) : `_reader_error' has value `A fatal error has occurred. Check the logfile above for details'
Attribute(string) : `fme_feature_type' has value `stg_open.mgt_tables_to_load'
Attribute(string) : `fme_geometry' has value `fme_undefined'
Attribute(entangled: string): `fme_type' has value `fme_no_geom'
entangled to postgis_type
Attribute(encoded: UTF-8) : `mgt_active' has value `T'
Attribute(encoded: UTF-8) : `mgt_category' has value `CITYPLAN'
Attribute(encoded: UTF-16LE): `mgt_insert_date' is <null>
Attribute(encoded: UTF-16LE): `mgt_last_update_date' is <null>
Attribute(encoded: UTF-8) : `mgt_process_delta' has value `F'
Attribute(encoded: UTF-8) : `mgt_type_verlading' has value `pg_to_pg'
Attribute(32 bit integer) : `multi_reader_full_id' has value `0'
Attribute(32 bit integer) : `multi_reader_id' has value `0'
Attribute(string) : `multi_reader_keyword' has value `POSTGIS_1'
Attribute(string) : `multi_reader_type' has value `POSTGIS'
Attribute(string) : `postgis_type' has value `postgis_none'
Attribute(encoded: UTF-8) : `src_table' has value `lab_cityplannen.output_clustersmetalles'
Coordinate System: `'
Geometry Type: IFMENull
===========================================================================
FeatureReader_<Rejected> (TeeFactory): FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader (QueryFactory): A fatal error has occurred. Check the logfile above for details
Stored 6425 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector_2.ffs'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Stored 1 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
A fatal error has occurred. Check the logfile above for details
Sadly, no solution yet. We are still facing this issue, it seems especially when the database is handling a lot of requests at the same time.
Userlevel 1
+5
- Contributor
- 69 replies
-
24 December 2021
I got the same error here, using a FeatureReader on a Postgis database. Here is the setup:
- Using Postgis Reader, read a list of tables to process
- Use for each feature a FeatureReader to read all features from the table name in the list
- Transform it using SchemaMapper (on a Postgis table...)
- Write the features read using a postgis writer in dynamics mode
It seems to run fine for about 10 out of 23 tables, then crashes, complaining (at line 39):
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_ontwerp_deelname_clusters_crsr" cursor for select "naam","clusterid","primarykey","deelname_scenario","toelichting_deelname","tips_omtrek","input_bd","snelmt","peakshaving_percentage","cityplan","scenario","datum_versie","mgt_checksum","mgt_insert_date","mgt_job_id","mgt_last_update_date","mgt_tijdstempel","mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."ontwerp_deelname_clusters"'
Reading POSTGIS table: 'lab_cityplannen.ontwerp_deelname_clusters'...
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_21
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Merged 1 schema features read from 1 datasets into 1 resulting feature types
SchemaMapper_Mapper (SchemaMappingFactory): Splitting bulk mode features into features
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_22
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Performing query against POSTGIS dataset `geodata_dev_svc_fme'
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_23
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_output_clustersmetalles_crsr" cursor for select "clusterid","deelname_scenario",...,"mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."output_clustersmetalles"'
Reading POSTGIS table: 'lab_cityplannen.output_clustersmetalles'...
Error executing SQL command ('FETCH 10000 IN "lab_cityplannen_output_clustersmetalles_crsr"'): 'Error connecting to database'
A fatal error has occurred. Check the logfile above for details
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Storing feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
The below feature caused the translation to be terminated
Storing feature(s) to FME feature store file `C:\Users\503898\Documents\Klus\User Story 53842 Herbruikbare FME template workspace om (historie)tabel Snowflake in de spoelen\2_execute_copy_to_pg_log.ffs'
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `FEATURE_TYPE'
Attribute(string) : `_reader_error' has value `A fatal error has occurred. Check the logfile above for details'
Attribute(string) : `fme_feature_type' has value `stg_open.mgt_tables_to_load'
Attribute(string) : `fme_geometry' has value `fme_undefined'
Attribute(entangled: string): `fme_type' has value `fme_no_geom'
entangled to postgis_type
Attribute(encoded: UTF-8) : `mgt_active' has value `T'
Attribute(encoded: UTF-8) : `mgt_category' has value `CITYPLAN'
Attribute(encoded: UTF-16LE): `mgt_insert_date' is <null>
Attribute(encoded: UTF-16LE): `mgt_last_update_date' is <null>
Attribute(encoded: UTF-8) : `mgt_process_delta' has value `F'
Attribute(encoded: UTF-8) : `mgt_type_verlading' has value `pg_to_pg'
Attribute(32 bit integer) : `multi_reader_full_id' has value `0'
Attribute(32 bit integer) : `multi_reader_id' has value `0'
Attribute(string) : `multi_reader_keyword' has value `POSTGIS_1'
Attribute(string) : `multi_reader_type' has value `POSTGIS'
Attribute(string) : `postgis_type' has value `postgis_none'
Attribute(encoded: UTF-8) : `src_table' has value `lab_cityplannen.output_clustersmetalles'
Coordinate System: `'
Geometry Type: IFMENull
===========================================================================
FeatureReader_<Rejected> (TeeFactory): FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader (QueryFactory): A fatal error has occurred. Check the logfile above for details
Stored 6425 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector_2.ffs'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Stored 1 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
A fatal error has occurred. Check the logfile above for details
Same!
- Contributor
- 8 replies
-
24 December 2021
I got the same error here, using a FeatureReader on a Postgis database. Here is the setup:
- Using Postgis Reader, read a list of tables to process
- Use for each feature a FeatureReader to read all features from the table name in the list
- Transform it using SchemaMapper (on a Postgis table...)
- Write the features read using a postgis writer in dynamics mode
It seems to run fine for about 10 out of 23 tables, then crashes, complaining (at line 39):
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_ontwerp_deelname_clusters_crsr" cursor for select "naam","clusterid","primarykey","deelname_scenario","toelichting_deelname","tips_omtrek","input_bd","snelmt","peakshaving_percentage","cityplan","scenario","datum_versie","mgt_checksum","mgt_insert_date","mgt_job_id","mgt_last_update_date","mgt_tijdstempel","mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."ontwerp_deelname_clusters"'
Reading POSTGIS table: 'lab_cityplannen.ontwerp_deelname_clusters'...
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_21
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Merged 1 schema features read from 1 datasets into 1 resulting feature types
SchemaMapper_Mapper (SchemaMappingFactory): Splitting bulk mode features into features
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_22
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Performing query against POSTGIS dataset `geodata_dev_svc_fme'
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_23
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_output_clustersmetalles_crsr" cursor for select "clusterid","deelname_scenario",...,"mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."output_clustersmetalles"'
Reading POSTGIS table: 'lab_cityplannen.output_clustersmetalles'...
Error executing SQL command ('FETCH 10000 IN "lab_cityplannen_output_clustersmetalles_crsr"'): 'Error connecting to database'
A fatal error has occurred. Check the logfile above for details
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Storing feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
The below feature caused the translation to be terminated
Storing feature(s) to FME feature store file `C:\Users\503898\Documents\Klus\User Story 53842 Herbruikbare FME template workspace om (historie)tabel Snowflake in de spoelen\2_execute_copy_to_pg_log.ffs'
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `FEATURE_TYPE'
Attribute(string) : `_reader_error' has value `A fatal error has occurred. Check the logfile above for details'
Attribute(string) : `fme_feature_type' has value `stg_open.mgt_tables_to_load'
Attribute(string) : `fme_geometry' has value `fme_undefined'
Attribute(entangled: string): `fme_type' has value `fme_no_geom'
entangled to postgis_type
Attribute(encoded: UTF-8) : `mgt_active' has value `T'
Attribute(encoded: UTF-8) : `mgt_category' has value `CITYPLAN'
Attribute(encoded: UTF-16LE): `mgt_insert_date' is <null>
Attribute(encoded: UTF-16LE): `mgt_last_update_date' is <null>
Attribute(encoded: UTF-8) : `mgt_process_delta' has value `F'
Attribute(encoded: UTF-8) : `mgt_type_verlading' has value `pg_to_pg'
Attribute(32 bit integer) : `multi_reader_full_id' has value `0'
Attribute(32 bit integer) : `multi_reader_id' has value `0'
Attribute(string) : `multi_reader_keyword' has value `POSTGIS_1'
Attribute(string) : `multi_reader_type' has value `POSTGIS'
Attribute(string) : `postgis_type' has value `postgis_none'
Attribute(encoded: UTF-8) : `src_table' has value `lab_cityplannen.output_clustersmetalles'
Coordinate System: `'
Geometry Type: IFMENull
===========================================================================
FeatureReader_<Rejected> (TeeFactory): FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader (QueryFactory): A fatal error has occurred. Check the logfile above for details
Stored 6425 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector_2.ffs'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Stored 1 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
A fatal error has occurred. Check the logfile above for details
I managed a workaround (see earlier post). Has anyone tried this out?
To resolve this I used an SQL Executor trying to set the idle parameter value (see: idle_in_transaction_session_timeout - pgPedia - a PostgreSQL Encyclopedia) for 1 specific query :
FME_SQL_DELIMITER /
SET idle_in_transaction_session_timeout = '20s'/
SELECT *
FROM
<table>
+10
- Contributor
- 42 replies
-
24 December 2021
I got the same error here, using a FeatureReader on a Postgis database. Here is the setup:
- Using Postgis Reader, read a list of tables to process
- Use for each feature a FeatureReader to read all features from the table name in the list
- Transform it using SchemaMapper (on a Postgis table...)
- Write the features read using a postgis writer in dynamics mode
It seems to run fine for about 10 out of 23 tables, then crashes, complaining (at line 39):
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_ontwerp_deelname_clusters_crsr" cursor for select "naam","clusterid","primarykey","deelname_scenario","toelichting_deelname","tips_omtrek","input_bd","snelmt","peakshaving_percentage","cityplan","scenario","datum_versie","mgt_checksum","mgt_insert_date","mgt_job_id","mgt_last_update_date","mgt_tijdstempel","mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."ontwerp_deelname_clusters"'
Reading POSTGIS table: 'lab_cityplannen.ontwerp_deelname_clusters'...
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_21
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Merged 1 schema features read from 1 datasets into 1 resulting feature types
SchemaMapper_Mapper (SchemaMappingFactory): Splitting bulk mode features into features
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_22
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Performing query against POSTGIS dataset `geodata_dev_svc_fme'
Successfully closed POSTGIS database reader
FME Configuration: Using connection values from 'geodata_dev_svc_fme' for R_23
Creating reader for format: PostGIS
Trying to find a DYNAMIC plugin for reader named `POSTGIS'
FME API version of module 'POSTGIS' matches current internal version (3.8 20200115)
Opening POSTGIS reader for dataset 'geodata_dev'
PostGIS client encoding: 'UTF8'
PostgreSQL 12.8 (Ubuntu 12.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
3.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
POSTGIS reader: Building features into bulk mode features
Executing data query: 'declare "lab_cityplannen_output_clustersmetalles_crsr" cursor for select "clusterid","deelname_scenario",...,"mgt_verwijderd_uit_bron", ST_AsEWKB("geom") as "geom" from "lab_cityplannen"."output_clustersmetalles"'
Reading POSTGIS table: 'lab_cityplannen.output_clustersmetalles'...
Error executing SQL command ('FETCH 10000 IN "lab_cityplannen_output_clustersmetalles_crsr"'): 'Error connecting to database'
A fatal error has occurred. Check the logfile above for details
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
Storing feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
The below feature caused the translation to be terminated
Storing feature(s) to FME feature store file `C:\Users\503898\Documents\Klus\User Story 53842 Herbruikbare FME template workspace om (historie)tabel Snowflake in de spoelen\2_execute_copy_to_pg_log.ffs'
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `FEATURE_TYPE'
Attribute(string) : `_reader_error' has value `A fatal error has occurred. Check the logfile above for details'
Attribute(string) : `fme_feature_type' has value `stg_open.mgt_tables_to_load'
Attribute(string) : `fme_geometry' has value `fme_undefined'
Attribute(entangled: string): `fme_type' has value `fme_no_geom'
entangled to postgis_type
Attribute(encoded: UTF-8) : `mgt_active' has value `T'
Attribute(encoded: UTF-8) : `mgt_category' has value `CITYPLAN'
Attribute(encoded: UTF-16LE): `mgt_insert_date' is <null>
Attribute(encoded: UTF-16LE): `mgt_last_update_date' is <null>
Attribute(encoded: UTF-8) : `mgt_process_delta' has value `F'
Attribute(encoded: UTF-8) : `mgt_type_verlading' has value `pg_to_pg'
Attribute(32 bit integer) : `multi_reader_full_id' has value `0'
Attribute(32 bit integer) : `multi_reader_id' has value `0'
Attribute(string) : `multi_reader_keyword' has value `POSTGIS_1'
Attribute(string) : `multi_reader_type' has value `POSTGIS'
Attribute(string) : `postgis_type' has value `postgis_none'
Attribute(encoded: UTF-8) : `src_table' has value `lab_cityplannen.output_clustersmetalles'
Coordinate System: `'
Geometry Type: IFMENull
===========================================================================
FeatureReader_<Rejected> (TeeFactory): FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader_<Rejected>: Termination Message: 'FeatureReader output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
FeatureReader (QueryFactory): A fatal error has occurred. Check the logfile above for details
Stored 6425 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector_2.ffs'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-inspectors-UxPgeR\inspector.fsi'
Stored 1 feature(s) to FME feature store file `C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.ffsupdating'
Saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Finished saving spatial index into file 'C:\Users\503898\AppData\Local\Temp\wb-cache-2_execute_copy_to_pg-jZOjob\Main_FeatureReader -1 5 fo 3 _lt_REJECTED_gt_ 0 86131318f19a3462305ce86ca57d37ca37580197.fsi'
Successfully closed POSTGIS database reader
Successfully closed POSTGIS database reader
A fatal error has occurred. Check the logfile above for details
Hi we're for sure going to try that [when I'm back from holidays] 😁
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.