Skip to main content
Question

Copy multiple CSV files on PostgreSQL database

  • February 24, 2025
  • 4 replies
  • 56 views

simeon.simic
Contributor
Forum|alt.badge.img+2

Hello, i am working on project and i need help.
I want to copy multiple csv files from my pc, to postgreSql database.
I have a folder on my computer and i want all files from that folder just to copy on postgre.

I dont know how many files will be in  folder on pc, so i cannot predict, but they all have some column names sames, some diferent, and some of them even have diferent  number of columns 

 

So basically, i have set up a reader in  FME workbench, in reader i selected which folder i want to read all csv files from, then reader is connected to schemaValidator , and that is connected on writer,  that wrriter should write in postgresql

My problem is that, when i run that workspace , in my postgre database writer writes right names in right schema  with right data, but with columns from all files 

I need help, what can i do that if its exactly 5 files in folder that i read , that all 5 files in exactly how they are in folder to be copied on postgresql ?
 

I have been try transformer FileCopy, but FileCopy cannot work on PostgreSql, or i dont know how

I appreciate it in advance.

 

4 replies

j.botterill
Influencer
Forum|alt.badge.img+34
  • Influencer
  • February 24, 2025

Do the CSV’s all have the same schema/columns?

Under the parameters of the CSV reader, you can preview the schema. Test the manual vs automatic preview to see if the files are consistent. 

If there are (any) differences, you can edit the input csv’s manually, or you can learn about Dynamic reading and writing, where you will need to expose attributes (if using the generic output port of a featureReader. I make use of inspecting the Feature information and the fme_feature_type for each CSV file.

Then rename attributes that need them, so you can  present the same attributes to the writer, where the should match with the user attributes you have manually set.


simeon.simic
Contributor
Forum|alt.badge.img+2
  • Author
  • Contributor
  • February 24, 2025

CSV’s have diferent columns, and some diferent schemas , but i want all of them in the same schema in postgresql


I understand that, but my problem is that i wont be able to see how many files are in the folder, because it can be 2,3,6 or even more ….

So i just need to copy all of them into postgresql database

 


j.botterill
Influencer
Forum|alt.badge.img+34
  • Influencer
  • February 25, 2025

dynamic reading is the answer...with some attributeRenaming of know instances… perhaps the SchemaMapper if you want to control the mapping outside of FME….here are some other examples 

https://www.youtube.com/watch?v=Qt01oKI5Ut0 

the pattern you are after may have some of these elements,

If you have coodinateFields, set that in the Reader parameters, then you can write the geom into an attribute. Schema scanner is an excellent transformer for dynamic workflows. However, manipulating the schema has to be performed in the data and schema port data streams. Schema is list manipulation

 


simeon.simic
Contributor
Forum|alt.badge.img+2
  • Author
  • Contributor
  • February 25, 2025

Thank you for your help, i will give it a try


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings