Skip to main content
  • EN

I need to transform a big ECW (30GB) into a GeoPackage. I think I do not need any kind of processing of the data, so the workspace is pretty simple:

 

That works fine with small rasters, but the big one is draining all my computer resources (being honest, those are not that great...). The obvious solution is executing this workspace in a better computer with enough RAM, but I wanted to try something else.

I have created a "parent" workspace that builds a grid. Then, I call the "child" workspace and clip the ECW with a cell of that grid using the Search Envelope parameters of the ECW Reader.

 

If the Tile Set Name of the GeoPackage Writer is a static value, the result only contains the data of the last cell of the grid. The data is overwritten even though I have made sure the checkbox for Overwrite Database is unchecked:

I guess, that check is not for what I thought.

If I use a dynamic Tile Set Name for the GeoPackage Writer, all the data is stored; but each cell in different tables.

Is there a way of forcing that all the cells of the grid are stored in the same table of the GeoPackage without overwriting the data?

I'm using FME 2019.2 64-bits, by the way.

 

 

Thanks for any help provided!

Try to replace the WRITER with a Featurewriter to see if this improves. And also replace the READER with a FeatureReader. Curious

Try to replace the WRITER with a Featurewriter to see if this improves. And also replace the READER with a FeatureReader. Curious

Tried the approach on a 2GB ECW-file - seems to work ok. Currently I use 7,5 GB of RAM - so that might still be an issue on your side. Attached is a workspace. ecw_to_gpkg.fmw

By the way @oscard is your ECW file on a LOCAL SSD disk?

By the way @oscard is your ECW file on a LOCAL SSD disk?

No. I don't have one in this computer.

The final workspace will have to be run in a much better PC than mine if they plan to process such big files.

 

 

Thanks for your help!

Tried the approach on a 2GB ECW-file - seems to work ok. Currently I use 7,5 GB of RAM - so that might still be an issue on your side. Attached is a workspace. ecw_to_gpkg.fmw

Just out of curiosity: How long has it taken? How much RAM does the computer have?

 

 

I'm bulding the minimal hardware requirements for this workspace and that could help me a lot.

No. I don't have one in this computer.

The final workspace will have to be run in a much better PC than mine if they plan to process such big files.

 

 

Thanks for your help!

I see on my computer it reads/writes 400-800MB/sec (MB per sec) so if you dont have an SSD this will make a huge difference. I would also suggest you can try to run the workspace on FMECloud as these are also on SSD. For testing you get 250USD in free credit - just remember to STOP your instance when you are done. (https://console.fmecloud.safe.com/signup?msp_referrer=mvbgZKXwhGNba2IDO9N2iWKQpYI )

Just out of curiosity: How long has it taken? How much RAM does the computer have?

 

 

I'm bulding the minimal hardware requirements for this workspace and that could help me a lot.

Hehe, forgot to reply to this, and forgo to turn it off. Now after 4 days of running it is 42% done 😛 - and uses 32GB of RAM currently - so I guess there have to be a better way to do this. Will try to figure it out - but not sure if I have the time right now.

You can also do some fast testing by selecting a different layer (lower resolution) to read - for instance 5. This will reduce the amount of data read alot (5*worse resolution) so you will be able to test the procedure at least.

You can also do some fast testing by selecting a different layer (lower resolution) to read - for instance 5. This will reduce the amount of data read alot (5*worse resolution) so you will be able to test the procedure at least.

I'll try this. Thanks!

Reply


Community Stats

30,110
Posts
114,303
Replies
37,745
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.