Hi @rumle Tricky one. I've spoken to our development team and we feel this is a bug at this stage. We'll assess the issue and report back here once we have an update. For reference our internal issue tracking number is: FMEENGINE-62240. Sorry that you have encountered this.

Hi @brianatsafe.

Thanks for replying. I will look forward to hear more from you, when you have looked into the issue. I have ECW files of up to 4GB I wish to send through the writer.

@rumle we may have a workaround: it seems that we only limit to 200 MB when there is an ECW Reader in the workspace, and as a side-effect, the writer is affected.

As a workaround may be the user can write with a sub-workspace that doesn’t have an ECW reader?

Out of curiousity. Will it work if you have the ECW Reader in the main workspace - then send the data to a Linked Custom Transformer with an ECW-writer inside. Or will this also pick up the same "parameters" @brianatsafe

Hi @brianatsafe.

Thanks for your update.

My workspace is a simple conversion of an ECW file from one UTM zone to another. Your workaround might work if I first write to a non-ECW raster format in one workspace. Then back to ECW in a second separate workspace. I fear I would lose some information there.

I tried @sigtills suggestion with a linked (not embedded) custom transformer. Sadly, that does not work, but I like the thinking. Thanks.

You could write to FFS as the intermediate format. Since it's FME's internal format, it should perfectly preserve the raster.

I also encountered this issue in 2019.2, but with a jp2000 writer, no ECW in the workspace at all.

Yes, this problem stems from both the ECW and JPEG2000 readers, which share an underlying implementation.

I'm surprised by the number 3.8MB though; I would have expected 200MB in all cases where a JPEG2000 reader or ECW reader is in the workspace.

This was filed as FMEENGINE-62320, and involves a geotiff reader and jp2000 writer.

Hi @jakemolnar

I tried writing to the FFS format as you suggested, as I like the idea of zero data loss. This does work, but writing takes a lot of time. The file size is 40 times larger on the FFS file than the ECW, so it is not a perfect solution with multiple ECW files of 1-4 GB each. My poor laptop is in pain. I need to move the process to an application server for it to run at all.

Another workaround is that I split the output ECW in smaller files, but that is not perfect either.

I hope Safe will implement a solution for this.

You can look forward to a much easier process in the next release of FME (2020.0). The problem should not usually occur anymore. Even if it does, then we have added a way of manually specifying the memory limit (an advanced setting in the ECW writer).

Thanks for your patience with the current workaround @rumle !

Hi everyone, I'm pleased to announce that the memory allocation error reported as (FMEENGINE-62240) has been resolved. You can find the fix in the FME 2020 betas.

Hi, I am currently using FME 2020.0.0.0 Beta (20191229 - Build 20153 Win64) still have the same error. I already tired twice, all with the same error code.

ECW writer: The ECWJP2 SDK only has 200.0 MB of memory allocated to it and this is not enough to compress this dataset. Please set the SDK's memory limit above 710.0 MB

2020-01-16 13:46:51|591626.9| 0.0|ERROR |ECW writer: Failed to write dataset 'D:\\ECW.ecw'. This may occur if you have insufficient privileges to write this file, or if insufficient disk space is available

2020-01-16 13:47:11|591646.8| 19.9|ERROR |A fatal error has occurred. Check the logfile above for details

2020-01-16 13:47:11|591646.8| 0.0|ERROR |A fatal error has occurred. Check the logfile above for details

Hi @brandonguo,

Are you trying this out with a brand new workspace or an existing one?

If you are using an existing workspace, please upgrade your writer first to see if this resolves the issue.

If it doesn't resolve the error, please try the setting the parameter "Compression Cache Size" to a size the log file suggests - in the log snippet you shared for this example it would be 710 MB or greater.

This new parameter allows a compression cache size to be set manually, in the event the auto calculated size does not work.

Some more details about the settings for the cache size calculation parameter settings:

Hope this helps!

- Andrea