When I try to write an ECW file of around 1 GB, I get the following error from the ECW writer:
ECW writer: The ECWJP2 SDK only has 200.0 MB of memory allocated to it and this is not enough to compress this dataset. Please set the SDK's memory limit above 203.9 MBThat sounds fair enough. Does anyone know how to allocate more memory?
Using FME Desktop 2019.2.
Best answer by jovitaatsafe
Hi everyone, I'm pleased to announce that the memory allocation error reported as (FMEENGINE-62240) has been resolved. You can find the fix in the FME 2020 betas.
Thank you @brianatsafe and @jakemolnar!
+11Hi @rumle Tricky one. I've spoken to our development team and we feel this is a bug at this stage. We'll assess the issue and report back here once we have an update. For reference our internal issue tracking number is: FMEENGINE-62240. Sorry that you have encountered this.
Hi @rumle Tricky one. I've spoken to our development team and we feel this is a bug at this stage. We'll assess the issue and report back here once we have an update. For reference our internal issue tracking number is: FMEENGINE-62240. Sorry that you have encountered this.
Hi @brianatsafe.
Thanks for replying. I will look forward to hear more from you, when you have looked into the issue. I have ECW files of up to 4GB I wish to send through the writer.
+11@rumle we may have a workaround: it seems that we only limit to 200 MB when there is an ECW Reader in the workspace, and as a side-effect, the writer is affected.
As a workaround may be the user can write with a sub-workspace that doesn’t have an ECW reader?
Can you try that? We are still looking to fix the issue.
+24@rumle we may have a workaround: it seems that we only limit to 200 MB when there is an ECW Reader in the workspace, and as a side-effect, the writer is affected.
As a workaround may be the user can write with a sub-workspace that doesn’t have an ECW reader?
Can you try that? We are still looking to fix the issue.
Out of curiousity. Will it work if you have the ECW Reader in the main workspace - then send the data to a Linked Custom Transformer with an ECW-writer inside. Or will this also pick up the same "parameters" @brianatsafe
@rumle we may have a workaround: it seems that we only limit to 200 MB when there is an ECW Reader in the workspace, and as a side-effect, the writer is affected.
As a workaround may be the user can write with a sub-workspace that doesn’t have an ECW reader?
Can you try that? We are still looking to fix the issue.
Hi @brianatsafe.
Thanks for your update.
My workspace is a simple conversion of an ECW file from one UTM zone to another. Your workaround might work if I first write to a non-ECW raster format in one workspace. Then back to ECW in a second separate workspace. I fear I would lose some information there.
I tried @sigtills suggestion with a linked (not embedded) custom transformer. Sadly, that does not work, but I like the thinking. Thanks.
Hi @brianatsafe.
Thanks for your update.
My workspace is a simple conversion of an ECW file from one UTM zone to another. Your workaround might work if I first write to a non-ECW raster format in one workspace. Then back to ECW in a second separate workspace. I fear I would lose some information there.
I tried @sigtills suggestion with a linked (not embedded) custom transformer. Sadly, that does not work, but I like the thinking. Thanks.
You could write to FFS as the intermediate format. Since it's FME's internal format, it should perfectly preserve the raster.
+28I also encountered this issue in 2019.2, but with a jp2000 writer, no ECW in the workspace at all.
ERROR |JPEG2000 writer: The ECWJP2 SDK only has 3.8 MB of memory allocated to it and this is not enough to compress this dataset. Please set the SDK's memory limit above 86.5 MB
I also encountered this issue in 2019.2, but with a jp2000 writer, no ECW in the workspace at all.
ERROR |JPEG2000 writer: The ECWJP2 SDK only has 3.8 MB of memory allocated to it and this is not enough to compress this dataset. Please set the SDK's memory limit above 86.5 MB
Yes, this problem stems from both the ECW and JPEG2000 readers, which share an underlying implementation.
I'm surprised by the number 3.8MB though; I would have expected 200MB in all cases where a JPEG2000 reader or ECW reader is in the workspace.
+28Yes, this problem stems from both the ECW and JPEG2000 readers, which share an underlying implementation.
I'm surprised by the number 3.8MB though; I would have expected 200MB in all cases where a JPEG2000 reader or ECW reader is in the workspace.
This was filed as FMEENGINE-62320, and involves a geotiff reader and jp2000 writer.
The same file on a different computer running 2019.2 had no issue.
The same file on the problem computer has no issue with 2019.0.
You could write to FFS as the intermediate format. Since it's FME's internal format, it should perfectly preserve the raster.
Hi @jakemolnar
I tried writing to the FFS format as you suggested, as I like the idea of zero data loss. This does work, but writing takes a lot of time. The file size is 40 times larger on the FFS file than the ECW, so it is not a perfect solution with multiple ECW files of 1-4 GB each. My poor laptop is in pain. I need to move the process to an application server for it to run at all.
Another workaround is that I split the output ECW in smaller files, but that is not perfect either.
I hope Safe will implement a solution for this.
Hi @jakemolnar
I tried writing to the FFS format as you suggested, as I like the idea of zero data loss. This does work, but writing takes a lot of time. The file size is 40 times larger on the FFS file than the ECW, so it is not a perfect solution with multiple ECW files of 1-4 GB each. My poor laptop is in pain. I need to move the process to an application server for it to run at all.
Another workaround is that I split the output ECW in smaller files, but that is not perfect either.
I hope Safe will implement a solution for this.
You can look forward to a much easier process in the next release of FME (2020.0). The problem should not usually occur anymore. Even if it does, then we have added a way of manually specifying the memory limit (an advanced setting in the ECW writer).
Thanks for your patience with the current workaround @rumle !
+11Hi everyone, I'm pleased to announce that the memory allocation error reported as (FMEENGINE-62240) has been resolved. You can find the fix in the FME 2020 betas.
Thank you @brianatsafe and @jakemolnar!
+3Hi everyone, I'm pleased to announce that the memory allocation error reported as (FMEENGINE-62240) has been resolved. You can find the fix in the FME 2020 betas.
Thank you @brianatsafe and @jakemolnar!
Hi, I am currently using FME 2020.0.0.0 Beta (20191229 - Build 20153 Win64) still have the same error. I already tired twice, all with the same error code.
ECW writer: The ECWJP2 SDK only has 200.0 MB of memory allocated to it and this is not enough to compress this dataset. Please set the SDK's memory limit above 710.0 MB
2020-01-16 13:46:51|591626.9| 0.0|ERROR |ECW writer: Failed to write dataset 'D:\\ECW.ecw'. This may occur if you have insufficient privileges to write this file, or if insufficient disk space is available
2020-01-16 13:47:11|591646.8| 19.9|ERROR |A fatal error has occurred. Check the logfile above for details
2020-01-16 13:47:11|591646.8| 0.0|ERROR |A fatal error has occurred. Check the logfile above for details
+10Hi, I am currently using FME 2020.0.0.0 Beta (20191229 - Build 20153 Win64) still have the same error. I already tired twice, all with the same error code.
ECW writer: The ECWJP2 SDK only has 200.0 MB of memory allocated to it and this is not enough to compress this dataset. Please set the SDK's memory limit above 710.0 MB
2020-01-16 13:46:51|591626.9| 0.0|ERROR |ECW writer: Failed to write dataset 'D:\\ECW.ecw'. This may occur if you have insufficient privileges to write this file, or if insufficient disk space is available
2020-01-16 13:47:11|591646.8| 19.9|ERROR |A fatal error has occurred. Check the logfile above for details
2020-01-16 13:47:11|591646.8| 0.0|ERROR |A fatal error has occurred. Check the logfile above for details
Hi @brandonguo,
Are you trying this out with a brand new workspace or an existing one?
If you are using an existing workspace, please upgrade your writer first to see if this resolves the issue.
If it doesn't resolve the error, please try the setting the parameter "Compression Cache Size" to a size the log file suggests - in the log snippet you shared for this example it would be 710 MB or greater.
This new parameter allows a compression cache size to be set manually, in the event the auto calculated size does not work.
Some more details about the settings for the cache size calculation parameter settings:
Auto (default): FME will calculate a cache size that is appropriate for the size of the raster data. This option should be preferred.Manual: FME will use the value of the Compression Cache Size (MB) parameter. This option should be used if a cache error occurs. The error will contain information about the prescribed manual cache size.Hope this helps!
- Andrea
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
