A direct utf-8 text file to utf-8 text file works perfectly but when I try to write to a system character encoding text file I get an error after a while.
I include the log file in the workspace (https://goo.gl/DF46Yx)
Any suggestions?
Page 1 / 1
Hi @j4
ifile.txt is not your source dataset, is it? It is a sample dataset, am I correct?
As you deal with UTF-8 data that you are attempting to transcode into Latin-1 (Win-1252), I suspected that you might have a UTF-8 character in your data that is not supported in Win-1252. However, the log doesn't provide enough evidence that this indeed is the case.
Could you please take another look at your source data? If you have a text editor that allows changing data encoding, you might want to try to open the file and display it in Win-1252 - will the data look right? Or, in your workspace, could you please disable the Writer that writes in system encoding and check if writing in UTF-8 will be OK?
Hi @j4,
I think this might be due to a memory issue. I was able to reproduce the failure in 32 bit FME 2017.1, but not in the 64 bit version, which has access to more memory. The input is a single feature with a 400 MB text attribute, which is likely stressing out the AttributeEncoder.
The good news is that when I tried this workspace out in the latest FME 2018 32 bit beta, it ran successfully. We have been working hard on performance improvements in FME 2018, and this to have helped in this case. Please try your workspace in FME 2018, which can be downloaded from here.
Hi @j4
ifile.txt is not your source dataset, is it? It is a sample dataset, am I correct?
As you deal with UTF-8 data that you are attempting to transcode into Latin-1 (Win-1252), I suspected that you might have a UTF-8 character in your data that is not supported in Win-1252. However, the log doesn't provide enough evidence that this indeed is the case.
Could you please take another look at your source data? If you have a text editor that allows changing data encoding, you might want to try to open the file and display it in Win-1252 - will the data look right? Or, in your workspace, could you please disable the Writer that writes in system encoding and check if writing in UTF-8 will be OK?
Hi @j4,
I think this might be due to a memory issue. I was able to reproduce the failure in 32 bit FME 2017.1, but not in the 64 bit version, which has access to more memory. The input is a single feature with a 400 MB text attribute, which is likely stressing out the AttributeEncoder.
The good news is that when I tried this workspace out in the latest FME 2018 32 bit beta, it ran successfully. We have been working hard on performance improvements in FME 2018, and this to have helped in this case. Please try your workspace in FME 2018, which can be downloaded from here.
Thanks for the reply. I'll migrate to the 64-bit version. Anyway, a little more information in the log would help with memory errors.
The log file is usually quite verbose about errors, but memory failures are quite catastrophic, and so don't give FME time to say anything. When I see an abrupt failure in the log like this, the issue is usually memory related.
Hi @j4,
The log file is usually quite verbose about errors, but memory failures are quite catastrophic, and so don't give FME time to say anything. When I see an abrupt failure in the log like this, the issue is usually memory related.