Solved

Errors every 10,000 records in FME Desktop 2022.0. How to fix this?

sussexpete
Rank
Badge +1

I have recently installed FME Desktop 2022.0 (Database edition – Build 22337 – WIN64) and want to migrate old workspaces to this latest version.

 

I have a workspace that was written and runs in FME Desktop 2019.0 (Database edition). It reads in about 120,000 records via SQL Creator (non-spatial Microsoft SQL data format), does some transformations, and writes to a CSV file. It works perfectly in FME Desktop 2019.0.

 

However, when I open and run the same workspace in FME Desktop 2022.0 it is not working correctly.

 

I have tried this on Windows 10 Enterprise and Windows Server 2012 R2.

 

I have tried with Feature Cache on and off.

 

I have tried running as a batch process.

 

I have tried running the workspace in 2022.0 unchanged (as a 2019.0 workspace).

 

I have tried running the workspace in 2022.0 unchanged (as a 2022.0 workspace).

 

I have tried running in 2022.0 with transformers upgraded to the latest 2022.0 versions.

 

In all cases after every 10,000 records, several records are corrupted on the output. Some column values are missing or else filled with unprintable characters. Some column values are from other records. The actual corruption differs depending on whether the workspace is run with feature caching, without feature caching, or batch run from command line.

 

Using Feature Cache, inspecting the records output from the SQL Creator, all records are present and clean. However, viewing the records after an AttributeSplitter transformer, records are corrupt.

 

I have also tried this on other workspaces and the same problem exists. I have tried using the standard MS SQL Reader and have the same problem.

 

I wonder if anyone else has experienced this and if so, what can be done to resolve the issue.

 

I attach the workspace and log file. I am unable to upload data because it is in an MS SQL database and is confidential.

 

In the screen grab below, the 2019 processed data is on the left and the 2022 processed data is on the right.

 

image 

 

icon

Best answer by JennaKAtSafe 31 October 2022, 18:05

View original
1 Attachment

12 replies

redgeographics
Rank
Userlevel 5
Badge +25

That's really weird. Is it always exactly after 10.000 records?

 

At first glance I would say it might be something with the encoding, but why would that change after 10.000 records?

sussexpete
Rank
Badge +1

Yes, record 10,001 is corrupt and the next correct is 10,125. Then corruption starts again at 20,001 with the next correct at 20,125.

 

I tried writing from SQL to FFS in a separate workspace and then using the FFS as input to the failing process and the errors were still there.

 

I then tried writing SQL to CSV in a separate workspace and using that as input and it worked! I can only think that is due to the CSV reader loading the whole file in memory?

redgeographics
Rank
Userlevel 5
Badge +25

Just an idea, since it seems to be happening very regularly, could it be that the SQLCreator is hitting a timeout? Try increasing the timeout value in it's parameters (it's in the Microsoft SQL parameters inside the SQLCreator's parameters) and see if that makes a difference.

sussexpete
Rank
Badge +1

Just an idea, since it seems to be happening very regularly, could it be that the SQLCreator is hitting a timeout? Try increasing the timeout value in it's parameters (it's in the Microsoft SQL parameters inside the SQLCreator's parameters) and see if that makes a difference.

No better, I'm afraid. I increased the timeout from 300 to 3000 but still the same errors at the same point.

 

In the Visual Preview window, the Results from the SQL Creator are correct. However, the output from the first transformer is wrong.

 

Yes, this is really weird!

sussexpete
Rank
Badge +1

I think I've found the answer. It looks like a problem with BULK MODE.

 

Reading through How FME Improves Performance with Bulk Mode (safe.com) I followed the advice to turn it off. I added a Decelerator after the SQL Creator. My output data is now identical to my reference set from FME 2019.0. I can only assume that Bulk Mode has been added to transformers I am using since FME 2019.0 and that is why it was working in the old release and not in FME 2022.0.

 

I would be very interested to learn if others have this issue too.

 

image

redgeographics
Rank
Userlevel 5
Badge +25

I think I've found the answer. It looks like a problem with BULK MODE.

 

Reading through How FME Improves Performance with Bulk Mode (safe.com) I followed the advice to turn it off. I added a Decelerator after the SQL Creator. My output data is now identical to my reference set from FME 2019.0. I can only assume that Bulk Mode has been added to transformers I am using since FME 2019.0 and that is why it was working in the old release and not in FME 2022.0.

 

I would be very interested to learn if others have this issue too.

 

image

It does sound like something Safe would be interested in seeing, so I would recommend you make a support case of it.

JennaKAtSafe
Rank
Badge +6

Hi @sussexpete​, Thank you for bringing this to our attention and also figuring out a workaround! I will create a case on your behalf for any follow-ups regarding this issue.

M

Thanks a lot @sussexpete​ for the workaround. I had similar problems with FME(R) 2022.0.0.1 (20220505 - Build 22339 - WIN64). We will update to 2022.0.0.2 in the next few days and hope it works again. Otherwise we could try the 2022.1 (beta).​ 

sussexpete
Rank
Badge +1

Hi @sussexpete​, Thank you for bringing this to our attention and also figuring out a workaround! I will create a case on your behalf for any follow-ups regarding this issue.

Thank you @JennaAtSafe for taking the time to look at this. I'm glad (in a way) that you've been able to recreate the issue at Safe. Disappointed that I cannot safely move to 2022.0. This is such an important issue for us that I will uninstall 2022.0 as it cannot be fully trusted to produce correct output for my organization. I will stick with 2021.1 for now, pending patch in 2022.1.

sussexpete
Rank
Badge +1

Thanks a lot @sussexpete​ for the workaround. I had similar problems with FME(R) 2022.0.0.1 (20220505 - Build 22339 - WIN64). We will update to 2022.0.0.2 in the next few days and hope it works again. Otherwise we could try the 2022.1 (beta).​ 

You're welcome @mschorcht and thanks for posting. This is a real, potentially serious, problem. Fix should be in 2022.1 beta.

B
Badge +1

I think I've found the answer. It looks like a problem with BULK MODE.

 

Reading through How FME Improves Performance with Bulk Mode (safe.com) I followed the advice to turn it off. I added a Decelerator after the SQL Creator. My output data is now identical to my reference set from FME 2019.0. I can only assume that Bulk Mode has been added to transformers I am using since FME 2019.0 and that is why it was working in the old release and not in FME 2022.0.

 

I would be very interested to learn if others have this issue too.

 

image

I am also having a similar issue dealing with 90,000 + features from 2 readers and using change detection. This post is extremely helpful! I added decelerators to both reader and worked great -- I am using Desktop 2022.0.0.2.

JennaKAtSafe
Rank
Badge +6

I am also having a similar issue dealing with 90,000 + features from 2 readers and using change detection. This post is extremely helpful! I added decelerators to both reader and worked great -- I am using Desktop 2022.0.0.2.

This issue was identified as a bug and the fix was verified in FME 2022.1 build 22609.  Let us know if you're running into the same issue in newer versions of FME!

Reply


Community Stats

28,688
Posts
109,427
Replies
36,543
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.