Hi @skormt​ ,

FME Flow Schedule should be able to read the CSV file without caching... What version of FME Workbench and FME Flow are you using? I can try for a repro on my end based on the version you're experiencing this behavior in.

Are you seeing any unusual messages logged in the scheduled job log?

Do you notice any differences between the job logs (manual run and schedule run)?

Testing out this Change Detection tutorial but help with troubleshooting.

Hope this helps!

Kezia

Workbench: FME(R) 2022.1.0.0 (20220704 - Build 22618 - WIN64)

(Flow?) FME Server: FME Server 2022.1 Build 22618 - win64

Most unusual message I see in then job log is:

• CSV reader: Detected excess data and/or delimiters in addition to the defined schema in line 2 with value 
  • Though this too is an odd message as the column is has double quotes (") around the extra comma, and seems to read fine in workbench as it loads the data

Logs at an eyeball glance look the same not extra "colored" messages between logs, just the one mentioned above.

Some other trouble shooting I have done:

• re-uploaded and overwrite from work bench. -- This seems to fix on day 1 of a new run, but still falls back to seeming to be cached from that last update.
• recreated Job and disabled older one. No change even on day 1 of new job.

Thanks for the update. A couple more questions:

• If you fixed line 2, does the issue resolve? Is it an issue of delimiter or excess data?
• Do you see the same error/warning message in FME Workbench translation log file?
• Are you able to share some sample data which includes the line 2 issue for me to test with?
• How often is the schedule run and with what Schedule type? Basic, Repeat or CRON?

Kezia

• Upon pulling out lines 1 and 2 from the file into a separate file, it all then parsed fine with no warnings
• I do see the same error message, but with a more subdued text highlight than that of the FME server. And it seems to not have any issues with generating the file.
• I can't attach the data here as it contains PII. And I'm less suspect of the data source than I am of the FME Server "holding" data instead of doing a "re-read" every time. For the most part the data changes daily, but I'm not seeing that on the final output.

Thanks for testing. Can you share some dummy data instead for me to reproduce the issue?

I'd advise submitting a case here. This way you can share your logs, workspaces, and dummy data for us to dig deeper.

Wanted to follow up as I've seemed to have fixed the issue.

Though it "may" be 1 of 2 possibilities -- scenario is that the CSV Writer was dumping a named file into the temp folder that would then be picked up by the FPT Transformer to put on a remote site. Well after working with a local person who looked over the shoulder so to speak, he pointed out that I was uploading the file to the FME Server when I would deploy it, which when looking at the logs, it almost looked like it was FTPing before ever processing (still new to reading those verbose logs)

My solution was 2 fold.

1. Remove the check option to upload temporary file.
2. In the FTP Transformer, I also made it "use" the output file path from the CSV writer that was given in the "dataset".

Which is the winner to me? Unknown, and too nervous to toy with it right now, as it's working nightly like I originally expected. I suspect the temp file upload is probably the biggest culurate as I think FME server stored that off somewhere else and then FTP tool just always used that one sample despite the file being updated daily in where ever the server was storying the newly generated file daily (probably still c:\\temp)