Solved

Custom transformer doesn't work on Cloud?

  • 28 August 2018
  • 7 replies
  • 0 views
jatoxa
Rank
Badge +2

Hi,

 

 

I made a workspace that creates folders and uploads files on a web API with the HTTP caller.

 

The first version created folders at one level then uploaded files. Then it went on to do the next level and so on. So it was basically a series of HTTPcallers which worked fine on both desktop and Cloud.

 

 

However since it's impossible to know how many folder levels there are in advance I made it with a custom loop transformer and just one HTTP caller.

 

This worked even better on desktop but trying to run it on FME Cloud it seems it never gets done.

 

Something that takes 8 seconds in desktop isn't ready after half an hour on Cloud.....seems it never will be.

 

 

I checked the webconnection and I uploaded the custom transformer both to the repository and to the transformers folder. I run the same version of Desktop and Cloud 2018.1.0.1(18528). It initially didn't accept the custom transformer on an earlier Cloud version.

Any ideas to what I might want to check or anyone with similar problems?

 

 

Thanks!

 

 

/Jacob

 

 

icon

Best answer by jatoxa 30 August 2018, 19:12

View original

7 replies

virtualcitymatt
Rank
Userlevel 4
Badge +26

Hi @j2,

 

I would set a limit to the number of loops (just for testing purposes). This should allow you to verify the same behavior on Desktop and Cloud. Keep in mind too that FME Cloud is on Linux and playing with file paths can be a real pain when testing between windows and Linux because of the difference in paths.
jatoxa
Rank
Badge +2

Hi @j2,

 

I would set a limit to the number of loops (just for testing purposes). This should allow you to verify the same behavior on Desktop and Cloud. Keep in mind too that FME Cloud is on Linux and playing with file paths can be a real pain when testing between windows and Linux because of the difference in paths.
Hi virtualcitymatt,

 

 

Thanks for the reply.

 

 

At first I thought it was a problem with the looping transformer but it seems it might be more related to what you're saying.

 

 

When running on Desktop I use the File/Directory reader and choose a path like

 

C:\\Projects\\testproject\\subfolders where "testproject" becomes level 0 or the root of the folder.

 

 

On Cloud it starts an infinite loop if I select a path like:

 

$(FME_SHAREDRESOURCE_DATA)/testproject/subfolders

 

 

However it does make the first loop if I change it to:

 

$(FME_SHAREDRESOURCE_DATA)/testproject/testproject/subfolders

 

 

Then it goes into the infinite loop again.

 

 

 

Edit:

 

I now tested the first version of the workflow without the loop on the 18528 version of Cloud and it doesn't work either....Seems like File/Directory reader doesn't really work on Server/Cloud?

 

 

 

 

 

G
Badge

Hi @j2,

can you share a simple workspace (without looping) and complete log files that demonstrate & reproduce the problem on FME Server 18528?

There is no indication that the Directory and File Pathnames is not working on FME Server or FME Cloud right now, but I am happy to test your scenario if you can share a simple workflow.

jatoxa
Rank
Badge +2

Hi @j2,

can you share a simple workspace (without looping) and complete log files that demonstrate & reproduce the problem on FME Server 18528?

There is no indication that the Directory and File Pathnames is not working on FME Server or FME Cloud right now, but I am happy to test your scenario if you can share a simple workflow.

Hi GerhardAtSafe,

 

 

Thank you for your reply.

 

 

I just tried both workspaces on FME Server 18528 on a Windows machine and both worked perfectly. So it seems the problem is related to what virtualcitymatt is talking about regarding the file paths? Are there any general guidelines for using Directory and File Pathnames when it's running on a Linux machine?

 

 

I will make a simpler version of the workflow to share.

 

 

 

virtualcitymatt
Rank
Userlevel 4
Badge +26
Hi virtualcitymatt,

 

 

Thanks for the reply.

 

 

At first I thought it was a problem with the looping transformer but it seems it might be more related to what you're saying.

 

 

When running on Desktop I use the File/Directory reader and choose a path like

 

C:\\Projects\\testproject\\subfolders where "testproject" becomes level 0 or the root of the folder.

 

 

On Cloud it starts an infinite loop if I select a path like:

 

$(FME_SHAREDRESOURCE_DATA)/testproject/subfolders

 

 

However it does make the first loop if I change it to:

 

$(FME_SHAREDRESOURCE_DATA)/testproject/testproject/subfolders

 

 

Then it goes into the infinite loop again.

 

 

 

Edit:

 

I now tested the first version of the workflow without the loop on the 18528 version of Cloud and it doesn't work either....Seems like File/Directory reader doesn't really work on Server/Cloud?

 

 

 

 

 

Hmmm, I use that on FME cloud often without issues (except me), I suppose there could be an issue with it on the version of FME on Linux but unlikely. Check with Loggers in your workflow to see what the path reader is actually reading. Also put loggers before the reader (if it's a feature reader). Loggers are your friend here when working on FME Server. Also you need to make sure that if you're using the pathname which is read that you use the _path_unix rather than the _path_windows attributes.

 

 

jatoxa
Rank
Badge +2

Ultimately it had nothing to do with the transformer but it seems paths are interpreted marginally different on Cloud compared to Windows Desktop/Server?

 

 

I ended up using a generic reader which reads zip files instead which uploads and extracts the files to Cloud/Server and then run the HTTP looper.

 

 

Thanks for the hints and replies!

 

G
Badge

Ultimately it had nothing to do with the transformer but it seems paths are interpreted marginally different on Cloud compared to Windows Desktop/Server?

 

 

I ended up using a generic reader which reads zip files instead which uploads and extracts the files to Cloud/Server and then run the HTTP looper.

 

 

Thanks for the hints and replies!

 

Hi @j2,

 

glad to hear you found a solution!

 

The main difference on FME Cloud is that the underlying OS is Linux and therefore unix paths are used instead of windows paths. Unix paths use / as separator and windows paths use \\. The Directory and File Pathnames reader returns both options and therefore you would have to switch to the path_unix attribute before you publish your workspace to FME Cloud. Another common issue when dealing with pathnames is spaces and special characters. Using quotes around pathnames can help with this.

 

Reply


Community Stats

28,670
Posts
109,365
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.