Skip to main content
  • EN
Solved

Creating files in Network Shared Folders using FME Flow

  • 14 June 2024
  • 7 replies
  • 110 views

Hello,

I’m trying to write files into a Shared Network Folders via an automation in FME Flow running a workspace.

The Shared Network Folders were mounted onto the VM Server and then added to the Resources in the FME Flow app with the credentials from “user1”.

After adding the shared network folders to the Resources in FME Flow I can see the files and folders inside them and I can download and read the files and folders via the FME Flow interface without problems. But when I’m trying to upload files/folders or create a folder directly in the share it gives me an error indicating issues with access rights. I find it weird that I can see, read and download folders but not upload them, but I guess it is because the credentials used by FME Flow to access the shared network folders are not the ones from “user1” but from the service user account that FME Flow uses, like “fmflow” user or something like that.

Question is how can I find out what is the user used by FME Flow to access the shared network drives?.

Thanks in advance.

RZ.-

7 replies

virtualcitymatt
Rank
Userlevel 5
Badge +32

Right, it does seem like the user which FME Flow is running as only has read access to those data. 

You can look at the services to see which user FME Flow Services as. In this case they are running as the LocalSystem. 

The specific doc about changing the user is here: https://docs.safe.com/fme/html/FME-Flow/AdminGuide/Running_System_Services_Under_Different_Accounts.htm
 

rzel24
Rank
Badge +3

Hi @virtualcitymatt my server is running on Debian 11, which is why I’m having a harder time pinpointing which user is doing what.

Any idea how to check the same in Deb 11?.

Thanks again.

Randall Z.-

steveatsafe
Rank
Userlevel 1
Badge +11

@rzel24, Linux. That’s interesting question that I do not have a precise answer for… or tested in this way before that I can clearly remember. I’ve only ever run jobs that connected with mounted shares.

I’m wondering if Tomcat (FME Flow Application Server) is using a user account that is not ‘fmeflow’ on Linux, that doesn’t have access to the Share… but for sure the Core & Engines are using ‘fmeflow’ user so it will need to be given access to RW (if it doesn’t already have).  Agree, it is interesting that you can can ‘see’, ‘read’ and ‘download’ folders but not upload them.

Can you test that the FME Flow Engine can create a file in the same location? So publish a workspace that is using the share and see if it can create the file, if it can’t then the ‘fmeflow’ user has read access but no write access and once this is resolved it will likely resolve the Web UI issue too. 

If that test is successful, then I am suspect of the user behind the FME Flow Application Server… and we will need to confirm that user.

SteveatSafe
rzel24
Rank
Badge +3

Hi @steveatsafe, the reason I found out we weren’t able to write into the shared network drives was because I have a workspace schedule in an automation that came back with errors:
 

146 2024-6-14 13:23:11 | File Copy Writer: Error attempting to copy from '/tmp/user/997/fmeengines/localhost_Engine1/1718360564692_1757186_d/tor400/ac_qual.csv' to '/mnt/share4/SANDBOX/ALE/AMP/CSV/ac_qual.csv'. Please check for sufficient permissions on the source and destination. Alternatively, if Overwrite Existing File is set to 'No' please ensure output file does not already exist

 

The workspace reads from a shared network folder (added as a resources), then after processing the information from the file it writes 3 different files in a temporary folder in the FME server (the temp folder is created using the TempPathnameCreator transformer), after the files are created we copy the files from the temp folder to the final destination in a different shared network folder (added as a separate resource from where the file is read).

I decided to try and see if I could write directly to the shared network folder (I tried in 2 different locations, one at a time) so I removed the fileCopy transformers as well as the TempPathnameCreator to read the file from the shared network folder and write the 3 x CSV files directly to the shared network folder. This time the workspace came back without the error above (I guess because I am not using the FileCopy writer) and in fact no error was reported for the CSV writer (I’m using a featurewriter configured for CSV files), but when I checked the shared network folders I cannot see any of the files there (nothing was created there).

I did this with 2 different locations on the shared network folders and despite not getting any errors in the log from the job (in fact the log shows the files are created):

*********************************************************************

114 2024-6-18 11:01:50 | CSV writer: Writing 5085 feature(s) to file '$(SharedDrive4)/SANDBOX/AMP/CSV/ac_all_test.csv'
115 2024-6-18 11:01:53 | CSV writer: Opening dataset '$(SharedDrive4)/SANDBOX/AMP/CSV/ac_qual_test.csv'
116 2024-6-18 11:01:53 | CSV writer: Writing 53600 feature(s) to file '$(SharedDrive4)/SANDBOX/AMP/CSV/ac_qual_test.csv'
117 2024-6-18 11:01:56 | Creator_XML_Creator (CreationFactory): Created 1 features
118 2024-6-18 11:01:56 | Creator_Cloner (TeeFactory): Cloned 1 input feature(s) into 1 output feature(s)
119 2024-6-18 11:01:56 | Creator_CREATED Brancher -1 4 (BranchingFactory): Branched 1 input feature -- 1 feature routed to the target factory, and 0 features routed to the fallback factory.
120 2024-6-18 11:01:56 | _CREATOR_BRANCH_TARGET (TeeFactory): Cloned 1 input feature(s) into 1 output feature(s)
121 2024-6-18 11:01:56 | Excel Reader: Closing dataset '/tmp/user/997/fmeengines/localhost_Engine1/TempFS_1718697693088_2132548/AMP All.xlsx'...
122 2024-6-18 11:01:56 | Excel Writer: Workbook contains 128 fonts and 374 cell formats after writing
123 2024-6-18 11:01:56 | Excel Reader: Closing dataset '/tmp/user/997/fmeengines/localhost_Engine1/TempFS_1718697693088_2132548/AMP All.xlsx'...
124 2024-6-18 11:01:56 | =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
125 2024-6-18 11:01:56 | Feature output statistics for `CSV2' writer using keyword `FeatureWriter_0':
126 2024-6-18 11:01:56 | =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
127 2024-6-18 11:01:56 | Features Written
128 2024-6-18 11:01:56 | =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
129 2024-6-18 11:01:56 | ac_all_test 5317
130 2024-6-18 11:01:56 | ac_live_test 41362
131 2024-6-18 11:01:56 | ac_qual_test 53600
132 2024-6-18 11:01:56 | ==============================================================================
133 2024-6-18 11:01:56 | Total Features Written 100279
134 2024-6-18 11:01:56 | =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
135 2024-6-18 11:01:56 | Destination Feature Type Routing Correlator (RoutingFactory): Tested 0 input feature(s), wrote 0 output feature(s): 0 matched merge filters, 0 were routed to output, 0 could not be routed.
136 2024-6-18 11:01:56 | Final Output Nuker (TeeFactory): Cloned 0 input feature(s) into 0 output feature(s)
137 2024-6-18 11:01:56 | =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
138 2024-6-18 11:01:56 | Features Read Summary
139 2024-6-18 11:01:56 | =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
140 2024-6-18 11:01:56 | ==============================================================================
141 2024-6-18 11:01:56 | Total Features Read 0
142 2024-6-18 11:01:56 | =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
143 2024-6-18 11:01:56 | =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
144 2024-6-18 11:01:56 | Features Written Summary
145 2024-6-18 11:01:56 | =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
146 2024-6-18 11:01:56 | ==============================================================================
147 2024-6-18 11:01:56 | Total Features Written 0
148 2024-6-18 11:01:56 | =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
149 2024-6-18 11:01:56 | Translation was SUCCESSFUL with 5 warning(s) (0 feature(s) output)
150 2024-6-18 11:01:56 | FME Session Duration: 24.2 seconds. (CPU: 20.5s user, 1.4s system)
151

2024-6-18 11:01:56 | END - ProcessID: 2132548, peak process heap usage: 1393865 kB, current process heap usage: 28366 kB

**********************************************************************

The files were not created in any of the tests I did.

I tried to create the file in the FME Server folder and that worked without problems.

Any thoughts?

RZ.-

 

steveatsafe
Rank
Userlevel 1
Badge +11

@rzel24 

Could you turn on Log Debug in the Workspace? Might get some more information?
Could you test with the CSV Writer vs the FeatureWriter?
I believe we will have to look into this more. 

Are you able to create a ticket? That way we can ask you a few more detailed questions and if necessary connect via a screen share. 

In any case, we’ll need to test this so we can give you conclusive answer - could be a bug - seems kind of odd that the CSV Writer isn’t being more helpful and reporting an error… or is it writing somewhere else?

SteveatSafe
rzel24
Rank
Badge +3

​
@steveatsafe 

I’m new at FME and I can’t seem to find this Log Debug option or setting, how do I turn this on?.

I tested the workspace with a CSV writer and I get an error indicating that it cannot write in the destination:
******************************************************************************************************

024-06-19 07:48:46|  15.4|  0.0|INFORM|CSV writer: Opening dataset '/mnt/share/AMP/Roster/Archive/ac_all_Test.csv'
2024-06-19 07:48:46|  15.4|  0.0|ERROR |CSV writer: Failed to open file '/mnt/share/ AMP/Roster/Archive/ac_all_Test.csv' for writing.  Please ensure that you have sufficient privileges to write this file.  If the file already exists, please ensure it is not open in another application
2024-06-19 07:48:46|  15.4|  0.0|ERROR |CSV2 writer: An error has occurred. Check the logfile above for details
2024-06-19 07:48:46|  15.4|  0.0|INFORM|Excel Reader: Closing dataset '/tmp/user/997/fmeengines/localhost_Engine1/TempFS_1718783309939_2216839/AMP All Candidates.xlsx'...
2024-06-19 07:48:46|  15.4|  0.0|INFORM|Excel Writer: Workbook contains 129 fonts and 375 cell formats after writing
2024-06-19 07:48:47|  15.8|  0.4|INFORM|Excel Reader: Closing dataset '/tmp/user/997/fmeengines/localhost_Engine1/TempFS_1718783309939_2216839/AMP All Candidates.xlsx'...
2024-06-19 07:48:47|  15.8|  0.0|INFORM|=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
2024-06-19 07:48:47|  15.8|  0.0|INFORM|Feature output statistics for `CSV2' writer using keyword `CSV2_1':
2024-06-19 07:48:47|  15.8|  0.0|STATS |=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
2024-06-19 07:48:47|  15.8|  0.0|STATS |                               Features Written
2024-06-19 07:48:47|  15.8|  0.0|STATS |=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
2024-06-19 07:48:47|  15.8|  0.0|STATS |ac_all_Test                                                                232
2024-06-19 07:48:47|  15.8|  0.0|STATS |==============================================================================
2024-06-19 07:48:47|  15.8|  0.0|STATS |Total Features Written                                                     232
2024-06-19 07:48:47|  15.8|  0.0|STATS |=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
2024-06-19 07:48:47|  15.8|  0.0|INFORM|Translation FAILED with 2 error(s) and 5 warning(s) (232 feature(s) output)
2024-06-19 07:48:47|  15.8|  0.0|INFORM|FME Session Duration: 18.1 seconds. (CPU: 14.4s user, 1.4s system)
2024-06-19 07:48:47|  15.8|  0.0|INFORM|END - ProcessID: 2216839, peak process heap usage: 1392194 kB, current process heap usage: 28355 kB
 

******************************************************************************************************

I’ll ask the account to see if they can open a ticket, not sure how long will that take but in the mean time if you have any other inputs it would be great.

Thanks again.

RZ.-

​

rzel24
Rank
Badge +3

Hello,

So I wasn’t able to get the ticket with safe created for this issue. Not sure why but it just didn’t happen.

In any case my issue with creating the files is that the fmeflow user doesn’t have any access to the Share Network Drives.

Once that is fixed I expect the problem to be solved. For now I will mark this as best answer.

Thanks for all your help.

RZ.-

Reply


Community Stats

29,620
Posts
112,572
Replies
37,269
Members
Terms & ConditionsCookie settings