Solved

Unable to create a workbench directory (error 500)

  • 12 February 2021
  • 6 replies
  • 16 views
smol
Rank

Greetings,

 

Once in a while, our workbench using the "dataupload" service seems to return an error 500. We do not see any logs related to disk space, nor should it as it is only using ~20% of the available space. The following log is shown in the logs of "Resources -> Logs -> service -> current -> fmeserverweb_dataupload.log":

 

Fri-12-Feb-2021 07:55:46.873 AM   INFORM   http-nio-8080-exec-22   419011 : Creating new request object: path level = "1", requested path = "", session ID = "f257b28e-bdd2-49df-84eb-007d69dabc5e", uploaded files = [], workspace path = "/<repository>/<wb_name>.fmw", retrieve absolute paths = "true", extract files = "false"
java.io.IOException: Unable to create directory: /data/fmeserverdata/resources/system/temp/upload/<repository>/<wb_name>.fmw/f257b28e-bdd2-49df-84eb-007d69dabc5e
	at COM.safe.web.upload.StoreManager.setup(StoreManager.java:232)
	at COM.safe.web.upload.StoreManager.<init>(StoreManager.java:226)
	at COM.safe.web.upload.StoreManager.acquire(StoreManager.java:253)
	at COM.safe.webservices.upload.VolatileUploadServlet.doPost(VolatileUploadServlet.java:245)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
	at COM.safe.webservices.upload.VolatileUploadServlet.service(VolatileUploadServlet.java:72)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at COM.safe.webservices.utility.ServiceAuthFilter.doFilter(ServiceAuthFilter.java:64)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.ebaysf.web.cors.CORSFilter.handleNonCORS(CORSFilter.java:640)
	at org.ebaysf.web.cors.CORSFilter.doFilter(CORSFilter.java:173)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at COM.safe.webservices.AcceptMIMEAdapterFilter.doFilter(AcceptMIMEAdapterFilter.java:409)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at COM.safe.web.servlet.CharsetEncodingFilter.doFilter(CharsetEncodingFilter.java:541)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Fri-12-Feb-2021 07:55:46.874 AM   ERROR    http-nio-8080-exec-22   419012 : Request failure, status 500, Unable to create directory: /data/fmeserverdata/resources/system/temp/upload/<repository>/<wb_name>.fmw/f257b28e-bdd2-49df-84eb-007d69dabc5e

 

 Note, if we try this again it seems to work fine. Is there anything specific reason for this behaviour, or is there a lead that I may follow for debugging?

 

icon

Best answer by jlutherthomas 13 April 2021, 03:11

View original

6 replies

hkingsbury
Rank
Userlevel 5
Badge +29

The fact its saying that it can't create a directory suggest to me that the service account running FME Server doesn't have permissions to do so. However the fact that its only happening sporadically complicates things.

 

have a look here: https://docs.safe.com/fme/html/FME_Server_Documentation/AdminGuide/Running_System_Services_Under_Different_Accounts.htm

Check out the Abley FME Firesides on Youtube: https://www.youtube.com/@ableyltd
smol
Rank

The fact its saying that it can't create a directory suggest to me that the service account running FME Server doesn't have permissions to do so. However the fact that its only happening sporadically complicates things.

 

have a look here: https://docs.safe.com/fme/html/FME_Server_Documentation/AdminGuide/Running_System_Services_Under_Different_Accounts.htm

Thanks for your comment. Sadly, as you also mentioned, it indeed works when retrying with exact same `POST`. The users of FME Core and FME Server are running with exact same `UID` (Using the FME Docker setup) and can correctly write files to the shared file.

hkingsbury
Rank
Userlevel 5
Badge +29

Thanks for your comment. Sadly, as you also mentioned, it indeed works when retrying with exact same `POST`. The users of FME Core and FME Server are running with exact same `UID` (Using the FME Docker setup) and can correctly write files to the shared file.

Hmmm, never really used Docker so i'm running out of ideas. Have you opened a support ticket?

Check out the Abley FME Firesides on Youtube: https://www.youtube.com/@ableyltd
J
Badge +2

This issue occurs when two data upload tasks occur at the same time/very close together, which results in a both requests trying to make a folder at the same time (which fails, as seen in the log file). At the moment, the best 'workaround' is for the user that experiences a failed upload is to try again.

 

This has been filed as FMESERVER-16637 if anyone else experiences this.

smol
Rank

This issue occurs when two data upload tasks occur at the same time/very close together, which results in a both requests trying to make a folder at the same time (which fails, as seen in the log file). At the moment, the best 'workaround' is for the user that experiences a failed upload is to try again.

 

This has been filed as FMESERVER-16637 if anyone else experiences this.

Thank you for the investigation. The workaround is sufficient enough for now.

 

Last question, what is the process of these (internal) tickets? Is there a way for us to follow this ticket (FMESERVER-16637) about the discussion, or is the fully private and internal process?

 

J
Badge +2

Thank you for the investigation. The workaround is sufficient enough for now.

 

Last question, what is the process of these (internal) tickets? Is there a way for us to follow this ticket (FMESERVER-16637) about the discussion, or is the fully private and internal process?

 

Yes, tickets are private, but if you (or anyone else) wants to know the status of it you can ask us and with that reference number it's faster for us to look it up.

 

Also, as you raised a support case your case is linked to that issue internally, so you will get an automated email when it's fixed and what build of FME it's fixed in. If any other customer experiences this and would also like to get notified, they can create a support case, reference that issue number and we can link their case too.

Reply


Community Stats

28,716
Posts
109,494
Replies
36,561
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.