Skip to main content
Solved

Unable to create a workbench directory (error 500)

  • February 12, 2021
  • 6 replies
  • 76 views
smol
Contributor
  • smolContributor
  • Contributor

Greetings,

 

Once in a while, our workbench using the "dataupload" service seems to return an error 500. We do not see any logs related to disk space, nor should it as it is only using ~20% of the available space. The following log is shown in the logs of "Resources -> Logs -> service -> current -> fmeserverweb_dataupload.log":

 

Fri-12-Feb-2021 07:55:46.873 AM   INFORM   http-nio-8080-exec-22   419011 : Creating new request object: path level = "1", requested path = "", session ID = "f257b28e-bdd2-49df-84eb-007d69dabc5e", uploaded files = [], workspace path = "/<repository>/<wb_name>.fmw", retrieve absolute paths = "true", extract files = "false"
java.io.IOException: Unable to create directory: /data/fmeserverdata/resources/system/temp/upload/<repository>/<wb_name>.fmw/f257b28e-bdd2-49df-84eb-007d69dabc5e
	at COM.safe.web.upload.StoreManager.setup(StoreManager.java:232)
	at COM.safe.web.upload.StoreManager.<init>(StoreManager.java:226)
	at COM.safe.web.upload.StoreManager.acquire(StoreManager.java:253)
	at COM.safe.webservices.upload.VolatileUploadServlet.doPost(VolatileUploadServlet.java:245)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
	at COM.safe.webservices.upload.VolatileUploadServlet.service(VolatileUploadServlet.java:72)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at COM.safe.webservices.utility.ServiceAuthFilter.doFilter(ServiceAuthFilter.java:64)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.ebaysf.web.cors.CORSFilter.handleNonCORS(CORSFilter.java:640)
	at org.ebaysf.web.cors.CORSFilter.doFilter(CORSFilter.java:173)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at COM.safe.webservices.AcceptMIMEAdapterFilter.doFilter(AcceptMIMEAdapterFilter.java:409)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at COM.safe.web.servlet.CharsetEncodingFilter.doFilter(CharsetEncodingFilter.java:541)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Fri-12-Feb-2021 07:55:46.874 AM   ERROR    http-nio-8080-exec-22   419012 : Request failure, status 500, Unable to create directory: /data/fmeserverdata/resources/system/temp/upload/<repository>/<wb_name>.fmw/f257b28e-bdd2-49df-84eb-007d69dabc5e

 

 Note, if we try this again it seems to work fine. Is there anything specific reason for this behaviour, or is there a lead that I may follow for debugging?

 

Best answer by jlutherthomas

This issue occurs when two data upload tasks occur at the same time/very close together, which results in a both requests trying to make a folder at the same time (which fails, as seen in the log file). At the moment, the best 'workaround' is for the user that experiences a failed upload is to try again.

 

This has been filed as FMESERVER-16637 if anyone else experiences this.

View original
Did this help you find an answer to your question?

6 replies

hkingsbury
Celebrity
Forum|alt.badge.img+54
  • hkingsburyCelebrity
  • Celebrity
  • February 14, 2021

The fact its saying that it can't create a directory suggest to me that the service account running FME Server doesn't have permissions to do so. However the fact that its only happening sporadically complicates things.

 

have a look here: https://docs.safe.com/fme/html/FME_Server_Documentation/AdminGuide/Running_System_Services_Under_Different_Accounts.htm

smol
Contributor
  • smolContributor
  • Author
  • Contributor
  • February 22, 2021
hkingsbury wrote:

The fact its saying that it can't create a directory suggest to me that the service account running FME Server doesn't have permissions to do so. However the fact that its only happening sporadically complicates things.

 

have a look here: https://docs.safe.com/fme/html/FME_Server_Documentation/AdminGuide/Running_System_Services_Under_Different_Accounts.htm

Thanks for your comment. Sadly, as you also mentioned, it indeed works when retrying with exact same `POST`. The users of FME Core and FME Server are running with exact same `UID` (Using the FME Docker setup) and can correctly write files to the shared file.

hkingsbury
Celebrity
Forum|alt.badge.img+54
  • hkingsburyCelebrity
  • Celebrity
  • February 22, 2021
smol wrote:

Thanks for your comment. Sadly, as you also mentioned, it indeed works when retrying with exact same `POST`. The users of FME Core and FME Server are running with exact same `UID` (Using the FME Docker setup) and can correctly write files to the shared file.

Hmmm, never really used Docker so i'm running out of ideas. Have you opened a support ticket?

J
Forum|alt.badge.img+2

This issue occurs when two data upload tasks occur at the same time/very close together, which results in a both requests trying to make a folder at the same time (which fails, as seen in the log file). At the moment, the best 'workaround' is for the user that experiences a failed upload is to try again.

 

This has been filed as FMESERVER-16637 if anyone else experiences this.

smol
Contributor
  • smolContributor
  • Author
  • Contributor
  • April 13, 2021
jlutherthomas wrote:

This issue occurs when two data upload tasks occur at the same time/very close together, which results in a both requests trying to make a folder at the same time (which fails, as seen in the log file). At the moment, the best 'workaround' is for the user that experiences a failed upload is to try again.

 

This has been filed as FMESERVER-16637 if anyone else experiences this.

Thank you for the investigation. The workaround is sufficient enough for now.

 

Last question, what is the process of these (internal) tickets? Is there a way for us to follow this ticket (FMESERVER-16637) about the discussion, or is the fully private and internal process?

 

J
Forum|alt.badge.img+2
smol wrote:

Thank you for the investigation. The workaround is sufficient enough for now.

 

Last question, what is the process of these (internal) tickets? Is there a way for us to follow this ticket (FMESERVER-16637) about the discussion, or is the fully private and internal process?

 

Yes, tickets are private, but if you (or anyone else) wants to know the status of it you can ask us and with that reference number it's faster for us to look it up.

 

Also, as you raised a support case your case is linked to that issue internally, so you will get an automated email when it's fixed and what build of FME it's fixed in. If any other customer experiences this and would also like to get notified, they can create a support case, reference that issue number and we can link their case too.

Reply


Helpful Members This Week

Community Stats

31,334
Posts
118,864
Replies
38,957
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.