Skip to main content
Solved

Access denied when fanning out many folder based dataset

theogerritsen
Contributor
Forum|alt.badge.img+2

I have a long workflow that transforms some DWG (for buildings), GeoTIFF (for terrain) and FBX (for trees) in 3D datasets (IFC, OBJ, FBX, glTF). The whole process takes a bit more than a day because it's on a quite huge input dataset. The output is fanned out based on a 500m x 500m grid.

When I was testing my workflow on a smaller dataset, it worked fine on the four different file formats. Now that I tested it on a bigger dataset, it only works on the file based datasets (IFC and OBJ). The workflow is still able to write the first part (terrain) in the folder based datasets (FBX and glTF), but as soon as it starts writing the buildings in the all ready created fanned out folders, I get an Access Denied (that I didn't get when I was working on smaller datasets).

Does anybody have any idea why ?

Best answer by nielsgerrits

theogerritsen wrote:

I did a full run directly, without FeatureCaching. I'm not reading the wrote files during the process. Basically, the script creates 3 different feature types for every file, and merges them together:

  • Buildings
  • Terrain
  • Trees

It first writes the terrain in a glTF file, then the buildings and then the trees, in the same file again. The weird thing is, it works on small datasets (16km^2) but not on larger ones. That's why I wanted to know if there was a known issue with FME that resctricts some things based on memory usage, or maybe a UNIX problem that prevents writing too many files in the same folder. The latter shouldn't be the problem as the same script works for file based datasets (IFC and OBJ), where I output more than 20'000 files in the same folder.

No experience with these formats, but if I read:

It first writes the terrain in a glTF file, then the buildings and then the trees, in the same file again.

I wonder, can it be that multiple writers are writing to the same file, at the same time? Because if that is the case, this is probably causing the access denied error.

 

This can work in a small set where due to accidental timing it works fine, but with a production run, with a lot of data and different timing, this can fail.

 

It is possible to force one FeatureWriter to wait for the previous one to finish. Connect the second FeatureWriter input and the first FeatureWriter summary outputport to a Holder, then a Tester to remove the summary feature from the data, send the rest to the inputport of the second FeatureWriter. Downside is this might take a lot of memory, due to all features waiting before they can move on.

 

2023-11-15_12h05_16

View original
Did this help you find an answer to your question?

4 replies

nielsgerrits
VIP
Forum|alt.badge.img+54

Did you test a full run or a partial run using FeatureCaching?

Are you reading the files you are writing during runtime?

Hard to say what is happening but this feels like a workflow order conflict to me.

theogerritsen
Contributor
Forum|alt.badge.img+2
  • theogerritsenContributor
  • Author
  • Contributor
  • November 15, 2023

I did a full run directly, without FeatureCaching. I'm not reading the wrote files during the process. Basically, the script creates 3 different feature types for every file, and merges them together:

  • Buildings
  • Terrain
  • Trees

It first writes the terrain in a glTF file, then the buildings and then the trees, in the same file again. The weird thing is, it works on small datasets (16km^2) but not on larger ones. That's why I wanted to know if there was a known issue with FME that resctricts some things based on memory usage, or maybe a UNIX problem that prevents writing too many files in the same folder. The latter shouldn't be the problem as the same script works for file based datasets (IFC and OBJ), where I output more than 20'000 files in the same folder.

nielsgerrits
VIP
Forum|alt.badge.img+54
  • nielsgerritsVIP
  • Best Answer
  • November 15, 2023
theogerritsen wrote:

I did a full run directly, without FeatureCaching. I'm not reading the wrote files during the process. Basically, the script creates 3 different feature types for every file, and merges them together:

  • Buildings
  • Terrain
  • Trees

It first writes the terrain in a glTF file, then the buildings and then the trees, in the same file again. The weird thing is, it works on small datasets (16km^2) but not on larger ones. That's why I wanted to know if there was a known issue with FME that resctricts some things based on memory usage, or maybe a UNIX problem that prevents writing too many files in the same folder. The latter shouldn't be the problem as the same script works for file based datasets (IFC and OBJ), where I output more than 20'000 files in the same folder.

No experience with these formats, but if I read:

It first writes the terrain in a glTF file, then the buildings and then the trees, in the same file again.

I wonder, can it be that multiple writers are writing to the same file, at the same time? Because if that is the case, this is probably causing the access denied error.

 

This can work in a small set where due to accidental timing it works fine, but with a production run, with a lot of data and different timing, this can fail.

 

It is possible to force one FeatureWriter to wait for the previous one to finish. Connect the second FeatureWriter input and the first FeatureWriter summary outputport to a Holder, then a Tester to remove the summary feature from the data, send the rest to the inputport of the second FeatureWriter. Downside is this might take a lot of memory, due to all features waiting before they can move on.

 

2023-11-15_12h05_16

theogerritsen
Contributor
Forum|alt.badge.img+2
  • theogerritsenContributor
  • Author
  • Contributor
  • November 15, 2023

Ok thanks a lot. I'll try this for sure. For now, I created three different workflows (one for the trees, one for the buildings and one for the terrain) that will create three seperate glTF for every grid cell. I'll merge them together afterwards. See if that works. But I'll definitely try your solution.

I also met Jean-Luc Miserez by chance yesterday, who is the director at INSER (FME's distributor in Switzerland). He told me that a "hack" of FME desktop is to be able to run up to 8 workflow simultaneously (each workflow will use 1 Core) to greatly increase the speed of a big workflow. So my solution of splitting up the workflow in three is not the most convenient but it will maybe greatly increase the speed (it took almost 2 full days of processing before). I'll see and let you know :)

Reply


Helpful Members This Week

Community Stats

31,358
Posts
118,951
Replies
38,988
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.