Skip to main content
  • EN

I had a minor task of buffering just over 70 000 points in lat-long by half a meter. I thought I would try the new parameters on the Bufferer instead of using the GeographicBufferer and the performance was very disappointing.

 

Has anyone else experienced this?

I decided to run a few tests with the same data and the results were as follows.

 

Reader -> Bufferer rBuffer Distance: 5e-6, Buffer Distance Units: Ground Units (None)]

 

runtime: 38.9s

 

 

Reader -> Bufferer eBuffer Distance: 0.5, Buffer Distance Units: Meters]

 

runtime: 4h 16m

 

 

Reader -> GeographicBufferer uBuffer Distance: 0.5]

 

runtime: 2min 0.5s

 

 

Reader -> CommonLocalReprojectorR_AZMED_] -> Bufferer ;Buffer Distance: 0.5, Buffer Distance Units: Ground Units (None)] -> Reprojector eLL-WGS84]

 

runtime: 49.8s

 

 

Reader -> Reprojectort_AZMED_] -> Bufferer &Buffer Distance: 0.5, Buffer Distance Units: Ground Units (None)] -> Reprojector ;LL-WGS84]

 

runtime: Error, apparently bad things happen with the Reprojector when the features are in Bulk Mode - all the points became (nan, nan) instead of the expected (0,0)

I'm not sure if this needs a bump or something but I feel that this issue has just fallen through the cracks. I'm also surprised that for such a simple process that there hasn't been a little more attention towards this.

 

With the release of 2021.0 I thought I'd check to see whether the Bufferer (now version 17) performance has improved.

To put it simply, it's still extremely disappointing and cannot compare to the GeographicBufferer running in our 2019.1 workspaces.

 

I'm not sure if this needs a bump or something but I feel that this issue has just fallen through the cracks. I'm also surprised that for such a simple process that there hasn't been a little more attention towards this.

 

With the release of 2021.0 I thought I'd check to see whether the Bufferer (now version 17) performance has improved.

To put it simply, it's still extremely disappointing and cannot compare to the GeographicBufferer running in our 2019.1 workspaces.

 

Thought I'd create an idea for its return: https://community.safe.com/s/idea/0874Q000000LP1gQAG/detail

I'm not sure if this needs a bump or something but I feel that this issue has just fallen through the cracks. I'm also surprised that for such a simple process that there hasn't been a little more attention towards this.

 

With the release of 2021.0 I thought I'd check to see whether the Bufferer (now version 17) performance has improved.

To put it simply, it's still extremely disappointing and cannot compare to the GeographicBufferer running in our 2019.1 workspaces.

 

I take it you see the same poor performance? We traced it to an issue with our IT security policies, clean windows 10 machines without any company policies applied did not show the horrendous performance. We also have a significant difference in speed when adding readers between machines with and without policies.

I'm not sure if this needs a bump or something but I feel that this issue has just fallen through the cracks. I'm also surprised that for such a simple process that there hasn't been a little more attention towards this.

 

With the release of 2021.0 I thought I'd check to see whether the Bufferer (now version 17) performance has improved.

To put it simply, it's still extremely disappointing and cannot compare to the GeographicBufferer running in our 2019.1 workspaces.

 

Thanks @jdh​,

I have to admit, when I encountered this in multiple business environments I ruled that out the company IT sources but it's entirely possible that the environments have similar security policies.

I'll have to look at doing some testing external to the work environment. Were you able to solve the issue within your company? I fear that our IT security policies will be out of bounds on my side which leaves us with utilising a work around.

 

Thanks again.

We did believe we addressed this with FME 2021.0 but @milo89​ 's experience seems to indicate it isn't totally gone. I'm super shocked and surprised that IT security could in any way interact with this functionality. Will ask the team to dig in.

Just did some tests myself and am seeing reasonable performance. @milo89​ please do send in your log and data/workspace (if possible) to support@safe.com and we'll see what is going on.

Just did some tests myself and am seeing reasonable performance. @milo89​ please do send in your log and data/workspace (if possible) to support@safe.com and we'll see what is going on.

Thanks @Dale Lutz​.

I'll bundle some stuff together and send it through.

Cheers.

Thanks @jdh​,

I have to admit, when I encountered this in multiple business environments I ruled that out the company IT sources but it's entirely possible that the environments have similar security policies.

I'll have to look at doing some testing external to the work environment. Were you able to solve the issue within your company? I fear that our IT security policies will be out of bounds on my side which leaves us with utilising a work around.

 

Thanks again.

No, our IT security policies were out of bounds as well, other than in specific air gapped labs.

 

Our workaround for the main network is Reproject to azmed-> Bufferer -> reproject to original projection. If there is a group-by needed on the bufferer, we used the CommonLocalReprojector.

We did believe we addressed this with FME 2021.0 but @milo89​ 's experience seems to indicate it isn't totally gone. I'm super shocked and surprised that IT security could in any way interact with this functionality. Will ask the team to dig in.

I won't swear to this, but my suspicion is that Digital Guardian is the culprit. Maybe the bufferer in geographic modes writes to a cache that DG needs to inspect?

We did believe we addressed this with FME 2021.0 but @milo89​ 's experience seems to indicate it isn't totally gone. I'm super shocked and surprised that IT security could in any way interact with this functionality. Will ask the team to dig in.

So I just ran a quick simple test in FME2021.0

2DGridCreator->Bufferer

GridCreator was 1000x1000 at a spacing of 0.0001 in wgs84 for a million geographic points.

 

Bufferer 0.00005 Ground Units (None) took 32 seconds cpu 28.2s user, 0.5s system with a peak ram of 62296KB

 

Bufferer 5 meters, took 30 minutes 56.7s cpu 248.6s user, 1064.8s system with a peak ram of 107928 kB

 

 

We did believe we addressed this with FME 2021.0 but @milo89​ 's experience seems to indicate it isn't totally gone. I'm super shocked and surprised that IT security could in any way interact with this functionality. Will ask the team to dig in.

This looks consistent with some of the issues we're getting.

Thanks @jdh​,

I have to admit, when I encountered this in multiple business environments I ruled that out the company IT sources but it's entirely possible that the environments have similar security policies.

I'll have to look at doing some testing external to the work environment. Were you able to solve the issue within your company? I fear that our IT security policies will be out of bounds on my side which leaves us with utilising a work around.

 

Thanks again.

Yeah, sounds similar to our circumstances.

I did some initial testing outside our network (only had 2019.1 available) and it definitely wasn't as slow but still not as quick as the work around or GeographicBufferer.

 

Our workaround is much the same. I've bundled some workspaces together for a support case where the workaround doesn't work in 2021.0, so we'll see how we go and hope there is something simple that isn't IT security related.

Thanks @jdh​,

I have to admit, when I encountered this in multiple business environments I ruled that out the company IT sources but it's entirely possible that the environments have similar security policies.

I'll have to look at doing some testing external to the work environment. Were you able to solve the issue within your company? I fear that our IT security policies will be out of bounds on my side which leaves us with utilising a work around.

 

Thanks again.

I forgot to mention, if high precision isn't required, then buffering with 0.00001 as a proxy for 1m is faster than the above workaround.

Thanks @jdh​,

I have to admit, when I encountered this in multiple business environments I ruled that out the company IT sources but it's entirely possible that the environments have similar security policies.

I'll have to look at doing some testing external to the work environment. Were you able to solve the issue within your company? I fear that our IT security policies will be out of bounds on my side which leaves us with utilising a work around.

 

Thanks again.

Hi @jdh​ Can you please send me your workspace and log file where you're buffering 1M points at 5m? My translation finishes in @ 4 minutes. thanks

Thanks @jdh​,

I have to admit, when I encountered this in multiple business environments I ruled that out the company IT sources but it's entirely possible that the environments have similar security policies.

I'll have to look at doing some testing external to the work environment. Were you able to solve the issue within your company? I fear that our IT security policies will be out of bounds on my side which leaves us with utilising a work around.

 

Thanks again.

Hi Dan,

I'll send you an email, though it's literally a 2DGridCreator attached to a Bufferer.

Just updating folks (and thanks for highlighting the issue). We've definitely found the cause and a fix just went in to FME 2021.1 (and later) starting with build 21539. Basically our arc reprojection code was repeatedly searching through the coordinate system database for a tolerance. With modern FMEs, buffers start out as arcs and so there's a lot of arc reprojection going on if we're doing units - based buffering of lat/long input. On systems with super fast disk this is less noticeable but if anything impedes disk access the slowdown is crushing. Of course the fastest disk access is no disk access, which is what the fix results in.

 

Anyway sorry for this issue -- help is on the way soon.

Just updating folks (and thanks for highlighting the issue). We've definitely found the cause and a fix just went in to FME 2021.1 (and later) starting with build 21539. Basically our arc reprojection code was repeatedly searching through the coordinate system database for a tolerance. With modern FMEs, buffers start out as arcs and so there's a lot of arc reprojection going on if we're doing units - based buffering of lat/long input. On systems with super fast disk this is less noticeable but if anything impedes disk access the slowdown is crushing. Of course the fastest disk access is no disk access, which is what the fix results in.

 

Anyway sorry for this issue -- help is on the way soon.

Thanks @Dale Lutz​, @danatsafe​ and @jdh​ for all your efforts.

Looking forward to trying it out.

Breaking news -- you won't have to wait until FME 2021.1 -- just learned that the team also backported to 2021.0.1 -- anything later than build 21311 will have the fix as well. Release of 2021.0.1 should be on or about April 6, 2021.

Reply


Community Stats

30,179
Posts
114,558
Replies
37,844
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.