Workspace processing stalls at NeighborFinder in FME 2018.

If you have an option to use a Group By, the NeighborFinder performance increases by a lot (usually).

Still strange that in other versions this issue does not occur.

As you have run the workspace in FME 2017.1, I assume it is built using an older version.

Can the NeighborFinder transformer be upgraded (newer version of the transformer)?

Hope this helps.

To be clear, this is a 32 bit only issue with FME 2018? If that is so, I strongly suspect it had to do with an issue with our high data volume/low memory handling that we had in some circumstances (and it appears you were lucky enough to be in that select and <un>lucky group. If you're able to share the data and workspace with us we could verify. Sorry about that -- we do think we're through this issue now.

Unfortunately I can't use Group By option in this case.

Also I upgraded the transformers, but unfortunately it didn't help. It seems likely, that it is a version specific memory handling issue, as described in the comment above.

Nevertheless, thanks for the input!

Yes, it's an issue with 32 bit FME 2018 and the memory handling issues look likely to be the root of this problem. Unfortunately I can't share the data, as the contents of these databases aren't open access yet.

If it helps, then from FeatureReaders A, B, C comes polygon data, and from F line features. Furthermore, the NeighborFinder with ADS bookmark receives a little over 810 000 candidate and over 56 000 base features; AY gets 4 700 candidate and around 750 base features; KÜ1 gets over 710 000 candidate and around 16 500 base features and the problematic FeatureReader receives around 17 500 candidate and 16 500 base features.

I think I can get around the problem, if I fiddle around with FeatureReaders initiators a little so that all the data won't be read at once and some NeighborFinders will finish their job, before others get their input data.

Thanks for the reply!

Unfortunately your screenshot doesn't show up so it's hard to see if there's any changes you can make to the workflow to make it less memory-intensive.

One thing to consider, if you have the hardware, is to try the 64-bit version of FME. Alternatively... you mentioned the 2019 beta had no issue and the official release of that is due soon...

You need to create your own partitioning if you cannot use groupby. It will be worth the effort.

Do this by adding a code that does enable you to split up the data into groups. I do this when I use NeighborFinder on a national dataset of address points and roads. In my case one layer (roads) has a district code but the addresses do not. So I do an overlay first to add a district code.

Actually there are so many operations I create a workspace runner that runs the workbench repeatedly for each district by having an SQL filter that selects just the features for each district. Each process then runs on a small set very quickly so the overall time is not only much reduced, but it never fails due to memory overloads.

Don't have a suitable polygon layer? Then invent one. You could just make a grid, or a binary tree grid where the counts are more equal.

Hi,

I'm also facing the same issue...

Processing base feature(s)... is shown in log file.

I thought, it is memory issue and left for 12+ hours but no change.

@hlh, Is there any solution identified?

Pratap

Hi,

Unfortunately I have no solutions for the issue in FME desktop. However, when I ran the workspace on FME Server, it ran flawlessly and quickly. While in the desktop version I waited for hours without any luck, in FME Server I get a result in 10-15 minutes. I haven't tried with the latest FME release, but unfortunately I can't help you at the moment.

H-

I think it would help if you could add this as a new question (with a reference back to this one). It may not even be the same issue as here or there might be different information now that might help. If you can supply the log file and workspace it will help enormously in coming up with a solution.