Replace the LRS* hacks for linear referencing with a proper set of transformers

For example. The MeasureChecker has to resort to a PythonCaller to detect if measures are present. I would like to see better dimension types. Not just 2D and 3D, 2DM and 3DM.Setting the writer to output measures is an obscure writer setting. It should be automatic.

Hi @kimo, thanks for letting us know that we've got some work to do on the LRS side of things. I do know many customers achieve great results with what we've got, but am open to feedback as to what we could do better. Would you be willing to share some example problems that you're looking to solve where we fall short? Can you provide an example of where the NeighbourFinder doesn't return the measures as documented? Definitely interested in fixing/extending things, just do need some specific examples and workflows.

I would concur Kimo's comments and suggestions. I remember how much we struggled when developing the LRS transformers that are published on the Hub.

I would suggest treating measurements as another dimension to reckon with when it exists in the geometry throughout transformations. On top of that, let us add special functions such as calibration, route building, and route QC. Oh, there is also the need for functions to handle the aspatial events.

One example where I have had trouble is with intersecting two event tables (some call dynamic segmentation). By this I mean creating a new segment whenever an attribute changes. I have successfully completed it with the intersector, lineOnLineOverlayer, inline querier.

The issue i find though is whenever the datasets get large the runtime is too slow. The workflow attached is based on @MarkAtSafe https://knowledge.safe.com/questions/58052/is-it-possible-to-do-a-dynamic-segmentation-on-des.html post. It runs in seconds for small datasets, but is 4hrs on my 64bit fme for the large sample data attached. I have similar lengths of runs with all the other methods I have tested (LineOnLine, inLineQuerier, cross join and aggregate). Perhaps my expectations are too high, but I feel it can be better.

Thanks!

-Updated to add example data

the readers will need to be updated as I split the files to load on the post

this file is actually a .7z compressed file and requires 7zip to decompress (extension may need to be changed to .7z)

Rest of example data

Thanks, I updated with the example data. The workflow readers will need to be re-configured as I had to 7zip the large data set separately as a csv, to fit the size requirements.

@notmyname Thank you for supplying the example data. There is definitely some work that we can do in FME to make working with LRS and events easier. I would suggest adding some Ideas for the types of LRS tools you are looking for, to help us prioritize improvements to FME's LRS support.

The the example workspace using the Intersector does indeed take a long time with a larger dataset. In general, geometry operations are more computationally expensive than attribute operations so if you can replace a geometry operation with an equivalent attribute operation then usually performance will improve.

The example data, where we need to intersect two event tables without an existing 'route' is also interesting because their is minimal 'splitting' of the events (i.e. start/end do not overlap). In this example data, one of the events tables (Animal events) has very few, long segments and the second event table (Land events) has many, very short segments. In this case it's far more efficient to use a SQL query to intersect the two event tables:

You can use the InlineQuerier to do this and also identify events that need to be split. I'm reverting to FeatureJoiner to identify events that do not intersect. Intersector can be used to split the few events that intersect at the start/end locations (as per the original example). I'm sure someone with stronger SQL skills than I have could make this more compact and efficient. The attached example takes under 3 minutes (FME 2018.1): (it uses the same data as above - and )

I know this does not address @kimo's concerns about simplicity of the workflow, but it does address the performance issues. Again, it would really help us if you posted Ideas or vote and comment on existing Ideas to help direct improvements for LRS handling in FME.

@MarkAtSafe thanks for the detailed response. When I have time I will put together the LRS workflows I have made and where they reach the limitations. InlineQuerier is indeed quick when smaller sets are used, but is very slow when two large sets of segments are used (owing to the range condition as opposed to an equal condition). I think there is a desire for a ETL tool effective with ranges, as when it comes to LRS other options are proprietary databases designs and very slow software.

How do we transfer from one linear referencing system to another?

It can definitely be done and all the tools are there. I suggest you give a careful view to our recent webinar on linear referencing in FME -- that should get you started.

I think I've got it figured out, but it has not been as simple as a single transformer as I'd hoped.