Solved

Workspace Runner for Batch Processing by Clip Region


I'm using FME Desktop to run Validation Checks on a large ESRI SDE network. The intention is to run the checks daily on recently edited data.

 

While a filter can be applied on some checks (i.e.: checking attributes are valid), others need to be checked across the full network, i.e.: a Point Feature added today needs to be checked against the full Line dataset to ensure its location is valid - a Valve must be located at the end point of 2 Mains, etc.

These large filters are difficult to process so I was hoping to use Workspace Runner to split the workspaces by location, so instead of say:

 

A. Checking all Valves in the whole network against all Mains

We

B. Check all Valves in Polygon 1 against all Mains in Polygon 1

C. Then check all Valves in Polygon 2 against all Mains in Polygon 2

D. etc. etc.

I'm not that experienced with FME but are there any tutorials or can anybody tell me how to do this using the Workspace Runner and say the Clip transformer. I've seen the following but don't fully understand how I would edit the principles to do what I want, i.e.: what do I need to do in either Workspace so that the Workspace Runner only runs based on each given extent?

https://knowledge.safe.com/articles/1469/batch-processing-using-the-workspacerunner-1.html

icon

Best answer by ebygomm 20 May 2019, 09:59

View original

8 replies

Userlevel 3
Badge +13

Hi @dsald Are "Polygon 1" and "Polygon 2" attribute values on Valves and Mains or are they feature type names? Ideally you could use a NeighborFinder, setting a Maximum Distance and grouping by a common attribute value. That should be faster than tiling the entire dataset with a Tiler, and better than a Clipper. If you're trying trying to check the location of valves (points) against the ends of Mains (lines) then you might also want to try a TopologyBuilder or a Chopper with a Matcher.

Badge +10

I wouldn't personally use the clipper for this scenario if your data is in sde.

I'd do something along the lines of the following

Set up grid covering area of interest

Child workspace

  • Read grid into child workspace
  • Link tester up to a user parameter to pass only the grid value selected
  • Use a feature reader with a bounding box to only read in the data for that area to test against

Parent workspace

  • Read grid into workspace
  • Send each feature to workspace runner, use attribute value from grid feature as your test parameter in the workspace runner

 

Depending on your exact checks you may need to allow for the fact that features that fall across grid boundaries may be tested in multiple runs.

 

@egomm Thanks, I'm trying to apply your method as I think it may work for the entire (20 or so checks) but struggled with the Feature Reader.

 

 

I've applied it on a small workspace here. When I run Feature Reader on the SDE, I don't get any outputs. I've tested it here with a known Grid Reference Name (B2 of 3 x 3 grid therefore the one that should have the majority outputs) but I don't get anything in the Inspector. Is there potentially something I'm doing wrong/a step I'm missing here or is there an issue with reading SDE features in Feature Reader?

Workspace and settings are as below. All else is set to default.

 

Badge +10

@egomm Thanks, I'm trying to apply your method as I think it may work for the entire (20 or so checks) but struggled with the Feature Reader.

 

 

I've applied it on a small workspace here. When I run Feature Reader on the SDE, I don't get any outputs. I've tested it here with a known Grid Reference Name (B2 of 3 x 3 grid therefore the one that should have the majority outputs) but I don't get anything in the Inspector. Is there potentially something I'm doing wrong/a step I'm missing here or is there an issue with reading SDE features in Feature Reader?

Workspace and settings are as below. All else is set to default.

 

Have you tried bounding boxes intersect in your spatial filter?

@egomm Thanks, I'm trying to apply your method as I think it may work for the entire (20 or so checks) but struggled with the Feature Reader.

 

 

I've applied it on a small workspace here. When I run Feature Reader on the SDE, I don't get any outputs. I've tested it here with a known Grid Reference Name (B2 of 3 x 3 grid therefore the one that should have the majority outputs) but I don't get anything in the Inspector. Is there potentially something I'm doing wrong/a step I'm missing here or is there an issue with reading SDE features in Feature Reader?

Workspace and settings are as below. All else is set to default.

 

I tried it there with bounding boxes intersect and still no output. issue with reading SDE data?

Badge +10

I tried it there with bounding boxes intersect and still no output. issue with reading SDE data?

Is your grid in the same coordinate system? With the correct inputs, there should be no issue using a featurereader with SDE data

@egomm Thanks, I'm trying to apply your method as I think it may work for the entire (20 or so checks) but struggled with the Feature Reader.

 

 

I've applied it on a small workspace here. When I run Feature Reader on the SDE, I don't get any outputs. I've tested it here with a known Grid Reference Name (B2 of 3 x 3 grid therefore the one that should have the majority outputs) but I don't get anything in the Inspector. Is there potentially something I'm doing wrong/a step I'm missing here or is there an issue with reading SDE features in Feature Reader?

Workspace and settings are as below. All else is set to default.

 

I created the Grid as a FDB file over the SDE dataset so I assume it has the same co-ordinates. I have a Boundary layer in the SDE dataset and its working fine now using that as a filter.

@egomm Thanks, I'm trying to apply your method as I think it may work for the entire (20 or so checks) but struggled with the Feature Reader.

 

 

I've applied it on a small workspace here. When I run Feature Reader on the SDE, I don't get any outputs. I've tested it here with a known Grid Reference Name (B2 of 3 x 3 grid therefore the one that should have the majority outputs) but I don't get anything in the Inspector. Is there potentially something I'm doing wrong/a step I'm missing here or is there an issue with reading SDE features in Feature Reader?

Workspace and settings are as below. All else is set to default.

 

yeah, it was the projection... more my issues with ArcMap than FME. Grid was created as geographic coordinate system so changed it to projected coordinate system to match my layers. thanks, it works really well!

Reply