Skip to main content
Hi all, 

 

 

I found this wonderful piece of informatjon in the description of ArcSDE reader (http://docs.safe.com/fme/html/FME_ReadersWriters/Default.htm#sde30/sde30.htm%3FTocPath%3DFME%20Readers%20and%20Writers%20(formats%20supported%20by%20FME%202014)%7CEsri%20ArcSDE%20Reader%2FWriter%7C_____0):

 

The SDE reader can also perform multi-table join queries, thereby exploiting the full power of the underlying RDBMS.

 

 

Cool... I wanna do that, and most likely either the ArcSDEQuerier or ArcSDE reader, whichever is most convenient. 

 

 

But I can't seem to wrap my head around the syntax. Composing a valid JOIN statement in SQL should be fairly straightforward, but that doesn't really seem to fit into the concept of WHERE Clause. Or I cant figure it out... 

 

 

Problem description: I have a huge ArcSDE database. I and I want to extract all data from table A, and a subset of information from table B, only extracting those whose ROUTE_ID values match ROUTE_ID in table A.  In my head, this should translate to a  very straightforward SQL:

 

SELECT * FROM A, B 

 

WHERE A.ROUTE_ID = B.ROUTE_ID

 

 

And FME ArcSDE reader takes care of anything up to and including the "WHERE", so i should input the  "A.ROUTE_ID = B.ROUTE_ID" - part where it says "WHERE Clause". 

 

 

Apparently not. What am I missing here? Or are there other, quicker ways to achieve what I want? 

 

 

I've also tested extracting table A and then separately get the data from table A that satisfy my conditions. Both the Joiner and FeatureReader are more than happy to do what I want - but way to sloooow, reading one. single. feature. per. database-query. Table A is big enough to make this unacceptable slow. (Although I could do it over the week-end, I guess). 

 

 

SQL executor / creator doesn't seem to support arcSDE, or at least I can't find that option. 

 

 

Other options I've considered:

 

 - Read A and B into, say, sqlite, then perform magic with the SQL executor.

 

 - Read A and B into workspace, then use FeatureMerger. Which will work, but slooooooooow. This is probably what I'll do if I can't make any progress on the SQL magic, letting the server do the work it's supposed to be really good at doing. 

 

 

 

 

 

 

 

 

where clause in reader is limited.

 

 

You want to do a left outer join. Cant do that in the whereclause of the reader afaik.

 

I found this handy work on the net.

 

 

 

 


Dear Gio, 

 

 

Thanks for the informative and nice "Joins" - illustration. 

 

 

In my particular case an "inner join" is equivalent to "left outer join", since Table A.Route_ID is guaranteed to be a subset of B.Route_ID. But in the general case you'd be right (unless you specifically wanted to ditch the non-matching part of A). 

 

 

In a way, it is somewhat comforting for me to notice that the both of us seem stuck in our futile attempts of implementing the wonderfull multi-table join queries, thereby exploiting the full power of the underlying RDBMS ...I may be utterly wrong and way beyond my skills and understanding here, but at least I'm not alone. 

 

 

It would be incredible nice if the RDBMS - FME reader guru's at Safe could shed some light on this. 

 

 

 


Hi,

 

 

I just talked to somebody at Safe about this, and it confirmed my suspicions: the ability to do joins in the SDE30 reader is based on setting the parameter SDE30_ID in the mapping file.

 

 

The problem is that this parameter isn't accessible in the Workbench, so it's rather difficult to do. Not impossible, but a bit complicated and somewhat contrived.

 

 

Personally, I find the following more intuitive and easier to maintain: use the SQLCreator and cast the shape column into a Well-Known Binary (WKB) format, like this:

 

 

select

 

  objectid,

 

  mf1.myattrib,

 

  mf2.myotherattrib,

 

  ...etc...

 

  sde.ST_AsBinary(shape) as shape

 

from

 

  myfeatureclass1 mf1 join myfeatureclass2 mf2 on (mf1.pk = mf2.fk);

 

 

Then convert the WKB "shape" attribute into an FME geometry using the GeometryReplacer:

 

 

 

 

This makes for a very flexible solution and I've used it successfully many times.

 

 

David

 

 

 


Whoops, wrong parameter setting in the GeometryReplacer above. It should be like this:

 

 

 

 

David
Now that is absolutely übercool! I'll try that as soon as I can!

 

 

And thanks to FME's wonderfully verbose  logging routines I can get about 95% of the SQL syntax right out of the log. 

 

 

 


Of course I forgot: ArcSDE is *not* supported by the sqlCreator (or SQLExecutor). 

 

 

Most likely way out of this is to read both tables into workspace, then do my featureMerger-thing. This *will* work, but is a) slow, b) primitive. 

 

 

Subdivision iteration over the subsets shouild speed up things, but I have no common factor that is guaranteed to do subdivision correct among the two data sets.. 

 

 

Alternatively, I could dump each table  into a database that IS supported, and do joins on that. 
Hi,

 

 

that is true, SDE isn't supported by the SQLCreator. That is exactly why I do it the way I describe above 😉 Just select the reader that suits the underlying database, e.g. Oracle, and typecast the shape to a binary and off you go. Works very well for most cases.

 

 

David
Ah.. of course! *Blinding flash of the obvious* Thanks for feeding me this information with propery sized tea spoons

 

 

 

 

And the reason I didn't see Oracle among available options in SQLCreator is because on my system I need to switch over to my oracle license version to speak with Oracle. 

 

 

 
Hi,

 

 

that is true, SDE isn't supported by the SQLCreator. That is exactly why I do it the way I describe above 😉 Just select the reader that suits the underlying database, e.g. Oracle, and typecast the shape to a binary and off you go. Works very well for most cases.

 

 

David

We're doing the same thing and I noticed that the geometry replacer is taking a long time to run. I'm looking for ways to speed this up. Do you know of any, or am I stuck using the sdequerier and geometry replacer?


We're doing the same thing and I noticed that the geometry replacer is taking a long time to run. I'm looking for ways to speed this up. Do you know of any, or am I stuck using the sdequerier and geometry replacer?

Hi @runneals, thank you for your question. I see this original post is from quite a long time ago, and I'd encourage you to post this as a new question for better visibility. You're always welcome to link back to this post for the context. Thank you!


Reply