Skip to main content

OK, here goes.  I have some data in a SQL Server database but it is stored in an object-oriented fashion which means a normal SQL query doesn't work, I have to pivot data in order to create a traditional tabular view.  Now that in itself wouldn't be a problem, but all data is stored in the same location with each asset class (e.g. door, window etc.) having a different number of attributes, so depending on how I pivot I get the relevant attributes as fields.  All good so far?  

OK, so I want to push this data from the source database into the target database (Oracle in this case) and the method for doing this is currently using SQLCreator to create the fields by which I want to pivot (based on the asset class as a parameter) and then SQLExecutor to retrieve the pivoted table, which works, up to a point.  What doesn't work is that the attributes of the output from the SQLExecutor are not exposed, and I can't get them to be exposed without Populate from SQL Query option, which fails as it relies on SQLCreator output to input into the SQL Statement.  Still with me?

To illustrate the code for the SQLCreator looks like this:

Select distinct
    substring(
        (
            Select ',"'+ T1.char_name + '"' AS Âtext()]
            From characteristics T1
INNER JOIN class_attributes t2 ON t1.char_id = t2.char_id
INNER JOIN class_objects t3 ON t2.class_id = t3.class_id
WHERE t3.name = '$(Asset_Class)'
AND t3.name = t4.name
            For XML PATH ('')
        ), 2, 1000) IAttributes]
From class_objects T4
where t4.name = '$(Asset_Class)';

and my SQLExecutor code looks like this:

SELECT object_id,class_id,name,code,description, @Value(Attributes)
FROM
  (SELECT 
  T1.char_name, 
  t3.class_id,
  t3.name,
  T4.object_id,
  T5.name tag_name,
  T5.code,
  T5.description,
  case when T4.char_value is not null then T4.char_value
  when T4.number_value is not null then CAST(T4.number_value as nvarchar)
  when T4.date_value is not null then CAST(T4.date_value as nvarchar)
  else null
  end as output_value
FROM  char_data T4
INNER JOIN characteristics T1 ON t4.char_id = t1.char_id
INNER JOIN tags t5 ON t4.object_id = t5.tag_id
INNER JOIN class_objects t3 ON t5.class_id = t3.class_id
WHERE t3.name = '$(Asset_Class)'
  ) AS s
PIVOT
(
  min(output_value)
  FOR char_name IN (@Value(Attributes))
) AS pvt;

It returns all the right attributes in the Data Inspector, but I want to output this to another database, and I can't tell the writer what attributes it needs to create.  If I try populate from SQL Query I get "Query failed, possibly due to a malformed statement." - presumably because it is seeing the @Value(Attributes) which doesn't exist yet...

Have tried using AttributeExploder (did not go well), PythonCaller to expose the attributes (did nothing as again only executes at run-time and bashing my head against the screen (worked as well as AttributeExploder), so any ideas would be welcomed!

Oh P.S. SQL Server 2013, FME 2017.1.1

You can use the AttributeExposer and type the names of the attributes to expose (or type the names in the attributes to expose in the SQLExecutor).

Both methods require you to know the names of the attributes.


Hi Erik,

Unfortunately there are some 500 asset classes and each one has different attributes, the idea was to automatically generate the attributes within the workspace :(


Are you doing some processing inside FME on those features, or are you just sending them to the writer? If you're just writing them out, look into using a dynamic schema writer, that way you don't need to expose all those attributes:

https://knowledge.safe.com/articles/1051/index.html

https://knowledge.safe.com/articles/43899/dynamic-workflows-advanced-example-modifying-the-s.html

As a starting point, insert a SchemaSetter (from the FME Hub) before your dynamic writer. There are also several other posts here dealing with this exact issue.


Hi Erik,

Unfortunately there are some 500 asset classes and each one has different attributes, the idea was to automatically generate the attributes within the workspace :(

hi,

 

were you able to solve this problem?


Reply