Question

Updating 'reader as resource' for dynamic writing

  • 5 September 2017
  • 8 replies
  • 5 views
jneujens
Badge

Hi all,

I am using an excel file containing several tabs. Each tab corresponds with a feature type in my data. And each feature type/tab has its own specific set of attributes I would like to write to several formats.

 

 

Whenever changes are made to this excel file (changing names of attributes, adding attributes, adding tabs, etc.), it does not update automatically. Even if I set the parameter 'Scan Schema in Dynamic Workflow' to 'Yes'. Every time changes are made, I have to delete the workspace resource and add it as a new.

 

 

Anyone having similar problems? Or solutions to this problem?

Thanks!

8 replies

takashi
Rank
Userlevel 2
Badge +17

Hi @jneujens, I was able to reproduce the same symptom, but I don't know if it's a defect of FME. Hope someone from Safe clarifies this > @NatalieAtSafe

A possible workaround is to configure dynamic schema based on schema feature(s) read from the resource Excel file by a Schema (Any Format) reader. Note that the schema features have to arrive in the writer feature type before data features.

Alternatively, a different method described in this article can also be applied, if you can replace the resource with a newly created lookup table.

Dynamic Workflows: Destination Schema is Derived from a Lookup Table

Hope this helps.

jneujens
Badge

Hi @jneujens, I was able to reproduce the same symptom, but I don't know if it's a defect of FME. Hope someone from Safe clarifies this > @NatalieAtSafe

A possible workaround is to configure dynamic schema based on schema feature(s) read from the resource Excel file by a Schema (Any Format) reader. Note that the schema features have to arrive in the writer feature type before data features.

Alternatively, a different method described in this article can also be applied, if you can replace the resource with a newly created lookup table.

Dynamic Workflows: Destination Schema is Derived from a Lookup Table

Hope this helps.

Thanks for your help! Good to know you are facing similar issues.

 

If I find time, I might look into using lookup tables.

 

takashi
Rank
Userlevel 2
Badge +17

Hi @jneujens, I was able to reproduce the same symptom, but I don't know if it's a defect of FME. Hope someone from Safe clarifies this > @NatalieAtSafe

A possible workaround is to configure dynamic schema based on schema feature(s) read from the resource Excel file by a Schema (Any Format) reader. Note that the schema features have to arrive in the writer feature type before data features.

Alternatively, a different method described in this article can also be applied, if you can replace the resource with a newly created lookup table.

Dynamic Workflows: Destination Schema is Derived from a Lookup Table

Hope this helps.

I found that you can use the Schema (Any Format) reader as a workspace resource, instead of the Excel reader. Maybe this is the correct answer.

 

jneujens
Badge
I found that you can use the Schema (Any Format) reader as a workspace resource, instead of the Excel reader. Maybe this is the correct answer.

 

That's interesting... I will try this out!
jneujens
Badge

takashi Your solution solved the problem indeed! Thanks!

fmelizard
Rank
Userlevel 4
Badge +13

Hi @jneujens, I was able to reproduce the same symptom, but I don't know if it's a defect of FME. Hope someone from Safe clarifies this > @NatalieAtSafe

A possible workaround is to configure dynamic schema based on schema feature(s) read from the resource Excel file by a Schema (Any Format) reader. Note that the schema features have to arrive in the writer feature type before data features.

Alternatively, a different method described in this article can also be applied, if you can replace the resource with a newly created lookup table.

Dynamic Workflows: Destination Schema is Derived from a Lookup Table

Hope this helps.

Hi @jneujens and @takashi. It seems like when using a Reader as Resource, changes to attributes should be picked up on every read but if a new tab (I assume this means new sheet) is added that won't be picked up during schema discovery. Resource reader will only use the feature types (tabs/sheets) set in the `Feature Types to Read` parameter.

 

Also, Excel and CSV may be a special case where there is no schema when reading the source (or using it as a reader as resource option in dynamic writing). CSV only has one feature type name `CSV` or the file name (without the extension) depending on the option selected when reader is added.

 

This other question uses the Schema (any format) reader to get the schema of CSV files and pass that across to the destination CSV file. This approach may be suitable for you.

 

 

jneujens
Badge
Hi @jneujens and @takashi. It seems like when using a Reader as Resource, changes to attributes should be picked up on every read but if a new tab (I assume this means new sheet) is added that won't be picked up during schema discovery. Resource reader will only use the feature types (tabs/sheets) set in the `Feature Types to Read` parameter.

 

Also, Excel and CSV may be a special case where there is no schema when reading the source (or using it as a reader as resource option in dynamic writing). CSV only has one feature type name `CSV` or the file name (without the extension) depending on the option selected when reader is added.

 

This other question uses the Schema (any format) reader to get the schema of CSV files and pass that across to the destination CSV file. This approach may be suitable for you.

 

 

NatalieAtSafe ?? that is indeed what I was seeing. I could even add the new excel sheets as feature types in my 'reader as resource' but nothing was coming out.

 

I found a way by using the generic schema reader, but thanks for your answer! ;)

 

H

Hi @jneujens​ and @Takashi Iijima​,

 

So glad to find your posts & discussions here.

 

I'm facing the similar issue, the problem is that my input Excel is not directly output into another data format, but acts as the Supplier of a FeatureMerger transformer, merged with the other input data (IFC) and output to another data format (CityGML). It failed to dynamically read different Excel files (different for file name, sheet name, attribute name&values) if i just change the Excel Reader to Generic Reader (or Schema Reader) for this case. Do you have any ideas/suggestions for my issue? Thanks in advance!

07201I've done dynamic related settings, not sure if I did correctly:

  1. EXCEL Reader setting07202
  2. Writer setting07203

As I'm newbee to deal with EXCEL in FME, any detailed illustrations would be much appreciated.

Reply


Community Stats

28,723
Posts
109,511
Replies
36,559
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.