Skip to main content
Released

Add Output Port to FeatureWriter

Related products:FME Form

Untitled

The current FeatureWriter only have a summary port, so If I want to continue my translation after the FeatureWriter, I will need a featureholder to wait for the summary port. I propose to add an output port to FeatureWriter with all the features, so I can continue to process my data after the data is safely written with FeatureWriter. 
This post is closed to further activity.
It may be a question with a best answer, an implemented idea, or just a post needing no comment.
If you have a follow-up or related question, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

27 replies

fmelizard
Safer
Forum|alt.badge.img+19
  • Safer
  • February 14, 2016

So to be clear, you want an output port for every input port, and the features will only be emitted AFTER the transformer has shut down and the writer has completed doing the writing. I think we'd set this up so that we'd always output the Summary feature first, and then any of the input features that were desired.

We'd probably need to think carefully about how to present this because there is a significant expense in storing the features up to be emitted at the end. So we'd not want this to happen unless there was a specific user configuration to do it.


jdh
Contributor
Forum|alt.badge.img+28
  • Contributor
  • February 15, 2016

I have a few scenarios where this behaviour is an asset. They are currently being handled with multiple workspaces chained together with python shutdown scripts.


erik_jan
Contributor
Forum|alt.badge.img+18
  • Contributor
  • February 15, 2016

I can see scenario's for this.

But I would like to see this only if needed (so controlled by a setting "output feature types", if possible a checkbox for selecting input feature type for output ) as it will consume a lot of resources.


  • Author
  • February 16, 2016

Yes, that is what I intend to achieve, a way to get rid of pyton and chain up workspace with safe data storage at each step.So the data from step one will be written even step 2 fail to run, it could be really helpfull when debugging huge amount of data.


fmelizard
Safer
Forum|alt.badge.img+19
  • Safer
  • February 17, 2016

Agree. And I like the idea of selecting the feature types to output as well. By default, none would be. But you could then select the ones you wanted, and they'd have output ports added for them.

You are correct, it is going to be a resource intensive thing to pile them up, but I can also see scenarios where it will be useful.


Forum|alt.badge.img
  • February 17, 2016

Ya, something like that would be very useful. If nothing else could the port create maybe a list instead and then could use a list exploder to get the feature. Something then can choose the attributes in the exploder Also with 64 bit FME and computer carrying 32+ gigs of ram, does it really matter to much on how much resources are used. I can see in the 32bit version but the 64 bit is all hard drive space for temp files and most computers/desktops carry the needed ram

.


Forum|alt.badge.img
  • February 17, 2016

Also I wanted to say, that is something that could be very useful. I mean to have a writer in the middle of the workspace without having to use a workspace runner would be nice


geosander
Forum|alt.badge.img+7
  • February 21, 2016

@daleatsafe: Since FME does not consume extra memory if the user will not connect anything to the Output port, a word of advice in the transformer Help would then suffice, I think. When the user decides to use the Output port, performance might degrade.

 

Actually, your remark made me wonder: would memory consumption of the FeatureWriter be any different from that of the StatisticsCalculator for instance, that also has a Summary and an Output port?

geosander
Forum|alt.badge.img+7
  • February 21, 2016

At the expense of adding at least 1 extra transformer, you could also have a FeatureMerger after the FeatureWriter to achieve this - which is what I'm doing now. This is similar to your FeatureHolder approach, but the amount of output features stay the same and you will have the summary on all your input features. Be sure to connect the Summary to the Supplier port and the incoming features (before they are written) to the Requestor port. Specify the same constant (e.g. 1) for the Join On attributes if you only write 1 dataset.

 

If you write to multiple datasets, it becomes a little more tedious:

'


fmelizard
Safer
Forum|alt.badge.img+19
  • Safer
  • February 21, 2016

You are correct that it would consume the same as the Stats Calculator does when the output of originals is done. Really it is that we have to pile up all the input data to let loose at the end. If the data volumes are moderate (measured in 10s of thousands or less) this is not interesting, but when they go into the millions (and there are those of you that do this), then even on 64 bit machines there is a tax to be paid.

The other complication is that very often there are several input ports coming into a Feature Writer -- so by default I'd prefer not to cloud the interface with an output port for each input port. So that is why I think we'd make this scenario an "opt in" in the UI, and if the toggle was set, then provide all the output ports. Also because if you don't care about getting those input features until after the writing is done (if it is okay to get them any time), then you can simply just send a copy past the FeatureWriter using normal FME workflows. So the FeatureWriter "hold" mode would only be for situations where you absolutely don't want data to flow until you know for sure that step of writing is complete. These do exist for sure.


geosander
Forum|alt.badge.img+7
  • February 21, 2016

Thanks for clearing that up!


scyphers
Participant
Forum|alt.badge.img+5
  • Participant
  • March 5, 2016

I think I might like also to see a featurewriter cache available in the DataDownload selections on Server. I think only exit writers turn up here. But the 'selected feature types' as Erik describes could be abstracted to the 'include' list. I think that would be interesting.

'


  • May 23, 2016

For expample I have A list of files in SharePoint. They all has the status "ToCheck".

I am staring the FME Schema that would search for all files with status "ToCheck" and separatly start another FME Scema for files check.

The time of checking of 1 file is about 2-3 seconds. SO If I would have 60 files all checks will take about 2 or 3 minutes. It's a lot for time for another user to start the first schema once more.

To solve that I need to update Status field to "Checking" for every file before I will start the checking.

Previously I was need to write one more schema for that. Now I can use the Feature Writer. But the problem is that i need my features to pass thru feature writer (like roger7467 suggest).

If you do not want to add such functionality to common FeatureWriter you can easyly add FeatureWriterWithOutput for such tasks?


revesz
Contributor
Forum|alt.badge.img+15
  • Contributor
  • August 30, 2016

Apart from further processing, another reason to pass features through the FeatureWriter is checking if the data was written successfully and if not, why.

It is also quite usual when I'd like to do further processing depending on the successful or failed write.

Eg.: I really want to know which features were rejected and why by the database even I sent the features the through a series of GeometryValidators .

Another scenario is that I want to process csv data exported from a 3rd party system and move them to the archive. I want to move to the archive when uploading all lines to the database is successful. If not I want to advise someone which file and which line dropped the error and preferably why.

Now I need to analyse the log for rejected features and causes in both cases.

There are formats where data can be written at one, eg. file based ones. But for databases or File Copy features could go on like in case of the Cumulative port in StatisticsCalculator.


Forum|alt.badge.img+2
  • August 31, 2016
@revesz we're looking at adding a <rejected> port to FeatureWriter to handle features that were rejected by the writer. Would that meet this use case?

 


david_r
Celebrity
  • September 1, 2016

Lot's of great feedback and discussion here. I agree wholeheartedly that the FeatureWriter ought to have an (optional) output port for the individual features, not just the summary. That would really make the FeatureWriter a game changer compared to the traditional writers, in my opionion.

The features output here should be features that have been written to the database, so they shouldn't be output before their transaction has been committed.

Why? Because it would allow you to perform very powerful post processing operations on the newly written features, e.g. updating existing master/detail relationships or retrieveing the primary key given by the database (OBJECTID is a prime example, but also sequences) to create new relationships. If features are continually output per commited transaction it would also reduce the memory overhead as the transformer would only be blocking for a number of features equal to the transaction size.


revesz
Contributor
Forum|alt.badge.img+15
  • Contributor
  • September 2, 2016
The <rejected> port alone would be a dream! (Especially with error messages...)

 

If this information is on the canvas, the rest is just drawing :)

 


fmelizard
Safer
Forum|alt.badge.img+19
  • Safer
  • February 7, 2017
Safe PR#69303

 

 


ebygomm
Influencer
Forum|alt.badge.img+39
  • Influencer
  • February 7, 2017
Agree that holding the output until all features had been successfully written would be very useful. Various ways and means to ensure that all features are written before continuing but they tend to be a bit convoluted.

 

 


All I am looking for is that summary be a continuous stream of Summary features. Something like a running log for each file in the DataSet that was written. Attrributes can be things like any file write errors, the current count of files written, etc. I am using the FeaureWriter to put out something like 100K files, and I want to send out every Module(50) files that it writes an HTTPPost message telling me how the writing is going, and what progress it is making.


Oh yes, I just tried to look at the summary port of the FeatureWriter saving a KMZ file. I have about 1000 files going in, and I would like to report the progress as this occurs. Unfortunately, only one feature comes out from the summary port.


rylanatsafe
Safer
Forum|alt.badge.img+13
  • Safer
  • December 20, 2017

@roger7467 et al... The FeatureWriter now has an Output Port! Please check out the current FME 2018.0 beta (Build 18195 or Newer) and let us know what you think! safe.com/beta

Edit: There are a couple important notes regarding this feature which I must stress is currently in beta and not finalized. There is not yet any Rejected Output Port and it currently will only output the Features that are passed through it.


david_r
Celebrity
  • December 21, 2017
Nice! Christmas came early this year :-)

rylanatsafe
Safer
Forum|alt.badge.img+13
  • Safer
  • December 21, 2017
Hi @david_r - Just wanted to ping you that I updated my comment! If you are planning to test out the FeatureWriter in 2018.0, please review the Edit so you are aware of what has not been implemented yet.

 


david_r
Celebrity
  • December 21, 2017
Thanks for the heads up. Will both these issues be adressed before the final release of FME 2018?

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings