Question

FeatureHolder waiting for WRITE

  • 30 April 2013
  • 6 replies
  • 32 views
sigtill
Rank
Badge +21
Have anyone come up with a solution to this problem:

 

 

http://fmepedia.safe.com/AnswersQuestionDetail?id=906a0000000cg4TAAQ

 

 

Except using "Daves method"?

 

 

It would be very handy to have an option on the FeatureHolder like this:

 

- Wait for Writers (YES/NO).

 

 

This would be very handy on FMEServer to be able to WRITE some features, then do some more Custom Transformers and then WRITE some other features to another WRITER. The alternative way is to create separate workspaces for each and use either a WorkspaceRunner or FMEServerWorkspaceRunner, however this creates the need for saving and loading features inbetween the Workspaces.

 

 

Another option would be for the FMEServerWorkspaceRunner to also work on FMEServer when selecting "Wait for Result" to not worry about the line:

 

 

"When you run a workspace on FME Server that contains this transformer, and the workspace sends a job to the same FME Server that is running the original workspace: depending on the availability of FME Engines, the original workspace may never finish."

 

 

 

The real SigTill

6 replies

david_r
Rank
Userlevel 4
Hi Sigbjørn,

 

 

here is an untested suggestion off the top of my head... :-)

 

 

Between the reader and a writer, create a custom transformer with one input port (i1) and two output ports (o1, o2).

 

 

On (i1) there is a StatisticsCalculator that takes the total count attribute "_count". All features entering (i1) goes directly to (o1), where they pass on to the writer.

 

 

After the StatisticsCalculator, there is a SQLExecutor that does a "select count(*) from myWriterTable" to check the record count in the output table. When the recordcount = _count, a dummy feature is sent to (o2) to signal to trigger the rest of the workspace (like a Creator would). If recordcount < _count, it loops back to the SQLExecutor via a ModuloCounter to avoid swamping the database.

 

 

It is very from elegant and makes a whole lot of assumptions, but it miiiiiight perhaps work?

 

 

Personally, though, I'd much rather use a WorkspaceRunner than maintaining such a byzantine solution ;-)

 

 

David
david_r
Rank
Userlevel 4
And here is another untested suggestion:

 

 

Replace your writer with a PythonCaller and write all your features using the FMEUniversalWriter class.

 

 

Using the class handler of the PythonCaller, make sure to call pyoutput() only once when all the features have been written. You could use a StatisticsCalculator before the PythonCaller to let your Python code know how many features to expect.

 

 

That should be quite doable for somebody half decent with Python :-)

 

 

David
sigtill
Rank
Badge +21
Thanks for all the good tips David R.!

 

 

The reason not to use a WorkspaceRunner or FMEServerWorkspaceRunner when it is ported to FME Server is the file-handling. The user needs to be able to download the first file, and the second file (badly explained - however..) - ie: The user wants one download link with outputfiles from "workspace1" and also files from "workspace2" - and both of these reads from a logfile created with both workspace1 and 2...

 

 

So to your suggeestions:

 

1. The "CustomTransformer"-solution: If I understand you this works if you have 2 WRITERS, and you first want to update the database, and after that write to some other format (ie Shape etc.). It actually might be worth a shot! The issue is then if my first WRITER is not a Database (but rather a SHAPE) - I guess it will not work the same way. 

 

 

2. Coding? :) As The Safe-guys says: "If there is a Transformer for that, dont use python:)". 

 

 

Also I would this solution to both work on Desktop and Client with no change for the user during run-time.

 

 

Would be very interested in an Advanced FME Server Webinar. Perhaps with examples from users all around the world demo-ing some of their "tricks". I know there is a lot of fmepedia and articles out there, however it would be nice to see some other tricks also.
The real SigTill
david_r
Rank
Userlevel 4
Hi Sigbjørn,

 

 

regarding 1, I guess this would be less than ideal for file-based databases with no SQL support.

 

 

Regarding 2, I think that this is extactly the kind of scenario where coding a solution would be a good idea, but of course it would depend on how you feel about coding in general :-)

 

 

Anyway, I suspect this is an excellent case for a proof-of-concept before you commit to either solution...

 

 

David
sigtill
Rank
Badge +21
So basically I dont want FME to be an ETL-tool, however I want it to be an ETLTLTLTLTLTLTLTLTL tool :)
The real SigTill
G
Hey David,

 

 

your first idea sounds great!

 

Do you have a workbench with this idea for example?

 

 

Georg

 

 

Reply


Community Stats

28,727
Posts
109,522
Replies
36,566
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.