Skip to main content
  • EN

Hi!

 

I am transforming data from different sources. The data travels at different speeds through their own workflows and end up at my Excel Writer at different points in time. This in itself is not the problem.

 

It’s because I have specified a Fanout function in the name of the file all the data/features are being written to that the problem appears. The Fanout contains hours and minutes, the name of the file begins with:

“@Timestamp(^Y).@Timestamp(^m).@Timestamp(^d)_kl.@Timestamp(^H).@Timestamp(^M)...“

 

Since the data gets to the Writer roughly a minute a part I end up with two written files with only part of the results in them. The two files get different names with only a digit’s difference. For example:

File 1: “2024.05.30_kl.12:55...”

File 2: “2024.05.30_kl.12:56...”

 

I can obviously solve the problem by removing the hour and minutes from the Fanout in the file’s name. But I would really prefer to keep it. 

 

I have been looking for something to pause and sync the data before putting it through to the writer. It might be important to know that it is not known exactly how many features will be coming through to the writer. I could calculate a “max” but I would never know if/when the max would be reached.

 

I think that using a breakingpoint would solve the problem, but scripts aren’t always ran with these enabled/activated and since it will mean having to inspect the data and run the last part, I would rather not use breakpoints.

 

It feels like there should be an easy solution that I just don’t know, and I can’t figure out what to search for. I hope someone has a quick and easy solution for this in their nifty metaphorical toolbox. 

 

Thanks for your time!

/Sofia

 

I would suggest another way: get the timestamp once (you can use a DateTimeStamper to write it to an attribute), and use that timestamp for the fanout and naming the files.

add a creator and join in the date time with a featuremerger that way the date time is only created once not over multiple seconds?

I think the other answers are probably what you need, but I do have a couple of other thoughts.

You could keep the hours/minutes in the fanout, but round them off to the nearest (for example) 1 hour, using the DateTimeRounder transformer. If you don’t mind how accurate the datetime is, that might be the simplest method. You could even round it to 15 minute intervals. You would have to choose that depending on how long your workspace takes to run in total.

Just make sure to round off to “nearest” (not “up” or “down”) else you might still get multiple files.

Another option might be the TimeWindower transformer. Usually that is for when you want to create multiple files at x intervals, but I think it could be used to create a single, time-based, result.

Reply


Community Stats

30,108
Posts
114,286
Replies
37,742
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.