Skip to main content
Solved

Using a calculated Private parameter as path to a Data Reader

P
Forum|alt.badge.img

I'm having some difficulty trying to use a private parameter as a path for a Data Reader. The difficulty stems from the fact that the value in the parameter is not literal but is calculated at run time (based on date).

The parameter name is SourcePath and its value is a concatenation of a literal string and two other private parameters, which are based on a date. At runtime the value in SourcePath would look something like this:

P:\\GIS Development\\Vicmap_Downloads\\2017\\20170606\\

The workspace itself is very simple, consisting of a Directory and File Pathnames Reader, and a corresponding File Copy Writer. The reader obtains a listing of files in the directory, as specified in the SourcePath parameter. It then supplies this list of files to the File Copy Writer to perform the copy/paste.

If I set the Data Reader folder path to a literal string then the workspace runs fine. However this is not workable as the path must be determined at runtime because it's based on current date.

So the question remains: how do I specify a path that is obtained at runtime to a Data Reader?

Best answer by mark_f

What about using. Python Scripted Parameter for the two date values?

View original
Did this help you find an answer to your question?

12 replies

M
Forum|alt.badge.img+2
  • mark_f
  • Best Answer
  • June 7, 2017

What about using. Python Scripted Parameter for the two date values?

takashi
Influencer
  • takashiInfluencer
  • June 7, 2017

Agree. FME Date/Time functions within a user parameter value seems not to be evaluated while FME is configuring reader parameters. You can use a scripted parameter instead. e.g.

# Scripted Python Parameter Example
import datetime
t = datetime.date.today() 
return 'P:/GIS Development/Vicmap_Downloads/%04d/%04d%02d%02d' % (t.year, t.year, t.month, t.day)
P
Forum|alt.badge.img
  • peterz
  • Author
  • June 7, 2017
takashi wrote:

Agree. FME Date/Time functions within a user parameter value seems not to be evaluated while FME is configuring reader parameters. You can use a scripted parameter instead. e.g.

# Scripted Python Parameter Example
import datetime
t = datetime.date.today() 
return 'P:/GIS Development/Vicmap_Downloads/%04d/%04d%02d%02d' % (t.year, t.year, t.month, t.day)
Thanks for the ideas and the example code! I have not yet tried Python in FME so will hit some examples/tutorials online first and then give this a try, but sounds like the way to go!

 

 

BTW, do Python Scripted Parameters continue to work in FME Server?

 

 

P
Forum|alt.badge.img
  • peterz
  • Author
  • June 7, 2017
mark_f wrote:

What about using. Python Scripted Parameter for the two date values?

Thanks Mark!

 

 

takashi
Influencer
  • takashiInfluencer
  • June 7, 2017
peterz wrote:
Thanks for the ideas and the example code! I have not yet tried Python in FME so will hit some examples/tutorials online first and then give this a try, but sounds like the way to go!

 

 

BTW, do Python Scripted Parameters continue to work in FME Server?

 

 

Yes, I believe that scripted parameters should work in FME Server as well.

 

 

david_r
Celebrity
  • david_rCelebrity
  • June 7, 2017
peterz wrote:
Thanks for the ideas and the example code! I have not yet tried Python in FME so will hit some examples/tutorials online first and then give this a try, but sounds like the way to go!

 

 

BTW, do Python Scripted Parameters continue to work in FME Server?

 

 

Can confirm that Python scripted parameters work in FME Server, yes. I use them a lot.
P
Forum|alt.badge.img
  • peterz
  • Author
  • June 8, 2017
takashi wrote:

Agree. FME Date/Time functions within a user parameter value seems not to be evaluated while FME is configuring reader parameters. You can use a scripted parameter instead. e.g.

# Scripted Python Parameter Example
import datetime
t = datetime.date.today() 
return 'P:/GIS Development/Vicmap_Downloads/%04d/%04d%02d%02d' % (t.year, t.year, t.month, t.day)
The scripted python parameter worked like a charm! Thanks everyone, and thanks Takashi for the sample code!

 

 

ngoorman
Contributor
Forum|alt.badge.img+3
  • ngoormanContributor
  • Contributor
  • June 8, 2017

For a solution without Python, you could also keep just the static part of the path in the private parameter, then use a Timestamper and an AttributeCreator to create the full path in the workspace. Then feed that to a FeatureReader to actually read your data, instead of a normal reader.

P
Forum|alt.badge.img
  • peterz
  • Author
  • June 8, 2017
ngoorman wrote:

For a solution without Python, you could also keep just the static part of the path in the private parameter, then use a Timestamper and an AttributeCreator to create the full path in the workspace. Then feed that to a FeatureReader to actually read your data, instead of a normal reader.

Thanks ngoorman, I'll keep that one in the back of my mind for future reference.

 

 

In my case, I'm simply copying dozens of Mapinfo layers, which could be in nested subdirectories from the base path. So the Directory and File Pathnames Reader and the File Copy Writer is much more suited to that as I can specify a search pattern. Also by doing the copy at a file systems level, it runs about 10 times faster than reading each record at a time with a conventional Mapinfo reader and then writing one record at a time with a MapInfo writer.

 

 

takashi
Influencer
  • takashiInfluencer
  • June 9, 2017
ngoorman wrote:

For a solution without Python, you could also keep just the static part of the path in the private parameter, then use a Timestamper and an AttributeCreator to create the full path in the workspace. Then feed that to a FeatureReader to actually read your data, instead of a normal reader.

Hi @peterz, just for your information, the Directory and File Pathnames (PATH) reader can also be used in conjunction with the FeatureReader and you can specify the reader parameters including Path Filter as well. For example, the FeatureReader with this setting reads all *.dat, *.id, *.map, *.tab file paths in the $(SourcePath) directory and its subdirectories.

 

 

P
Forum|alt.badge.img
  • peterz
  • Author
  • June 9, 2017
ngoorman wrote:

For a solution without Python, you could also keep just the static part of the path in the private parameter, then use a Timestamper and an AttributeCreator to create the full path in the workspace. Then feed that to a FeatureReader to actually read your data, instead of a normal reader.

Thanks for the clarification Takashi, I see what you mean now.

 

 

So I was able to get this to work without any parameters and just using inline attribute creation to build the source path, then feed that to the FeatureReader, and the rest stayed the same as my original post.

 

 

 

Although it achieves the same result (albeit using 6 steps), I still prefer the simpler approach of using the Scripted Parameter (using just 3 steps). It was a good exercise to go through anyway to increase my knowledge nevertheless! :)
gio
Contributor
Forum|alt.badge.img+15
  • gioContributor
  • Contributor
  • June 9, 2017

@peterz

Or use a tcl scripted parameter:

[string map {" \\\\ " "\\\\"} [concat "P:\\\\GIS DEVELPENT\\\\Vicmap_Downloads" \\\\ [clock format [clock scan today] -format {%Y}] \\\\ [clock format [clock scan today] -format {%Y%m%d}]]]

Actually just

P:\\\\GIS DEVELPENT\\\\Vicmap_Downloads\\\\[clock format [clock scan today] -format {%Y}]\\\\[clock format [clock scan today] -format {%Y%m%d}]

should work, but fme2016 appears top process it incorrectly.

It works in a attributecreator (etc.) (string editor) with 2x @Evaluate

P:\\\\GIS DEVELPENT\\\\Vicmap_Downloads\\\\@Evaluate([clock format [clock scan today] -format {%Y}]) \\\\@Evaluate([clock format [clock scan today] -format {%Y%m%d}])

Reply


Helpful Members This Week

Community Stats

31,203
Posts
118,291
Replies
38,807
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.