Question

Dynamic string concatanation in FME DESKTOP 2013

  • 20 September 2013
  • 10 replies
  • 7 views
S
Badge
I an reading a feature table having 3 string columns say a,b,c and column X from another feature. Columns a,b,c holds string value(nullable). X has value like "Concat(Concat(a,b),c)". I need to derive final string without running sqlexecutor.

 

 

Please let me know how to solve this.

 

 

Thanks.

 

Subrat

10 replies

fmelizard
Rank
Userlevel 4
Badge +13
Hi Subrat,

 

Have you tried using the stringconcatenator ? 
S
Badge
Hi Itay,

 

In StringConcatenator, i have to hard code attribute names and their sequence. but here, X attribute may contain

 

Concat(Concat(a,b),c)

 

or

 

Concat(Concat(b,a),c)

 

or

 

Concat(b,c)

 

or

 

b

 

 

I have to evaluate this expression and write it.

 

I tried with inline querier but unable to use feature attributes.

 

Please let me know how to solve this.

 

 

Thanks.

 

takashi
Rank
Userlevel 2
Badge +17
Hi Subrat,

 

 

You can use the TestFilter to branch the feature flow into four flows according to X value, and then use four StringConcatenators to do different concatenating operations.

 

 

Alternatively, if you are using FME 2013 SP1 or later, you can also use "Conditional Mapping" functionality of the AttributeCreator, like this.

 

Select "Set To Conditional Value..." for setting Value parameter.

 

 

 

  I like this functionality very much.

 

See also this article: FME 2013-SP1: Conditional Processing in FME http://evangelism.safe.com/fmeevangelist113/

 

 

Takashi
fmelizard
Rank
Userlevel 4
Badge +13
Yes conditional mapping has my preference too
takashi
Rank
Userlevel 2
Badge +17
This is an experimental solution. A PythonCaller with this script might work. -----

 

import fmeobjects, re   def processFeature(feature):     x = str(feature.getAttribute('X'))     s = ''     for name in re.sub('.*\\(|\\)|\\s', '', x).split(','):         v = feature.getAttribute(name)         if v: s += str(v)     feature.setAttribute('concatenated', s)

 

-----

 

S
Badge
Hi Takashi,

 

  I am using FME(R) 2013 (20130207 - Build 13264 - WIN32) version of FME desktop,but i am not able to open screens shown by you for AttributeCreator.

 

 

Please let me know whether i am using a wrong version else how can i open "Conditional Mapping" functionality of the AttributeCreator.

 

 

Actually i have around 8-10 columns and their combination can be one or more,also  in any order. So python may suit my requirement. 

 

My feature has 2 million records. Will usage of python hamper my execution timings??

 

Thanks.

 

 

 

takashi
Rank
Userlevel 2
Badge +17
Hi,

 

 

The minimum requirement for "Conditional Mapping" is FME 2013 SP1 (released in late of March), I don't think your FME version supports it unfortunately. I recommend you to upgrade your FME to the latest version if possible.

 

FME Downloads

 

http://www.safe.com/support/support-resources/fme-downloads/   > My feature has 2 million records. Will usage of python hamper my execution timings?

 

Depending on the environment. If the performance of execution may be too inefficient, I would also consider to use two SQLExecutors in a series. Naturally, this approach can be applicable only if the source dataset is a database.   1st SQLExecutor: select distinct X from table_name   2nd SQLExecutor: select id, foo, bar, @Value(X) as concatenated from table_name where X = '@Value(X)'

 

Takashi
S
Badge
Hi, I am currently using Sqlexecuter approach. But it is taking much time. Hence please let me know any work around. Thanks.
takashi
Rank
Userlevel 2
Badge +17
Hi,

 

 

I currently have no other ideas. In my experiences, querying with SQL statement seems to achieve the best performance in many cases. But I have never used SQL statement containing "concat" function in a workspace, the Python approach might be better in this case. The exact answer can not be known until you measure time actually.   About general issues on FME performance, see Dave's posts and related links in this Q&A: Dynamic Schema and Memory issues https://safe.secure.force.com/AnswersQuestionDetail?id=906a0000000coggAAA

 

 

Takashi
david_r
Rank
Userlevel 4
Hi,

 

 

the SQLExecutor is bound to be slow for thousands / millions of features as it makes a separate call to the database for each feature.

 

 

You could use the dynamical nature of Python to your advantage here, though. It should be much faster. Example:

 

 

import fmeobjects   def Concat(part1, part2):     return str(part1) + str(part2)   def FeatureProcessor(feature):     text = feature.getAttribute("text")     a = feature.getAttribute("a")     b = feature.getAttribute("b")     c = feature.getAttribute("c")     concatenated = eval(text)     feature.setAttribute("result", concatenated)

 

 

Given a feature with the following attributes:

 

 

text: "Concat(Concat(a,b),c)"

 

a: "aaa"

 

b: "bbb"

 

c: "ccc"

 

 

The above script will create the attribute

 

 

result: "aaabbbccc"

 

 

David

 

 

Reply


Community Stats

28,670
Posts
109,366
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.