Question

Text line aggregation

10 years ago
Oct 28, 2016 11:39 AM UTC
7 replies
421 views

+21

davideagle
Contributor

Hi All,

I have a text file with a variable quantity of lines, separated by an empty line. The lines adjacent to one another need to be grouped together and then concatenated. So for example:

Test1

Test2

Test3

---

testy1

testy2

---

test

---

(where the hyphens are the blank lines, I can't format this page with return lines it seems)

Would need to become:

Test1 Test2 Test3

testy1 testy2

test

I've tried to do this a couple of ways so far, and variables seemed to be the right choice initially but now I'm not so sure. As I also have a format attribute on each line that contains the text_line_length, I was trying to use this as the delimiter when it was = to 0. It's interesting because the logic is a bit like using the PointConnector to create line geometries and what I need is to use the text_line_length as the 'connection break attribute'. Anyway, that's a digression. Does anyone have any simple tricks I could deploy to do this?

Thanks, Dave

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

+22

revesz
Supporter
10 years ago
Oct 28, 2016 12:06 PM UTC

Hi Dave

What I would try:

Read the file as text line.
Trim the lines.
Get rid of empty lines: simply by Attribute has a value / Attribute Is Empty String.
StringSearcher: extract the digits from the end using the RegExp \\d+$ (\\d: digit, +: any number - of digits, $: at the end)
Copy the original string to a new attrib
With StringReplacer (or other way) replace the extracted digits from the copied attribute
(Sort if you need to)
Aggregator: group by this new attribute, concatenate the original string attrib.

Is this what you mean?

Cheers,

András

Upvote

+21

davideagle
Author
Contributor
10 years ago
Oct 28, 2016 12:07 PM UTC

Ok, I persisted with Variables and sorted it out.

Test for a blank line, send this to the counter and then set a variable to the count value. On the failed port of the Tester, use a VariableRetriever to get the count value and glue it onto the features that pass through. Everytime a row with nothing on it is seen, the count ID increments. Then just use the Aggregator to group on the count ID and do the concatenation.

Thanks for listening, I helps to talk it through :0)

Upvote

+63

redgeographics
VIP
10 years ago
Oct 28, 2016 12:19 PM UTC

I had a different approach:

Replace empty lines with a character that won't occur in your dataset, then aggregate them all and specify that they should be concatenated, split on that character, explode the list, cleanup a bit et voila! Hope this helps.

concat-text.fmwt

FME rocks! \m/

Upvote

+54

ebygomm
Evangelist
10 years ago
Oct 28, 2016 12:30 PM UTC

I'd assign a value to the blank lines and then use the adjacent feature support in the attribute creator to increment a group number after every blank line, then aggregate/concatenate based on this value

Upvote

takashi
10 years ago
Oct 28, 2016 1:16 PM UTC

Hi @1spatialdave, assuming that a newline is represented by two special characters - [CR] (\\r) and [LF] (\\n), this might also be a possible way.

Text File reader: Read whole file at once.
StringReplacer (1): Replace two consecutive newlines i.e. [CR][LF][CR][LF] with a specific special character e.g. [BEL] (\\a).
StringReplacer (2): Replace every remaining newline with a white space.
StringReplacer (3): Replace the special character [BEL] with a single newline.

Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?