Archived

Word Frequency Transformer

Forum|Forum|8 years ago
September 28, 2017
8 replies
70 views

+22

fmelizard
Safer

I was just thinking it would be great if I could determine the word frequency of some text held in an attribute.

I bet there is a python function to do that or a number of different transformers could be put together to figure it out. Alas, I'm a simple lizard with limited time and knowledge...

Maybe someone can whip something up for me? Oh! Oh! and maybe add it to the FME Hub for all to enjoy? That would be great!

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

david_r
Forum|Forum|8 years ago
September 28, 2017

Great idea. I think the challenge should include a sample text with a given result to aim for. The text should include stuff like international characters, apostrophes, hyphens etc. to make sure non-trivial edge cases are treated correctly.

Possible starting point: http://www.nltk.org/

Upvote

+21

denniswilhelm
Contributor
Forum|Forum|8 years ago
September 28, 2017

This would be really easy to implement with just two standard FME Transformers.

Use an AttributeSplitter to get every word in your text as a list. After that you can use a ListHistogrammer to get the actual statistic.

Upvote

+26

fgiron
Enthusiast
Forum|Forum|8 years ago
September 28, 2017

Well, you would need a couple of extra transformers to exclude some not very wordy characters before calculating the word frequency (commas, dots, brackets, inverted commas, etc). Putting back together words split in two lines would also be nice.

Anyway, what is a word? https://www.sussex.ac.uk/webteam/gateway/file.php?name=essay---what-is-a-word.pdf&site=1

Upvote

+15

ulfme
Contributor
Forum|Forum|8 years ago
September 28, 2017

I vote for this one and at the same time this WordCloud idea :

https://knowledge.safe.com/idea/51622/wordcloud-transformer.html

I think they are related in many ways. To create WorldClouds we would really need wordcounting. So a nice and fast WordCounter where exceptions can be included in a nice way for common, not important words would rock.

Upvote

+14

rylanatsafe
Safer
Forum|Forum|8 years ago
September 28, 2017

FME Hub transformer UniqueValueLogger may be of assistance here too... Lots of great idea on how to implement!

Upvote

+22

chriswilson
Enthusiast
Forum|Forum|8 years ago
September 29, 2017

...and/or something to read a text file/Word document for the number of times a word or phrase (or sentence) repeats. Having put together a lengthy python script for this in the past it would be great!