I am wondering if anyone have any suggestions to use FME to process live data streams that are funneled into various text files, server log files being one of the examples. Thanks!
Solved
How to read live data streams in text files
+10- Contributor
- 79 replies
Best answer by fmelizard
Hi @bo,
The Directory Watch has a min poll of 1 minute, whereas a schedule can run every second if needed. Another option is to use a looping custom transformer to re-read the file over and over again in an endless cycle (it would require an engine (or workspace) to run constantly though . I put one together which you can test out.
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+18- Safer
- 1542 replies
Hi @bo,
You can use WebSockets to capture these messages/data streams. Here are some resources to get you started:
- What is Message Streaming (from the Server Authoring manual)
- Exercise: Handling Emergency Phone Call Streams (from the Server Authoring manual)
- Streaming features from PostGIS to FME using WebSockets
- Weather Updates and FME Server (Tutorial)
- High Capacity Message Streaming
- Live Spatial Dashboard
Hopefully that helps, let us know if you need anymore information.
-Liz
+10- Contributor
- 79 replies
Hi @bo,
You can use WebSockets to capture these messages/data streams. Here are some resources to get you started:
- What is Message Streaming (from the Server Authoring manual)
- Exercise: Handling Emergency Phone Call Streams (from the Server Authoring manual)
- Streaming features from PostGIS to FME using WebSockets
- Weather Updates and FME Server (Tutorial)
- High Capacity Message Streaming
- Live Spatial Dashboard
Hopefully that helps, let us know if you need anymore information.
-Liz
+18- Safer
- 1542 replies
Thanks Liz. In this particular case, we have a log file that is constantly being appended by other processes. And we need to use the log file as the data source. Then the question becomes how can we open the log file for read as a stream.
If you have access to FME Server then you can create a schedule to run your workspace to grab the appended data. You can set up the schedule to run until cancelled so your output dataset will be continually updating from the log file.
Let me know if I'm not on the right track of what you are asking.
+10- Contributor
- 79 replies
Hi @bo,
You can use WebSockets to capture these messages/data streams. Here are some resources to get you started:
- What is Message Streaming (from the Server Authoring manual)
- Exercise: Handling Emergency Phone Call Streams (from the Server Authoring manual)
- Streaming features from PostGIS to FME using WebSockets
- Weather Updates and FME Server (Tutorial)
- High Capacity Message Streaming
- Live Spatial Dashboard
Hopefully that helps, let us know if you need anymore information.
-Liz
+18- Safer
- 1542 replies
Hi @lizsanderson, this process could work even though it is not "real-time" stream. Wonder if we could efficiently grab only the "newly-appended" data since last read?
This article Directory Watch Publisher with Idle Time Delay (Advanced) (2017) updates your output file by only grabbing the newly appended data. It works on a Modify trigger and the file can be locally or within FME Server. It writes to Google Fusion Tables, but ultimatly it can be written to format.
I'm going to pass this question on to one of our experts and see if they have a better solution for you.
+2Hi,
If you read in the latest file, and had a copy of the log file in it's previous state (last time you read it) you could use a duplicatefilter transformer to only work with the new, unique rows/lines in the log file. You'd then have to schedule the workspace to run really frequently, so it wouldn't quite be live streaming but fairly close.,
Hi,
If you on a frequent schedule read in the text file, but also kept a record of the previous state of the text file (reading them in with a feature for every line) you could use a DuplicateFilter to only process the new lines added. You'd have to run the job via a schedule very frequently, I don't know of a way that you could live stream a text file.
+10- Contributor
- 79 replies
Hi,
If you read in the latest file, and had a copy of the log file in it's previous state (last time you read it) you could use a duplicatefilter transformer to only work with the new, unique rows/lines in the log file. You'd then have to schedule the workspace to run really frequently, so it wouldn't quite be live streaming but fairly close.,
Hi,
If you on a frequent schedule read in the text file, but also kept a record of the previous state of the text file (reading them in with a feature for every line) you could use a DuplicateFilter to only process the new lines added. You'd have to run the job via a schedule very frequently, I don't know of a way that you could live stream a text file.
+10- Contributor
- 79 replies
Hi @bo,
You can use WebSockets to capture these messages/data streams. Here are some resources to get you started:
- What is Message Streaming (from the Server Authoring manual)
- Exercise: Handling Emergency Phone Call Streams (from the Server Authoring manual)
- Streaming features from PostGIS to FME using WebSockets
- Weather Updates and FME Server (Tutorial)
- High Capacity Message Streaming
- Live Spatial Dashboard
Hopefully that helps, let us know if you need anymore information.
-Liz
+20- Safer
- 3719 replies
- Best Answer
Hi @bo,
The Directory Watch has a min poll of 1 minute, whereas a schedule can run every second if needed. Another option is to use a looping custom transformer to re-read the file over and over again in an endless cycle (it would require an engine (or workspace) to run constantly though . I put one together which you can test out.
+10- Contributor
- 79 replies
Hi @bo,
The Directory Watch has a min poll of 1 minute, whereas a schedule can run every second if needed. Another option is to use a looping custom transformer to re-read the file over and over again in an endless cycle (it would require an engine (or workspace) to run constantly though . I put one together which you can test out.