Question

Building Line from large volume streaming data

  • 30 December 2022
  • 1 reply
  • 0 views
boubcher
Rank
Badge +10

Hello There

we are consuming streaming data coming from the GPS location of a large number of assets, we are getting around 4 Million records a day all those records represent the location of the asset moving in reel time

we need to store this data in the database to get the path for each asset.

due to the size of the data, it takes a long time to query the database and retrieve the path of the assets during a period of time

is there anyway, instead of storing the Point we store line, and this will reduce the number of records

how could build a line path that will have the same alignment as the road

or any other solution that will solve this issue

Thanks

1 reply

M
Badge +2

@boubcher​ If you generate a line for each asset you have to decide:

  • how long the line should be
  • do you want FME to block the data flow, because creating a line implies holding all the point features until the line is complete.

I think I would suggest that you dump your points into the database as you are now. Then periodically query for a time period (previous 24 hours say) for each asset and then create the lines. Then purge the point table for that time period. In the meantime, the points will keep loading into the point table.

Reply


Community Stats

28,723
Posts
109,511
Replies
36,560
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.