Question

big data -> big headache

G
Badge
Afternoon All,

 

 

I was spouting how FME could deal with any format and size GIS data, so a friend decided to make me sweat a bit. He provided me with  a csv of over 3 billion gps points logged by vehicles to have a play with and laughed at me!

 

 

I have been scratching my head over this one....

 

 

My options seem to be

 

1) bring up a massive FME_CLOUD Fme server ENTERPRISE with 64 GIGS of ram and try and wrestle it into shape with various fme transformers. 

 

 

2)Try and utilise one of the online services to load and query the data for me. Amazon , Google cloud services. (havent succeeded here) 

 

 

3)Find a Hadoop cluster to throw it at

 

 

4) admit defeat and enjoy a beer, its sunday after all. 

 

 

Has anyone had any sucess using Google BIGQUERY for GIS data ?Any tutorials for this in FME, success stories ? i have tried to have a play with this and FME without luck.. It seems to only be able to use non spatail data. i have read ALL the GIS Big data articles on the web without finding how to proceed.

 

 

Has anyone tried to wrestle data of this size within fme and had sucess?

 

 

Any other suggestions for Big data. 

 

 

Thanks for your help

 

 

Steve

 

 

 

 

 

 

 

6 replies

fmelizard
Rank
Userlevel 4
Badge +13
Hi Steve,

 

I have had in the past processed large xyz csv files with FME desktop.

 

That was before the introduction of the xyz and PointCloud readers, so I would try them first to see if FME can handle the size. 
B
Badge
You can use Point Cloud XYZ reader to read your BIG data.
G
Badge
thanks for both your suggestions, once read into FME im scared what will happen when i try and run any transformers on it. Ie spatial join to roads...... i will have a try and see if the magic smoke appears out of my pc.  (250g SSD,64 bit windows and fme, 16gig ram and good processor.

 

 

Thanks steve
fmelizard
Rank
Userlevel 4
Badge +13
Hi Steve, Any smoke signs yet? :)

 

You could try to convert the point first to a more efficient format (FFS) and see if that helps.

 

Itay
G
Badge +3
sounds like a tiling strategy may be usefull when you want to do spatial operations on such a big pointcloud
G
Badge
Thanks all for your suggestions. I have been pulled into another project for a few days (this one pays). Once i have done this i will report back my findings with all your suggestions. 

 

 

Keep them coming. Thanks Steve

Reply


Community Stats

28,753
Posts
109,617
Replies
36,587
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.