Skip to main content
  • EN
Afternoon All,

 

 

I was spouting how FME could deal with any format and size GIS data, so a friend decided to make me sweat a bit. He provided me with  a csv of over 3 billion gps points logged by vehicles to have a play with and laughed at me!

 

 

I have been scratching my head over this one....

 

 

My options seem to be

 

1) bring up a massive FME_CLOUD Fme server ENTERPRISE with 64 GIGS of ram and try and wrestle it into shape with various fme transformers. 

 

 

2)Try and utilise one of the online services to load and query the data for me. Amazon , Google cloud services. (havent succeeded here) 

 

 

3)Find a Hadoop cluster to throw it at

 

 

4) admit defeat and enjoy a beer, its sunday after all. 

 

 

Has anyone had any sucess using Google BIGQUERY for GIS data ?Any tutorials for this in FME, success stories ? i have tried to have a play with this and FME without luck.. It seems to only be able to use non spatail data. i have read ALL the GIS Big data articles on the web without finding how to proceed.

 

 

Has anyone tried to wrestle data of this size within fme and had sucess?

 

 

Any other suggestions for Big data. 

 

 

Thanks for your help

 

 

Steve

 

 

 

 

 

 

 

Hi Steve,

 

I have had in the past processed large xyz csv files with FME desktop.

 

That was before the introduction of the xyz and PointCloud readers, so I would try them first to see if FME can handle the size. 
You can use Point Cloud XYZ reader to read your BIG data.
thanks for both your suggestions, once read into FME im scared what will happen when i try and run any transformers on it. Ie spatial join to roads...... i will have a try and see if the magic smoke appears out of my pc.  (250g SSD,64 bit windows and fme, 16gig ram and good processor.

 

 

Thanks steve
Hi Steve, Any smoke signs yet? :)

 

You could try to convert the point first to a more efficient format (FFS) and see if that helps.

 

Itay
sounds like a tiling strategy may be usefull when you want to do spatial operations on such a big pointcloud
Thanks all for your suggestions. I have been pulled into another project for a few days (this one pays). Once i have done this i will report back my findings with all your suggestions. 

 

 

Keep them coming. Thanks Steve

Reply


Community Stats

30,364
Posts
115,370
Replies
38,141
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.