Wow scary amounts of data.

 

A couple of possible approaches.......

 

1. If you have access to a database program, import tables into the database. Set up views (queries) in the database (eg to join the tables). This approach is making the database to do all the hard work. You can still usually use the views (queries) in FME if required further down your data processing line eg using the Joiner and other Transformers. You may even at this point have created a new table containing all merged sub tables. Using FME alone to do the hard work will hammer your PC with such large amounts of data. I suspect your machine is running out of memory.

 

2. Point cloud Transformers in FME may (should) be optimised to handle huge datasets. Point clouds are essentially text files of XYZ coordinates. You may be able to utilise these Transformers somehow to help process your data. You may have to create some new fields first in your sub tables (eg XYZ).

 

..............

 

Hope this helps

 

Howard L'

---

Hi Howard,

 

 

I though the InlineQuerier was actually creating a SQLite db and that the join was performed only inside that db.

 

 

I imported the same tables in a spatialite db outside FME, and did the same query. It was successful in 3 minutes.

 

 

I dont understand why InlineQuerier would take 4 hours (and stop without error message) when the same query in SQLite took 3 minutes.

 

 

Any idea?

 

 

---

Hi,

 

 

sounds eerily familiar to some similar problems I experienced with the InlineQuerier in FME 2013. I suggest you report them to Safe, just as I did.

 

 

David

---

Hi David,

 

 

Thank you for your feedback. I have reported this issue to my local FME supplier from whom I receive technical support.

 

 

Does Safe has provided solution or any information when you did report a similar issue?

---

Hi,

 

 

unfortunately, Safe were unable to reproduce the issue. Also, I couldn't send them any test data, for various reasons, so I had to let it go.

 

 

It would be great if you could contribute to resolving this!

 

 

David

---

HI,

 

 

I have reported the issue to Safe and their explanation about memory problem is explained by the fact that the InlineQuerier puts all field (geometry and description) in a single blob field instead of recreating the table schema in the SQLite db. I don't think they will put their investigations any further. They suggest to run the transformer in 64bits with more memory. I am running on 64bit with 8Go of RAM but that seems not enough. I think the solution then would be to import the tables in SQLite myself and do the join into SQLite, or to use chained FeatureMergers in FME.

---

Hi,

 

 

that's very interesting, thanks for posting this followup.

 

 

I think you're on a good track in your thoughts about using SQLite like this. Although I would recommend you do your joins within SQLite as much as possible, it will probably be an order of magnitude quicker than using chained FeatureMergers. It will probably also be a lot easier on your memory.

 

 

David

---