Skip to main content
  • EN

Good afternoon all,

I'm trying to create a large nested JSON document from SQL to load into MongoDB. Currently I have managed to create the JSON file but i'm not able to load it into Mongo based on the size of the documents. The maximum size for documents is 16MB while my average document size is over 50MB.

A solution I have thought of is to bucket each of the sub files so that instead of thousands of documents being embedded into each root there is a maximum limit of say 500 and then a new document is created with the same root, but a continuation of the Sub documents. In the end instead of a single 50MB document I would be left with say 10 identical documents that would hold different Sub documents from one another. I could then query the set of 10 documents using $unwind so that it treated them as a single document of sorts.

Does anyone know of a way to achieve this within FME Workbench instead of loading all sub documents into a root?

Update: I know i can use the AttributeCreator to make groupID's by setting the attribute value as '@Evaluate(@int(@Count()/500))' - but im not any closer on working out how to feed each group individually through the JSONTemplater to create separate documents.

 

 

My thoughts are that I would set a counter value as 0, pass all documents with the groupID equal to the counter through the templater, write these to JSON and then loop back to the start, incrementing the counter by 1 and running the process repeatedly until it reaches the end. However, I have no idea if this is feasible in FME.

 

 

Any suggestions?

 

 

Update #2: I have managed to create what I am looking for manually thus far - but it is entirely unscalable. I've attached a couple of images to show the growth of the files from 1 root document to 2.

 

First the manual solution, that returns the data in the manner that I would be able to use it:

 

Then what happens when I attempt it for 2 files:

 

Considering that there are approx. 29,500 files to go through, I am hoping that someone can see a way to clear the mess down and suggest a simpler solution.

 

Hi @majobajob, sounds you are looking for a way to create copies of a common root feature for every group. If I understood the requirement correctly, this mockup might help you. You can use the FeatureMerger to merge attributes from a single Supplier feature to every Requestor feature, if you set an identical constant value (e.g. 1) to the Join On parameter for both Requestor and Supplier.

Hi @majobajob, sounds you are looking for a way to create copies of a common root feature for every group. If I understood the requirement correctly, this mockup might help you. You can use the FeatureMerger to merge attributes from a single Supplier feature to every Requestor feature, if you set an identical constant value (e.g. 1) to the Join On parameter for both Requestor and Supplier.

Hi Takashi, Thank you so much for sharing this. The logic seems to show exactly what I want to achieve. I'll build it in and test it later this evening and let you know the result. Thanks again!

 

 

Hi @majobajob, sounds you are looking for a way to create copies of a common root feature for every group. If I understood the requirement correctly, this mockup might help you. You can use the FeatureMerger to merge attributes from a single Supplier feature to every Requestor feature, if you set an identical constant value (e.g. 1) to the Join On parameter for both Requestor and Supplier.

Hi again Takashi, this was a great answer. Did everything I needed with little to no amendments

 

 

Reply


Community Stats

30,108
Posts
114,295
Replies
37,744
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.