Skip to main content
Solved

Does JSONTemplater have the ability to create buckets?

M

Good afternoon all,

I'm trying to create a large nested JSON document from SQL to load into MongoDB. Currently I have managed to create the JSON file but i'm not able to load it into Mongo based on the size of the documents. The maximum size for documents is 16MB while my average document size is over 50MB.

A solution I have thought of is to bucket each of the sub files so that instead of thousands of documents being embedded into each root there is a maximum limit of say 500 and then a new document is created with the same root, but a continuation of the Sub documents. In the end instead of a single 50MB document I would be left with say 10 identical documents that would hold different Sub documents from one another. I could then query the set of 10 documents using $unwind so that it treated them as a single document of sorts.

Does anyone know of a way to achieve this within FME Workbench instead of loading all sub documents into a root?

Best answer by takashi

Hi @majobajob, sounds you are looking for a way to create copies of a common root feature for every group. If I understood the requirement correctly, this mockup might help you. You can use the FeatureMerger to merge attributes from a single Supplier feature to every Requestor feature, if you set an identical constant value (e.g. 1) to the Join On parameter for both Requestor and Supplier.

View original
Did this help you find an answer to your question?

5 replies

M
  • majobajob
  • Author
  • October 4, 2018
Update: I know i can use the AttributeCreator to make groupID's by setting the attribute value as '@Evaluate(@int(@Count()/500))' - but im not any closer on working out how to feed each group individually through the JSONTemplater to create separate documents.

 

 

My thoughts are that I would set a counter value as 0, pass all documents with the groupID equal to the counter through the templater, write these to JSON and then loop back to the start, incrementing the counter by 1 and running the process repeatedly until it reaches the end. However, I have no idea if this is feasible in FME.

 

 

Any suggestions?

 

 

M
  • majobajob
  • Author
  • October 4, 2018
Update #2: I have managed to create what I am looking for manually thus far - but it is entirely unscalable. I've attached a couple of images to show the growth of the files from 1 root document to 2.

 

First the manual solution, that returns the data in the manner that I would be able to use it:

 

Then what happens when I attempt it for 2 files:

 

Considering that there are approx. 29,500 files to go through, I am hoping that someone can see a way to clear the mess down and suggest a simpler solution.

 

takashi
Influencer
  • takashiInfluencer
  • Best Answer
  • October 5, 2018

Hi @majobajob, sounds you are looking for a way to create copies of a common root feature for every group. If I understood the requirement correctly, this mockup might help you. You can use the FeatureMerger to merge attributes from a single Supplier feature to every Requestor feature, if you set an identical constant value (e.g. 1) to the Join On parameter for both Requestor and Supplier.

M
  • majobajob
  • Author
  • October 5, 2018
takashi wrote:

Hi @majobajob, sounds you are looking for a way to create copies of a common root feature for every group. If I understood the requirement correctly, this mockup might help you. You can use the FeatureMerger to merge attributes from a single Supplier feature to every Requestor feature, if you set an identical constant value (e.g. 1) to the Join On parameter for both Requestor and Supplier.

Hi Takashi, Thank you so much for sharing this. The logic seems to show exactly what I want to achieve. I'll build it in and test it later this evening and let you know the result. Thanks again!

 

 

M
  • majobajob
  • Author
  • October 5, 2018
takashi wrote:

Hi @majobajob, sounds you are looking for a way to create copies of a common root feature for every group. If I understood the requirement correctly, this mockup might help you. You can use the FeatureMerger to merge attributes from a single Supplier feature to every Requestor feature, if you set an identical constant value (e.g. 1) to the Join On parameter for both Requestor and Supplier.

Hi again Takashi, this was a great answer. Did everything I needed with little to no amendments

 

 

Reply


Helpful Members This Week

Community Stats

31,105
Posts
117,925
Replies
38,857
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.