Skip to main content
  • EN

Hi folks,

I got the attached error, very basic process, I'm joining a table loaded from a CSV with another CSV. It should be easy and my transformers are correct.

https://imgur.com/P3jkoQM

Hi @osarhan,

Are you try to merger between two CSV files?

If yes, try with the transformer FeatureMerger.

Thanks,

Danilo

Hi @osarhan,

Are you try to merger between two CSV files?

If yes, try with the transformer FeatureMerger.

Thanks,

Danilo

I was originally doing this, but both files are extremely big so it it hits memory constraints 😞
I was originally doing this, but both files are extremely big so it it hits memory constraints :(
Perfect. Could you share the print inside your transformer Paramenters Joiner and share it?

 

 

Thanks,

 

Danilo

 

 

I don't think the Joiner is particularly well suited to join two huge CSV files, personally.

I would use the FeatureMerger, but it's important to make sure that you use two separate readers, one for the supplier CSV and one for the requestor CSV.

In the Navigator, make sure the suppliers CSV comes at the very top, this means that dataset will be read before the requestors.

You can then tell the FeatureMerger that the suppliers come first, which will "unblock" then transformer making it a lot more efficient memory-wise.

Another alternative could be to use the InlineQuerier, but it's slightly more complicated. Let us know if the "non-blocking" FeatureMerger doesn't work, and we'll help you if needed.

Reply


Community Stats

30,189
Posts
114,582
Replies
37,853
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.