Hello Everyone,
I have 17,000 TIFF files(~3Mb each) that I need to merge to make just one file. While I create the workbench and read the data as Single Merged Feature Type, it takes a lot of time to read the data (feature types to read) and once I run the wb, it ends up crashing or the output(zipped) is not merged.
Any solution for this?
A rough calculation is that you've got about 50 Gb worth of TIFF files. That's most likely causing memory issues.
Does the output have to be TIFF? How about ECW or JPG2000?
A rough calculation is that you've got about 50 Gb worth of TIFF files. That's most likely causing memory issues.
Does the output have to be TIFF? How about ECW or JPG2000?
So I asked the Gurus here and they think this should work. Lots of disk space will be needed though. If you could post the log from a failed run (or follow up with support) that would be helpful. We think very little memory would actually be in use at any given time. Assuming you have 3-band uncompressed data as input, if the mosaicked image is roughly square, it would be around 133000x133000. We would have to keep at least one tile worth of data in memory; the size of that tile can be configured on the tiff writer, but defaults to one row (1x133000) which is not much memory at all. So it should be okay. We would suggest trying to write to just an uncompressed Tiff to begin with just to be sure its not the compression getting you at the end, provided you've got the disk space.
(May be relevant to find out what FME Version you're running too).
Good challenge.
Oh and one last tip -- when generating the workspace, you could just pick one file when you're generating (and say single merged feature type) and then go and update the source dataset afterwards to *.tiff since the schema isn't something you care about (by default we will be checking the schema of all the input files when you say "single merged" in case they have interesting attributes, and then merging all that we find...great for DWG files, not so much for GeoTIFF)
So I asked the Gurus here and they think this should work. Lots of disk space will be needed though. If you could post the log from a failed run (or follow up with support) that would be helpful. We think very little memory would actually be in use at any given time. Assuming you have 3-band uncompressed data as input, if the mosaicked image is roughly square, it would be around 133000x133000. We would have to keep at least one tile worth of data in memory; the size of that tile can be configured on the tiff writer, but defaults to one row (1x133000) which is not much memory at all. So it should be okay. We would suggest trying to write to just an uncompressed Tiff to begin with just to be sure its not the compression getting you at the end, provided you've got the disk space.
(May be relevant to find out what FME Version you're running too).
Good challenge.
Oh and one last tip -- when generating the workspace, you could just pick one file when you're generating (and say single merged feature type) and then go and update the source dataset afterwards to *.tiff since the schema isn't something you care about (by default we will be checking the schema of all the input files when you say "single merged" in case they have interesting attributes, and then merging all that we find...great for DWG files, not so much for GeoTIFF)
I am doing *.tif for reading the file and the output should be merged it directly into the writer. But, what unlikely I am facing is the output is unmerged and just same as the input, just that there is one more 1kb file (TIFF_1.tifw). I am sharing the log file in the attachment. Please spare some time to review it.tiff2tiff.zip
I am doing *.tif for reading the file and the output should be merged it directly into the writer. But, what unlikely I am facing is the output is unmerged and just same as the input, just that there is one more 1kb file (TIFF_1.tifw). I am sharing the log file in the attachment. Please spare some time to review it.tiff2tiff.zip