Skip to main content
Solved

ZipArchiver truncates xml file while zipping

  • Oct 14, 2026 2:15 PM UTC
  • 2 replies
  • 28 views

s.jager
Influencer
Forum|alt.badge.img+27

So I have a process that extracts data from a database, converts that data into a fairly complex xml, downloads a PDF from Sharepoint, and at the end it uses the ZipArchiver to add both PDF and XML to a zipfile. This works fine, except that at some point in time the XML in the zipfile is truncated. This of course results in invalid xml, so it is not accepted downstream. The file itself is fine, since I leave that in it’s original location that’s easy to check. Somewhere inside the ZipArchiver or inside the zip the XML is cut off. It does not seem to have to do with the number of characters in the xml, since it differs between various zipfiles as tohow many characters of the xml have survived. Strangely enough the PDF inside the same zipfile is fine, usually many times larger than the xml (one example has a 39.2 Mb PDF, and a 349kB xml...), so I don’t think it’s the filesize (unless 349 is too small...).

 

Has anybody else seen this before? I’ve been looking at the Python-code inside the ZipArchiver, but I do not see anything suspicious.

Best answer by s.jager

Just found the culprit: it was a race condition. Since this process has been in development for a few years now, there is quite a bit of stuff in there that was quite old. And that was using a regular TextWriter to write the xml out to disk. I’m fairly certain that the TextWriter wasn’t finished when the ZipArchiver tried to zip the results. Because as soon as I replaced the TextWriter with an AttributeFileWriter (thus not continuing until the xml is completely written to disk!), the problem disappeared…

So I’m 99% convinced that was the problem (it also explains why it did not happen with the PDF, since that is downloaded by the SharepointOnlineConnector - which also does not continue until the doc is downloaded)… Leaving this out here just in case others run into similar issues. And as a reminder to myself to keep a closer eye on race conditions...

View original

2 replies

s.jager
Influencer
Forum|alt.badge.img+27
  • Author
  • Influencer
  • Oct 15, 2026 8:13 AM UTC

Could that be a memory issue? Trying to reproduce the issue, I made a little test-workspace which uses a few attribute creators to create attributes with filenames and the ZipArchiver. And to my surprise no truncated xml! The original workspace however is still truncating the xml (in this case it cuts off 982 characters from what should be a total of 357311 characters), but there’s also a few warnings relating to the ZipArchiver:

918405  ResourceManager: Detected system low memory state while using 363589632 of memory
918421  ResourceManager: Reducing process memory usage. Start freeing memory at: 363589631
918427  Memory usage above the limit was detected: Either 346.75 MB exceeds 332.88 MB memory or 5.38 GB exceeds 16777216.00 TB address space
        ResourceManager: Freeing! Over private memory limit: 346.79 MB > 346.75 MB
        ResourceManager: Freed 14606580 bytes of table recycled block data (memory decreased by 25559040 bytes)
17007   Schema scanning encountered features with different coordinate systems. Preserving 'Netherlands-RDNew-2008' and ignoring ''
 

Could it be that Python’s memory is cut short when adding the file to the zip, thus truncating it? Would rather surprise me, but the fact that the xml on disk is fine but the xml in the zip isn’t already caught me by surprise… If that is the case, how would I prevent that?


s.jager
Influencer
Forum|alt.badge.img+27
  • Author
  • Influencer
  • Best Answer
  • Oct 15, 2026 8:59 AM UTC

Just found the culprit: it was a race condition. Since this process has been in development for a few years now, there is quite a bit of stuff in there that was quite old. And that was using a regular TextWriter to write the xml out to disk. I’m fairly certain that the TextWriter wasn’t finished when the ZipArchiver tried to zip the results. Because as soon as I replaced the TextWriter with an AttributeFileWriter (thus not continuing until the xml is completely written to disk!), the problem disappeared…

So I’m 99% convinced that was the problem (it also explains why it did not happen with the PDF, since that is downloaded by the SharepointOnlineConnector - which also does not continue until the doc is downloaded)… Leaving this out here just in case others run into similar issues. And as a reminder to myself to keep a closer eye on race conditions...