When support for ZipCrypto was added, an internal StringIO buffer was
added to Deflater, in order to fix a decryption bug. While this worked,
it caused unlimited memory growth when compressing large files. The
proper fix is to writer the encryption header in init_next_entry instead
of finalize_current_entry, so the headers are written before any
encrypted data. Because of this fix we can remove the buffering in
Deflater, which keeps memory usage low and allows to stream compressed
data while it is written.
This should fix issue #233.
Adding the io parameter to OutputStream::write_buffer breaks backward
compatibility with v1.1.0. Adding a default value reinstates backward
compatibility.
The local header size computed from the central directory entry
is incorrect due to the Zip64Placeholder in the local entry.
This caused us to seek to the wrong location when copying an
unchanged compressed data stream.
(The same problem could occur when modifying any zip file where
the local header and central directory header contain different
variable-sized fields, so it's a good idea not to trust the CD
to tell us the local header size in any case.)
This commit adds the capability of creating archives larger than
4GB via zip64 extensions. It also fixes bugs reading archives of
this size (specifically, the 64-bit offset of the local file
header was not being read from the central directory entry).
To maximize compatibility, zip64 extensions are used only when
required. Unfortunately, at the time we write a local file header,
we don't know the size of the file and thus whether a Zip64
Extended Information Extra Field will be required. Therefore
this commit writes a 'placeholder' extra field to reserve space
for the zip64 entry, which will be written if necessary when
we update the local entry with the final sizes and CRC. I use
the signature "\x99\x99" for this field, following the example
of DotNetZip which does the same.
This commit also adds a rake task, zip64_full_test, which
fully tests zip64 by actually creating and verifying a 4GB zip
file. Please note, however, that this test requires UnZip
version 6.00 or newer, which may not be supplied by your OS.
This test doesn't run along with the main unit tests because
it takes a few minutes to complete.
Sebastian Henke
Robert Haines
Takumasa Ochi
Felix Bünemann
Pavel Lobashov
Vít Ondruch
Pavel.Lobashov
Alexander Simonov
Shigeaki Matsumura
Johnny Shields
Max Cantor
Orien Madgwick
Jeremy Stanley
René Sprotte