[opencms-dev] speed of CmsExport / ZipOutputStream bug

[Ruben Malchow]

hi alex,

i haven't found a detailed description anywhere, but you may want to 
play around with the attached example. try running it in 1.5.x and 1.4.x 
and watch the difference ... it really is amazingly inefficient, 
especially when you bear in mind that the older implementation (in 
1.4.x) does it far better.

.rm



Alexander Kandzior schrieb:
> Ruben,
> 
> Thanks for your note about the export speed.
> 
>> this is a serious bug in java, verfied to exist in 1.5.0_07,
> 
> Can you provide any pointers (URLs) where this bug in Java is described? I
> would like to verify this before including such a patch in the official
> release.
> 
> Kind Regards,
> Alex.
> 
> -------------------
> Alexander Kandzior
>                                                                
> Alkacon Software GmbH  - The OpenCms Experts                        
> http://www.alkacon.com - http://www.opencms.org               
>  
> 
>> -----Original Message-----
>> From: opencms-dev-bounces at opencms.org 
>> [mailto:opencms-dev-bounces at opencms.org] On Behalf Of Ruben Malchow
>> Sent: Wednesday, August 09, 2006 10:50 PM
>> To: The OpenCms mailing list
>> Subject: Re: [opencms-dev] speed of CmsExport / ZipOutputStream bug
>>
>>
>> hello list,
>>
>> i have written a little patch for the problem described below. built 
>> from the 6.0.x and 6.2.x branches from the cvs, respectively:
>>
>> 	http://disco.kalkatraz.de/opencms/opencms-6.0.x-rm1.jar
>> 	http://disco.kalkatraz.de/opencms/opencms-6.2.x-rm1.jar
>>
>> and the corresponding sources:
>>
>> 	http://disco.kalkatraz.de/opencms/opencms_6.0.x-rm1_src.zip
>> 	http://disco.kalkatraz.de/opencms/opencms_6.2.x-rm1_src.zip
>>
>> the only change is in
>>
>> 	/org/opencms/importexport/CmsExport.java
>>
>> if you are running jdk1.5* and want to give it a try, replace the 
>> original opencms.jar with the matching ones here.
>>
>> .rm
>>
>>
>>
>> Ruben Malchow schrieb:
>>>
>>> hello list,
>>>
>>> i can't stop wondering: this time, i wonder if it's possible that 
>>> exporting 10 files around 3mb each can take more than 15 
>> minutes? so, 
>>> this is a 3ghz p4, 2g ram: no swapping going on, and - this 
>> is weird - 
>>> not even a lot of disk activity, the system is at 100% ... 
>> what's taking 
>>> so long?
>>>
>>> i dived into this, and did a little test: the ZipInputStream is 
>>> EXTREMELY inefficient when it's being confronted with LARGE 
>> byte arrays.
>>> so, instead of <1sec when reading from a file and writing 
>> to a ZipEntry 
>>> in 1024-byte chunks, it took 60ms to read the entire ~3mb to a byte 
>>> array, and then MORE THAN ONE HUNDRED SECONDS to write it 
>> to the zip entry.
>>> now, i don't use capital letters very often, but the 
>> ZipOutputStream 
>>> seem to be incredibly inefficient. so basically, i propose 
>> to replace 
>>> this code in CmsExport.java:
>>>
>>>     getExportZipStream().write(file.getContents());
>>>
>>> with this code:
>>>
>>>     byte[] buff=file.getContents();
>>>     int offset=0;
>>>     int chunkSize=1024;
>>>     while((buff.length-offset)>0) {
>>>         getExportStream.write(
>>>             buff,
>>>             offset,
>>>             Math.min(chunkSize,(buff.length-offset))
>>>         );
>>>         offset+=chunkSize;
>>>     }
>>>
>>> to speed the export up by an incredible 10000 percent - 
>> especially if 
>>> you have lots of large files. this is a serious bug in 
>> java, verfied to 
>>> exist in 1.5.0_07, but not in 1.4.2_08 on win32, but 
>> anticipating this 
>>> wouldn't hurt very much. in an example on java.sun.com, 
>> they even used a 
>>> bigger buffer (4096 byte) ...
>>>
>>>
>>> .rm
>>>
>>>
>>>
>>> _______________________________________________
>>> This mail is sent to you from the opencms-dev mailing list
>>> To change your list options, or to unsubscribe from the 
>> list, please visit
>>> http://lists.opencms.org/mailman/listinfo/opencms-dev
>>>
>> _______________________________________________
>> This mail is sent to you from the opencms-dev mailing list
>> To change your list options, or to unsubscribe from the list, 
>> please visit
>> http://lists.opencms.org/mailman/listinfo/opencms-dev
>>
>>
> 
> 
> _______________________________________________
> This mail is sent to you from the opencms-dev mailing list
> To change your list options, or to unsubscribe from the list, please visit
> http://lists.opencms.org/mailman/listinfo/opencms-dev
> 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ZipTest.java
URL: <https://webmail.opencms.org/pipermail/opencms-dev/attachments/20060810/8630a0a1/attachment.ksh>