[opencms-dev] Large files upload fails

[Paul-Inge Flakstad]

Hi again Marc

I contacted our in-house DB guru, he helped me dig deeper.

There were some strange things going on. Although I had set the max_allowed_packet variable in MySQL config file, this setting was not employed as intended. 

The return value of the command (in bash):
mysql --verbose --help | grep max_allowed_packet

was inconsistent with the return value of the query (in mysql shell):
show variables like 'max_allowed_packet';

We had to add the max_allowed_packet variable in a different part of the config file file - specifically, in the [mysql] section instead of the [mysqld] section. After this small change, the return values are consistent, and large file upload is working fine (yay!).

For reference: 
1) We ran a describe query on the CMS_OFFLINE_CONTENTS table, which revealed that the data field is in fact a longblob field and thus does not have to be altered.
2) I had a problem with the publishing of large files as well, even though the files got uploaded correctly. Checking the OpenCms log file, I identified the cause to be 'java.lang.OutOfMemoryError: Java heap space'. This was easily solved by increasing the Java heap space.

Many thanks to Marc and Sébastien for useful input. :-)

Cheers,
Paul


> -----Original Message-----
> From: opencms-dev-bounces at opencms.org 
> [mailto:opencms-dev-bounces at opencms.org] On Behalf Of Marc Johnen
> Sent: 25. juni 2009 13:54
> To: opencms-dev at opencms.org
> Subject: Re: [opencms-dev] Large files upload fails
> 
> 
> Hi Paul,
> are you sure the changes to MySql are active? Did you edit the
> correct config-file and restart MySql? I never had to change
> anything more than that property.
> Greetings
> Marc
> 
> 
> Paul-Inge Flakstad wrote:
> > 
> > Thanks for the reply, Marc. 
> > 
> > First, a correction: My previous mail said 
> "max_upload_packet", which was
> > a typo - it should've said "max_allowed_packet". It is set 
> to 64M so I
> > should have that case covered. 
> > 
> > I tried uploading files of various sizes yesterday, and it 
> is pretty clear
> > that there's some kind of ceiling at 16MB. All is well for 
> files <16MB,
> > but files that exceed 16MB (e.g. PDF files) cause problems.
> > 
> > Any other ideas? I read a post somewhere where someone 
> suggested changing
> > a MySQL field definition from (MEDIUM)BLOB to LONGBLOB - 
> could that be it
> > (sounds pretty awkward)?
> > 
> > -Paul
> > 
> >> -----Original Message-----
> >> From: opencms-dev-bounces at opencms.org 
> >> [mailto:opencms-dev-bounces at opencms.org] On Behalf Of Marc Johnen
> >> Sent: 25. juni 2009 12:01
> >> To: opencms-dev at opencms.org
> >> Subject: Re: [opencms-dev] Large files upload fails
> >> 
> >> 
> >> Hi Paul,
> >> yes you could have found the solution by searching in the 
> mailing list
> >> first.
> >> This is probably connected to the max_allowed_packet property 
> >> of MySql.
> >> Also I'd remove the "-Xms8m" from your config, because it 
> >> potentially create
> >> problems, but not related to uploading.
> >> Greetings
> >> Marc Johnen
> >> 
> >> 
> >> 
> >> Paul-Inge Flakstad wrote:
> >> > 
> >> > Hi all
> >> > 
> >> > I'm pretty sure this question is asked on a regular basis 
> >> in here, and my
> >> > apologies for asking it again. I just couldn't find any solution.
> >> > 
> >> > OpenCms misbehaves when I try to upload large files. One of 
> >> two scenarios
> >> > occur: either 1.) The upload applet closes and no file is 
> >> visible in the
> >> > workplace, or 2.) The upload applet closes and the file is 
> >> visible in the
> >> > workplace, but a CmsVfsResourceNotFoundException is thrown 
> >> when I try to
> >> > view the file.
> >> > 
> >> > No exception is thrown during the applet upload. If I 
> >> disable "Use upload
> >> > applet", a CmsVfsResourceNotFoundException is thrown.
> >> > 
> >> > Now, I've set/checked some values to try and fix the problem:
> >> > 
> >> > 1.) In my.cnf: max_upload_packet=64M
> >> > 2.) In opencms-workplace.xml: 
> >> <maxfileuploadsize>-1</maxfileuploadsize>
> >> > 3.) In catalina.bat: CATALINA_OPTS="-Xmx256m -Xms8m"
> >> > 
> >> > After changing to this, an upload of a 2.7MB .pdf file 
> worked (which
> >> > didn't work prior to the change), but I still can't upload 
> >> a 16.0MB .pdf
> >> > file... Is there anything else I need to check?
> >> > 
> >> > Any solution, tip or idea is most welcome.
> >> > 
> >> > Cheers,
> >> > Paul
> >> > 
> >> > 
> >> > _______________________________________________
> >> > This mail is sent to you from the opencms-dev mailing list
> >> > To change your list options, or to unsubscribe from the 
> >> list, please visit
> >> > http://lists.opencms.org/mailman/listinfo/opencms-dev
> >> > 
> >> 
> >> -- 
> >> View this message in context: 
> >> http://www.nabble.com/Large-files-upload-fails-tp24187053p2420
> > 0400.html
> >> Sent from the OpenCMS - Dev mailing list archive at Nabble.com.
> >> 
> >> 
> >> _______________________________________________
> >> This mail is sent to you from the opencms-dev mailing list
> >> To change your list options, or to unsubscribe from the list, 
> >> please visit
> >> http://lists.opencms.org/mailman/listinfo/opencms-dev
> >> 
> > 
> > _______________________________________________
> > This mail is sent to you from the opencms-dev mailing list
> > To change your list options, or to unsubscribe from the 
> list, please visit
> > http://lists.opencms.org/mailman/listinfo/opencms-dev
> > 
> > 
> 
> -- 
> View this message in context: 
> http://www.nabble.com/Large-files-upload-fails-tp24187053p2420
1834.html
> Sent from the OpenCMS - Dev mailing list archive at Nabble.com.
> 
> 
> _______________________________________________
> This mail is sent to you from the opencms-dev mailing list
> To change your list options, or to unsubscribe from the list, 
> please visit
> http://lists.opencms.org/mailman/listinfo/opencms-dev
>