[opencms-dev] Queue time is getting longer

Achim Westermann achim.westermann at questico.de
Fri Dec 4 09:30:22 CET 2009 

Hi Lorenzo,

exactly as Christian said. If it is possible to find proper export rules to publish faster.

Disadvantage:

within your export rule config there is knowlege: What collector list pages (<contentload collector="allIinxxx"...) drag in what contents 
(e.g. news articles). If you collect all news below "/" for such a list page publishing a news will force to re-export "/".
Also you have to maintain those rules whenever your site structure changes.

That's why personally I prefer using static export on demand. But I can understand that some projects require static export after publish. 
After all that export handler would not have been written if not.

kind regards,

Achim

Lorenzo Sala wrote:
> Ciao Achim,
>                may be I found the real problem that will solve most of our
> long waits. There is only 1 project that is exporting only the modified
> files and it takes about 1 second to export them. This project is the only
> one that has a <export-rule> configuration in the opencms-importexport.xml.
> All the other projects doesn't appear in this file at all. Could this be
> what causes CMS to republish ALL the files of all the projects when we do an
> export?
> 
> I'm setting up some <export-rule> for the rest of the projects and tomorrow
> we'll see if that worked.
> 
> Lo'
> 
> 
> Achim Westermann wrote:
>> Ciao Lorenzo,
>>
>> I know this level of system-architecture talking and working from my
>> projects. And sometimes I wasn't loud & clear enough (or my customer 
>> was brave enough to ignore me) to have a very simple basic setup (that may
>> be scaled up by clustering of opencms and mysql transparently).
>>
>> Perhaps if you start a new website project you could start with a single
>> machine again and then do it as described. Or hire some very 
>> experienced consultant to get the setup started. It's worth the money.
>>
>> kind regards,
>>
>> Achim
>>
>>
>> Lorenzo Sala wrote:
>>> Ciao Achim,
>>> yes.. we are using the badstyle.. we have several crontabs scheduled, one
>>> for every project.. and for the links most are absolute and only direct
>>> link
>>> to docs like pdf files are relative but on subfolders.
>>>
>>> It looks like we didn't set up in the right way our first projects.. it
>>> would be a mess to rebuild them all now...
>>>
>>> Lo'
>>>
>>>
>>> Achim Westermann wrote:
>>>> Hi Lo,
>>>>
>>>> great, now you need 1 second!
>>>>
>>>> Read more how this export handler exports files to the harddrive:
>>>> http://opencms-wiki.org/Static_Export_On_Demand
>>>>
>>>> Some words by me: You have to wrap all links to exportable resources in
>>>> <cms:link>link</cms:link> tags. Files go to
>>>>
>>>> <tomcat>/webapps/<webappname>/export/...
>>>>
>>>> Now I fear you have a setup where your web-server is on a different
>>>> machine. That will not work very good. The webservers doc root should 
>>>> always have the doc root of <tomcat>/webapps/<webappname>/
>>>> because <cms:link> creates links to the export subfolder and only if
>>>> those
>>>> files are requested apache webserver will request their export 
>>>> from tomcat.
>>>>
>>>> See this:
>>>> http://opencms-wiki.org/MultiSite_configuration_instructions_%28apache_%2B_mod_jk%29
>>>>
>>>> Especially take a close look at the section "Defining the virtual hosts"
>>>> where "magic" rules are used to trigger static export on demand 
>>>> requests to the tomcat.
>>>>
>>>> If you cannot have the webserver and appserver on one machine (believe
>>>> me,
>>>> that is faster than on two machines) then you had to set up some 
>>>> rsync cronjob  (rsync -auv
>>>> root at oherserver:/opt/tomcat/webapps/opencms/export/) on the webserver
>>>> machine. But that is bad style.
>>>>
>>>> good luck,
>>>>
>>>> Achim
>>>>
>>>>
>>>>
>>>>
>>>> Lorenzo Sala wrote:
>>>>> Hello Achim,
>>>>>                  we tried to use the CmsOnDemandStaticExportHandler.
>>>>> Rebooted, tried creating a new file and publishing it. It took 1 second
>>>>> to
>>>>> run the publishing and this is the log:
>>>>>
>>>>> Publishing resources ...
>>>>> Publishing files ...
>>>>> ( 1 / 1 ) Publishing file /prova20091203.html ... o.k.
>>>>> ... finished publishing files
>>>>> Statistics: published files: 1, published folders: 0, deleted folders:
>>>>> 0,
>>>>> duration: 00:00:00
>>>>> ... the resources have been published
>>>>> Updating search index "Online project EN (VFS)"
>>>>> ( 1 ) Indexing file /sites/extranet/prova20091203.html ... o.k.
>>>>> ... finished updating search index "Online project EN (VFS)"
>>>>> Updating search index "Online project DE (VFS)"
>>>>> ( 1 ) Indexing file /sites/extranet/prova20091203.html ... o.k.
>>>>> ... finished updating search index "Online project DE (VFS)"
>>>>>
>>>>> The problem is that on the export server the file doesn't exist, it
>>>>> didn't
>>>>> publish it but it says it did.
>>>>>
>>>>> Are we doing something wrong?
>>>>>
>>>>> Lo'
>>>>>
>>>>>
>>>>>
>>>>> Achim Westermann wrote:
>>>>>> Hi Lo,
>>>>>>
>>>>>> that is not normal behaviour of OpenCms.
>>>>>>
>>>>>> I assume you are using the CmsAfterPublishStaticExportHandler or a
>>>>>> custom
>>>>>> similar implementation in opencms-importexport.xml. If you are 
>>>>>> free in your setup (no "OpenCms MUST NOT BE IN PRODUCTION Mr.
>>>>>> knowitalls
>>>>>> in charge") try using CmsOnDemandStaticExportHandler.
>>>>>>
>>>>>>
>>>>>> kind regards,
>>>>>>
>>>>>> Achim
>>>>>>
>>>>>> Lorenzo Sala wrote:
>>>>>>> Hi everyone, 
>>>>>>>                 it's 4 months now that we have implemented OpenCMS in
>>>>>>> our
>>>>>>> web sites and we've been creating new Projects since then. We are
>>>>>>> getting
>>>>>>> worried about two major problems now:
>>>>>>>
>>>>>>> 1) All the projects use the same queue to publish, so if many users
>>>>>>> sends
>>>>>>> a
>>>>>>> request to publish a document while working on different projects the
>>>>>>> queue
>>>>>>> tends to get really long.
>>>>>>>
>>>>>>> 2) The time to processa each single publishing request is getting
>>>>>>> longer,
>>>>>>> it
>>>>>>> was 5 minutes and now it's 13 minutes.
>>>>>>>
>>>>>>> Is there a way to better distribute or make faster the publishing
>>>>>>> queue?
>>>>>>> The
>>>>>>> publishing time of a single request doesn't change if you asked to
>>>>>>> publish 1
>>>>>>> or 500 files, now they all take 13 minutes, meaning that if 5 people
>>>>>>> sent
>>>>>>> a
>>>>>>> publishing request the last one will have to wait about 1 hour to see
>>>>>>> it
>>>>>>> published.
>>>>>>>
>>>>>>> Any suggestion will be greatly appreciated.
>>>>>>>
>>>>>>> Lo'
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> This mail is sent to you from the opencms-dev mailing list
>>>>>> To change your list options, or to unsubscribe from the list, please
>>>>>> visit
>>>>>> http://lists.opencms.org/mailman/listinfo/opencms-dev
>>>>>>
>>>>>>
>>>> _______________________________________________
>>>> This mail is sent to you from the opencms-dev mailing list
>>>> To change your list options, or to unsubscribe from the list, please
>>>> visit
>>>> http://lists.opencms.org/mailman/listinfo/opencms-dev
>>>>
>>>>
>> _______________________________________________
>> This mail is sent to you from the opencms-dev mailing list
>> To change your list options, or to unsubscribe from the list, please visit
>> http://lists.opencms.org/mailman/listinfo/opencms-dev
>>
>>

More information about the opencms-dev mailing list