[opencms-dev] CmsCollector performance
marcio.camurati
marcio.camurati.opencms at gmail.com
Wed Mar 5 22:02:18 CET 2008
Ok, yes it answer me, at the afternoon I read more the CORE code and see that
with this I will only add more filters possibilities direct to SQL, at my
little tests it really return the resources with a best performance.
The problem that I see on it is that is necessary alter codes direct at the
CORE of OpenCMS and it's a bad thinking that one day we will need to upgrade
this version, but for now this help is very preciously.
Thanks every one.
Regards,
Marcio Camurati
Sebastian Himberger wrote:
>
> Hi again,
>
> haven't i just perfectly ignored your simple question and given a maybe
> useless explanation?! :D. To answer it directly: No OpenCms will not
> always hit the database if you use FlexCache. It will serve the output
> from the cache.
>
> best regards,
> sebastian
>
>
>
>
> Sebastian Himberger schrieb:
>> Hi,
>>
>> Flex is more about caching views (page elements, output) than about
>> caching result sets. They sit more or less on top of each other. There's
>> a dedicated db cache for queries.
>>
>> i hope this helps.
>>
>> sebastian
>>
>>
>> marcio.camurati schrieb:
>>> Oh, no problem about it the source that you send give me the macro
>>> vision of
>>> what is needed, and see it and think that it really help about
>>> performance
>>> because with it we can get only what is really necessary fo show and not
>>> all
>>> to filter after.
>>>
>>> The other doubt that I have is, using this new custom query the OpenCMS
>>> will
>>> use the FlexCache to cache this custom query ? Or make this all request
>>> will
>>> be direct to the database ?
>>>
>>> Do you know about it ?
>>>
>>> Best regards,
>>> Marcio Camurati
>>>
>>>
>>>
>>> Shi Yusen wrote:
>>>> Sorry for not open the performance module. We have to make a living.
>>>>
>>>> Anyway, here are the steps to build your own module:
>>>> 1. Add some new filters in CmsResourceFilter. For example, I added two
>>>> new filter methods: addTopLatest(int top) and addPagedLatest(int
>>>> startRow, int rowsInPage).
>>>>
>>>> 2. Add new filter branches in
>>>> org.opencms.db.CmsDriverManager.readResources(...)
>>>>
>>>> 3. Add new functions to implements the new filter branches in
>>>> org.opencms.db.oracle.CmsVfsDriver
>>>>
>>>> 3. Add new query strings to /org/opencms/db/oracle/query.properties.
>>>> For
>>>> example, you can use
>>>> # patterns for statements to select resources/folders (= selections
>>>> without content)
>>>> # THINGS TO KNOW: don't select the project-ID attrib. of the structure
>>>> table per default!
>>>> # There are cases, where the project-ID attrib. of the
>>>> project-resources
>>>> tab. is used
>>>> # as the project-ID!
>>>> C_ORACLE_RESOURCES_SELECT_ATTRIBS_LEVEL1=\
>>>> STRUCTURE_ID,\
>>>> RESOURCE_ID,\
>>>> RESOURCE_PATH,\
>>>> STRUCTURE_STATE,\
>>>> DATE_RELEASED,\
>>>> DATE_EXPIRED,\
>>>> STRUCTURE_VERSION,\
>>>> RESOURCE_ID_2,\
>>>> RESOURCE_TYPE,\
>>>> RESOURCE_FLAGS,\
>>>> RESOURCE_STATE,\
>>>> DATE_CREATED,\
>>>> DATE_LASTMODIFIED,\
>>>> USER_CREATED,\
>>>> USER_LASTMODIFIED,\
>>>> LOCKED_IN_PROJECT,\
>>>> RESOURCE_SIZE,\
>>>> DATE_CONTENT,\
>>>> SIBLING_COUNT,\
>>>> RESOURCE_VERSION
>>>>
>>>> # patterns for statements to select resources/folders (= selections
>>>> without content)
>>>> # THINGS TO KNOW: don't select the project-ID attrib. of the structure
>>>> table per default!
>>>> # There are cases, where the project-ID attrib. of the
>>>> project-resources
>>>> tab. is used
>>>> # as the project-ID!
>>>> C_ORACLE_RESOURCES_SELECT_ATTRIBS_LEVEL2=\
>>>> CMS_${PROJECT}_STRUCTURE.STRUCTURE_ID AS STRUCTURE_ID,\
>>>> CMS_${PROJECT}_STRUCTURE.RESOURCE_ID AS RESOURCE_ID,\
>>>> CMS_${PROJECT}_STRUCTURE.RESOURCE_PATH AS RESOURCE_PATH,\
>>>> CMS_${PROJECT}_STRUCTURE.STRUCTURE_STATE AS STRUCTURE_STATE,\
>>>> CMS_${PROJECT}_STRUCTURE.DATE_RELEASED AS DATE_RELEASED,\
>>>> CMS_${PROJECT}_STRUCTURE.DATE_EXPIRED AS DATE_EXPIRED,\
>>>> CMS_${PROJECT}_STRUCTURE.STRUCTURE_VERSION AS STRUCTURE_VERSION,\
>>>> CMS_${PROJECT}_RESOURCES.RESOURCE_ID AS RESOURCE_ID_2,\
>>>> CMS_${PROJECT}_RESOURCES.RESOURCE_TYPE AS RESOURCE_TYPE,\
>>>> CMS_${PROJECT}_RESOURCES.RESOURCE_FLAGS AS RESOURCE_FLAGS,\
>>>> CMS_${PROJECT}_RESOURCES.RESOURCE_STATE AS RESOURCE_STATE,\
>>>> CMS_${PROJECT}_RESOURCES.DATE_CREATED AS DATE_CREATED,\
>>>> CMS_${PROJECT}_RESOURCES.DATE_LASTMODIFIED AS DATE_LASTMODIFIED,\
>>>> CMS_${PROJECT}_RESOURCES.USER_CREATED AS USER_CREATED,\
>>>> CMS_${PROJECT}_RESOURCES.USER_LASTMODIFIED AS USER_LASTMODIFIED,\
>>>> CMS_${PROJECT}_RESOURCES.PROJECT_LASTMODIFIED AS LOCKED_IN_PROJECT,\
>>>> CMS_${PROJECT}_RESOURCES.RESOURCE_SIZE AS RESOURCE_SIZE,\
>>>> CMS_${PROJECT}_RESOURCES.DATE_CONTENT AS DATE_CONTENT,\
>>>> CMS_${PROJECT}_RESOURCES.SIBLING_COUNT AS SIBLING_COUNT,\
>>>> CMS_${PROJECT}_RESOURCES.RESOURCE_VERSION AS RESOURCE_VERSION
>>>>
>>>> #
>>>> # General subtree selection statement
>>>> #
>>>> C_ORACLE_RESOURCES_READ_TREE_PAGED=\
>>>> SELECT \
>>>> ${C_ORACLE_RESOURCES_SELECT_ATTRIBS_LEVEL1} \
>>>> FROM (\
>>>> SELECT \
>>>> ${C_ORACLE_RESOURCES_SELECT_ATTRIBS_LEVEL1},\
>>>> ROWNUM AS ROW_NUMBER \
>>>> FROM (\
>>>> SELECT \
>>>> ${C_ORACLE_RESOURCES_SELECT_ATTRIBS_LEVEL2},\
>>>> CMS_${PROJECT}_RESOURCES.PROJECT_LASTMODIFIED \
>>>> FROM \
>>>> ${C_RESOURCES_SELECT_TABLES} \
>>>> WHERE \
>>>> ${C_JOIN_RESOURCE_STRUCTURE}
>>>>
>>>> #
>>>> # Resources order by DATE_LASTMODIFIED AND ROWNUM
>>>> #
>>>> C_ORACLE_RESOURCES_PAGED_ORDER_BY_DATELASTMODIFIED=\
>>>> ORDER BY CMS_${PROJECT}_RESOURCES.DATE_LASTMODIFIED DESC\
>>>> )\
>>>> ) \
>>>> WHERE ROW_NUMBER >=? AND ROW_NUMBER <?
>>>>
>>>> to get paged latest resources.
>>>>
>>>> Use
>>>> #
>>>> # General subtree selection statement
>>>> #
>>>> C_ORACLE_RESOURCES_READ_TREE=\
>>>> SELECT \
>>>> ${C_ORACLE_RESOURCES_SELECT_ATTRIBS_LEVEL1} \
>>>> FROM (\
>>>> SELECT \
>>>> ${C_ORACLE_RESOURCES_SELECT_ATTRIBS_LEVEL2},\
>>>> CMS_${PROJECT}_RESOURCES.PROJECT_LASTMODIFIED \
>>>> FROM \
>>>> ${C_RESOURCES_SELECT_TABLES} \
>>>> WHERE \
>>>> ${C_JOIN_RESOURCE_STRUCTURE}
>>>>
>>>> #
>>>> # Resources order by DATE_LASTMODIFIED
>>>> #
>>>> C_ORACLE_RESOURCES_ORDER_BY_DATELASTMODIFIED=\
>>>> ORDER BY CMS_${PROJECT}_RESOURCES.DATE_LASTMODIFIED DESC\
>>>> ) \
>>>> WHERE ROWNUM<=?
>>>>
>>>> to get top latest resources.
>>>>
>>>> To filter the resources by sql rather than by Java. That's the trick.
>>>> Too simple, right?
>>>>
>>>> Good luck,
>>>>
>>>> Shi Yusen/Beijing Langhua Ltd.
>>>>
>>>>
>>>> 在 2008-03-05三的 05:28 -0800，marcio.camurati写道：
>>>>> Shi Yusen,
>>>>>
>>>>> The velocity is really very nice !
>>>>>
>>>>> Can you give more informations about this project, do you use only
>>>>> simple
>>>>> collectors like allInFolderDateReleasedDesc or
>>>>> allInSubTreeDateReleasedDesc
>>>>> for get this information at the OpenCMS resources?
>>>>>
>>>>> You said at the end for don't worry about the sql problem, this
>>>>> problem
>>>>> really exit if we use the OpenCMS at CORE it´s really necessary any
>>>>> modification about sql operation to resolve the performance problem
>>>>> that
>>>>> Martin post to us ?
>>>>>
>>>>> Best regards,
>>>>> Marcio Camurati
>>>>>
>>>>>
>>>>> Shi Yusen wrote:
>>>>>> Here is a website we almost completed with more than 50k pages now,
>>>>>> and
>>>>>> about 15k-20k more annually. You'll find it's quite fast.
>>>>>> http://www2.scnjw.com/scnjw/index.html
>>>>>>
>>>>>> CentOS + Squad + Apache + Tomcat + OpenCms 7.0.3 + Oracle.
>>>>>>
>>>>>> Don't worry about OpenCms performance. For the sql problem, you can
>>>>>> write your own performance module to improve the sql operation. It's
>>>>> not
>>>>>> difficult.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Shi Yusen/Beijing Langhua Ltd.
>>>>>>
>>>>>>
>>>>>> 在 2008-03-05三的 03:47 -0800，marcio.camurati写道：
>>>>>>> Hi Martin,
>>>>>>>
>>>>>>> I read you post, and see you cenario at our future project that is
>>>>>>> at
>>>>> the
>>>>>>> begin yet, the performance is one of our concern about use the
>>>>>>> OpenCMS
>>>>> to
>>>>>>> manage the content of the site, do you have any good news about your
>>>>>>> performance problem ? or it continuos ?
>>>>>>>
>>>>>>> If you can tallk more obut it or about the solution that you do for
>>>>> it.
>>>>>>> Best regards,
>>>>>>> Marcio Camurati
>>>>>>>
>>>>>>>
>>>>>>> Martin Bednář wrote:
>>>>>>>> I have problems with very poor performance of Collectors, specialy
>>>>> with
>>>>>>>> allInFolderDateReleasedDesc and allInSubTreeDateReleasedDesc. I
>>>>>>>> have
>>>>>>>> site with about >10000 articles categorized in folders, so i have
>>>>> this
>>>>>>>> structure
>>>>>>>>
>>>>>>>> /Categories
>>>>>>>> /Categories/Cat1
>>>>>>>> /Categories/Cat2
>>>>>>>> /Categories/CatX
>>>>>>>> ...
>>>>>>>>
>>>>>>>> in Cat1, CatX i have page which shows articles in whole category
>>>>> (with
>>>>>>>> paging by 20 articles)
>>>>>>>> on homepage i have 20 newest articles from all categories
>>>>>>>> I use something like:
>>>>>>>> <cms:contentload
>>>>>>>> collector="allInSubTreeDateReleasedDesc"
>>>>>>>> param="/Categories/|magArticle|20"
>>>>>>>> editable="true">
>>>>>>>>
>>>>>>>> respectively in cat1
>>>>>>>> <cms:contentload
>>>>>>>> collector="allInSubTreeDateReleasedDesc"
>>>>>>>> param="/Categories/Cat1/|magArticle"
>>>>>>>> pageSize="20" pageIndex="1" editable="true">
>>>>>>>>
>>>>>>>> Performance is very poor, I looked to source code and see that CMS
>>>>>>> works
>>>>>>>> (for my HP for exmaple) in this way:
>>>>>>>> Load /Categories resource from DB, Create CmsResource
>>>>>>>> Load all 10000 under /Categories resources from DB and Create
>>>>>>>> CmsResource objects for it
>>>>>>>> Sort all 10000 CmsResources by ReleaseDate
>>>>>>>> throw 9980 unneeded objects !!!
>>>>>>>> return 20 CmsResources
>>>>>>>>
>>>>>>>> It's really crazy.
>>>>>>>>
>>>>>>>> Is there a way how to optimize it ?
>>>>>>>>
>>>>>>>> Why data is not sorted on SQL server and returned only 20 items in
>>>>>>>> recordset ?
>>>>>>>>
>>>>>>>> I thing that it's really performance problem, waste of CPU and
>>>>> memory.
>>>>>>>> This operation takes about 2,3min on my server (CPU 2xQuadCore
>>>>> 2,4GHz,
>>>>>>>> 4GB RAM, 8xHDD on 3WareCard in RAID6) on my old server with AMD64
>>>>>>>> Opteron and 4HDD in SW RAID it takes about 14minutes !
>>>>>>>>
>>>>>>>> Martin
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> This mail is sent to you from the opencms-dev mailing list
>>>>>>>> To change your list options, or to unsubscribe from the list,
>>>>>>>> please
>>>>>>> visit
>>>>>>>> http://lists.opencms.org/mailman/listinfo/opencms-dev
>>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>>
>>>>> http://www.nabble.com/CmsCollector-performance-tp14931100p15848425.html
>>>>>>> Sent from the OpenCMS - Dev mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> This mail is sent to you from the opencms-dev mailing list
>>>>>>> To change your list options, or to unsubscribe from the list, please
>>>>>>> visit
>>>>>>> http://lists.opencms.org/mailman/listinfo/opencms-dev
>>>>>> _______________________________________________
>>>>>> This mail is sent to you from the opencms-dev mailing list
>>>>>> To change your list options, or to unsubscribe from the list, please
>>>>> visit
>>>>>> http://lists.opencms.org/mailman/listinfo/opencms-dev
>>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/CmsCollector-performance-tp14931100p15850039.html
>>>>> Sent from the OpenCMS - Dev mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> This mail is sent to you from the opencms-dev mailing list
>>>>> To change your list options, or to unsubscribe from the list, please
>>>>> visit
>>>>> http://lists.opencms.org/mailman/listinfo/opencms-dev
>>>> _______________________________________________
>>>> This mail is sent to you from the opencms-dev mailing list
>>>> To change your list options, or to unsubscribe from the list, please
>>>> visit
>>>> http://lists.opencms.org/mailman/listinfo/opencms-dev
>>>>
>>
>>
>> _______________________________________________
>> This mail is sent to you from the opencms-dev mailing list
>> To change your list options, or to unsubscribe from the list, please
>> visit
>> http://lists.opencms.org/mailman/listinfo/opencms-dev
>
>
> _______________________________________________
> This mail is sent to you from the opencms-dev mailing list
> To change your list options, or to unsubscribe from the list, please visit
> http://lists.opencms.org/mailman/listinfo/opencms-dev
>
--
View this message in context: http://www.nabble.com/CmsCollector-performance-tp14931100p15860187.html
Sent from the OpenCMS - Dev mailing list archive at Nabble.com.
More information about the opencms-dev
mailing list