[opencms-dev] OpenCms 9.5.2: Not all files get statically exported with static export after publish

Tue Aug 27 13:52:12 CEST 2019

Hi Alex,

we finally found the problem. The flex cache params in opencms-system.xml where set too low AND there where problems in opencms-core which where only partially fixed in the current opencms-core branch_11_0_x on GitHub.
We added a pull request with the full fix. Please follow this link to see the details: https://github.com/alkacon/opencms-core/pull/666

So the problem occured, because the customer copied a folder with about 1500 files within the VFS. And this was enough to go over the limit set in opencms-system.xml

Best regards
Kai

Von: opencms-dev-bounces at opencms.org <opencms-dev-bounces at opencms.org> Im Auftrag von Schliemann, Kai
Gesendet: Mittwoch, 14. August 2019 16:06
An: The OpenCms mailing list <opencms-dev at opencms.org>
Betreff: Re: [opencms-dev] OpenCms 9.5.2: Not all files get statically exported with static export after publish

Hi Alex,

Thanks for the hint. We checked that, but it seems not to be the problem.
I am currently checking all folders in the content area by activating / deactivating the static export (by setting the property „export“ to „“ or „true“).
I already found out, that one main folder breaks the export. When I exclude that one, the export works fine.

By doing all the research now I found it hard to read the log files, because OpenCms very often gives me a 404 error where I would expect another or let’s say a better error message. (I have to admit, that all my resarch bases on the old and in the meantime deprecated ressource type xmlpage. So may be, some of my findings are not the case anymore.)

I get a 404 error in the following situations, where another or an additional info would make debugging easier:

  1.  When setting export to false for a resource (folder or single file) I get 404 errors for every file in all subfolders. Why not showing a message like „no export neccessary“. I it is not possible to avoid a 404, an additional info like the one I mentioned would be nice.
  2.  When there is no template set in the property „template“. Additional info could be „no template set for this file“.
  3.  The template produces a rendering error. This may be difficult. If possible an additional info could be „an error occured rendering this page with template „abc.jsp““.

I know that this is a rare case. But I would appreciate it, when the error messages would be more precise. Especially because I can follow the export info in the workplace, when triggering a „full static render“ in database management.

Best regards
Kai

Von: opencms-dev-bounces at opencms.org<mailto:opencms-dev-bounces at opencms.org> <opencms-dev-bounces at opencms.org<mailto:opencms-dev-bounces at opencms.org>> Im Auftrag von Jensen Asger
Gesendet: Montag, 12. August 2019 18:35
An: The OpenCms mailing list <opencms-dev at opencms.org<mailto:opencms-dev at opencms.org>>
Betreff: Re: [opencms-dev] OpenCms 9.5.2: Not all files get statically exported with static export after publish

I actually observed something similar at one point, but that was due to an attachment (.doc) having been uploaded with .docx as extension without actually being a .docx.  (or was it the other way around?)

The indexer did not find that amusing, and stopped indexing at random places.

From: <opencms-dev-bounces at opencms.org<mailto:opencms-dev-bounces at opencms.org>> on behalf of Alexander Kandzior <alex at opencms.org<mailto:alex at opencms.org>>
Reply to: The OpenCms mailing list <opencms-dev at opencms.org<mailto:opencms-dev at opencms.org>>
Date: Monday, 12 August 2019 at 16.42
To: The OpenCms mailing list <opencms-dev at opencms.org<mailto:opencms-dev at opencms.org>>
Subject: Re: [opencms-dev] OpenCms 9.5.2: Not all files get statically exported with static export after publish

Kai,

Regarding your static export issue:

There is a scheduled task in OpenCms which does a „full static render“ every evening. This worked perfectly for years. Customer said, that there where no changes on the system OpenCms is running on.

I would bet there have been changes in the system environment configured with OpenCms, regardless of what the customer says.
Static export after publish will request all the URLs of the published pages quickly, so there may be some new firewall settings that „prevent“ a DoS Attack.

Kind regards,
Alex.

-------------------
Alexander Kandzior

Alkacon Software - The OpenCms Experts
http://www.alkacon.com<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.alkacon.com&data=02%7C01%7C%7Cf03713dbd5a24759f45b08d71f333fd3%7C9e4eb25883584a33b3e9b734c884fc4b%7C0%7C0%7C637012177291274510&sdata=ogbT3eS68r3%2FZgGyPkSVEdqxYEzejeHQdOcDVrJlOY0%3D&reserved=0> - http://www.opencms.org<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.opencms.org&data=02%7C01%7C%7Cf03713dbd5a24759f45b08d71f333fd3%7C9e4eb25883584a33b3e9b734c884fc4b%7C0%7C0%7C637012177291284504&sdata=4eWnQggnbgwtAR0ch7qjk41pSLGmgwuoz5Yicc1bGu4%3D&reserved=0>

Am 12.08.2019 um 15:39 schrieb Schliemann, Kai <K.Schliemann at comundus.com<mailto:K.Schliemann at comundus.com>>:

Hi list,

we have a very strange problem in one of our customers installations with the full static export.
This is our setup:

Configuration in opencms-importexport.xml:

…
     <staticexport enabled="true">
     <staticexporthandler>org.opencms.staticexport.CmsOnDemandStaticExportHandler</staticexporthandler>
     <linksubstitutionhandler>org.opencms.staticexport.CmsDefaultLinkSubstitutionHandler</linksubstitutionhandler>
          <exportpath>export</exportpath>
          <exportworkpath>temp</exportworkpath>
          <exportbackups>2</exportbackups>
          <defaultpropertyvalue>false</defaultpropertyvalue>
          <defaultsuffixes>
                …
          </defaultsuffixes>
          <rendersettings>
                <rfs-prefix>${CONTEXT_NAME}/export</rfs-prefix>
                <vfs-prefix>${CONTEXT_NAME}${SERVLET_NAME}</vfs-prefix>
                <userelativelinks>true</userelativelinks>
               <exporturl>http://127.0.0.1:8080${CONTEXT_NAME}/handle404</exporturl><http://127.0.0.1:8080$%7bcontext_name%7d/handle404%3c/exporturl%3e>
                <plainoptimization>true</plainoptimization>
                <testresource uri="/system/shared/page.dtd" />
                <resourcestorender>
                     <regex>/sites/aaa/.*</regex>
                     <regex>/system/galleries/.*/aaa/.*</regex>
                     <regex>/system/galleries/.*/bbb/.*</regex>
                     <regex>/system/modules/de.mymodule.extranet/resources/.*</regex>
                </resourcestorender>
          </rendersettings>
     </staticexport>
…
In OpenCms several folders have set the export property to true and the exportname property to the name of the folder (there are no duplicates).

There is a scheduled task in OpenCms which does a „full static render“ every evening. This worked perfectly for years. Customer said, that there where no changes on the system OpenCms is running on.

The problem:
From one day to the other not all contents are statically exported.  We watched some folders. Some folders are complete, some only contain two files of lets say about 45 files in the VFS. After every export there are two other files in the folder. They seem to be randomly created.
The log throws a 404 for all not exported files.
We checked all configuration files and property settings in the VFS of the folder but could not find any hint on the problem.
The weird thing is, when I request a file in the online project from my browser (a file which had a 404 error after the full static export) it is correctly rendered and written to the export folder. So „on demand static export“ is working correctly.

Last thing to say is, that we have a kind of mirrored system, which has the same settings. Everything works perfect here.
The only difference:
The contents of that system are not the same. As the working system is an integration environment, the contents are older.
So we assume, that the problem must be in the content. As we have thousands of files, finding the one which is the problem seems to be impossible.

So any hint is highly appreciated.

Thanks a lot in advance.

Best regards
Kai

_______________________________________________
This mail is sent to you from the opencms-dev mailing list
To change your list options, or to unsubscribe from the list, please visit
http://lists.opencms.org/cgi-bin/mailman/listinfo/opencms-dev<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.opencms.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fopencms-dev&data=02%7C01%7C%7Cf03713dbd5a24759f45b08d71f333fd3%7C9e4eb25883584a33b3e9b734c884fc4b%7C0%7C0%7C637012177291284504&sdata=Or8jyIBkfj2YiqsKadk8gjxpSJD57MyKUewa88B0%2Bew%3D&reserved=0>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://webmail.opencms.org/pipermail/opencms-dev/attachments/20190827/1b8811bf/attachment.htm>