[opencms-dev] Problem displaying non ISO-8859-1 charsets

[Dimitri Bougoulias]

Hello,

In my previous mail to the list 
(http://mail.opencms.org/pipermail/opencms-dev/2003q2/004231.html)
I mentioned a problem (and potential workaround) for using non ISO-8859-1
(in my case ISO-8859-7 Greek).
In this mail I include the (updated) patch I'm currently using - applies to
current CVS as of April 28th.

This patch makes OpenCMS usable for Greek (and other charsets).

My setup is on windows, linux and HPUX with mysql 3.x and J2SE-1.4.1

*** For OpenCms to become usable for other languages at least content, title
and nav-text should allow non ISO-8859-7 characters.

Changed files:
a) mostly added missing Content Encoding Meta
opencms/etc/ocsetup/vfs/system/workplace/administration/project/currentproje
cts/administration_content_delete_project
opencms/etc/ocsetup/vfs/system/workplace/administration/project/currentproje
cts/administration_content_lockchange
opencms/etc/ocsetup/vfs/system/workplace/administration/project/currentproje
cts/administration_content_projectresent
opencms/etc/ocsetup/vfs/system/workplace/administration/project/currentproje
cts/administration_content_publish_project
opencms/etc/ocsetup/vfs/system/workplace/administration/project/historyproje
cts/administration_content_projecthistory
opencms/etc/ocsetup/vfs/system/workplace/administration/project/newproject/a
dministration_content_projectnew
opencms/etc/ocsetup/vfs/system/workplace/administration/user/groups/administ
ration_content_groups
opencms/etc/ocsetup/vfs/system/workplace/administration/user/usermessages/ad
ministration_content_usermessages
opencms/etc/ocsetup/vfs/system/workplace/administration/user/users/administr
ation_content_users
opencms/etc/ocsetup/vfs/system/workplace/templates/explorer_files_new_folder

b) Eliminated javascript encodeURIcomponent stuff. since it causes some
problems
opencms/etc/ocsetup/vfs/system/workplace/templates/chnav 
opencms/src/com/opencms/workplace/CmsChnav.java

c) modified so that escapeHtml and escapeXml do not encode already encoded
strings
opencms/src/com/opencms/util/Encoder.java

d) escapeHtml for nicenames[] since it would not update NAVPOS properly if
navigation text was non ISO-8859-7
opencms/src/com/opencms/workplace/CmsNewResourceFolder.java 
opencms/src/com/opencms/workplace/CmsNewResourceLink.java 
opencms/src/com/opencms/workplace/CmsNewResourcePage.java
For the above files there is still problems with encodeURIcomponent (see
below)

e)  Added ISO-8859-7 and other charsets  that include ASCII charset to
content-encoding directive so that it can be usable with other charsets.
(International English version) 
opencms/src/com/opencms/workplace/workplace.properties
opencms/src/com/opencms/workplace/workplace_en.properties
I believe this is correct approach since the english version should be able
to edit non english content as it applies to most software apps.

Still there are problems with username etc. in non ISO-8859-1 - especially
for the username (first, last name, etc.) the size of db fields should be
increased so that it can accommodate URLencoded strings. Etc.

In part (d) above -> CmsNewResourceXXX.java there is still a problem with
characters 212,213,222 (integer values of Greek capital Tau, capital Ipsilon
and lowercase accented eta) since they turn into double questionmarks. This
is coused somehow by the encodeURIcomponent Javascript function. Had the
same problem in CmsChnav (Change navigation) but I solved it through getting
rid of javascript. 

The good part is that even if the strings (title, navigation text) get
corrupt I can fix using the property tab which works OK after my patch.

The funny part is that when I use the same javascript function in another
web application within tomcat but outside OpenCms there is no problem with
the three characters I mentioned above.

To fully solve charset problems some overhaul must probably be performed in
CmsRequestBroker and in the com.opencms.file subpackage.

I have not seen any CVS changes since April 3, when is version 5 going to be
released???

Best regards,

Dimitri




-------------- next part --------------
A non-text attachment was scrubbed...
Name: opencms.patch
Type: application/octet-stream
Size: 20253 bytes
Desc: not available
URL: <https://webmail.opencms.org/pipermail/opencms-dev/attachments/20030429/39aa13dd/attachment.obj>