[opencms-dev] Sort on NavPos using getNavigationForFolder or using SOLR and <cms:search>

[Hein van der Kallen]

I want to share the results of the question: can I efficiently replace 

retrieving data using getNavigationForFolder from the Vfs with 

retrieving the same data using <cms:search> from the SOLR index. 

The main result is a deeper understanding what seems NOT possible with
<cms:search>.

Background: 

At the moment for all our generated navigations (menu's) and lists
(news-lists) we use jsp's in the elements subdirectory that use the method
getNavigationForFolder(String folder) in CmsJspNavBuilder. ( the sitemap and
<cms:navigation> use the same foundation). 

The results of the element-jsp's with getNavigationForFolder  are included
in our pages using 

<cms:include file="<my element-jsp>" >
  <cms:param name="folder" value="<my folder>" /> 
</cms:include>
 
getNavigationForFolder roughly selects all direct children of the folder
from the vfs that have a NavPos property, sorted on the value of that NavPos
property.
The properties NavInfo and NavText can play a secondary role (For background
see
<https://documentation.opencms.org/opencms-documentation/creating-a-website/
building-a-page-navigation/changing-the-navigational-behavior-of-pages/>
https://documentation.opencms.org/opencms-documentation/creating-a-website/b
uilding-a-page-navigation/changing-the-navigational-behavior-of-pages/ ) but
to keep it clear I will ignore that here. 
 
Sometimes we use getNavigationForFolder nested: we repeat
getNavigationForFolder for found subfolders. 
 
Sometimes in a navigation we need the direct child index.html as the Home of
the navigation. 
And have to correct for the fact that  getNavigationForFolder does not
return the direct child index.html in the folder if that file has no
(inherited) NavPos. 
 
The question: 
The modern way to create lists is to use a search based on a SOLR query.
That could be an efficient alternative. 
So Is a search on SOLR suitable to replace some or all of our usage of
getNavigationForFolder?
 
I don't want to compile java code, so I will have to work with what is
loaded in the solr index using 
<fieldconfiguration
class="org.opencms.search.solr.CmsSolrFieldConfiguration">

I would be willing to change the SOLR schema, but found no way where that
helps. 

I would be willing to change the opencms-search.xml, but found no way where
that helps. 

 
As I found out, that gives some serious limitations:
 
1.
Most resourcetypes are loaded in SOLR with their content and properties,
based on the documenttypes in opencms-search.xml.
But no folders are loaded and htmlredirect is also not loaded. 
So when I need the folders and their properties, I can only get them where I
followed the convention to add an index.html file to every folder (of a
resourcetype indexed in SOLR), without direct properties in the index.html
file. 
 
2. 
In SOLR you can search all descendants of a given folder, but I found no way
to limit the depth of the results. 
If for instance you need a two level menu, you don't want to be overloaded
with results for the third level. 
 
3. The NavPos property is a Double, but is loaded in SOLR as a string.
So sorting the results in SOLR on the NavPos gives a wrong ordering (even
for the direct children of the same folder). 
Sorting results in the NavPos way is tricky anyhow. Because the ordering on
NavPos should only be applied on the resources in the same folder. 
And the NavPos is not guaranteed to be unique, even within a folder. 
 
As an exercise I tried to make a rather generic solution for replacing a
(nested) getNavigationForFolder in 3 steps
 
a.     Perform a search using <cms:search>
b.     Store the results for each folder in a map sorted on the NavPos
c.      Hierarchically present the results starting from the first folder. 
 
( the code is appended at the end of this post)
And yes that works within the limitations.
But the limitations are too severe. 
 
There is a problem with sorting on NavPos anyhow:  
Providing a NavPos was in the old workplace always supported with a special
(new) property dialog with "insert after" for ALL resources. 
But in the new workplace it is supported only for folders in the sitemap. 
So when we want to change to the new workplace we at least have to abandon
selecting and sorting news-lists on NavPos.
 
Conclusion: 
For generated navigations we can better keep what we have. 
For generated news-lists we can use <cms:search> but will have to provide a
newsDate field and sort the results on that.   
 
The code: 
 
<%@ page
    buffer="none"
    session="false"
    trimDirectiveWhitespaces="true"%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %>
<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions"%>
<%@ taglib prefix="fmt" uri="http://java.sun.com/jsp/jstl/fmt"%>
<%@ taglib prefix="cms" uri="http://www.opencms.org/taglib/cms"%>
 
<%@ page import="java.util.* , org.opencms.util.CmsStringUtil,
org.opencms.jsp.search.result.CmsSearchResourceBean" %>
 
<%-- a.  Perform a search using <cms:search> -->
 
<%-- Get the path under which the resources should be searched (including
subpathes) --%>
<c:set var="folder">${param.folder}</c:set>
 
 
<%-- add site root and trailing slash --%>
<c:set var = "basePath">${cms.requestContext.siteRoot}${folder}/</c:set>
 
<%-- filter on descendants --%>
<c:set var="filter1">fq=parent-folders:\"${basePath}\"</c:set>
 
<%-- filter on direct navPos  not empty or index.html and (inherited navPos
not empty or direct child of requested folder)--%>
<c:set
var="filter2">fq=((NavPos_dprop_s:*)OR((filename:\"index.html\")AND((NavPos_
prop:*)OR(path:\"${basePath}\index.html\"))))</c:set>
 
<%-- fields to query --%>
<c:set var="fieldList"
>fl=id,path,filename,type,link,Title_dprop_s,Description_dprop_s,NavText_dpr
op_s,NavPos_dprop_s,Title_prop_s,Description_prop_s,NavText_prop_s,NavPos_pr
op_s</c:set>
 
<%-- Create the configuration for <cms:search> --%>
<c:set var="searchconfig">
  {
  "ignorequery" : true,
  "extrasolrparams" : "${filter1}&${filter2}&${fieldlist}",
  "pagesize" : 50
  }  
</c:set>
 
<%-- Perform the search, the result object is stored in the variable
"search" --%>
<cms:search configString="${searchconfig}" var="search"
      addContentInfo="true" />
 
<c:set var="results" value= "${search.searchResults}"/>
 
<%-- switch to servlet style   --%>
 
<%-- b.  Store the results for each folder in a map sorted on the NavPos
--%>
 
<%
Collection<CmsSearchResourceBean> results =
(Collection<CmsSearchResourceBean>) pageContext.getAttribute("results");
 
// create a map that for each found folder stores the direct children in a
treemap that sorts the results on the NavPos property 
TreeMap<String,TreeMap<Double,CmsSearchResourceBean>> resultsMapped = new
TreeMap<String,TreeMap<Double,CmsSearchResourceBean>>();
// walk through the results
  Iterator rI = results.iterator(); 
   while (rI.hasNext()) {
   CmsSearchResourceBean result = (CmsSearchResourceBean) rI.next();
   Map<String,String> fields = result.getFields();
   String path = fields.get("path");
   String filename = fields.get("filename");
   String folder = path;
  
   // treat results with filename index.html as the folder above it
   if (filename.equals("index.html"))
   {folder = folder.substring(0,folder.lastIndexOf("/") ) ;}
   
   // get folder above it , with a trailing "/"
   folder = folder.substring(0,folder.lastIndexOf("/")+ 1 ) ;
   
   String NavPos ;
   if (filename.equals("index.html")) {
     NavPos = fields.get("NavPos_prop_s");
  if (NavPos == null) {NavPos = "1.0";}
  }
   else {
       NavPos = fields.get("NavPos_dprop_s");
  }
   Double key = Double.parseDouble(NavPos);
   // store folder in the resultsMapped
   TreeMap<Double,CmsSearchResourceBean> innerMap;
   if (!(resultsMapped.containsKey(folder))) {       
       resultsMapped.put (folder, new
TreeMap<Double,CmsSearchResourceBean>());
   }
  innerMap = resultsMapped.get(folder);
  // force the key to be unique, without disturbing the logical ordering in
the folder
     while (innerMap.containsKey(key)) { 
    key = key *1.000000001 ;
  } 
  //store key and result in innerMap   
  innerMap.put(key, result);
  }
    
// switch back to EL style
 
%>
<c:set var="resultsMapped" value="<%= resultsMapped %>" />
 
<%-- c.  Hierarchically present the results starting from the first folder.
--%>
 
<p> initialize output </p>
<p> intialize folder level 1 </p>
<c:set var="outerEntry1" value="${resultsMapped.firstEntry()}" />    
  <c:forEach var="innerEntry1" items="${outerEntry1.value}">
    <c:set var= "result1" value = "${innerEntry1.value}" />
    <p> show output for result with path ${result1.fields.path} </p>
    <%-- check for level 2 --%>
    <c:if test ="${result1.fields.filename.equals('index.html')}" >     
      <c:set var= "path1" value = "${result1.fields.path}" />
      <c:set var= "folder2" value =
"${path1.substring(0,path1.lastIndexOf('/')+ 1)}" />
      <c:if test = "${resultsMapped.containsKey(folder2)}" > 
        <p> intialize folder level 2 </p>
        <c:forEach var="innerEntry2" items="${resultsMapped.get(folder2)}">
            <c:set var= "result2" value = "${innerEntry2.value}" />
          <p> show output for result with path ${result2.fields.path} </p>
          <%-- check for level 3 --%>
          <c:if test ="${result2.fields.filename.equals('index.html')}" >

            <c:set var= "path2" value = "${result2.fields.path}" />
            <c:set var= "folder3" value =
"${path2.substring(0,path2.lastIndexOf('/')+ 1)}" />      
            <c:if test = "${resultsMapped.containsKey(folder3)}" > 
              <p> intialize folder level 3 </p>
              <c:forEach var="innerEntry3"
items="${resultsMapped.get(folder3)}">
                <c:set var= "result3" value = "${innerEntry3.value}" />
                <p> show output for result with path ${result3.fields.path}
</p>
                <%--  if wanted, check for level 4 etc --%>
              </c:forEach>
              <p> finalize folder level 3 </p>
            </c:if>
          </c:if>
        </c:forEach>
        <p> finalize folder level 2 </p>
      </c:if>
    </c:if>
  </c:forEach>  
 
<p> finalize folder level 1 </p>
<p> finalize output </p>

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.opencms.org/pipermail/opencms-dev/attachments/20210519/d662832a/attachment.htm>