[opencms-dev] getting recognized by robots and crawlers

[Joe Desbonnet]

Normally search engines will spider the content of the site unless
forbidden by the robots.txt file or the page header.

It is important however to have links on the home page. A Flash splash
page without any links is going to be a dead end for search engines.

If the site is public, can you send me the URL and I'll check if there
is anything wrong.

Joe.


On 10/21/05, Christoph P. Kukulies <kuku at physik.rwth-aachen.de> wrote:
> One reason why I'm doing all this "getting rid" stuff now is, that
> my customer wants his site to be better recognized by robots.
>
> At the moment the site is practically not visited.
>
> I see google and other robots visiting the site, reading robots.txt and
> after that nothing happens.
>
> When I do a wget -r on the site, just the index.html is fetched,
> that's all. The content seems to be totally hidden to the outer world.
>
> What would be the way to proceed from here? What I want to achieve is,
> that all relevant paths are traversed by robots and documents (PDF)
> are also searched through. (opencms search module, htdig? that other thing
> with 'L', forgot the name for the moment).
>
> Another point is, and that's probably another reason why the site
> is practically not mentioned in search results - I told that to the
> customer's admins already - is that there is no RR PTR record (reverse lookup)
> in their DNS server. I assume that search engines leave fingers off of sites
> that do not reverse map (often sites of questionable contents, mildly
> speaking :)
>
> I took care that they added one now but it still is not migrated downstream
> to other DNS servers.
>
>
> Help appreciated.
>
> Thank you.
>
>
> --
> Chris Christoph P. U. Kukulies kukulies (at) rwth-aachen.de
>
>
> _______________________________________________
> This mail is send to you from the opencms-dev mailing list
> To change your list options, or to unsubscribe from the list, please visit
> http://mail.opencms.org/mailman/listinfo/opencms-dev
>