ALeadSoftbot is ALeadSoft's web-crawling robot. It collects
documents from the web to build a searchable index to build
site search engine. On this page, you'll find answers to the
most commonly asked questions about how our web crawler
works.
1. Why is
ALeadSoftbot asking for a file named "robots.txt" on my
server?
"robots.txt"
is a standard document that can tell search engine robots not to
index some or all information from your web server. For
information on how to create a robots.txt file, see
The Robot Exclusion Standard.
If there is no "robots.txt" file on your server and you just
want to prevent the 404 "file not found" error messages in
your web server log, you can create an empty file named "robots.txt"
and upload to your server.
Example: If you want to disallow all search engine robots to
index the path "cgi-bin", you can write down the following "robots.txt":
User-agent: *
Disallow: /cgi-bin/
If you only want to disallow search engine builder to index
the path "cgi-bin", you can write down the following "robots.txt":
User-agent: ALeadSoftbot
Disallow: /cgi-bin/
2. What
kinds of links does ALeadSoftbot follow?
It
follows HREF links and SRC links.
3. How do
I prevent ALeadSoftbot from indexing or following links on my
pages?
To
keep search engine robots from indexing text or following links on your
pages to other pages or documents, you'd place the following
meta tag in the head of your HTML document:
<meta name="robots" content="noindex,nofollow">
To learn more about meta tags, please refer to
The Robots META tag.
Remember, changes to your site won't be immediately
reflected in ALeadSoft; they'll be discovered and propagate
when ALeadSoftbot next crawls your site.
