Crawling Your Site

Insty-Site contains a search engine which allows your visitors to search for content on your website. Before the search engine will work, it is necessary to crawl your website on a regular basis.

Crawler Start
Enter the complete URL to your homepage. The crawler will start at this point.

Proper Domain
Enter the proper domain for your website. This is needed to determine what links are local and what links are not.

The Crawler
When you click "Crawl Now," a new window will open and the crawler will begin indexing your website. This takes several minutes. You don't need to do anything here. Just wait while the crawler works. Once the crawl is complete, you will be instructed to close the window.

Important Notes!

1. The crawler is server intensive and should not be done more often than necessary or during peak traffic hours.
2. The crawler will execute page hits all over the website and may appear in any traffic reporting you may have.
3. You may turn off the CMS traffic reporting in Settings & Options so that the crawler will not be reflected in it.
4. The crawler views web pages from the perspective of a guest user (not logged in), so member-only content will not be indexed.
5. Links that contain the attribute rel="nofollow" will be ignored by the crawler. This is critical for some pages like the calendar where an indefinite number of pages can be created.

See also:
Search Log

Back to Table of Contents