Configuring mySpider's Spider Engine

Using the configure spider tool (see the screen shot) you can:
  • Set the domains you want spidered.
  • Set the URLs of the entry points in those domains.
  • Set the file extensions you don't want spidered.
  • Set the files you don't want spidered by either the full URL, the full file name or a common string.
  • Set the URLs and respective usernames and passwords to enable the spidering of protected areas of the site.

Configuring mySpider's Search Engine

Using the configure search tool (see the screen shot) you can:
  • Set the values of the parameters in the ranking algorithm to change the ranking or scoring of search results.
  • Set the names and URL patterns of the categories or sections of the site to allow searching within one or more categories.
  • Set the number of results to display per page.
  • Set the maximum number of chunks of text or body excerpts returned in each results set.
  • Set the maximum number of characters in each chunk or excerpt.
  • Set the target frame or window the linked titles will open in.
  • Customize the look of the search page by using the "myspider_style.css" file to adjust the width of the results sets and the font colors and sizes. Mouseover different areas of the search results example page to see how to make changes.

And We Can Do More Customization ...

We can further customize for your specific web site's code or for your specific needs by changing or adding to the source code of the mySpider search and spider tools.

Example 1: Your site doesn't have a site map page and pages are linked primarily by Dynamic HTML navigation menus. We can integrate your specific DHTML menu code into the Spider Engine code.

Example 2: Your site has links where the URL's query string calls a function that dynamically creates the link's real URL. Spiders cannot capture and then visit these dynamic links. However, we can code a work-around in the Spider Engine.