Comments on: Avoiding session IDs when spidering http://www.gsadeveloper.com/2006/04/20/avoiding-session-ids-when-spidering/ Google Search Appliance and Google Mini development Mon, 08 Sep 2008 08:11:53 +0000 http://wordpress.org/?v=2.5.1 By: Paul http://www.gsadeveloper.com/2006/04/20/avoiding-session-ids-when-spidering/#comment-510 Paul Thu, 15 Feb 2007 11:07:04 +0000 http://www.gsadeveloper.com/2006/04/20/avoiding-session-ids-when-spidering/#comment-510 If you don't exclude the session ID, it will spider the pages. I suggest you set the host load to be 1, which might help it keep to the same session ID as it effectively only sends one connection to spider the site rather than 4 (the default host load.) You can't get it to ignore part of the URL, it only understands inclusion and exclusion based on parameters. If you have to have a session ID, then you may need to look at changing the CMS or whatever runs the site so it will always feed the same session ID to the Mini when it is spidering. Basically, spiders hate session IDs, so if you have to have one, you're always going to be in a bit of trouble. Hmm... you could set up a page of links to every page on your site, all with a session ID appended to them, then set that page as the place the Mini starts spidering. That might then allow it access to the whole site without getting too confused. Unfortunately this is just guesswork from me, usually I deal with people who can get rid of the sessions, or places where we exclude them entirely. If you don’t exclude the session ID, it will spider the pages. I suggest you set the host load to be 1, which might help it keep to the same session ID as it effectively only sends one connection to spider the site rather than 4 (the default host load.)

You can’t get it to ignore part of the URL, it only understands inclusion and exclusion based on parameters.

If you have to have a session ID, then you may need to look at changing the CMS or whatever runs the site so it will always feed the same session ID to the Mini when it is spidering.

Basically, spiders hate session IDs, so if you have to have one, you’re always going to be in a bit of trouble. Hmm… you could set up a page of links to every page on your site, all with a session ID appended to them, then set that page as the place the Mini starts spidering. That might then allow it access to the whole site without getting too confused. Unfortunately this is just guesswork from me, usually I deal with people who can get rid of the sessions, or places where we exclude them entirely.

]]> By: Ty C. http://www.gsadeveloper.com/2006/04/20/avoiding-session-ids-when-spidering/#comment-509 Ty C. Wed, 14 Feb 2007 23:37:55 +0000 http://www.gsadeveloper.com/2006/04/20/avoiding-session-ids-when-spidering/#comment-509 What if the website requires the session ID? Is there a way to tell GSA to ignore a specific querystring parameter but still index the page? Otherwise the entire site will be ignored, won't it? What if the website requires the session ID? Is there a way to tell GSA to ignore a specific querystring parameter but still index the page? Otherwise the entire site will be ignored, won’t it?

]]>