Comments on: How do I spider a protected area with a Google Mini? http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/ Google Search Appliance and Google Mini development Fri, 21 Nov 2008 12:08:40 +0000 http://wordpress.org/?v=2.5.1 By: Ed http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-3325 Ed Wed, 17 Oct 2007 15:50:26 +0000 http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-3325 I want the password-protected results to show up in the results, with an icon showing it is Members Only contetn. Is this possible with the Google Mini. thanks I want the password-protected results to show up in the results, with an icon showing it is Members Only contetn. Is this possible with the Google Mini.

thanks

]]> By: shauna durham http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-1767 shauna durham Thu, 05 Jul 2007 20:39:43 +0000 http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-1767 I have set up the username and password for the crawl and it is working however because it is logging into our login page instead of the name of the page showing, which is what I want, it is showing "Member login..." for all the protected areas of our site. Not very user friendly. Does anyone know how to fix this problem? I have set up the username and password for the crawl and it is working however because it is logging into our login page instead of the name of the page showing, which is what I want, it is showing “Member login…” for all the protected areas of our site. Not very user friendly. Does anyone know how to fix this problem?

]]> By: Fatimah http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-1322 Fatimah Thu, 24 May 2007 14:18:17 +0000 http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-1322 Did anyone fine a solution to this? Did anyone fine a solution to this?

]]> By: Paul http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-260 Paul Wed, 20 Dec 2006 17:49:38 +0000 http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-260 When you put the password in, it goes in to a 'password' type field, so it's shown as *s. Once the password is stored, it is not shown in the form again (i.e. when checking the settings or editing them) so you can't get it out that way. The username is shown. There's no way of getting to the information on the hard disk unless you take the Mini apart and plug the hard disk in to another system, so I have no idea how it stores the passwords. Depending on the authentication system, it may need to send the passwords to the site as clear text, as it won't be able to send them encrypted if the site is expecting to receive them unencrypted. So if it does store the passwords encrypted, it will not be in a secure one-way system. When you put the password in, it goes in to a ‘password’ type field, so it’s shown as *s. Once the password is stored, it is not shown in the form again (i.e. when checking the settings or editing them) so you can’t get it out that way. The username is shown.

There’s no way of getting to the information on the hard disk unless you take the Mini apart and plug the hard disk in to another system, so I have no idea how it stores the passwords.

Depending on the authentication system, it may need to send the passwords to the site as clear text, as it won’t be able to send them encrypted if the site is expecting to receive them unencrypted. So if it does store the passwords encrypted, it will not be in a secure one-way system.

]]> By: Anon http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-259 Anon Wed, 20 Dec 2006 17:25:29 +0000 http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-259 By entering in user/password for the crawler to use, is the password encrypted in anyway when stored? By entering in user/password for the crawler to use, is the password encrypted in anyway when stored?

]]> By: Jason Divis http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-135 Jason Divis Wed, 23 Aug 2006 17:41:57 +0000 http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-135 Thanks for pointing me in the right direction. If I can get to the header before you, I'll have to post my solution here. Thanks for pointing me in the right direction. If I can get to the header before you, I’ll have to post my solution here.

]]> By: Paul http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-134 Paul Wed, 23 Aug 2006 16:06:27 +0000 http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-134 I think the process I've listed only works with standard user/password combinations as set up by the web server (e.g. Apache or IIS). To get through your security, I think you'll need to look at the HTTP Headers bit of 'Crawl and Index' (Mini v2) or 'Configure Crawl' -> 'Crawler Parameters' (in Mini v1). You can use this to either send a cookie through with the spider, or some special bit of information which matches some code on your pages and will let it in without needing the standard cookie set when people login. I do need to look at this some more and write up an example, I just haven't had time recently as I've been busy on client work. I think the process I’ve listed only works with standard user/password combinations as set up by the web server (e.g. Apache or IIS). To get through your security, I think you’ll need to look at the HTTP Headers bit of ‘Crawl and Index’ (Mini v2) or ‘Configure Crawl’ -> ‘Crawler Parameters’ (in Mini v1). You can use this to either send a cookie through with the spider, or some special bit of information which matches some code on your pages and will let it in without needing the standard cookie set when people login.

I do need to look at this some more and write up an example, I just haven’t had time recently as I’ve been busy on client work.

]]> By: Jason Divis http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-133 Jason Divis Wed, 23 Aug 2006 15:28:35 +0000 http://www.gsadeveloper.com/2006/01/30/how-do-i-spider-a-protected-area-with-a-google-mini/#comment-133 Is this the same process that I would go through if one has to login to our site (ASP based)? I have specified the user name and password that I use to login to our site, and the pattern of the URL to where our login page is, but it doesn't look like Mini is able to gain access to our secure site. It only crawls the 10 pages on the non-secure side, and is redirected when it tries to crawl the secure pages... Is this the same process that I would go through if one has to login to our site (ASP based)? I have specified the user name and password that I use to login to our site, and the pattern of the URL to where our login page is, but it doesn’t look like Mini is able to gain access to our secure site. It only crawls the 10 pages on the non-secure side, and is redirected when it tries to crawl the secure pages…

]]>