[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Re: HGRAB. Syndication. Google. Grey area.

• To: "K. Ari Krupnikov" <ari@c...>
• Subject: Re: HGRAB. Syndication. Google. Grey area.
• From: Paul T <pault12@p...>
• Date: Tue, 08 Jan 2002 15:28:09 -0800
• Cc: xml-dev@l...
• References: <02e301c19889$99cf3680$2cd0a340@paultx><3C3B7134.D418F69B@c...>

> > So, what HGRAB actually does? It polls
> > the HTML pages ( once in a while,
> > no harm done to the load of original
> > website ). Then it places some part of the
> > content into HGRAB database (for future
> > searching). Then it provides the end-user
> > with some 'part' of the original news item
> > and with the URL to the original news source.
> > 
> > Google does *exactly* this (and also Google
> > provides a cached copy of the original content)
> > 
> > That means:
> > 
> > Either both HGRAB and Google should be sued,
> > because they both sell the content
> > *which does not belong to them*, or both
> > HGRAB and Google should be considered
> > 'just a service'.
> 
> 
> Have a look at http://www.google.com/robots.txt

I don't understand your point. Could you pelase 
explain?

So, some websites may have robots.txt, 
saying that some pages should not be 
indexed by robots.

Because HGRAB, for example, is 
usually polling only home page of the website, 
they are all allowed for polling.

Also, I'm not sure if search engines do 
really care about the robot.txt, but that's another 
story.

Also, the interesting twist is that when the 
robot encounters the website with *no*
robots.txt ( most of the sites have no robots.txt ) 
the robot assumes that it is *safe* for him to 
'steal' the content.

Shouldn't it be 'if there is no robots.txt - 
get out of here'?

I think this is really gray area and 
robots.txt is not a solution. 
At the moment, at least.

What I'm missing?

Rgds.Paul.

• Follow-Ups:
  • OT: web crawling (was: Re: HGRAB. Syndication. Google. Grey area.)
    • From: "K. Ari Krupnikov" <ari@c...>

• References:
  • HGRAB. Syndication. Google. Grey area.
    • From: Paul T <pault12@p...>
  • Re: HGRAB. Syndication. Google. Grey area.
    • From: "K. Ari Krupnikov" <ari@c...>

• Prev by Date: Re: [ANN] XML Limerick Competition
• Next by Date: Re: [ANN] XML Limerick Competition
• Previous by thread: Re: HGRAB. Syndication. Google. Grey area.
• Next by thread: OT: web crawling (was: Re: HGRAB. Syndication. Google. Grey area.)
• Index(es):
  • Date
  • Thread