next »

[netmktg]

I was just looking at the Blurb of serpscraper.com and was intrigued by the mention of a BH secret that "cannot be entirely reviled" which works around the Google IP-ban. I'm sure software by Earl cannot be "Reviled" but I was hoping this dark secret can be "Revealed" by the gurus@Perkiset

[vsloathe]

There are only a few factors.

For one, you need a large word list. For another, you need a number of common useragents. Your queries should be as unique as possible to avoid banning.

Here's a piece of code to get your gears turning:

Code:

<?php
class scrapeGoogle{
private $arrResults = array();
private $arrWordlist = array();
private $googleURL = 'http://www.google.com/search?hl=en&q=[QUERY]&start=[START]&num=100&btnG=Google+Search';
public function __construct(){
$this->arrWordlist = file('data/wordlist.txt');
foreach($this->arrWordlist as $key => $word){
$this->arrWordlist[$key] = trim($word);
}
}
public function scrape($term, $page = 1){
$keyword = $this->arrWordlist[array_rand($this->arrWordlist)]
." "
.$this->arrWordlist[array_rand($this->arrWordlist)]
." "
.$this->arrWordlist[array_rand($this->arrWordlist)]
." "
.$this->arrWordlist[array_rand($this->arrWordlist)]
." "
.$this->arrWordlist[array_rand($this->arrWordlist)];
$query = urlencode($term.' '.$keyword);
$scrapeURL = str_replace('[QUERY]', $query, $this->googleURL);
$scrapeURL = str_replace('[START]',(($page - 1) * 100) ,$scrapeURL);
$ch = curl_init();
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1)Gecko/20061010 Firefox/2.0');
curl_setopt($ch, CURLOPT_URL, $scrapeURL);
$response = curl_exec($ch);
while(strstr($response, '403 Forbidden')){
sleep(rand(1,60));
$response = curl_exec($ch);
}
curl_close($ch);
$cleanHTML = tidy_parse_string($response);
tidy_clean_repair($cleanHTML);
$DOMDoc = new DOMDocument();
@$DOMDoc->loadHTML($cleanHTML);
@$nodelist = $DOMDoc->getElementsByTagName('a');
foreach($nodelist as $node){
if($node->getAttribute('class') == 'l'){
if(!array_keys($this->arrResults, $node->getAttribute('href'))){
echo $this->arrResults[] = $node->getAttribute('href');
echo "\n";
flush();
ob_flush();
}
}
}
}
}
?>

It requires PHP5 (PHP DOM) and the Tidy extensions to work properly.

[dink]

Hey netm . . .  you could also try your scraper on AOL search.  Or, code the scraper to pull requests for different G datacenters on each run.

[DangerMouse]

I guess it depends on the purpose of the scrape - rank checking versus target acquisition.

I've heard that sending multiple requests in the same packet a la the HTTP 1.1 specification works - pipelining. I've not really tried it enough to confirm though.

DM

[Bompa]

I guess it depends on the purpose of the scrape - rank checking versus target acquisition.

I've heard that sending multiple requests in the same packet a la the HTTP 1.1 specification works - pipelining. I've not really tried it enough to confirm though.

DM

I spent a lot of time researching this.

Multiple requests with one tcp connection; the Keep-Alive directive of Apache.

Anyways, it does work, in fact, browsers already use the feature. 

You can see the:

Connection: Keep-Alive
Keep-Alive: 300

with liveheaders.  The 300 is seconds, not requests heh.

And Apache has this feature enabled by default.



But there are several problems:

1. Google does not use the out of the box Apache, LOL.  Their version is heavily modified and renamed: GWS.

2. Apache's default setting is a maxiimum of five requests per tcp connect which sorta defeats
the whole purpose, but what do I know.

3.  Any savvy admin can lower the max requests or simply disable the Keep-Alive altogether.



thank you and come again,
Bomps

[nutballs]

i am so glad I stocked up on API keys a long time ago. LOL

[mampy]

LOL....

Does anyone have any knowledge on the availability of the MAC address with each packet received from apache
and could this be a method of banning?

[vsloathe]

Apache can't specify packets on that low a level. The first few bytes of an IP header is what you're looking for, the OS controls everything that low. Apache doesn't even handle the TCP handshake, just the HTTP serving.

I think, anyway.

[perkiset]

umm... dunno that VS... Apache is responsible to open and handle the socket, only the the event is vectored to it by the OS... I think 

[vsloathe]

Yes I think you're correct perks. I was thinking of it too much from a client perspective.