The Cache: Technology Expert's Forum
 
*
Welcome, Guest. Please login or register. September 22, 2019, 11:55:04 PM

Login with username, password and session length


Pages: [1]
  Print  
Author Topic: Detecting proxies requiring authentication  (Read 4045 times)
Pidea
n00b
*
Offline Offline

Posts: 6


View Profile
« on: July 26, 2007, 02:05:35 PM »

First post here . . .

I've developed a proxy checking and rating system that basically hits a known URL directly and then via the proxy that I'm testing.  I time both so can work out how much slower the proxy is compared to direct access and this is all stored in a database along with the overall reliability (number of tests passed in the last 24 hours).

The problem I have is sometimes a proxy that I have previously tested causes my script to completely hang.  When I access the proxy directly it's because the admin has introduced authentication.  Obviously I want to detect that the proxy now requires authentication so that I can mark it as inactive but how can I do this ?

Here's the code that I'm using to use the proxy:

Code:
function useProxy($url, $proxy, $port) {
// Function to retrieve page content using the CURL library compiled into PHP

// Build proxy string
$proxyAddress=$proxy . ":" . $port;

$ch = curl_init ();
// We're  going to use a proxy
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 0); // Note 0=on, 1=off

// Specify proxy location
curl_setopt($ch, CURLOPT_PROXY, $proxyAddress);

// Specify that we want output rather than report on success or failure
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);

// Define URL to get
curl_setopt ($ch, CURLOPT_URL, $url);

// Follow any headers that redirect us
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);

// Set timeout in case we don't get anything back
curl_setopt ($ch, CURLOPT_TIMEOUT, 30);

// Masquerade as IE6 on Windows XP :-)
curl_setopt ($ch, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 1.0.3705)');

// Go and get it !
$content = curl_exec ($ch);
curl_close ($ch);

return $content;
}

Although I'm fairly au-fait with PHP, this bloody cURL stuff is baffling !  Is there an option that I should set to catch the authentication request ?  Because the script hangs $content doesn't return any value so I can't scan it looking for any tell tale clues ?

Or should I dump cURL and use something else ?  Thanks in advance !
Logged

No links in signatures please
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #1 on: July 26, 2007, 02:13:51 PM »

So the timeout has no effect, because the problem is not a hang, but there is an auth request? Is that the issue?

Also, have you looked at CURLOPT_HTTPAUTH and such  - although I have not used that stuff, I would assume that an "invalid auth" response would at least handle the hang...
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
Pidea
n00b
*
Offline Offline

Posts: 6


View Profile
« Reply #2 on: July 26, 2007, 02:21:34 PM »

Quote
So the timeout has no effect, because the problem is not a hang, but there is an auth request? Is that the issue?

Absolutely right.

Quote
Also, have you looked at CURLOPT_HTTPAUTH and such  - although I have not used that stuff, I would assume that an "invalid auth" response would at least handle the hang...

From what I can work out CURLOPT_HTTPAUTH is used to specify a username and password to use.  I've not figured out how catch the response.
Logged

No links in signatures please
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #3 on: July 26, 2007, 02:55:19 PM »

I guess what I'm suggesting, is if you send an invalid username and password, can you get it to bomb - so that the response would at least give you a failure or something?

Other than that, I have not seen what an auth request looks like in HTTP... what you might try, is to ask for headers only and see what you get - I am guessing that passing your initial request but seeing an auth request (or whatever it is) in the headers would let you know that you can't go any further. I'm personally in uncharted waters, but that's the angle I'd take to pull the challenge apart.
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
Bompa
Administrator
Lifer
*****
Offline Offline

Posts: 564


Where does this show?


View Profile
« Reply #4 on: July 26, 2007, 11:04:08 PM »

I'm suprised that you don't get a 401 status code.



Bompa
Logged

"The most beautiful and profound emotion we can experience is the sensation of the mystical..." - Albert Einstein
Pidea
n00b
*
Offline Offline

Posts: 6


View Profile
« Reply #5 on: July 27, 2007, 01:57:01 AM »

Quote
I guess what I'm suggesting, is if you send an invalid username and password, can you get it to bomb - so that the response would at least give you a failure or something?

Good idea, I'll try that now

Quote
I'm suprised that you don't get a 401 status code.

What I should get is a 407 and if I switch on LiveHeaders in Firefox that's what I see.  I'll try perkiset suggestion and will let you know how I get on.

Thanks guys
Logged

No links in signatures please
Bompa
Administrator
Lifer
*****
Offline Offline

Posts: 564


Where does this show?


View Profile
« Reply #6 on: July 27, 2007, 11:17:03 PM »

Quote
I guess what I'm suggesting, is if you send an invalid username and password, can you get it to bomb - so that the response would at least give you a failure or something?

Good idea, I'll try that now

Quote
I'm suprised that you don't get a 401 status code.

What I should get is a 407 and if I switch on LiveHeaders in Firefox that's what I see.  I'll try perkiset suggestion and will let you know how I get on.

Thanks guys

Ok, 407.  So, you *are* getting that with Firefox, but not with your code?

In perl we just use like $status = $res->status_line;

Why doesn't your code return the same status code as Firefox?

Bompa
Logged

"The most beautiful and profound emotion we can experience is the sensation of the mystical..." - Albert Einstein
mrsdf
Rookie
**
Offline Offline

Posts: 20



View Profile
« Reply #7 on: July 31, 2007, 12:18:17 PM »

The easy way is to check with invalid user/password. The other way is to actually check the headers, this may be useful when you just want to get something out of the headers.

After looking at the curl error codes I didn't see any 407 reference, and the curl authors don't really care about making the lib useful and easy to use at the same time
( http://curl.haxx.se/mail/lib-2006-09/0272.html ) , here's how to actually check the headers for the error.

php curl_setopt  ->> CURLOPT_HEADERFUNCTION
quote from the manual:
Quote
The name of a callback function where the callback function takes two parameters. The first is the cURL resource, the second is a string with the header data to be written. The header data must be written when using this callback function. Return the number of bytes written.
I know it says 'written' and not 'read', but this function actually checks the headers receiver from the server.

The headerfunction allows you to check for some text in the headers, but unfortunately the only way (I know) to get data out of it is to use a global var. I've been using something like this to get the cookies out of the header into an array. So here is what it should look like:

function headerfunc($ch,$data)
{
        global $somevar;
         //replace function with stristr 407 or preg_match or something
        if(something_looks_wrong_in_headers($data))
        {
              $somevar=TRUE;
        }
        return strlen($data);
}

Initialize $somevar with FALSE just before doing curl_exec and check its value afterwards.

$ch=curl_init();
.....
.....
curl_setopt($ch,CURLOPT_HEADERFUNCTION,'headerfunc');
$somevar=FALSE;
$response=curl_exec($ch);
if($somevar)
{
   do_something();
}
Logged

We're sp4mmin', we're sp4mmin', I hope you like sp4mmin' too...
majiren
n00b
*
Offline Offline

Posts: 3


View Profile
« Reply #8 on: July 07, 2008, 11:45:35 PM »

Hey,

I am having a similar problem, i do not understand why I am not receiving the 407  Sad at least I would know I was on the right track! I seem to have hit a wall with getting my simple curl script to use a proxy...

I have read into the CURLOPT_HEADERFUNCTION but seem to be getting swfa from it! (Its not being called!)

here is my code, hope someone can help - thanks

Code:
function titties($ch,$data)
{
        global $somevar;

    if(stristr('407', $data))
        {
              $somevar='TRUE';
        }
        return strlen($data);
}


$ch = curl_init();
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 0);
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP);
curl_setopt($ch, CURLOPT_PROXYAUTH, CURLAUTH_BASIC); /////CURLAUTH_NTLM//////CURLAUTH_BASIC
curl_setopt($ch, CURLOPT_PROXY, '74.54.199.194:8080');
curl_setopt($ch, CURLOPT_PROXYUSERPWD, "user:pass");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4');
curl_setopt($ch, CURLOPT_URL, 'http://www.digg.com');
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$somevar = 'FALSE';
curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'titties');


if (!$page = curl_exec($ch)) {
echo 'cURL Error No. ';
echo  curl_error($ch);
echo '<br />';

print_r(curl_getinfo($ch));
}

curl_close($ch);


echo '<hr>';
echo 'Found 407: '.$somevar;
echo '<hr>';

Logged

No links in signatures please
Pages: [1]
  Print  
 
Jump to:  

Perkiset's Place Home   Best of The Cache   phpMyIDE: MySQL Stored Procedures, Functions & Triggers
Politics @ Perkiset's   Pinkhat's Perspective   
cache
mart
coder
programmers
ajax
php
javascript
Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC
Seo4Smf v0.2 © Webmaster's Talks


Valid XHTML 1.0! Valid CSS!