next »

[skyts]

Perk, I don't see where the java section is, I am talking about java, not javascript - But couldn't find a section fit for it, So feel free to move it to where it belongs.

OK, So I developed an application that is working with proxies. Each time I connect to a site with a proxy, And then I hang up the connection. However, When seeing programs like Hreferer for example, It seems that their program stays connected to the proxy, And do not close connection after each request for a page. It seems a lot faster, But still I'm not sure how to do it - Because persoanlly I know that for each request a proxy is connecting to the internet, performing the job it needs to, and then closes connection.

Anybody have a clue how to do it, And if it worth the time investing in it?

[perkiset]

Hey Skyts -

Good eye - didn't have Java listed anywhere - as you can see, I have renamed this board to handle it.

However I do not have an answer for you since I know just .0001% more than squat about Java...

/p

[skyts]

Hey Skyts -

Good eye - didn't have Java listed anywhere - as you can see, I have renamed this board to handle it.

However I do not have an answer for you since I know just .0001% more than squat about Java...

/p

Cheers perk, Maybe someone else will know the answer

[thedarkness]

Pretty sure that this is in the underlying proxy connection. Do a search for "Proxy-Connection: Keep-Alive" (actual header) and post back if you need more. We're open all hours 

Good on you skyts, good luck.

I'll be taking advantage of this in the near future but my rig is built in C++, should be similar in implementation though.

Cheers,
td

[arms]

if you use this,

Code:

System.setProperty("http.proxyHost", "127.0.0.1");
System.setProperty("http.proxyPort", "8118");

you only have to set it once. if you're using proxies with authentication i don't remeber what the properties are.

if you're using httpclient ( http://jakarta.apache.org/commons/httpclient/ ),

Code:

HttpClient client = new HttpClient();
client.getHostConfiguration().setProxy("127.0.0.1", 8118);

// if you need authentication add this:
client.getState().setProxyCredentials(
            new AuthScope(null, -1),
            new UsernamePasswordCredentials("username", "password")
);

you can reuse the client so you don't have to reset it

[skyts]

if you use this,

Code:

System.setProperty("http.proxyHost", "127.0.0.1");
System.setProperty("http.proxyPort", "8118");

you only have to set it once. if you're using proxies with authentication i don't remeber what the properties are.

if you're using httpclient ( http://jakarta.apache.org/commons/httpclient/ ),

Code:

HttpClient client = new HttpClient();
client.getHostConfiguration().setProxy("127.0.0.1", 8118);

// if you need authentication add this:
client.getState().setProxyCredentials(
            new AuthScope(null, -1),
            new UsernamePasswordCredentials("username", "password")
);

you can reuse the client so you don't have to reset it

Globally, My Connector class uses the same mechanism - just without authentication... So as I see it, the client do not make the decision about permanent connection - it is the decision of the proxy server, or do I understand it wrong?

[thedarkness]

Clues to be had here;

http://forum.java.sun.com/thread.jspa?threadID=608874&messageID=3332680

http://www.squid-cache.org/mail-archive/squid-users/200704/0282.html

http://books.google.com/books?id=iq10UfwhosoC&pg=PA98&lpg=PA98&dq=header+connection+%22proxy+connection+keep+alive%22&source=web&ots=FnPA5598HZ&sig=AB8MilwLVnf9tS_6Ng9cJWrB6tE

http://www.codeproject.com/cs/internet/Crawler.asp#points

Perhaps you need to target a lower level object/call that allows you to send a header directly out the socket if this option is not supported by your HttpClient object?

HTH,
td

[arms]

all i know is you set it once, whether or not you stay connected i don't know.
if your using a URLConnection (or HttpURLConnection) object or HttpClient you can set/get the headers yourself.
so i guess you need to set the "Proxy-Connection: Keep-Alive" or "Connection: Keep-Alive" header or both and in the response check for "Connection: close" to see if the server refuses it.

[skyts]

Clues to be had here;

http://forum.java.sun.com/thread.jspa?threadID=608874&messageID=3332680

http://www.squid-cache.org/mail-archive/squid-users/200704/0282.html

http://books.google.com/books?id=iq10UfwhosoC&pg=PA98&lpg=PA98&dq=header+connection+%22proxy+connection+keep+alive%22&source=web&ots=FnPA5598HZ&sig=AB8MilwLVnf9tS_6Ng9cJWrB6tE

http://www.codeproject.com/cs/internet/Crawler.asp#points

Perhaps you need to target a lower level object/call that allows you to send a header directly out the socket if this option is not supported by your HttpClient object?

HTH,
td

My client supports it, but I'm not sure a public proxy like I use support this kind of beahviour. But then again, If you've seen Hreferer once, It looks like it is using this mecahnisem (at least visually it looks like it), Like it stays connected to the proxy all the time, And just make different queries in google.

[thedarkness]

Use something like wireshark to see what headers it's actually sending.

Cheers,
td