Working with proxies

skyts

Perk, I don't see where the java section is, I am talking about java, not

javascript

- But couldn't find a section fit for it, So feel free to move it to where it belongs.

OK, So I developed an application that is working with proxies. Each time I connect to a site with a proxy, And then I hang up the connection. However, When seeing programs like Hreferer for example, It seems that their program stays connected to the proxy, And do not close connection after each request for a page. It seems a lot faster, But still I'm not sure how to do it - Because persoanlly I know that for each request a proxy is connecting to the inte

rnet

, performing the job it needs to, and then closes connection.

Anybody have a clue how to do it, And if it worth the time investing in it?

perkiset

Hey Skyts -

Good eye - didn't have Java listed anywhere - as you can see, I have renamed this board to handle it.

However I do not have an answer for you since I know just .0001% more than squat about Java...

/p

skyts

quote author=perkiset link=topic=414.msg2691#msg2691 date=1185205202

Hey Skyts -

Good eye - didn't have Java listed anywhere - as you can see, I have renamed this board to handle it.

However I do not have an answer for you since I know just .0001% more than squat about Java...

/p

Cheers perk, Maybe someone else will know the answer

thedarkness

Pretty sure that this is in the underlying proxy connection. Do a search for "Proxy-Connection: Keep-Alive" (actual header) and post back if you need more. We're open all hours Applause

Good on you skyts, good luck.

I'll be taking advantage of this in the near future but my rig is built in C++, should be similar in implementation though.

Cheers,
td

arms

if you use this,

System.setProperty("http.proxyHost", "127.0.0.1");
System.setProperty("http.proxyPort", "8118");

you only have to set it once. if you're using proxies with authentication i don't remeber what the properties are.

if you're using httpclient ( http://jakarta.

apache

.org/commons/httpclient/ ),

HttpClient client = new HttpClient();
client.getHostConfiguration().setProxy("127.0.0.1", 8118);

// if you need authentication add this:
client.getState().setProxyCredentials(
new AuthScope(null, -1),
new UsernamePasswordCredentials("username", "password" Applause

);

you can reuse the client so you don't have to reset it

skyts

quote author=arms link=topic=414.msg2698#msg2698 date=1185246515

if you use this,

System.setProperty("http.proxyHost", "127.0.0.1");
System.setProperty("http.proxyPort", "8118");

you only have to set it once. if you're using proxies with authentication i don't remeber what the properties are.

if you're using httpclient ( http://jakarta.

apache

.org/commons/httpclient/ ),

HttpClient client = new HttpClient();
client.getHostConfiguration().setProxy("127.0.0.1", 8118);

// if you need authentication add this:
client.getState().setProxyCredentials(
new AuthScope(null, -1),
new UsernamePasswordCredentials("username", "password" Applause

);

you can reuse the client so you don't have to reset it

Globally, My Connector class uses the same mechanism - just without authentication... So as I see it, the client do not make the decision about permanent connection - it is the decision of the proxy server, or do I understand it wrong?

thedarkness

Clues to be had here;

http://forum.java.sun.com/thread.jspa?threadID=608874&messageID=3332680

http://www.squid-cache.org/mail-archive/squid-users/200704/0282.html

http://books.google.com/books?id=iq10UfwhosoC&pg=PA98&lpg=PA98&dq=header+connection+%22proxy+connection+keep+alive%22&source=web&ots=FnPA5598HZ&sig=AB8MilwLVnf9tS_6Ng9cJWrB6tE

http://www.codeproject.com/cs/inte

rnet

/Crawler.

asp

#points

Perhaps you need to target a lower level object/call that allows you to send a header directly out the socket if this option is not supported by your HttpClient object?

HTH,
td

arms

all i know is you set it once, whether or not you stay connected i don't know.
if your using a URLConnection (or HttpURLConnection) object or HttpClient you can set/get the headers yourself.
so i guess you need to set the "Proxy-Connection: Keep-Alive" or "Connection: Keep-Alive" header or both and in the response check for "Connection: close" to see if the server refuses it.

skyts

quote author=thedarkness link=topic=414.msg2701#msg2701 date=1185273599

Clues to be had here;

http://forum.java.sun.com/thread.jspa?threadID=608874&messageID=3332680

http://www.squid-cache.org/mail-archive/squid-users/200704/0282.html

http://books.google.com/books?id=iq10UfwhosoC&pg=PA98&lpg=PA98&dq=header+connection+%22proxy+connection+keep+alive%22&source=web&ots=FnPA5598HZ&sig=AB8MilwLVnf9tS_6Ng9cJWrB6tE

http://www.codeproject.com/cs/inte

rnet

/Crawler.

asp

#points

Perhaps you need to target a lower level object/call that allows you to send a header directly out the socket if this option is not supported by your HttpClient object?

HTH,
td

My client supports it, but I'm not sure a public proxy like I use support this kind of beahviour. But then again, If you've seen Hreferer once, It looks like it is using this mecahnisem (at least visually it looks like it), Like it stays connected to the proxy all the time, And just make different queries in google.

thedarkness

Use something like wireshark to see what headers it's actually sending.

Cheers,
td