The Cache: Technology Expert's Forum
 
*
Welcome, Guest. Please login or register. September 23, 2019, 12:12:19 AM

Login with username, password and session length


Pages: [1]
  Print  
Author Topic: Working with proxies  (Read 6517 times)
skyts
Rookie
**
Offline Offline

Posts: 22


View Profile
« on: July 23, 2007, 03:31:21 AM »

Perk, I don't see where the java section is, I am talking about java, not javascript - But couldn't find a section fit for it, So feel free to move it to where it belongs.

OK, So I developed an application that is working with proxies. Each time I connect to a site with a proxy, And then I hang up the connection. However, When seeing programs like Hreferer for example, It seems that their program stays connected to the proxy, And do not close connection after each request for a page. It seems a lot faster, But still I'm not sure how to do it - Because persoanlly I know that for each request a proxy is connecting to the internet, performing the job it needs to, and then closes connection.

Anybody have a clue how to do it, And if it worth the time investing in it?

Logged
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #1 on: July 23, 2007, 08:40:02 AM »

Hey Skyts -

Good eye - didn't have Java listed anywhere - as you can see, I have renamed this board to handle it.

However I do not have an answer for you since I know just .0001% more than squat about Java...

/p
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
skyts
Rookie
**
Offline Offline

Posts: 22


View Profile
« Reply #2 on: July 23, 2007, 11:50:10 AM »

Hey Skyts -

Good eye - didn't have Java listed anywhere - as you can see, I have renamed this board to handle it.

However I do not have an answer for you since I know just .0001% more than squat about Java...

/p

Cheers perk, Maybe someone else will know the answer Smiley

Logged
thedarkness
Lifer
*****
Offline Offline

Posts: 585



View Profile
« Reply #3 on: July 23, 2007, 05:38:14 PM »

Pretty sure that this is in the underlying proxy connection. Do a search for "Proxy-Connection: Keep-Alive" (actual header) and post back if you need more. We're open all hours  ROFLMAO

Good on you skyts, good luck.

I'll be taking advantage of this in the near future but my rig is built in C++, should be similar in implementation though.

Cheers,
td
Logged

"I want to be the guy my dog thinks I am."
 - Unknown
arms
Expert
****
Offline Offline

Posts: 235



View Profile
« Reply #4 on: July 23, 2007, 08:08:35 PM »

if you use this,
Code:
System.setProperty("http.proxyHost", "127.0.0.1");
System.setProperty("http.proxyPort", "8118");
you only have to set it once. if you're using proxies with authentication i don't remeber what the properties are.

if you're using httpclient ( http://jakarta.apache.org/commons/httpclient/ ),
Code:
HttpClient client = new HttpClient();
client.getHostConfiguration().setProxy("127.0.0.1", 8118);

// if you need authentication add this:
client.getState().setProxyCredentials(
            new AuthScope(null, -1),
            new UsernamePasswordCredentials("username", "password")
);
you can reuse the client so you don't have to reset it
Logged
skyts
Rookie
**
Offline Offline

Posts: 22


View Profile
« Reply #5 on: July 24, 2007, 02:38:12 AM »

if you use this,
Code:
System.setProperty("http.proxyHost", "127.0.0.1");
System.setProperty("http.proxyPort", "8118");
you only have to set it once. if you're using proxies with authentication i don't remeber what the properties are.

if you're using httpclient ( http://jakarta.apache.org/commons/httpclient/ ),
Code:
HttpClient client = new HttpClient();
client.getHostConfiguration().setProxy("127.0.0.1", 8118);

// if you need authentication add this:
client.getState().setProxyCredentials(
            new AuthScope(null, -1),
            new UsernamePasswordCredentials("username", "password")
);
you can reuse the client so you don't have to reset it

Globally, My Connector class uses the same mechanism - just without authentication... So as I see it, the client do not make the decision about permanent connection - it is the decision of the proxy server, or do I understand it wrong?

Logged
thedarkness
Lifer
*****
Offline Offline

Posts: 585



View Profile
« Reply #6 on: July 24, 2007, 03:39:59 AM »

Clues to be had here;

http://forum.java.sun.com/thread.jspa?threadID=608874&messageID=3332680

http://www.squid-cache.org/mail-archive/squid-users/200704/0282.html

http://books.google.com/books?id=iq10UfwhosoC&pg=PA98&lpg=PA98&dq=header+connection+%22proxy+connection+keep+alive%22&source=web&ots=FnPA5598HZ&sig=AB8MilwLVnf9tS_6Ng9cJWrB6tE

http://www.codeproject.com/cs/internet/Crawler.asp#points

Perhaps you need to target a lower level object/call that allows you to send a header directly out the socket if this option is not supported by your HttpClient object?

HTH,
td
Logged

"I want to be the guy my dog thinks I am."
 - Unknown
arms
Expert
****
Offline Offline

Posts: 235



View Profile
« Reply #7 on: July 24, 2007, 09:10:23 AM »

all i know is you set it once, whether or not you stay connected i don't know.
if your using a URLConnection (or HttpURLConnection) object or HttpClient you can set/get the headers yourself.
so i guess you need to set the "Proxy-Connection: Keep-Alive" or "Connection: Keep-Alive" header or both and in the response check for "Connection: close" to see if the server refuses it.
Logged
skyts
Rookie
**
Offline Offline

Posts: 22


View Profile
« Reply #8 on: July 24, 2007, 09:20:20 AM »



My client supports it, but I'm not sure a public proxy like I use support this kind of beahviour. But then again, If you've seen Hreferer once, It looks like it is using this mecahnisem (at least visually it looks like it), Like it stays connected to the proxy all the time, And just make different queries in google.
Logged
thedarkness
Lifer
*****
Offline Offline

Posts: 585



View Profile
« Reply #9 on: July 24, 2007, 02:24:07 PM »

Use something like wireshark to see what headers it's actually sending.

Cheers,
td
Logged

"I want to be the guy my dog thinks I am."
 - Unknown
Pages: [1]
  Print  
 
Jump to:  

Perkiset's Place Home   Best of The Cache   phpMyIDE: MySQL Stored Procedures, Functions & Triggers
Politics @ Perkiset's   Pinkhat's Perspective   
cache
mart
coder
programmers
ajax
php
javascript
Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC
Seo4Smf v0.2 © Webmaster's Talks


Valid XHTML 1.0! Valid CSS!