next »

[herbacious]

I need to create an entier static copy of a site that has a login.

thought wget might be up to the task - anyone know the best way to do it?

[perkiset]

--username= --password= (could be just --user=, I don't remember ATM)

Then look here: http://www.veen.com/jeff/archives/000573.html

[vsloathe]

Code:

$ man wget

[krustee]

wget --username= --password= -r -p -k urlhere -e robots=off

[herbacious]

thanks for the replies

as I understood it the username and password switches are for htaccess password protection, not login form protection.

Just wondering if anyone had a any great ideas there.

I had thought of writing a little PHP script to handle the login, create a cookie and then pass over to wget - would that work do you think?

[perkiset]

Sorry, missed the login form part.

No, WGet will not be able to help you with that because it's programmatic ... somehow, via a session cookie, or a GET parameter, the server will want to verify that you have a valid session, approved via a password login. I don't believe there's any WGet stuff for that ... it'd need to be WPost for that

So this will need to be a PHP (or a language you're comfortable in) effort. Log in, most likely get a cookie for your efforts, then you'll need to send that cookie up with each request so that the server knows you're logged in. You can do that by hand with sockets, you can try my webRequest class here or, probably the best advice, become conversant with cURL which is complicated and probably way more than you need, but would do the job.

[vsloathe]

You can still do it in wget. Ignore perk.

wget can do cookies, POSTs, whatever. Anything that cURL can do, wget can do faster. Anything that HTTPClient can do, wget can do faster.

EDIT: The above to say; anything a browser can do, wget can do faster.

[perkiset]

Then I am an idiot and looking forward to you spilling the beans VS ... please, do it!

[vsloathe]

Bah, come and correct him and the man puts you to work! For the record I was just giving you a hard time. I'm right, but you're right as well in that it's probably more straightforward for him to learn how to do it in a language, especially if he's not looking to become a sysadmin.

I'll take a stab...

Code:

wget --post-data="username=guywhodoesntusemanpages&password=flamemeonforumspls" --keep-session-cookies --save-cookies myCookieFile.txt --load-cookies myCookieFile.txt http://domain.com/page

Seriously though, he needs to look at the man page. Without knowing exactly what he wants to do, it's pretty pointless for me to guess at it.

The ten minutes it will take you to skim through wget's man pages are worth it, trust me. It's an invaluable little tool.

[perkiset]

The ten minutes it will take you to skim through wget's man pages are worth it, trust me. It's an invaluable little tool.

that much is certain... wget is under appreciated IMO.

[herbacious]

thanks a lot

like perks I had come to the conclusion that wget wasn't able to do what i needed on its own..

did have a look through the man pages but i seem to have some kind of man page blindness like i read 30% of it and then my brain stops taking it in way before ive got to the bit that im actually looking for...

anyway - cheers for that 

[vsloathe]

...i read 30% of it and then my brain stops taking it in way before ive got to the bit that im actually looking for...

lol you know who you're talking to? I probably have some sort of world record in ADD.

Code:

man wget | grep post
man wget | grep cookie
man wget | grep session
...etc.