The Cache: Technology Expert's Forum
 
*
Welcome, Guest. Please login or register. September 18, 2019, 12:32:56 AM

Login with username, password and session length


Pages: [1]
  Print  
Author Topic: wget and copying an entier password (not htacces) protected site  (Read 4589 times)
herbacious
Journeyman
***
Offline Offline

Posts: 51


View Profile
« on: September 17, 2009, 03:19:38 PM »

I need to create an entier static copy of a site that has a login.

thought wget might be up to the task - anyone know the best way to do it?
Logged
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #1 on: September 17, 2009, 03:32:28 PM »

--username= --password= (could be just --user=, I don't remember ATM)

Then look here: http://www.veen.com/jeff/archives/000573.html

Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
vsloathe
vim ftw!
Global Moderator
Lifer
*****
Offline Offline

Posts: 1669



View Profile
« Reply #2 on: September 18, 2009, 10:03:33 AM »

Code:
$ man wget
Logged

hai
krustee
Rookie
**
Offline Offline

Posts: 33


View Profile
« Reply #3 on: September 20, 2009, 08:12:12 PM »

wget --username= --password= -r -p -k urlhere -e robots=off
Logged
herbacious
Journeyman
***
Offline Offline

Posts: 51


View Profile
« Reply #4 on: September 21, 2009, 02:53:35 AM »

thanks for the replies Smiley

as I understood it the username and password switches are for htaccess password protection, not login form protection.

Just wondering if anyone had a any great ideas there.

I had thought of writing a little PHP script to handle the login, create a cookie and then pass over to wget - would that work do you think?
Logged
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #5 on: September 21, 2009, 10:24:24 AM »

Sorry, missed the login form part.

No, WGet will not be able to help you with that because it's programmatic ... somehow, via a session cookie, or a GET parameter, the server will want to verify that you have a valid session, approved via a password login. I don't believe there's any WGet stuff for that ... it'd need to be WPost for that Wink

So this will need to be a PHP (or a language you're comfortable in) effort. Log in, most likely get a cookie for your efforts, then you'll need to send that cookie up with each request so that the server knows you're logged in. You can do that by hand with sockets, you can try my webRequest class here or, probably the best advice, become conversant with cURL which is complicated and probably way more than you need, but would do the job.
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
vsloathe
vim ftw!
Global Moderator
Lifer
*****
Offline Offline

Posts: 1669



View Profile
« Reply #6 on: September 21, 2009, 11:02:55 AM »

You can still do it in wget. Ignore perk.

wget can do cookies, POSTs, whatever. Anything that cURL can do, wget can do faster. Anything that HTTPClient can do, wget can do faster.

EDIT: The above to say; anything a browser can do, wget can do faster.
Logged

hai
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #7 on: September 21, 2009, 11:04:23 AM »

Then I am an idiot and looking forward to you spilling the beans VS ... please, do it!
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
vsloathe
vim ftw!
Global Moderator
Lifer
*****
Offline Offline

Posts: 1669



View Profile
« Reply #8 on: September 21, 2009, 11:13:00 AM »

Bah, come and correct him and the man puts you to work! For the record I was just giving you a hard time. I'm right, but you're right as well in that it's probably more straightforward for him to learn how to do it in a language, especially if he's not looking to become a sysadmin.

I'll take a stab...

Code:
wget --post-data="username=guywhodoesntusemanpages&password=flamemeonforumspls" --keep-session-cookies --save-cookies myCookieFile.txt --load-cookies myCookieFile.txt http://domain.com/page

Seriously though, he needs to look at the man page. Without knowing exactly what he wants to do, it's pretty pointless for me to guess at it.

The ten minutes it will take you to skim through wget's man pages are worth it, trust me. It's an invaluable little tool.
Logged

hai
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #9 on: September 21, 2009, 11:21:29 AM »

The ten minutes it will take you to skim through wget's man pages are worth it, trust me. It's an invaluable little tool.

that much is certain... wget is under appreciated IMO.
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
herbacious
Journeyman
***
Offline Offline

Posts: 51


View Profile
« Reply #10 on: September 22, 2009, 01:59:05 PM »

thanks a lot Smiley

like perks I had come to the conclusion that wget wasn't able to do what i needed on its own..

did have a look through the man pages but i seem to have some kind of man page blindness like i read 30% of it and then my brain stops taking it in way before ive got to the bit that im actually looking for...

anyway - cheers for that  Grin
Logged
vsloathe
vim ftw!
Global Moderator
Lifer
*****
Offline Offline

Posts: 1669



View Profile
« Reply #11 on: September 24, 2009, 10:07:04 AM »

...i read 30% of it and then my brain stops taking it in way before ive got to the bit that im actually looking for...

lol you know who you're talking to? I probably have some sort of world record in ADD.

Code:
man wget | grep post
man wget | grep cookie
man wget | grep session
...etc.
Logged

hai
Pages: [1]
  Print  
 
Jump to:  

Perkiset's Place Home   Best of The Cache   phpMyIDE: MySQL Stored Procedures, Functions & Triggers
Politics @ Perkiset's   Pinkhat's Perspective   
cache
mart
coder
programmers
ajax
php
javascript
Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC
Seo4Smf v0.2 © Webmaster's Talks


Valid XHTML 1.0! Valid CSS!