The Cache: Technology Expert's Forum
 
*
Welcome, Guest. Please login or register. September 18, 2019, 12:37:17 PM

Login with username, password and session length


Pages: [1]
  Print  
Author Topic: Multi threaded php apps?  (Read 6256 times)
m0nkeymafia
Expert
****
Offline Offline

Posts: 240


Check it!


View Profile
« on: June 12, 2007, 03:22:54 PM »

Can it be done?
If so does anyone wanna give me a few pointers or some code to look at?
I tried googling but the results simply sucked

What, if any, are the implications regarding outputting to screen, storing info in variables, locking, etc
Cheers!
Logged

I am Tyler Durden
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #1 on: June 12, 2007, 03:40:25 PM »

If you're using PHP from the console, it can be done just fine.

If you're hinting at spinning off threads while in the middle of a call from Apache, it's a big no-no.

The only way that I've reliably been able to simulate this is to fire off shell_exec() functions so that the shell does it for me... it is inelegant but works. I use semaphore files, the database for communication between processes, as well as using parameters on the exec line and return values from the called procedure.

You can still take advantage of thread-safe semaphores and such using the semV library (compile line switch on PHP) so you can talk (minimally) between applications. I do not know if you can use APC in this instance - I know you cannot use it directly from the shell, but using it in a executed script under a child of Apache MIGHT be able to - which would give you much more robust inter-process communication capabilities.
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
georgiecasey
Rookie
**
Offline Offline

Posts: 16


View Profile
« Reply #2 on: August 01, 2007, 08:40:55 PM »

perk,
im running php off the shell, accessing a mysql database to get a url to post to, then update that record with set done='true' or something, then accessing the next mysql record. i dont use cron to run the shell script, just start a screen and run a bash script that keeps running the php shell script in a loop.

is this an efficient way to do things? wat do other people use, cron? i want to go multi-threaded as one script is very slow obviously, wats the best way to do this. i tried just starting more screen sessions and more bash scripts but something crossed wires and i was getting weird errors.

so just wondering what other peoples setups are, so i can copy em :-)
Logged

No links in signatures please
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #3 on: August 01, 2007, 10:01:53 PM »

im running php off the shell, accessing a mysql database to get a url to post to, then update that record with set done='true' or something, then accessing the next mysql record. i dont use cron to run the shell script, just start a screen and run a bash script that keeps running the php shell script in a loop.

is this an efficient way to do things? wat do other people use, cron? i want to go multi-threaded as one script is very slow obviously, wats the best way to do this. i tried just starting more screen sessions and more bash scripts but something crossed wires and i was getting weird errors.

Completely fine way of doing things... I do exactly that all the time. But for things that are really big, or easily parsed into little chunks, I'll use the DB to keep state of where I am in the job, then cron me a script that does a part of the work. I just mentioned in another thread that my crawler and eblaster work this way:
cron calls eblaster.php
eblaster.php looks to see if there are any mails that need dispatching
I keep a table of "handle" records and a config record that tells me how many handles <this> mailer application is allowed to have (different clients get different dispatch rates)
eblaster dispatches up to (max handles) dispatchlet.php instances to handle the mails in queue
eblaster busy waits for up to 50 seconds and watches handles and queue... if there's a free handle and a mail to dispatch then do another
at about 50 seconds it quits, cron will call it again in a bit.

Although a bit klunky, this allows for all kinds of throttling options, as well as very easy telemetry and easy pause without any great affect - and since I often have many such jobs going at once, it allows me to balance the load across a particular machine.

@ What's the best for multithreaded: Nop would say Python  ROFLMAO ... PHP has threading but it is less than perfect. As mentioned above, I just use shells and let the kernel do the scheduling for me... poor man's threading Wink

/p
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
nop_90
Global Moderator
Lifer
*****
Offline Offline

Posts: 2203


View Profile
« Reply #4 on: August 02, 2007, 01:23:58 AM »

@ What's the best for multithreaded: Nop would say Python  ROFLMAO ... PHP has threading but it is less than perfect. As mentioned above, I just use shells and let the kernel do the scheduling for me... poor man's threading Wink
Erlang on paper is the best for multiple threads. Unfortunately there are no libraries for it Smiley. So unless u are a telecom company or doing specific work .....

Python has the limitation that even if it is multithreaded it can only use one processor. The reason behind this is that only one process can access python code at a given time. (What they call the global interpretor lock). Since most processing takes place outside of python code this is not really an issue. For example the waiting for a socket and recieving data on a socket is all done in C code. Most other languages despite what they say have similar problems.

All conventional languages require global variables, the mean mutexes, which mean mysterious crashes, lock ups heap corruption etc  ROFLMAO ROFLMAO

In almost all cases threading is not required (did perks say most of the time u should rethink design), you are better off to do what perks calls poor man threading Smiley.

The only case where threading is actually needed is when dealing with GUI.
But in almost all gui regardless of language only the main thread can modify/update the gui.
Ussually in this case I have the threads do the heavy work when they are done they place results into a threadsafe queue.
Main thread then has a timer, timer polls the queue and if needed updates gui, and also timer can do otherwise and place data on queues.
That way gui runs smooth, you do not get nasty lock up etc.

For educational purposes i uploaded somewhere here a perl proggie that uses same technique to upload files to multiple ftp sites at once Smiley

If you where really hardcore for PHP create a class that uses fork.
the main process would be like a controller, and it would have ability to create subprocess using fork, it would then transparently (to the user) talk to the subprocess using either sockets/IPC.
this is perl's version http://search.cpan.org/~rybskej/forks-0.24/lib/forks.pm

For me i stay away from threads unless i absolutely need them Cheesy





Logged
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #5 on: August 02, 2007, 08:01:00 AM »

In almost all cases threading is not required

So true. People confuse concurrent processing with threading. As you mention Nop, what he needs here is a really simple way to get multiple things done at one time, not multithreading.

The few times I've needed for-reals threading was writing network service daemons. Other than that, shell execs are just fine methinks.

/p
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
mrsdf
Rookie
**
Offline Offline

Posts: 20



View Profile
« Reply #6 on: August 02, 2007, 10:58:15 AM »

Here's some code I use to avoid using curl_multi. The first php calls the second php at a specific time interval. The second one runs in the background (not sure if it works on windows) and gets each url from the command line argument and processes it. The code is much more simple than curl_multi, and it works for me.

start.php:
Code:
<?php
    $urls
=file('input',FILE_IGNORE_NEW_LINES|FILE_SKIP_EMPTY_LINES);
    foreach (
$urls as $url)
    {
        
popen('php ./get.php '.escapeshellarg($url).' &',"r");
        
usleep(300000);
    }
?>


get.php:
Code:
<?php
    $data
=download_page_with_curl($argv[1]);
    
do_something_with_data($data);
?>


Logged

We're sp4mmin', we're sp4mmin', I hope you like sp4mmin' too...
webprofessor
Rookie
**
Offline Offline

Posts: 15


View Profile
« Reply #7 on: August 30, 2007, 10:15:05 AM »

There is no such thing as mutithreaded apps in PHP. Your confusing mutiple processes with multiple threads. There is a big differnce between the two.

A thread:
- is a resource cheap separate path of execution
- has access to its parent processes memory -
can communicate with its parent easily.

A process in a multiprocess program has:
- its own seperate copy of all the resources that the parent process had
- its own memory space/resource seperate from the parent
- can communicate to the parent process through IPC

Logged

No links in signatures please
JasonD
Expert
****
Offline Offline

Posts: 100


View Profile
« Reply #8 on: August 30, 2007, 03:08:55 PM »

As the old saying goes:

Even a bad fork is better than a good day at work
Logged
webprofessor
Rookie
**
Offline Offline

Posts: 15


View Profile
« Reply #9 on: August 31, 2007, 09:05:51 AM »

 D'oh!
Logged

No links in signatures please
vsloathe
vim ftw!
Global Moderator
Lifer
*****
Offline Offline

Posts: 1669



View Profile
« Reply #10 on: August 31, 2007, 09:31:25 AM »

Most of my webservers run Apache, but at work our dev server runs IIS. Through a little playing with COM objects I was able to hook into a DLL that let me screw around with named pipes and I can communicate between disparate threads wicked fast that way. I'm still screwing around with it and I'm still at the stage of a caveman throwing rocks at stuff to see what yelps.
Logged

hai
webprofessor
Rookie
**
Offline Offline

Posts: 15


View Profile
« Reply #11 on: August 31, 2007, 09:38:10 AM »

You can start a thread in PHP?

Please post a code example then I'd love to see proof of that.
Logged

No links in signatures please
vsloathe
vim ftw!
Global Moderator
Lifer
*****
Offline Offline

Posts: 1669



View Profile
« Reply #12 on: August 31, 2007, 10:09:13 AM »

You can start a thread in PHP?

Please post a code example then I'd love to see proof of that.
No, I'm talking about calling it from the command line but using named pipes to communicate between the thread and the base process or the thread and the other thread processes.
Logged

hai
webprofessor
Rookie
**
Offline Offline

Posts: 15


View Profile
« Reply #13 on: August 31, 2007, 10:16:36 AM »

So your talking about talking to processes not threads.
Logged

No links in signatures please
vsloathe
vim ftw!
Global Moderator
Lifer
*****
Offline Offline

Posts: 1669



View Profile
« Reply #14 on: September 04, 2007, 06:35:24 AM »

yah
Logged

hai
Pages: [1]
  Print  
 
Jump to:  

Perkiset's Place Home   Best of The Cache   phpMyIDE: MySQL Stored Procedures, Functions & Triggers
Politics @ Perkiset's   Pinkhat's Perspective   
cache
mart
coder
programmers
ajax
php
javascript
Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC
Seo4Smf v0.2 © Webmaster's Talks


Valid XHTML 1.0! Valid CSS!