The Cache: Technology Expert's Forum
 
*
Welcome, Guest. Please login or register. September 22, 2019, 06:22:17 PM

Login with username, password and session length


Pages: [1]
  Print  
Author Topic: Launch N-processes based on existing number of processes?  (Read 3272 times)
nutballs
Administrator
Lifer
*****
Offline Offline

Posts: 5627


Back in my day we had 9 planets


View Profile
« on: May 15, 2009, 08:45:31 PM »

I was about to embark on this on my own, but decided against it.

What I want is to call a script every minute via cron. - no problems there...

I want the script to check to see how many instances there are of a specific process, in particular a PHP script I am running from the CLI.
I want the script to maintain a total of 10 running. So if there are 7 already going, i want it to run that command 3 more times.
Ideally, I also want it to pay attention to the Load values. Lets say, 1min avg<1 (dual quads so.. technically 8 should be max i think)

I am not sure how to count the existing procs from script. I know how to view them, but not count them.
ps aux | grep "some special string"

also I have no idea how to do a loop. I am guessing its similar syntax actually to PHP. And frankly, i assume it could even be a PHP file that I run as the script from CLI instead of a .sh right? So if I can do PHP, I might be able to figure it out on my own. But a quick VS style 4 character magic code would be cool too. Smiley

Logged

I could eat a bowl of Alphabet Soup and shit a better argument than that.
vsloathe
vim ftw!
Global Moderator
Lifer
*****
Offline Offline

Posts: 1669



View Profile
« Reply #1 on: May 16, 2009, 07:46:15 PM »

Code:
ps aux | grep -c 'script_name'

Will return the number of processes running, plus one for the grep itself.
Logged

hai
nutballs
Administrator
Lifer
*****
Offline Offline

Posts: 5627


Back in my day we had 9 planets


View Profile
« Reply #2 on: May 16, 2009, 09:17:06 PM »

-c    DUH
thanks V

seriously I think My --help keys are stuck
Logged

I could eat a bowl of Alphabet Soup and shit a better argument than that.
nop_90
Global Moderator
Lifer
*****
Offline Offline

Posts: 2203


View Profile
« Reply #3 on: May 16, 2009, 09:20:24 PM »

WTF is this love of cron people have ?

Better to have one program.
Have the program fork/spawn all the child programs you need.

That way you can control the input/output/what ever.
You can also check exit code of function.

Fork is the best. As perks said, because of the way linux is setup. Extra forks do not take any extra memory.
(Hard to believe but it is true).
Logged
nutballs
Administrator
Lifer
*****
Offline Offline

Posts: 5627


Back in my day we had 9 planets


View Profile
« Reply #4 on: May 16, 2009, 11:01:56 PM »

im actually going to be moving towards that. An infinite loop that checks for processes and launches as needed, probably going to do it in PHP-cli.

I just have been currently using cron because it was there and worked with what i already had with little effort.
Logged

I could eat a bowl of Alphabet Soup and shit a better argument than that.
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #5 on: May 17, 2009, 08:22:23 PM »

WTF is this love of cron people have ?

LOL Because I don't want to have to re-code 5,10,25,30,55 * 1,3,5,6 1-6, *

cron is awesome as a timer when you don't want something to run * * * * *

Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
nop_90
Global Moderator
Lifer
*****
Offline Offline

Posts: 2203


View Profile
« Reply #6 on: May 17, 2009, 11:12:03 PM »

WTF is this love of cron people have ?
LOL Because I don't want to have to re-code 5,10,25,30,55 * 1,3,5,6 1-6, *

cron is awesome as a timer when you don't want something to run * * * * *
Ok so I was not the only one who experienced that Smiley

I remember once i played with cron. I wasted like 2 hours trying to get it work.
After I read about how great it was.
processes = []
while 1 :
        for p in processes :
             if p.exitcode == 0 :
                  do something
        time.sleep(30)
there all your logic done Smiley
Logged
vsloathe
vim ftw!
Global Moderator
Lifer
*****
Offline Offline

Posts: 1669



View Profile
« Reply #7 on: May 18, 2009, 09:55:49 AM »

I've always found CRON pretty straightforward...
Logged

hai
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #8 on: May 18, 2009, 10:23:47 AM »

I agree, and I think that writing your own forker is important when you want sub-minute dispatch, but anything other than that cron is a gem IMO.

I write forkers only when I need something either interrupt driven or running much more than once per minute.
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
arms
Expert
****
Offline Offline

Posts: 235



View Profile
« Reply #9 on: May 19, 2009, 12:51:23 PM »

python has a multi processing library that has a process pool implementation.
this way you can manage processes like a thread pool with n amount of worker processes.

if your language of choice doesn't have something this you could throw it together in python and have that script manage your php scripts.

http://docs.python.org/library/multiprocessing.html

backport for python 2.5:
http://code.google.com/p/python-multiprocessing/
Logged
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #10 on: May 19, 2009, 01:03:07 PM »

Hmmm... so you'd use Python as your daemon management daemon, correct? Is this what you'd do in the same situation?
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
arms
Expert
****
Offline Offline

Posts: 235



View Profile
« Reply #11 on: May 19, 2009, 01:26:18 PM »

possibly. it's basically how i started using python. i wanted a shell script for something when i was still on windows and found python more intuitive and flexible than ms' bat syntax. then in linux, rather than learn sh i just converted over those python scripts.

but actually the library i linked to is not apropriate for this. it appears it's for running python functions in a subprocess.
 D'oh!

the snippet nop posted is almost working python code and would be an easy solution.
Logged
nop_90
Global Moderator
Lifer
*****
Offline Offline

Posts: 2203


View Profile
« Reply #12 on: May 19, 2009, 04:49:39 PM »

For what I am doing I use the multiprocessing module that arms talks about.
If you peek inside the code it just basically wraps the "fork" function.

As perks said. Amazingly enuff with increased forks your memory ussuage hardly goes up.
Ur processor ussuage will but not ur memory.

multiprocessing module is good if u are running same group of python "functions" etc.
There are a few crevates. Also if there is possibily that one of the subprocess with seg fault do not use shared memory.

If u want to do run "non-python" programs.
Then use subprocess
Advantage is u automatically have the stdin/stdout/stderr pipes attached
Code:
def runGocr(pilImage):
    io = StringIO.StringIO()
    pilImage.save(io,"ppm")
    imgStr = io.getvalue()
    io.close()
   
    p = subprocess.Popen(("gocr", "-"),
                          stdin = subprocess.PIPE,
                          stdout = subprocess.PIPE,
                          stderr = subprocess.PIPE,
                          close_fds = True)
   
    stdout_text, stderr_text = p.communicate(imgStr)
    return stdout_text.strip()

notice in the above snippit how the image is directly piped into GOCR using stdin.
Notice how the result is pulled out from stdout
In the above case it will block (which is what i want it to do) but you can check out module.

Logged
arms
Expert
****
Offline Offline

Posts: 235



View Profile
« Reply #13 on: May 20, 2009, 05:55:56 AM »

i always use os.popen and just pass the command as a string, but this would save me a step of writing to file before.
Logged
Pages: [1]
  Print  
 
Jump to:  

Perkiset's Place Home   Best of The Cache   phpMyIDE: MySQL Stored Procedures, Functions & Triggers
Politics @ Perkiset's   Pinkhat's Perspective   
cache
mart
coder
programmers
ajax
php
javascript
Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC
Seo4Smf v0.2 © Webmaster's Talks


Valid XHTML 1.0! Valid CSS!