next »

[kurdt]

I have been thinking about job queue and how to do it. My idea is that I do a simple table with columns "task" and "status". Then I have a task_delegator script that reads that table and it has preconfigured tasks built-in. Task column tells it what needs to be done and status tells it whether task has been started by another task_delegator already or is it available. When task is done, it's logged and removed from queue.

But how do I keep this under control so that server doesn't get bogged down because there just happens to be X amount of resource intensive tasks running? One idea is that I could read some current stats for CPU in task_delegator and then decide if to proceed. Then what to do if some task goes to infinite loop? Here I can't think anything except that task scripts should be done so well that they don't get stuck in infinite loop or something

[Phaëton]

im curious to see what is said about this as well... doesnt the system handle cpu
load balancing? Well.. all the tasks that start something your afraid of entering the infinite loop
could throw out a 'hey im going in the cave' record somewhere.. and then a seperate task
once or twice a day to make sure that everyone that went into a cave came out and
took down their own flag.

interested to see where this one goes... is this on a *amp box?

[kurdt]

im curious to see what is said about this as well... doesnt the system handle cpu
load balancing?

It does but the point is that it just doesn't start new tasks if there's already 10 tasks taking 100% CPU. Instead get those done before getting next one in works.

Well.. all the tasks that start something your afraid of entering the infinite loop could throw out a 'hey im going in the cave' record somewhere.. and then a seperate task
once or twice a day to make sure that everyone that went into a cave came out and took down their own flag.

That could work. Mainly I'm afraid of the 3rd party stuff that might go apeshit. I have some exec stuff there that I just can't avoid.

is this on a *amp box?

Kinda.. basic PHP & MySQL on mainly Debian.

[perkiset]

@ loading the CPU: What kind of tasks are we talking about K? If they are network tasks, then the natural bottleneck of the 'net will help you out. Other than that, if you cron something to hit once per minute and then only do (n) jobs, perhaps even instituting a little bit of sleep in the process, you'll be alright.

This is pretty much exactly how my queuing systems work. I upgraded a bit ago to implement stored procedures in MySQL to make sure there's never a race condition, and I can go from one handler to any number of handlers without having DB problems.

[nutballs]

i have a task that fires every 5 minutes.
however, its a beast of a task, and if the server is under load, it will cause a domino effect that will never recover, getting slower and slower.

So... I check the server load in /proc/loadavg and make sure its under an upper limit.
Actually a few tasks do this, but it was done initially for a specific one.

[perkiset]

My eblaster does that, but just a little differently. I do a shell_exec('ps aux') and see if there are any of (me) running and if so, halt - making it sort of ghetto singleton.

[Phaëton]

I was gonna suggest a ps aux but i wasnt sure if its available on debian...

[walrus]

Code:

//this gets the current load - but what nutballs said with proc/loadavg might be better
function get_load()
{
preg_match('/average: (.+?),/',`uptime`,$out);
return $out[1];
}

//this counts how many times $filename is already running
$running_count = `ps aux | grep -i $filename |grep -v grep| wc -l`;