next »

[netmktg]

A long while back I had a discussion on this board about writing a mysql-based multitasking system. And it has been holding out well so far 

In my existing system, the Client server "pops" the job of the mysql Task queue and executes it but it REQUIRES all the function definitions. That means the entire code has to be duplicated & updated on each client server.

I wanted to move over to something more independent/stateless. I thought Gearman was the solution, but its not... each Gearman Worker App also requires all the functions to be defined on the worker-side.


It does seem a rather far fetched idea in terms of PHP ... but is there something out there already that can also "pickup" the compiled function definitions from the job server and execute on the client server. That would mean the client server would need just a couple of lines to pop a job, no other files should be required.

[perkiset]

If the dispatching server also popped code that was eval'd and run on the fly, then you'd not need the code locally, but then you're essentially sending your entire app with every job.

I don't know of any simple way for a local app to call remote functions in php, if I understand your request correctly.

[intel352]

Why not remote file inclusion (via HTTP), or inclusion of code from a central shared drive?
Assuming your workers are on a private network, either method could work fine.

If you are not on a private network, then you would need to go to extra lengths to ensure that HTTP inclusion is secure.

Read more about HTTP inclusion here: http://php.net/manual/en/function.include.php

[lamontagne]

Cant you just do this with git and write a batch file to push to remote nodes after changes are made?

Honestly i can think of a few ways to do it and each has a different benefit. I'll list a few:

GIT - You're going to be pushing the files containing functions onto the client machines. This still requires duplication but will be much easier to maintain and make changes with. Bulky on file storage space but each nodes code can be individualized down to the most granular detail.

SOAP/REST API - A call is made to a function on a remote machine sending the parameters in an XML/REST format. This works best if a central server needs to dispatch jobs to multiple spiders. Network load increases

Remote File Include (what intel352 is talking about) - If the script is long running after it is fired up (loops forever reading from db) this is your best bet but provide a central way to kill the app on all machines (basically check if file exists on each loop, when you need to kill all clients delete the file , or create individual file for each node using a unique id, if file does not exist the loop simply does a sleep(30)... so it will check if file exists again every 30 seconds.. upon restart re-include central functions).. basically you build it to sit idle if you want it to. The file you remote include will be stored in memory on the server but not on disk.

[netmktg]

Why not remote file inclusion (via HTTP), or inclusion of code from a central shared drive?
Assuming your workers are on a private network, either method could work fine.

Can't sleep with the devil... the files are not kept publicly accessible.

Yep, revision control is meant for code updation. But my code still is open for all to see.
Maybe I just need a Php compiler. My original line of thought was to avoid floating my entire codebase around on multiple vps servers.

I am already using a custom mysql-based Task system which bundles parameters. The execution can be disabled for all Clients in the central database itself by changing a "Flag".

[lamontagne]

Oh... well if you need to keep the code private on the clients yet distribute it you're going to have to compile/encrypt in some way and with php it's never going to be 100% as someone will always be able to nullify the script or decompile... if you can somehow use a REST/SOAP api and put the confidential parts of those functions on the central server (as other functions) and simply pass back the results that would work, but of course you'll have a network load and the design may end up very awkward... At the end of the day php will read it as php so leaving it as php will always be reversible... I suppose you could create a php extension (dll) and simply distribute that and use it in the php code (http://www.php.net/manual/en/function.dl.php) but even then others would just have to implement the dll as well, unless you include a few routines to phone home and do some checks (send remote ip and remote os and whatnot)... never built a php extension but it seems plausible...


Yeah, come to think of it a php extension that phones home would be your best bet....

[perkiset]

The only issue with the php extension is that you need to put it into the php.ini and restart Apache for it to take effect. I've written a few and they ROCK - i nfact, this is my standard trick when PHP just doesn't have the juice to do (something) really fast. No worries for someone like me, but if you are to sell it to people that are using normal hosting, then getting the hosting company to install a custom php extension might prove ... difficult. In fact, unless you really know the details of the target machine and can precompile it then you'd want to distribute the source and let it get built on the destination machine ... which eliminates the advantage of compiled obfuscation entirely.