Comments on: Parallel Processing in Python with processing http://mtrr.org/blog/?p=91 Notes on physics, computers, and physics with computers. Mon, 06 Sep 2010 00:24:25 +0000 http://wordpress.org/?v=2.5.1 By: Paul Boddie http://mtrr.org/blog/?p=91#comment-9534 Paul Boddie Fri, 16 Nov 2007 16:59:01 +0000 http://mtrr.org/blog/?p=91#comment-9534 I responded to Valery about the issues with pickling certain things. SQLAlchemy does some exotic things which don't help with pickling, and I suggested avoiding passing such objects between processes (or across "process boundaries", if you like). Without knowing more, I can't say whether that's an acceptable solution or not, however. I responded to Valery about the issues with pickling certain things. SQLAlchemy does some exotic things which don’t help with pickling, and I suggested avoiding passing such objects between processes (or across “process boundaries”, if you like). Without knowing more, I can’t say whether that’s an acceptable solution or not, however.

]]> By: stan http://mtrr.org/blog/?p=91#comment-9521 stan Sun, 04 Nov 2007 18:38:12 +0000 http://mtrr.org/blog/?p=91#comment-9521 It is possible that the object doesn't pickle properly, which is how all pretty much all the parallel processing modules pass arguments from one process to the other. Without knowing more about the details of your problem, that's a total guess. It is possible that the object doesn’t pickle properly, which is how all pretty much all the parallel processing modules pass arguments from one process to the other. Without knowing more about the details of your problem, that’s a total guess.

]]> By: Valery http://mtrr.org/blog/?p=91#comment-9520 Valery Sun, 04 Nov 2007 17:45:06 +0000 http://mtrr.org/blog/?p=91#comment-9520 Paul Boddie wrote really an excellent module. I am happy to use it already. Finally I met the situation, when my function supplied as a first argument into the pmap call returns an class instance. When I try to access the result the exception is thrown. The unlucky class is "SQLSoup" of SqlSoup extension of well-known SQLAlchemy ( http://www.sqlalchemy.org/docs/04/documentation.html#plugins_sqlsoup) I can't reproduce situation yet in a simpler form. Maybe anyone could give me a hint? Valery Paul Boddie wrote really an excellent module. I am happy to use it already.

Finally I met the situation, when my function supplied as a first argument into the pmap call returns an class instance. When I try to access the result the exception is thrown. The unlucky class is “SQLSoup” of SqlSoup extension of well-known SQLAlchemy ( http://www.sqlalchemy.org/docs/04/documentation.html#plugins_sqlsoup)

I can’t reproduce situation yet in a simpler form. Maybe anyone could give me a hint?

Valery

]]> By: Jesse http://mtrr.org/blog/?p=91#comment-9478 Jesse Wed, 10 Oct 2007 19:47:28 +0000 http://mtrr.org/blog/?p=91#comment-9478 I pinged the processing module author - he's game for a PEP to be put together I pinged the processing module author - he’s game for a PEP to be put together

]]> By: stan http://mtrr.org/blog/?p=91#comment-9477 stan Tue, 09 Oct 2007 19:00:04 +0000 http://mtrr.org/blog/?p=91#comment-9477 I haven't posted it yet since I need to take another pass through it to make sure I didn't break anything. Then I can send it to the processing developer to see what he thinks. I haven’t posted it yet since I need to take another pass through it to make sure I didn’t break anything. Then I can send it to the processing developer to see what he thinks.

]]> By: Peer http://mtrr.org/blog/?p=91#comment-9476 Peer Tue, 09 Oct 2007 06:51:58 +0000 http://mtrr.org/blog/?p=91#comment-9476 So where can we find your modified processing.PipeQueue code? Did you submit it to the processing developers? So where can we find your modified processing.PipeQueue code? Did you submit it to the processing developers?

]]> By: stan http://mtrr.org/blog/?p=91#comment-9472 stan Mon, 08 Oct 2007 00:02:54 +0000 http://mtrr.org/blog/?p=91#comment-9472 Oh, and I finally hacked processing so that processing.PipeQueue has infinite queue depth. So on a dual core Linux system, you get the following results: wf-1-gala.py 3.422 wf-2.py 1.049 wf-3.py 1.223 wf-4.py 1.103086 wf-5.py 0.7303121 wf-6.py 0.5413789 wf-4-processing.py 0.7166 So as you might have expected, processing is on par with the multi-process version that used pipes (wf-5) and slower than the version (wf-6) which used shared memory. Oh, and I finally hacked processing so that processing.PipeQueue has infinite queue depth. So on a dual core Linux system, you get the following results:

wf-1-gala.py 3.422
wf-2.py 1.049
wf-3.py 1.223
wf-4.py 1.103086
wf-5.py 0.7303121
wf-6.py 0.5413789
wf-4-processing.py 0.7166

So as you might have expected, processing is on par with the multi-process version that used pipes (wf-5) and slower than the version (wf-6) which used shared memory.

]]> By: stan http://mtrr.org/blog/?p=91#comment-9471 stan Sun, 07 Oct 2007 23:58:31 +0000 http://mtrr.org/blog/?p=91#comment-9471 pprocess looks pretty neat! Definitely a different approach than trying to emulate Thread(). The common use cases seem to be very streamlined. pprocess looks pretty neat! Definitely a different approach than trying to emulate Thread(). The common use cases seem to be very streamlined.

]]> By: Paul Boddie http://mtrr.org/blog/?p=91#comment-9470 Paul Boddie Sun, 07 Oct 2007 22:50:42 +0000 http://mtrr.org/blog/?p=91#comment-9470 I had to try this with the pprocess module [1], too, and I've uploaded my version here: http://www.boddie.org.uk/python/pprocess/pwf-4.py I've taken the liberty to divide the chunks into sizes proportional to the number of processes involved. See below for a discussion of this modification. Running on a 2 CPU machine I get 0.95s with 1 computation process, 0.60s with 2 processes, 0.62s with 4 processes. Meanwhile, taking wf-4.py as a comparison, I get 1.07s for 1 thread, 1.01s for 2 threads, 1.00s for 4 threads. This is on a machine with decent I/O and with the disk cache presumably nicely warmed up. Thus, the threaded version doesn't really get much benefit when running with lots of threads. If I use the default chunk size, more parallel invocations or communications are required, whether or not processes are reused or spawned anew. With the pprocess module, this has a large impact on the performance, probably because the inter-process communications could do with some optimisation. Still, if you diff wf-4.py with pwf-4.py, you'll see that I actually remove code because of certain conveniences provided in the pprocess module, as described in the tutorial [2] [1] http://www.python.org/pypi/pprocess [2] http://www.boddie.org.uk/python/pprocess/tutorial.html I had to try this with the pprocess module [1], too, and I’ve uploaded my version here:

http://www.boddie.org.uk/python/pprocess/pwf-4.py

I’ve taken the liberty to divide the chunks into sizes proportional to the number of processes involved. See below for a discussion of this modification.

Running on a 2 CPU machine I get 0.95s with 1 computation process, 0.60s with 2 processes, 0.62s with 4 processes. Meanwhile, taking wf-4.py as a comparison, I get 1.07s for 1 thread, 1.01s for 2 threads, 1.00s for 4 threads. This is on a machine with decent I/O and with the disk cache presumably nicely warmed up. Thus, the threaded version doesn’t really get much benefit when running with lots of threads.

If I use the default chunk size, more parallel invocations or communications are required, whether or not processes are reused or spawned anew. With the pprocess module, this has a large impact on the performance, probably because the inter-process communications could do with some optimisation.

Still, if you diff wf-4.py with pwf-4.py, you’ll see that I actually remove code because of certain conveniences provided in the pprocess module, as described in the tutorial [2]

[1] http://www.python.org/pypi/pprocess
[2] http://www.boddie.org.uk/python/pprocess/tutorial.html

]]> By: stan http://mtrr.org/blog/?p=91#comment-9468 stan Sun, 07 Oct 2007 18:07:46 +0000 http://mtrr.org/blog/?p=91#comment-9468 I think starting the PEP process (well, with the author's blessing) would be a great idea. This really does belong in the standard library alongside the threading module, so we have a simple alternative to threads when appropriate. Certainly, when forking on a single machine, it seems perfect. I need to play more to see how easy working with remote clients is. The remote execution problem is harder, since there is no inter-machine fork (unless you are using some kind of single-system-image operating system). You have to start separate processes on the other machines, worry about authentication, and all that mess. Here the slightly more awkward approach of Parallel Python seems sensible since it works the same way for local or remote worker processes. Too bad there is no obvious way to streamline the processing module for remote usage. I do have this strange desire to try to use Stackless with this module to distribute tasklets automatically to worker threads... I think starting the PEP process (well, with the author’s blessing) would be a great idea. This really does belong in the standard library alongside the threading module, so we have a simple alternative to threads when appropriate. Certainly, when forking on a single machine, it seems perfect.

I need to play more to see how easy working with remote clients is. The remote execution problem is harder, since there is no inter-machine fork (unless you are using some kind of single-system-image operating system). You have to start separate processes on the other machines, worry about authentication, and all that mess. Here the slightly more awkward approach of Parallel Python seems sensible since it works the same way for local or remote worker processes.

Too bad there is no obvious way to streamline the processing module for remote usage. I do have this strange desire to try to use Stackless with this module to distribute tasklets automatically to worker threads…

]]>