I posted this message to the x264 list this evening (see below).

I just had the thought that there were a couple of folks in
ua-developers interested in cluster programming techniques. For all it's
faults, MPI (with C, C++ or Fortran) is still the 'gold standard' in
academia.

While my code is pretty bad, it just may provide a simple introduction
to message passing with MPI for ridiculously parallel jobs (which I made
video encoding into by arbitrarily chunking the input).

It's not much, but maybe someone wants to pick this project up on their
shoulders? Internet fame and glory await (distributed encoding is always
a popular topic on the internet, piracy being what it is these days).

The only thing this really needs is a frame server (define MPI message
that properly represents a frame, then some simple logic to send them)
and then working to make the frame server intelligent (i.e. work closely
with the deeper logic bits of x264) [Well, maybe some optimization to
make sure that the frame server overhead doesn't clobber the
parallelization gains].

I am too busy at work to make this a priority, but I'd be willing to
help out where I can. Hopefully, soon enough you guys will have a proper
cluster at HACKS on which to try this stuff out.

Peace,
Shawn

Subject: Re: [x264-devel] implementing Cluster farming
From: Shawn Nock <nock@nocko.net>Date: Tue, 17 Mar 2009 23:06:42 -0700
To: Mailing list for x264 developers <x264-devel@videolan.org>

I actually worked on this for a bit as a weekend hack / POC. I
implemented a *very basic* [read: not looking for criticism, I know it
is terrible and borderline worthless] x264-mpi workflow last year.

If anyone cares I put it on my gitweb: http://git.nocko.net/?p=x264-mpi
The primary commitdiff of interest is here:
http://git.nocko.net/?p=x264-mpi;a=commitdiff;h=2200ac1a260a20085b2df588936911338f486f95
although there are useful patches (multipass support) later on (look for
commits by 'Shawn Nock').

There is no frame server (so it relies on shared storage). I am sure
that it breaks a lot of optimizations (specifically scene detection,
which is right out).

In a nutshell, It counts the frames and splits the frames into groups
equal to the number of requested MPI processes. Then each process
encodes a frame group separately, outputting to a file with a sequential
extension. Concatenating these files (I don't think I implemented
concatenation in the primary process... memory fades) produces coherent
output, but nothing resembling the baseline output of a normal x264 run.

Caveats aside, It compiles on x86_64 (mpich2) and ia64 (sgi propriatary
mpi for it's numalink technology). If you don't care that it butchers
encoding efficiency (and ultimately the output file), the raw fps
numbers are encouraging.

I'd love to see a proper MPI support and I could provide several testing
platforms if someone was seriously interested in doing this.

Peace,
Shawn