Archive | Featured

First Work Day of Fall ‘09

Posted on 30 August 2009 by Shawn

Work day, come one come all. We’re going to get this cluster working and setup a modern webserver on which we my host our sites.

Details:
Sunday Sep. 6th
ECE232
Noon – Until we all get tired and leave!

Comments (0)

Tags: , ,

x264-MPI Rough Demo Code

Posted on 17 March 2009 by Shawn

I posted this message to the x264 list this evening (see below).

I just had the thought that there were a couple of folks in
ua-developers interested in cluster programming techniques. For all it's
faults, MPI (with C, C++ or Fortran) is still the 'gold standard' in
academia.

While my code is pretty bad, it just may provide a simple introduction
to message passing with MPI for ridiculously parallel jobs (which I made
video encoding into by arbitrarily chunking the input).

It's not much, but maybe someone wants to pick this project up on their
shoulders? Internet fame and glory await (distributed encoding is always
a popular topic on the internet, piracy being what it is these days).

The only thing this really needs is a frame server (define MPI message
that properly represents a frame, then some simple logic to send them)
and then working to make the frame server intelligent (i.e. work closely
with the deeper logic bits of x264) [Well, maybe some optimization to
make sure that the frame server overhead doesn't clobber the
parallelization gains].

I am too busy at work to make this a priority, but I'd be willing to
help out where I can. Hopefully, soon enough you guys will have a proper
cluster at HACKS on which to try this stuff out.

Peace,
Shawn
Subject: Re: [x264-devel] implementing Cluster farming From: Shawn Nock <nock@nocko.net>Date: Tue, 17 Mar 2009 23:06:42 -0700 To: Mailing list for x264 developers <x264-devel@videolan.org> I actually worked on this for a bit as a weekend hack / POC. I implemented a *very basic* [read: not looking for criticism, I know it is terrible and borderline worthless] x264-mpi workflow last year. If anyone cares I put it on my gitweb: http://git.nocko.net/?p=x264-mpi The primary commitdiff of interest is here: http://git.nocko.net/?p=x264-mpi;a=commitdiff;h=2200ac1a260a20085b2df588936911338f486f95 although there are useful patches (multipass support) later on (look for commits by 'Shawn Nock'). There is no frame server (so it relies on shared storage). I am sure that it breaks a lot of optimizations (specifically scene detection, which is right out). In a nutshell, It counts the frames and splits the frames into groups equal to the number of requested MPI processes. Then each process encodes a frame group separately, outputting to a file with a sequential extension. Concatenating these files (I don't think I implemented concatenation in the primary process... memory fades) produces coherent output, but nothing resembling the baseline output of a normal x264 run. Caveats aside, It compiles on x86_64 (mpich2) and ia64 (sgi propriatary mpi for it's numalink technology). If you don't care that it butchers encoding efficiency (and ultimately the output file), the raw fps numbers are encouraging. I'd love to see a proper MPI support and I could provide several testing platforms if someone was seriously interested in doing this. Peace, Shawn

Comments (0)

Tags: , , , , ,

03/08/2009 Work Day Post-game!

Posted on 08 March 2009 by Shawn

This is a basic summary, a more thorough post is available here.

The head node now has a POC nfsroot that boots into init (which it pulls via NFS). We are having some reliability problems after that (NFS time-out errors at different places in the init script). However, we learned a lot about how disk-less Linux works and we have a plan to fix it in the near future! The boot server can now boot the nodes, and give us BIOS, kernel, and (if we can sort out the NFS time-out) login on the serial server.

We also did quite a bit of work on the EVA5000 and come up empty handed. I tried to access the serial interface, only to learn that it was diagnostics only (no documented management interface) and Jason stalled on the Windows2000 management utilities. We’ll try again soon, I suppose that in the worst case we could use the one giant LUN as-is and not break it up… but let’s hope we can use the space a little more intelligently.

I also fixed the boot-time networking config on the server we made for the ua-developers club. I fat-fingered a conf file entry and the network didn’t come back after Jason power cycled the rack (QA testing I am sure :) ).

Stay tuned, the fun stuff is on the horizon.

Comments (1)

HACKS Jabber/XMPP Server

Posted on 05 March 2009 by Shawn

A few weeks ago I set-up Openfire Jabber Server for HACKS. Really I set this up for Jason, he was looking for a server-side way to login to all his IM accounts and log a history. In any case, now it is up and anyone with a HACKS LDAP account can login. Just fire up a Jabber/XMPP client and use the JID <HACKS username>@hacks.arizona.edu with your LDAP password and you are in. It has most common transports enabled (AIM, Yahoo, Jabber, and ICQ) as well as the IRC transport. S2S is confirmed working with jabber.org, gtalk, and my domain (nocko.net).

For those that are interested it is running IgniteRealtime’s OpenFire Server 3.6.3 and it is running in tugarin.hacks.arizona.edu. If you are having trouble or you don’t have a HACKS LDAP account send me and e-mail (n...@hacks.arizona.edu). If you decide to use the service, give me a holla. My JID is nock@nocko.net .

Comments (0)

HACKS CA!

Posted on 04 January 2008 by admin

SSL is really neat. What sucks is that SSL certs cost monies and self-signed certs aren’t respected by browsers. Enter Shawn and I concocting a master plan where we create a CA for HACKS. Now, this does NOT mean you get free SSL certificates that validate in browsers out of the box. What it means is, you have a CA cert you can import into your browser and THEN when you use a HACKS issued SSL cert, you won’t get any “untrusted cert” alerts.

While I know this is pretty much non-news, I run some stuff under cover of SSL and I’m pretty stoked. I’ve attached the HACKS CA certificate to this post (strip the txt extension, Drupal is retarded). You download this file and then in your browser’s preferences (not going into detail here, be clever) you import it as a trusted CA, bingo bango, valid certificates fall from the heavens, or whatever.

Comments (0)

Tags:

LDAP HA Cluster is Operational

Posted on 09 October 2007 by Shawn

In order to ensure continued reliability for the LDAP directory server. I have configured the DNS cluster with a new service IP served by all the machines. Kiyo and Bahamut are now read-only slave servers to Smaug.

If the master fails, the others will answer read request and allow auth to continue to function. All machines and the main website have been updated to use the new dns entry ldap.hacks.arizona.edu->150.135.84.4 . Please update any applications that use Smaug directly to query the service address to take advantage of the new fault tolerance.

Comments (0)

Tags:

What is LDAP?

Posted on 05 September 2007 by Shawn

There has been a lot of talk about LDAP since we decided to roll it out for central HACKS authentication.

LDAP stands for Lightweight Directory Access protocol. LDAP is a protocol for exchanging objects over a variety of links. LDAP does not specify how the information is stored. So, an LDAP server is just a computer that takes formatted text objects and presents them over a socket in an organized way.

The primary uses for LDAP are central authentication, online phone books, and cryptographic keyservers.

Examples of LDAP servers @ the University of Arizona include NetID (authentication server for many campus services including e-mail, WebReg, employee link, …), UA Phonebook (ldap.arizona.edu; an online phonebook containing information about students, faculty and staff).

While the physical storae of the data is unregulated by the LDAP standards docs, the logical representation is tightly governed. Schema are defined that restrict and mandate what types of objects may/can exist in the directory.

When manually interacting with an LDAP compatible server, you use a file called an ldif. An ldif file defines an instance of an object and describes its place in the directory looks like this (a user object):

dn: cn=Shawn Nock,ou=members,dc=hacks,dc=arizona,dc=edu
givenName: Shawn
sn: nock
cn: Shawn Nock
uid: nock
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: top
mail: n...@hacks.arizona.edu
userPassword: deadbeef(password hash)deadbeef
shell:/usr/local/bin/bash
home: /home/nock

Description of terms (for an object of type posixUser):
cn= “Common Name” The name of the object instance
ou= “Organizational Unit” A container for objects
dn: “Distinguished Name” Describes where the object exists in the directory heirarchy. In the above example: Theobject named “Shawn Nock” exists in a container called “members” in the directory “dc=hacks,dc=arizona,dc=edu”
sn: “Surname” Last Name
givenName: First Name
objectClass: The object type. The above object inherits the requirements for a person (inetOrgPerson; allowing surname, givenname, mail), posix account (means the object get a uid [login name] userPassword [login password], login shell, home directory, etc.) and “top” a basic toplevel object which all objects inheirit (allows common name and distinguished name)

Once in the directory, HACKS uses a PAM (pluggable authentication module) to query the directory for a uid (login name) and compare the psasword for login to multiple machines. Assuming the password is correct, the server can pull all the relevant information (where the users home directory is , what shell they like, etc. and use it to customize their environment.

For example, the main web site uses uid and userPassword to authenticate a user and pulls in mail to allow the user to be contacted through a web form. given and surnames are used to mark a post by a user so others know who posted it.

More information about LDAP can be found from various sources on the internet. Try searching for OpenLDAP (the name of the LDAP server/client software HACKS uses).

Comments (0)

Tags:

HACKS Virtualization Environment

Posted on 09 June 2007 by admin

HACKS has rolled out Xen virtualization in order to facilitate easy provisioning of servers to club members.

Hardware:

  • 2x 8-way 700Mhz Xeon; 4GB RAM
  • 1x 8-way 550Mhz Xeon; 6GB RAM
  • 500GB Fibre Channel Multipath SAN storage

Comments (0)