Let's say you want to regularly synchronize a directory from one machine to another.  One common solution is to setup SSH keys with no passwords and then schedule rsync to push the files over the network.  

The major problem with this is someone could “steal” the private key from the source machine and use it to access the destination machine anytime.  The solution I will present involves using ssh-agent via keychain to automate setting up a password protected private key that cron can be used in cron scripts.

First create an SSH keypair with ssh-keygen.  Use the defaults and enter a reasonable password when it asks:

$ ssh-keygen

This creates an RSA keypair of the default length with the default name of id_rsa and id_rsa.pub.

If you already have a keypair, you can add or change the password with ssh-keygen like so:

  $ ssh-keygen -f ~/.ssh/id_rsa -p

Next append the public key to the ~/.ssh/authorized_keys files for the desired user on the destination host.  One not so well known but very convenient way to do this is with ssh-copy-id:

 $ ssh-copy-id user@machine

This will copy our id_rsa.pub file to the specified user on the specified host.  See the manual page for ssh-copy-id for more options.  ssh-copy-id will append the public key only if it does not already exist in the authorized_keys file.  It will also create the necessary directories and

I recommend using the least privilege principle here and not allowing anything to ssh into a host as root.  In other words the destination user should never be root.  You should try ssh with this key and make sure it works as expected.

Next setup keychain.  Keychain is the glue between ssh-agent which stores ssh keys and UNIX execution environment.  Keychain is included in most distros package repositories.  On modern Debian style machines you can add it with apt-get:

  $ apt-get install keychain

Keychain hooks into your shell environment via the .rc file.  Most linux users use bash so add this to either .bashrc or .profile:

  eval `/usr/bin/keychain --quiet --eval`

This executes the keychain program that searches for an ssh-agent and starts one if one is not already running.  Keychian emits some shellcode which defines two shell variables SSH_AUTH_SOCK and SSH_AGENT_PID.  The eval runs this code in your current environment and makes these variables available to commands executed later.  

You should fire up another terminal window and log in to the source box a second time.  Don’t close the first window until you have logged in and proven you don’t have any errors preventing you from logging in.  

Once you login a second time, add the key to your keychain:

  $ ssh-add

You will be asked for the keys password and it will be saved by the ssh-agent which keychain made for you.  You can view which ssh keys an agent has with `ssh-add –l` and you can delete a key from an agent with `ssh-add –d `.  See the man page for more options including how to add more than a single key.

The value add of keychain is that it finds the same ssh-agent for you the next time you login.  Try sshing to the destination machine and you should not get asked for a password.  Logout and login again the ssh a second time and again you will not be asked for a password because ssh reuses the same ssh-agent which already as the opened private key in memory.

Since the ssh key is password protected it is useless to a rouge user unless they can also get the password to unlock it.  Of course they can use the current environment to launch attacks but restricting the commands allowed in the destination authorized_keys file can mitigate this as well.

Finally we will hook this into cron.  Keychain creates a couple shell scripts that aid in setting up the proper environment when .rc files are not used.  Follow this example for cron:

  # Prepend this to your cron jobs

   . ~/.keychain/myhostname-sh && rsync ...

Where myhostname is the actual name returned by the hostname command.  

If you are in a shell script try this:

  HOSTNAME=`hostname`

  KEYCHAIN=${HOME}/.keychain/${HOSTNAME}-sh

  if [ -f $KEYCHAIN ]

  then

    . ${HOME}/.keychain/${HOSTNAME}-sh

  fi

Congratulations, you now have an environment that can automatically sync files without human intervention using a secure key system that is password protected yet still automated.  

Next steps would be restricting the keys use on the destination host by adding restrictions to the authorized_keys file.

This is my recipe for tom ka gai.  I use ingredients which I am more likely to have on hand.  The authentic Thai recipe has galangal and kaffir lime.  I still call for lemongrass which is a little exotic, but you could probably get away without it.

I got the basic recipe from a Thai lady who owned a restaurant near my old apartment.  I had it there for the first time and I was hooked.  She kindly told me the ingredients and a basic method and I have been fine tuning it for almost 10 years. Don't overcook.

 The flavor is sour and spicy carried by the richness and slight sweetness of the coconut milk. 

Ingredients:

• 1.5 pound chicken meat.  breast or leg meat cut in medium chunks.
• 2 limes.  we'll use the juice and the zest so get fresh limes.  organic preferred (no pestisides).
• 2 stalks of lemon grass. crush it with your knife and then cut into match sticks.
• 2 thumb size pieces of ginger cut into thin slices.
• 1/2 teaspoon of dried red pepper.  the stuff you shake on pizza.  I add it for extra heat.
• 1 large white or yellow onion cut into medium chunks
• 2 cups of button mushrooms.  I cut them in half lengthwise so they still look like mushrooms.
• 1-2 tomatoes cut into chunks.  about the same amount as the onion.
• 3-4 jalapeno peppers cut in rings.  I keep the seeds for heat.
• cilantro for decoration and aroma.
• 1 can low sodium chicken broth.
• 2 cans of coconut milk.

Method:

• Marinate the chicken in the lime juice and lime zest.  It will denature slightly from the lime juice but not a lot.
• Simmer the lemongrass, ginger and red pepper in the chicken broth and a little water.  Get the flavor out of the lemongrass and ginger.
• Strain the above keeping the liquid.  You can discard solids.
• Add the coconut milk to the broth.  Bring up to a simmer and add onions.  We want to soften them but not untill they are mushy.
• Add the jalapenos.
• Put the chicken and lime marinade in.  
• Add the mushrooms.  We want to cook out the raw flavor but leave the toothsome body.
• Add the tomato.  Don't simmer too long.  Can pretty much kill the flame and let the residual heat do the job.

Serve in large bowl with a few cilantro leaves for garnish.  I scoop in a little fragrant Thai jasmine rice and call it a meal.

Just a little story about some of the silly things I do.

I log onto this production box which has an app server and a database.  Now this box is a linux box and it is on the AWS cloud.  AWS had some serious issues a few weeks before this so this box had been reuild in a hurry, rode hard and put away wet.  A coworker tells me that psql the postgresql client is erroring out when he truies to run it.

So I log in and try it myself and sure enough I get this error:

/usr/lib/postgresql/8.4/bin/psql: symbol lookup error: /usr/lib/postgresql/8.4/bin/psql: undefined symbol: rl_filename_completion_function

I know postgres was installed via backports (this box predates me) so there are some issues with the dynamic libraries. First thing I try is an apt-get update; apt-get upgrade to pull it forward.  This causes 50 or so packages to get upgraded but it does not solve the problem.

So we go to the next level. I do some googling on rl_filename_completion_function and I come across this page on readline where it says:

Function: char * rl_filename_completion_function (const char *text, int state)
A generator function for filename completion in the general case. text is a partial filename. The Bash source is a useful reference for writing custom completion functions (the Bash completion functions call this and other Readline functions).

I go looking around in headers and ldd psql to see what libraries it pulls in directly but to no avail. I just need this to work for a little longer so time to start "hacking".

What we're going to do is replace that readline function with a stub and prelink our stub into psql so it will run.  Fire up vi (vim really), paste the function definition in from the man age and edit until it looks like this:

char _blank = ' ';
char * rl_filename_completion_function (const char *text, int state)
{
  return &_blank;
}

Next we have to compile it into a share library

linux #>gcc -fPIC -c dwa.c 

linux #>gcc -shared -Wl,-soname,libdwa.so.1 -o libdwa.so.1.0 dwa.o

Now we have a shared library libdwa.so.1 which "implements" the function we need to make this damn thing work, but the linker has no reason to ever pull this library in because it is not referenced by the psql libraries or their decendants.

At this point I could go whole hog and rebuild the libreadline.so with my stub in there but this entails taking the libreadline.a, un-ar-ing it to get a bunch of .o files then putting them back together as a .so. Prey that the object files were compiled relocatable as they often are.

Instead we'll us a little known trick LD_PRELOAD. This specifies shared libraries that override the normal ones thus changing the default behavior.

linux# export LD_PRELOAD=/root/dwa/libdwa.so.1.0 
linux# psql -U postgres
psql (8.4.7)
Type "help" for help.

postgres=#

At this point we can run the postgres client and get our work done. I am realizing what a POS this machine is and why do I subject myself to these things?

I needed a way to mirror a directory between to shared nothing servers.  I started out uing cron to fire off rsync, but the application required the files be synched "immediately".

Enter inotify.  I installed the inototify-tools go get a shell interface to filesystem events from the kernal.  A couple lines of shell script later and we have a daemon like utility which waits for relevant events and rsyncs the required minimum files.

Run like so:

nohup sync.sh /full/path/to/dir user@host & > /dev/null 2>&1

After I wrote this I found a couple other scripts and programs which do the same or similar things, but this one is pretty minimal and I like that.

• http://code.google.com/p/lsyncd/
• http://ebalaskas.gr/blog/?page=PIrsyncD
• http://www.linuxawy.org/node/13

#!/bin/sh

set -e
set -u

BASEDIR="$1"
REMOTE_HOST="$2"
RSYNC="rsync -av -e ssh --delete "

# Initial sync
$RSYNC ${BASEDIR}/* ${REMOTE_HOST}:${BASEDIR}/

# Wait for individual file events and keep in sync
inotifywait --format '%e %w' -e close_write -e move -e create -e delete -qmr $BASEDIR | while read EVENT DIR
do
  # Fork off rsync proc to do sync
  $RSYNC ${DIR}/* ${REMOTE_HOST}:${DIR}/ &
done

In some ways it's great when people dont understand something but maybe you do.  Like the internet in 1999.  Some people said it was "CB radio with computers" and it would fade away.  I knew that wasn't true, and that gave me power.  I feel the same way now about the cloud.

This week at work we got a request to process 7000 documents off a partners web site.  Problem is getting each of these documents requires a multistep navigation process to find and download the correct page.  I asked the engineer who wrote this program how long it took on average to fetch each of these.  "About a minute" she replied.

A little back of the envelope math says it would take about 5 days to fetch all of these serially.  The problem is we were supposed to deliver the updated data on Monday and it was Friday.  Typical.  We could divide and conquer the problem by running on multiple computers in parellel, but then we would have to source extra computers, set up a bunch of software on them and baby sit them over the weekend.  This still wouldn't work since we couldn't get the hardware and set them up one by one in time.

To The Cloud!

So you're getting done quickly but are you saving work?  YES!  I was able to spin up a single test instance and the engineers teamed up to instal and debug the custom software.  Once we had it working, we tested it by loading a dozen documents and bundled a complete machine image of our test machine.  We were able to test, debug and build a gold master out of that one machine.

The final obsticle was the fact our software was actually driving a web browser in a windowing environment.  Can you run desktop software in the cloud?  YES!  I installed FreeNX server on the prototype worker machine and the client on my laptop and soon enough I had a desktop machine running somewhere in a datacenter I've never set foot in on a virtual machine I just ordered an hour ago with a custom image my coworkers and myself threw together in a couple hours.

I did fire up those 10 workers and connected to their desktop remotely.  They chomped away on the work in parellel over Friday night and it was done early Saturday.  I collected the data and shut the instances down.  No scrounging for spare hardware, no installing software over and over, no all nighter, no drama.

This experience has left me with that feeling that I had years ago talking to the internet doubters.  People may say it's a solution looking for a problem or it's all hype.  I think it will fundamentally change the way we work.  Imagine all the data on your harddrive stored securely (encrypted) in the cloud.  It's getting backed up, always available from anywhere.  Now imagine your desktop computer has been replaced by a super computer with virtually limitless CPU, and memory.  Your spreadsheets crunch numbers so hard the digits are sobbing slightly.  This power is available to you on your home computer, work computer, laptop in the cafe and your smartphone everywhere.

So don't fear the cloud.  Your data is just about as safe in the cloud on Amazons servers or Google as it is on your harddrive.  It is certainly getting backed up better and much better protected from loss.  Your data are much less likely to "get hacked" on the cloud with professionals securing it that it is on your virus infected PCwhich your allow some PC handyman full access to tune up for you.

Once you stop worrying and feel the awesomeness, you will love the cloud.

I've been seeing a lot of videos from friends on FB, so I give you this Beastie Boys Daft Punk