SourceForge Logo
ssh_tunnel


HISTORY
ssh_tunnel has been developed over a three year period. It was first written as a hideous tcsh script that actually worked. The tcsh version was 1.x. It was then ported and mostly rewritten for use with the bash shell. This was a huge improvement in reliability and maintainability. The bash version was 2.x. Next it was ported to perl, its current form, as version 3.x. It has been extensively tested on Linux, Solaris, and Cygwin. It is very stable on Linux and Solaris. Cygwin works okay, but it is less stable.


DESCRIPTION

ssh_tunnel is a cron driven perl script that establishes and maintains an ssh tunnel between a client and a server. The tunnel provides a secure connection between an ssh client inside the firewall to an ssh server outside the firewall. It is assumed that the user has configuration control of the ssh server. Local and remote port forwarding through the tunnel make it possible to connect to any machine on the LAN side of the firewall. The tunnel uses RSA public/private keys for authentication. ssh_tunnel has been used successfully on Cygwin, Solaris, and Linux systems. Since it is written in perl it should be portable to other unix based systems. ssh_tunnel uses standard perl functionality. No additional perl modules are required. It depends on native programs such as ssh, netstat, ps, etc. It checks for these programs when starting up and exits with an error if they aren't found.

ssh_tunnel has several unique features. Its operation is controlled by a configuration file "tunnel.conf" on the ssh server machine which is outside the firewall. The client periodically does a binary compare of the tunnel.conf on the server with the local copy. If they are different ssh_tunnel on the client shuts down the tunnel and restarts the tunnel using the new tunnel.conf. ssh_tunnel can be disabled until needed by simply setting tunnel state to "disabled" in tunnel.conf.

ssh_tunnel can update a webpage with its current status. The webpage is updated using scp and can be on any machine accessible using RSA public/private keys. Here is a sample webpage. The webpage contains a lot of useful information about the tunnel's operation and the system on which it is running. It has current users and uptime, current netstat, current processes running, statistics for the loader and the tunnel, the contents of the crontab file, the contents of loader.conf, and the contents of tunnel.conf.

ssh_tunnel sends email to one or more email addresses when events occur such as the tunnel becoming disconnected or when restarting. Here is a sample email. The subject line has "ssh:" in it to make it easy to filter in an email client.

ssh_tunnel maintains logfiles on both the server and client. These logfiles are automatically compressed and rotated weekly with the last four being retained. Tailing the logfile makes it possible to figure out what is going on when problems arise. Debuggin is tricky to do for cron started processes without logfiles. Debugging information can be turned on by setting bits in a file "debug_flags". Invoke ssh_tunnel with the "-hd" option to see a description of how this works. Debug can be turned on and off for a running ssh_tunnel simply by changing the contents of "debug_flags". Here is part of a logfile.

ssh_tunnel.pl is a single perl file that runs in one of four modes depending on how the script is invoked. The script is well documented and has built in help text explaining step by step how to setup and test a tunnel. A brief description of the four operating modes is given below:

  • "cron" - cron Runs on ssh client. When the program name on the command line is ssh_tunnel.pl it is internally changed to "cron". This is the normal way to start ssh_tunnel. It is intended to be done from a crontab. It checks if "loader" is still running and launches a new loader if no loader is running or if loader hangs. Cron time between runs must be longer than sleep time. All the other programs are invoked automatically by ssh_tunnel as it starts up.

  • "loader" - loader runs on ssh client. cron starts the loader. It launches "tunnel". It checks ssh server for updated tunnel.conf and restarts "tunnel" when new file is found. It does server heartbeat verification on the client side and restarts "tunnel" upon heartbeat failure or if "tunnel" pid is inactive.

  • "tunnel" - tunnel runs on ssh client. Started by "loader". It establishes the ssh link with the ssh server using port forwardings defined in tunnel.conf. It writes the client heartbeat to the server through the ssh tunnel. It runs forever until killed by "loader" or kills itself if it detects incorrect pid or that the client heartbeat can't be written through the ssh tunnel.

  • "pulse" - pulse runs on ssh server. Started by "tunnel" when it establishes the ssh link. It does client heartbeat verification on the server side. It writes the server heartbeat to the client through the ssh tunnel. It runs forever until the tunnel is killed, heartbeat verification fails, it detects incorrect pid, or detects that the server heartbeat can't be written through the ssh tunnel.

    ssh_tunnel reliably detects when the connection has been broken and automatically restores the connection. It does this by periodically writing a "heartbeat" timestamp in both directions through the ssh tunnel. The "sleep time" between heartbeats is a configurable item in tunnel.conf. A typical sleep time is around four minutes, but it can be set much longer if desired. Shorter sleep times are required for connections that are prone to frequent drop outs. The last four timestamps are compared to each other. If the tunnel is working correctly the timestamps will be different. If they are all the same on the client side the tunnel is stopped and restarted. If they are all the same on the server side then pulse exits which will force the tunnel to restart.


    INSTALLATION

    Before this tunnel will work there are several configuration steps that must be done on both the ssh client and ssh server machines. Follow these steps starting on the ssh client.

    1. All files are in the same directory as ssh_tunnel.pl on both the client and server. The directory names are important. It must have the name of the other machine in it. For example, if the server's name is "screamer" and the client's name is "vorkosigan" then the installation path on the client could be:

        /home/jlarsen/ssh_tunnel/screamer

    On the server the installation path could be:

        /home/jlarsen/ssh_tunnel/vorkosigan

    The machine name is used when searching the output of "ps". On Solaris machines the full path to the ssh_tunnel.pl program must be no more than 66 characters. Solaris 8 truncates redirected ps output to 127 characters. The machine's name will be lost if the absolute path is longer than 66 characters.

    2. On the client execute ssh_tunnel.pl for the first time in the installation directory. You might need to edit the first line of ssh_tunnel.pl so that the path to perl is correct. The first time you run ssh_tunnel.pl it creates the following files and symbolic links. Click on the links to see samples of the files:
        @loader
        loader.conf
        loader.pid
        logfile
        @pulse
        ssh_tunnel.conf
        @tunnel
        tunnel.conf
        tunnel.pid

    3. Edit the files "ssh_tunnel.conf", "loader.conf", and "tunnel.conf". Template files are created automatically when ssh_tunnel.pl is run for the first time. Follow the instructions in each file. Step 9 copies your edited "tunnel.conf" and "loader.conf" from the ssh client to the ssh server. The control copy of tunnel.conf is on the server. During tunnel operation if this file is modified on the server, it is copied by the "loader" to the client, and the "loader" stops and restarts the "tunnel". This makes it possible from the server side to make changes in the port forwardings, email addresses, and email thresholds. The "loader" does a binary compare of the remote and local copies of tunnel.conf. If the content changes then the file is copied over and the tunnel is restarted.

    4. The next step is to get all the ssh connections working. Use the "ssh-keygen" utility to generate an rsa public/private key pair. Name the key something like "id_rsa.client_name" so it is easily identifiable. Do not give the key a pass phrase. The ssh connections all require a public/private key. The ssh client's public RSA key must be copied to the ~/.ssh/authorized_keys file on the ssh server and the ssh client. This key must not have a passphrase as that would require entering a password, which can't be done in a cron job. The corresponding private key must be on the client and server in the location specified in the loader.conf and tunnel.conf files. Directory and file permissions are important. The RSA private key file must only be readable by the owner of the file, ie. "chmod 400 filename". The directory must only be writeable by the owner, ie. 700. Use chmod to set permissions accordingly.

    5. There are ssh host definition sections in loader.conf and tunnel.conf. Once the information is entered it is time to verify that ssh connections can be made. Start on the client. First verify that a connection can be made using the "loader" host definition in the loader.conf file. On the command line use this command:

        ssh -F loader.conf loader

    The above command verifies the settings in the loader.conf file. You will be warned that the ssh server's key doesn't exist and you will be asked if you want to add it. Answer yes. Once this key is in the known_hosts file you won't be asked for it anymore. Type exit to return to the ssh client shell.

    6. Next verify that the "tunnel" host definition in the tunnel.conf file works. Use this command:

        ssh -F tunnel.conf tunnel

    The above command verifies the settings in the tunnel.conf file. You might be warned that the ssh server's key doesn't exist and you will be asked if you want to add it. Answer yes. Once this key is in the known_hosts file you won't be asked for it anymore. Don't exit this ssh connection at this time. It has set up port forwardings that are needed in the next step.

    7. Next verify that the "heartbeat" host definition in tunnel.conf works. Open another shell on the ssh client and cd to the installation directory. With the connection made in step 6 you can verify that the "heartbeat" host settings work. Use this command:

        ssh -F tunnel.conf heartbeat

    The above command verifies the settings in the tunnel.conf file. You will be warned that the localhost's key doesn't exist and you will be asked if you want to add it. Answer yes. Once this key is in the known_hosts file you won't be asked for it anymore. This step verifies that you are able to use the ssh port forwarding setup in step 6 to establish another ssh connection. If you can't automatically login to the ssh server you must resolve the problems before continuing. The program won't work unless you can successfully perform steps 5 through 11. Common problems are permissions issues on directories and files. Another common problem is mixing up the names of ssh client and ssh server in directory path names. Another common problem is not correctly putting the rsa public key in the authorized_keys file. Use the "-v" option on the ssh command line to turn on ssh verbose mode. This will provide information about what is wrong.

    8. Next verify the "webpage" host definition in tunnel.conf works. Use this command:

        ssh -F tunnel.conf webpage

    The above command verifies the settings in the tunnel.conf file. You might be warned that the ssh server's key doesn't exist and you will be asked if you want to add it. Answer yes. Once this key is in the known_hosts.webpage file you won't be asked for it anymore. Type exit to return to the ssh client shell. Note: You don't need to do this step if you aren't using the webpage feature.

    9. The next step is to copy all the needed files from the ssh client to the ssh server. On the server make sure that the installation directory defined in loader.conf exists. Use these commands to copy the files from the client to the server:

    scp -F loader.conf ssh_tunnel.pl loader:/server/installation/path/ssh_tunnel.pl
    scp -F loader.conf loader.conf loader:/server/installation/path/loader.conf
    scp -F loader.conf tunnel.conf loader:/server/installation/path/tunnel.conf
    scp -F loader.conf id_rsa.name loader:/server/installation/path/id_rsa.name
    scp -F loader.conf id_rsa.name.pub loader:/server/installation/path/id_rsa.name.pub

    The permissions of the files should be maintained by the scp transfer, but check to be sure that ssh_tunnel.pl is executable and that id_rsa.name is "read only" by the owner.

    10. On the ssh server you now need to run ssh_tunnel.pl so that it creates all the needed files and symbolic links. You might need to edit the first line of ssh_tunnel.pl so that it correctly gives the path to perl on your machine. ssh_tunnel.pl won't overwrite existing loader.conf and tunnel.conf files. It creates ssh_tunnel.conf which must be edited to match the host OS on which it is running.

    11. Now it is time to verify the "pulse" host definition in tunnel.conf. On the ssh client establish the "tunnel" connection to the server with the following command:

        ssh -F tunnel.conf tunnel

    The connection should be made without requiring any password. Change directory to the installation directory on the ssh server. Verify that the "pulse" host definition can be made using this command:

        ssh -F tunnel.conf pulse

    The connection back to the ssh client should be made without any password. You will be warned that the ssh client's key doesn't exist and you will be asked if you want to add it. Answer yes. Once this key is in the known_hosts file you won't be asked for it anymore. Type exit to return to the ssh server's shell and then type exit again to return to the ssh client's shell.

    12. The next step is to manually startup the ssh_tunnel on the ssh client and tail the logfile to verify proper operation. The logfile is the default location for all program output. In one shell on the client use the command "tail -f logfile" to follow what gets entered there. In another shell on the ssh client enter this command:

        /full/path/to/directory/ssh_tunnel.pl

    If all is working well you will see lots of information about "cron" checking for the "loader" and restarting it because it is dead. You will see log entries from "loader" as it checks for a live "tunnel" and restarts it because it is dead. You will then see the "tunnel" start up and it will launch an ssh link with the ssh server. You will see client and server heartbeats in the log. If you also tail the logfile on the ssh server you will see the "pulse" start up and see heartbeat information there.

    13. Once manual operation is confirmed it is time to create a crontab entry so the tunnel will be started automatically. A typical crontab entry looks like this:

        0,10,20,30,40,50 * * * * /full/path/to/directory/ssh_tunnel.pl

    This will verify the "loader" is running every 10 minutes and restart it if the "loader" pid isn't active. When everything is working correctly the cron task simply exits.

    14. Webpage update. There is a section in tunnel.conf to enable updating a webpage at a periodic rate. Read the instructions in tunnel.conf. The webserver can be on the same machine as the ssh server or it can be on a totally different machine. Configure the ssh host definition for "webpage" with the correct values. If the webserver is on the same machine as the ssh server then these values will be the same as those in loader.conf for the "loader" definition. If a different machine is used you will need to put the RSA public key in ~/.ssh/authorized_keys file as described in step 4. Use the following command to verify that you can establish an ssh connection to the webserver machine:

        ssh -F tunnel.conf webpage

    The above command verifies the settings in the tunnel.conf file. You will be warned that the webserver's key doesn't exist and you will be asked if you want to add it. Answer yes. Once this key is in the known_hosts.webpage file you won't be asked for it anymore.

    15. You can kill all processes associated with an ssh_tunnel with this command:

        /full/path/to/directory/ssh_tunnel.pl -k


    SUPPORTED PLATFORMS:
    This script has been used successfully on the following operating systems:
        Solaris 8
        Linux 2.x on Mandrake 9.1, Yellow Dog 4.1, Gentoo
        Cygwin 1.5.x on Windows 2000 and Windows XP

    Copyright: 2004-2006 by John R Larsen
    http://larsen-family.us     john@larsen-family.us
    Released under the same Artistic license as perl