In order to make the Grid Engine internal communication more secure, Grid Engine has been enhanced by a certificate based security layer that sits on top between the GDI and comlib layer. The security layer is based on OpenSSLs libcrypto and uses RSA for secret key exchange and rc4 (keylength 128bit) for encryption. The implementation originated from a diploma thesis realised in 1996. Instead of transfering messages in clear text, the messages are encrypted with a secret key. The secret key is exchanged via a public/private key protocol. The user presents its certificate to SGE to prove its identity and receives the certificate from SGE to be sure he is communicating to the correct system. After this initial announcement phase the communication is transparently continued in encrypted form. The session is valid only for a certain period then session has to be reannounced. Below you can find the steps to setup a CSP secured system.
The CSP mode is enabled in the product_mode file ($SGE_ROOT/{default | $SGE_CELL/common/product_mode) by adding the suffix -csp to sge or sgeee.
To setup the Certificate Security Protocol enhanced version of Grid Engine, the steps involved are very similar to the standard setup. Apart from the standard setup of Grid Engine the following additional steps are necessary:
The security enhancement can be either added to a previously configured system or the system can be setup to support CSP security during the installation. The installation is performed as usual except that the install script is called with the additional parameter -csp.
To generate the CSP certificates and keys the following information must be supplied:
Country code |
C=US |
The screen shots of an example installation outline the steps that are performed. If the system is already installed, the procedure looks very similar. Instead of using install_qmaster and install_execd, the script $SGE_ROOT/util/sgeCA/sge_ca -init is used. The screens below appear in the same manner.
First the Certificate Authority is created. The approach taken here, is to have a Grid Engine specific CA at the master host. The directory structure that contains security relevant information consists of two parts. Under $SGE_ROOT/{default | $SGE_CELL}/common/sgeCA the publicly accessible CA and daemon certificate are stored. The corresponding private keys are stored under /var/sgeCA/{sge_service | port$COMM_PORT}/{default | $SGE_CELL}/private. User keys and certificates go into /var/sgeCA/{sge_service | port$COMM_PORT}/{default | $SGE_CELL}/userkeys/$USER.
Initializing Certificate Authority (CA) for OpenSSL security framework ---------------------------------------------------------------------- Creating /scratch2/eddy/sge_sec/default/common/sgeCA Creating /var/sgeCA/port6789/default Creating /scratch2/eddy/sge_sec/default/common/sgeCA/certs Creating /scratch2/eddy/sge_sec/default/common/sgeCA/crl Creating /scratch2/eddy/sge_sec/default/common/sgeCA/newcerts Creating /scratch2/eddy/sge_sec/default/common/sgeCA/serial Creating /scratch2/eddy/sge_sec/default/common/sgeCA/index.txt Creating /var/sgeCA/port6789/default/userkeys Creating /var/sgeCA/port6789/default/private Hit to continue >> |
After setting up the directory structure the CA specific certificate and private key are generated. We use either pseudo random data from a special file or if available /dev/random for seeding the PRNG (pseudo random number generator, see http://www.openssl.org/support/faq.html and http://www.cosy.sbg.ac.at/~andi for additional info on random numbers)
Creating CA certificate and private key --------------------------------------- Please give some basic parameters to create the distinguished name (DN) for the certificates. We will ask for - the two letter country code - the state - the location, e.g city or your buildingcode - the organization (e.g. your company name) - the organizational unit, e.g. your department - the email address of the CA administrator (you!) Hit to continue >> |
Please enter your two letter country code, e.g. >US< >> DE Please enter your state >> Bavaria Please enter your location, e.g city or buildingcode >> Regensburg Please enter the name of your organization >> Gridware Please enter your organizational unit, e.g. your department >> Griders Please enter the email address of the CA administrator >> admin@griders.org You selected the following basic data for the distinguished name of your certificates: Country code: C=DE State: ST=Bavaria Location: L=Regensburg Organization: O=Gridware Organizational unit: OU=Griders CA email address: emailAddress=admin@griders.org Do you want to use these data (y/n) [y] >> |
Creating RANDFILE from >/kernel/genunix< in >/var/sgeCA/port6789/default/private/rand.seed< 1513428 semi-random bytes loaded Creating CA certificate and private key Using configuration from /tmp/sge_ca14364.tmp Generating a 1024 bit RSA private key .....++++++ ................++++++ writing new private key to '/var/sgeCA/port6789/default/private/cakey.pem' ----- Hit to continue >> |
After the installation of the CA infrastructure application and user certificates and private keys are created and signed by the CA for the admin user, for the pseudo daemon user and for the user root. Currently we use the outdated uniqueIdentifier field for tieing the Unix user name to the certificate, this will change in a future implementation.
Creating Daemon certificate and key ----------------------------------- Creating RANDFILE from >/kernel/genunix< in >/var/sgeCA/port6789/default/private/rand.seed< 1513428 semi-random bytes loaded Using configuration from /tmp/sge_ca14364.tmp Generating a 1024 bit RSA private key ...............++++++ ................++++++ writing new private key to '/var/sgeCA/port6789/default/private/key.pem' ----- Using configuration from /tmp/sge_ca14364.tmp Check that the request matches the signature Signature ok The Subjects Distinguished Name is as follows countryName :PRINTABLE:'DE' stateOrProvinceName :PRINTABLE:'Bavaria' localityName :PRINTABLE:'Regensburg' organizationName :PRINTABLE:'Gridware' organizationalUnitName:PRINTABLE:'Griders' uniqueIdentifier :PRINTABLE:'root' commonName :PRINTABLE:'SGE Daemon' emailAddress :IA5STRING:'none' Certificate is to be certified until Mar 5 13:50:57 2003 GMT (365 days) Write out database with 1 new entries Data Base Updated created and signed certificate for SGE daemons Creating RANDFILE from >/kernel/genunix< in >/var/sgeCA/port6789/default/userkey s/root/rand.seed< 1513428 semi-random bytes loaded Using configuration from /tmp/sge_ca14364.tmp Generating a 1024 bit RSA private key ............++++++ .................++++++ writing new private key to '/var/sgeCA/port6789/default/userkeys/root/key.pem' ----- Using configuration from /tmp/sge_ca14364.tmp Check that the request matches the signature Signature ok The Subjects Distinguished Name is as follows countryName :PRINTABLE:'DE' stateOrProvinceName :PRINTABLE:'Bavaria' localityName :PRINTABLE:'Regensburg' organizationName :PRINTABLE:'Gridware' organizationalUnitName:PRINTABLE:'Griders' uniqueIdentifier :PRINTABLE:'root' commonName :PRINTABLE:'SGE install user' emailAddress :IA5STRING:'none' Certificate is to be certified until Mar 5 13:50:59 2003 GMT (365 days) Write out database with 1 new entries Data Base Updated created and signed certificate for user >root< in >/var/sgeCA/port6789/default/userkeys/root< Creating RANDFILE from >/kernel/genunix< in >/var/sgeCA/port6789/default/userkeys/eddy/rand.seed< 1513428 semi-random bytes loaded Using configuration from /tmp/sge_ca14364.tmp Generating a 1024 bit RSA private key .............++++++ .....................................................++++++ writing new private key to '/var/sgeCA/port6789/default/userkeys/eddy/key.pem' ----- Using configuration from /tmp/sge_ca14364.tmp Check that the request matches the signature Signature ok The Subjects Distinguished Name is as follows countryName :PRINTABLE:'DE' stateOrProvinceName :PRINTABLE:'Bavaria' localityName :PRINTABLE:'Regensburg' organizationName :PRINTABLE:'Gridware' organizationalUnitName:PRINTABLE:'Griders' uniqueIdentifier :PRINTABLE:'eddy' commonName :PRINTABLE:'SGE admin user' emailAddress :IA5STRING:'none' Certificate is to be certified until Mar 5 13:51:02 2003 GMT (365 days) Write out database with 1 new entries Data Base Updated created and signed certificate for user >eddy< in >/var/sgeCA/port6789/default/userkeys/eddy< Hit to continue >> |
The security related setup of sge_qmaster is now complete and the installation procedure continues as usual:
SGEEE startup script -------------------- Your system wide SGEEE startup script is installed as: "/scratch2/eddy/sge_sec/default/common/rcsge" Hit to continue >> |
If the shared filesystem is not secure enough to hold the CSP security information in a place that can be accessed by the execution daemons as well, it is necessary to transfer the directory containing the daemon's private key and the random file to the execution host. Here it is installed locally before the setup of the execution daemon can be performed.
Copy the private keys to the execd host On the master machine as user root: # umask 077 # (cd /; tar cvpf /var/sgeCA/port6789.tar /var/sgeCA/port6789/default) On the execution host machine as user root: # scp /var/sgeCA/port6789.tar@<master host> / # (cd /; tar xvpf /var/sgeCA/port6789.tar ) Verify the file permissions: # ls -lR /var/sgeCA/port6789/ /var/sgeCA/port6789/: total 2 drwxr-xr-x 4 eddy other 512 Mar 6 10:52 default /var/sgeCA/port6789/default: total 4 drwx------ 2 eddy staff 512 Mar 6 10:53 private drwxr-xr-x 4 eddy staff 512 Mar 6 10:54 userkeys /var/sgeCA/port6789/default/private: total 8 -rw------- 1 eddy staff 887 Mar 6 10:53 cakey.pem -rw------- 1 eddy staff 887 Mar 6 10:53 key.pem -rw------- 1 eddy staff 1024 Mar 6 10:54 rand.seed -rw------- 1 eddy staff 761 Mar 6 10:53 req.pem /var/sgeCA/port6789/default/userkeys: total 4 dr-x------ 2 eddy staff 512 Mar 6 10:54 eddy dr-x------ 2 root staff 512 Mar 6 10:54 root /var/sgeCA/port6789/default/userkeys/eddy: total 16 -r-------- 1 eddy staff 3811 Mar 6 10:54 cert.pem -r-------- 1 eddy staff 887 Mar 6 10:54 key.pem -r-------- 1 eddy staff 2048 Mar 6 10:54 rand.seed -r-------- 1 eddy staff 769 Mar 6 10:54 req.pem /var/sgeCA/port6789/default/userkeys/root: total 16 -r-------- 1 root staff 3805 Mar 6 10:54 cert.pem -r-------- 1 root staff 887 Mar 6 10:54 key.pem -r-------- 1 root staff 2048 Mar 6 10:53 rand.seed -r-------- 1 root staff 769 Mar 6 10:54 req.pem Continue with the installation of sge_execd # cd $SGE_ROOT # ./install_execd -csp |
In order to let users use the CSP secured system, the user must have access to a user specific certificate and private key. They are generated by the administrator logged in as the user root on the master machine. To generate certificates and private keys for the user it is most convenient to create a file of the following format and to execute the commands in the next box.
Create a file containing the following information, e.g. myusers.txt eddy:Eddy Smith:eddy@griders.org sarah:Sarah Miller:sarah@griders.org leo:Leo Lion:leo@griders.org where the fields are: Unix user:Gecos field:email address As user root on the master machine do: % $SGE_ROOT/util/sgeCA/sge_ca -usercert myusers.txt % ls -l /var/sgeCA/port6789/default/userkeys dr-x------ 2 eddy staff 512 Mar 5 16:13 eddy dr-x------ 2 sarah staff 512 Mar 5 16:13 sarah dr-x------ 2 leo staff 512 Mar 5 16:13 leo Every user can then install its security related files in $HOME/.sge by: % source $SGE_ROOT/default/common/settings.csh % $SGE_ROOT/util/sgeCA/sge_ca -copy Certificate and private key for user eddy have been installed For every Grid Engine installation a subdirectory for the corresponding COMMD_PORT number is installed: % ls -lR $HOME/.sge /home/eddy/.sge: total 2 drwxr-xr-x 3 eddy staff 512 Mar 5 16:20 port6789 /home/eddy/.sge/port6789: total 2 drwxr-xr-x 4 eddy staff 512 Mar 5 16:20 default /home/eddy/.sge/port6789/default: total 4 drwxr-xr-x 2 eddy staff 512 Mar 5 16:20 certs drwx------ 2 eddy staff 512 Mar 5 16:20 private /home/eddy/.sge/port6789/default/certs: total 8 -r--r--r-- 1 eddy staff 3859 Mar 5 16:20 cert.pem /home/eddy/.sge/port6789/default/private: total 6 -r-------- 1 eddy staff 887 Mar 5 16:20 key.pem -r-------- 1 eddy staff 2048 Mar 5 16:20 rand.seed |
For checking and looking at certificates the following commands might be helpful.
% setenv ARCH `$SGE_ROOT/util/arch` Display a certificate: % $SGE_ROOT/utilbin/$ARCH/openssl x509 -in ~/.sge/port6789/default/certs/cert.pem -text Check issuer % $SGE_ROOT/utilbin/$ARCH/openssl x509 -issuer -in ~/.sge/port6789/default/certs/cert.pem -noout issuer= /C=DE/ST=Bavaria/L=Regensburg/O=Griders Org/OU=Testsystem at port 6789/CN=SGE Certificate Authority/uniqueIdentifier=CA/uniqueIdentifier=eddy/Email=eddy@griders.org Check subject % $SGE_ROOT/utilbin/$ARCH/openssl x509 -subject -in ~/.sge/port6789/default/certs/cert.pem -noout subject= /C=DE/ST=Bavaria/L=Regensburg/O=Griders Org/OU=Testsystem at port 6789/CN=eddy donetti/Email=eddy@gridders.org Show email of cert % $SGE_ROOT/utilbin/$ARCH/openssl x509 -email -in ~/.sge/port6789/default/certs/cert.pem -noout eddy@griders.org Show validity % $SGE_ROOT/utilbin/$ARCH/openssl x509 -dates -in ~/.sge/port6789/default/certs/cert.pem -noout notBefore=Sep 25 14:48:38 2001 GMT notAfter=Sep 25 14:48:38 2002 GMT Show fingerprint % $SGE_ROOT/utilbin/$ARCH/openssl x509 -fingerprint -in ~/.sge/port6789/default/certs/cert.pem -noout MD5 Fingerprint=F9:FA:AB:86:F3:71:E4:7F:18:82:78:7D:5E:51:7B:B5 |
Copyright 2002 Sun Microsystems, Inc. All rights reserved.