Merlin log

Status

Merlin is the name of the PC that will run labkey. It can be accessed via:

username@merlin.fmf.uni-lj.si

Contact me if you want to have a username, however a valid reason should be specified.

The ownership status of merlin is somewhat mixed. It belongs to the meteorology research group and is as such under scrutiny of prof. Nedeljka Žagar. However, she does appreciate the usage of PC as opposed to its sitting under the table unused. The PC was used by a former researcher gone to pursue other interests.

The PC is a quad core Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz machine, with 16 GB of memory and a 106 GB disk under a single partition mounted on /. The disk is currently unused (4 % usage listed, mostly system related stuff).

There are four (4) 3 TB disks in the housing, however they don't seem to be mounted.

Looking at kernel messages with dmesg the following jumped out:

[    2.514731] md: md0 stopped.
[    2.515854] md: bind<sdb1>
[    2.515968] md: bind<sdd1>
[    2.516073] md: bind<sde1>
[    2.516191] md: bind<sdc1>
[    2.516210] md: kicking non-fresh sdb1 from array!
[    2.516215] md: unbind<sdb1>
[    2.531573] md: export_rdev(sdb1)

Especially the part about kicking a disk out of array. There is a simple solution via web. Following the instructions, I did:

mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1

which reported an error

Device '/dev/sdb1' not present.

But then I went on and did:

mdadm /dev/md0 --add /dev/sdb1

which didn't fail. Kernel was happy to report:

[1300661.965415] md: export_rdev(sdb1)
[1300662.629125] md: bind<sdb1>
[1300662.651294] RAID conf printout:
[1300662.651296]  --- level:5 rd:4 wd:3
[1300662.651298]  disk 0, o:1, dev:sdb1
[1300662.651299]  disk 1, o:1, dev:sdc1
[1300662.651300]  disk 2, o:1, dev:sdd1
[1300662.651300]  disk 3, o:1, dev:sde1
[1300662.651377] md: recovery of RAID array md0
[1300662.651379] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[1300662.651380] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[1300662.651382] md: using 128k window, over a total of 2930133504k.

More on RAID management.

Still, df -h does not see the disk (ie. it is not mounted). I found out (via blkid) that /dev/md0/ is formated to xfs:

/dev/md0: UUID="276bf256-553e-4a15-a202-e0ce57b70a9b" TYPE="xfs"

There are no xfs tools (xfsprogs).

There is no apache or other web servers (httpd).

There are directories in /dev/md0:

52K     aqmeii
145M    luka
1.7T    rahela

Actions

Mount array

Manually by

mkdir /data0
mount /dev/md0 /data0

Filesystem type (xfs) determined automatically by mount. Also added a line to /etc/fstab:

UUID=276bf256-553e-4a15-a202-e0ce57b70a9b /data0 xfs defaults,errors=remount-ro 0 0

Used UUID from blkid command.

Erase data

cd /data0
rm -rf rahela luke aqmeii

Disk status:

root@merlin:/data0# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/dm-0       106G  3.8G   97G   4% /
udev             10M     0   10M   0% /dev
tmpfs           3.2G   58M  3.1G   2% /run
tmpfs           7.9G   68K  7.9G   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda1       236M   34M  191M  15% /boot
tmpfs           1.6G  4.0K  1.6G   1% /run/user/116
tmpfs           1.6G  8.0K  1.6G   1% /run/user/1000
tmpfs           1.6G     0  1.6G   0% /run/user/1001
/dev/md0        8.2T   34M  8.2T   1% /data0

Install web server

Install:

apt-get install apache2

Starts also the apache2 deamon. Fiddling a bit with modules allows user web-pages to pop up:

   a2enmod userdir
   service apache2 restart

so I can look at http://localhost/~andrej.

Install java SRE

Go to oracle.com and download the latest java SRE server. Go to /usr/local, untar and unzip. Following suggestions by LabKey I created a link:

lrwxrwxrwx 1 root staff 22 Mar 22 15:41 /usr/local/jdk-current -> /usr/local/jdk1.8.0_74

Directory:

/usr/local/jdk1.8.0_74

contains subdirectories such as bin, jre and others.

There is a local version of java running:

root@merlin:/usr/local# /usr/bin/java -version
java version "1.7.0_95"
OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-1~deb8u1)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)

However, it is the OpenJava, which is not supported by LabKey.

Install tomcat

apt-get install tomcat8

I choose version 8 on a whim. It works with 7 also. I follow LabKey suggestion to modify tomcat startup script /etc/default/tomcat8 to use the selected java JRE by setting:

JAVA_HOME=/usr/local/jdk-current

By default tomcat8 is run as user tomcat8 with a default home at /usr/share/tomcat8.

SSL

Certificate Authority (CA), server and user certificates

Edit the configuration file in /etc/ssl; default is openssl.cnf; I copied it to openssl_tomcat8.cnf. Most notable changes:

 dir           = ./tomcat8CA           # Where everything is kept
 dir           = ./tomcat8CA           # TSA root directory
 private_key     = $dir/signing-ca-1.key# The private key
 certificate     = $dir/signing-ca-1.crt         # The CA certificate
 serial          = $dir/ca.db.serial             # The current serial number
 RANDFILE        = $dir/ca.db.rand       # private random number file
 database        = $dir/ca.db.certs      # database index file.
 new_certs_dir   = $dir/ca.db.new_certs          # default place for new certs.
 ...
 [ req_distinguished_name ]
 countryName_default           = SI
 stateOrProvinceName_default     = Slovenija
 localityName_default            = Ljubljana
 0.organizationName_default    = FMF
 organizationalUnitName_default        = Medicinska Fizika
 ...
 [usr_cert]
 nsCaRevocationUrl               = http://merlin.fmf.uni-lj.si/tomcat8CA/tomcat8CA-crl.pem

Once the configuration is set, start by generating keypair:

openssl genrsa -des3 -out root-ca.key 1024

Then, generate the self-signed CA ROOT certificate:

openssl req -new -x509 -days 3650 -key root-ca.key -out root-ca.crt -config openssl_tomcat8.cnf

The root-ca.key password needs to be supplied. Copy created key and certificate to tomcat8CA directory:

mv root-ca.crt tomcat8CA/signing-ca-1.crt
mv root-ca.key tomcat8CA/signing-ca-1.key

This are now the CA certificate and key. Our CA just came into existence.

Server certificate

Create a new certificate:

openssl req -newkey rsa:1024 -keyout /usr/share/apache2/server/server.key -nodes -config
openssl_tomcat8.cnf -out /usr/share/apache2/server/server.csr

Sign via CA:

openssl ca -config openssl_tomcat8.cnf -out /usr/share/apache2/server/server.crt 
-infiles /usr/share/apache2/server/server.csr

User certificate

Generate a request and sign it with CA's certificate. The last command also archives the generated certificate:

cd /usr/share/apache2/users
openssl req -newkey rsa:1024 -keyout astuden.key -config /etc/ssl/openssl_tomcat8.cnf
-out astuden.req
openssl ca -config /etc/ssl/openssl_tomcat8.cnf -out astuden.crt -infiles astuden.req

It is usually easier to install PKCS12 formated certificates on most browsers. Hence we combine the generated key and certificate to a single file:

openssl pkcs12 -export -clcerts -in astuden.crt -inkey astuden.key 
-out astuden.p12 -name "astuden@merlin"

To revoke certificate, first find the certificate in the database. List of certificates is stored in /etc/ssl/tomcat8CA/ca.db.new_certs. The certificates there are stored by number rather than name. Find name of the issuee by greping. Once a certificate is identified, you can revoke it. Unfortunately it doesn't work on the crt-s stored in /usr/share/apache2/users.

#>cd /etc/ssl/tomcat8CA/ca.db.new_certs
#>for f in `ls *.pem` ; do echo $f; openssl x509 -in $f -text -noout | grep Subject | grep CN; done;
# Assume XX.pem is the one you want to revoke
#>openssl ca -config /etc/ssl/openssl_tomcat8.cnf -revoke XX.pem

Manage certificate revocation list (CRL)

Create the CRL, and copy it to accesible location (again, on merlin).

#>mkdir /var/www/html/tomcat8CA
#create an empty file
#>echo 01 > /etc/ssl/tomcat8CA/crlnumber
#>openssl ca -config /etc/ssl/openssl_tomcat8.cnf -gencrl -out /var/www/html/tomcat8CA/tomcat8CA-crl.pem
#check the crl
#>openssl crl -in /var/www/html/tomcat8CA/tomcat8CA-crl.pem -noout -text

You should re-create the CRL at regular intervals. By default, the CRL expires after 30 days. This is controlled by the default_crl_days option in the [ CA_default ] section.

Enable SSL in apache2

Add SSL support:

a2enmod ssl

If you want apache2 to listen to specific ports (8443 comes to mind), then you should enable them in /etc/apache2/ports.conf:

<IfModule ssl_module>
  <!-- Listen 8443   Don't enable 8443 for apache2; used by tomcat8! -->
  Listen 443
</IfModule>

Enable site ssl by cd /etc/apache2/site-available and editing default-ssl.conf; in fact I cp default-ssl.conf ssl.conf and edited the latter. Some notable changes

ServerAdmin tomcat8
SSLEngine on
SSLCertificateFile      /usr/share/apache2/server/server.crt
SSLCertificateKeyFile   /usr/share/apache2/server/server.key

Both must be set. If key is omitted, errors like:

SSL Library Error: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag

appear in /var/log/apache2/error.log.

Add ssl site control (from /etc/apache2/site-available):

a2ensite ssl.conf

This creates a file ssl.conf in /etc/apache2/site-enabled/. All later edits should be in this new file. Restart apache:

/etc/init.d/apache2 restart

Test the setup via 

openssl s_client -connect localhost:443 -cert /usr/share/apache2/server/server.crt

This should output a complain about the certificate that doesn't belong to any established CA authority.

To limit access to certified users only (etc/apache2/site-enabled/ssl.conf):

SSLCACertificateFile   "/etc/ssl/tomcat8CA/signing-ca-1.crt
SSLVerifyDepth  2
SSLVerifyClient require

IP tables

Since I had troubles accessing tomcat8 through port 8443, I put an entry in a routing table routing port 443 to 8443. 

iptables -t nat -p tcp -A PREROUTING --dport 443 -j REDIRECT --to-ports 8443

[//]: # See this image iptables

A nice diagram about routing can be found here. By stating https://merlin.fmf.uni-lj.si you actually access port 8443 and consequentially tomcat8 service.

Enable SSL in tomcat8

TOMCAT8 server certificate

Do the same as for apache2, ie.

openssl req -newkey rsa:1024 -keyout /usr/share/tomcat8/server/server.key -nodes 
-config /etc/ssl/openssl_tomcat8.cnf -out /usr/share/tomcat8/server/server.csr
openssl ca -config /etc/ssl/openssl_tomcat8.cnf -out /usr/share/tomcat8/server/server.crt
-infiles /usr/share/tomcat8/server/server.csr

Tomcat8 settings

Make sure APR (libapr-1) and tcnative are installed. Add their paths to java.library.path and $LD_LIBRARY_PATH via the following lines in /usr/share/tomcat8/bin/setenv.sh:

LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/apr-util-1:/usr/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH}
CATALINA_OPTS=-Djava.library.path=${LD_LIBRARY_PATH}

In $CATALINA_BASE, fiddle with $CATALINA_BASE/conf/server.xml configuration file. I resorted to APR based SSL engine. Since we are using the native APR library, the settings are nearly equal to apache2 settings: 

<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />

<Connector port="8443"
protocol="org.apache.coyote.http11.Http11AprProtocol"
    maxThreads="150" SSLEnabled="true"
    scheme="https" secure="true"
SSLCertificateFile="/usr/share/tomcat8/server/server.crt"
    SSLCertificateKeyFile="/usr/share/tomcat8/server/server.key"
    SSLCACertificateFile="/etc/ssl/tomcat8CA/signing-ca-1.crt"
    SSLVerifyClient="required"  #that means that access is only allowed for certified users
SSLVerifyDepth="2"
    SSLProtocol="all"
SSLCARevocationFile="/var/www/html/tomcat8CA/tomcat8CA-crl.pem"
 />

Restart tomcat8:

/etc/init.d/tomcat8 restart

Check /var/log/tomcat8/catalina.out for error messages, they seem to be quite informative. Some possible errors:

  • java.lang.Exception: An invalid value [TLSv1+TLSv1.1+TLSv1.2] was provided for the SSLProtocol attribute This is a tcnative issue of 1.1.32. Change it to "all", see above.
  • ssl_error_rx_record_too_long reported on web page: Error in tomcat8; look at /var/log/tomcat8/catalina.out
  • start failed: java.lang.Exception: Socket bind failed: [98] Address already in use. Port already registered by apache2. Edit /etc/apache2/sites-enabled/ssl.conf.
  • Error initializing endpoint java.lang.Exception: Unable to load certificate key. You should set permissions for tomcat8 to be able to access CA directory. I changed the ownership of /etc/ssl/tomcat8CA and made tomcat8 part of the ssl-cert group.
    chown root:ssl-cert /etc/ssl/tomcat8CA
usermod -a -G ssl-cert tomcat8

Install postgreSQL

apt-get install postgresql

Configure postgreSQL

The configuration for posgresql is in /etc/postgresql/9.4/main/postgresql.conf. To change location of the database, modify data_directory field. In our case, we want to keep data on the RAID storage, hence

#data_directory = '/var/lib/postgresql/9.4/main'         # use data in another directory
data_directory='/data0/postgresql/9.4/main'

I also modified join_collapse_limit following this suggestion.

Following a further Labkey suggestion, I also increased logging; ie uncommented the following lines:

track_counts = on
autovacuum = on        
client_min_messages = notice 
log_min_messages = warning
log_min_error_statement = error
log_statement = 'mod'

Prepare the environment

Create the data folder and make sure the postgres user account can write to that directory. Then create the database.

mkdir /data0/postgresql 
chown postgres:postgres /data0/postgresql
su postgres         #become postgres
mkdir -p /data0/postgresql/9.4/main
/usr/lib/postgresql/9.4/bin/initdb /data0/postgresql/9.4/main
exit                #become root again

Start the database

/etc/init.d/postgresql start

Check for error messages:

cat /var/log/postgresql/postgresql-9.4-main.log

The user password for accessing postgreSQL database must be defined. Set it through:

psql -U postgres
psql$ ALTER USER postgres WITH PASSWORD 'VeryVerySecret';

Very important - don't forget the semicolon!

The following is not needed, but currently I believe it does no harm: Create a database labkey owned by postgres

psql$ CREATE DATABASE labkey OWNER postgres

Check whether access is functioning by:

anyuser$> psql -U postgres -h localhost -p 5432
Password for user postgres:

For upgrades, multiple versions of postgresql might be running at the same time. Check by

   $> sudo pg_lsclusters

To copy database from one version to the other, before the update to

postgres$>pg_dumpall >> database.out

Then, when the new is running, check the port the upgraded version is running on, and do:

postgres$> psql -p 5433 -d postgres -f database.out

Further notes

A useful tool for database management is phpPgAdmin. This site tells you how to manage it through web app.

Labkey

Use the version that needs registration. Below are some missteps. The instructions are for the registered version. Then unpack it and follow instructions below.

Don't do any of the following:

svn version:

 cd /home/andrej/software/src
 svn co --username cpas --password cpas https://hedgehog.fhcrc.org/tor/stedi/trunk svnlabkey

packaged version: 

 wget https://www.labkey.org/download/16.1/LabKey16.1-42836.12-src.tar.gz
 cd ~/software/src
 mkdir labkey
 cd labkey
 tar xvzf ~/Downloads/LabKey16.1-42836.12-src.tar.gz

Configure labkey

Tomcat

Following Labkey instructions. tomcat-home referrs to CATALINA_BASE; see below

 cd /var/lib/tomcat8/lib
 cp /home/andrej/software/src/labkeyRegister/LabKey16.1-42836.12-community-bin/tomcat-lib/*.jar .

LabKey home

While setting it to /usr/local/labkey seems fine, the tomcat8 set home at /usr/share/tomcat8 also has its merrits. I decided that the actual labkey home will be /usr/share/labkey; however a symbolic link was inserted at /usr/local/labkey pointing to the same disk position.

lrwxrwxrwx 1 root staff 17 Mar 22 16:41 /usr/local/labkey -> /usr/share/labkey

Make sure that tomcat8 has write permissions on /usr/share/labkey.

Then proceed with copying

cp -rp /home/andrej/software/src/labkeyRegister/LabKey16.1-42836.12-community-bin/bin .
cp -rp /home/andrej/software/src/labkeyRegister/LabKey16.1-42836.12-community-bin/labkeywebapp .
cp -rp /home/andrej/software/src/labkeyRegister/LabKey16.1-42836.12-community-bin/modules .
cp -rp /home/andrej/software/src/labkeyRegister/LabKey16.1-42836.12-community-bin/pipeline-lib .

Third party components

There is a full list of additional 3rd party components. So here it goes:

apt-get install r-base
apt-get install graphviz

Download Trans Proteomic Pipeline from here. Follow instrucions. I've set TPP_ROOT=/usr/local/tpp and TPP_WEB=/web/. Then the standard make all and make install. Update tomcat8's path by adding 

export PATH=/usr/local/tpp/bin:${PATH}

to /usr/share/tomcat8/bin/setenv.sh.

Continue:

apt-get install libpwiz #for proteowizzard

Then, as andrej:

cd ~/Downloads
wget https://www.labkey.org/download/pepmatch/pepmatch.tar.gz
cd ~/software/src
tar xvzf ~/Downloads/pepmatch.tar.gz
make

As root:

cd /usr/local/tpp/bin
ln -vs /home/andrej/software/src/pepmatch/pepmatch

The instructions say 

Copy pepmatch binary to your pipeline tools directory

and the only pipeline tools are the trans proteomic pipeline tools quoted above.

Moving LabKey configuration script

Here we are instructed to move the labkey.xml file to a path under <tomcat_home>. However, tomcat is a peciuliar beast, and two variables need to be considered.

  • CATALINA_HOME is the place where tomcat8 itself has its binaries and enviromental variables (path, java, etc.)

  • CATALINA_BASE is the location where tomcat8 looks for applications. In this terminology, LabKey is the application that tomcat8 will run, so it should be placed there.

It was only through looking at 'http://localhost:8080' that I found out that they are not the same, ie.

CATALINA_HOME=/usr/share/tomcat8
CATALINA_BASE=/var/lib/tomcat8

Personally, /var/lib strikes me as the most unlikely place to put applications into. Especially since /var is sometimes a separate partition. In my case, / is a rather big place (100 GB of SSD, I am guessing), so I am happy to go with that. But do consider changing the variables in $CATALINA_HOME/bin/setenv.sh if needed.

Setting up tomcat8 heap size

Edit setenv.sh in $CATALINA_HOME/bin:

root@XXX:/usr/share/tomcat8# cat bin/setenv.sh 
JAVA_OPTS="$JAVA_OPTS -Xms128m -Xmx2048m -XX:-HeapDumpOnOutOfMemoryError"
    PATH=/usr/local/tpp/bin:${PATH}

Set password and user for the database

As instructed, values in $CATALINA_BASE/conf/Catalina/localhost/labkey.xml must be adjusted to local install. On top of the suggested changes I found that in setting url tag, the port specification was important

 url="jdbc:postgresql://localhost:5432/labkey"

Start labkey server

Restart tomcat8

/etc/init.d/tomcat8 restart

Connect via

https://merlin.fmf.uni-lj.si/labkey

Install SMTP send only server

Potentially, merlin could be used to send out labkey related messages. However, as this post shows, it does get quite hairy if you want your mail to be assumed valid, ie. not spam. So, at the moment, I relay labkey mail to FMF mail server through my username. Probably, a separate user could be created to keep things separate. On the TODO list.

Access to files via WebDAV

One can directly mount files, say for processing, via WebDAV infractructure. The following is for linux, but similar steps can be taken in Windows.

To have files from database appear as mount point in your directory structure, the client to use is davfs. Once you install davfs, mount command will mask your remote access to a local directory. First, you will need an entry in the /etc/fstab file. If you are not root, have your administrator edit the file for you. The additional line has to be:

https://merlin.fmf.uni-lj.si/labkey/_webdav     /data0/studen/data/merlin       davfs   noauto,user     0 0

where /data0/studen/data/merlin is an appropriate mount point. You should also be a member of the davfs2 group, so your administrator has to add you:

/sbin/usermod -g davfs2 username

where username is your identity. You 'll also need the root CA (public) certificate, available here, and your private certificate. According to this, private certificate goes into ~/.davfs2/certs/private and CA certificate into ~/.davfs2/certs. In fact, clicking on the above link will place the CA certificate in your browser's Authorities certificate list, should you put a tick next to Trust this CA to identify websites option. Then you should locate it in the list and export it, as a pem certificate, to the ~/.davfs2/certs directory. Then you should let davfs know which certificates to use, by editing ~/.davfs2/davfs2.conf file, specifying

trust_ca_cert    ca.pem
clientcert      user.p12

where ca.pem is the exported CA certificate and user.p12 is your private certificate. Then you start the mounting procedure by

mount /data0/studen/data/merlin

The algorithm will ask you for your username, which is the email address you used to collect labkey invitation, the password, which is the password you use to enter labkey, and then the password for your private certificate. If all passwords are corrected, you should be able to see all data files from the server from your mountpoint; ie. doing

ls /data0/studen/data/merlin

should list all projects.

Todo

Most users will not have access to the core webdav directory, but only to respective projects. In that case, the mount point is slightly different (and contains spaces). According to this post spaces are notorioulsy difficult to manage; however they give some suggestions as how to avoid the worse. All come around to it, should it be neccesary.

links

social