New merlin

Just got a new virtual machine running on FMF's servers installed.

Some data from Feri:

med0.ipv6.fmf.uni-lj.si

u: xxxxxxx
p: xxxxxxx

Ta uporabnik ima polne sudo pravice.

Trenutno ipv6 naslov ni dostopen iz Interneta, samo iz FMF omrezja.

Za virtualko je skonfiguriran reverse proxy (na ipv4) na naslovu 
https://med0.fmf.uni-lj.si. Nastiman je tudi Letsencrypt certifikat. 
K temu certifikatu lahko dodamo dodatna imena, v kolikor bi se odlocili za uporabo.

Some basic parameters:

debian@med0:~$ uname -r
4.9.0-2-amd64
andrej@med0:~$ lsb_release -da
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.0 (stretch)
Release:        9.0
Codename:       stretch

The mount map include '/var' and '/home' disks. They were removed. To help system realize that /home and /var are now part of '/', one should run:

debian@med0:~$ sudo systemctl daemon-reload

This helps /etc/init.d/apache2 start service to run without mounting the '/var' partition.

Certificate authority CA

CA is still on the old merlin (med1.fmf.uni-lj.si). Useful scripts (in scripts):

#>generateUserReq.sh
#>generateServerReq.sh
#>signUserReq.sh
#>signServerReq.sh
#>convertCert.sh

This is also works for certificate revocation and such. More details on merlin

Installation of required components

Java

Download Java SRE server from Java. Copy .tar.gz to /home/andrej/software/packages. Unzip to /home/andrej/software/install. Now I have a new directory:

/home/andrej/software/install/jdk1.8.0_131

This will be used later on for other software components.

Tomcat

Use apt-get:

apt-get install tomcat8

Correct JAVA_HOME in etc/default/tomcat8:

JAVA_HOME=/home/andrej/software/install/jdk

and restart tomcat:

sudo \etc\init.d\tomcat8 restart

Check tomcat version by running:

cd /usr/share/tomcat8/lib
/home/andrej/software/install/jdk/bin/java -cp catalina.jar org.apache.catalina.util.ServerInfo
Server version: Apache Tomcat/8.5.12 (Debian)
Server built:   Apr 18 2017 07:53:23 UTC
Server number:  8.5.12.0
OS Name:        Linux
OS Version:     4.9.0-2-amd64
Architecture:   amd64
JVM Version:    1.8.0_131-b11
JVM Vendor:     Oracle Corporation

Verify presence of libapr and tcnative:

sudo dpgk-query -l | grep apr
sudo dpgk-query -l | grep tcn

Find their libraries and append paths to setenv.sh in /usr/share/tomcat8/bin directory. There might be no setenv.sh file yet. Typically:

debian@med0:/usr/share/tomcat8/bin$ cat setenv.sh 
LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/apr-util-1:/usr/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH}
CATALINA_OPTS=-Djava.library.path=${LD_LIBRARY_PATH}

Update tomcat8 configuration file (which is in CATALINA_BASE); on debian:

CATALINA_HOME=/usr/share/tomcat8
CATALINA_BASE=/var/lib/tomcat8

Actual path is ${CATALINA_BASE}/conf/server.xml, but conf actually points to /etc/tomcat8, so you can edit

sudo vi /etc/tomcat8/server.xml

You should edit it until you come up with something like below:

<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />

<Connector port="8443"
protocol="org.apache.coyote.http11.Http11AprProtocol"
    maxThreads="150" SSLEnabled="true"
    scheme="https" secure="true"
    SSLCertificateFile="/usr/share/tomcat8/server/server.crt"
    SSLCertificateKeyFile="/usr/share/tomcat8/server/server.key"
    SSLCACertificateFile="/etc/ssl/labkeyAtMed0CA/labkeyAtMed0CA.crt"
    SSLVerifyClient="required"  #that means that access is only allowed for certified users
SSLVerifyDepth="2"
    SSLProtocol="all"
SSLCARevocationFile="/var/www/html/labkeyAtMed0CA/labkeyAtMed0CA-crl.pem"
 />

The certificates in /usr/share/tomcat8/server are just copies of certificates in /usr/share/apache2/server, but with ownership set to tomcat8 user. This is a security feature. Make sure you do:

sudo mkdir /usr/share/tomcat8/server
sudo cp /usr/share/apache2/server/server.{key,crt} /usr/share/tomcat8/server
sudo chown -R tomcat8:tomcat8 /usr/share/tomcat8/server
sudo chown -R root:ssl-cert /etc/ssl/labkeyAtMed0CA
sudo /usr/sbin/usermod -G ssl-cert tomcat8

This has tomcat respond to 8443 http querries. Test if the configuration is at leas semantically correct by restarting tomcat8. You should also see response by opening a browser to look at:

https://med0.ipv6/fmf.uni-lj.si:8443

If the message It works appears, the tomcat8 is set correctly.

Postgresql

Install from apt-get:

apt-get install postgresql

Update configuration file to point to new location on '/data':

#data_directory = '/var/lib/postgresql/9.6/main'         # use data in another directory
data_directory='/data/postgresql/9.6/main'
#update other variables
track_counts = on
autovacuum = on
client_min_messages = notice
log_min_messages = warning
log_min_error_statement = error
log_statement = 'mod'

Prepare new database:

sudo mkdir /data/postgresql
sudo chown postgres:postgres /data/postgresql
sudo su postgres         #become postgres
mkdir -p /data/postgresql/9.6/main
/usr/lib/postgresql/9.6/bin/initdb /data/postgresql/9.6/main
exit                #become debian

If the database already exists, delete it with

sudo /etc/init.d/postgres stop
sudo su postgres
rm -rf /data/postgresql/9.6/main/*
export LC_CTYPE="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
/usr/lib/postgresql/9.6/bin/initdb /data/postgresql/9.6/main
exit

(Re)start the database and check for errors:

sudo /etc/init.d/postgresql start
cat /var/log/postgresql/postgresql-9.6-main.log

Copy old database from merlin:

#on old merlin
pg_dumpall -c -p 5432 > /mnt/disk/merlin/postgresql.backup
scp /mnt/disk/merlin/postgresql.backup andrej@med0.ipv6.fmf.uni-lj.si:~/
ssh andrej@med0.ipv6.fmf.uni-lj.si
su debian
sudo su postgres
/usr/lib/postgresql/9.6/bin/psql -f /home/andrej/postgresql.backup postgres

Labkey

Copy files!

IP configuracija

This is how it looks like on merlin (old)

$> less /etc/init.d/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
allow-hotplug eth0
iface eth0 inet dhcp

auto eth0:1
allow-hotplug eth0:1
iface eth0:1 inet static
        address 193.2.67.69
        netmask 255.255.255.0
        gateway 193.2.67.1

This is the new merlin file:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# Primary static ipv6 address
auto eth0
iface eth0 inet6 static
  address 2001:1470:fff0:830::a01:134/64
  gateway 2001:1470:fff0:830::1

# Additional autoconfigured IPv6 interface
#iface eth0 inet6 auto

# Additional dynamic ipv4 address
#auto eth0:1
#iface eth0:1 inet dhcp

For static ipv6 using the one given to merlin:

root@merlin:/home/andrej# /sbin/ifconfig
eth0      Link encap:Ethernet  HWaddr bc:5f:f4:60:40:92
          inet addr:193.2.67.198  Bcast:193.2.67.255  Mask:255.255.255.0
          inet6 addr: 2001:1470:fff0:830:be5f:f4ff:fe60:4092/64 Scope:Global
          inet6 addr: fe80::be5f:f4ff:fe60:4092/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:338363536 errors:0 dropped:0 overruns:0 frame:0
          TX packets:254659687 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:214088917675 (199.3 GiB)  TX bytes:316323797689 (294.5 GiB)

eth0:1    Link encap:Ethernet  HWaddr bc:5f:f4:60:40:92
          inet addr:193.2.67.69  Bcast:193.2.67.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:881159085 errors:0 dropped:0 overruns:0 frame:0
          TX packets:881159085 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:167540609835 (156.0 GiB)  TX bytes:167540609835 (156.0 GiB)

Change hostname in /etc/hostname

(merlin->med1) (med0->merlin)

Change name in /etc/hosts. This is how it looks like on med0:

127.0.0.1       localhost
127.0.1.1       med0.fmf.uni-lj.si      med0

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

And this is how it looks on merlin:

127.0.0.1       localhost
127.0.1.1       merlin.fmf.uni-lj.si    merlin

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Changing 127.0.1.1 lines to (med0->merlin) and (merlin->med1).

To change hostname:

invoke-rc.d hostname.sh start
invoke-rc.d networking force-reload

To reload the network configuration:

/etc/init.d/networking restart

Backup

Use tar via ssh.

tar zcvf - ${TARDIR} | ssh med1.fmf.uni-lj.si "cat > ${BACKUP_DIR}/${TARFILE}.tar.gz"

Restore via:

$> ssh med1.fmf.uni-lj.si "cat ${BACKUP_DIR}/${TARFILE}.tar.gz" | tar zxvf - ${TARDIR}

To use ssh tunnel, exchange keys on the client (merlin) and host (med1) machine. On merlin, do:

ssh-keygen -t rsa
ssh-copy-id med1.ipv6.fmf.uni-lj.si

Restarting after a shutdown

The following steps should be executed on reboot:

Make sure /data is mounted; it points to /dev/xvde. The entry in /etc/fstab is:

UUID=c3bd9b07-9073-48f8-9e20-478668c4511d /data         xfs     defaults        0       1

Restart postgresql and check that is starts ok:

#/etc/init.d/postgresql restart
#tail /var/log/postgresql/postgresql-9.6-main.log

#revised version
$debian>sudo su postgres
$postgres>cd
$postgres>bin/startup.sh
$postgres>less log/postgres.log

When default postgresql.config from Labkey is used, bear in mind that postgresql above version 9.5 or so doesn't use checkpoint_segments any longer and postgres startup may fail [Here][postgresqlCheckpointSegments] is a comment on that and a workaround. I decided to comment out that particular setting.

Do port forwarding:

iptables -t nat -p tcp -A PREROUTING -i eth0 --dport 443 -j REDIRECT --to-ports 8443

To make the rules execute at reboot, do :

$root>iptables-save > /etc/iptables.up.rules

Create file /etc/network/if-pre-up.d/iptables with a content:

#!/bin/sh
/sbin/iptables-restore < /etc/iptables.up.rules

and allow the user to execute it:

chmod u+x /etc/network/if-pre-up.d/iptables`

(Re)start tomcat8:

$debian> sudo su tomcat8
$tomcat8> cd
$tomcat8> bin/startup.sh
#make sure tomcat8 is running by looking at the log file
$tomcat8> less log/catalina.out
#once tomcat is running, check labkey's log
$tomcat8> less log/labkey.log

Mask system-based tomcat8 and postgresql services:

sudo systemctl stop tomcat8.service
sudo systemctl mask tomcat8.service
sudo systemctl disable tomcat8.service
sudo systemctl stop postgresql.service
sudo systemctl mask postgresql.service
sudo systemctl disable postgresql.service
systemctl daemon-reload

If SSL is being obnoxiuos, maybe block certificate revocation list (CRL) checking (/etc/apache2/sites-enabled/default-ssl.conf):

    #SSLCARevocationFile /var/www/html/labkeyAtMed0CA/labkeyAtMed0CA-crl.pem
    #SSLCARevocationCheck chain

Blog

Setup of new merlin

New merlin

Certificate authority CA

Installation of required components

Java

Tomcat

Postgresql

Labkey

IP configuracija

Backup

Restarting after a shutdown

New merlin

Certificate authority CA

Installation of required components

Java

Tomcat

Postgresql

Labkey

IP configuracija

Backup

Restarting after a shutdown

links

social