ARC Configuration

/etc/arc.conf example configuration directives to be added. It should work with both 0.6.5 and 0.8.1 arc releases.


[vo]
id="atlas"
vo="atlas"
source="vomss://voms.cern.ch:8443/voms/atlas"
mapped_unixid="griduser03"

[group]
name="atlas-lcgadmin"
voms="atlas * lcgadmin *"

[group]
name="atlas-production"
voms="atlas * production *"

[group]
name="atlas-users"
voms="atlas * NULL *"

[gridftpd]
allowunknown="no"
unixmap="griduser01 group atlas-lcgadmin"
unixmap="griduser02 group atlas-production"
unixmap="griduser03 group atlas-users"
#unixgroup="atlas-users simplepool /var/spool/nordugrid/atlas-users"
unixmap="griduser:gridgroup all" # see the note below

[grid-manager]
maxloadshare="10 voms:role"

# Register to Atlas
[infosys/cluster/registration/GrisToAtlas]
targethostname="atlasgiis.nbi.dk"
targetport="2135"
targetsuffix="Mds-Vo-name=Atlas, o=grid"
regperiod="600"

[infosys/cluster/registration/AtlasBackup]
targetsuffix="mds-vo-name=Atlas,o=grid"
targethostname="arcgiis.titan.uio.no"
targetport="2135"
regperiod="600"


where
# cat /var/spool/nordugrid/atlas-users/pool
atlas001
atlas002
atlas003
atlas004
atlas005

For a working voms support, make sure the following two files are present in the /etc/grid-security/vomsdir/atlas directory:
# cat /etc/grid-security/vomsdir/atlas/voms.cern.ch.lsc
/DC=ch/DC=cern/OU=computers/CN=voms.cern.ch
/DC=ch/DC=cern/CN=CERN Trusted Certification Authority

# cat /etc/grid-security/vomsdir/atlas/lcg-voms.cern.ch.lsc
/DC=ch/DC=cern/OU=computers/CN=lcg-voms.cern.ch
/DC=ch/DC=cern/CN=CERN Trusted Certification Authority

In addition, the following two files should be present in /etc/vomses/ directory:
# cat /etc/vomses/atlas-voms.cern.ch
"atlas" "voms.cern.ch" "15001" "/DC=ch/DC=cern/OU=computers/CN=voms.cern.ch" "atlas"

# cat /etc/vomses/atlas-lcg-voms.cern.ch
"atlas" "lcg-voms.cern.ch" "15001" "/DC=ch/DC=cern/OU=computers/CN=lcg-voms.cern.ch" "atlas"

Notes:

  • [vo] part is used to generate gridmap-file, needed for infosys/brokering, with a default mapping to the least-privileges username
  • [group] part is for voms Role based authorization, based on current proxy Roles
  • unixmap remaps the default local user as provided in gridmap-file to the proper local user as defined by proxy Roles
  • simplepool will remove the dynamic mappings after 10 days of user's inactivity. Mappings are stored in the same directory as "pool" file containing the list of local pool usernames
  • maxloadshare will only work with ARC 0.8.1. The number (10) is the maximum number of jobs in processing per voms role.
  • the last unixmap line maps all users-not-matched by voms rules to a single username "griduser". The line should be skipped if the original mapping from grid-mapfile should be preserved.
  • The unixmap order is essential: lcgadmin first, then production, users last. voms proxy has several attributes, so any proxy would match the users rule.

Auto-installation requirements

Disk space requirements

  • 200GB for software release
  • 1TB for cooldata, initially 300GB should be enough

Local user privileges

username with Role=lcgadmin (griduser01 from arc.conf above ) needs read-write permissions to
  • $VO_ATLAS_SW_DIR, a top directory for software releases and cooldata from every node through a grid job
  • APPS/HEP must be writeable in the same way
Make sure, the local username with privileges is not mapped from any other than lcgadmin Role. I would suggest not to use the pool accounts for this particular role. If pool accounts are there, the usernames must end with a number (like atlassgm001, atlassgm002, ...)

APPS/HEP/ATLAS-AUTOINSTALL

This rte is a dummy one. The clusters having this available will be managed through grid jobs.

#!/bin/bash
# dummy RTE for auto-installation

APPS/HEP/ATLAS-SITE

Site definitions as now for Frontier/squid and any additional site-specific settings. This RTE should be modified by site admins and never by grid jobs. It should define the $VO_ATLAS_SW_DIR

#!/bin/bash
#
#......ATLAS Site configuration
#
if [ -z $1 ] || [ $1 == 1 ] ; then
  shift
  export VO_ATLAS_SW_DIR=/net/pikolit/d0/nfs/grid/arcswdir/atlas
  export FRONTIER_SERVER="(serverurl=http://lcgft-atlas.gridpp.rl.ac.uk:3128/frontierATLAS)(serverurl=http://ccsqfatlasli01.in2p3.fr:23128/ccin2p3-AtlasFrontier)(proxyurl=http://db-atlas-squid.ndgf.org:3128)"
  export ATLAS_POOLCOND_PATH="$VO_ATLAS_SW_DIR/data"
fi

ENV/PROXY

a simple RTE script to copy the proxy and ca certs to the node, which are needed to access grid services.

#!/bin/bash

x509_cert_dir="/etc/grid-security/certificates"

case $1 in
  0) mkdir -pv $joboption_directory/arc/certificates/
     cp -rv $x509_cert_dir/ $joboption_directory/arc
     cat ${joboption_controldir}/job.${joboption_gridid}.proxy >$joboption_directory/user.proxy
     ;;
  1) export X509_USER_PROXY=$RUNTIME_JOB_DIR/user.proxy
     export X509_USER_CERT=$RUNTIME_JOB_DIR/user.proxy
     export X509_CERT_DIR=$RUNTIME_JOB_DIR/arc/certificates
     ;;
  2) :
     ;;
esac

Memory scaling

Atlas jobs use a lot of virtual memory. Typical memory usage for reconstruction jobs is 1.8GB of physical and 2.4GB of virtual memory while the jobs are submitted with a 2GB memory requirement. Such a job can run efficiently with a reservation of 2GB of memory without swapping.Since a typical node has 2GB/core memory installed, for an efficient cluster usage, some workaround is needed on LRMS submission scripts. The virtual memory should be scaled for at least a factor of 1.25.

For torque, the submit-pbs-job script should be changed in one of the following ways:

Memory reservation without enforcing the limits:

if [ ! -z "$joboption_memory" ] ; then
  # echo "#PBS -l pvmem=${joboption_memory}mb" >> $LRMS_JOB_SCRIPT
  # echo "#PBS -l pmem=${joboption_memory}mb" >> $LRMS_JOB_SCRIPT
  echo "#PBS -l mem=${joboption_memory}mb" >> $LRMS_JOB_SCRIPT
fi

Scaling up the virtual memory for a factor of two

if [ ! -z "$joboption_memory" ] ; then
  echo "#PBS -l pmem=${joboption_memory}mb" >> $LRMS_JOB_SCRIPT
  joboption_memory=`expr $joboption_memory \* 2`
  echo "#PBS -l pvmem=${joboption_memory}mb" >> $LRMS_JOB_SCRIPT
fi

-- AndrejFilipcic - 18 Dec 2009

This topic: Main > TWikiUsers > AndrejFilipcic > AtlasNG > AtlasNGSetup
Topic revision: 07 Jun 2011, AndrejFilipcic
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback