Milu

Table of Contents


What is Milu?

Milu is the MIramare Lite User interface developed within the Sissa/eLab. It is based and it is an evolution of the ICTP_UI and EGRID_UI and is currently maintained by Tyanko Aleksiev. For further information and requests on this project please contact S.Cozzini ( cozzini _at_ democritos _dot_ it)

The Milu3.1 is a re-packaging of the gLite UI software version 3.1; its features are the following:

  • Installing it WON'T pollute your environment with tens of environment variables and path components; all you need is to add just 1 directory to your ``PATH``! This directory will be automatically added during the execution of the install script.
  • It has been tested and is known to be working on several GNU/Linux distributions, including:
    • Scientific Linux 4.6
    • CentOS 5.3
    • Ubuntu 9.04
    • Fedora 11

Note: The software is working properly on Scientific Linux 4.6: for other Linux distributions some additional settings and packages must be necessary for the correct functionality. These settings are described below in the section Additional Settings.

Download


You can download the latest version of the Miramare Lite User Interface from this URL: http://www.escience-lab.org/software/Milu31-20091210-2.tar.gz

Older releases can be found at the URL: http://euindia.ictp.it/the-ictp-ready-ui.

Milu and ARC


The NorduGrid middleware (or Advanced Resource Connector, ARC) is an open source software solution. Current production release of ARC middleware is ARC v0.6.5. It is sometimes called "NorduGrid middleware". Currently we integrated inside Milu software the ARC UI solution. It is a light-weight command line client. The ARC User Interface is a set of tools to submit, monitor and manage jobs on the Grid, move data around and query resource information. The User Interface comes with a built-in broker, which is able to select the best matching resource for a job. Grid job requirements are expressed via xRSL. Currently we tested the ARC UI compatibility with the following GNU/Linux distributions:

  • CentOS 5.3

So, in order to have ARC UI on your desktop you must simply install tha Milu3.1 software following the instructions bellow.

Watch Out!


The Miramare Lite User Interface software is very picky and may fail mysteriously if your host does not comply with the following requirements:

The system time must be correct! That is, your timezone must be correctly set, and your host's idea of the current time must be within five minutes from the actual correct time. If the time is not correct, you will most likely get authentication related errors.

Your firewall must allow *outgoing* connections to the configured grids. In particular, you will need to connect:

  • to TCP port 2170 of the BDII host;
  • to TCP port 7513 of the MyProxy host;
  • to TCP ports 7772,9000,9001,9002 to the RB and WMS host;
  • to TCP ports 2119,2811,20000--25000 of any CE;
  • to TCP ports 2811,5001,8443,20000--25000 of any SE;
  • to TCP ports 8080,8088,8443 of the MON and REG hosts;
  • to TCP ports 5010 of the LFC host.

For a complete listing, please see: https://twiki.cern.ch/twiki/bin/view/LCG/LCGPortTable#The_middleware_port_list

In actual practice, it is simpler to just allow *all outgoing* connections.

Installation


This section explains the installation process:

1. These instruction assume that you are running '/bin/bash' as your shell. To ensure you run bash, just type:

 bash

2. Untar the distribution package 'Milu31.tar.gz'.

 tar -xzvf Milu31.tar.gz -C /path/to/install_dir

This will extract all files in the directory /path/to/install_dir/Milu3.1

3. Cd to the installation directory:

 cd /path/to/install_dir/Milu3.1

4. Run the install script:

install.sh

or

./install.sh "VO_name" 

If you use the second method you automatically choose "VO_name" as your default VO. To change the VO after the installation please see: How to change the configuration

"VO_name" is one of following VOs:

  • "egrid" - for configuring the UI to connect to the ICTP production Grid in INFNGrid
  • "euindia" - the EUIndiaGrid testbed
  • "eela" - E-infrastructure shared between Europe and Latin America
  • "gilda" - Grid Infn Laboratory for Dissemination Activities
  • "gridseed" - a Demo Grid developed from eLab and used for educational purpose.
  • "compchem" EGEE CompChem Virtual Organization (support added with the help of Alessandro Costantini)

5. ICTP software requires a properly setup environment; to ensure your execution environment is properly set up, please log out and log back in again. If something has gone wrong, just run the installation script again.

 bash

Milu3.1 as standard UI for all the users of a Linux Box

(It is still under construction but the final version will be soon available. )

If you want to use Milu3.1 as UI interface for all the users on you Linux Box you can proceed as follow:

1. Login as root

2. Download the package Milu31.tar.gz.

3. Create a directory where to unpack Milu (for instance ui in the /opt directory):

 [root@ui-2 /]# mkdir /opt/ui
 [root@ui-2 /]# cd /opt/ui

4. Untar the package:

[root@ui-2]# tar -xzvf Milu31.tar.gz

5. Go to the Milu3.1 directory

[root@ui-2 ui]# cd Milu3.1/

and install the Milu3.1 software how described in the above Installation section. Before continue be sure that the Milu3.1 software is working properly.

6.To enable every user to use the Muli3.1 software on her/his login place the following simple script in the:

/etc/profile.d/ 

directory.

>cat milu31exp.sh
export MILU_LOCATION=/opt/ui 
export ICTP_INSTALL_DIR=$MILU_LOCATION/Milu3.1
export PATH=$PATH:$MILU_LOCATION/Milu3.1/bin
export MANPATH=$MANPATH:$MILU_LOCATION/Milu3.1/sw/edg/share/man:$MILU_LOCATION/Milu3.1/sw/glite/share/man:
$MILU_LOCATION/ui/Milu3.1/sw/glite/man:$MILU_LOCATION/Milu3.1/sw/globus/man:$MILU_LOCATION/Milu3.1/sw/gpt/man:
$MILU_LOCATION/Milu3.1/sw/lcg/man:$MILU_LOCATION/Milu3.1/sw/lcg/share/man

Be sure that the permissions of the script are right. For example:

-rwxr-xr-x  1 root root  280 Sep 23 14:39 milu31exp.sh

7. You must change some permissions in order to get the users' access to the software resources. Go to the /ui/Milu3.1/sw/glite/ directory change the permissions of the /etc directory like this:

[root@ui-2 glite]# chmod 755 etc/

Remove the file vomses

[root@ui-2 glite] rm vomses
[root@ui-2 glite] mkdir vomses
[root@ui-2 glite] chmod 755 vomses/

Must me incuded the definition of the gridbox voms. The file is called gridbox-voms.grid.box

[root@ui-2 vomses]# cat gridbox-voms.grid.box
"gridbox" "voms.grid.box" "15000" "/O=GRIDBOX/DC=box/DC=grid/OU=Host Certificate/CN=voms.grid.box" "gridbox"

[root@ui-2 vomses]# chmod 644 gridbox-voms.grid.box

8. Go to the following directory:

~/ui/MIlu3.1/support/ 

it contains a script called Usersupp.sh. Executing it you must find out this output:

1. all
2. single
3. list
Please digit all to add support of Milu3.1 for all users?
Please digit single to add support for a single user.
Please difit list to get the list of all the users and the relative support
followed by [ENTER]:

It is used to add support for all, single user or simply to see which user support the Milu3.1 software.

Additional Settings/Specific Instructions for Different Distribution

Generic Linux Distribution


  • Java Installation:

The Miramare Lite User Interface apparently depends on the Java VM 1.5.x or Java VM 1.5.x. The install script will issue warnings if it is not installed. With most modern distributions of Linux, Java comes already pre-installed, so there is no need to do anything. Anyway for Java Generic installation you can use standard installation tools ( apt-get for debian based distribution / yum for Redhat based distribution).

For debian based:

apt-get install j2re 

For RedHat based:

yum install j2re 
  • Missing packages:

Some distribution cannot have installed by default some packages required by gLite middleware. Here below the list of the most common ones and how to install them.

glite UI needs the the perl-ldap a collection of perl modules which provide an object oriented interface to LDAP servers. To install them (as root):

On Debian/Ubuntu:

sudo apt-get install libauthen-simple-ldap-perl 

On Redhat/Fedora:

yum install perl-LDAP 
  • We report on all distribution the following (harmless as far as we noted) warning messages issued by some gLite commands:
 1.Creating a proxy certificate con voms-proxy-init
Cannot find file or dir: /home/user/Milu31/sw/opt/glite/etc/vomses
 2. During a job submission con glite-wms-job-submit
Warning - Unable to find glite installation
no installation in /opt/glite or in /usr/local ; neither GLITE_WMS_LOCATION nor GLITE_LOCATION are set

Scientific Linux 4.6


The Milu3.1 project is based and developed under the Scientific Linux 4.6, so it should run without huge problems on that distribution. Common problems' solutions are presented in the Generic Linux Distribution section.

When using glite-wms-job-submit complain for a missing libcurl.so.3 library, to bypass this problem go in the $MILU_INSTALL_DIR/sw/symlibs and create the following symbolic link:

ln -s /usr/local/lib/libcurl.so.4  libcurl.so.3

CentOS 5.3


Milu3.1 is tested on the CentOS 5.3 distribution. No particular problems are verified. Common problems' solutions are presented in the Generic Linux Distribution

Ubuntu 9.04


1. For newly installed distribution, the Milu3.1 software complains for missing Java Runtime Environment. For quick installation just type:

sudo apt-get install sun-java6-jre

this will automatically install JRE on your computer.

2. Milu3.1 complains for missing a collection of perl-ldap modules. For installation see the Missing packages section.

3. On testing the software complains for missing some cURL libraries. For installation:

sudo apt-get install curl

4. When using voms-proxy-init complain for a missing libexpat.so.0 library, to bypass this problem go in the $MILU_INSTALL_DIR/sw/symlibs and create the following symbolic link:

ln -s /usr/lib/libexpat.so.1 libexpat.so.0

5. When using the glite-wms-job-status complain for missing python2:

/usr/bin/env: python2: No such file or directory

to bypass this problem please go in the $MILU_INSTALL_DIR/bin and create the following symbolic link:

ln -s ../sw/external/usr/bin/python python2

Fedora 11


1. Milu3.1 complains for missing a collection of perl-ldap modules. For installation see the Missing packages section.

2. When using voms-proxy-init complain for a missing libexpat.so.0 library, to bypass this problem go in the $MILU_INSTALL_DIR/sw/symlibs and create the following symbolic link:

ln -s /lib/libexpat.so.1 libexpat.so.0

3. When using glite-wms-job-submit complain for a missing libcurl.so.3 library, to bypass this problem go in the $MILU_INSTALL_DIR/sw/symlibs and create the following symbolic link:

ln -s /usr/lib/libcurl.so.4 libcurl.so.3

4. When using the glite-wms-job-status a harmless python's version mismatching appears.

To bypass this problem please go in the $MILU_INSTALL_DIR/bin and create the following symbolic link:

ln -s ../sw/external/usr/bin/python python2

Testing


Please follow the following information to test the correct functionality of the Milu UI.

1. Be sure that you have your personal certificates in the ~/.globus/ directory. During the installation Milu will alert you if the certificates are missing.

2. The Grid Information System permits you to find out the resources available for a specific VO. There are two fundamental command to be tested: lcg-infosites and lcg-info. The output produced by the commands should be simmilar to:

lcg-infosites --vo <vo> all
valor del bdii: bdii.grid.seed:2170
#CPU	Free	Total Jobs	Running	Waiting	ComputingElement
----------------------------------------------------------
   2	   2	   0	          0	   0	ce-2.grid.seed:2119/jobmanager-lcgpbs-gridbox
   4	   4	   0	          0	   0	ce-1.grid.seed:2119/jobmanager-lcgpbs-gridbox
Avail Space(Kb) Used Space(Kb)  Type	SEs
----------------------------------------------------------
47628920        86084           n.a 	se.grid.seed
lcg-info --vo <vo>  --list-ce --attrs 'RunningJobs,FreeCPUs'
- CE: ce-1.grid.seed:2119/jobmanager-lcgpbs-gridbox
  - RunningJobs         0
  - FreeCPUs            4

- CE: ce-2.grid.seed:2119/jobmanager-lcgpbs-gridbox
  - RunningJobs         0
  - FreeCPUs            4

where the <vo> is the name of your Virtual Organization.

3. For testing and then using the UI you must have a valid user certificate. When you have your credentials installed correctly you must generate a Grid proxy using the the following command:

voms-proxy-init -voms  <vo>

where <vo> is the VO name. The output must be similar to:

Cannot find file or dir: ~/Milu3.1/sw/opt/glite/etc/vomses
Enter GRID pass phrase:
Your identity: /O=GRIDSEED/DC=seed/DC=grid/OU=Personal Certificate/CN=Tyanko Aleksiev
Creating temporary proxy ......................................... Done
Contacting  voms.grid.seed:15000 [/O=GRIDSEED/DC=seed/DC=grid/OU=Host Certificate/CN=master.grid.seed] "gridseed" Done
Creating proxy ..................................................... Done
Your proxy is valid until Tue Jul  7 23:41:42 2009

4. We'll submit a very simple job to test the UI. A valid proxy certificate must be created first, as shown in section 3.

  • Create a file hello.jdl using your favourite text editor (pico, nano, emacs, vi).
[
  Executable = "/bin/echo";
  Arguments = "Hello from $HOSTNAME";
  StdOutput = "hello.out";
  StdError = "hello.err";
  OutputSandbox = {"hello.out", "hello.err"};
]
  • Submit the job using the following command
glite-wms-job-submit -a -o jobid  hello.jdl

The otput should be similar to this:

Connecting to the service https://wms.grid.seed:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://wms.grid.seed:9000/EeIEZVi31yv6as1TA6BLiA

The job identifier has been saved in the following file:
/home/user01/jobid
  • Check job status command con:
glite-job-status -i jobid

*************************************************************
BOOKKEEPING INFORMATION:

Status info for the Job : https://wms.grid.:9000/EeIEZVi31yv6as1TA6BLiA
Current Status:     Done (Success)
Logged Reason(s):
    - 
    - Job terminated successfully
Exit code:          0
Status Reason:      Job terminated successfully
Destination:        ce-1.grid.box:2119/jobmanager-lcgpbs-gridbox
Submitted:          Sat Sep 13 16:56:15 2008 CEST
*************************************************************

5. Check a simple data management command.

lfc-ls -l /grid/<vo>

You would see a listing of the contents of the /grid/<vo> directory.

Automatic Testing


In the $MILU_INSTALL_DIR/test/ directory you should find a script called test_ui.sh.
It is still in its beta version. The script tests not only the functions described in the Testing section but also some others. After finishing, in the $MILU_INSTALL_DIR/test/ directory you should find a report.DATA file which reports the parts where some problem occurred.
In the $MILU_INSTALL_DIR/test/log/ directory a report.DATA.tar.gz file is created containing the output and the errors produced by all the included parts.

How to Change Virtual Organization


The command:

milu_change_VO

allows you to change the actual configuration. The Milu3.1 comes with 5 available configurations:

  • euindia
  • egrid
  • eela
  • gridseed
  • gilda
  • compchem ( recently added with the help of Alessandro Constantini)

invoke milu_change_VO with the name of one of the above-listed configurations to change your UI settings. For instance:

milu_change_VO gridseed

How to support a new Virtual Organizzation


Other VO support can be added simply taking the following steps:

1. Go to $MILU_INSTALL_DIR/sw/etc/profile.d/VO directory. You have to add a supporting script for the VO you want to add. The following main parameters must be added:

export BDII_HOST=
export CE_HOST=
export LCG_CATALOG_TYPE=
export LCG_GFAL_INFOSYS=
export LCG_GFAL_VO=
export LFC_HOST=
export PX_HOST=
export RB_HOST=
export SE_HOST=
export VO_"NAME"_DEFAULT_SE=
export VO_"NAME"_VOMSES=
export VO_"NAME"_VOMS_SERVERS=
export VO_NAME=
export WMS_HOST=

where "NAME" is the name of the VO which must be supported. If some other parameters are specified please include them. An example of the euindia.sh script is reported below.

export BDII_HOST=eu-india-03.pd.infn.it
export CE_HOST=prod-ce-01.pd.infn.it
export LCG_CATALOG_TYPE=lfc
export LCG_GFAL_INFOSYS=eu-india-03.pd.infn.it:2170
export LCG_GFAL_VO=$VO_NAME
export LFC_HOST="lfcserver.cnaf.infn.it"
export PX_HOST=myproxy.cnaf.infn.it
export RB_HOST=eu-india-02.pd.infn.it
export SE_HOST=prod-se-02.pd.infn.it
export VO_EUINDIA_DEFAULT_SE="$SE_HOST"
export VO_EUINDIA_VOMSES="euindia voms2.cnaf.infn.it 15010 /C=IT/O=INFN/OU=Host/L=CNAF/CN=voms2.cnaf.infn.it euindia"
export VO_EUINDIA_VOMS_SERVERS="vomss://voms2.cnaf.infn.it:8443/voms/euindia?/euindia"
export VO_NAME=euindia
export WMS_HOST=eu-india-02.pd.infn.it

2. Give the right permissions of the file using:

 chmod 644 VO_name.sh

3. Add the name ot the VO in the $MILU_INSTALL_DIR/sw/etc/profile.d/VirtualOrganizations file. Save and close the file.

4. You must modify the milu_change_VO.sh script also. It is in the $MILU_INSTALL_DIR/bin directory. The name of the VO must be included in the list

if [ "$vo" = "egrid" ] || ... || [ "$vo" = "gilda" ]

so please add:

|| [ "$vo" = "VO_name" ]

where "VO_name" is the name of the VO. Save and close the script.

5. Modify the install.sh script in the $MILU_INSTALL_DIR/ directory. You must add the vomses for the VO you want to include. At the end of the install.sh script (line 577) are difined the vomses of the VO supported by the UI. For example the vomses of euindia is like this:

"euindia" "voms2.cnaf.infn.it" "15010" "/C=IT/O=INFN/OU=Host/L=CNAF/CN=voms2.cnaf.infn.it" "euindia"'

Please add the vomses to the VO you want to include. WARNING! Please write the definition of the vomses on the same line, if not some strange problems of authorisation may occur. Save and close the install.sh script.

6. You must add the configuration files for the WMSclient. Go to the following directory:

$MILU_INSTALL_DIR/sw/glite/etc/ add a directory with the name of the VO you want to support for instance:

mkdir euindia

this directory must contain the following two files:

-rw-r--r--  1 tyanko tyanko 572 May  8 14:54 glite_wms.conf
-rw-r--r--  1 tyanko tyanko 263 May  8 14:54 glite_wmsui.conf

The contens of the glite_wms.conf is like this:

[
    WmsClient = [
        virtualorganisation = "euindia";
        requirements = other.GlueCEStateStatus == "Production";
        MyProxyServer = "myproxy.cnaf.infn.it";
        WMProxyEndpoints = {
            "https://eu-india-02.pd.infn.it:7443/glite_wms_wmproxy_server"
        };
        ListenerStorage = "/tmp";
        ErrorStorage = "/tmp";
        ShallowRetryCount = 10;
        AllowZippedISB = true;
        PerusalFileEnable = false;
        rank =- other.GlueCEStateEstimatedResponseTime;
        OutputStorage = "/tmp";
        RetryCount = 3;
    ];
]

The contens of the glite_wmsui.conf is like this:

[
        VirtualOrganisation = "euindia";
        HLRLocation = "";
        NSAddresses = {
            "eu-india-02.pd.infn.it"
        };
        LBAddresses = {
            "eu-india-02.pd.infn.it"
        };
        MyProxyServer = "myproxy.cnaf.infn.it";
]

7. Pay attention that during the installation are downloaded certificates only for the voms server of egrid, euindia and gridats. So you must download the voms server certificate you need manualy and put it in the $MILU_INSTALL_DIR/sw/etc/grid-security/vomsdir/ directory.

8. Remove the actual installation (see uninstallation) and reinstall the UI. Using the install.sh script

 sh install.sh

or

./install.sh "VO_name"

Uninsallation


To uninstall the distribuition of the Miramare Lite UI on your machine you must:

1. Go to your home directory.

2. Remove the symbolic link to glite directory.

rm .glite

3. Remove the part riguarding the UI installation in your $HOME/.bashrc.

4. Remove the whole distribution from your machine

rm -rf MILU_INSTALL_DIR 

MILU_INSTALL_DIR is the directory where all the UI software is.