Basic Data Management

Table of Contents

1. Introduction
2. Environment variables
3. Listing file and directory
4. Create directory
5. Upload file into SE
6. Get the file SURL
7. Replicate file between SE
8. Create symbolic link
9. Download a file from SE to UI
10. Clean all !

1. Introduction

The aim of this tutorial is to introduce file management in gLite. We will use running two sets of commands:

"lfc-" commands interact with the LFC catalogue server that maps logical filenames to "site URLs".
"lcg-" commands include ones used to copy files to and from an SE, and to replicate files.

The lcg-commands performs also interaction with both the SEs and the catalogue server.

2. Environment variables

Several environment variables need to be checked before you start to ensure that the correct catalog service is used. The default settings for these variables in your account should be correct. They sould have the following values:

 [user01@ui-1 ~]$ echo $LCG_GFAL_INFOSYS
 test-bdii.grid.seed
 [user01@ui-1 ~]$ echo $LCG_CATALOG_TYPE 
 lfc
 [user01@ui-1 ~]$echo $LFC_HOST
 central-1.grid.seed

If one or more of them has different or empty value, please set it (them) in this way:

export LCG_GFAL_INFOSYS=test-bdii.grid.seed:2170
export LCG_CATALOG_TYPE=lfc
export LFC_HOST=central-1.grid.seed

3. Listing file and directory

In order to use the LFC and LCG commands you should have a valid certificate generated using the command:

voms-proxy-init -voms gridseed

For each of the supported VO a separate "top level" directory exists under the "/grid" directory. You can see all the files that are stored for the gridseed VO. First ensure you have a running VOMS proxy and then type:

[user01@ui-1 ~]$ lfc-ls -l /grid/gridseed

You will see a listing of the contents of the /grid/gridseed directory.

4. Create directory

Before creating and uploading any of your own files create a personal directory for storage by typing (of course, here and in the next examples, you have to replace userdirectory with your account name ; so, don't just copy and paste!

[user01@ui-1 ~]$ lfc-mkdir /grid/gridseed/tutorials/userdirectory

To check that you have created your directory type:

[user01@ui-1 ~]$ lfc-ls -l /grid/gridseed/tutorials

and you should see your directory (plus possibly those of other attendees).

5. Upload file into SE

The next step is to upload a file into the directory you just created. First create locally a simple text file:

[user01@ui-1 ~]$ echo "Put something here" > text_file.txt

In order to find which SE are available you can use the following command:

[user01@ui-1 ~]$ lcg-infosites --vo gridseed se

You should see the similar results:


Avail Space(Kb) Used Space(Kb)  Type	SEs
----------------------------------------------------------
47628920        86084           n.a 	se-3.grid.seed
47628920        86084           n.a 	se-3.grid.seed
48772014080     88150016        n.a 	se-2.grid.seed
47628864        86140           n.a 	se-1.grid.seed
48772014080     88150016        n.a 	se-2.grid.seed
47628864        86140           n.a 	se-1.grid.seed

The command used for this is lcg-cr (LCG copy and register). Type the following to store this file on the se-3.grid.seed Storage Element.

[user01@ui-1 ~]$ lcg-cr --vo gridseed  -l lfn:/grid/gridseed/tutorials/userdirectory/text_file.txt
-d se-1.grid.seed file://$HOME/text_file.txt

The output should be something like this:

[user01@ui-1 ~]$ lcg-cr --vo gridseed -l lfn:/grid/gridseed/tutorials/userdirectory/text_file.txt  
-d se-1.grid.seed file://$HOME/text_file.txt 
guid:c8da3f42-9a84-47de-9f32-f10ef93c8daa

Of course, the guid you will have is different, since it is an unique identifier for each file (except for the case when you are replicating a file, as we will see). Check that the file is there by listing the contents of your directory.

[user01@ui-1 ~]$ lfc-ls -l /grid/gridseed/tutorials/userdirectory/

Before continuing it is worth noting the difference between the command used to store the file and the creation of the directory in previous. The directory created is just a virtual directory and only exists within the catalog of lfn's. On the other hand the file physically exists on an SE but has an additional "virtual" filename in the catalog. This is connected to why the commands just handling the lfn namespace tend to start with "lfc" whilst the commands manipulating the file directly tend to start with "lcg".

6. Get the file SURL

For several purpose, FTS for instance, is useful to know the file SURL (they can be many if the file has replicas somewhere). The appropriate command is lcg-lr (list-replicas) [lfn | guid]

[user01@ui-1 ~]$ lcg-lr --vo gridseed lfn:/grid/gridseed/tutorials/userdirectory/text_file.txt
srm://se-1.grid.seed/gridseed/generated/2008-09-15/file890e1013-7df5-4239-9250-b51360ec6387:

7. Replicate file between SE

gLite supports file replication. A file can be stored on multiple SE's and then a running job can access the closest SE with the file on it, thus giving faster access times to the data. This also helps protect against failures/access difficulties with a particular SE. To find the list of SEs available to you see the tutorial on lcg-infosites command. We will replicate the file just created to the SE se-1.grid.seed with the command

[user01@ui-1 ~]$ lcg-rep --vo gridseed -d dpm.grid.seed lfn:/grid/seed/tutorials/userdirectory/text_file.txt

There is no output from this command on success, but you can check that the replica was created by listing all the replicas of your file, that is done by using the LCG list replicas command:

[user01@ui-1 ~]$ lcg-lr --vo gridseed lfn:/grid/gridseed/tutorials/userdirectory/text_file.txt

You should get two replicas listed, as here:

[user01@ui-1 ~]$ lcg-lr --vo gridseed lfn:/grid/gridseed/tutorials/userdirectory/text_file.txt 
srm://dpm.grid.seed/gridseed/generated/2008-09-15/file890e1013-7df5-4239-9250-b51360ec6387
srm://se-1.grid.seed/gridseed/generated/2008-09-15/file4d64e218-2179-4aa1-8454-a5dd191c3441

Note how the path to where each file is stored is different. This demonstrates how the use of a "lfn" avoids the need to understand the local filesystem where the replica is actually stored.

8. Create symbolic link

As an example of some of the other functionality available a second lfn for the file you uploaded is created. This is very similar to the Unix symbolic link. A second lfn is created with the command:

[user01@ui-1 ~]$ lfc-ln -s /grid/gridseed//text_file.txt /grid/gridseed/tutorials/userdirectory/text_file_symlink.txt

Note the file path for the source file was used so as to avoid problems with relative links being used from the wrong directory. If you now list the contents of your directory you should see both names, but the symbolic link has 0 size and also the link target is shown:

[user01@ui-1 ~]$ lfc-ls -l  /grid/gridseed/tutorials/userdirectory
 -rw-rw-r--   1 554      102              19 Jul 19 15:36 text_file.txt
 lrwxrwxrwx  1 554      102                0 Jul 19 15:37 text_file_symlink.txt -> 
/grid/gridseed/tutorial/userdirectory/text_file.txt

9. Download a file from SE to UI

Having already uploaded a file the next step is to show downloading a file. To download the file you already uploaded using the new lfn you have just created use the command:

[user01@ui-1 ~]$ lcg-cp --vo gridseed lfn:/grid/gridseed/tutorials/userdirectory/text_file.txt 
file://$HOME/text_file_copy.txt

You can check that the file you just downloaded is the same as the file you uploaded with cat command, you should find the same text you entered below.

[user01@ui-1 ~]$ cat text_file_copy.txt

10. Clean all !

You can delete a file from SE with lcg-del

[user01@ui-1 ~]$ lcg-del -a  --vo gridseed lfn:/grid/gridseed/tutorials/userdirectory/text_file.txt

This will remove File Catalog entries as well

[user01@ui-1 ~]$ lfc-ls /grid/gridseed/tutorials/userdirectory
[user01@ui-1 ~]$

complete by removing the working directory on file catalog

[user01@ui-1 ~]$ lfc-rm -r /grid/gridseed/tutorials/userdirectory
[user01@ui-1 ~]$ lfc-ls /grid/gridseed/tutorials | grep userdirectory