Submitted by furano on

This article summarizes the steps that are necessary in order to federate S3 and/or Azure storage to a Dynafed.

The outcome is that we are able to make these storage providers appear in the final federated namespace, in the desired subdirectory. Subdirectories in the cloud storage (S3/Azure) will work out of the box, with no action needed. Read/write access to those buckets will be managed by Dynafed through the usage of short term delegations, hence the keys will remain secret and managed by the manager of the Dynafed instance, which can be even local to a site or a group of sites.

The integration is totally dynamic, no catalogues are involved. One can integrate any number of buckets in Dynafed just by repeating the steps and giving different internal names to the instances. The instances of course can be installed in overlapping directory subtrees (or even the same one) and Dynafed will take care of properly presenting their content as merged on the fly. A bucket in principle can be added multiple times into the same federation, and our best recommendation is just to deploy them in a way that makes sense.

The authentication method exposed by the Dynafed instance will be the same authentication method that was choosen when configuring the Apache daemon running Dynafed. By default it will be X509/VOMS.

 

What's needed is:

  • A working, bassic Dynafed installation (please refer to the installation TWiki: https://svnweb.cern.ch/trac/lcgdm/wiki/Dynafeds
  • For S3: the two keys that give access to the bucket. For MS Azure the denomination is different, yet the analogous two keys are needed.
  • Using the Amazon or MS tools please doublecheck that the keys are correct and that they work.

 

Example:

We want to add the Amazon galadriel bucket into out dynafed, and make it appear in a directory names /s3-federation. The sysadmin needs to create a configuration file in the dynafed config directory (usually /etc/ugr.conf.d). The file name chosen here is /etc/ugr.conf.d/myclouds.conf

The keys listed in this example are just examples! They must be substituted with the actual S3 keys that need to be kept secret under the control of the sysadmin of the Dynafed instance.

 

# the galadriel bucket will be accessible in /s3 federation
glb.locplugin[]: libugrlocplugin_s3.so s3_aws_galadriel 15 s3://galadriel-bucket.s3.amazonaws.com
locplugin.s3_galadriel.xlatepfx: /s3-federation /
locplugin.s3_galadriel.s3.priv_key: 5plqAriNv7L2A5UdyK8lUu8X38fysAs7Th75Bmae
locplugin.s3_galadriel.s3.pub_key: AKIAIZFAIDN23NDBVYTQ

 

Analogously, if we want to do the same with an Azure share, a new little file /etc/ugr.conf.d/myazure.conf has to be created with content that is analogous to what follows:

 

## first bucket 
glb.locplugin[]: libugrlocplugin_azure.so myazure1 15 https://yeah.blob.core.windows.net/yeahcontainer
locplugin.myazure1.xlatepfx: /azure-s3-cephs3-clouds-together /
locplugin.myazure1.status_checking: false
locplugin.myazure1.writable: true
locplugin.myazure1.azure.key: da0FdqpI67GUXoKsfRflGLJD05zb4+gDJ+L6eWM7f3702czcAW3X6azuFBrjOv3OXKL9o2jXAMl5+cMgp8UgJg==

 

 

Caveats:

  • The above examples use the Amazon AWS protocol version 2, that is the default for Dynafed. There may be the need of using the Amazon AWS protocol version 4. In this case the syntax differs very slightly, and we refer the reader to the Dynafed documentation.
  • For a more in-depth description of what the configuration directives do, please refer to the documentation: http://lcgdm.web.cern.ch/dynafeds-text-documentation-white-paper
  • The authorization rules applied by Dynafed can be customised in a very flexible way. Please refer to the official documentation to do that.
  • The S3 delegation support works by signing temporary keys, hence the validity of the signed key (default: 3600 secs) must be longer than the maximum cache time. The chosen value must also accommodate a very likely time skew between the client and the server. In practice, some good values for S3 support are:

locplugin.*.s3.signaturevalidity: 7200
infohandler.itemttl: 600
infohandler.itemmaxttl: 1200
extcache.memcached.ttl: 3600 

 

Tags