How to add a /robots.txt to DPM, LFC or DYNAFED for excluding HTTP bots

Configuring Apache so that it exposes a file names /robots.txt implies these two steps:

Add a line in the config file zlcgdm_dav.conf like the following:

# Robots.txt
Alias /robots.txt /usr/share/lcgdm-dav/robots.txt

Add a file named /usr/share/lcgdm-dav/robots.txt *readable by the user running the apache daemon with this content:


# Disallow all bots
User-agent: *
Disallow: /

The current trunk release 0.16 already has this. Pre-existing installations should do the manual thing anyway, as the config files cannot be overwritten.

Tags: 

You are here