Hello, we have pushed to EPEL the dmlite packages of DPM 1.14.0. As usual, It may take two or three days for it to be available in the various mirrors. I wish to warmly thank all those who have contributed to it with their opinions, advice, testing and direct contributions. Attached are the release notes for this release Cheers Fabrizio
--------------------------------------------------------
Release Notes - LCGDM Development - Version dmlite 1.14.0
** Bug
* [LCGDM-2696] - Small leak when the verification of a macaroon fails
* [LCGDM-2842] - dpm-storage-summary.py not invoked correctly in the case of longer domain names
* [LCGDM-2843] - If it fails the mkdirp, the DomeIOHandler ctor prints some garbage in the log
* [LCGDM-2875] - Revenge of the globus-gridftp race condition
* [LCGDM-2886] - lcgdm-dav custom errorDocuments are not loaded on Apache 2.4
* [LCGDM-2887] - Errors on recursive ACL commands do not report the file/folders involved
* [LCGDM-2891] - Possible DomeMetadataCache crash
* [LCGDM-2892] - Dirty double execution of TaskExec tasks
* [LCGDM-2893] - Newly inserted files/dirs should not show atime/mtime/ctime = 0
* [LCGDM-2899] - Allow deleting a quotatoken also if it points to a non existing pool
* [LCGDM-2906] - Add "before" as default caveat to the produced macaroons
* [LCGDM-2908] - WebDAV X.509 delegation disabled by wrong Authorization header
* [LCGDM-2909] - The disk server appears not to honour 'Credential: none' ?
* [LCGDM-2917] - Symbol conflict between json-c and jansson libraries
* [LCGDM-2923] - rmdir -r does not work
* [LCGDM-2924] - MKCOL should return 409, not 405 for an already existing directory
* [LCGDM-2932] - dmlite-puppet-dpm-1.14.0-1.el6.noarch has dependency /opt/puppetlabs/puppet/bin/ruby which is not available.
** New Feature
* [LCGDM-2880] - Configurable WebDAV TPC speed limits
* [LCGDM-2881] - Add support for RemoteConnections performance markers
** Task
* [LCGDM-2849] - Puppet modules in DMLite: add a script that creates the tarballs
* [LCGDM-2879] - Don't allow to specify quotatoken size smaller than associated pool minimum free space
* [LCGDM-2885] - Add SSLCARevocationCheck conf for Apache 2.4
* [LCGDM-2888] - Implement getLFN from replica inode
* [LCGDM-2903] - Add to puppet a switch to override the memory allocator library on centos7
* [LCGDM-2905] - add to puppet a switch to tune xrootd idle threads timeout
** Improvement
* [LCGDM-2493] - Fix Coverity defects
* [LCGDM-2563] - Make sure that DPM dome-only has a working argus poller
* [LCGDM-2634] - Add tests for the xrootd checksums to dpm-tester
* [LCGDM-2788] - dpm-listspaces should report to bdii if dome is active or not
* [LCGDM-2882] - Avoid exception trying to get file descriptor from NULL
* [LCGDM-2883] - Correct use of strerror_r function
* [LCGDM-2902] - Poor dome-checksum performance with default buffer size
* [LCGDM-2910] - WebDAV with tokens should use secure redirection by default
* [LCGDM-2911] - Update list of supported TLS protocols and ciphers
* [LCGDM-2918] - Configurable TPC trusted certificates
* [LCGDM-2937] - DomeAdapter should honour the groupnames suggested by getIdMap
Follows the list of commits, representing various enhancements
(HEAD -> master, tag: v1.14.0e, origin/master, origin/HEAD) - Requires: xrootd >= 1:4.12.3 (Fabrizio Furano)
- Merge branch 'puppet-boolean' into 'develop' (Fabrizio Furano)
- Add headnode FQDN in list of allowed OIDC audiences (Petr Vokac)
- LCGDM-2911 Puppet/hiera configurable Apache SSLProtocol (Petr Vokac)
- LCGDM-2935 Enable GridFTP redirection by default DPM DOME default config should not provide non-optimal configuration for GridFTP protocol where all data gets tunneled through DPM headnode. Enable GridFTP redirection by default, because this configuration is better for DOME and should work fine also with legacy DPM (SRM). (Petr Vokac)
- MKCOL on an existing resource now returns 405: METHOD_NOT_ALLOWED (Fabrizio Furano)
(tag: v1.14.0d, tag: v1.14.0c) - Replace legacy puppet fact (Petr Vokac)
- Apache is using MPM event by default on CentOS8 (Petr Vokac)
(tag: v1.14.0b) - Improve puppet configuration for EGI StAR accounting (Petr Vokac)
- Fix misspelled xrootd_async in puppet headnode classes (Petr Vokac)
- Forgotten file for typed puppet classes parameters (Petr Vokac)
- Better support for python3 build (Petr Vokac)
- Replace print with python logging for DB utils (Petr Vokac)
- Make xrootd_async configurable by puppet disknode declaration (Petr Vokac)
- Use typed parameter for main puppet classes (Petr Vokac)
- Prevent duplicit declaration of ca-policy-egi-core This package is normally declared/installed directly by puppet-fetchcrl (Petr Vokac)
- Remove dependency on finger package also from puppet (Petr Vokac)
- Update DPM puppet dependencies and drop support for puppet 4 (Petr Vokac)
- Use puppet-lint to unify module formatting (Petr Vokac)
- Add missing OIDC allowed audience puppet configuration (Petr Vokac)
- Allow multiple OIDC issuers in puppet configuration (Petr Vokac)
- Allow to configure jemalloc also on SLC6 (Petr Vokac)
- Remove redundant config.h generated by cmake (Petr Vokac)
- Enable debug logging also for HEAD request htext download (Petr Vokac)
- LCGDM-2927 protect fsadd to add special prefix /dpm (Petr Vokac)
- Don't assume that an authorization header is a macaroon (Fabrizio Furano)
- Better macaroon detection in the Auth header field (Fabrizio Furano)
- Puppet merge (Fabrizio Furano)
- Merge branch 'develop' of ssh://gitlab.cern.ch:7999/lcgdm/dmlite into develop (Fabrizio Furano)
- Bump version to 1.14.0 (Fabrizio Furano)
- Minimalistic support for pure python mysql client (Petr Vokac)
- By default overwrite existing destination file during WebDAV TPC https://twiki.cern.ch/twiki/bin/view/LCG/HttpTpcTechnical (Petr Vokac)
- dmlite packages doesn't realy depend on finger (Petr Vokac)
- Use VAR_STRING instead of VARCHAR type in MySQL prepare statement (Petr Vokac)
- LCGDM-2921 Allow access to parent directories with macaroons for HEAD, PROPFIND and MKCOL (Petr Vokac)
- Compilation fix (Fabrizio Furano)
- LCGDM-2924 GridFTP directory listing should not fail with wrong nlink (Petr Vokac)
- Fix compilation (Fabrizio Furano)
- Return NOT_ALLOWED if MKCOL of an existing dir (Fabrizio Furano)
- fix makedir (root)
- mod_dav: MKCOL should return 409, not 405 if the directory already exists (root)
- Compilation fix (Fabrizio Furano)
- MKCOL - try to catch the case where the parent is missing (Fabrizio Furano)
- dome_makedir: do it in a transaction, to protect from mkdir storms (Fabrizio Furano)
- Compilation fix (Fabrizio Furano)
- Allow MKCOL operations with a macaroon that allows PUT or MANAGE. Be less restrictive on matching the path. (Fabrizio Furano)
- Avoid sfn2lfn exception with missing parents (inconsistent db) (Petr Vokac)
- smuggle prometheus monitoring script into dmlite sources (Petr Vokac)
- Puppet template endl cleanup (Petr Vokac)
- LCGDM-2920 Basic OIDC authentication configuration (Petr Vokac)
- Fix typo in dmlite puppet hiera parameter (Petr Vokac)
- LCGDM-2919 Make XRootD request signing configurable by puppet (Petr Vokac)
- Prevent missing endl for long RemoteConnections performance marker (Petr Vokac)
- Update dirsize also for directories without any files (Petr Vokac)
- Added missing dir-size-offline fix for dirspaces script (Petr Vokac)
- Fixed wrong / reverse path size sorting used to match the closest quotatoken (Petr Vokac)
- LCGDM-2914 Avoid stat errors fo GridFTP uploads (Petr Vokac)
- Avoid GNU_SOURCE define in GridFTP plugin (Petr Vokac)
- LCGDM-2918 Configurable TPC trusted certificates Avoid hardcoded paths in source file and instead use apache configuration with compatible default values (Petr Vokac)
- Support for trace log of the TPC request (Petr Vokac)
- LCGDM-2917 Replace json-c with jansson (symbol conflicts) (Petr Vokac)
- LCGDM-2905 Puppet/hiera configurable DPM headnode xrd.timeout (Petr Vokac)
- LCGDM-2908 Failed X.509 delegation is OK with TransferHeaderAuthorization (not Authorization) header (Petr Vokac)
- LCGDM-2909 Use Credential header to avoid X.509 delegation (deprecate X-No-Delegate) (Petr Vokac)
- Enable WebDAV secure redirection by default to protect tokens (Petr Vokac)
- Enable jemalloc by default with puppet (Petr Vokac)
- Allow operations on the parent dir of a macaroon (Fabrizio Furano)
- Make macaroons easily detectable in the authorization field (Fabrizio Furano)
- ALso makedir shall support the oidc auth (Fabrizio Furano)
- Alwasy add before: caveat in macaroons with default validity 5min (Petr Vokac)
- Macaroons time should be UTC (Petr Vokac)
- Add before: to avoid immortal macaroons (Petr Vokac)
- Check the oidc_authorized flag also in dome_getdir (Fabrizio Furano)
- the last word about failing has to be given by the usual group based logic, not by the OIDC logic (Fabrizio Furano)
- Add storage.write to the scopes that can write (Fabrizio Furano)
- cleanup unused code (Petr Vokac)
- remove duplicate dome_addfstopool command (Petr Vokac)
- Document the new oidc config directives (Fabrizio Furano)
- Remove curl build files from git (Fabrizio Furano)
- Change the OIDC directives into head.oidc.* (Fabrizio Furano)
- Always pass exception by reference (Petr Vokac)
- Add basic support for builds with address sanitizer (Petr Vokac)
- LCGDM-2878: update directory size to the defined level (not level-1) (Petr Vokac)
- copatibility: even with python logging use stdout by default (Petr Vokac)
- More cmake fun with xrootd5 (Fabrizio Furano)
- Don't look specifically for xrootd v4 (Fabrizio Furano)
- Support of the openid-connect bearer tokens. First alpha implementation (Fabrizio Furano)
- Don't try the delegation if an Authorization field is present Honour the default non-grid ca-bundle in a third party copy (Fabrizio Furano)
- Be consistent with 10kB/s speed limit everywhere (Petr Vokac)
- Fix default 2 minute curl speed limit to 10kB/s (Petr Vokac)
- LCGDM-2902 change dome-checksum read buffer size to 1M (Petr Vokac)
- Allow deleting QTs that point to non existing pools. Just print a clear warning in the log and continue. (Fabrizio Furano)
- Better error message if permission denied on the parent path (Fabrizio Furano)
- Make sure that splitting a path never creates empty components (Fabrizio Furano)
- Better log reporting for the case of replicas in disabled or broken filesystems (Fabrizio Furano)
- Always use python logging withing dmlite-shell allow easier integration of existing external tools (dpm-dump, dmlite-mysql-dirspaces, ...) that already use standard python logging (Petr Vokac)
- Fix help formatting for dpm-tester default value (Petr Vokac)
- New files/dirs will not appear anymore with atime/mtime/ctime set to 0 (Fabrizio Furano)
- parse file status data from json response (Petr Vokac)
- add kDeleted file status also in C API (Petr Vokac)
- TaskExec: submitting a command should not start it Related to https://its.cern.ch/jira/browse/LCGDM-2892 (Fabrizio Furano)
- Protect the lfnfileidcache when removing info from the cache (Fabrizio Furano)
- LCGDM-2563 add argus banning support to puppet modules (Petr Vokac)
- LCGDM-2784 wrap php default config to be used only with prefork Event MPM is incompatible with mod_php - module is not loaded unless apache use prefork configuration. Unfortunatelly additional default php config file is installed without PHP specific config options wrapper in <IfModule prefork.c> section. (Petr Vokac)
- Update dmlite-shell Argus banning All basic functionality with user and group banning tested Changed format for alias selection, because '#' can't be used in shell (Petr Vokac)
- Forgot to assign user/group for cleaned serialization (Petr Vokac)
- Added dmlite-shell support for Argus banning Simplified support for Argus banning users (subect) and groups (vo,fqdn,pfqdn) with cli interface that look like (Petr Vokac)
- Log adding duplicate key in the Extensible (Petr Vokac)
- avoid duplicate keys while updating user/group info (Petr Vokac)
- it doesn't make sense to catch & raise exception (Petr Vokac)
- LCGDM-2887: print where the exception setting the ACLs happens (Andrea Manzi)
- Avoid partial regex matches (related to LCGDM-2884) (Petr Vokac)
- LCGDM-2886: add SSILegacyExprParser to the conf (Andrea Manzi)
- add SSILegacyExprParser on to properly handle custom error pages on apache 2.4 (Andrea Manzi)
- update metadata.json (Andrea Manzi)
- update source on metadata.json (Andrea Manzi)
- update deps, remove Puppetfiles (Andrea Manzi)
- Validate CRLs also with apache 2.4 Apache 2.4 comes with new configuration option SSLCARevocationCheck For details see https://httpd.apache.org/docs/2.4/upgrading.html (be aware that apache 2.2 ignored missing CRLs while apache 2.4 fails in case of missing CRL - this could cause troubles for CAs that doesn't provide CRLs - e.g. LE - but they are not IGFT certified, full compatibility with 2.2 can be archieved only with apache 2.4.21 and additional flag no_crl_for_cert_ok) (Petr Vokac)
- Fix prettySize formatting Function prettySize is called also with string argument and recent commit (70085b5430f4086a15a04ad6e10726c5e8061c08) did not took that into account (Petr Vokac)
- package puppet voms as an external dep (Andrea Manzi)
- LCGDM-2884: remove check for empty ACL (Andrea Manzi)
- LCGDM-2884: remove check for empty ACL (Andrea Manzi)
- Attempt to update packaged config files with recomended options (Petr Vokac)
- Allow to getlfn also by fileid (Petr Vokac)
- Fix signed datatype formatting (Petr Vokac)
- Python script minor cleanup with flake8 (Petr Vokac)
- Remove unused legacy code SELECT INTO FILE (Petr Vokac)
- Format python script according pep8 (Petr Vokac)
- LCGDM-2878 atomic parent directory size updates (Petr Vokac)
- Avoid exception trying to get file descriptor from NULL This is not full solution for useless backtraces during file upload with davs and gsiftp, but at least we don't try to use NULL while calling dmlite_fileno (Petr Vokac)
- Correct use of strerror_r function Function strerror_r has quite unexpected behavior in the _GNU_SOURCE (glibc) that is quite different from its POSIX variant - this is an attempt to use strerror_r correctly regardless enabled features. This should silence compiler warnings and also it is hopefuly correct solution for LCGDM-2843 (Petr Vokac)
- Add basic Perf Marker input validation (Petr Vokac)
- Working support for splitted Perf Marker data (Petr Vokac)
- Avoid bogus parsing warning (Petr Vokac)
- Fix problem with buffer usage while parsing Perf Markers (Petr Vokac)
- Remove useless logging (Petr Vokac)
- Initial support for RemoteConnections performance markers (Petr Vokac)
- Fix compiler warning for WebDAV speed limits (Petr Vokac)
- add puppetfile (Andrea Manzi)
- Provide also queue length with dome_info (Petr Vokac)
- fix few misspelled references in python shell (Petr Vokac)
- Avoid exception while listing empty pool (Petr Vokac)
- Add missing file metadata status (Petr Vokac)
- Checksum calculation for zero size files (Petr Vokac)
- Attempt to distinguish recently succeeded task vs. abort and timeout (Petr Vokac)
- Don't mix monotonic clock with time(0) (Petr Vokac)
- Speed limit configuration must be defined per directory / location (Petr Vokac)
- Log failed WebDAV transfers (Petr Vokac)
- Configurable WebDAV TPC speed limits Allow user to specify curl speed limits for TPC in the apache configuration file zlcgdm-dav.conf (Petr Vokac)
- LCGDM-2874 don't log timeout for finished tasks (Petr Vokac)
- don't allow to specify quotatoken size smaller than associated pool minimum free space (Petr Vokac)
- improve log level and log messages (Petr Vokac)
- bump release (Andrea Manzi)
- create base puppet folder (Andrea Manzi)
- Merge branch 'include_puppet' into 'develop' (Andrea Manzi)
- Sophisticate the workaround for the "gridftp race condition" in the checksum calculation Before starting, wait for the filesize to be 0 and at least one replica in status '-' LCGDM-2875 (Fabrizio Furano)
- Do not abort the build if aprutil is not found (Fabrizio Furano)
- LCGDM-2869: report an error in case the utime update is failing (Andrea Manzi)
(origin/include_puppet) - Add 'src/puppet/lcgdm/' from commit '24f5b354283f37765333d7164d81c91e3da13df6' (Andrea Manzi)
- Add 'src/puppet/gridftp/' from commit '50d7c7db3990e01f13d5b071d2a52df4c8739926' (Andrea Manzi)
- Add 'src/puppet/voms/' from commit '9ee4bc432fe117d25e2a690db0714be1c338c817' (Andrea Manzi)
- Add 'src/puppet/xrootd/' from commit '752869e519b2ab4b9b4558ce77ae916591f38c03' (Andrea Manzi)
- Add 'src/puppet/dmlite/' from commit '3d277eee5fd51bd0759861b24e572df2a7ba594b' (Andrea Manzi)
- Add 'src/puppet/dpm/' from commit 'e3064859f1cac581c59ccc9ff2440ee520439fe3' (Andrea Manzi)
- remove puppet modules as better import them from repo (Andrea Manzi)
- LCGDM-2849: move puppet moduled sources under dmlite source tree (Andrea Manzi)
- update puppet modules (Andrea Manzi)
- update puppetfile (Andrea Manzi)
- fix merge (Andrea Manzi)
- update README (Andrea Manzi)
- update changelog (Andrea Manzi)
- update readme (Andrea Manzi)
- update readme (Andrea Manzi)
- update README (Andrea Manzi)
- update changelog (Andrea Manzi)
- update dependency version (Andrea Manzi)
- update version (Andrea Manzi)
- update version (Andrea Manzi)
- new version (Andrea Manzi)
- update version and CHANGELOG (Andrea Manzi)
- disable star accouting by default and update version (Andrea Manzi)
- enable/disable star using configure_star parameter (Andrea Manzi)
- add a way to remove packages and conf if accounting is disabled (Andrea Manzi)
- fix syntax (Andrea Manzi)
- fix parameter name (Andrea Manzi)
- add new parameters for accounting class (Andrea Manzi)
- add db conf to accounting script, purge old cron (Andrea Manzi)
- Merge pull request #8 from vokac/update_start_accounting (Andrea Manzi)
- Merge pull request #30 from vokac/update_start_accounting (Andrea Manzi)
- simplify APEL StAR accounting usage (Petr Vokac)
- simplify APEL StAR accounting usage (Petr Vokac)