NEWS

psmgmt NEWS -- history of user-visible changes.

Copyright (C) 2010-2021 ParTec Cluster Competence Center GmbH, Munich
Copyright (C) 2021-2024 ParTec AG, Munich

Please send bug reports, questions and suggestions to <support@par-tec.com>


Version 5.1.60-1:
=================
Bugfixes:
 - psslurm: respect step core map with cpu-bind=sockets (psc:#449)
 - psslurmgetbind: Adjust coremap generation (psc:#446)
 - psslurmgetbind: Allow option combinations (psc:#451)
Enhancements:
 - Print message if core requested to pin does not match step core map
 - psslurmgetbind: Print core map if PSSLURM_PRINT_COREMAPS set
Additional changes:
 - psslurmgetbind: Change default Slurm version to 23.02


Version 5.1.60:
===============
Bugfixes:
 - Don't stop psaccount collect scripts on temporary error (#186)
 - Ensure PSP_DD_CHILDRESREL is sent without delay (#177)
 - Ensure -x and -H (-N, -f) work together in mpiexec (#160)
 - psiadmin has to expect local answer on plugin (#167)
 - Rework mpiexec's environment handling (#182)
 - Ensure building works even with missing NUMA support (#12)
 - Fix potentail memory leak and unexpected NULL attribute unveiled by Clang
 - Fix warnings unveiled by recent Clang versions
 - Fix psidsession information when spawning w/out Slurm
Enhancements:
 - Add psslurm configuration option SSTAT_USERS (psc:#438)
 - Add extra variables to client environment (#188)
   * PS_SESSION_ID, PS_JOB_ID, PS_RESERVATION_ID, PS_JOB_RANK, PS_JOB_SIZE
 - Implement --pset option for mpiexec (#110)
 - Add support for OCI containers to psslurm (via Slurm's oci.conf)
 - Add PAM session support for user processes in psslurm
 - Prepare for PMIx_Spawn() support -- this is not yet complete
 - Close psidforwarder's logger connection on dropped FINALIZE message
Additional changes:
 - Get rid of the OpenMPI tweeks in mpiexec
   * This is obsoleted for a long time by native Slurm support
 - Get rid of starters for ancient MPI versions
 - PMIx tests: Add spawn and simple abort tests
 - Add script to calculate cpumap from /proc/cpuinfo
 - Rename __PMI_SPAWN_SERVICE_RANK to __SPAWNER_SERVICE_RANK
 - Introduce PSIDHOOK_EXEC_CLIENT_EXEC and bump plugin API to 141
 - Push rrcomm's plugin version to 2
 - Make PSID_execFunc() usable outside psid itself
 - Publish loadPlugin(), findPlugin() as PSIDplugin_load(), PSIDplugin_find()
 - Move spawn definitions into common pluginspawn.[ch]
 - Rework psidspawn's testExecutable() and changeToWorkDir()
 - Introduce mmapFile() in pluginhelper
 - Rework env_t to make it opaque, rework filter functionality, etc.
 - Allow stealing of the whole array from strv_t
 - IWYU fixes (after update to version 0.22)
 - Add Intel Xeon MAX topology w/ and w/out flat mode recorded with v2.2


Version 5.1.59-5:
=================
Bugfixes:
 - Ensure reading from cached pack info messages start from the start


Version 5.1.59-4:
=================
Bugfixes:
 - Adopt to changed return code in libbpf 1.0.0 and above (jwt:#26118)
 - Fix handling of srun's --threads-per-core for pinning (psc:#443)
 - Rework psslurm's fillHints() avoids random 'invalid hint' (psc:#440)
Enhancements:
 - Introduce PSSLURM_VERBOSE_PINNING to print additional information
 - Suppress more error messages if quiet flag is given in bpf_loader


Version 5.1.59-3:
=================
Enhancements:
 - Update bpf_cgroup_device for compatibility with Rocky 9
Additional changes:
 - Rename bpf_cgroup_device to prevent automatic removal (#176)


Version 5.1.59-2:
=================
Bugfixes:
 - Evict false positive warnings from syslog (#178)


Version 5.1.59-1:
=================
Bugfixes:
 - Stop hardly blocking build of BPF support in RHEL9, Rocky9, etc.


Version 5.1.59:
===============
Bugfixes:
 - Adopt to new error behavior of Slurm 23.02 (jwt:#24558)
 - Execute SPANK hooks in batchscript context (deep:#3325)
 - Prevent possible DOS in psserial (#172)
 - Ensure pluginforwarder sends sufficient data
 - Prevent segfault when psaccount gets insufficient answer
 - Prevent memory leak when pscommon's logger is re-initialized
 - Set correct size for allocation of Req_Signal_Tasks_t
 - Fix various memory leaks, double free(), etc. found during development
 - Various fixes unvealed by scan-build
Enhancements:
 - Add a generic energy readout script (psc:#411)
 - Optimize PSI_spawnRsrvtn() (#155)
 - Ensure slurmutils is installed with matching Slurm
 - Set correct job and step memory limits for SPANK
 - RRComm v2 supporting messages across jobs (created by MPI_Comm_spawn())
 - Allow psmgmt to be built without SPANK support (#171)
Additional changes:
 - Remove the cgroup plugin (obsoleted by jail) (#174)
 - Rework PStask_t's handling of extra information (#156)
 - Drop support for Slurm 19.05 protocol
 - Drop support to spawn to PSProtocolVersion < 341 (before 2019)
 - Use PS_DataBuffer for unpacking messages using psserial functions
 - Enable the compiler to verify types for more psserial functions
 - Introduce PSID_dbg() an replace PSID_log() as much as possible
 - Introduce psslurmprototypes
 - Remove unused PSI_spawnSingle()
 - Various IWYU fixes


Version 5.1.58-1:
=================
Bugfixes:
 - sss-nss re-uses closed fd (#159/jwt:#20747)
 - prevent segfault from termJail() (jwt:#24361)
 - ensure nLocalSlots is initialized (#162/jwt:#24377)
 - POSIX does not guarantee the getpwuid_r() et al. sets errno
Enhancements:
 - librrcomm needs -fPIC to be linked into libpscom


*************************************************************************
*************************************************************************
**                                                                     **
**  Since the pspelogue executable was moved, the slurmctld prologue   **
**  has to be adapted accordingly. The new location is libexecdir,     **
**  i.e. /opt/parastation/libexec/psmgmt/pspelogue                     **
**                                                                     **
**  psmgmt 5.1.58-0 will require psconfig 5.2.10 or later to work      **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.1.58:
===============
Bugfixes:
 - Pin only to cores allowed by the step in psslurm
 - Fix support for spawn in PMI
 - Ensure step forwarder task owns its strings
Enhancements:
 - Add user-facing error messages to psslurm's pinning
 - Handle hint compute_bound in psslurm
 - Print coremaps to stderr for debugging if PSSLURM_PRINT_COREMAPS is set
 - Make energy reporting managable by srun (meluxina #1242)
 - Cache user-facing messages until connection to srun appears
 - Add support for libbpf versions 1.0.0 and above (cgroup v2 on Rocky 9)
Additional changes:
 - Use share/psconfig/dumps for main psconfig dumps (requires psconfig 5.2.10)
 - Add flag termAfterFWmsg to job and step to stop immediately after
   delivery of queued messages to the user
 - Move psid to sbin directory
 - Move various helpers (pspelogue,ps_acc,psaccounter,psilogger) to libexecdir
 - Major rework psidsession (adressing tasks via session/job/app/rank)
 - Utilize psid (instead of psilogger) to check validity of
   installation directory
 - Introduce PSIDHOOK_LAST_RESRELEASED, PSIDHOOK_LAST_CHILD_GONE and
   bump plugin API version to 140
 - Introduce the concept of sister partitions and push daemon protocol
   to version 416
 - Allow PSIDpart_register() to extend a partition
 - Introduce PSID_flog() and PSID_fdbg()
 - Introduce PSIDfwd_inForwarder()
 - Move clrPartQueue() to PSpart_clrQueue() and make it public
 - Rework strv_t behavior


Version 5.1.57-3:
=================
Bugfixes:
 - Various fixes for cgroup v2 (jwt:#23342)
   * Sanitize calculations of jail limits, especially memory
   * Prevent race conditions leading to concurrent modifications of cgroup

Version 5.1.57-2:
=================
Bugfixes:
 - Delete allocation before sending MESSAGE_EPILOG_COMPLETE (jwt:#23342)

Version 5.1.57-1:
=================
Bugfixes:
 - Set correct state of allocation to ensure proper cleanup of processes
 - Set the correct sequence when setting cgroup memory limits
 - Ensure correct locking in cgroupv1
 - Fix race condition and ensure all SSH processes get jailed into cgroups
 - Ensure cgroupv2 controllers get enabled correctly
 - Fix quoting and other errors unveiled by shellcheck in jail scripts
 - Ensure local variable does not influence top level behavior
 - Various checks if data was unavailable from message
 - Avoid array boundary violation
Enhancements:
 - Print messages from PMIx server log interface
 - Improve cleanup of leftover cgroupv2 directories
 - Add caller name to jail log functions
 - Avoid superfluous mdsave call in cgroup jail script
 - Avoid temporary buffers of excessive size
Additional changes:
 - Various improvements in the build system (e.g. to support Rocky 9)
 - Fix gcc-11 warning emitted via rpmbuild on Rocky 9
 - Improvements of documentation comment

*************************************************************************
*************************************************************************
**                                                                     **
**  psmgmt 5.1.57-0 introduces support for cgroup v2. Since enhanced   **
**  cgroup support was moved into the jail-plugin machinery away from  **
**  the original cgroup-plugin, loading both plugins was useless but   **
**  not really harmful. This changes with psmgmt 5.1.57-0 when usage   **
**  of cgroup v2 support is enabled on the system and within Slurm.    **
**  In this specific case default settings for the cgroup-plugin will  **
**  clash with the expectations of the jail-plugin's cgroup machinery  **
**  and the standard setup of cgroups found on the systems             **
**                                                                     **
**  Therefore it is highly recommended to:                             **
**                                                                     **
**  - properly setup the cgroups machinery of the jail-plugin          **
**  - disable the use of the old cgroup-plugin                         **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.1.57:
===============
Bugfixes:
 - Fix jail memory check (#118)
 - Deny access to all devices not allocated by the user via cgroups
 - Cleanup leftover cgroupv1 psid directories
 - Do not let psslurm init fail if optional Spank plugins are broken
 - Do not overwrite jail information if multiple GRes have devices
 - Fix inet_ntop()'s second argument
Enhancements:
 - Introduce support for cgroup v2
 - Set default cgroup version to autodetect
 - Add psslurm set options SPANK_LOAD, SPANK_UNLOAD and SPANK_FIN (#pct:428)
 - Introduce support for port ranges in SlurmctldPort in psslurm
 - Send SIGTERM only in the first round of killing cgroup processes
 - psslurm takes hwloc info as default if SKIP_CORE_VERIFICATION is given
 - Avoid lapsed constant PMIX_ERR_NOT_IMPLEMENTED in PMIx 4
 - Mark processes started with PSID_execFunc() as non daemon
Additional changes:
 - Add support for changing IP addresses delivered in psmgmt-dynip
 - Introduce PSC_traverseHostInfo() to avoid getaddrinfo()'s boylerplate code
 - Introduce PSC_isDaemon() and rework PSC_setDaemonFlag()
 - Add adjusted psconfig dumps for base and defaults configuration
 - Refactor and streamline RDP


Version 5.1.56-2:
=================
Bugfixes:
 - Ensure cached PSP_PACK_INFO info is handled (#131)
 - Execute SPANK prologue/epilogue hooks without the presence of
   a prologue/epilogue script
 - Let doGetMsgBuf() (behind PSP_getMsgBuf() et al.) warn even if header.len is 0
Enhancements:
 - Ensure cores is not used in gres.conf
 - Include UID in PMIx' temp session dir name to help with node sharing
 - Add sentence on ParaStation ID -1 to psiadmin's inline help


Version 5.1.56-1:
=================
Bugfixes:
 - Set correct localNodeId for jobs on sister nodes (psc:#427)
 - Prevent spank_api.so from being dlopen()ed twice
 - Fix RPM's postun scriptlet
Enhancements:
 - Add debugging output in psslurm's findGresCred()


Version 5.1.56:
===============
Bugfixes:
 - Prevent segfault if resInfo is broken (jwt:#21216)
 - Don't let exiting shell spoiling output for steps with pty (jwt:#21114)
 - Fix quoting prevents device jail script to generate unexpected files (dt:3151)
 - Set correct cgroup configuration default if no cgroup.conf is present
 - Ensure configuration updates will remove obsolete entries
 - psslurm: Move SLURM_TRES_* setting place
 - Fix one improbable segfault unveiled by gcc 13.1
Enhancements:
 - pspmix: Use PSPMIX_ENV_TMOUT to steer environment timeout
 - Add jail configuration option JAIL_INIT_SCRIPT and the corresponding script
 - Save main psid PID to /run for jail scripts and pshc
 - Let pluginforwarder jail pspmix server processes
 - psslurm: Rework GPU pinning
 - Avoid silent fails in GPU pinning
 - Introduce spank_prepend_task_argv() expected for Slurm 23.11
 - Let RPMs depend on PMIx minor but release version
Additional changes:
 - pspmix: Print input to server_grp_cb()


Version 5.1.55-2:
=================
Bugfixes:
 - Prevent free() on uninitialized pointer (jwt:#20926)


Version 5.1.55-1:
=================
Bugfixes:
 - Ensure SPANK calls (esp. TASK_EXIT) are not called for service procs
Enhancements:
 - Extra analysis checks for jwt:#20747


Version 5.1.55:
===============
Bugfixes:
 - Ensure the correct order of SPANK hook calls
 - Add SPANK option handling at all necessary places
 - Actual use optional keyword of plugstack.conf
Enhancements:
 - Add support for Slurm 23.02 protocol
 - Add support for include statement in slurm.conf
 - Set all Slurm configuration files to case-insensitive
 - Allow SPANK hooks spank_init, spank_init_post_opt and spank_user_init
   to modify the environment of corresponding user processes
 - Rework and improve jail scripts
 - Introduce PSIDnodes_lookupHostname() to internally resolve hostnames;
   this utilizes <Psid.NetworkName>.Hostname in psconfig's node objects
 - Add support for S_TASK_ARGV (might be in future Slurm vers) to psslurmspank
Additional changes:
 - Add psslurm option SKIP_CORE_VERIFICATION as preparation for AWS pClusters
 - Add ReFrame testsuite and first PMIx test
 - Make pluginconfig's Config_t opaque


Version 5.1.54-3:
=================
Bugfixes:
 - Ensure downward messages are not handled as upward (jwt:#19720)
Enhancements:
 - Support srun option --threads-per-core in psslurmgetbind
 - Reject mututally exclusive srun options in psslurmgetbind


Version 5.1.54-2:
=================
Bugfixes:
 - slurm.conf might hold absolute or relative path to spank configuration


Version 5.1.54-1:
=================
Bugfixes:
 - Fix possbile segfault when parsing GRes usage in psslurm
 - Update srun options for user triggered spawn in psslurm


Version 5.1.54:
===============
Bugfixes:
 - Fix possible segfault if psslurm verifies an allocation (#96)
 - Fix handling of hint nomultithread in pspmix (#104)
 - Don't destroy steps in Job_delete()
 - Ensure all delayed tasks get removed when allocation is gone
 - Prevent premature exit of psilogger on short jobs
 - Don't load psslurm if jail function handles are not avaiable
 - Ensure all data gets distribute during PMIx fence operation
Enhancements:
 - Add cgroup support to psslurm (might limit various system resources)
 - Major rework of jail plugin and scripts to allow node sharing
 - Introduce automatic PMIx process sets to pspmix when acting as v4 server
 - More scalable, tree protocol based PMIx fence operation in pspmix
 - Introduce psslurm configuration option SLURM_CONFIG_DIR
 - Rename psslurm configuration option SLURM_CONF_DIR to SLURM_CONF_CACHE
 - Warn about missing multi gpu per tasks support in psslurm (mlx#842)
 - Verify sender, dest and user IDs in pspmix communication
 - Add new psslurm configuration option DENIED_USERS
 - Speedup completing phase of steps which failed to start
 - Introduce __PSI_LOGGER_IO_FILE to detour psidlogger's outputs
Additional changes:
 - Show senders TID for failed psaccount update messages
 - Give some indication why psidforwarder's connection to logger gets lost
 - Move jail scripts to separate folders
 - Use pscompress and psexpand in jail scripts
 - Switch psid's Timer facility to POSIX per-process timer
 - Rename Step_clearByJobid() to Step_destroyByJobid()
 - Rename traverseHostList() to traverseCompList()
 - Add functionality to find entry in vector
 - Print pmix version at the end of configure
 - Avoid %exclude statement in spec file for each .la (#100)


Version 5.1.53-1:
=================
Bugfixes:
 - Avoid double/wrong free with PMIx_Cpuset_destruct() in pspmix


Version 5.1.53:
===============
Bugfixes:
 - Fix filter logic bug unvealed by Clang
 - Fix syntax error not complained by gcc-11 or later
 - Don't mix loop variables
 - Fix memory leak in pspmix
 - Use PMIx_Cpuset_destruct() in current version in pspmix
 - Add validity checks for array lengths in pspmix messages
 - Sanitize file descriptor handling in pspmi
 - Call GC for cbInfoPool only when necessary
 - Fix print_array_if macro in gdbinit definitions
Enhancements:
 - Set PMIX_LOCALITY_STRING using hwloc in pspmix
 - Make logger print prefix look nicer and sort compatible
 - Introduce rrcomm plugin and user-space library for Rank Router Communication
 - Introduce mpiexec's --fullpartition (-P) option
 - Use local reservation info for pinning (obsoletes PSP_DD_SPAWNLOC msgs)
 - Support more fence info types used by openpmix
 - Remove unnecessary initializations and reduce noise during pspmix's operation
 - Add minRank/maxRank to PSIDsession's PSresinfo_t to speedup search)
 - Add memory cleanup functionality to PSIDpart module
 - Introduce delayPSPMsg plugin (delays specific message types for debugging)
Additional changes:
 - Introduce PSCio_recvBufB() for blocking receive
 - Introduce tryRecvFragMsg() in psserial
 - Introduce PSIDHOOK_SPAWN_TASK and bump plugin API version to 137
 - Move pspmix server start to new hook HOOK_SPAWN_TASK
 - Add hook PSIDHOOK_FRWRD_SETUP and bump plugin API version to 138
 - Enhance PSCio_setFDblock() to report old setting on return
 - Send local reservation infos to nodes (push PSDaemonProtocolVersion to 415)
 - Use common definitions of crucial environment names
 - Introduce Timer_restart()
 - Introduce PSP_resolveType() and PSDaemonP_resolveType()
 - Remove size limitation from fragmented messages
 - Do not prevent sending 0-length fragmented messages
 - Count number of selectors and make it available via Selector_getNum()
 - Let psidforwarder rely on reported number of Selectors to decide on exit
 - Store size in PSIDmsgbuf_t explicitly
 - Make psserial's fragment types public
 - Refactor PSIDsession to make use of PSitems
 - Rework selector
 - Remove obsolete remnants of psicomm (early sketches of rank-routing idea)
 - Improve lots of documentation comments


Version 5.1.52-5:
=================
Bugfixes:
 - prevent segfault due to late REQUEST_LAUNCH_TASKS message


Version 5.1.52-4:
=================
Bugfixes:
 - Prevent Selector deadlock (don't awaitWrite() on disabled Selector)


Version 5.1.52-3:
=================
Bugfixes:
- Ensure psslurm is fully initialized if config-less mode is combined
  with Slurm healthcheck
- Allow to call psslurm's cleanup() even if not fully initialized


Version 5.1.52-2:
=================
Enhancements:
 - Allow higher pmix release versions in pspmix RPM requirements


Version 5.1.52-1:
=================
Bugfixes:
 - Adjust psslurmgetbind to Slurm's interpretation of -B option (#89)
 - Fix hint nomultithread together with 'exact'


Version 5.1.52:
===============
Bugfixes:
 - Allow directory as destination for srun's --bcast option (pct:#404)
 - Use configured nodename for psmix's namespace procmap (#82)
Enhancements:
 - Add support for Slurm 22.05
 - Add IPMI support to psslurm's energy monitoring
 - psslurmgetbind: Add --exact option (implied by -c for Slurm >= 22.05)
 - Try to autodetect Slurm version in psslurmgetbind if possible
 - Show Slurm protocol version in config request log message
Additional changes:
 - pspmix RPM require pmix version used to build


Version 5.1.51-2:
=================
Bugfixes:
 - pspmix: Create namespace with only info arrays (works around OpenPMIx #2791)
 - psslurm: Do not filter PMIX_MCA_* variables from user env
 - pspmix: Fix memory leak


Version 5.1.51-1:
=================
Bugfixes:
 - Fix round counter for pluginforwarder children
   * This is used by pelogue to determine if to run as root

*************************************************************************
*************************************************************************
**                                                                     **
**  psmgmt 5.1.51-0 renames psconfig parameters (i.e.:                 **
**       RdpStatusTimeout => StatusTimeout                             **
**       RdpStatusDeadLimit => DeadLimit                               **
**       RdpStatusBroadcasts => StatusBroadcasts                       **
**  This must be reflected in the psconfig database. To adapt the      **
**  defaults:psid object accordingly, the script                       **
**  update_defaults_5.1.51.sh deployed in                              **
**  /opt/parastation/share/doc/psmgmt/psconfig                         **
**  must be run with corresponding rights on the system hosting the    **
**  psconfig database. If your configuration contains custom setting   **
**  of either parameter listed above, those have to be adapted, too.   **
**  Starting with version 5.1.51 the ParaStation daemon will refuse    **
**  kick off if one of the now obsolete psconfig parameters is found   **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.1.51:
===============
Bugfixes:
 - Prevent writing behind end of buffer in spawner (#84)
 - Reset FPE exceptions on spawn and unload if ENABLE_FPE_EXCEPTION is enabled
 - Allow to run PMIx jobs as root
 - Prevent possible segfault in psgw at unload when it was uninitialized
 - Various potential bugs unveiled by scan-build
 - Fix format errors unveiled by cppcheck
 - Ensure array size for gethostname() is sufficient
 - Ensure debug loops have no bad side effects
Enhancements:
 - Let MAP_LDOM pinning use physical domain numbers
 - Support topology.conf and set SLURM_TOPOLOGY_* in rank environments
 - Extend psslurm's accounting towards energy, interconnect and I/O
   * Set their poll intervals from slurm.conf
   * Add psaccount configuration option MONITOR_SCRIPT_PATH
   * Introduce psaccount debug masks PSACC_LOG_FILESYS and PSACC_LOG_INTERCON
 - Unload psslurm if Slurm configuration parsing in configless mode fails
 - Enhance PMIx 4 support
 - Add PMIx singleton support to pspmix
 - Add default session ID explicitly (might be required by openpmix 4.1.2rc1)
Additional changes:
 - Rename (misleading) psconfig parameters to match psiadmin
 - Make use of PSC_concat() less error-prone (remove need for trailing 0L)
 - Make flog() and and fdbg() accessible to other plugins
 - Introduce FW_CHILD_INFINITE to restart plugin forwarder's child endlessly
 - Also call hookFWInitUser if user is root
 - Add const qualifier to traverseHostList()
 - Add some test programs for PMI


Version 5.1.50-5:
=================
Bugfixes:
 - Do not pass node info array in pspmix with PMIx < 4 (jwt:#16581)
 - Fix handling of PSPMIX_CLIENT_INIT/FINALIZE[_RES] message type
Enhancements:
 - Integrate pspmix' info arrays output into debug logging


Version 5.1.50-4:
=================
Bugfixes:
 - Rework request handling for Slurm messages messed up in 5.1.50-3


Version 5.1.50-3:
=================
Bugfixes:
 - Ensure ptid is interpreted correctly if node goes down (jwt:#16170)
 - Fix re-sendind requests originally sent via sendSlurmctldReq() (jwt:#16170)


Version 5.1.50-2:
=================
Bugfixes:
 - Ensure reqKeys.strings gets initialized (jwt:#16557)


Version 5.1.50-1:
=================
Bugfixes:
 - Enforce PSP_DD_DAEMONCONNECT to be the first message delivered via RDP
Enhancements:
 - Introduce PSSLURM_FAKE_UPTIME to help supporting Slurms power saving feature
 - Add example scripts for Slurm power saving feature
Additional changes:
 - Introduce RDP_getState() and RDP_getNumPend()


Version 5.1.50:
===============
Bugfixes:
 - Reorganize psslurm's con->info handling (#77)
 - Let MASK_LDOM pinning use physical domain numbers (#72)
 - Release delayed spawn requests in the step callback (meluxina:#219)
 - Ensure an allocation is defined for spawn request (meluxina:#219).
 - Do not crash psid when SLURM_OVERCOMMIT is not set
 - Ensure psslurm's help will not crash psid if no key is given
 - Avoid use-after-free for config.psiddomain
 - Avoid memory leak upon connection reset in psslurm
 - Fix memory leak in pspmix' fence data handling
 - Substantial hardening of pspmix
 - Retry sending Slurm message even if nothing is sent yet
 - Verify an allocation if no job/step is started
 - Ensure openSlurmctldConEx() returns an error
Enhancements:
 - Add basic support for PMIx 4.0 and bump pspmix plugin version to 2
 - Continue pinning on full nodes when overcommit is set in psslurm
 - Use psid domain as cluster ID in pspmix
 - Introduce PSC_getwd()
 - Introduce addStringArrayToMsg()
Additional changes:
 - Adapt strv_t in analogy to env_t and add compatibility to getStringArrayM()
 - Replace add/getEnviron() by addStringArrayToMsg() and getStringArrayM()
 - Remove obsolete info parameter from sendSlurmMsg()


Version 5.1.49-5:
=================
Bugfixes:
 - Fix user ID of response message for a BCast RPC (pct:#405)


Version 5.1.49-4:
=================
Bugfixes:
 - Don't delete the allocation before all terminate messages are send


Version 5.1.49-3:
=================
Bugfixes:
 - Delete allocation if the slurmctld prologue fails (jwt:#14755)
 - Also delete lingering allocation if result cannot be sent to pspelogue
Enhancements:
 - Add psslurm kvs set commands DEL_ALLOC, DEL_JOB, DEL_STEP
 - Add hook PSIDHOOK_PELOGUE_DROP and bump plugin API version to 136


Version 5.1.49-2:
=================
Bugfixes:
 - Fix possible memory leak if slurmctld misses to send a reply to request
 - Prevent segfault when cleaning up clients in PMIx server
 - Fix memory corruption if number of requested GRes is too large
 - Fix memory leaks unveiled by valgrind
 - Fix warnings unveiled by cppcheck
Enhancements:
 - Add the ability to query psslurm's active Slurm connections via psiadmin
 - Allow PMI client to connect multiple times (via TCP)
 - Overwrite default PMI connection method by PMI_ENABLE_TCP environment
Additional changes:
 - Raise preference of CONNECTED above RELEASED for PMI(x) status


Version 5.1.49-1:
=================
Messed up the tags, thus, need a new version

Version 5.1.49:
===============
Bugfixes:
 - Avoid reading beyond end of buffer
 - Ensure SIGPIPE is not ignored if started via systemd
 - Prevent PSsignal_get() from interupt by RDP timeouts, too
Enhancements:
 - Major rework of pspmix to a concept of one server per user
 - Add function to deregister client from PMIx server library
 - Unload psslurm if Slurm's healthcheck fails
 - Introduce PStask_destroy() and rework PSIDtask_clearMem() to use it
 - Plugin's help directive now takes an argument
 - Rework update caching in providerloop.c
Additional changes:
 - Store reservations in jobs as sets grouped by spawner
 - Fix description of DEBUG_MASK in pspmix.conf
 - Fix various gcc-12 warnings
 - Split psidsession.[ch] from psidspawn.[ch]
 - Introduce PSC_getVersionStr() and make use of it at various places
 - Make PSID_findJobInSession() public
 - Rename (original) PSjob_t to PSsession_t and PSresset_t to PSjob_t
 - Rename PSI_sendSpawnReq() to sendSpawnReq() and make it private
 - Consolidate env.[ch] into psenv.[cv]
 - Centralize fixList() into list_fix() in list.h
 - Add print_array_if macro to gdbinit definitions


Version 5.1.48-6:
=================
Bugfixes:
 - Attach Slurm message hash to munge credential (psc:#402)
Enhancements:
 - Allow to encode payload in psMungeEncodeRes()
 - Improve debug log (unify resID print, minor fixes)

*************************************************************************
*************************************************************************
**                                                                     **
**  psmgmt 5.1.48-5 adapts psslurm to critical bugfixes in Slurm       **
**  (CVE-2022-29500, CVE-2022-29501, CVE-2022-29502). Therefore        **
**  psslurm in this and further versions is only compatible with       **
**  Slurm versions 20.11.9 and 21.08.8 and beyond                      **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.1.48-5:
=================
Bugfixes:
 - Allow only specific users to decode munge messages send by psslurm
 - Cleanup possibly sensitive information when spawning step- or job-forwarder


Version 5.1.48-4:
=================
Bugfixes:
 - Set correct exit status for steps hitting the walltime limit
 - Ensure symbol is found after after second %, too
Enhancements:
 - Add `--exact` option to srun when spawning additional processes


Version 5.1.48-3:
=================
Bugfixes:
 - Prevent double sending of signals in psslurm


Version 5.1.48-2:
=================
Bugfixes:
 - Fix slurmcltd (v21.08) warning on empty gres links statement (psc:#400)
 - Consider all spawn info objects during MPI_Comm_spawn_multiple (#69)
Enhancements:
 - Add support for links parameter of gres.conf


Version 5.1.48-1:
=================
Bugfixes:
 - Ensure to receive all pending ACKs from srun (#2977)
 - Force spawning srun to use configuration cache in Slurm configless-mode
 - Ensure to close all connections via Step_destroy() on step termination
 - Allow NodeName entry of slurm.conf without additional options
Enhancements:
 - Let (pmi[x])spawn fail early if ressources are insufficient
 - Add psslurm configuration option SRUN_BINARY used for spawning processes


Version 5.1.48:
===============
Bugfixes:
 - Fix rank label output for pack jobs (jrt:13143)
 - Fix ldom mask|map pinning with uncommon CPU maps in psslurm
 - Register to slurmctld only after health-check at startup phase is finished
 - Only release forwarder when all children are gone
 - Don't use cpu map with map_cpu and mask_cpu in psslurmgetbind
Enhancements:
 - Prevent sending superfluous CHILDDEAD messages
 - Combine consecutive calls to sendCHILDRESREL() into one message
 - Avoid double bookkeeping of parent/child signals
 - Use $PWD over CWD provided in step if both resolve to same directory
 - psslurmgetbind: Include --cpumap|-M in --help output
 - Re-work messaging in between psid and pluginforwarder avoiding PSLog
Additional changes:
 - Push daemon-protocol version to 414
 - Get rid of now obsolete psidtimer.h
 - Cosmetic fix of memory mask length in output of psslurmgetbind


Version 5.1.47-2:
=================
Bugfixes:
 - Fix core credential mapping to nodes (#61)
 - Fix forwarding of (ps)PMIx server TID to psid forwarders


Version 5.1.47-1:
=================
Bugfixes:
 - Call PSIDHOOK_JAIL_CHILD after hookFWInit() to provide all information
 - Test for valid child PID in jail scripts
 - Don't expect a terminate script for every jail module
 - Pass forwarder's PID to hook PSIDHOOK_JAIL_CHILD instead of UID
Enhancements:
 - Call sinfo with SLURM_CONF_SERVER set for Slurm protocol version detection
 - Switch off debug output in jail scripts as default
 - Introduce hookFWInitUser() to pluginforwarder

*************************************************************************
*************************************************************************
**                                                                     **
**  psmgmt 5.1.47 introduces NIC binding. If psid is configured via    **
**  psconfig, this has to be reflected in psid's default object in the **
**  database. Therefore, on the node hosting psconfig's master db run: **
**                                                                     **
**      psconfig -- set 'defaults:psid' 'BindNics' 'yes'               **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.1.47:
===============
Bugfixes:
 - Fix possible segfault when a step is deleted (jwt:#12795)
 - Only use errno value if kill() actually failed
 - Check for uid_t consistency in msg_SIGNAL
 - Ensure PMIx-server shutdown and add kill timer (steered via SERVER_KILL_WAIT)
Enhancements:
 - Add NIC pinning with correct mapping to device names
 - Speedup psid's shutdown/reset process
 - pspmix: Add option to terminate job on server fail
 - pspmix: Drop root priviliges in pspmix server (shall help with #43)
 - psslurm: BCast forwarder runs as user
 - pspmix: Detect PMIx server usage within psid
 - Allow to run plugin forwarder as user in a more natural way
 - NIC device name fetching via hwloc
 - Change nodeinfo's NICSort default from BIOS to PCI (only for new installs)
 - Introduce PSCio_recvBufS(): suppress warning on closed psslurm PTY connection
 - Make psid depending on munge when using systemd
Additional changes:
 - Add SLURM_CONF_SERVER and SLURM_CONF_BACKUP_SERVER to example psslurm.conf
 - Move psmunge plugin into psslurm sub-RPM
 - Determine systemd's and sysctl's config directory via pkg-config
 - Introduce PSIDclient_getNum()
 - Introduce PSCfgHelp_getObject() into libputil
 - Several improvements on cppcheck and scanbuild warning
 - Tons of IWYU cleanup


Version 5.1.46-4:
=================
Enhancements:
 - Execute Slurm health-check when psslurm is loaded
   * slurm.conf option "HealthCheckInterval" does *not* influence the
     health-check execution on startup
   * psslurm.conf option SLURM_HC_STARTUP may prevent execution on startup


Version 5.1.46-3:
=================
Bugfixes:
 - Extract Slurm Protocol Version by calling sinfo
 - Ensure initSerial() and finalizeSerial() are balanced


Version 5.1.46-2:
=================
Bugfixes:
 - do not exec client process with blocked SIGCHLD


Version 5.1.46-1:
=================
Bugfixes:
 - Avoid NULL pointer de-reference (jwc:#12570)


Version 5.1.46:
===============
Bugfixes:
 - Ensure psidclient will not crash after PSIDclient_clearMem()
 - Fix various memory leaks
 - Don't request configuration if config-server was not set
 - Fix packing of flags for job info request
 - Add sanity checks to execBatchJob()
 - psslumgetbind supports CPU mapping and fix up
 - Fix several scanbuild warnings
Enhancements:
 - Add support for Slurm protocol 21.08
 - Add support for Slurm health-check (#31)
 - Add support for REQUEST_REBOOT_NODES (#36)
 - Make RPC RESPONSE_CONFIG compatible with Slurm 21.08
 - Allow to use Prolog in addition to PrologSlurmctld in slurm.conf
 - Remove the need for [prologue|epilogue].parallel
 - Set SLURM_CONF to config-cache in configless mode (for external programs)
 - Unset SLURM_CONF for jobs and steps in configless mode
 - Replace getpwuid() with PSC_getpwuidBuf() in psidspawn
 - Add timeout monitor (60 sec/SLURM_HC_TIMEOUT) to Slurm health-check
 - Let psslurm parse SlurmdParameters of slurm.conf
 - Add RPCs REQUEST_JOB_REQUEUE and REQUEST_KILL_JOB,
   REQUEST_LAUNCH_PROLOG and REQUEST_COMPLETE_PROLOG
 - Use sendSlurmReq() for RPCs REQUEST_UPDATE_NODE and MESSAGE_EPILOG_COMPLETE
 - Set SHOW_LOCAL flag for Req_Job_Info_Single
 - Synchronize slurmd prolog and batch job startup
 - Cleanup memory in psslurm job/step forwarder
 - Improve PSID_exec[script|func]): clear mem, jail forked procs, kill procgroup
 - pscommon functions to query user database in plugins prevents psid bloating
 - Avoid detour when deleting a job or step
Additional changes:
 - Include SPANK error codes for Slurm protocol 21.08
 - Rework the Slurm message unpack code
 - Rework slurmctld request/response message handling
 - Introduce destroyBCastByJobid() to kill running BCast forwarders
 - Introduce Job_destroy() to kill all remaing processes of a job
 - Introduce Step_destroy() to kill all remaing processes of a step
 - Some refactoring in psslurm and function naming consolidation
 - IWYU fixes


Version 5.1.45-4:
=================
Bugfixes:
 - Set users supplementary groups in psidforwarder (j3:#1382)


Version 5.1.45-3:
=================
Bugfixes:
 - Ensure to use the correct nodelist for shutting down step forwarders


Version 5.1.45-2:
=================
Bugfixes:
 - Don't wait for finalize message for tasks that could not be spawned (#45)
 - Ensure heterogeneous steps will shutdown correctly if executable is missing
 - Fix possbile segmentation fault if psslurm connections get re-used
 - Ensure correct termination of error message
Enhancements:
 - Adapt visible copyright messages to ParTec AG
 - Fix warnings emitted by clang13


Version 5.1.45-1:
=================
Bugfixes:
 - Ensure slurm.conf's hash is defined and correct (#34)
 - Ensure psidforwarder isn't caught in releaseLogger
 - Fix memory leak
Enhancements:
 - Expose MPI_* envvars only when PMI or PMIx is to be used
 - Make slurm.conf hash and update time accessible via psiadmin


Version 5.1.45:
===============
Bugfixes:
 - Ensure PATH resolution ignores non-executable files (jwt:#11381)
 - Check for valid protocol version before tweaking fragment (jwt:#11068)
 - Prevent inherited __PSI_NO_MEMBIND interfering with memory pinning (#20)
 - Ensure psserial's byteorder is never messed up
 - Ensure SIGTERM is not unblocked in main daemon
 - Ensure forwarder ignores ghost messages
 - psgw must unregister all message handlers on unload
 - psexec shall not clear psslurm's dropper
 - Fix buildsystem when slurm-devel is not installed
 - Fix various bugs unveiled by scan-build
Enhancements:
 - Trace PMIx initialization state to improve client release (#24)
 - Catch down nodes early in psslurm's nodeinfo handling
 - Update psroute.py to python3
 - Hide psslurm's output of node_iter_next behind PSSLURM_LOG_PART
Additional changes:
 - Rework psid's message handling
   * Multiple handler might be registered for a given type
   * Handlers called in reverse order of registration (latest registered first)
   * Steer further handling by their return value (true => fully handled)
   * PSID_clearMsg() now unregisters a specific handler
 - Refactor psidforwarder to utilize PSID_handleMsg() and
   bump plugin API version to 133
 - Cleanup psslurm's nodeinfo interfaces
 - Rework PSIDHOOK_FRWRD_CLNT_RLS to support for multiple plugins and
   bump plugin API version to 134
 - Rework PSIDHOOK_FRWRD_EXIT's interface
 - Obsolete PSIDHOOK_FRWRD_KVS, PSIDHOOK_FRWRD_SPAWNRES, PSIDHOOK_FRWRD_CC_ERROR
 - Introduce PSCio_recvMsgT(), i.e. receive message with timeout
 - Introduce PSIDcomm_registerSendMsgFunc()
 - Introduce recvDaemonMsg() to replace recvMsg()
 - Add list_item macro to gdbinit definitions


Version 5.1.44-2:
=================
Bugfixes:
 - Let visspank start without additional parameters


Version 5.1.44-1:
=================
Bugfixes:
 - Various fixes on input forwarding in psidforwarder
 - Fix various warning emitted by rpmbuild
Enhancements:
 - Use mallinfo2() if available (#19)


Version 5.1.44:
===============
Bugfixes:
 - Use correct pack size for interactive steps
 - step follower need to send step complete messages for pack jobs
 - Prevent possible segfault when a pack job is aborted at startup
 - Ensure nodes with different Slurm protocols can use tree forwarding
 - Prevent segfault when psslurm gets unloaded and protocol < 20.11
Enhancements:
 - pspam: add option auth_groups to pam module
 - Optimize partition creation in psslurm
Additional changes:
 - psslurm: Rename some variables to better reflect their meaning


Version 5.1.43:
===============
Bugfixes:
 - Let psslurm report the real memory of the local node in Megabytes
 - Fix partition creation for job packs in psslurm
 - Allow PAM SSH connections when cpuacc module is loaded
 - Make RPC REQUEST_JOB_NOTIFY compatible with Slurm 20.11
 - Make psgw option --gw_debug work with --gw_psgwd_per_node
 - Don't send signal twice on scancel
 - Do not prevent signal delivery in hetjobs
Enhancements:
 - Support interactive steps (#16)
 - Add support for --gpu-bind=map_gpu in psslurm
 - Add support for RPC REQUEST_RECONFIGURE_WITH_CONFIG and REQUEST_RECONFIGURE
 - Support hetjobs in pspmix
   * For this, distribute reservations to all nodes in partition
 - Rework map string parsing and support multiplying '*' in psslurm
 - Rename CPU env variables and leave in user env
   * Rename __PSSLURM_STEP_CORE_BITMAP to PSSLURM_STEP_CPUS
   * Rename __PSSLURM_JOB_CORE_BITMAP to PSSLURM_JOB_CPUS
 - Pass psid's log destination to plugins
Additional changes:
 - Add STEP_CPUS to main jail script
 - Prevent jail plugin from spamming the log
 - psslurm now manages job infos in list but array
 - Utilize different PMIX-macros within the code
 - Add NVIDIA Tesla V100 SXM2 32GB to nodeinfo config


Version 5.1.42-1:
=================
Bugfix:
 - Fix possible segmentation fault in x11spank
 - Change psgw configuration option GATEWAY_ENV to change compute
   process' environment instead of psgwd


Version 5.1.42:
===============
Bugfixes:
 - Fix bug in jail script to set the oom score
 - Fix various memory leaks
Enhancements:
 - Add new psgw configuration option GATEWAY_ENV to set environment for psgwd
 - psslurm checks if PrologSlurmctld is set in slurm.conf
 - Improved syslog messages
 - Replace getdtablesize() by sysconf(_SC_OPEN_MAX) in psmom, too
Additional changes:
 - Merge fwCMD_printMessage() and fwCMD_printJobMsg() into fwCMD_printMsg()
 - Move doRead() et al. from psserial to PSCio_recvBuf()
   * Call PSCio_recvBuf() directly instead via PSID_readall()
 - Introduce PSCio_recvMsg() family of functions
 - Use PSCio_setFDblock() instead of fcntl()


Version 5.1.41-3:
=================
Bugfixes:
 - Ignore if a spank plugin registers spank options only to srun
 - x11spank: fix handling of connect() return code
 - x11spank: use correct display string for xauth


Version 5.1.41-2:
=================
Bugfixes:
 - Ensure to call the correct callback for spank options


Version 5.1.41-1:
=================
Bugfixes:
 - Ensure the environment is setup properly for Spank
 - Forward runtime variables to spank_exit hook
Enhancements:
 - Use PSIDHOOK_EXEC_CLIENT_PREP in psslurm to call Spank hook SPANK_TASK_INIT
 - Make psslurm plugin init never fail without message
Additional changes:
 - Add hook PSIDHOOK_EXEC_CLIENT_PREP and bump plugin API version to 132


Version 5.1.41:
===============
Bugfixes:
 - pspelogue removes "SPANK_" prefix from already prefixed variables (jwt:#9228)
 - Ensure PSP_SMP_NODE_ID is kept (#2911, meluxina:#92)
Enhancements:
 - Add support for spank options (spank_option_register(),
   spank_option_getopt(), and spank_options symbol)
 - Add support for slurm_spank_log()


Version 5.1.40-8:
=================
Bugfixes:
 - Fix setting of slurm node info file flag value (jwt:#9188)
 - Fix potential segfault in GPU pinning code
Enhancements:
 - Ensure close() does not spoil errno in pskill()


Version 5.1.40-7:
=================
Bugfixes:
 -  include msg creation in pspmix send mutexes


Version 5.1.40-6:
=================
Bugfixes:
 - Correct handling of pipe ends
Enhancements:
 - Introduce PSP_SMP_NODE_ID (#2911)
Additional changes:
 - Remove quickhack unused for ages


Version 5.1.40-5:
=================
Enhancements:
 - Allow for multiple PMIx servers on a single node


Version 5.1.40-4:
=================
Enhancements:
 - Forward special slurmctld prologue environment variables to job environment


Version 5.1.40-3:
=================
Bugfixes:
 - Ensure STDIN_FILENO stays open after fork()
Additional changes:
 - Move pse.h and psi.h into psmgmt-devel, too


Version 5.1.40-2:
=================
Bugfixes:
 - Accept option "Features" in the Slurm host configuration
 - Accept empty directories in spank include directive
Additional changes:
 - RPMs reorganized


Version 5.1.40-1:
=================
Bugfixes:
 - Fix tree forwarding with mixed protocol versions and main protocol 20.11


Version 5.1.40:
===============
Bugfixes:
 - Remove interactive steps immediately after completion (pct:389)
 - Bugfixes in psslurm's handling of configless mode
 - Set default GRes count to 1 if unspecified
 - Use psconfig DevIPAddress to generate the routing file
 - Ensure StrBuffers are correctly initialized
Enhancements:
 - Apply filename pattern on job/step current working directory (pct:#387)
 - Use NUMA domain distances for pinning to close GPUs
 - Bind lone task to all GPUs in psslurm pinning
 - Make RPC REQUEST_REATTACH_TASKS compatible with Slurm 20.11
 - Add support for TaskProlog/TaskEpilog option of slurm.conf
 - Enhance jail plugin for more sophisticated cgroups (prepare for shared nodes)
 - Ensure psid and psslurm have the same hardware view
Additional changes:
 - Change psgw --gw_debug filename to JOB-$JOBID-psgwd_$PSGW_ID-$HOSTNAME.debug
 - Add hook PSIDHOOK_JAIL_TERM and bump plugin API version to 129
 - Add JSC's spank plugins
 - Improve psslurm's kvs output on hardware threads
 - Improve error logging of Slurm return codes
 - Improve error messages of psmunge
 - Add new step states to strStepID()


Version 5.1.39-3:
=================
Bugfixes:
 - Don't overwrite LD_LIBRARY_PATH on gateway nodes with an empty value (#2893)
 - Fix unexpected line breaks in output when using srun's label option (#2868)
Enhancements:
 - Do not send SIGWINCH again if already sent by ioctl(TIOCSWINSZ)


Version 5.1.39-2:
=================
Bugfixes:
 - Prevent termination of a job/step when sending a non fatal signal (#2885)
 - Fix resizing of terminal windows via TIOCSWINSZ


Version 5.1.39-1:
=================
Bugfixes:
 - Use srun-address instead of submit host to setup X11 forwarding (pct:#92)


Version 5.1.39:
===============
Bugfixes:
 - psslurm should not wait for a failed psgw prologue that never fully started
 - Test psgwd binary as user and not as root
Enhancements:
 - Add psgw options --gw_debug and --gw_inherit
 - Introduce special handling of obsolete hooks (#2883)
 - Rework global to local rank conversion
 - Emit more documentation on psgw plugin
 - Add spank module for Slurm X11 forwarding
Additional changes:
 - Put currently unused RPC data fields in the corresponding structures
 - Introduce PSCio and move setFDblock() there
 - Replace hand-crafted while() write()/send() loops by PSCio_send*()
 - Extensive code modernization in PSIDclient, PSIDcomm and PSIDRDP


Version 5.1.38-3:
=================
Bugfixes:
 - Do no put reservation still in list (jwt:#8344)


Version 5.1.38-2:
=================
Bugfixes:
 - Re-add support for Slurm 18.08 authentication


Version 5.1.38-1:
=================
Bugfixes:
 - Fill proto-version cache prior to spawn requests
Enhancements:
 - Fix format of various PSI_log() output


Version 5.1.38:
===============
Bugfixes:
 - nodeinfo plugin's message type only contains LSB
 - Fix several potential byte-order conflict
 - Ensure correct errno in doRead()/doWrite() family
Enhancements:
 - Implement distance handling in PSIDhw and nodeinfo plugin
 - Introduce PSIDnodes_distances() and PSIDnodes_distance()
 - Enhance statistics reports for all users of PSitems
 - Move fragment header fetching into fetchFragHeader()
 - Make psiadmin's delay between starting nodes configurable
Additional changes:
 - Create new sub-RPM w/ plugins for test and debug (e.g. fakenode)
 - Rework psidhw's API and introduce PSIDhw_reInit()
 - Refactor PSIDcomm to make use of PSitems
 - Introduce PSPARM_uintSet to handle unsigned parameters in psiadmin
 - Remove obsolete PSIDHOOK_FRWRD_CINFO and PSIDHOOK_FRWRD_DSOCK
 - Add more hwloc topologies recorded on different systems


Version 5.1.37:
===============
Bugfixes:
 - Prevent recursive entering of killSession()
 - Fix invalid calls to pskill() in pluginforwarder
 - Prevent segfault and add warning if Slurm RPC has invalid authentication
 - Fix GRes step credential unpacking
 - Remove false indirection when accessing specific message type
Enhancements:
 - Add support for Slurm 20.11
 - Allow psslurm to start without psslurm.conf
 - Install psslurm.conf and psgw.conf as config file in rpm
Additional changes:
 - Drop Slurm 18.08 support
 - Add documentation to node registration structure
 - Add sources of pspmix mpi plugin for Slurm
 - Add more hwloc XML files for KNL


Version 5.1.36-3:
=================
Bugfixes:
 - Fix building on CentOS 8


Version 5.1.36-2:
=================
Enhancements:
 - Removes psroute's dependency on ClusterShell and make it more
   python2/3 agnostic


Version 5.1.36-1:
=================
Bugfixes:
 - Avoid segfault when using GRes in combination with job packs (jwt:#7902)


*************************************************************************
*************************************************************************
**                                                                     **
**  From psmgmt 5.1.36 on the nodeinfo plugins has to be configured    **
**  via psconfig. Use 'source psconfig/nodeinfo.dump' to update the    **
**  psconfig database and refer to README.nodeinfo for details.        **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.1.36:
===============
Bugfixes:
 - Ensure fwdata is available to forward PMIx msg (jwt:#7750)
 - Ensure 'all' nodes are accepted in psiadmin
 - Lots of minor bugfixes in psid's psconfig parsing
Enhancements:
 - Display nodelist hash in psiadmin's "list version" and "list summary" (#2870)
 - Use POLL_INTERVAL to steers psaccount's polling (#2513)
 - Warn about a slow name resolver (#2872)
 - Make PCIe IDs and sort order configurable via nodeinfo plugin
 - Add NVIDIA V100-SXM2 (JUWELS GPU nodes) to nodeinfo's default
 - Make mixed protocol usage visible in psiadmin
Additional changes:
 - Move pspelogue into main RPM (no direct relation to psslurm plugin)
 - Get rid of psh (more recent version in pscluster RPM)
 - Enable psserial to send extra data with each fragment
 - Implement fully cached PSI_protocolVersion()
 - Cleanup lots of obsolete code and protocol


Version 5.1.35-1:
=================
Bugfixes:
 - Prevent syslog for nodeinfo plugin (#2871)
 - Do not set GPU pinning variables if no GPUs are assigned
 - Fix compilation without spank support
Enhancements:
 - Print only existing hardware threads in PSID_CPU_PINNING
 - Add JUSUF GPU (V100 PCIe 16GB) to PCI ID list
 - Redistribute spank stuff over main, psslurm and spank sub-RPMs
Additional changes:
 - Rename -spank sub-RPM to -slurmutils


Version 5.1.35:
===============
Bugfixes:
 - Delay fw->callback until connection is closed (jwt:#7136)
Enhancements:
 - psslurm: Implement NUMA based GPU pinning
 - Introduce psid's nodeinfo plugin to sync hwloc info between nodes
 - Add PSP_OP_TRIGGER_DIST and PSIDHOOK_DIST_INFO; bump pluginAPIVersion to 129
Additional changes:
 - Move spank plugins into their own sub-RPM
 - Rename __PSI_USE_GPUS to __PSID_USE_GPUS
 - Rename envar PSI_CLOSE_GPUS to PSID_CLOSE_GPUS
 - Add plugin-dependency psslurm->nodeinfo
 - Create new psid module for pinning stuff
 - Add PSCPU_overlap() and PSCPU_disjoint()
 - Rename PSP_CC_PLUG_* message types to PSP_PLUG_*
 - Rename PSIDnodes_[gs]etVirtCPUs() to PSIDnodes_[gs]etNumThrds()
 - Rename PSIDnodes_[gs]etPhysCPUs() to PSIDnodes_[gs]etNumCores()
 - Add various hwloc topologies recorded on different systems


Version 5.1.34-2:
=================
Bugfixes:
 - Accept node registration RPC from any Slurm protocol version (juwels:#7423)


Version 5.1.34-1:
=================
Bugfixes:
 - Quickfix for jwt:#7436


*************************************************************************
*************************************************************************
**                                                                     **
**  Remark: From psmgmt 5.1.34 on support for Slurm 17.11 is dropped   **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.1.34:
===============
Enhancements:
 - Add 20.02 protocol definition and drop 17.11 support
 - Add support for Slurm configless mode
 - SLURM_UPDATE_CONF_AT_STARTUP chooses update of Slurm configuration
   cache at startup
 - Cleanup of Slurm configuration cache via psiadmin's "set CLEAR_CONF_CACHE"
 - Support "Flags" option in gres.conf
Additional changes:
 - Remove special treatment for protocol 18.08
 - Install psmgmt.gdb to /etc/gdbinit.d/
 - Drop psstrings.c since it contained only dead code


Version 5.1.33-2:
=================
Bugfixes:
 - Improve PMIx_Abort handling to avoid double free()


Version 5.1.33-1:
=================
Bugfixes:
 - Abort loop over procList if unused snapshot is found (#2858)
 - Do not unblock SIGTERM in pskill() if it was blocked before (#2856)
 - Ensure pluginforwarder catches the correct SIGCHLD (#2856)
 - Only forward pelogue output in plugins which actually requested it
Enhancements:
 - Fix debugging output in psslurm's handleCreatePart()
 - More proper cleanup in more plugins


Version 5.1.33:
===============
Bugfixes:
 - Remove incorrect check in handleFrwrdMsgReply() (jw:#8622)
 - Hide PSP_CC_ERROR messages from hook handleMthrMsg() (j3:#1359)
 - Fix blocking of SIGUSR introduced in 99d9b252 (j3:#1359)
 - Block signals to prevent asynchronous call via signal handler (#2856)
Enhancements:
 - psslurm: Rename PSSLURM_[JOB|STEP]_GPUS to SLURM_[JOB|STEP]_GPUS
 - psaccount: Ensure signals are delivered only once per round
Additional changes:
 - Eliminate obsolete psAccountSignalChildren


*************************************************************************
*************************************************************************
**                                                                     **
**  Remark: From psmgmt 5.1.32 on Psid.BindGpus has to be defined in   **
**  psconfig or psid will not start                                    **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.1.32:
===============
Bugfixes:
 - Ensure mother superior cancels a pelogue script even without local child
 - psslurmgetbind: Fix segfault by incomplete pininfo
 - Fix distribution *:fcyclic:cyclic issue in (psc#350)
 - psslurm: Set only existing cpus in slots with bind type none
 - Prevent out of bound accesses to PSCPU_set_t (#2855)
Enhancements:
 - Add support for binding processes to local GPUs; might be set
   persistently (psconfig/parastation.conf) or at runtime (psiadmin)
 - Add support for REQUEST_UPDATE_NODE to psslurm
 - Add support for include statement in Slurm's plugstack.conf
 - Option DIRECT_DRAIN lets psslurm drain nodes without psexec's help
 - psslurm: offline a node if name resolution of job owner fails (pct:#374)
 - psslurm: offline a node if the slurmctld prologue fails
 - Enable GPU remapping by envvar PSI_GPUMAP
 - psslurm: report out of bound problems with --cpu-bind=map_cpu:*
 - psslurm: add support for --ntasks-per-socket option
 - Improve psgw error logging
 - Add support for spank API calls slurm_spank_init() and
   slurm_spank_slurmd_exit() in slurmd context
 - Add PSSLURM_JOB_GPUS, PSSLURM_STEP_GPUS, and PSSLURM_CLOSE_STEP_GPUS to env
 - Avoid compile problems with hwloc >= 2.0
Additional changes:
 - Improve pspam's documentation
 - Add hook PSIDHOOK_PELOGUE_GLOBAL and bump pluginAPIVersion to 128
 - Add hook PSIDHOOK_PELOGUE_GLOBAL
 - Introduce PSIDnodes_unmapCPU()
 - Use __AUTO_CUDA_VISIBLE_DEVICES, etc. to protect user-defined env
 - psslurm: optimize building reservations significantly
 - Replace obsolete blockSignal() with PSID_blockSig()


Version 5.1.31-2:
=================
Bugfixes:
 - Ensure epilogue will not be started while the prologue is running (#2788)
Enhancements:
 - Let pspelogue resolve hostnames using psconfig
Additional changes:
 - Update psslurm configuration template


Version 5.1.31-1:
=================
Bugfixes:
 - Extended error handling for ENOENT after sendto() is on CentOS8 required
 - Set correct spank context for spank callbacks
Additional changes:
 - Rename msgbuf_t to PSIDmsgbuf_t avoids clash with RDP
 - Improve naming and documentation of findStepByFwPid() and
   findStepByTaskPid() (#2847)


*************************************************************************
*************************************************************************
**                                                                     **
**  Remark: From psmgmt 5.1.31 on the slurm user on the slurmctld node **
**  needs to be a psid admin user for pspelogue to work                **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.1.31:
===============
Bugfixes:
 - Report node failure in job error output (pct:#298)
 - Report to jobs stderr if job was canceled due to wall-time limit (pct:#219)
 - Allow admin users direct use of mpiexec (#2845)
 - Fix possible segfault in psgw plugin
Enhancements:
 - Add support for RPC REQUEST_SUSPEND_INT in psslurm (pct:#174)
 - Add support for REQUEST_JOB_NOTIFY (pct:#175)
 - New plugin pam_service (pct:#362) starts PAM session for all
   children of psid and psslurm. Activate by loading the plugin
 - Add support to psaccount to read energy data collected using a script
 - Save output of prologue/epilogue scripts on mother superior node
   * to enable for prologue with pspelogue option --fwPrologueOE
   * to enable for epilogue with psslurm option PELOGUE_LOG_OE
 - Let psgw plugin forward stdout/stderr of pelogue scripts
 - Execute epilogue on gateway nodes
 - Verify message sender is a privileged user in all plugins
 - Add support for magic Slurm signal SIG_TERM_KILL
 - Update magic Slurm signals
 - Replace kill() with pskill() and PSID_kill() in the plugins
Additional changes:
 - Rename PStask_cleanup() -> PSIDtask_cleanup() for consistency
 - Push automake to 1.16.2


Version 5.1.30:
===============
Bugfixes:
 - Protocol messed up between psid and psiadmin (#2841)
 - Do not set SLURM_NTASKS and SLURM_NPROCS for jobs (psc:#355)
 - Fix spank plugin argument parsing
 - Do not use putenv() with an automatic variable (prevents export of
   multiple environments in task prolog)
Enhancements:
 - Complete rework of pinning within psslurm (psc:#349)
 - Add psslurmgetbind tool
 - Add support for file pattern "%x" to psslurm (psc:#353)
 - Add epilogue.finalize script
 - Add support for "print" and "unset" in psslurm task prologue
 - Use hwloc to detect HW-threads and cores (#2833)
 - Activate spank hook SPANK_INIT_POST_OPT and warn on spank hooks return error
 - Skip undefined nodes faster in case of split brain situations
Additional changes:
 - Add psslurm epilogue.finalize example script
 - Add debugging output for multiprog step handling
 - Add PSIDHOOK_GETRESERVATION and push API version to 125
 - Fix warnings unveiled by cppcheck, clang, et al.


Version 5.1.29-1:
=================
Bugfixes:
 - Avoid RDP-Timer warnings in psidforwarder (jwt:#4473)
 - Add missing newlines to psidforwarder's logs
Enhancements:
 - Add option DISABLE_SPANK to prevent loading of spank plugins in psslurm
 - Let psslurm execute an epilogue finalize script on mother superior


Version 5.1.29:
===============
Bugfixes:
 - Don't send infinite SPAWNFAILED messages (jwt:#5160)
 - Fix segfault for long spank cmdline options (jrt:#10196)
 - Don't prevent PSID_execFunc() et al. to report failed execution
Enhancements:
 - Improve reporting of node failures to slurmctld
 - Add support for spank plugins to psslurm. Supported hooks are:
   SPANK_INIT, SPANK_JOB_PROLOG, SPANK_LOCAL_USER_INIT, SPANK_USER_INIT,
   SPANK_TASK_INIT_PRIVILEGED, SPANK_TASK_INIT, SPANK_TASK_POST_FORK,
   SPANK_TASK_EXIT, SPANK_JOB_EPILOG, SPANK_EXIT
 - Show spank plugins via psiadmin
 - Reduce memory footprint of various forwarder processes
 - Let the psgw plugin signal the process group when stopping the psgwd
 - psslurm: Print task distribution mask in debugging output
 - Remove obsolete tweaks for automounts
Additional changes:
 - Bump pluginAPIVersion to 124
 - Add aggressive flag to PSID_clearMem()
 - Refactor psslurmio: move forwarder communication to psslurmfwcomm, etc.
 - PSIDHOOK_FRWRD_CLNT_RES reports clients exit status to attached plugins
 - Rename PSIDHOOK_FRWRD_RESCLIENT to PSIDHOOK_FRWRD_EXIT
 - Rename PSIDHOOK_FRWRD_CLIENT_STAT to PSIDHOOK_FRWRD_CLNT_RLS
 - Ensure RPM build work without slurm-devel and/or systemd-devel
 - Remove obsolete build dependency systemd-devel on OpenSUSE
 - Fix various warning unveiled by -Wshadow and cppcheck


Version 5.1.28:
===============
Bugfixes:
 - Fix possible segfault when the TRes database is not configured
 - When running in VM, recvfrom() might return ENOENT
Enhancements:
 - Add OMPI_* variables to client environment in pspmix (psc#326)
 - Limit PIDs to 32768 via sysctl configuration (#2826)
 - Improve Slurm message reply code of psslurm
 - Make slurmctld prologue work with the gcc address sanitizer on
Additional changes:
 - Unifiy prototype of PSID_sendSignal()
 - Initialize various structures detected by valgrind
 - Ensure building RPMs works without pam-devel
 - Fix some warnings unveiled by gcc -Wshadow option
 - Revert "Enhancement: set SLURM_NTASKS_PER_NODE if it was not set by sbatch"
 - Fix a number of pylint warnings.
 - has_key() is deprecated
 - Use more Pythonic way of string formatting
 - Check for None with 'is'
 - Make Python version used explicit


Version 5.1.27-1:
=================
Bugfixes:
 - Fix possible segfault when accounting data is packed (j3t:#1345)


Version 5.1.27:
===============
Bugfixes:
 - Fix possible segfault in psgw startup
 - Make --gw_env option work without specifying export
Enhancements:
 - Add support for slurm.conf SlurmctldHost option (pct:#336)
 - Set SLURM_NTASKS_PER_NODE if not set by sbatch (pct:#339)
 - Add accounting of various min/max task values with 19.05 (pct:#337)
Additional changes:
 - Add two new functions to .gdbinit: reverse_print_list[_entry]
 - Fix various cppcheck warning


Version 5.1.26:
===============
Bugfixes:
 - Prevent psgw plugin from crashing the daemon (j3t:#329)
 - Fix segfault when late srun replies arrive after step is gone (jrt:10050)
 - Ensure PSI_recvMsg() ignores interupted read() (jwt:#2494)
 - Let psslurm delay tasks via PSIDHOOK_RECV_SPAWNREQ (jwt:#2515)
 - Prevent segmentation fault if username resolution failes (jwt:#4234)
 - Ensure step in callback is still valid (jrt:#10122)
 - Consider byte-order when dropping SPAWNREQUEST (jwt:#4282)
 - Make PMI parameters fit into the line (psc:#332)
 - Add missing offset to SLURM_GTIDS for pack jobs (pct:#334)
 - Fix segfault in gres environment parsing
 - Ensure dupSlurmMsg() copies the complete structure
 - Prevent possible segfault in psgw plugin
 - Fix potential memory leaks unveiled by scan-build
 - Ensure to actually exit on PSIlog_exit().
Enhancements:
 - Add option --gw_psgwd_per_node start multiple psgwd on a gateway node
 - Show gres IDs at psslurm startup
 - Only report step timeout message on mother superior
 - Unify gres error reporting
 - Add option --gw_verbose to report psgw startup errors to file
 - Improve handling of psgw error files
 - Don't send message to parent known to be dead
Additional changes:
 - Adopt psroute.py to start multiple psgwd on a gateway node


Version 5.1.25:
===============
Bugfixes:
 - Fix segfault in psslurm when multiple steps are signaled (jwt:#3999)
 - pspmix: Reduce timeout for PSID_HOOK_EXEC_FORWARDER (jwt:#3515)
 - Cleanup PMIx environment variables to fix interactive jobs (jrt:#9831)
 - Make SLURM_CPU_BIND_TYPE reflect the actually used pinning
 - Prevent unnecessary calls to Selector_awaitWrite()
 - Correct invalid pack node-list in rare conditions
Enhancements:
 - Add support of Slurm 18.08 and 19.05 protocols
 - Add support for new Slurm gpu submit options to psslurm
 - Add TRes support
 - Update GRES environment and credential unpacking for 19.05
 - Implement tresBind and tresFreq
 - Implement job energy accounting in psslurm via psaccount
 - Also add energy configuration to psaccount.conf template
 - Add support for RESPONSE_NODE_REGISTRATION
 - Update unpack BCast credential
 - Let pspelogue forward SLURM_SUBMIT_DIR to prologue and epilogue
Additional changes:
 - Make switchUser() available to all plugins via pluginhelper
 - Refactor psslurmio and consolidate step input/output redirection
 - Drop support for 17.02 protocol version


Version 5.1.24:
===============
Bugfixes:
 - Answer terminate requests correctly when waiting for other nodes (jwt:#3712)
 - Actually break recursion when sending CC_MSG to daemon (psc:#320)
 - Do not call shutdownForwarder() for already gone jobserver (psc:#320)
 - Ensure to send forwarder messages to valid TIDs only (psc:#320)
 - Propoer handling of obsolete task structures (jwt:#3709)
Enhancements:
 - Enable psiadmin to count and reset obsoleteTasks
 - pspmix: More debugging output in fence handling
Additional changes:
 - Fix compiler warning emmited by clang-8.0.1
 - Use type bool at more places


Version 5.1.23:
===============
Bugfixes:
 - Ensure SIGALRM is unblocked in pspelogue (jwt:#3664)
 - Fix potential race in pspmix modex data handling
 - Fix modex response handling in pspmix
 - Fix race in fence handling in pspmix
 - Fix possible segmentation fault in Slurm protocol error reporting
Enhancements:
 - Move default location of psgw's routing file to submit's CWD (pct:#307)
 - Add support for SLURM 18.08 protocol
 - First steps to support GRES cores in psslurm
 - Avoid warnings from pspmix by ignoring control messages
 - Improve psslurm I/O message debugging
 - Add missing call output and clarify server not found output in pspmix
 - Better debugging output in fence handling in pspmix
 - Ensure noiseless use of psserial in child processes
Additional changes:
 - Disable printing process map every time in pspmix


Version 5.1.22:
===============
Bugfixes:
 - Each pack job's mother superior must send REQUEST_STEP_COMPLETE (j3t:#1293)
 - Fix handling of invalid namespace in pspmix' sendModexDataRequest()
 - Fix passing of uid and gid to PMIx_server_init()
Enhancements:
 - Add automatic detection of Slurm protocol to psslurm (new  default or set
   "SLURM_PROTO_VERSION = auto" in psslurm's config explicitly)
 - Add psslurm config option SINFO_BINARY to specify absolute path to sinfo
 - Let job start fail in psslurm if user ID cannot be resolved (#2765)
 - Avoid "cannot convert" messages for missing keys in psconfig parser
 - Allow to log interesting environment variables in pspmix
 - Return PMIX_OPERATION_SUCCEEDED in pspmix' server_fencenb_cb()
 - Improved some psslurm log messages
Additional changes:
 - Add function strStepID() to easily print step IDs in psslurm
 - Fix more minor issues unveiled by cppcheck
 - Fix some NULL pointer format errors unveiled by gcc 9.1.0


Version 5.1.21-1:
=================
Bugfixes:
 - Remove dependency of psgw to psslurm


*************************************************************************
*************************************************************************
**                                                                     **
**  Remark: psmgmt 5.1.21 changes the expected content of psconfig     **
**  that might be incompatible with former configuations:              **
**   - the key NodeNo is now called Psid.NodeId (but NodeNo might be   **
**     used if no Psid.NodeId is found)                                **
**   - If Psid.Domain is set in a node's host object, only other nodes **
**     with the very same Psid.Domain will be seen                     **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.1.21:
===============
Bugfixes:
 - Fix segfault in pelogue plugin on race condition
 - Don't remove pspelouge config in pelogue needed by psgw callback
 - Fix various warnings brought by cppcheck and gcc-8.3
Enhancements:
 - Introduce psid domain concept (pt#2631)
 - Use Psid.NodeId in favour of NodeNo in psconfig
 - Add psgw route plugin "plugin02.py"
 - Make pspmix configurable via file and psiadmin
 - Add rudimentary support of psiadmin plugin commands to psgw
 - Improve log output in pelogue
Additional changes:
 - Lift required pmix-Version to 3.1.3
 - Add Uint64Array and Int64Array support to psserial


Version 5.1.20-1:
=================
Bugfixes:
 - Fix handling of SLURM_MPI_TYPE=none (srun --mpi=none)


Version 5.1.20:
===============
Bugfixes:
 - Break infinite loop when sending PSP_CC_MSG to daemon (j3t:#1305)
 - Fix potential memory leak in psenv
Enhancements:
 - Report node failure in step error output (pct:#298)
 - Support to start modular jobs: add psgw and psgw_spank plugins
 - Support openMPI by adding pspmix pluging adding PMIx support
 - Distribute reservation infomation to all involved node. Needed for pspmix
Additional changes:
 - Introduce PSSLURM_PMI_TYPE to choose the PMI version to use
 - Move hook PSIDHOOK_EXEC_CLIENT_USER after child born
 - Set task->fd to daemonfd before PSIDHOOK_EXEC_FORWARDER
 - Introduce PSIDHOOK_RECV_SPAWNREQ
 - Improve psidhook
 - Additional code streamlining


*************************************************************************
*************************************************************************
**                                                                     **
**  Remark: psmgmt 5.1.19 changes psslurms prologue behavior. Prologue **
**  is no longer started by psslurm directly but only via the prologue **
**  plugin mechanism of the slurmctld.                                 **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.1.19:
===============
Bugfixes:
 - Protect PMI_BARRIER_ROUNDS from user modification in psslurm (pct:#251)
 - Use effective user ID to test working directory in psslurm (pct:#293)
 - Fix for hanging job packs (pct:#296)
 - Fix for interactive jobs using gres resources (juwelstrac:#2365)
 - Re-enable coredump capability after seteuid() (#2769)
 - Check if file-descriptor is valid (#2766)
Enhancements:
 - Set local node offline via psexec if fork() fails (pct:#299)
 - Export SLURM_GID, SLURM_JOB_STARTTIME and SLURM_PACK_JOBID to epilogue
 - Improve psslurm plugin interface: 'show resolveIDs' (juwelstrac:#2360)
 - Improve psslurm plugin interface: 'show tasks' and pack information
Additional changes:
 - Extract node information more efficiently in mpiexec
 - Drop support for prologue started via psslurm; use slurmctld prologue instead


Version 5.1.18:
===============
Bugfixes:
 - Fix wrong status reporting of pack leader job (pct:#287)
 - Fix broken requeue logic for batch jobs (juwelstrac:#2045)
 - Prevent infinite recursion in CHILDDEAD handling (#2740)
 - Prevent double execution of offline-script if prologue fails on
   mother superior
 - Don't use daemon functions from within forwarders
 - Fix various minor bugs unveiled by cppcheck
Enhancements:
 - Replace very slow expandHostList() by hostlist traversal (#2747)
   * Warning: psslurm option WEAK_NODEID_CHECK no longer supported!
 - Add option --psid_resolve to pspelogue (#2749)
 - Add support for arbitrary script path to psexec
 - Improve psserial's usablility for multiple users
Additional changes:
 - Fix problems unloading pelogue and psslurm
 - Let pelogue call hook PSIDHOOK_PELOGUE_RES
 - Remove obsolete logic for supporting old slurm versions
 - Cleanup, improved logging and fixing typos


*************************************************************************
*************************************************************************
**                                                                     **
**  Remark: psmgmt 5.1.17 changes the protocol between pspelogue and   **
**  pelogue. This new version is *NOT* compatible with ParaStation     **
**  versions 5.1.16 or older when using the slurmctld prologue with    **
**  pspelogue.                                                         **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.1.17:
===============
Bugfixes:
 - Ensure errno will not get overwritten in different functions (jwt:#1189)
 - Prevent memory corruption in pelogue drop handler for fragmented messages
 - Fix logic error in psslurm unveiled by clang
 - Fix various memory leaks and bugs unvealed by cppcheck
 - Fix segfault in spawner for gdb, valgrind or mpichComp option set
Enhancements:
 - Use SLURM_PACK_JOB_NODELIST for pack jobs in pspelogue (jrt:#8923)
 - Major spawn speedup through:
   * Don't parse unused job core bitmap in psslurm
   * Pass step core bitmap without conversation to pinning code in psslurm
   * Adjust pinning to use unconverted step core bitmap
 - Ensure hanging jobs get released after MAX_TERM_REQUESTS
 - Check mandatory configuration options in pelogue
 - Let psslurm handle arbitrarily RCP responses (e.g. RESPONSE_JOB_INFO)
 - Export environment with prefix _PSSLURM_ENV_ from prologue to user procs
 - Let psexec return output from scripts
 - Fix error msg in psid forwarder
Additional changes:
 - Add PSIDHOOK_PELOGUE_RES and PSIDHOOK_PSSLURM_FINALLOC
 - Optimize addStrBuf(), str2Buf() and strn2Buf() and avoid strncpy()
 - Fix documentation in pluginforwarder.h


Version 5.1.16-1:
=================
Bugfixes:
 - Initialize PS_DataBuffer_t correctly (jwt:#1230)


Version 5.1.16:
===============
Bugfixes:
 - Fix reverse hostname lookup which broke Slurm's tree communication
 - Use array pack index to build a composite partition
Additional changes:
 - More accurate tracking of initialization of PSitems_t


Version 5.1.15-1:
=================
Bugfixes:
 - Workaround for orphan jobs using job packs (pct:#126)

*************************************************************************
*************************************************************************
**                                                                     **
**  Remark: psmgmt 5.1.15 drops support of ParaStation daemon         **
**  protocol prior to version 411. Therefore compatibility with        **
**  versions prior to psmgmt-5.0.47 is dropped by this release.        **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.1.15:
===============
Bugfixes:
 - Fix segfault with sub pack groups (j3t:#1246,#1247)
 - Ensure correct byte-order between pspam and PAM module
 - Fix memory leak in psslurm's pinning code
 - Fix logical errors unveiled by scan-build
 - Fix bookkeeping if timesUsed has to be corrected in releaseThreads()
Enhancements:
 - Use fragmented PSP_CD_SPAWNREQUEST messages to speedup spawning
 - Add "_SLURM_SPANK_OPTION_*" to default filter of pspelogue
 - Enhance psserial for multile message handling
 - Introduce PSitems to abstract chunks of items
 - Improve logging for releaseThreads()
Additional changes:
 - Drop support for daemon protocol prior to 411 (released at 20.2.2015)
 - Refactor psaccountproc, Selector, PSIDrsrvtn, PSIDMsgbuf, PSIDFlwCntrl,
   and PSsignal to make use of PSitems
 - Introduce PSI_sendSpawnMsg()
 - Refactor pspmi plugin to use PSI_sendSpawnMsg()


Version 5.1.14:
===============
Bugfixes:
 - Fix pinning with different TPP for job packs (pct:#272, pct:#273)
 - Fix Slurm option cpus-per-task in combination with job packs (pct:#272)
 - Some minor improvement in error handling starting a job pack
Enhancements:
 - Report errors spawning a jobscript in psslurm to the user (pct:#262)
 - Use hash table and binary search to convert hostnames <=> PS nodeIDs
Additional changes:
 - Remove cast disliked by EL7's gcc 4.8.5 for stdc>89
 - Change the LIBEXECDIR define to PKGLIBEXECDIR, which is more precise


Version 5.1.13:
===============
Bugfixes:
 - Let psslurm drop messages if munge service is not available (#2727)

Enhancements:
 - Finalize support for slurmctld parallel prologue
   * Allocation information is distributed from the slurmctld prologue
     via pelogue to psslurm
   * The prologue and epilogue is now tracked in the allocation for jobs
     and steps
   * Every node is starting a local epilogue when receiving a terminate
     request and sending the result to the allocation leader
   * The allocation leader will track the epilogue results and offline
     unresponsive nodes
   * The psslurm configuration options DISABLE_PROLOGUE and
     DISABLE_EPILOGUE were removed
   * If slurmctld prologue is not used psslurm will automatically start
     the parallel prologue the usual way
 - Add timing information to psslurm tree forwarding
 - Add MEASURE_RPC configuration option to psslurm
 - Add MEASURE_MUNGE configuration option to psslurm
 - Add psMungeMeasure() to psmunge for measuring encode/decode times spend in
   libmunge


Version 5.1.12-1:
=================
Bugfixes:
 - Report all known jobs/steps in node registration status (#jrt:8639)


Version 5.1.12:
===============
Bugfixes:
 - Make sure srun terminates if spawning gets interrupted (juwelstrac:#683)
 - Fix memory corruption in psslurm if forwarded messages get dropped
 - Set correct node ID for unused slurmBackupController
 - Use dynamic memory allocation for psexec environment
 - Evaluate ENFORCE_BATCH_START also in handleCreatePart()

Enhancements:
 - Print user-visible message if execv() fails (juwelstrac:#629)
 - Use host tag to identify the local gres configuration (juwels:#440)
 - Accept fully qualified host names in slurm.conf
 - Use local addresses to identify the host configuration in psslurm.conf
 - Use host lookup table also for resolving node IDs in psslurm
 - PSConfig: Ignore host objects without NodeName set (#2631)
 - More detailed error reporting in PSI_sendMsg()
 - Add missing configuration options to psslurm.conf.template (#2719)

Additional changes:
 - Add PSIDHOOK_PELOGUE_START
 - Make isLocalNode() functionality public (#2711)
 - Remove obsolete ChangeLog-History of psslurm (#2720)
 - Prevent mpiexec et al. from unnecessary resolving (#2710)
 - Get rid of obsolete tests in dospawn()
 - Add psBlackHole debug plugin (#2721)
 - Push build environment to automake 1.16.1


Version 5.1.11:
===============
Bugfixes:
 - Reporting of invalid jobids might lead to orphan jobs (jrt:#3565)
 - Prevent possible segmentation fault when loading psslurm
 - Ensure pslog-protocol will not mess up (#1321)
 - Fix various possible problems unveiled by scan-build

Enhancements:
 - Disable prologue for batch jobs if DISABLE_PROLOGUE option is given
 - psconfig parser: try to cut hostname to short
 - Remove PMI environment variables if MPI support is disabled (psc:#251)
 - Add support for NodeAddr, ControlAddr, and BackupAddr in slurm.conf (#2687)
 - Retire signal() from all places of psmgmt (#516)

Additional changes:
 - Cleanup and improved documentation of Job_t, Step_t and Alloc_t
 - Ensure pspelogue and pelogue plugin have a common understanding of
   the protocol in between
 - Fix warnings unveiled by Clang6


Version 5.1.10:
===============
Bugfixes:
 - Disable kvsprovider if --pmidisable is given (psc:#264)
 - Make sure unknown jobs get released for REQUEST_KILL_TIMELIMIT
 - Avoid double free() if loading psmunge fails
 - Ensure correct use PSC_concat()
 - Fix various potential memory leaks

Enhancements:
 - Add support for SLURM 17.11 protocol including job packs
   * This drops support for SLURM 16.05 protocol
   * Standard protocol is SLURM 17.02
   * The SLURM protocol can be changed using the SLURM_PROTO_VERSION
     option of the psslurm configuration
 - Add pspelogue to allow slurmctld parallel prologues
 - Change psslurm CPU load reporting to one minute average (psc:#265)
 - Introduce jail plugin (#2666)
 - Detect the use of compression in psslurm bcast request
 - Improve handling of fragmented messages

Additional changes:
 - Move various functionality from libplugincommon to libpscommon. This
   includes environment and hostlist handling, serialization, etc.
 - Remove now obsolete add*ToMsgBuf() calls
 - Fix various warning unvieled by gcc-8


Version 5.1.9:
==============
Bugfixes:
 - Adapt psslurm to changed signal handling starting with v17.02.8
   (psc:#22, psc:#133)
 - Handle absence of Frequency statements in /proc/cpuinfo (#2621)

Enhancements:
 - Fix CPU frequency determination for ARM platforms
 - Add implementation for cpuBindType CPU_BIND_LDRANK to psslurm
 - Modified socket and ldom pinning

Additional changes:
 - Move verbose step launch infos to printLaunchTasksInfos()
 - Add test program for local pinning tests to simulate different
   pinning scenarios and see the resulting pinning.


Version 5.1.8:
==============
Bugfixes:
 - Fix segmentation fault in psslurm step callback (jrt:#6980)
 - Fix segmentation fault if epilogue start messages get dropped (jrt:#7129)
 - Ensure timer does not expire in localtime() (#2636)
 - Store and resend important Slurm messages if the slurmctld is
   temporally unavailable (jrt:#3389).
 - Switch to non-blocking read/write for Slurm messages (psc:#255).
 - Ignore flow-control messages withing libpsi (#261)
 - Handle executable names including ')' correctly in psaccount plugin
 - Fix message unregistration in plugins
 - Drop use of SIGUNUSED which was removed from glibc 2.26
 - Fix various bugs unveiled by gcc 7

Enhancements:
 - Add support for SLURM 17.02 in psslurm
 - Modify psslurm plugin's socket and ldom pinning
 - Show active and registered jobs in pelogue plugin's show query
 - Add support for RPC DAEMON_STATUS in psslurm (pct:#134)
 - Add support for RPC REQUEST_SHUTDOWN in psslurm (pct:#178)
 - Add support to change pelogue plugin's debug mask during runtime
 - Introduce PSP_CD_UNKNOWN message type to tell sender about not
   handling some messages
 - Allow to set the task-group via PSI_spawnService()
 - Split mpiexec into three executables: mpiexec, kvsprovider, and spawner
 - Hide connection reset with debug mask PSSLURM_LOG_COMM
 - Improve config parsing for various plugins
 - Improved debug output in psslurm

Additional changes:
 - Push automake to version 1.15.1
 - Drop support for SLURM 14.05
 - Major code refactoring, cleanup and streamlining concentrating on psslurm


Version 5.1.7:
==============
Bugfixes:
 - Prevent illegal nodes in clearRDPMsgs() (jrt:#6829)
 - Reusing PIDs for the psilogger in a single job corrupt psslurms
   task management (jrt:#6713)
 - Cleanup steps properly if master superior node disappears

Enhancements:
 - More efficient extension of buffers

Additional changes:
 - Unify PS ID range check in PSC_validNode()


Version 5.1.6:
==============
Bugfixes:
 - Prevent stuck psid if MPI_Comm_spawn requests wrong HW-type
 - Rectify pointer issues in pspmi
 - Fix various potential memory errors found by address sanitizer
 - Allow psidforwarder to create coredumps again

Enhancements:
 - Switch to non-blocking read/write for Slurm messages (psc:#255)
 - Store and resend SLURM messages to slurmctld (jrt:#3389).
 - Allow to pass extra info to per-node environment creator
 - First implementation of SLURM's 17.02 protocol in psslurm

Additional changes:
 - Eliminate obsolete dependency on ps_search_libs.m4
 - Get rid of SVN revision strings
 - Remove support of priviledged options from mpiexec
 - Hide connection reset with debug mask PSSLURM_LOG_COMM
 - Update regression test suite


Version 5.1.5:
==============
Bugfixes:
 - Ensure jobs get released if mother superior is unknown (jrt:#6645)
 - Catch rare race-condition (jrt:#6680)
 - Ensure srun will get an exit notification for all tasks (#6421)
 - Don't try to pack empty data if slurm message forwarding fails
 - Release resources assigned to failed spawns
 - Correct release of overcommited resources
 - Prevent inheritance of forwarder processes (#2629)
 - Check that task is still present (#2629)

Enhancements:
 - More scalable passing of children to grandparents upon parent's release

Additional changes:
 - Add new plugin delegateStat to peek into resource delegates for debugging


Version 5.1.4:
==============
Bugfixes:
 - Counting in CUDA_VISIBLE_DEVICES starts with zero


Version 5.1.3:
==============
Bugfixes:
 - Reworked shutdown process of psslurm and pelogue (jrt:#6401)
 - Fix bug introduced in 89783b4dda7f88b9f5 (jrt:#6536)
 - Fix segfault on psslurm unloading without a gres configuration

Enhancements:
 - Allow psid to manage its own cgroup hierarchy, without systemd interfering
 - Improved error output for invalid slurm protocol version
 - Suppress warning messages for admin users starting jobs via mpiexec (#2617)

Additional changes:
 - Change defaults for CGROUP_{ROOT,NAME} if systemd is used (#2566)


Version 5.1.2:
==============
Bugfixes:
 - Fix segfault in broken connection handling (jrt:#6428)
 - Set psid's loginuid to fix accounting (jrt:#6429)
 - Bitstrings are packed in hex format since Slurm 16.05 (jrt:#6426)
 - Fix memory corruption in psslurm gres processing

Enhancements:
 - Add psslurm-option 'ENABLE_FPE_EXCEPTION' to trap FPE instantly (jrt:#6290)
 - Add support for PMI 'abort' command (jrt:#6427)
 - Give client processes and their forwarders correct loginuid

Additional changes:
 - Make PSID_adjustLoginUID() part of psid's offical interface


Version 5.1.1:
==============
Bugfixes:
 - Grab correct exit code for pelogue scripts (jrt:#6406)
 - Fix pelogue's traverseJobs() (jrt:#6405)
 - Fix segfault in psslurm when I/O connections breaks (jrt:#6396)

Enhancements:
 - Adapt psid version information to missing SVN revs
 - Adapt psiadmin version information to missing SVN revs
 - Add new test tool to list all info types in protocol

Additional changes:
 - pluginfowarder's estatus and ecode validity now flagged
 - Major code cleanup dropping ancient protocol versions
 - Mark Sselect() as deprecated
 - Flatten psmom's directory tree


Version 5.1.0:
==============
Bugfixes:
 - pluginforwarder shall ignore PSP_CC_ERROR messages
 - Fix delayed spawns waiting for step (#6354)
 - Fix memory allocation in pluginstrv
 - Fix memory allocation in PMI spawn handling

Enhancements:
 - New function dupDataBuffer()
 - Remove PSI_INPUTDEST in rank environment instead of mpiexec

Additional changes:
 - Merge most recent version of psslurm plugin from psslurm-devel
 - Allow psmgmt RPM to replace psmgmt-slurm1605
 - Move psslurm plugin into its own sub-RPM
 - Push plugin API version to 117 for PSIDspawn_localTask()
 - Add new testplugin to delay SLURM messages


*************************************************************************
*************************************************************************
**                                                                     **
**  Remark: psmgmt 5.0.54 changes the default behavior of psaccount.   **
**  From now on accountpoll = 0 will be considered again, i.e. no      **
**  polling on used resources will take place if accountpoll is set    **
**  to 0. Since 0 is the default setting this change might break the   **
**  expected behavior of resource accounting. Nevertheless, psmom      **
**  (and also psslurm) will overrule this setting on startup.          **
**  The system's  settings for accountpoll shall be reconsidered,      **
**  anyhow.                                                            **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.0.54:
===============
Bugfixes:
 - Let mpiexec propagate PSI_TPP (#2594)
 - Various critical fixes for psaccount
   - Ensure that a job's first accounting update is always sent
   - Ensure that the last update is sent even though all clients are disabled
   - Don't let the jobscript resources spoil the minCputime value
   - Get rid of an useless limitation when forwarding aggregated data
 - Allow accounting of pluginforwarder's child
 - Prevent pspam from killing random processes
 - Determine UID more reliably in psaccount
 - Help psmomforwarder to escape loop on SIGCHLD
 - Don't set PMI_APPNUM if PMI is disabled (merge from psslurm)
 - Ensure \0 terminated strings at several places
 - Omit leading garbage in printRefList() (visible during shutdown)
 - Ensure PMI actually disables accounting temporarily during initial KVS update
 - Prevent psid from crashing when unloading certain plugins
 - Make psaccount summing up maximum threads correctly

Enhancements:
 - Compute configuration hash value once psslurm is merged (PCT:#254)
 - Let mpiexec provide appnum independently of PMI in PSI_APPNUM
 - Merge psaccount, psexec, pspam, and pelogue plugins from psslurm
   - This includes major refactorings
 - Reorganize all plugins according to #2580
 - Enable pspam to support shared nodes
 - pspam has a KVS interface now
 - New pspam API function psPamFindSessionForPID() as required by psmom
 - Replaces psmom's according functionality by pspam
 - psaccount can ignore procs with loginuid -1 or uid 0, zombies, etc.
   - Ignoring root processes configurable via IGNORE_ROOT_PROCESSES
 - Allow changing psaccount's debug-mask during runtime
 - pelogue has a KVS interface now that also provides some statistics info
 - Introduce new task-group TG_PLUGINFW
 - Replace libreadline by improved version of linenoise

Additional changes:
 - Fix and simplify spec file
 - Add psmgmt.vipsconfig to documentation
 - Merge .gitignore from psslurm-devel
 - Partly refactoring of psmom
 - psmom's pam_psmom.so is replaced by a symlink to pam_pspam.so
 - Allow building RPMs even if pam-devel is missing
 - Add PSIDHOOK_PELOGUE_PREPARE hook
 - Merge major parts of the plugincommon library
   - Incl. pluginforwarder, plugincomm
   - Major refactoring of merged parts allowing for better scalability, etc.
 - Make psaccount using pre-allocated structs to store /proc snapshot
 - Rework psaccount's data aggregation
 - Allow NULL as buf in addDataToMsg()
 - Allow to receive 0-length strings in plugincomm
 - Add PSIDspawn_localTask() to be used by pelogue, pluginforwarder, etc.


Version 5.0.53:
===============
Bugfixes:
 - Merge bugfix from psslurm-devel with some changes (jrt:#6067)

Enhancements:
 - Make spawnTasks public for use within plugins
 - Introduce PSIDHOOK_CLEARMEM
 - Allow cleanup of futile spawn information (jrt:#5792)
 - Replace obsolete gethostbyaddr() by getnameinfo() (#2589)
 - Replace obsoletet gethostbyname() by getaddrinfo() again (#654)
 - Don't let psidclient and psidcomm spoil the global namespace
 - Initialize fragmentation layer explicitly in psmom

Additional changes:
 - More booleanization


Version 5.0.52:
===============
Bugfixes:
 - Ensure all fds are handled if number is increased in config (jrt:#5879)
 - Resolve byte-ordering clash between psmom and pam_psmom (dt:#1567)
 - Fix merge done in r9400
 - Fix minor memory leak
 - Prevent potential segfault in common node down hook handler

Enhancements:
 - Get rid of libreadline dependencies, use linenoise (#1690)
 - Don't let psidutil spoil the global namespace
 - Use a logger for psiadmin's errors

Additional changes:
 - Minor improvements if msg components are not densely packed
 - Merge various changes from psslurm-devel
 - Get rid of some useless snprintf() in pluginmalloc.c
 - Add plugin debug function to log binary data in hex format


*************************************************************************
*************************************************************************
**                                                                     **
**  Remark: psmgmt 5.0.51 dropps support of RDP SYNNACK messages.      **
**  Therefore compatibility with versions prior to psmgmt-5.0.48 is    **
**  dropped by this release. This new version is *NOT* compatible      **
**  with ParaStation versions 5.0.47-3 or older. A setup of the        **
**  current version mixed with ones prior to 5.0.48 will *NOT* work.   **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.0.51:
===============
Bugfixes:
 - epoll requires special handling of stdin in psilogger (#2561)
 - Prevent partition registrations from handled as normal requests (jsc:#181)
 - Ensure that resource delegates never get released outside of psmom (dt:#1495)
 - Ensure psid has correct permissions to call openpty() (#2549)
 - Never exit in hook PSIDHOOK_EXEC_CLIENT_USER
 - Fix memory leak in selector
 - Fix some more file lists for the build-system

Enhancements:
 - New mechanism for task epilogue execution
 - First working version of the cgroup plugin (jrt:#5439)
 - Unload plugin even if cgroup got removed (jrt:#5680)
 - 'plugin show cgroup key status' prints cgroup's status (#2558)
 - Add PSIDHOOK_JAIL_CHILD for cgroup plugin. Push plugin API version to 115
 - Improve usability of mpiexec's --valgrind option
 - Allow const tags in logging facility
 - PSP_strLen() will keep its argument thus allow const arguments
 - Toughen pscpu for different sizes of PSCPU_mask_t

Additional changes:
 - Remove obsolete RDP_SYNNACK message type from RDP (#2426)
 - Merge large parts libplugincommon from the psslurm-devel branch
 - More use of PSP_putMsgBuf()/PSP_getMsgBuf() in psidpartition
 - More use of bool instead of int all over the code
 - Adapt various plugins to libplugincommon changes
 - Streamline psresport plugin


Version 5.0.50-1:
=================
Bugfixes:
 - Fix MPI_Comm_spawn()
 - Fix memory leak in pssh

Version 5.0.50:
===============
Bugfixes:
 - Printing PSCPU_set_t via PSCPU_print() might be broken (psc:#243)
 - Fix mpiexec's -e option for environment propagation
 - Actually adapt psiadmin's "list proc" directive to screen-width
 - Prevent sendReqList() from messing up the stack for nodes >72 cores
 - Create pipe()s with correct euid between forwarder and client
 - Also unblock SIGUSR[12] in PSID_resetSigs()
 - Catch rare case of new parent's node down during process inheritance
 - Ensure all file-descriptors are closed when exec()ing logger
 - Fix logic in error reporting while releasing task
 - Check for failed allocation in putil/env
 - Fix parser-syntax of psiadmin's 'list jobs' directive

Enhancements:
 - Selector now uses epoll() (psc:#231)
 - Remove mpiexec's limitation on number of executables (jrt:#5402)
 - Use signalfd() to handle SIGCHLD, SIGUSR1 and SIGUSR2 in psid
 - Take IP addresses from psconfig avoiding name resolution bursts
 - Clone environment optimization during spawn
 - Count the number of sent NACKs per connection within RDP
 - Catch and forward stray SIGTERM in psidforwarder
 - Delayed finalization of psid's parent process (#2522)
 - Report daemon failure via psid's parent process (#2522)
 - Activate PSID_clearMem() during task-spawn and PSID_execScript() (#1218)
 - free() all memory used by RDP on exit or by request (#1013)
 - Suppress warning on membind if using SLURM (psc:#226)
 - Reorder psid's module initialization
 - Don't wait too long in forwarder befores re-sending FINALIZE message
 - Make PSID_cancelCB() working for file-descriptors beyond FD_SETSIZE
 - Merge KVS related changes from psslurm
 - Add new hooks PSIDHOOK_EXEC_CLIENT_USER, PSIDHOOK_FRWRD_DSOCK
     and PSIDHOOK_PELOGUE_FINISH required by psslurm
 - Let psilogger handle daemon messages via Selector.
 - Re-organize psidforwarder: Use signalfd, improved error messages, etc.
 - Prepare for asynchronouos inheritance of children (??)

Additional changes:
 - Add various systemd example override files (jurecatrac:ticket:4718)
 - Simplify handling of the pam config file
 - Refactoring of psidforwarder, psilogger (booleanize, Swait(), etc.)
 - Use a dispatcher registered in RDP to handle RDP messages.
 - Add PSP_tryPutMsgBuf()
 - Get selector structures in chunks.
 - Make PSpart_encodeReq() compatible to PSP_putMsgBuf() et.al.
 - Suppress one warning coming out of psh
 - Include psconfig docs for "make dist"
 - psprotocol.h now only depends on pstaskid.h instead of pstask.h
 - Mark kvsprovider's terminateJob() to be 'noreturn'
 - Add annotation for iwyu tool.


Version 5.0.49:
===============
Bugfixes:
 - Check Selector's file-descriptors if Sselect() returns with error (#2498)
 - Save errno before doing further calls at various places (jrt:#3316)
 - Guard dynamic memory related calls in msg_SPAWNREQ() (psc:#214)
 - Make psid's behavior under SIGUSR2 bombardment nicer (jrt:#3694)
 - Prevent duplicating messages on ENOBUFS in RDP's sendto()
 - Fix memory corruption if realloc() actually moves data
 - Ensure fd is in valid range within the selector facility

Enhancements:
 - Configurable delay of SIGKILL after signals from relatives
 - Support of a more fault tolerant MPI developed in the context of DEEP-ER
 - Replace getdtablesize() by sysconf(_SC_OPEN_MAX) and call less frequently
 - Improve psiadmin's output of 'list load' for load > 100

Additional changes:
 - Change RPM's name when built without psconfig support (#2502)
 - Adapt testacct to newly available accounting data


Version 5.0.48:
===============
Bugfixes:
 - Suppress the disfigurement discussed in jrt:#394
 - Catching SIGFPE for a normal shutdown of psid is a bad idea (jrt:#424)
 - Ensure myUse is big enough for huge partThrds (jrt:#273)
 - Guard psaccount's periodicMain (j3t:#679)
 - Prevent getPSIEnv() from wrongly matching longer names (psc:#183)
 - Block SIGCHLD at various places (psc:#192)
 - Enforce RDP-connections getting closed on both sides (#1681)
 - Make pssh using the remote settings for a user's shell (#2414)
 - Handling of AdminUsers configuration in psconfig
 - Present error message when dynamic reservation fails
 - Fix resource bookkeeping when tpp is different from 1
 - Fix reading of resource limits with psconfig
 - Fix psconfig configuration template
 - Fix logic in logger's panic output
 - Prevent segfault if libedit (not updating rl_line_buffer) is used

Enhancements:
 - Support environment setting in KEY=VALUE style in psconfig
 - Allow to suspend and resume jobs and resources with reservations
 - Enable to count released resources within a reservation.
 - Provide more information to the RM when releasing dynamical
     resources (PSIDHOOK_RELS_PART_DYNAMIC)
 - Add vipsconfig template for psconfig configuration
 - Implement mpiexec's -ppn option (#845)
 - Close unneeded file-descriptor before exec()ing the actual client process
 - Allow to print smaller CPU-sets via PSCPU_print_part()

Additional changes:
 - Adapt dyn-resource example plugin to new HOOK parameters
 - Expanding RCS-style keywords again
 - Clean up PSP_INFO_LIST_GETNODES (never used due to PSP_INFO_LIST_RESNODES)


Version 5.0.47-3:
=================

Bugfixes:
- Leave handling of resDelegate to psmom/psslurm (#2412)
- Close potential memory leak.

Additional changes:
- Optional build rpms without psconfig support.
- Do not filter out -Wall and -Wextra


Version 5.0.47-2:
=================

Bugfixes:
- Re-enable OMP_NUM_THREADS handling in mpiexec.

Enhancements:
- Implement MPI_Info_set(<info>, "tpp", "<N>")


Version 5.0.47-1:
=================

Bugfixes:
- Before deleting a delegate psmom must remove all references (#2409)
- Fix some more warning notified by clang's scan-build.

Enhancements:
- Add alternative debugging access to psilogger
- Suppress psilogger's output on SIGUSR1 when SLURM is detected


Version 5.0.47:
===============

Bugfixes:
- Init script should be able to restart the psid after a package update
- PMI plugin: Fix receiving of large PMI messages
- Fix problem when sharing a batch-partition between several MPI-jobs (jt:#7554)
- Actually respect the chosen hardware type in mpiexec
- Ensure that HW-types are handled correctly during spawn
- Bind RDP's UDP socket to the configured IP address
- Fix malloc() size error in mpiexec (allocation was too large)
- Fix many memory leaks and warning notified by clang's scan-build
- Close unneeded file to pid_max as reported by D. Krause.
- Fix location of parastation.conf.tmpl on RedHat based distributions
- timer.h needs to include stdio.h for FILE definition

Enhancements:
- New resource handling: partition / reservation / slots
- Improved integration of resources handling with psmom
- Add support for dynamic resource allocation as required in DEEP
  * PSIDHOOK_XTND_PART_DYNAMIC, PSIDHOOK_RELS_PART_DYNAMIC added
  * PSIDpart_extendRes() added
  * Therefore push plugin API to 112
- PSIDpart_register() allows to register resources provided by external RMS
- Upgrade psmom to just register partitions. Shall help with #1187 and #2049
- PSI_DYNAMIC allows to enable dynamic resource allocation
- Add support for -tpp (threads per process) to mpiexec
- Support per spawn tpp and options in mpiexec
- Release resources in partition management (#1988)
- Allow to set psid's resource-limits via psiadmin (#1042)
- Mark kvs-providers in psiadmin accordingly
- Improve warning in PSP_putMsgBuf() and PSP_putTypedMsgBuf()
- Prepare spec-file for /etc/SuSE-release to disappear

Additional changes:
- Remove support to consider resources of ancient (+10 years old) applications
- Add example plugin 'dyn_resources' to test dynamic resource allocation
- PSP_getMsgBuf() / PSP_tryGetMsgBuf() help to fetch data from message buffers
- Mark psmgmt.sh and psmgmt.csh as %config(missingok,noreplace) (psc:#150)
- Rename psmom's ChangeLog-History, README and LICENSE.OpenPBS since they
  end up in their own directory, anyhow.
- Add option to build without psconfig support


*************************************************************************
*************************************************************************
**                                                                     **
**  Remark: psmgmt 5.0.46 enables configuration via psconfig.          **
**    Nevertheless, for compatibility reasons psconfig is ignored as   **
**    long as /etc/parastation.conf exists and has not zero length.    **
**    Thus, updating to psmgmt-5.0.46 requires to either move the      **
**    configuration file saved to /etc/parastation.conf.rpmsave back   **
**    to its  original location /etc/parastation.conf or to update     **
**    the information in the psconfig database accordingly.            **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.0.46:
===============

Bugfixes:
- Return full info on errors and TIDs from PSI_spawn() et al. (#2348)
- Prevent re-enabling stdin after all input was read (#2331)
- Fix relative executable path in MPI_Comm_spawn()
- Fix argument manipulation in psid (#2334)
- Enable --envall even on further steps of spawning (#2332)
- pspmi might segfault on psiadmin's 'plugin set/show/unset' directives
- Ensure to handle sufficient number of service processes (#2230)
- Ensure input forwarding works even if clients join late
- Prevent segfaults by corrupted PSP_DD_GETPARTNL messages (#2288)
- Catch segfault in the rare case of apps connecting to psid without
    sending data (#2252)
- Fix race condition in registerClient() when spawning multiple KVSs
    concurrently: assure to assign unique service ranks
- Ensure all nodes are updated on changed hardware-status
- Fix mpiexec's --merge option for the case of dynamic spawns
- Make logger work even if no stdin is provided and psid is connected
    via fd 0 (#2273)
- Improve robustness of PSI_spawn() et al.
- Several minor fixes highlighted by clang's scan-build.

Enhancements:
- Add support for MPI_Comm_spawn_multiple()
- Start multiple binaries via mpiexec using the colon syntax (#2154)
- Support of 'path' and 'arch' info keys of MPI_Comm_spawn().
- Add support for psconfig
- Improved flow-control which hopefully fixes the problem of psid bloats
- Let kvsprovider kill all its children on SIGTERM (#2201)
- Rely on psid's PSID_checkPrivilege() for some privilege checking
- Make pamdir (installation path of pspam plugin) configurable (#2302)
- Add valgrind support also for spawned processes
- More intelligent way of psmom to obtain jobs from multiple PBS servers
- Switch %postinstall action to 'service inetd try-restart' (#1932)
- Be more relaxed to leave the logger's raw-mode.
- Enforce client messages to have correct sender information
- PSID_cancelCB() allows cancellation of callbacks for scripts or
    functions executed asynchronously (#1503)
- Streamline propagation of resource-limits.
- New function PSCPU_cmp() to compare CPU-sets
- Add Selector_isActive() to check a selector's status from outside
    the selector facility.

Additional changes:
- Remove support of mixed operation with ancient versions of psmgmt
- Remove hard-coded /etc (#2301)
- Remove all non-GNU paths from the build system (#2302)
- Add dependencies on PAM and NUMActl in spec file (#2349)
- Add popt-devel to general BuildRequires (#2284)
- Improvements in the build system
- Adapt the no_flowcontrol plugin to the new flow-control
- Forward APPNUM parameter from mpiexec to pmiclient.
- Add print_list_entry macro to the gdbinit definitions
- Add print_msg_list macro to the gdbinit definitions


Version 5.0.45:
===============

Bugfixes:
- psaccount: update used cpu time periodically (#2147)
- psmom: prevent protocol deadlocks from third party programs (jt:#6971)
- psmom: Don't remove jobinfo for jobs with no walltime limit.
- Fix psiadmin's output for 'list jobs' and 'list plugins'
- Fix accounting again. This was broken by r8746.
- Re-add psaccview.
- psmom: fixed race condition when verfiying the scratch directory (DIR_TEMP).
- Fix many bugs detected by LLVM's scan-build.
- Correct cleanup of processes spawned via PMI_spawn()
- Fix possible segfault in mpiexec's output merging (mpiexec -m)

Enhancement:
- Allow more than 32 cores per node
- Add valgrind support to mpiexec
- Enable PSP_SCHED_YIELD in mpiexec, if overbooking is requested (#151)
- Remove hardcoded paths in pspmi and psmom (#2074)
- Short proc title of psidfw to give more info on the actual process
- psmom: Only root uses the master UNIX socket for local communication
- New psiadmin directive 'list installdir'
- Tell MPI processes to use PMI version 1
- Start KVS service process on starter node
- Move also spawner process of PMI_spawn() when __MPIEXEC_DIST_START is set
- Resolving nodelist's hostnames only required for OpenMPI
- Add lightweight memory debugging for psaccount, pspmi and psmom


Version 5.0.44:
===============

Bugfixes:
- Prevent segfault due to access of outdated task-structure (#2060)
- kvsprovider should report to stderr not syslog (#2040)
- kvsprovider has to exit on itself if pspmi plugin is absent
- Propagate PMI_BARRIER_TMOUT and PMI_BARRIER_ROUNDS to all kvsproviders
- psmom: Flag copy forwarder and its children always valid
- psaccount: Free all config objects when the plugin gets unloaded
- psresport: Unregister hooks and message handler if initialisation fails
- psmom: Unregister hooks and message handler if initialisation fails
- More minor bugs and warnings detected by LLVM's scan-build fixed

Enhancements:
- psmom version 5.2.10-20 merged into psmgmt
- Place spawner/kvsprovider on own nodes when __MPIEXEC_DIST_START is set
- Let mpiexec output the TID of service processes in verbose mode
- psmom: Moving the psid spawner process is now save (#2071)
- psmom: Try every known pbs server on job obit for failover
- Meaningful process title for the kvsprovider
- More self-explanatory CC_ERROR log messages

Additional changes:
- Omit psid's dependency on libpsi and libpse.


Version 5.0.43-1:
=================

Bugfixes:
 - prevent kvs-provider getting stuck in loop

Enhancements:
 - Finally, handle logger's prfx, too. This should help with #1087
 - Use malloc_trim() instead of malloc()/free() on  SIGUSR2


Version 5.0.43:
===============

Bugfixes:
 - Fix segmentation fault when growing client structures for negative
   ranks in the logger
 - Prevent PSP_DD_SENDCONT-storms possibly initiated by forwarded
   PSP_CD_RELEASE messages
 - Prevent false RELEASERES messages when child is inherited several times
 - Fix possible race whithin deleteClient(): SIGCHLD might lead to premature
   invalidation of task structure.
 - Don't release parent before all children are inherited. This should fix
   a possible race unveiled by the new kvs-provider.
 - Pre-allocate some fmt for logger (#1087)
 - Drop PSP_CD_RELEASE correctly to prevent hanging processes
 - Drop PSP_CD_NOTIFYDEAD correctly and don't send PSP_NOTIFYDEADRES twice

Enhancements:
 - Support for MPI_Comm_spawn
 - KVS code moved from logger into service process (kvs-provider)


*************************************************************************
*************************************************************************
**                                                                     **
**  Remark: Starting with psmgmt 5.0.42 the pspmi plugin is mandatory  **
**  for the MPI2 startup. Therefore, it has to be added to the         **
**  parastation configuration.                                         **
**                                                                     **
*************************************************************************
*************************************************************************


*************************************************************************
*************************************************************************
**                                                                     **
**  Remark: psmgmt 5.0.42 pushed pslog to version 3. This new version  **
**  is *NOT* compatible with older parastation versions. Therefore, a  **
**  setup of current version mixed with ones prior to 5.0.42 will      **
**  *NOT* work                                                         **
**                                                                     **
*************************************************************************
*************************************************************************

Version 5.0.42:
===============

Remark: The pspmi plugin is now mandatory for the MPI2 startup and should
be added as default to the parastation configuration.

Remark: Pushed pslog to version 3. This new version is *NOT* compatible
with older parastation versions.

Bugfixes:
 - Various changes to prevent psid from bloating (#1087)
 - print all help messages of mpiexec to stdout (#1926)
 - Avoid directory clash of /var/account (#1930)
 - Fix possible segmentation-fault in psiadmin's auto-completion
 - Fix memory-leak in psiadmin's parser.

Enhancements:
 - Keep a history of last accounted jobs to suppress confusing error messages
 - Make hooks used by pspmi plugin official. Pushed plugin API to 110
 - Let the psaccounter started by psid run as root
 - Ordinary users only get their own accounting records using psaccounter
 - PMI layer moved from the psidforwarder into the pspmi plugin
 - Optimized distribution of PMI values and keys over daisy chain

Additional changes:
 - Let MPI processes fail if no pspmi plugin is loaded.
 - Support for PMI spawn prepared (actual spawning still missing) (#1539)


Version 5.0.41-2:
=================
Additional changes:
 - Enable coredumps of forwarder on SIGPIPE for debugging purposes.


Version 5.0.41-1:
=================
Bugfixes:
 - Prevent call of iterations over message-buffers recursively. This
    should fix new problems reported on JUDGE (JUDGE-TRAC #412)


Version 5.0.41:
===============
Bugfixes:
 - Prevent lists of messages-buffers from corruption due to
   asynchronous operations. (JUDGE #412)
 - Add missing header include for setrlimit().

Enhancements:
 - Speedup (un-)blocking of SIGCHLD by preventing actual syscalls.
 - Speedup (un-)blocking of timers by cashing the latest search-result.


Version 5.0.40-2:
=================
Bugfixes:
 - Prevent from message floods due to flow-control. (#1907)
 - Block any asynchronous action while doing garbage collection on
   message buffers. (#1904 / #1907)
 - Prevent race-condition that might lead to double free() of msgbufs
   (#1904 / #1907)
 - Don't let the init script confuse psidforwarder with the main psid
   process (#1907)

Enhancements:
 - Only do PStask_gcSig() with expensive blocking of signals, if
   required.


Version 5.0.40-1:
=================
Bugfixes:
 - Fix cpuinfo calls on Intel platforms (#1896)


Version 5.0.40:
===============
Bugfixes:
 - Allow to overbook nodes without pinning (#1877).
 - Don't spawn broken tasks, if unable to determine logger's TID (#1844).
 - Fix possible deadlock in psidforwarder's signal handler (#1708)
 - Be more careful when asynchronously modifying daemon's signal-lists.
 - Don't let send stop/cont messages disturb the partition request process.
 - Guard PStask_clone() from rare races on list iterations.
 - Ensure keeping the last chunk of msgbufs.
 - Don't drop message, if buffers are temporarily unavailable in
   the IP stack. Instead try to re-send.
 - Ensure that timeout might expire several times in PSLog_read().
 - Prevent futile daemon message on selector during shutdown
 - Fix one possible deadlock in flow-control between forwarder and
    logger on heavy traffic from both sides. For this, new messages
    types PSP_CD_SENDSTOP and PSP_CD_SENDCONT were introduced.
 - psh: omit newline after prompt when using "-s" (#1797)
 - Ensure to really use the correct abstract UNIX socket as the local
   service port
 - forward client data before cleanup up jobs
 - fixed segfault/list corruption in psaccount plugin (#1711)
 - account jobscripts and children from serial jobs
 - correct accounting of multiple mpiexec calls in a parallel job

Enhancements:
 - Enable binding to memory-nodes as default (#1747).
 - Added support for the MVAPICH PMI process map extension.
 - Added support for MPI_LOCALNRANKS and MPI_LOCALRANKID needed by libpsm
 - Use Selector facility within psidforwarder (#1648)
 - Implement 'parameter' directive in psiadmin to allow modification of
   local parameters within the admin (#633).
 - Allow psiadmin's 'list' directive to show hostnames (#490).
 - Allow psiadmin's 'show' directive to print hex-numbers (#612).
 - Check kernel.pid_max for compatibility with PStask_ID_t (#1767)
 - implemented psaccount plugin's set/unset commands
 - instantly cleanup jobs on psmom request
 - better cleanup of leftover jobs/clients
 - continously forward accounting information to logger node
 - implemented psaccount plugin's show and help calls of plugin API

Additional changes:
 - Use $localstatedir instead of /var.
 - Change defaults of $sysconfigdir and $localstatedir to /etc and /var,
   respectively.
 - Added PSIDHOOK_CREATEPARTNL and pushed the plugin API to 109.
 - Trigger a 1k malloc()/free() cycle via SIGUSR2 to psid.
 - added configuration file for psaccount plugin


Version 5.0.39:
===============
Bugfixes:
 - Make pssh's -e and psmstart's --exports options working.
 - Fix problem in RDP with window-size
 - Ensure message-buffers get free()ed when RDP fails to establish connection

Enhancements:
 - Startup support for OpenMPI executables (version 1.5 and higher) (#580)
 - Add new features required by psmom to solve #1477
 - Prevent forwarder from excessive logging.
 - More robust inline assembly in corePerPhysical()

Additional changes:
 - Move license to QPL


Version 5.0.38:
===============
Bugfixes:
 - syslog() might deadlock. Related to r8548 (#5285)

Enhancements:
 - Add support for GridEngine PE-files. (#932)
 - Improved clash-detection on resources between CLI, environment and
   batch-system
 - Add 'silent' option to psiadmin's shutdown directive and be more verbose
   during shutdown (#1572)
 - Master socket now uses Linux's abstract address. (#1573)
 - Get rid of explicit lock-file.

Additional changes:
 - Move legacy MPI starters into separate sub-package (#1584)
 - Get rid of ELAN-support, since Quadrics QsNet seems to be dead (#1591)


Version 5.0.37:
===============
Bugfixes:
 - Block SIGCHLD while doing syslog() in order to prevent psid from deadlocks.
   syslog() is not re-entrand! (JUDGE-TRAC #141)
 - Ensure, that all TASKDEAD-messages are delivered to new master-psid after
   failover. Additionally, don't tell new master about jobs currently shutting
   down.
 - Actually enable dumping of too-long messages tail. This completes the
   changes done in r8132.
 - Get rid of obsolete timer-entries after fork(). Might lead to strange
   behavior, if timer is pending during fork()
 - Connection to syslog has to be re-opened after cleanup for fork()
 - Selectors shall be removed when corresponding file-descriptor is closed.
 - Remove spurious TAB in output when talking to plugins (#1481).

Enhancements:
 - Prepare psiadmin to print-out hostnames instead of node IDs sometime (#490)

Additional changes:
 - Improve messages from psid's plugin-handling to psiadmin.
 - Include stub into test-plugin for extraordinary long help-messages.


Version 5.0.36:
===============
Bugfixes:
 - Beautify psiadmin's 'plugin help/show/...' output (#1481)
 - (quick fix) psaccount logging via mlog in the psmom forwarder can let
   it exit unexpectedly.
 - If old task-structure found during enqueue delete the right one.

Enhancements:
 - Replace gethostbyname() with getaddrinfo() (#654)
 - If PSI_LOG_SILENT is set, logger will suppress all output.
 - Next generation plugin API (#973)
 - New type of timeout-handlers receiving the timer ID and additional
   information upon elapse (#1361)
 - sendSignal2Session() returns number of children signal was sent to
 - psid shall not block SIGCHLD most of the time. Instead, only critical
   code-regions are fenced by blocked SIGCHLD (#1333)


Version 5.0.35:
===============
Bugfixes:
 - Correct detection of psiadmin and mpiexec started from within scripts
   run via pssh (#1280)
 - psilogger suppresses unnecessary warning on duplicate FINALIZE (#1304)

Enhancements:
 - 'pssh -r' enables pssh to be used as transport for rsync (#504)
 - Cancel PMI-barrier warning after all clients joined (#1172)
 - Let the forwarders set there process title
 - PSIDHOOK_SHUTDOWN realized (#1215)

Additional changes:
 - Add sample .gdbinit with some usefull functions for debugging psid.


Version 5.0.34:
===============
Bugfixes:
 - psiadmin prints out numerical IP address, if 'resolve' cannot get hostname
 - Fix pinning, if node 0 was not yet involved. Before wrong partition
   request was encoded with false protocol-version.

Enhancements:
 - New psmgmt-devel RPM holds additional headers for plugin development (#1100)
 - mpiexec accepts multiple occurrences of -e option (jt:#4515)


Version 5.0.33:
===============
Bugfixes:
 - Fix problem in flow-control disturbing the psaccounter daemon
 - Suppress unnecessary warning, if allocation of partition fails
 - Improved plugin handling
 - Fix psiadmin's 'range' command broken in last version

Enhancements:
 - Propagate PATH and LIBRARY_PATH environment automatically (#1173)
 - Add psaccount plugin providing advanced accounting functionality
 - More intuitive user-handling in configuration-file

Additional changes:
 - Version of the plugin API pushed to 103
 - Adapt for gcc 4.5.1 coming with OpenSUSE 11.4


Version 5.0.32:
===============
Bugfixes:
 - Fix pssh's behavior for interactive sessions
 - Fix race-conditions that might lock RDP's timer forever (#1077)
 - Don't try to read the executable, if there's no permission (#1058)
 - Allow plugin's initialize() method to fail (#1096)
 - Fix some minor memory-leaks (#1009, )

Enhancements:
 - Allow users to influence pinning via __PSI_CPUMAP (#1086)
 - Dynamical dropping of message within psid (#992)
 - psid now has a hook-facility; currently it support PSIDHOOK_NODE_UP
   and PSIDHOOK_NODE_DOWN
 - Implement logger_finalize() in order to allow dynamical loggers (#1010)

Additional changes:
 - Version of the plugin API pushed to 102.


Version 5.0.31:
===============
Bugfixes:
 - Remove limitations on length of arguments when spawning processes (#740).
 - psiadmin now accept range of the form last-first, too (#1014).
 - Include RDP-header in maximum message size for protocols (#459).
 - Fix crippled output of very long lines since 5.0.30.
 - Fix memory leak in partition handling (#794).
 - Catch possible segfaults within psid's logging.
 - Prevent psid from segfaulting, if PSI_TPP erroneously set to 0.
 - Don't crash psid, if client<->daemon protocol is messed up.

Enhancements:
 - Unload plugins before daemon shuts down (#562).
 - RDP statistics (mean-time to ACK, etc.).
 - psh: fix precedence of $RCMD, $AUTOCD and $MAXRCMD.
 - Nicer output of long lines with and without prefixes.
 - Better error-message from mpiexec, if spawning of service-process failed.
 - More detailed messages on late receives in psilogger.
 - Improve testing on failed initialization of loggers.
 - Initialize timer facility explicitly (#799).

Additional changes:
 - Remove extra debug in context of #442.


Version 5.0.30:
===============
Bugfixes:
 - Improve memory-handling when temporarily buffering messages (#689/jt:#2686).
 - Mark psiadmin's hex-output to be in hex (#612).
 - Improved error-messages, if PMI barrier timeout strikes (#827).
 - Improve logging with timestamps (#719).

Enhancements:
 - New psiadmin directive: echo (#632).
 - RLIMIT_NOFILE now configurable & propagated to remote processes
   (#792/jt:#2691).
 - Better handling of hanging file-systems while spawning processes (#879).
 - psid's environment might be changed during runtime (#815).
 - Enhanced plugin-interface within psid (#562). This is a pre-requisit for
   psmom and psaccount plugins.
 - Job's starttime is listed in psiadmin's 'list jobs' directive.

Additional changes:
 - Improved logging on flaws in psid's configuration file.
 - Add /opt/parastation/sbin to root's PATH (#893).


Version 5.0.29:
===============

Bugfixes:
 - psh name merging will not merge (#317).
 - psh's NameCompress create empty result (#318).
 - Performance enhancement in RDP: msgbufs put back into pool
      earlier (jt:#3239).
 - Performance enhancement in flow-control: Don't always wait
      for Sselect to time-out (jt:#3239).

Enhancements:
 - use selectors to handle clients' file-descriptors in psid.
 - allow to trigger Sselect() to start-over. Useful to enable modified
      fd-sets more quickly. Might help with jt:#3239.
 - allow to disable registered selectors temporarily.
 - enable stack protector, if supported (#796).

Additional changes:
 - less important RDP-messages suppressed per default. These might
      lead to bandwidth-problems on huge clusters due to created load.
 - psh now uses psnameexpand and psnamecompress from pscluster/bin/pstool
 - Obsolete NrofNodes removed from default configuration


Version 5.0.28:
===============

Bugfixes:
 - Various improvements in RDP protocol (#386).
 - Fix possible race-condition when resequencing RDP messages (#327).
    this might lead to the behavior described in #327.
 - Improved startup of psid on huge cluster with few active nodes.
 - psaccview: option --norcfile doesn't need a parameter
 - psaccview: don't display empty lines when using option --nospace

Enhancements:
 - Improved logging from RDP's state-machine.
 - psaccview: new option --last_days to show the last x days of accounting data.
 - psaccview: new option --timeout will stop parsing if the timeout is reached.
 - Allow service-tasks to explicitly request signals on children's end (#554).

Additional changes:
 - If core-dumps are enabled, psid does not catch SIGBUS any longer.
 - Extended accounting functionality moved into plugin.