ECN-related tasks are not always predictable. That is, the backups may occur in
the early morning (around 2 or 3am on peta,) but could also occur later
if there is some delay in the backups.
The times to generally watch out for are late evening (8pm or 10pm) and 2-3:30am in
the morning.
What is more, you cannot easily see if a backup is occuring by using "ps"
to look at the processes because they occur from remote machines.
peta is most affected by backups and remote accesses since it is home for a
software RAID array. Every access to a disk causes a CPU load, even when
the accesses are from a remote machine.
condor and PUNCH must have certain daemons running all the time or the process of the Hub will not work properly. You can see these jobs by doing 'ps' on the system (see the FAQ on "How can I tell if the machine that I'm using is heavily loaded?".)
Our systems are connected to a NFS meaning that the disks can be mounted by other systems. When a remote machine accesses the local disks the disk access latency increases. On peta, there is a CPU load for every access to the 32 GB of disk space because it uses a software RAID array.