System Administration Duties
Appointed by the Executive
The Executive must nominate one or more people to look after HUMBUG's computer systems.
The system administration team should submit a brief report to every Executive meeting in the following format:
Backups available (these must have been sighted): yyyy-mm-dd by Person.
Date a backup was tested by a complete restore from backup only: yyyy-mm-dd by Person.
- Disk utilisation:
Filesystem Size Used Avail Use% Mounted on
/dev/xvda 50G 6.9G 44G 14% /
- List of packages that aren't up-to-date security wise: package, ...
- Any major configuration changes since the last exec meeting: ...
- Changes to schedule of people with access to the box: ...
- Other significant changes, if any.
Email: <sasig AT humbug DOT org DOT au>
For all member requests, please login to Trac as user "guest" with password "guest" and log a ticket.
Please note that tickets and email to sasig are archived and publicly accessible.
The Backup System
Backups are implemented using the Rdiff-Image tools. As they aren't part of Debian Stable there are also in the Humbug Debian Repository. Backups are run hourly by cron(8) from /etc/cron.d/rdiff-image. The most recent backup is stored on excalibur itself and can be downloaded using http from the web site. Amazon S3 stores backups going back 6 months.
Read the rdiff-image-cron(8) man page to gain an understanding of how Rdiff-Image structures its backups. Underneath, the backup is just a tar image of the entire VM. That tar image is split into two. The main backup has all sensitive data stripped out, and all passwords replaced with x. Thus it is sanitised but remains bootable. The second backup (called the secret backup) has the original versions of the sanitised files, so that when restored over the main backup a faithful copy of the VM's file system is created. Both backups can be downloaded by anybody, but the secret backup is encrypted with several gpg keys.
Downloading and Booting Backups
We used to store the instructions for downloading and booting from the backups in the wiki. However, that's not much use if it's a live restore required as a result of excalibur disappearing for some reason.
Since we don't want to risk having two sets of instructions getting out of sync, all the instructions are now kept in a Mercurial repository on excalibur and all Sysadmins are expected to obtain the instructions and to follow at least the part under the heading Preliminary Steps - Do This Now in the file README that will be part of the repository that is cloned in the command below:
hg clone ssh://excalibur.humbug.org.au:24//etc/rdiff-image
If that README needs to be modified, make sure you both update the excalibur Mercurial repository and send a notice to the sasig mailing list so that the other Sysadmins will know they need to update their clone of the repository. The basic Mercurial commands needed for the above steps are discussed in the section on DNS below.
Maintenance of the backup system
Finally, the Amazon S3 costs are kept to a minimum by keep the differences between successive backups small. If they are growing more quickly than you expect, trying using rdiff-image-tarutil(1) to find out why. How do to that is explained in its man page. In fact, lots of important things not described here are in the various Rdiff-Image man pages. Read them.
Even if it is obvious, we are not all doing this stuff every day. Any common tasks could go here
Stopping and Starting services
Debian stretch uses systemd as it's init system, but some VPS's don't support it. To get around that use this command to start a service:
/usr/bin/rdiff-image-boot start-service service-name
service-name can by a systemd service (ie, there is a file called service-name.service), or a SysV init script called /etc/init.d/service-name.
zones are stored in mecurial dvcs.
- Get a clone of the repo if you have not already done so:
hg clone ssh://excalibur.humbug.org.au:24//etc/bind/hg/pri
- Update your repo:
hg pull hg update
- Edit the files in your repository
- Update Excalibur:
hg commit -m "A Meaningful Log Message" hg push ssh -p24 excalibur.humbug.org.au cd /etc/bind/hg/pri # or whatever location the repo is hg update sudo ./install.sh
We use postfix.
Edit /etc/postfix/aliases-humbug and make the appropriate change
run sudo postfix reload
Mailman and @humbug.org.au aliases
When subscribing an @humbug.org.au alias and the corresponding destination address you should set the "no email" option on the destination address not the alias.
The reason is that doing it the other way around (disable delivery for the alias, enable for the destination) may disable delivery for both addresses. At least one user (Matthew Franklin) has reported this problem, while at least one user (Raymond Smith) has it working with alias enabled and destination disabled.
To add/remove/update blogs edit /srv/http/planet.humbug.org.au/data/config.ini . The format is straightforward. Planet should pick up the changes on its hourly runs.
The credentials needed to login / control the VM excalibur runs on are attached to this page. Download. It is GPG encrypted to the same people who can decrypt the backup.
The payment gateway lives under /srv/http/payments. It is a mercurial repository. There are further instructions in the README.txt file in that directory.