Wed 26 Sep 2012
Tags: devops, sysadmin
If you're a modern sysadmin you've probably been sipping at the devops
koolaid and trying out one or more of the current system configuration
management tools like puppet or chef.
These tools are awesome - particularly for homogenous large-scale
deployments of identical nodes.
In practice in the enterprise, though, things get more messy. You can
have legacy nodes that can't be puppetised due to their sensitivity and
importance; or nodes that are sufficiently unusual that the payoff of
putting them under configuration management doesn't justify the work;
or just systems which you don't have full control over.
We've been using a simple tool called extract in these kinds of
environments, which pulls a given set of files from remote hosts and
stores them under version control in a set of local per-host trees.
You can think of it as the yang to puppet or chef's yin - instead of
pushing configs onto remote nodes, it's about pulling configs off
nodes, and storing them for tracking and change control.
We've been primarily using it in a RedHat/CentOS environment, so we
use it in conjunction with
rpm-find-changes,
which identifies all the config files under /etc that have been
changed from their deployment versions, or are custom files not
belonging to a package.
Extract doesn't care where its list of files to extract comes from, so
it should be easily customised for other environments.
It uses a simple extract.conf shell-variable-style config file,
like this:
# Where extracted files are to be stored (in per-host trees)
EXTRACT_ROOT=/data/extract
# Hosts from which to extract (space separated)
EXTRACT_HOSTS=host1 host2 host3
# File containing list of files to extract (on the remote host, not locally)
EXTRACT_FILES_REMOTE=/var/cache/rpm-find-changes/etc.txt
Extract also allows arbitrary scripts to be called at the beginning
(setup) and end (teardown) of a run, and before and/or after each host.
Extract ships with some example shell scripts for loading ssh keys, and
checking extracted changes into git or bzr. These hooks are also
configured in the extract.conf config e.g.:
# Pre-process scripts
# PRE_EXTRACT_SETUP - run once only, before any extracts are done
PRE_EXTRACT_SETUP=pre_extract_load_ssh_keys
# PRE_EXTRACT_HOST - run before each host extraction
#PRE_EXTRACT_HOST=pre_extract_noop
# Post process scripts
# POST_EXTRACT_HOST - run after each host extraction
POST_EXTRACT_HOST=post_extract_git
# POST_EXTRACT_TEARDOWN - run once only, after all extracts are completed
#POST_EXTRACT_TEARDOWN=post_extract_touch
Extract is available on github, and
packages for RHEL/CentOS 5 and 6 are available from
my repository.
Feedback/pull requests always welcome.
Tue 08 May 2012
Tags: centos, rhel, yum
Ok, this has bitten me enough times now that I'm going to blog it so I
don't forget it again.
Symptom: you're doing a yum update on a centos5 or rhel5 box, using rpms
from a repository on a centos6 or rhel6 server (or anywhere else with
a more modern createrepo available), and you get errors like this:
http://example.com/repodata/filelists.sqlite.bz2: [Errno -3] Error performing checksum
http://example.com/repodata/primary.sqlite.bz2: [Errno -3] Error performing checksum
What this really means that yum is too stupid to calculate the sha256
checksum correctly (and also too stupid to give you a sensible error
message like "Sorry, primary.sqlite.bz2 is using a sha256 checksum,
but I don't know how to calculate that").
The fix is simple:
yum install python-hashlib
from either rpmforge or epel, which makes the necessary libraries
available for yum to calculate the new checksums correctly. Sorted.
Tue 13 Mar 2012
I've been enjoying playing around with ZeroMQ
lately, and exploring some of the ways it changes the way you approach
system architecture.
One of the revelations for me has been how powerful the pub-sub (Publish-
Subscribe) pattern is. An architecture that makes it straightforward for
multiple consumers to process a given piece of data promotes lots of
small simple consumers, each performing a single task, instead of a
complex monolithic processor.
This is both simpler and more complex, since you end up with more
pieces, but each piece is radically simpler. It's also more flexible and
more scalable, since you can move components around individually, and it
allows greater language and library flexibility, since you can write
individual components in completely different languages.
What's also interesting is that the benefits of this pattern don't
necessarily require an advanced toolkit like ZeroMQ, particularly for
low-volume applications. Here's a sketch of a low-tech pub-sub pattern
that uses files as the pub-sub inflection point, and
incron, the 'inotify cron' daemon, as our
dispatcher.
Recipe:
Install incron, the inotify cron daemon, to monitor our data directory
for changes. On RHEL/CentOS this is available from the rpmforge or EPEL
repositories: yum install incron.
Capture data to files in our data directory in some useful format
e.g. json, yaml, text, whatever.
Setup an incrontab entry for each consumer monitoring CREATE operations
on our data directory e.g.
/data/directory IN_CREATE /path/to/consumer1 $@/$#
/data/directory IN_CREATE /path/to/consumer2 $@/$#
/data/directory IN_CREATE /path/to/consumer3 $@/$#
The $@/$# magic passes the full file path to your consumer - see man 5
incrontab for details and further options.
Done. Working pub-sub with minimal moving parts.
Fri 13 Jan 2012
Tags: aoe, coraid, centos, rhel
Updated 2012-07-24: packages updated to aoe-80 and aoetools-34,
respectively.
I'm a big fan of Coraid and their relatively
low-cost storage units.
I've been using them for 5+ years now, and they've always been pretty
well engineered, reliable, and performant.
They talk ATA-over-Ethernet (AoE),
which is a very simple non-routable protocol for transmitting ATA
commands directly via Ethernet frames, without the overhead of higher
level layers like IP and TCP. So they're a lighter protocol than
something like iSCSI, and so theoretically higher performance.
One issue with them on linux is that the in-kernel 'aoe' driver is
typically pretty old. Coraid's
latest aoe driver is version
78, for instance, while the RHEL6 kernel (2.6.32) comes with aoe v47,
and the RHEL5 kernel (2.6.18) comes with aoe v22. So updating to the
latest version is highly recommended, but also a bit of a pain, because
if you do it manually it has to be recompiled for each new kernel
update.
The modern way to handle this is to use a
kernel-ABI tracking kmod, which gives you
a driver that will work across multiple kernel updates for a given EL
generation, without having to recompile each time.
So I've created a kmod-aoe package that seems to work nicely here. It's
downloadable below, or you can install it from my
yum repository.
The kmod depends on the 'aoetools' package, which supplies the command
line utilities for managing your AoE devices.
kmod-aoe (v80):
aoetools (v34):
There's an init script in the aoetools package that loads the kernel module,
activates any configured LVM volume groups, and mounts any filesystems.
All configuration is done via /etc/sysconfig/aoe.
Fri 28 Oct 2011
Tags: ldap, openldap, rhel, centos, linux, sysadmin
Having spent too much of this week debugging problems around migrating
ldap servers from RHEL5 to RHEL6, here are some miscellaneous notes
to self:
The service is named ldap on RHEL5, and slapd on RHEL6 e.g.
you do service ldap start on RHEL5, but service slapd start
on RHEL6
On RHEL6, you want all of the following packages installed on your clients:
yum install openldap-clients pam_ldap nss-pam-ldapd
This seems to be the magic incantation that works for me (with real SSL
certificates, though):
authconfig --enableldap --enableldapauth \
--ldapserver ldap.example.com \
--ldapbasedn="dc=example,dc=com" \
--update
Be aware that there are multiple ldap configuration files involved now.
All of the following end up with ldap config entries in them and need to
be checked:
- /etc/openldap/ldap.conf
- /etc/pam_ldap.conf
- /etc/nslcd.conf
- /etc/sssd/sssd.conf
Note too that /etc/openldap/ldap.conf uses uppercased directives (e.g. URI)
that get lowercased in the other files (URI -> uri). Additionally, some
directives are confusingly renamed as well - e.g. TLA_CACERT in
/etc/openldap/ldap.conf becomes tla_cacertfile in most of the others.
:-(
If you want to do SSL or TLS, you should know that the default behaviour
is for ldap clients to verify certificates, and give misleading bind errors
if they can't validate them. This means:
if you're using self-signed certificates, add TLS_REQCERT allow to
/etc/openldap/ldap.conf on your clients, which means allow certificates
the clients can't validate
if you're using CA-signed certificates, and want to verify them, add
your CA PEM certificate to a directory of your choice (e.g.
/etc/openldap/certs, or /etc/pki/tls/certs, for instance), and point
to it using TLA_CACERT in /etc/openldap/ldap.conf, and
tla_cacertfile in /etc/ldap.conf.
RHEL6 uses a new-fangled /etc/openldap/slapd.d directory for the old
/etc/openldap/slapd.conf config data, and the
RHEL6 Migration Guide
tells you to how to convert from one to the other. But if you simply
rename the default slapd.d directory, slapd will use the old-style
slapd.conf file quite happily, which is much easier to read/modify/debug,
at least while you're getting things working.
If you run into problems on the server, there are lots of helpful utilities
included with the openldap-servers package. Check out the manpages for
slaptest(8), slapcat(8), slapacl(8), slapadd(8), etc.
Further reading:
Tue 18 Oct 2011
Tags: linux, sysadmin, centos, rhel
rpm-find-changes is a little script I wrote a while ago for rpm-based
systems (RedHat, CentOS, Mandriva, etc.). It finds files in a filesystem
tree that are not owned by any rpm package (orphans), or are modified
from the version distributed with their rpm. In other words, any file
that has been introduced or changed from it's distributed version.
It's intended to help identify candidates for backup, or just for
tracking interesting changes. I run it nightly on /etc on most of my
machines, producing a list of files that I copy off the machine (using
another tool, which I'll blog about later) and store in a git
repository.
I've also used it for tracking changes to critical configuration trees
across multiple machines, to make sure everything is kept in sync, and
to be able to track changes over time.
Available on github:
https://github.com/gavincarr/rpm-find-changes
Mon 08 Aug 2011
Tags: googleplus, hangout, centos
Kudos to Google for providing linux plugins for their
Google Plus+
Hangouts (a multi-way video chat system), for both debian-based and
rpm-based systems. The library requirements don't seem to be
documented anywhere though, so here's the magic incantation required
for installation on CentOS6 x86_64:
yum install libstdc++.i686 gtk2.i686 \
libXrandr.i686 libXcomposite.i686 libXfixes.i686 \
pulseaudio-libs.i686 alsa-lib.i686
Wed 29 Jun 2011
Tags: date, hacks, ntp
One of today's annoyances was a third-party complaining about clock
skew on a server at their site that they're testing against. No, they
don't have a local ntp server, and no, they couldn't allow us to
connect out to a designated ntp server externally. All we have is an
ssh forward in.
They wanted me to manually set the clock on the server whenever they
noticed it was out of synch! Real professionals.
I thought about tunneling an ntp stream out, but that requires
udp-to-tcp fudging at each end, or using ssh's Tunnel facility, which
requires root at both ends.
In the end, I settled for the low-tech approach - a once a day cron
job that resets the time based on a local clock. Ugly, but good enough:
ssh root@server date --utc $(date --utc "+%m%d%H%M%Y.%S")
Wed 23 Feb 2011
Tags: sysadmin, redis, ruby
When you have more than a handful of hosts on your network, you need to
start keeping track of what services are living where, what roles
particular servers have, etc. This can be documentation-based (say on a
wiki, or offline), or it can be implicit in a configuration management
system. Old-school sysadmins often used dns TXT records for these kind of
notes, on the basis that it was easy to look them up from the command
line from anywhere.
I've been experimenting with the idea of using lightweight tags attached
to hostnames for this kind of data, and it's been working really nicely.
Hosttag is just a couple of ruby command line utilities, one (hosttag
or ht) for doing tag or host lookups, and one (htset/htdel) for
doing adds and deletes. Both are network based, so you can do lookups
from wherever you are, rather than having to go to somewhere centralised.
Hosttag uses a redis server to store the hostname-tag
and tag-hostname mappings as redis sets, which makes queries lightning
fast, and setup straightforward.
So let's see it in action (rpms available in
my yum repo):
# Installation - first install redis somewhere, and setup a 'hosttag'
# dns alias to the redis host (or use the `-s <server>` option in
# the examples that follow). e.g. on CentOS:
$ yum install redis rubygem-redis
# Install hosttag as an rpm package (from my yum repo).
# Also requires/installs the redis rubygem.
$ yum install hosttag
# gem version coming soon (gem install hosttag)
# Setup some test data (sudo is required for setting and deleting)
# Usage: htset --tag <host> <tag1> <tag2> <tag3> ...
$ sudo htset --tag server1 dns dell ldap server centos centos5 i386 syd
$ sudo htset --tag server2 dns dell ldap server debian debian6 x86_64 mel
$ sudo htset --tag server3 hp nfs server centos centos6 x86_64 syd
$ sudo htset --tag lappy laptop ubuntu maverick i386 syd
# Now run some queries
# Query by tag
$ ht dns
server1 server2
$ ht i386
lappy server1
# Query by host
$ ht server2
debian debian6 dell dns ldap mel server x86_64
# Multiple arguments
$ ht --or centos debian
server1 server2 server3
$ ht --and dns ldap
server1 server2
# All hosts
$ ht --all
lappy server1 server2 server3
# All tags
$ ht --all-tags
centos centos5 centos6 debian debian6 dell dns hp i386 laptop ldap \
maverick mel nfs server syd ubuntu x86_64
An obvious use case is to perform actions on multiple hosts using your
ssh loop of choice e.g.
$ sshr $(ht centos) 'yum -y update'
Finally, a warning: hosttag doesn't have any security built in yet, so it
should only be used on trusted networks.
Source code is on github - patches
welcome :-).
Mon 31 Jan 2011
Tags: shares, investing, atom, rss, planetaux
I wrote a post a couple of years ago surveying
the 20 biggest Australian listed companies in terms of how they syndicate
the ASX announcements they make when they have news for the market.
All of the companies made their announcements available on their (or an
associated) website, but that obviously requires that an interested investor
or follower of the company regularly check the announcements page of each
company they're interested in - not ideal, timely, or scalable.
"Push" or broadcast approaches are much more useful for this sort of thing,
and back then 14 or the 20 did make their announcements available via email
- which is useful enough - and 7 of the 20 also published RSS or Atom feeds,
which are better still, because they're more lightweight, centralised, and
standardised/re-mixable.
On the other hand, 7/20 is a pretty ordinary score, especially considering
these are the biggest Australian listed companies. Smaller companies with
fewer resources are presumably going to fare even worse.
Two years on the number of ASX20 companies with announcements feeds has now
leapt to ... 9. :-/
And so towards the end of last year I decided to scratch this particular
itch, and built an announcement aggregator site called
PlanetAUX. It aggregates the announcements
feeds of the ASX20 companies with them, and generates announcements feeds
for those that don't have them.
Of the companies in the ASX20, there was only one whose HTML was so terrible
that I couldn't manage to parse it. I figured 19/20 isn't bad for starters.
Plans are to add more companies as time and interest demands, presumably
the remainder of the ASX50 initially, and then we'll see how things go.
Happy feeding.
Fri 14 Jan 2011
Tags: linux, rhel6, centos6, gdm
Update: ilaiho has provided a better solution in the comments,
which is to install the xorg-x11-xinit-session package, which adds
a "User script" session option. This will invoke your (executable)
~/.xsession or ~/.Xclients configs, if selected, and works well,
so I'd recommend you go that route instead of using this patch now.
The GDM Greeter in RHEL6 seems to have lost the ability to select
'session types' (or window managers), which apparently means you're
stuck using Gnome, even if you have other better options installed.
One workaround is to install KDM instead, and set DISPLAYMANAGER=KDE
in your /etc/sysconfig/desktop config, as KDM does still support
selectable session types.
Since I've become a big fan of
tiling window managers
in general, and ion
in particular, this was pretty annoying, so I wasted a few hours
today working through the /etc/X11 scripts and figuring out how
they hung together on RHEL6.
So for any other gnome-haters out there who don't want to have to
go to KDM, here's a patch to /etc/X11/xinit/Xsession that ignores
the default 'gnome-session' set by GDM, which allows proper window
manager selection either by user .xsession or .Xclients files,
or by the /etc/sysconfig/desktop DISPLAY setting.
diff --git a/xinit/Xsession b/xinit/Xsession
index e12e0ee..ab94d28 100755
--- a/xinit/Xsession
+++ b/xinit/Xsession
@@ -30,6 +30,14 @@ SWITCHDESKPATH=/usr/share/switchdesk
# Xsession and xinitrc scripts which has been factored out to avoid duplication
. /etc/X11/xinit/xinitrc-common
+# RHEL6 GDM doesn't seem to support selectable sessions, and always requests a
+# gnome-session. So we unset this default here, to allow things like user
+# .xsession or .Xclients files to be checked, and /etc/sysconfig/desktop
+# settings (via /etc/X11/xinit/Xclients) honoured.
+if [ -n "$GDMSESSION" -a $# -eq 1 -a "$1" = gnome-session ]; then
+ shift
+fi
+
# This Xsession.d implementation, is intended to obsolte and replace the
# various mechanisms present in the 'case' statement which follows, and to
# eventually be able to easily remove all hard coded window manager specific
Apply as root:
cd /etc/X11
patch -p1 < /tmp/xsession.patch
Sun 09 Jan 2011
Tags: linux, sysadmin
Here's what I use to take a quick inventory of a machine before a rebuild,
both to act as a reference during the rebuild itself, and in case something
goes pear-shaped. The whole chunk after script up to exit is
cut-and-pastable.
# as root, where you want your inventory file
script $(hostname).inventory
export PS1='\h:\w\$ ' # reset prompt to avoid ctrl chars
fdisk -l /dev/sd? # list partition tables
cat /proc/mdstat # list raid devices
pvs # list lvm stuff
vgs
lvs
df -h # list mounts
ip addr # list network interfaces
ip route # list network routes
cat /etc/resolv.conf # show resolv.conf
exit
# Cleanup control characters in the inventory
perl -i -pe 's/\r//g; s/\033\]\d+;//g; s/\033\[\d+m//g; s/\007/\//g' \
$(hostname).inventory
# And then copy it somewhere else in case of problems ;-)
scp $(hostname).inventory somewhere:
Anything else useful I've missed?
Sat 04 Dec 2010
Tags: sysadmin, cronologue, blosxom
Came across cronologger
(blog post)
recently (via Dean Wilson),
which is a simple wrapper script you use around your cron(8) jobs, which
captures any stdout and stderr output and logs it to a couchdb database,
instead of the traditional behaviour of sending it to you as email.
It's a nice idea, particularly for jobs with important output where it
would be nice to able to look back in time more easily than by trawling
through a noisy inbox, or for sites with lots of cron jobs where the sheer
volume is difficult to handle usefully as email.
Cronologger comes with a simple web interface for displaying your cron jobs,
but so far it's pretty rudimentary. I quickly realised that this was another
place (cf. blosxom4nagios) where
blosxom could be used to provide a pretty
useful gui with very little work.
Thus: cronologue.
cronologue(1) is the wrapper, written in perl, which logs job records and
and stdout/stderr output via standard HTTP PUTs back to a designated apache
server, as flat text files. Parameters can be used to control whether job
records are always created, or only when there is output produced. There's
also a --passthru mode in which stdout and stderr streams are still output,
allowing both email and cronologue output to be produced.
On the server side a custom blosxom install is used to display the job records,
which can be filtered by hostname or by date. There's also an RSS feed available.
Obligatory screenshot:
Update: I should add that RPMs for CentOS5 (but which will probably work on
most RPM-based distros) are available from
my yum repository.
Wed 24 Nov 2010
Tags: perl, parallel, fork
Needed to parallelise some processing in perl the last few days, and
did a quick survey of some of the parallel processing modules on CPAN,
of which there is the normal
bewildering diversity.
As usual, it depends exactly what you're trying to do. In my case I
just needed to be able to fork a bunch of processes off, have them
process some data, and hand the results back to the parent.
So here are my notes on a random selection of the available modules.
The example each time is basically a parallel version of the following
map:
my %out = map { $_ ** 2 } 1 .. 50;
Object oriented wrapper around 'fork'. Supports parent callbacks.
Passing data back to parent uses files, and feels a little bit clunky.
Dependencies: none.
use Parallel::ForkManager 0.7.6;
my @num = 1 .. 50;
my $pm = Parallel::ForkManager->new(5);
my %out;
$pm->run_on_finish(sub { # must be declared before first 'start'
my ($pid, $exit_code, $ident, $exit_signal, $core_dump, $data) = @_;
$out{ $data->[0] } = $data->[1];
});
for my $num (@num) {
$pm->start and next; # Parent nexts
# Child
my $sq = $num ** 2;
$pm->finish(0, [ $num, $sq ]); # Child exits
}
$pm->wait_all_children;
[Version 0.7.9]
Basically a parallel version of 'map'. Dependencies: none.
use Parallel::Iterator qw(iterate);
my @num = 1 .. 50;
my $it = iterate( sub {
# sub is a closure, return outputs
my ($id, $num) = @_;
return $num ** 2;
}, \@num );
my %out = ();
while (my ($num, $square) = $it->()) {
$out{$num} = $square;
}
[Version 1.00]
Provides parallel versions of 'foreach' and 'while'. It uses 'tie' to allow
shared data structures between the parent and children. Dependencies:
Parallel::ForkManager.
use Parallel::Loops;
my @num = 1 .. 50;
my $pl = Parallel::Loops->new(5);
my %out;
$pl->share(\%out);
$pl->foreach( \@num, sub {
my $num = $_; # note this uses $_, not @_
$out{$num} = $num ** 2;
});
You can also return values from the subroutine like Iterator, avoiding
the explicit 'share':
my %out = $pl->foreach( \@num, sub {
my $num = $_; # note this uses $_, not @_
return ( $num, $num ** 2 );
});
[Version 0.03]
Provides an interesting perlish forking interface using blocks. No built-in
support for returning data from children, but provides examples using pipes.
Dependencies: Exporter::Tidy.
use Proc::Fork;
use IO::Pipe;
use Storable qw(freeze thaw);
my @num = 1 .. 50;
my @children;
for my $num (@num) {
my $pipe = IO::Pipe->new;
run_fork{ child {
# Child
$pipe->writer;
print $pipe freeze([ $num, $num ** 2 ]);
exit;
} };
# Parent
$pipe->reader;
push @children, $pipe;
}
my %out;
for my $pipe (@children) {
my $entry = thaw( <$pipe> );
$out{ $entry->[0] } = $entry->[1];
}
[Version 0.71]
Like Parallel::ForkManager, but adds better signal handling. Doesn't
seem to provide built-in support for returning data from children.
Dependencies: Proc::Wait3.
[Version 0.08]
More complex module, loosely based on ForkManager (?). Includes better signal
handling, and supports scheduling and dependencies between different groups
of subprocesses. Doesn't appear to provide built-in support for passing data
back from children.
[Version 1.232]
Tue 16 Nov 2010
Tags: riak, sysadmin, brackup, rhel, centos
Been playing with Riak recently, which is
one of the modern dynamo-derived nosql databases (the other main ones being
Cassandra and Voldemort). We're evaluating it for use as a really large
brackup datastore, the primary attraction
being the near linear scalability available by adding (relatively cheap) new
nodes to the cluster, and decent availability options in the face of node
failures.
I've built riak packages for RHEL/CentOS 5, available at my
repository,
and added support for a riak 'target' to the
latest version (1.10) of brackup
(packages also available at my repo).
The first thing to figure out is the maximum number of nodes you expect
your riak cluster to get to. This you use to size the ring_creation_size
setting, which is the number of partitions the hash space is divided into.
It must be a power of 2 (64, 128, 256, etc.), and the reason it's important
is that it cannot be easily changed after the cluster has been created.
The rule of thumb is that for performance you want at least 10 partitions
per node/machine, so the default ring_creation_size of 64 is really only
useful up to about 6 nodes. 128 scales to 10-12, 256 to 20-25, etc. For more
info see the Riak Wiki.
Here's the script I use for configuring a new node on CentOS. The main
things to tweak here are the ring_creation_size you want (here I'm using
512, for a biggish cluster), and the interface to use to get the default ip
address (here eth0, or you could just hardcode 0.0.0.0 instead of $ip).
#!/bin/sh
# Riak configuration script for CentOS/RHEL
# Install riak (and IO::Interface, for next)
yum -y install riak perl-IO-Interface
# To set app.config:web_ip to use primary ip, do:
perl -MIO::Interface::Simple -i \
-pe "BEGIN { \$ip = IO::Interface::Simple->new(q/eth0/)->address; }
s/127\.0\.0\.1/\$ip/" /etc/riak/app.config
# To add a ring_creation_size clause to app.config, do:
perl -i \
-pe 's/^((\s*)%% riak_web_ip)/$2%% ring_creation_size is the no. of partitions to divide the hash
$2%% space into (default: 64).
$2\{ring_creation_size, 512\},
$1/' /etc/riak/app.config
# To set riak vm_args:name to hostname do:
perl -MSys::Hostname -i -pe 's/127\.0\.0\.1/hostname/e' /etc/riak/vm.args
# display (bits of) config files for checking
echo
echo '********************'
echo /etc/riak/app.config
echo '********************'
head -n30 /etc/riak/app.config
echo
echo '********************'
echo /etc/riak/vm.args
echo '********************'
cat /etc/riak/vm.args
Save this to a file called e.g. riak_configure, and then to configure a couple
of nodes you do the following (note that NODE is any old internal hostname you use
to ssh to the host in question, but FIRST_NODE needs to use the actual -name
parameter defined in /etc/riak/vm.args on your first node):
# First node
NODE=node1
cat riak_configure | ssh $NODE sh
ssh $NODE 'chkconfig riak on; service riak start'
# Run the following until ringready reports TRUE
ssh $NODE riak-admin ringready
# All nodes after the first
FIRST_NODE=riak@node1.example.com
NODE=node2
cat riak_configure | ssh $NODE sh
ssh $NODE "chkconfig riak on; service riak start && riak-admin join $FIRST_NODE"
# Run the following until ringready reports TRUE
ssh $NODE riak-admin ringready
That's it. You should now have a working riak cluster accessible on port 8098 on your
cluster nodes.
Fri 08 Oct 2010
Tags: sysadmin, centos, pxe
Problem: you've got a remote server that's significantly hosed, either
through a screwup somewhere or a power outage that did nasty things to
your root filesystem or something. You have no available remote hands,
and/or no boot media anyway.
Preconditions: You have another server you can access on the same
network segment, and remote access to the broken server, either through
a DRAC or iLO type card, or through some kind of serial console server
(like a Cyclades/Avocent box).
Solution: in extremis, you can do a remote rebuild. Here's the simplest
recipe I've come up with. I'm rebuilding using centos5-x86_64 version
5.5; adjust as necessary.
Note: dnsmasq, mrepo and syslinux are not core CentOS packages,
so you need to enable the rpmforge
repository to follow this recipe. This just involves:
wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm
rpm -Uvh rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm
1. On your working box (which you're now going to press into service as a
build server), install and configure dnsmasq
to provide dhcp and tftp services:
# Install dnsmasq
yum install dnsmasq
# Add the following lines to the bottom of your /etc/dnsmasq.conf file
# Note that we don't use the following ip address, but the directive
# itself is required for dnsmasq to turn dhcp functionality on
dhcp-range=ignore,192.168.1.99,192.168.1.99
# Here use the broken server's mac addr, hostname, and ip address
dhcp-host=00:d0:68:09:19:80,broken.example.com,192.168.1.5,net:centos5x
# Point the centos5x tag at the tftpboot environment you're going to setup
dhcp-boot=net:centos5x,/centos5x-x86_64/pxelinux.0
# And enable tftp
enable-tftp
tftp-root = /tftpboot
#log-dhcp
# Then start up dnsmasq
service dnsmasq start
2. Install and configure mrepo
to provide your CentOS build environment:
# Install mrepo and syslinux
yum install mrepo syslinux
# Setup a minimal /etc/mrepo.conf e.g.
cat > /etc/mrepo.conf
[main]
srcdir = /var/mrepo
wwwdir = /var/www/mrepo
confdir = /etc/mrepo.conf.d
arch = x86_64
mailto = root@example.com
smtp-server = localhost
pxelinux = /usr/lib/syslinux/pxelinux.0
tftpdir = /tftpboot
[centos5]
release = 5
arch = x86_64
metadata = repomd repoview
name = Centos-$release $arch
#iso = CentOS-$release.5-$arch-bin-DVD-?of2.iso
#iso = CentOS-$release.5-$arch-bin-?of8.iso
^D
# (uncomment one of the iso lines above, either the DVD or the CD one)
# Download the set of DVD or CD ISOs for the CentOS version you want
# There are fewer DVD ISOs, but you need to use bittorrent to download
mkdir -p /var/mrepo/iso
cd /var/mrepo/iso
elinks http://isoredirect.centos.org/centos/5.5/isos/x86_64/
# Once your ISOs are available in /var/mrepo/iso, and the 'iso' line
# in /etc/mrepo.conf updated appropriately, run mrepo itself
mrepo -gvv
3. Finally, finish setting up your tftp environment. mrepo should have copied
appropriate pxelinux.0, initrd.img, and vmlinuz files into your
/tftpboot/centos5-x86_64 directory, so all you need to supply is an
appropriate grub boot config:
cd /tftpboot/centos5-x86_64
ls
mkdir -p pxelinux.cfg
# Setup a default grub config (adjust the serial/console and repo params as needed)
cat > pxelinux.cfg/default
default linux
serial 0,9600n8
label linux
root (nd)
kernel vmlinuz
append initrd=initrd.img console=ttyS0,9600 repo=http://192.168.1.1/mrepo/centos5-x86_64
^D
Now get your server to do a PXE boot (via a boot option or the bios or whatever),
and hopefully your broken server will find your dhcp/tftp environment and boot up
in install mode, and away you go.
If you have problems with the boot, try checking your /var/log/messages file on the
boot server for hints.
Mon 22 Mar 2010
Tags: dell, omsa, centos, rhel, linux, sysadmin
Following on from my IPMI explorations, here's the next
chapter in my getting-down-and-dirty-with-dell-hardware-on-linux adventures.
This time I'm setting up Dell's
OpenManage Server Administrator
software, primarily in order to explore being able to configure bios settings
from within the OS. As before, I'm running CentOS 5, but OMSA supports any of
RHEL4, RHEL5, SLES9, and SLES10, and various versions of Fedora Core and
OpenSUSE.
Here's what I did to get up and running:
# Configure the Dell OMSA repository
wget -O bootstrap.sh http://linux.dell.com/repo/hardware/latest/bootstrap.cgi
# Review the script to make sure you trust it, and then run it
sh bootstrap.sh
# OR, for CentOS5/RHEL5 x86_64 you can just install the following:
rpm -Uvh http://linux.dell.com/repo/hardware/latest/platform_independent/rh50_64/prereq/\
dell-omsa-repository-2-5.noarch.rpm
# Install base version of OMSA, without gui (install srvadmin-all for more)
yum install srvadmin-base
# One of daemons requires /usr/bin/lockfile, so make sure you've got procmail installed
yum install procmail
# If you're running an x86_64 OS, there are a couple of additional 32-bit
# libraries you need that aren't dependencies in the RPMs
yum install compat-libstdc++-33-3.2.3-61.i386 pam.i386
# Start OMSA daemons
for i in instsvcdrv dataeng dsm_om_shrsvc; do service $i start; done
# Finally, you can update your path by doing logout/login, or just run:
. /etc/profile.d/srvadmin-path.sh
Now to check whether you're actually functional you can try a few of the
following (as root):
omconfig about
omreport about
omreport system -?
omreport chassis -?
omreport is the OMSA CLI reporting/query tool, and omconfig is the
equivalent update tool. The main documentation for the current version of
OMSA is here.
I found the CLI User's Guide
the most useful.
Here's a sampler of interesting things to try:
# Report system overview
omreport chassis
# Report system summary info (OS, CPUs, memory, PCIe slots, DRAC cards, NICs)
omreport system summary
# Report bios settings
omreport chassis biossetup
# Fan info
omreport chassis fans
# Temperature info
omreport chassis temps
# CPU info
omreport chassis processors
# Memory and memory slot info
omreport chassis memory
# Power supply info
omreport chassis pwrsupplies
# Detailed PCIe slot info
omreport chassis slots
# DRAC card info
omreport chassis remoteaccess
omconfig allows setting object attributes using a key=value syntax, which
can get reasonably complex. See the CLI User's Guide above for details, but
here are some examples of messing with various bios settings:
# See available attributes and settings
omconfig chassis biossetup -?
# Turn the AC Power Recovery setting to On
omconfig chassis biossetup attribute=acpwrrecovery setting=on
# Change the serial communications setting (on with serial redirection via)
omconfig chassis biossetup attribute=serialcom setting=com1
omconfig chassis biossetup attribute=serialcom setting=com2
# Change the external serial connector
omconfig chassis biossetup attribute=extserial setting=com1
omconfig chassis biossetup attribute=extserial setting=rad
# Change the Console Redirect After Boot (crab) setting
omconfig chassis biossetup attribute=crab setting=enabled
omconfig chassis biossetup attribute=crab setting=disabled
# Change NIC settings (turn on PXE on NIC1)
omconfig chassis biossetup attribute=nic1 setting=enabledwithpxe
Finally, there are some interesting formatting options available to both
omreport, for use in scripting e.g.
# Custom delimiter format (default semicolon)
omreport chassis -fmt cdv
# XML format
omreport chassis -fmt xml
# To change the default cdv delimiter
omconfig preferences cdvformat -?
omconfig preferences cdvformat delimiter=pipe
Thu 11 Mar 2010
Tags: linux, centos, rhel, ipmi, dell
Spent a few days deep in the bowels of a couple of datacentres last week,
and realised I didn't know enough about Dell's DRAC base management
controllers to use them properly. In particular, I didn't know how to
mess with the drac settings from within the OS. So spent some of today
researching that.
Turns out there are a couple of routes to do this. You can use the Dell
native tools (e.g. racadm) included in Dell's
OMSA product, or you can use
vendor-neutral IPMI,
which is well-supported by Dell DRACs. I went with the latter as it's
more cross-platform, and the tools come native with CentOS, instead of
having to setup Dell's OMSA repositories. The Dell-native tools may give
you more functionality, but for what I wanted to do IPMI seems to work
just fine.
So installation is just:
yum install OpenIPMI OpenIPMI-tools
chkconfig ipmi on
service ipmi start
and then from the local machine you can use ipmitool to access and
manipulate all kinds of useful stuff:
# IPMI commands
ipmitool help
man ipmitool
# To check firmware version
ipmitool mc info
# To reset the management controller
ipmitool mc reset [ warm | cold ]
# Show field-replaceable-unit details
ipmitool fru print
# Show sensor output
ipmitool sdr list
ipmitool sdr type list
ipmitool sdr type Temperature
ipmitool sdr type Fan
ipmitool sdr type 'Power Supply'
# Chassis commands
ipmitool chassis status
ipmitool chassis identify [<interval>] # turn on front panel identify light (default 15s)
ipmitool [chassis] power soft # initiate a soft-shutdown via acpi
ipmitool [chassis] power cycle # issue a hard power off, wait 1s, power on
ipmitool [chassis] power off # issue a hard power off
ipmitool [chassis] power on # issue a hard power on
ipmitool [chassis] power reset # issue a hard reset
# Modify boot device for next reboot
ipmitool chassis bootdev pxe
ipmitool chassis bootdev cdrom
ipmitool chassis bootdev bios
# Logging
ipmitool sel info
ipmitool sel list
ipmitool sel elist # extended list (see manpage)
ipmitool sel clear
For remote access, you need to setup user and network settings, either at boot time
on the DRAC card itself, or from the OS via ipmitool:
# Display/reset password for default root user (userid '2')
ipmitool user list 1
ipmitool user set password 2 <new_password>
# Display/configure lan settings
ipmitool lan print 1
ipmitool lan set 1 ipsrc [ static | dhcp ]
ipmitool lan set 1 ipaddr 192.168.1.101
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 192.168.1.254
Once this is configured you should be able to connect using the 'lan' interface
to ipmitool, like this:
ipmitool -I lan -U root -H 192.168.1.101 chassis status
which will prompt you for your ipmi root password, or you can do the following:
echo <new_password> > ~/.racpasswd
chmod 600 ~/.racpasswd
and then use that password file instead of manually entering it each time:
ipmitool -I lan -U root -f ~/.racpasswd -H 192.168.1.101 chassis status
I'm using an 'ipmi' alias that looks like this:
alias ipmi='ipmitool -I lan -U root -f ~/.racpasswd -H'
# which then allows you to do the much shorter:
ipmi 192.168.1.101 chassis status
# OR
ipmi <hostname> chassis status
Finally, if you configure serial console redirection in the bios as follows:
Serial Communication -> Serial Communication: On with Console Redirection via COM2
Serial Communication -> External Serial Connector: COM2
Serial Communication -> Redirection After Boot: Disabled
then you can setup standard serial access in grub.conf and inittab on com2/ttyS1
and get serial console access via IPMI serial-over-lan using the 'lanplus' interface:
ipmitool -I lanplus -U root -f ~/.racpasswd -H 192.168.1.101 sol activate
which I typically use via a shell function:
# ipmi serial-over-lan function
isol() {
if [ -n "$1" ]; then
ipmitool -I lanplus -U root -f ~/.racpasswd -H $1 sol activate
else
echo "usage: sol <sol_ip>"
fi
}
# used like:
isol 192.168.1.101
isol <hostname>
Further reading:
Fri 26 Feb 2010
Tags: linux, rpm, centos
Mock is a Fedora project that allows
you to build RPM packages within a chroot environment, allowing you to build
packages for other systems than the one you're running on (e.g. building CentOS 4
32-bit RPMs on a CentOS 5 64-bit host), and ensuring that all the required build
dependencies are specified correctly in the RPM spec file.
It's also pretty under-documented, so these are my notes on things I've figured out
over the last week setting up a decent mock environment on CentOS 5.
First, I'm using mock 1.0.2 from the EPEL repository, rather than older 0.6.13
available from CentOS Extras. There are apparently backward-compatibility problems
with versions of mock > 0.6, but as I'm mostly building C5 packages I decided to go
with the newer version. So installation is just:
# Install mock and python-ctypes packages (the latter for better setarch support)
$ sudo yum --enablerepo=epel install mock python-ctypes
# Add yourself to the 'mock' group that will have now been created
$ sudo usermod -G mock gavin
The mock package creates an /etc/mock directory with configs for various OS
versions (mostly Fedoras). The first thing you want to tweak there is the
site-defaults.cfg file which sets up various defaults for all your builds. Mine now
looks like this:
# /etc/mock/site-defaults.cfg
# Set this to true if you've installed python-ctypes
config_opts['internal_setarch'] = True
# Turn off ccache since it was causing errors I haven't bothered debugging
config_opts['plugin_conf']['ccache_enable'] = False
# (Optional) Fake the build hostname to report
config_opts['use_host_resolv'] = False
config_opts['files']['etc/hosts'] = """
127.0.0.1 buildbox.openfusion.com.au nox.openfusion.com.au localhost
"""
config_opts['files']['etc/resolv.conf'] = """
nameserver 127.0.0.1
"""
# Setup various rpm macros to use
config_opts['macros']['%packager'] = 'Gavin Carr <gavin@openfusion.com.au>'
config_opts['macros']['%debug_package'] = '%{nil}'
You can use the epel-5-{i386,x86_64}.cfg configs as-is if you like; I copied them
to centos-5-{i386,x86_64}.cfg versions and removed the epel 'extras', 'testing',
and 'local' repositories from the yum.conf section, since I typically want to build
using only 'core' and 'update' packages.
You can then run a test by doing:
# e.g. initialise a centos-5-i386 chroot environment
$ CONFIG=centos-5-i386
$ mock -r $CONFIG --init
which will setup an initial chroot environment using the given config. If that
seemed to work (you weren't inundated with error messages), you can try a build:
# Rebuild the given source RPM within the chroot environment
# usage: mock -r <mock_config> --rebuild /path/to/SRPM e.g.
$ mock -r $CONFIG --rebuild ~/rpmbuild/SRPMS/clix-0.3.4-1.of.src.rpm
If the build succeeds, it drops your packages into the /var/lib/mock/$CONFIG/result
directory:
$ ls -1 /var/lib/mock/$CONFIG/result
build.log
clix-0.3.4-1.of.noarch.rpm
clix-0.3.4-1.of.src.rpm
root.log
state.log
If it fails, you can check mock output, the *.log files above for more info, and/or
rerun mock with the -v flag for more verbose messaging.
A couple of final notes:
the chroot environments are cached, but rebuilding them and checking for updates
can be pretty network intensive, so you might want to consider setting up a local
repository to pull from. mrepo (available
from rpmforge) is pretty good for that.
there don't seem to be any hooks in mock to allow you to sign packages you've
built, so if you do want signed packages you need to sign them afterwards via a
rpm --resign $RPMS.
Wed 20 Jan 2010
Tags: backups, brackup, sysadmin
After using brackup for a while you find
you have a big list of backups sitting on your server, and start to think
about cleaning up some of the older ones. The standard brackup tool for this
is brackup-target, and the prune and gc (garbage collection)
subcommands.
Typical usage is something like this:
# List the backups for a particular target on the server e.g.
TARGET=myserver_images
brackup-target $TARGET list-backups
Backup File Backup Date Size (B)
----------- ----------- --------
images-1262106544 Thu 31 Dec 2009 03:32:49 1263128
images-1260632447 Sun 13 Dec 2009 08:19:13 1168281
images-1250042378 Wed 25 Nov 2009 06:25:06 977464
images-1239323644 Mon 09 Nov 2009 00:30:34 846523
images-1239577352 Thu 29 Oct 2009 13:03:02 846523
...
# Decide how many backups you want to keep, and prune (delete) the rest
brackup-target --keep-backups 15 $TARGET prune
# Prune just removes the brackup files on the server, so now you need to
# run a garbage collect to delete any 'chunks' that are now orphaned
brackup-target --interactive $TARGET gc
This simple scheme - "keep the last N backups" - works pretty nicely for
backups you do relatively infrequently. If you do more frequent backups,
however, you might find yourself wanting to be able to implement more
sophisticated retention policies. Traditional backup regimes often involve
policies like this:
- keep the last 2 weeks of daily backups
- keep the last 8 weekly backups
- keep monthly backups forever
It's not necessarily obvious how to do something like this with brackup, but
it's actually pretty straightforward. The trick is to define multiple
'sources' in your brackup.conf, one for each backup 'level' you want to use.
For instance, to implement the regime above, you might define the following:
# Daily backups
[SOURCE:images]
path = /data/images
...
# Weekly backups
[SOURCE:images-weekly]
path = /data/images
...
# Monthly backups
[SOURCE:images-monthly]
path = /data/images
...
You'd then use the images-monthly source once a month, the images-weekly
source once a week, and the images source the rest of the time. Your list
of backups would then look something like this:
Backup File Backup Date Size (B)
----------- ----------- --------
images-1234567899 Sat 05 Dec 2009 03:32:49 1263128
images-1234567898 Fri 04 Dec 2009 03:19:13 1168281
images-1234567897 Thu 03 Dec 2009 03:19:13 1168281
images-1234567896 Wed 02 Dec 2009 03:19:13 1168281
images-monthly-1234567895 Tue 01 Dec 2009 03:19:13 1168281
images-1234567894 Mon 30 Nov 2009 03:19:13 1168281
images-weekly-1234567893 Sun 29 Nov 2009 03:19:13 1168281
images-1234567892 Sat 28 Nov 2009 03:25:06 977464
...
And when you prune, you want to specify a --source argument, and specify
separate --keep-backups settings for each level e.g. for the above:
# Keep 2 weeks worth of daily backups
brackup-target --source images --keep-backups 12 $TARGET prune
# Keep 8 weeks worth of weekly backups
brackup-target --source images-weekly --keep-backups 8 $TARGET prune
# Keep all monthly backups, so we don't prune them at all
# And then garbage collect as normal
brackup-target --interactive $TARGET gc