collect server config files via rsync and commit them to a CVS with svn

I really like what RANCID does for routers and switches and wanted the same thing for my mail server configs. Here's what I put together with some help from the folks at freenode's #bash IRC channel. I hope someone else can find it a useful starting point. If you have any suggestions on how to improve it, I'd be grateful.

#!/bin/bash
#
# TODO:
#       *)make removing the working directories optional?
#

servers=("server-01" "server-02" "server-03" "server-04")
working_dir=/var/rancid/svn-temp/
rm -fr $working_dir/servers-cvs.mail
rm -fr $working_dir/servers-cvs2.mail

for i in "${servers[@]}"
 do
  echo '#########################################################' >>  $working_dir/servers-cvs.mail
  echo "$i: clearing working directories." >>  $working_dir/servers-cvs.mail
#  rm -rf $working_dir$i/\.*
#  rm -rf $working_dir$i/*
  cd /var/rancid/svn-servers/$i/
  echo "$i: checking out files from the repo." >>  $working_dir/servers-cvs.mail
  svn  checkout file:///var/rancid/svn-servers/$i $working_dir$i/
  echo "$i: copying files from $i into working directory." >>  $working_dir/servers-cvs.mail
  rsync  -q --no-motd --checksum -rptgoDLK --compress --recursive --delete  --relative --cvs-exclude  --exclude '/var/log/*'  root@$i:/etc/postfix :/etc/dovecot :/etc/mail :/var/www/html :/etc/snmp :/etc/mail :/etc/policyd :/etc/policyd-weight.conf  :/etc/fail2ban :/etc/amavisd.conf :/etc/clamd.conf :/mnt/lustre $working_dir$i/
  echo "$i: svn status:" >>  $working_dir/servers-cvs.mail
  svn status $working_dir$i   >>  $working_dir/servers-cvs.mail
  echo "$i: svn diff:"  >>  $working_dir/servers-cvs.mail
  svn diff $working_dir$i   >>  $working_dir/servers-cvs.mail
  echo "$i: committing changes to svn repo, if any."  >>  $working_dir/servers-cvs.mail
  IFS=$'\n'
  for new_line in $(svn status $working_dir$i); do
    file_svn_status=$(echo $new_line | cut -c1)         #get svn status of the file
    new_file=$(echo $new_line | cut -c2-)       #get the name of the file
    new_file="${new_file#"${new_file%%[![:space:]]*}"}" 
    echo '----------------------------------------------------------'
        case $file_svn_status in
        D)
                echo "this appears to be a DELETED file:$new_file.  Removing file from repo SVN" >>  $working_dir/servers-cvs2.mail
                svn remove /$new_file
                ;;
        M)
                echo "this appears to be a MODIFIED file:$new_file.  Committing changes to SVN" >>  $working_dir/servers-cvs2.mail
                ;;
        !)
                echo "this appears to have been removed from the working copy: $new_file .  Removing from repo SVN"  >>  $working_dir/servers-cvs2.mail
                svn remove /$new_file
                ;;
        ?)
                echo "this appears to be a new file: $new_file.  Committing new file to SVN"  >>  $working_dir/servers-cvs2.mail
                svn add /$new_file     
                ;;
        *)
                echo "Unknown fruit - sure it isn't toxic?"  >>  $working_dir/servers-cvs2.mail
        esac

  done  # done with 'for new_line  in $(svn status $working_dir$i)' loop
  svn commit $working_dir$i  -m 'auto-commit'
done     # done with 'for i in "${servers[@]}"' loop

#mail logic repurposed from RANCID
MAILHEADERS=""
    if [ -s $working_dir/servers-cvs2.mail ]  ; then
         echo "Today's date is $(date)" >> $working_dir/servers-cvs.mail
         echo "Today's date is $(date)" >> $working_dir/servers-cvs2.mail
        (
          echo "To: me@example.com"
          echo "Subject: changes in servers"
          echo "$MAILHEADERS" | awk '{L = "";LN = $0;while (LN ~ /\\n/) { I = index(LN,"\\n");L = L substr(LN,0,I-1) "\n";LN = substr(LN,I+2,length(LN)-I-1);}print L LN;}'
          echo ""
          cat $working_dir/servers-cvs.mail
          cat $working_dir/servers-cvs2.mail
        ) | /usr/sbin/sendmail -t
    fi

Blocking DNS requests with iptables

iptables -I INPUT 1 -p udp -m udp --dport 53 -m string --hex-string "|333032353904696e666f|" --algo kmp --from 30  --to 65535 -m comment --comment "drop 30259.info" -j DROP

Unfortunately that string is in hex, so you'll need to run a tcpdump to extract the hex-encoded domain name like so:

[bob@host01 ~]# tcpdump -nnX -c 10 -s 0 -l port 53 
05:18:53.307390 IP nnn.nnn.nnn.nnn.32825 > xxx.xxx.xxx.xxx.53:  7117+ [1au] ANY? 30259.info. (51)
        0x0000:  4500 004f 0000 4000 3f11 df23 cd86 d638  E..O..@.?..#...8
        0x0010:  d8b1 e009 8039 0035 003b 34c9 1bcd 0100  .....9.5.;4.....
        0x0020:  0001 0000 0000 0001 0533 3032 3539 0469  .........30259.i
        0x0030:  6e66 6f00 00ff 0001 0000 2910 0000 0000  nfo.......).....
        0x0040:  0000 0c50 fa00 0800 0120 000b 5dfb ed    ...P........]..

Notice that this happens to be a QTYPE 255: A request for all records (all cached records, not an AXFR which is type 252. I dunno if this QTYPE is useful in the real world or just a vector for attack. You can view all queries with QTYPE=255 with

tcpdump -c 10 -s 0 -l port 53 and ether[len - 3] == 0xff

find your public IP from behind a NAT device in linux

[me@host ~]# curl http://checkip.dyndns.org/

Current IP Address: 4.2.2.2

username, hostname and IP address have been changed to protect the innocent.

changes in CentOS logrotate archive naming convention

It looks like the archive naming scheme used by logrotated has changed from CentOS5 to CentOS6.  For any one doing any CentOS scripting, this may affect you.  The CentOS5 logrotated simply added a ".n" to the filename, where "n" is a number.  For example:

[root@mail01-01 ~]# ls -latrh /var/log/messages*
-rw------- 1 root root 4.5M Dec 23 04:11 /var/log/messages.5.gz
-rw------- 1 root root 4.8M Dec 24 04:10 /var/log/messages.4.gz
-rw------- 1 root root 4.4M Dec 25 04:09 /var/log/messages.3.gz
-rw------- 1 root root 4.4M Dec 26 04:08 /var/log/messages.2.gz
-rw------- 1 root root 4.5M Dec 27 04:09 /var/log/messages.1.gz
-rw------- 1 root root  17M Dec 27 08:50 /var/log/messages

In CentOS6, the file name is appended with the date it was rotated.  For example:

[root@radius1 radius]# ls -latrh /var/log/messages*
-rw-------. 1 root root 2.1K Dec  4 03:45 /var/log/messages-20111204
-rw-------. 1 root root 1.8K Dec 11 03:11 /var/log/messages-20111211
-rw-------. 1 root root  33K Dec 18 03:23 /var/log/messages-20111218
-rw-------. 1 root root  438 Dec 25 03:29 /var/log/messages-20111225
-rw-------. 1 root root  281 Dec 25 21:50 /var/log/messages

This is more efficient, to be sure, as the old method would rename all the archived files to make room for the newest one, changing messages.1.gz to messages.2.gz, messages.2.gz to messages.3.gz, etc.  This may however mess up some scripts if they're looking for the ".n" pattern.

Displaying Interface descriptions in SNMP traps using SNMPTT's PREXEC funtion

I collect snmp traps via snmptrapd which then hands them off to snmptt. Upon receipt of an interface down/up trap (.1.3.6.1.6.3.1.1.5.3, for example) snmptt then runs a quick snmpget request (unfortunately using SNMPv1 at the moment) to the reporting host to pull the description for the given interface like so:

EVENT ciscoConfigManEventDN .1.3.6.1.6.3.1.1.5.3 "Status Events" Normal
# use snmpget to fetch the interface description and save it as $p1, to be used it the FORMAT line.
PREEXEC /usr/bin/snmpget -v 1 -t 2 -Ovq -c snmptt $aA ifAlias.$1
FORMAT Link DOWN $2 - $p1
# OPTIONAL: do not process this event for VI interfaces (like PPPoE interfaces)
MATCH $2:!(Virtual-Access)
MATCH $2:!(Multilink)
MATCH MODE=and
#
EXEC /root/bin/sendemail.sh high traps@mydomain.com "ALERT-$s-$R-$Fz" "Agent $A at $aA reports $Fz$FnTrap $e:$D$Fn"

Note that I've created a special SNMP VIEW and community one our routers which allow access only to the ifAlias.* OID tree like so:

snmp-server view snmptt ifXEntry.* included
! note that ACL99 includes my snmp NMS only!
snmp-server community snmptt view snmptt RO 99

BackupPC client quickstart with improved security

To embellish and improve upon http://backuppc.sourceforge.net/faq/ssh.html#how_can_client_access_as_root_be_avoided I've been using the following procedure:

1.  Add the 'mybackuppc' user to the machine that is being backed up.  I try to avoid using "backuppc" as the username in case a common dictionary attack occurs. Then,  create a SSH key pair on the client as the 'mybackuppc' user (do not enter a password):
   

   useradd mybackuppc -c "Backup User"
   su - mybackuppc
   ssh-keygen -t rsa
   exit

2. 
   
3. Add the 'mybackuppc' user to the client's sudoers file using visudo to give the user sudo rights as follows:
   

   mybackuppc ALL=(ALL) NOPASSWD: /usr/bin/rsync --server *
   Defaults:mybackuppc !requiretty






4. Copy 'backuppc' users public key from the backup server:
   

   /var/lib/backuppc/.ssh/id_rsa.pub) to the mybackuppc's auth keys file on the client (/home/mybackuppc/.ssh/authorized_keys




5. SSH from the BackuPC server to the client once as the backuppc user (su - backuppc; ssh -l mybackuppc $host) to get the RSA fingerprint into the backup server's /var/lib/backuppc/.ssh/known_hosts file:
   

   bash-3.2$ /usr/bin/ssh -l mybackuppc client-host
   The authenticity of host 'client-host (x.x.x.x)' can't be established.
   RSA key fingerprint is xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added 'client-host, x.x.x.x' (RSA) to the list of known hosts.
   [mybackuppc@client-host ~]$ exit
   logout
   
   Connection to client-host closed.




6. Use the BackupPC web interface to add a new host. Notice in the comments at the bottom of the page that you can easily copy an existing host, or just add one from scratch with the defaults. You don't have to use an valid name available via DNS if you make sure the name is in /etc/hosts on the backup server. Alternatively, you could create a config file on the Backup Server in /etc/BackupPC/pc/ and then add them to /etc/BackupPC/hosts file.



7. Use the BackupPC server's web interface to initiate a full backup of the new client machine.

Site to Site VPN Worksheet

Periodically I find myself working with another party to establish a LAN to LAN, or Site to Site, IPSEC VPN tunnel and there are various parameters which must be agreed upon.  I have found that exchanging a VPN worksheet ahead of time helps both parties think through the process better and speeds up the VPN configuration.  This is preferable under all scenarios, but especially when one or more parties are paying by the hour to have someone configure it for them.  Feel free to download, alter and use it as you see fit:

https://docs.google.com/leaf?id=0Bw0IToZ5AnXTMjAzMzJiYmEtOWRjOC00OTY0LTg4ZjUtYzk4ZTk4MjViNTEx&hl=en_US&authkey=CMbmnecO