2013
04.09

On 2011-06-16 I wrote about a small PHP snippet to convert a MySQL database and all its tables to UTF-8/InnoDB. Yesterday, two years later, I was trying to accomplish the exact same thing using common_schema which I discovered recently.

According to it’s author(s) “common_schema is a framework for MySQL server administration“. In High Performance MySQL (3rd Edition), Baron Schwartz states that “common_schema is to MySQL as jQuery is to javaScript“. I don’t know about that, but it sure is really useful and fun to play with :-)

SET @script := "
  var $database := 'cacti';
  foreach ($table, $schema, $engine: table in :$database)
  {
    ALTER TABLE :$schema.:$table ENGINE=InnoDB;
    ALTER TABLE :$schema.:$table CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
  }
";
CALL common_schema.run(@script);
2013
04.06

Recently I had some jpg files (scans) that I wanted to combine to one pdf. With the convert utility it is as simple as:

convert *.jpg foo.pdf;

If you’re using Ubuntu (or Debian) you can get convert by installing ImageMagick:

sudo apt-get install imagemagick;
2013
03.26

Recently I was benchmarking several methods for exporting (dumping) and importing data. One tool that I wanted to test was mydumper. I started installing it the regular way:

sudo apt-get install mydumper;

Unfortunately Ubuntu’s version (0.5.1) did not work for me. This could be related to this bug, but also could have been caused by me using Percona Server instead of MySQL. Anyway, I decided to compile the latest version of mydumper myself:

wget https://launchpad.net/mydumper/0.5/0.5.2/+download/mydumper-0.5.2.tar.gz;
tar -xzvf mydumper-0.5.2.tar.gz;
cd mydumper-0.5.2;
 
cmake . -DCMAKE_INSTALL_PREFIX=~/bin/mydumper;
make;

Unfortunately that didn’t work either.

make[2]: *** No rule to make target `/usr/lib/libmysqlclient_r.so', needed by `mydumper'.

At first I didn’t know what to do, but after some Googling I found a similar problem. Examining /usr/lib revealed that libmysqlclient_r.so was pointing to a non-existing file.

Missing libmysqlclient_r.so

Missing libmysqlclient_r.so

After doing some research I decided to point libmysqlclient_r.so to libmysqlclient.so. Note that this is similar to libmysqlclient_r.a pointing to libmysqlclient.a.

cd /usr/lib;
ln -sf libmysqlclient.so libmysqlclient_r.so;

Doing so will (of course) fix the symlink.

Fixed libmysqlclient_r.so

Fixed libmysqlclient_r.so

Also the compilation of mydumper will succeed now!

make;
make install;
2012
12.20

As I wrote before, a few months ago my N900 (touchscreen) died and without a working touchscreen it proved to be impossible to operate it. Therefore I ordered a new one which could, but did not fix my phone.

In order to recover my text messages I wrote a simple Python script which reads the el-v1.db file in comm_and_cal.zip:/home/user/.rtcom-eventlogger/backup.tgz (which is an SQLite 3 database) and writes the messages to a CVS file (which are better handled by other phones).

#!/usr/bin/env python
# -*- coding: utf-8 -*-
 
import sys
import sqlite3
import os.path
import csv
from datetime import datetime
 
if len(sys.argv) != 3:
  print ""
  print "Usage:"
  print "%s [path to el-v1.db] [path to csv file]" % sys.argv[0]
  print ""
  sys.exit(1)
 
dbFileName  = sys.argv[1].strip()
csvFileName = sys.argv[2].strip()
 
if not os.path.exists(dbFileName):
  print ""
  print "Error:"
  print "Could not open '%s'" % dbFileName
  print ""
  sys.exit(1)
 
with sqlite3.connect(dbFileName) as connection, open(csvFileName, 'wb') as csvFd:
  cursor = connection.cursor()
  writer = csv.writer(csvFd, delimiter = '\t', quotechar = '"', quoting = csv.QUOTE_ALL)
 
  sql = "SELECT start_time, remote_uid, free_text FROM Events WHERE event_type_id = 7"
  for record in cursor.execute(sql):
    startTime, phoneNumber, textMessage = record
 
    startDateTime = datetime.fromtimestamp(startTime).strftime('%Y-%m-%d %H:%M:%S');
    phoneNumber   = phoneNumber.encode('utf-8')
    textMessage   = textMessage.encode('utf-8')
 
    writer.writerow([startTime, startDateTime, phoneNumber, textMessage])
 
connection.close()
csvFd.close()

Enjoy!

References:

2012
12.20

A few months ago my N900 (touchscreen) died. Without a working touchscreen it proved to be impossible to operate it. Although I could have logged in via SSH unfortunately my WIFI and 3G were of :-( So there was no possibly to create an export of all my data. Fortunately I had a backup. The only problem was that the backup was just a set of zip files:

# ll
totaal 4,0M
-rw-r--r-- 1 lorem ipsum 1,1K sep 25  2011 applications.zip
-rw-r--r-- 1 lorem ipsum  678 sep 25  2011 backup.metadata
-rw-r--r-- 1 lorem ipsum 233K sep 25  2011 bookmarks.zip
-rw-r--r-- 1 lorem ipsum 2,2M sep 25  2011 comm_and_cal.zip
-rw-r--r-- 1 lorem ipsum 1,6M sep 25  2011 settings.zip

containing “raw” dumps of directories. Not particularly useful for transferring to a new phone…

After some searching I found a file called addressbook.db (in comm_and_cal.zip:/home/user/.osso-abook-backup/db/) which should contain my contacts (vcard data). I read that addressbook.db was a Berkeley DB (BDB) database file. Because I was learning Python I wrote a simple script to extract the vcard data from the database and write it as separate vcards (which are easy to import on other devices) to a folder.

I hope it useful for someone else and please comment my python skills :-)

#!/usr/bin/env python
# -*- coding: utf-8 -*-
 
import sys
import vobject
from bsddb3 import db
 
if len(sys.argv) != 2:
  print ""
  print "Usage:"
  print "%s [path to addressbook.db]" % sys.argv[0]
  print ""
  sys.exit(1)
 
fileName = sys.argv[1].strip()
 
addressBookDb = db.DB()
try:
  # Try to open db file
  addressBookDb.open(fileName, None, db.DB_HASH, db.DB_DIRTY_READ)
 
  cursor = addressBookDb.cursor()
  record = cursor.first()
 
  # Fill a list with vcard strings from db 
  vCards = []
  while record:
 
    # Get vcard string and correct line endings
    vCardString = record[1].replace('\x00', '\r\n')
    vCards.append(vCardString)
 
    record = cursor.next()
 
  # Write vcard strings
  for vCard in vCards:
 
    parsedVCard = vobject.readOne(vCard)
 
    # Not all entries have a n(ame) attribute
    if hasattr(parsedVCard, 'n'):
      vCardName = str(parsedVCard.n.value).strip()
 
      # Open vcard for writing
      with open('vcards/%s.vcf' % vCardName, 'w') as fd:
        fd.write(vCard)
      fd.close()
 
except db.DBNoSuchFileError:
  print ""
  print "Error:"
  print "Could not open '%s'" % fileName
  print ""
  sys.exit(1)

References:

2012
12.20

Recently I needed to “rename” a user (mischa_new -> mischa-new). Personally I find changing only the name a bit ugly and confusing. So this is how I addressed it:

Change username

usermod -l mischa-new mischa_new;

Change home directory (move)

usermod -d /home/mischa-new -m mischa-new;

Change group name

groupmod -n mischa-new mischa_new;
2012
12.20

Personally I find Gnome’s `Recent documents` feature quit annoying. In earlier versions I always disabled it like this. Unfortunately that didn’t work anymore… Apparently .recently-used.xbel has moved to a new location (thanks ade). It is now in ~/.local/share/recently-used.xbel. So this should do the trick:

rm ~/.local/share/recently-used.xbel -Rf;
mkdir ~/.local/share/recently-used.xbel -p;
2012
11.18

Simple Network Management Protocol (SNMP) is an “Internet-standard protocol for managing devices on IP networks.”

Version 3:
Although SNMPv3 makes no changes to the protocol aside from the addition of cryptographic security, it looks much different due to new textual conventions, concepts, and terminology.

SNMPv3 primarily added security and remote configuration enhancements to SNMP.

Security has been the biggest weakness of SNMP since the beginning. Authentication in SNMP Versions 1 and 2 amounts to nothing more than a password (community string) sent in clear text between a manager and agent. Each SNMPv3 message contains security parameters which are encoded as an octet string. The meaning of these security parameters depends on the security model being used.

SNMPv3 provides important security features:

  • Confidentiality – Encryption of packets to prevent snooping by an unauthorized source.
  • Integrity – Message integrity to ensure that a packet has not been tampered with in transit including an optional packet replay protection mechanism.
  • Authentication – to verify that the message is from a valid source.

Configuration:
In Ubuntu 10.04 I used these settings which were working fine but not exactly secure:

/etc/default/snmpd

# This file controls the activity of snmpd and snmptrapd
 
# MIB directories.  /usr/share/snmp/mibs is the default, but
# including it here avoids some strange problems.
export MIBDIRS=/usr/share/snmp/mibs
 
# snmpd control (yes means start daemon).
SNMPDRUN=yes
 
# snmpd options (use syslog, close stdin/out/err).
SNMPDOPTS='-LS4d -Lf /dev/null -u snmp -g snmp -I -smux -p /var/run/snmpd.pid 127.0.0.1'
 
# snmptrapd control (yes means start daemon).  As of net-snmp version
# 5.0, master agentx support must be enabled in snmpd before snmptrapd
# can be run.  See snmpd.conf(5) for how to do this.
TRAPDRUN=no
 
# snmptrapd options (use syslog).
TRAPDOPTS='-Lsd -p /var/run/snmptrapd.pid'
 
# create symlink on Debian legacy location to official RFC path
SNMPDCOMPAT=yes

/etc/snmp/snmpd.conf

com2sec ro default public
com2sec rw localhost private
 
group public v1 ro
group public v2c ro
group public usm ro
group private v1 rw
group private v2c rw
group private usm rw
 
view all    included  .1                               80
view system included  .iso.org.dod.internet.mgmt.mib-2.system
 
access public "" any noauth exact all none none
access private "" any noauth exact all all all

In Ubuntu 12.04 I couldn’t get things to work in the first place. Not with the default settings and not with the above. Looking at /var/log/syslog made it clear that SNMPD wasn’t able to locate the MIBs.

Nov 17 22:27:59 hades snmpd[1066]: payload OID: prNames
Nov 17 22:27:59 hades snmpd[1066]: /etc/snmp/snmpd.conf: line 143: Error: unknown payload OID
Nov 17 22:27:59 hades snmpd[1066]: Unknown payload OID: prNames
Nov 17 22:27:59 hades snmpd[1066]: /etc/snmp/snmpd.conf: line 143: Error: Unknown payload OID
Nov 17 22:27:59 hades snmpd[1066]: payload OID: prErrMessage
Nov 17 22:27:59 hades snmpd[1066]: /etc/snmp/snmpd.conf: line 143: Error: unknown payload OID
Nov 17 22:27:59 hades snmpd[1066]: Unknown payload OID: prErrMessage
Nov 17 22:27:59 hades snmpd[1066]: /etc/snmp/snmpd.conf: line 143: Error: Unknown payload OID
Nov 17 22:27:59 hades snmpd[1066]: trigger OID: prErrorFlag
Nov 17 22:27:59 hades snmpd[1066]: /etc/snmp/snmpd.conf: line 143: Error: unknown monitor OID
...
Nov 17 22:27:59 hades snmpd[1066]: Turning on AgentX master support.
Nov 17 22:27:59 hades snmpd[1066]: net-snmp: 33 error(s) in config file(s)

After some searching I was able to get it running by setting MIBS to UCD-SNMP-MIB in /etc/default/snmpd.

In order to get SNMPv3 running only some minor adjustments were needed. I ended up with the following settings (which I hope are secure :-), see last two references (in Dutch)):

/etc/default/snmpd

# This file controls the activity of snmpd and snmptrapd
 
# Don't load any MIBs by default.
# You might comment this lines once you have the MIBs downloaded.
export MIBS=UCD-SNMP-MIB
 
# snmpd control (yes means start daemon).
SNMPDRUN=yes
 
# snmpd options (use syslog, close stdin/out/err).
SNMPDOPTS='-LS4d -Lf /dev/null -u snmp -g snmp -I -smux -p /var/run/snmpd.pid'
 
# snmptrapd control (yes means start daemon).  As of net-snmp version
# 5.0, master agentx support must be enabled in snmpd before snmptrapd
# can be run.  See snmpd.conf(5) for how to do this.
TRAPDRUN=no
 
# snmptrapd options (use syslog).
TRAPDOPTS='-Lsd -p /var/run/snmptrapd.pid'
 
# create symlink on Debian legacy location to official RFC path
SNMPDCOMPAT=yes

/etc/snmp/snmpd.conf

agentAddress udp:161,udp6:[::1]:161
 
createUser internalUser  MD5 "mypassword3"
createUser cacti SHA "mypassword4" AES "passphrase4"
 
view   systemonly  included   .1.3.6.1.2.1.1
view   systemonly  included   .1.3.6.1.2.1.25.1
 
rouser   cacti
 
iquerySecName   internalUser
rouser          internalUser
 
defaultMonitors          yes
linkUpDownNotifications  yes
 
master          agentx

References:

2012
08.18

While applying some (550) changesets I needed some good tools to show me the differences between files and revisions.

The first one I found very useful was:

svn diff --diff-cmd diff -x -uw -r REV1:REV2 FILE;

which is even better in combination with colordiff:

apt-get install colordiff;
svn diff --diff-cmd diff -x -uw -r REV1:REV2 FILE | colordiff;

An other great tool is vimdiff, which starts Vim as usual, and additionally sets it up for viewing the differences between files. Here’s how I use it:

sudo apt-get install vim;
vimdiff -O -c 'set diffopt+=iwhite,filler' FILE1 FILE2;

Some of you will probably prefer Meld, which is a visual diff and merge tool.

To combine the powers of all three I created a simple (Bash) wrapper. It downloads two revisions (svn) of a file and compares them using a diff tool of choice:

#!/bin/bash
#
# set -x;
#
if [ -z "${1}" ] || [ -z "${2}" ] || [ -z "${3}" ]; then
 
  echo "";
  echo "Use the script as discribed below:";
  echo "$0 [ filename ] [ revision1 ] [ revision2 ] [ diff tool (optionally) ]";
  echo "";
 
else
 
  pid=$$;
 
  filename="${1}";
  revision1="${2}";
  revision2="${3}";
  diff_command="${4}";
 
  temp1="/tmp/${pid}.${revision1}.${filename}";
  temp2="/tmp/${pid}.${revision2}.${filename}";
 
  trap trap_function 0 1 2 15;
 
  trap_function()
  {
    if [ -f "${temp1}" ]; then
 
      rm "${temp1}";
 
    fi
 
    if [ -f "${temp2}" ]; then
 
      rm "${temp2}";
 
    fi
  }
 
  svn cat "${filename}" -r "${revision1}" > "${temp1}";
  svn cat "${filename}" -r "${revision2}" > "${temp2}";
 
  if [ "${diff_command}" == "vimdiff" ]; then
 
    vimdiff -O -c 'set diffopt+=iwhite,filler' "${temp1}" "${temp2}";
 
  elif [ "${diff_command}" == "meld" ]; then
 
    meld --diff "${temp1}" "${temp2}";
 
  else
 
    diff -uw "${temp1}" "${temp2}" | colordiff;
 
  fi
 
fi
2012
08.18

Recently I needed to install PHPUnit to write some test for an application I was working on. Here I will describe how I installed it on Ubuntu.

First of all, you’ll need to install PEAR, which provides a distribution system for PHP packages, and is shipped with every release of PHP since version 4.3.0.

apt-get install php5-dev php-pear

Subsequently you’ll install PHPUnit through the PEAR installer. To do that run the following:

pear upgrade PEAR
pear config-set auto_discover 1
pear install pear.phpunit.de/PHPUnit

After the installation you can find the PHPUnit source files inside your local PEAR directory; the path is usually /usr/lib/php/PHPUnit.