Debian Wheezy ships with Dovecot 2.x which has a different config layout to the 1.x verion in Lenny and Squeeze. In response, I've created a wheezy branch of postfixadmin-installer (there's an issue for it, too) which configures Dovecot 2.x and it's actually been a really easy switch.

In much the same way as the current version generally does away with the heavily commented documentation masquerading as a config file, this one simply moves /etc/dovecot out of the way and writes two files into it - dovecot.conf and dovecot-sql.conf (which are the same as for 1.x). This causes a pretty hilarious reduction in filesize, too:

[email protected]:~# find /etc/dovecot/ -type f -exec cat {} \; | wc -l
48
[email protected]:~# find /etc/dovecot_2013-01-29/ -type f -exec cat {} \; | wc -l
1772
[email protected]:~#

Anyway, with some incredibly limited testing, and assuming you have already installed dovecot, this seems to work. If you want to test it (please!), enable Wheezy backports in Squeeze and then:

apt-get install libwww-perl mysql-server postfix
apt-get -t squeeze-backports install dovecot-common dovecot-imapd dovecot-pop3d
perl ./postfixadmin-installer

And, finally, here's that working config I'm using, in case that's what you're after:
/etc/dovecot/dovecot.conf

protocols = imap pop3
log_timestamp = "%Y-%m-%d %H:%M:%S "
mail_location = maildir:/var/vmail/%d/%n
mail_privileged_group = vmail
# This should match that of the owner of the /var/lib/vmail hierarchy, and
# be the same as the one postfix uses.
first_valid_uid = 999
# Allow people to use plaintext auth even when TLS/SSL is available (you
# might not want this but it is handy when testing):
disable_plaintext_auth = no
# Uncomment this to get nice and verbose messages about authentication
# problems:
# auth_debug=yes

ssl = no

protocol imap {
}

protocol pop3 {
pop3_uidl_format = %08Xu%08Xv
}

# 'plain' here doesn't override the disble_plaintext_auth_default of 'yes'.
# you should add any other auth mechanisms you want
#auth_mechanisms = plain
userdb {
driver = sql
args = /etc/dovecot/dovecot-sql.conf
}
passdb {
driver = sql
args = /etc/dovecot/dovecot-sql.conf
}

service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0660
# yes, 'postfix' (or the user that owns the above socket file), not vmail
user = postfix
group = postfix
}
}

/etc/dovecot/dovecot-sql.conf

connect = host=localhost dbname=vmail user=vmail password=1lgI2ehK6aEqytjkeDFT4Z7Pq
driver = mysql
default_pass_scheme = MD5-CRYPT
user_query = SELECT maildir, 999 AS uid, 122 AS gid FROM mailbox WHERE username = '%u' AND active='1'

I've *finally* merged about a billion changes into master in postfixadmin installer, chief amongst them is that most of the boring output now goes to a logfile, the vacation plugin might work after install and it the setup password is randomised. This is all procrastination in order to avoid working out how to configure Dovecot on Wheezy.

It's still a big pile of poor hacks rather than a 'proper' script, but if you just don't look at the source you'll be fine!

# Sitecreator

I've just spent a few days using up spare holiday, which means I've been making things for work that work doesn't want but I do. This time it's sitecreator, a tool for configuring websites and all their dependencies (Unix users, databases, ssh keys, DNS records etc.) on servers.

Since there's so many possible things for the site to rely upon, and I'm not *that* fond of reinventing the wheel, all it really does is generate passwords and call scripts. There's a configuration file that tells it how many passwords to generate, how to work out what the username should be and perhaps to generate a couple of other things (like database names) if needed. Another bit of the config then explains which scripts to call and with which arguments (including these recently-generated passwords and usernames), and at the end it tells you what it thinks it did. I've written a few scripts for it already (mirroring what I want to do with it).

For example, here's a relatively simple config file with some explanation of what's going on, and some output with that configuration:

[email protected]:~sitecreator example.com
MySQL:
database: example

SSH:

MySQL dev:
database: example

[email protected]:~$ And there's at least another example config file in etc/config/. Anyway, hopefully this'll be useful to somebody else who isn't quite into automation enough to have already done this (or to have started using puppet or similar), but does have enough users or systems to configure that some automation would be good. Oh, it's not very tested yet, and I've still not come up with a sane thing to do with the output from the scripts :) # Finding exploited wordpress pages WordPress seems to be hilariously easy to compromise (this might be a bad place to write that) and the general form of an exploit is to inject code like this < ?php$a = base64_decode(YSBsb25nIHN0cmluZyBvZiBiYXNlNjQgdGV4dAo=.......);

right at the top of a script. base64_decode is rarely used by the Good Guys outside of mailers and doing tricks with images, but it's almost never found right at the top of a script. I did write a really convoluted script that found calls to base64_decode and exec and guessed whether they were nefarious (generally, for example, base64_decode is called with a variable (base4_decode($mailBody)), not just a string (base64_decode(dGV4dAo=)) but that just ate all my I/O and didn't really work. So I came up with a much cruder way of doing it. Have a script called ~/bin/base64_in_head #! /bin/bash file=1 headfile | grep base64 2>&1 >/dev/null || exit 1; echo$file
exit 0;


And then run it like this:

$ionice -c3 find /home/user/public_html/ -name \*.php -exec ~/bin/base64_in_head {} \; I've not yet had a situation where that's missed a file that later manual greps have found. # Per-extension logging in MediaWiki This is another of those things that took me rather longer to work out than I would have liked, so hopefully this'll appear in the sorts of searches I should have done. MediaWiki has this nifty feature where you can split the logging for particular extensions out into individual files by doing things like this: $wgDebugLogGroups = array(        'SomeExtension'     => '../logs/wiki_SomeExtension.log',);

What's not made overly clear (well, with hindsight, it is implied by the manual) is that the keys of the hash don't necessarily have anything to do with the name of the extension. I assumed that, in debugging SimpleCaptcha, what I wanted was

 $wgDebugLogGroups = array( 'SimpleCaptcha' => '../logs/wiki_SimpleCaptcha.log',);  But not so! What I actually wanted was $wgDebugLogGroups = array(        'captcha'     => '../logs/wiki_confirmedit.log',);

And, as far as I can find, this isn't documented *anywhere*. For other extensions lacking in documentation so, you can find this out by poking around in the code, and looking for where the extension does this sort of thing:

 function log( <span class="katex math inline">message ) {       wfDebugLog( 'captcha', 'ConfirmEdit: ' .</span>message . '; ' .  $this->trigger );}  That first argument to wfDebugLog is what you want as the key in the hash. Why it can't just use the name of the class invoking it, which is the name used to configure the rest of the extension, I've no idea. # Allowing uploads of arbitrary files in MediaWiki I did RTFM and I did what it said, and still my Mediawiki complained when I tried to upload executable files and things with funny file extensions or mime types. if$wgFileExtensions is empty but $wgEnableUploads = true and$wgStrictFileExtensions = false it should just let me upload anything. I can't think what other behaviour one would expect there, but set like that I can't upload my dodgy files.

So I've removed the code it uses to check.

Here's a pair of diffs if you'd also like to do this. These are on version 1.17.0 but I suspect it's not changed very much.

This just comments out the two blocks of code in UploadBase.php which check whether files are considered safe and warn if they're not - it prevents the checking and the warning:

wiki:/home/wiki/public_html# diff includes/upload/UploadBase.php includes/upload/UploadBase.php.bak447,455c447,454< // ## Avi Commented this out so that we can upload whatever we like to our server. That was nice of him< //            // Check whether the file extension is on the unwanted list< //            global $wgCheckFileExtensions,$wgFileExtensions;< //            if ( $wgCheckFileExtensions ) {< // if ( !$this->checkFileExtension( <span class="katex math inline">this->mFinalExtension,</span>wgFileExtensions ) ) {< //                            $warnings['filetype-unwanted-type'] =$this->mFinalExtension;< //                    }< //            }< //--->               // Check whether the file extension is on the unwanted list>               global <span class="katex math inline">wgCheckFileExtensions,</span>wgFileExtensions;>               if ( <span class="katex math inline">wgCheckFileExtensions ) {>                       if ( !</span>this->checkFileExtension( <span class="katex math inline">this->mFinalExtension,</span>wgFileExtensions ) ) {>                               <span class="katex math inline">warnings['filetype-unwanted-type'] =</span>this->mFinalExtension;>                       }>               }> 557,570c556,569< // ## Avi Commented this out so that we can upload whatever we like to our server. That was nice of him< //            /* Don't allow users to override the blacklist (check file extension) */< //            global $wgCheckFileExtensions,$wgStrictFileExtensions;< //            global $wgFileExtensions,$wgFileBlacklist;< //            if ( $this->mFinalExtension == '' ) {< //$this->mTitleError = self::FILETYPE_MISSING;< //                    return $this->mTitle = null;< // } elseif ($this->checkFileExtensionList( <span class="katex math inline">ext,</span>wgFileBlacklist ) ||< //                            ( $wgCheckFileExtensions &&$wgStrictFileExtensions &&< //                                    !$this->checkFileExtension( <span class="katex math inline">this->mFinalExtension,</span>wgFileExtensions ) ) ) {< //$this->mTitleError = self::FILETYPE_BADTYPE;< //                    return $this->mTitle = null;< // }< //---> > /* Don't allow users to override the blacklist (check file extension) */> global <span class="katex math inline">wgCheckFileExtensions,</span>wgStrictFileExtensions;> global <span class="katex math inline">wgFileExtensions,</span>wgFileBlacklist;> if ( <span class="katex math inline">this->mFinalExtension == '' ) {></span>this->mTitleError = self::FILETYPE_MISSING;> return <span class="katex math inline">this->mTitle = null;> } elseif (</span>this->checkFileExtensionList( <span class="katex math inline">ext,</span>wgFileBlacklist ) ||> ( <span class="katex math inline">wgCheckFileExtensions &&</span>wgStrictFileExtensions &&> !<span class="katex math inline">this->checkFileExtension(</span>this->mFinalExtension, <span class="katex math inline">wgFileExtensions ) ) ) {></span>this->mTitleError = self::FILETYPE_BADTYPE;> return$this->mTitle = null;>               }>

And this just stops Setup.php making-safe the $wgFileExtensions array by removing whatever's in$wgFileBlacklist from it, which I think wouldn't complain had I not already done Bad Things to those two variables, but it's late and it can't hurt to turn this off, too:

wiki:/home/wiki/public_html# diff includes/Setup.php includes/Setup.php.bak296,298c296,297< // ## Avi Commented this out so we can upload whatever we like to our server. That was nice of him< //# Blacklisted file extensions shouldn't appear on the "allowed" list< //$wgFileExtensions = array_diff ($wgFileExtensions, $wgFileBlacklist );---> # Blacklisted file extensions shouldn't appear on the "allowed" list> <span class="katex math inline">wgFileExtensions = array_diff (</span>wgFileExtensions,$wgFileBlacklist );

# Tagging images by path in Shotwell

I've finally decided to use an image manager, and since it comes with Ubuntu this week I've gone with Shotwell. I've got a directory hierarchy containing most of my images which is sort-of sorted already, and I'm probably going to keep adding to it, if for no other reason than force of habit.

I know that one of the wonderful features of these photo managers is that you can tag photos, and obviously a photo can be in more than one tag rather more easily than it can be in several directories. That said, all photo managers seem to have decided that an easy, fast way to tag photos isn't what's needed.

Additionally, shotwell's got this weird thing for hiding the fact that there's a filesystem from you, and I can't find any way to tag files by directory. So I've poked around the database and written a script to do it, which is below and here and pasted below in case I change my mind about file hierarchies later.

The oddest bit is the way the filenames are linked to the tags. The TagTable table has a field photo_id_list which contains a list of photo IDs in a format that I've not found anywhere else in the (admittedly not very extensive) db.

They're created by taking the id of the image (its value in the id field of the PhotoTable table), converting it to a hex value, padding it out to 16 characters with leading zeroes, and then concatenating it onto the string 'thumb':

my <span class="katex math inline">hexPhotoId = sprintf("%x",</span>photoId);my <span class="katex math inline">thumbString = "thumb".sprintf('%016s',</span>hexPhotoId);

Anyway, the script's a bit simple because bash is quite good at handling loads of files; usage is like this to tag the contents of ~/Pictures/2011-france/ with the tag 'morzine':

[email protected]:~$find ~/Pictures/2011-france/ -type f -exec ./shotwell-tag {} morzine \; Creating tag morzine tagged /home/avi/Pictures/2011-france/R0012810.JPG with morzine tagged /home/avi/Pictures/2011-france/R0012850.JPG with morzine tagged /home/avi/Pictures/2011-france/R0012911.JPG with morzine tagged /home/avi/Pictures/2011-france/R0012931.JPG with morzine tagged /home/avi/Pictures/2011-france/R0012921.JPG with morzine tagged /home/avi/Pictures/2011-france/R0012794.JPG with morzine tagged /home/avi/Pictures/2011-france/R0012883.JPG with morzine tagged /home/avi/Pictures/2011-france/R0012881.JPG with morzine  I've no idea if it breaks anything - I wrote it about an hour ago, have tagged ~500 photos with it since, and Shotwell doesn't seem to be annoyed. YMMV. Here's the script:  #! /usr/bin/perl # shotwell-tag## Tags files specified by filename in shotwell. Handy for # getting round shotwell's attempts at hiding the filesystem.# # Avi 2011 use strict;use DBI; my <span class="katex math inline">file = shift;my</span>tag = shift; if (<span class="katex math inline">tag !~ /.+/){ print "Usage:\n\n\tshotwell-tag [file] [tag]\n\n"; print "Tags [file] with [tag] in shotwell's db\n"; exit 1;} my</span>dbfile = <span class="katex math inline">ENV{'HOME'}."/.shotwell/data/photo.db";my</span>dbh = DBI->connect("dbi:SQLite:dbname=<span class="katex math inline">dbfile","",""); # Each tag has a string of photo 'ids'. These are generated# by taking the ID of the photo from PhotoTable, representing# it in hex, padding that out to 16 characters with leading # zeroes and then appending it to the string 'thumb'my</span>sth = <span class="katex math inline">dbh->prepare("select id from PhotoTable where filename='</span>file'");<span class="katex math inline">sth->execute();my</span>row = <span class="katex math inline">sth->fetch;my</span>photoId = <span class="katex math inline">row->[0];unless(</span>photoId =~ /\d+/){print "<span class="katex math inline">file is not in shotwell library\n"; exit 0;}my</span>hexPhotoId = sprintf("%x", <span class="katex math inline">photoId);my</span>thumbString = "thumb".sprintf('%016s', <span class="katex math inline">hexPhotoId);</span>sth = <span class="katex math inline">dbh->prepare("select id from TagTable where name='</span>tag'");<span class="katex math inline">sth->execute();</span>row = <span class="katex math inline">sth->fetch;my</span>tagId = <span class="katex math inline">row->[0];unless(</span>tagId =~ /\d+/){ print "Creating tag <span class="katex math inline">tag\n"; my</span>sth = <span class="katex math inline">dbh->prepare("insert into TagTable (name) values('</span>tag')"); <span class="katex math inline">sth->execute;}</span>sth = <span class="katex math inline">dbh->prepare("Select photo_id_list from TagTable where name='</span>tag'");<span class="katex math inline">sth->execute();</span>row = <span class="katex math inline">sth->fetch;my</span>photoList = <span class="katex math inline">row->[0];if(</span>photoList !~ /,<span class="katex math inline">/ &&</span>photoList =~ /._/){ <span class="katex math inline">photoList.=',';}if(</span>photoList =~ /<span class="katex math inline">thumbString/){ print "</span>file is already tagged with <span class="katex math inline">tag\n"; exit 0;}else{</span>photoList.=<span class="katex math inline">thumbString.',';</span>sth = <span class="katex math inline">dbh->prepare("update TagTable set photo_id_list = '</span>photoList' where name='<span class="katex math inline">tag'");</span>sth->execute; print "tagged <span class="katex math inline">file with</span>tag\n"; exit 0;}  # Splitting massive MySQL dumps As I posted yesterday, I have a massive MySQL dump to import. I tried BigDump, but one of the tables kept producing errors and so BigDump would exit. I don't need the whole db imported, so I wrote this to split it by table. It produces a new sql file for every table it finds, numbered sequentially so if you process them in alphabetical order it's the equivalent of the whole dump. USE statements get their own files in the same sequence. #! /usr/bin/perl use strict;use warnings;use 5.010; my <span class="katex math inline">dump_file =</span>ARGV[0];&usage() if !<span class="katex math inline">dump_file; say "using ".</span>dump_file; my (<span class="katex math inline">line,</span>table,@query, <span class="katex math inline">file_number,</span>file_name);my <span class="katex math inline">line_number = 1;my</span>find_count = 0; open(DUMP_IN, "<$dump_file");        while(<DUMP_IN>){                my <span class="katex math inline">line =</span>_;                if (/^USE\s.(\w+)./){                        say "changing db: ".<span class="katex math inline">1;</span>file_name = &make_file_name("USE_<span class="katex math inline">1", "</span>find_count");                        &write_USE(<span class="katex math inline">file_name,</span>line);                        <span class="katex math inline">find_count++;                }elsif (/^-- Table structure for table .(.+)./){			## If the current line is the beginning of a table definition			## and @query is defined, then @query must be full of the previous			## table, so we want to process it now:                        if (@query){</span>file_name = &make_file_name("<span class="katex math inline">table", "</span>find_count");                                open(OUTPUT, "><span class="katex math inline">file_name");                                        foreach(@query){                                                print OUTPUT</span>_;                                        }                                close OUTPUT;                                undef @query;                        }                        <span class="katex math inline">table =</span>1;                        <span class="katex math inline">find_count++;                }                next unless</span>table;                push @query, <span class="katex math inline">line;</span>line_number++;        }close DUMP_IN;say <span class="katex math inline">line_number; ## Subroutines!sub write_USE() {        my(</span>filename, <span class="katex math inline">line) = @_[0,1];        open (OUTPUT, "></span>filename");        print OUTPUT <span class="katex math inline">line;        close OUTPUT;} sub make_file_name() {        my (</span>type, <span class="katex math inline">number) = @_[0,1];</span>number = sprintf("%05d", <span class="katex math inline">number);</span>file_name=<span class="katex math inline">number."_".</span>type.".sql";        return <span class="katex math inline">file_name;} sub usage() {        say "Error: missing arguments.";	say "Usage:";	say "</span>0 [MYSQL_DUMP]";        exit 1;}

A small downside is that this replaces my 2.5Gb file with about 1800 smaller ones. A scripted importer is to follow.

# Whoo! Theme update!

I've updated the theme, and applied my handy modifications that make it more grey. Here's the diff on the css file, if you're wondering what I did (and for next time when I forget). I despise CSS, so it's all nice and easy.

The changes are to give the <pre> tags a grey (#EEE) background colour, and to make the posts individual white boxes on grey, rather than an all-white page.