Massive dumps with MySQL

hurr. *insert FLUSH TABLES joke here*

I have a 2.5GB sql dump to import to my MySQL server. MySQL doesn't like me giving it work to do, and the box it's running on only has 3GB of memory. So, I stumbled across bigdump, which is brilliant. It's a PHP script that splits massive SQL dumps into smaller statements, and runs them one at a time against the server. Always the way: 10 lines into duct-taping together something to do the job for you, you find that someone else has done it rather elegantly.1

In short, we extract the directory to a publicly http-accessible location, stick the sql dump there and tell it to go.

In long, installation is approximately as follows:

  1. avi@jup-linux2:~$ cd www
  2. avi@jup-linux2:~/www$ mkdir bigdump
  3. avi@jup-linux2:~/www$ chmod 777 bigdump
  4. avi@jup-linux2:~/www$ cd bigdump/
  5. avi@jup-linux2:~/www$ wget -q
  6. avi@jup-linux2:~/www$ unzip
  7. avi@jup-linux2:~/www/bigdump$ ls
  8. bigdump.php

Where ~/www is my apache UserDir (i.e. when I visit http://localhost/~avi, i see the contents of ~/www). We need permissions to execute PHP scripts in this dir, too (which I have already). We also need to give everyone permissions to do everything - don't do this on the internet!2

Configuration involves editing bigdump.php with the hostname of our MySQL server, the name of the DB we want to manipulate and our credentials. The following is lines 40-45 of mine:

  1. // Database configuration
  3. $db_server = 'localhost';
  4. $db_name = 'KBDB';
  5. $db_username = 'kbox';
  6. $db_password = 'imnottellingyou';

Finally, we need to give it a dump to process. For dumps of less than 2Mb3, we can upload through the web browser, else we need to upload or link our sql dump to the same directory as bigdump:

  1. avi@jup-linux2:~/www/bigdump$ ln -s /home/avi/kbox/kbox_dbdata ./dump.sql

Now, we visit the php page through a web browser, and get a pretty interface:

BigDump lists all the files in its working directory, and for any that are SQL dumps provides a 'Start Import' link. To import one of them, click the link and wait.

  1. Yes, you Perl people, it's in PHP. But it's not written by me. So on balance turns out more elegant. []
  2. Those permissions aside - anyone can execute whatever SQL they like with your credentials through this page. Seriously, not on the internet! []
  3. Or whatever's smaller out of upload_max_filesize and post_max_size in your php.ini []

Whoo! Theme update!

I've updated the theme, and applied my handy modifications that make it more grey. Here's the diff on the css file, if you're wondering what I did (and for next time when I forget). I despise CSS, so it's all nice and easy.

The changes are to give the <pre> tags a grey (#EEE) background colour, and to make the posts individual white boxes on grey, rather than an all-white page.

  2. avi@avi:whiteasmilk_1.8$ diff style.css style.css.original
  3. 18,35d17
  4. < /* Added to make the code blocks pretty. Stupid CSS implementations mean this will break
  5. < any form of validity in order to actually make it work. Stupid browser people.
  6. < I can't remember where I got this from, but if you reckon I might've found it on your site
  7. < let me know and if I believe you I'll stick a URL here
  8. < */
  9. < pre {
  10. < border 0
  11. < padding: 0.2em 0.5em;
  12. < background-color:#EEE;
  13. < white-space:pre-wrap;
  14. < white-space: -moz-pre-wrap; /* Mozilla, since 1999 */
  15. < white-space: -pre-wrap; /* Opera 4-6 */
  16. < white-space: -o-pre-wrap; /* Opera 7 */
  17. < word-wrap: break-word; /* Internet Explorer 5.5+ */
  18. < }
  19. <
  20. <
  21. <
  22. 40c22
  23. < body {background-color:#c9c9c9;}
  24. ---
  25. > body {background-color:white;}
  26. 323,326c305
  27. < border-bottom:15px solid #c9c9c9;
  28. < padding-left:10px;
  29. < padding-right:10px;
  30. < background-color:#fff;
  31. ---
  32. > border-bottom:1px solid #999;

Generating Fluxbox menus for VNC (Vinagre) connections

One of the lovely things about Fluxbox is the text-driven menu. One of the nice things about Vinagre (Gnome's VNC client) is the xml-based bookmarks file. Here's a handy script to create a Fluxbox submenu out of your Vinagre bookmarks:

  2. #! /usr/bin/perl
  4. use strict;
  5. use warnings;
  6. use XML::Simple;
  7. my $HOME = $ENV{ HOME };
  9. my $bookmarks_file = "$HOME/.local/share/vinagre/vinagre-bookmarks.xml";
  10. my $menu_file = "$HOME/.fluxbox/vnc_menu";
  12. my $xml = new XML::Simple (KeyAttr=>[]);
  13. my $data = $xml->XMLin("$bookmarks_file");
  15. open(MENU, ">$menu_file") || die "Error opening \$menu_file: $menu_file $0";
  17. print MENU "[begin]\n";
  19. foreach my $b(@{$data->{"item"}}){
  20. print MENU "[exec] ($b->{name}) {vinagre $b->{host}:$b->{port}}\n";
  21. }
  22. print MENU "[end]\n";
  23. close MENU;

Dell Warranty Info

I hate navigating the Dell website. It's inconsistent and messy and noisy, and all I generally want is a single date (when the warranty expires or expired on a given box). So I wrote this. It scrapes the Dell website, and returns the warranty info for the service tag it's been passed.
I've CGI'd it here.

  1. #! /usr/bin/perl
  3. use strict;
  4. use warnings;
  6. die "$0\n\tGet warranty info from dell.\nUsage\n$0 [SERVICE TAG]\n" if !$ARGV[0];
  8. my $service_tag = $ARGV[0];
  10. use LWP::Simple;
  11. use HTML::TableExtract; # Is in the CPAN, and exists in the debian repositories as libhtml-tableextract-perl
  13. ## Make a URL:
  14. my $url_base = "";
  15. my $url_params = "?c=uk&cs=ukbsdt1&l=en&s=gen";
  16. my $url = $url_base.$url_params."&servicetag=".$service_tag;
  17. my $content = get($url);
  19. # Tell HTML::TableExtract to pick out the table(s) whose class is 'contract_table':
  20. my $table = HTML::TableExtract->new( attribs => { class => "contract_table" } );
  21. $table->parse($content);
  23. ## Gimme infos!
  24. foreach my $ts ($table->tables) {
  25. foreach my $row ($ts->rows) {
  26. print "", join("\t", @$row), "\n";
  27. }
  28. }

Getopt in Perl

Oddly, it's taken me until this afternoon to have real need for using getopts in Perl. After a not-overly-brief look around, I've settled on Getopt::Long for the purpose. It's marginally more complicated than the alternative (Getopt::Std), but more flexible and better at error checking.

To use it, you pass a hash of valid options to GetOptions, where the keys are options and the values are the references to variables in which to put their arguments.
The name of the option dictates what value(s) it can hold: the final character indicates type (i - integer, f - float, s - string), and the penultimate whether it is optional or not (= - required, : - optional). Flags are indicated by not following this pattern - they're just given a name with no symbols.

Getopt::Long allows for the shortest unambiguous switch to be used, doesn't distinguish between -o and --o, and allows for the negation of flags (if -flag sets a flag to 1, -noflag will set it to 0). It also doesn't touch @ARGV when it's done getting its flags out of it.

Here's a brief script hopefully helping explain the above:

  1. #!/usr/bin/perl
  2. use strict;
  3. use warnings;
  4. use Getopt::Long;
  6. # This is only neccesary when using strict. Which is always.
  7. my ($flag, $compulsory_string, $optional_string, $compulsory_integer, $optional_integer, $compulsory_
  9. GetOptions(
  10. "o"=>\$flag,
  11. "cfloat=f"=>\$compulsory_float,
  12. "cint=i"=>\$compulsory_integer,
  13. "cstring=s"=>\$compulsory_string,
  14. "ofloat:f"=>\$optional_float,
  15. "oint:i"=>\$optional_integer,
  16. "ostring:s"=>\$optional_string,
  17. );
  19. print "flag set\n" if $flag;
  20. print $compulsory_float."\n" if $compulsory_float;
  21. print $compulsory_integer."\n" if $compulsory_integer;
  22. print $compulsory_string."\n" if $compulsory_string;
  23. print $optional_float."\n" if $optional_float;
  24. print $optional_integer."\n" if $optional_integer;
  25. print $optional_string."\n" if $optional_string;

Giving Android a swap file

I don't know if it's because I'm doing more with it than I used to, or the rose-tinted specs that come with the novelty value have worn off, or if the later updates have been designed for more powerful hardware, but my G1's been lagging a bit recently, so I figured I'd have a look at what I can do to make it faster. Turns out it's quite receptive to the idea of a swap file.
You'll need root access for this - everything below is to be run as root. Following is how I did it on my cyanogen'd G1, but I don't see why it wouldn't work on anything else - it makes use of standard linux tools. This will likely advance the death of your sdcard - it increases the reads and writes by some large margin. I don't know enough about sdcards to know how much quicker it will die, but modern ones will likely deal with it better than older ones.

Anyway, in your terminal emulator:

  1. dd if=/dev/zero of=/sdcard/swap bs=1M count=100

You can call it whatever you like. The 'count' value (100 above) is how many MB of swap space you want. The above will create 100MB of swap.
This will take a few seconds (of the order of 50), so if it's 'hung' it's probably not crashed.

Then we make it into a swap file and activate it as swap space:

  1. mkswap /sdcard/swap
  2. swapon /sdcard/swap

Finally, we want to change the swappiness of the system. When the system memory is completely full, and more is requested, the kernel needs to work out how to create some space. It, generally, can do one of two things: drop some filesystem cache, or move some application memory to swap.
The swappiness value dictates the priority of either - a value of 0 means it will always try to drop fs cache rather than use swap, and a value of 100 means it will always try to use swap and preserve the fs cache. Only values between 0 and 100 inclusive are allowed, and neither guarantees any action - you can still find yourself swapping out at 0, for example - just the priority afforded each solution.
The default value for the Linux kernel is 60, which is generally not appropriate for something running completely from flash media (where the fs cache isn't so important). So I'm dropping it to 20, which seems to work for me. If it doesn't for you, try some other values, it's an on-the-fly change.

  1. echo 20 > /proc/sys/vm/swappiness

On rebooting the phone, the above settings wont hold. The swap file will still be available (assuming the SD card is still inserted), but on reboot you'll need to run

  1. swapon /sdcard/swap
  2. echo 20 > /proc/sys/vm/swappiness

On the plus side, you'll be rebooting less.

Cyanogen on my G1

I've just upgraded to Cyanogen on my G1 and it's lovely. Well, I got root, which is basically what I always wanted.

I basically followed the instructions on the Cyanogen wiki and everything worked exactly as described, I've nothing really to add here except to say it's brilliantly easy and everyone should do it.
One small note, though, the final reboot into Cyanogen takes a long time. Mine took just shy of thirty minutes, and I've read of others taking anything above about 15.

The first obvious benefits are root access, exchange activesync support, five workspaces (as against the stock 3) and a bunch of useful apps already installed, including a nifty power control widget which replaces my collection of 5 distinct widgets. I'll write more on it when I've used it more.
In case, as I did, you've been pondering this and wondering what the end result is like, it's almost exactly as previously, but with some new features, there's some screenshots on the webpage. I've not yet found anything to have been degraded by it, and it all looks and feels the same, but a bit more polished in areas. Though it only ships with one ringtone.

PHP error on fresh install of PHPWiki:
Non-static method _PearDbPassUser::_PearDbPassUser() cannot be called statically

I've just installed PHPWiki 1.3.14-4 from the debian repositories and out of the box I got the following message on trying to log in to it:

Fatal error: Non-static method _PearDbPassUser::_PearDbPassUser() cannot be called statically, assuming $this from incompatible context in /usr/share/phpwiki.bak.d/lib/WikiUserNew.php on line 1118

The problem appears to be that, as of PHP 5.something, you're not allowed to have a function with the same name as a class. Apparently it's been a failure in E_STRICT mode for a while.

Anyway, the solution is to rename _PearDbPassUser() to something else, and then replace all calls to it with this new name.

I've done this and, so far, everything appears to work.

The function is defined in /usr/share/lib/phpwiki/lib/WikiUser/PearDb.php:

  2. jup-linux2:/usr/share$ diff phpwiki.bak.d/lib/WikiUser/PearDb.php phpwiki/lib/WikiUser/PearDb.php
  3. 18c18
  4. < function _PearDbPassUser($UserName='',$prefs=false) {
  5. ---
  6. > function _PearDbPassUserFoo($UserName='',$prefs=false) {

and is called in /usr/share/WikiUserNew.php:

  2. jup-linux2:/usr/share$ diff phpwiki.bak.d/lib/WikiUserNew.php phpwiki/lib/WikiUserNew.php
  3. 1118c1118
  4. < _PearDbPassUser::_PearDbPassUser($this->_userid, $this->_prefs);
  5. ---
  6. > _PearDbPassUser::_PearDbPassUserFoo($this->_userid, $this->_prefs);
  7. 1157c1157
  8. < _PearDbPassUser::_PearDbPassUser($this->_userid, $prefs);
  9. ---
  10. > _PearDbPassUser::_PearDbPassUserFoo($this->_userid, $prefs);
  11. 2120c2120
  12. < function PearDbUserPreferences ($saved_prefs = false) {
  13. ---
  14. > function PearDbUserPreferencesFoo ($saved_prefs = false) {

Munin plugins are really easy to write

Munin plugins basically need to output variable names and values, and a little bit of config. They're tremendously easy to write.

My plugin is mostly useless - it graphs the value returned by /dev/urandom, and the random constants from debian and dilbert. Current graph is here and the code is as follows:

  1. #! /bin/bash
  3. case $1 in
  4. config)
  5. cat < < EOF
  6. graph_category amusement
  7. graph_title Random numbers
  8. graph_vlabel value
  9. debian.label debian
  10. dilbert.label dilbert
  11. urandom.label /dev/urandom
  12. graph_scale no
  13. EOF
  14. exit 0
  15. esac
  17. urandom=$(cat /dev/urandom | tr -dc '0-9' | head -c2)
  19. echo "urandom.value " $urandom
  20. echo "debian.value 4"
  21. echo "dilbert.value 9"

Munin's plugins live in /etc/munin/plugins/, most of which are symlinks to scripts in /usr/share/plugins/. On restart, munin-node rechecks the plugins directory and loads any new plugins.
For a plugin called foo, munin-node will run foo configure first to get the configuration of the graph (which is passed to munin-graph), and then foo. For information as to graph configuration, see here.
It takes about 15 mins of collection for it to start making a graph, and you'll get more data every 5mins thereafter.

The script itself is mostly self-explanatory, except for:

- The values and the labels are linked by what occurs before the dot. If you define foo.label in the config output, that is what will be used to label the number that comes after foo.value in the 'normal' output. The munin tutorial sort-of hints at this, but only uses one variable.

- Munin doesn't care what order the variables come out in, it uses the labels to determine who's who. Similarly, it doesn't seem particularly fussed as to which flavour of horizontal whitespace is used.

Android G1 factory reset

With the phone off and the battery in, press and hold <power>+<back> for 20 seconds or so. You'll get the standard G1 splash screen, then a screen with an exclamation mark.

Press <alt>+<L> to get to the System Recovery screen. Here you're presented with four options:

reboot system now [Home+Back]
apply [Alt+S]
wipe data/factory reset [Alt+W]
wipe cache partition

Which you should be able to choose from with the scrollball or use the key combinations for.