Drupal themeing is addictive. I’ve been so busy mucking with the new layout for one of my sites that I still haven’t gotten all of the content moved over, categorized, etc. I was really itching to fix “one last thing” on my site, and that was the front page.

My new layout uses the litejazz theme from RoopleThemes (though this tip should work with any theme, with only minor modification), which is super, super nice. It has three different color schemes, a million or so user regions to use, nice block formatting, lots of options and features… the works (thanks very much to the Roople folks for their work, btw).

Anyway, one thing I haven’t seen *any* themes do is provide an option to turn *off* the main content area on the home page (or any other page, for that matter). I wanted my home page to consist only of snapshots of various areas or aspects of the site, and the main navigation. I’m not doing some big news site or anything, so everything I’m doing fits into blocks. I don’t need the main content area. Turns out there’s a dead-simple way to get rid of it.

In your theme folder, whatever theme it is, there’s a file called page.tpl.php. Drupal (actually, PHPTemplate, Drupal’s theme engine) applies this page template to all of your site’s pages… unless it’s overriden!

Drupal’s theme system has a naming convention you can use to create page templates for any page or class of pages you want. So, if you want /node/8 to have a special look or feature that doesn’t exist anywhere else on the site, you can make a copy of page.tpl.php called page-node-8.tpl.php, alter the required parts, and /node/8, and only /node/8, will reflect those changes.

The same works for page-front.tpl.php, which is the home page of a drupal site. If you want to remove the main content, you can delete just that portion by removing lines that reference the $content variable. In the case of litejazz, I removed the entire ’squeeze’ div, and now my footer touches the boxes in the top header area. Perfect!

addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F12%2F10%2Fhacking-drupal-6xs-page-or-any-other-page%2F'; addthis_title = 'Hacking+Drupal+6.x%26%238217%3Bs+%3Cfront%3E+page+%28or+any+other+page%29'; addthis_pub = 'jonesy';

I’ve done some theme customization for Drupal, and I’ve used drupal as “just a user” for years. Recently, I took on a project to take a designer’s theme and make it work with Drupal. I’m pleased to say that, minus a couple of little quirks, it’s been relatively simple. Here are a couple of tips for those starting out:

• Use CSS for the main layout. If you must use tables for some small part to get it just right, fine — but the high-level layout should be CSS. There are a million reasons for doing this that have nothing to do with drupal, but in addition, the way drupal manages the pieces that it manages (like menus and blocks created in the admin interface) encourages CSS use.
• Don’t hard code anything that you don’t have to. Use the ‘base_path()’ and ‘path_to_theme()’ functions in page.tpl.php so that if you decide to base another theme off of this one by making a copy of the theme directory, it’ll ‘just work’, instead of having to go through and change every reference to every ‘img src’ in all of your files.
• Get to know Drupal. You might be thinking you have a div that’s being hidden or isn’t showing up at all. In reality, it might be that you have a drupal block that your current login has no permissions to use, or a menu item that doesn’t apply to the current login, in which case it won’t show up in the menu listing. Also, you might have drupal’s caching enabled, which it shouldn’t be if you’re building a theme or doing development in general. To turn off caching, go to Admin->Site Configuration->Performance, disable caching, and then click on the ‘clear cache’ button.
• Also under ‘get to know drupal’: depending on where your images are referenced, you need to use different paths. So, if your images are all in a directory under your theme dir called ‘images’, then images referenced in page.tpl.php need to be relative to the drupal top-level directory (so, ‘/sites/all/themes/<theme>/images/<imgfile>’). But in styles.css, you can use a directory relative to the styles.css file (so, ‘/images/<imgfile>’). I’m not 100% sure what the difference is, but presumably page.tpl.php gets ‘include’d into the main PHPTemplate theme engine code, which is somewhere else under the drupal top-level dir, which is why you need the full path. Correct me or clarify this in the comments if you know for sure.
• Try to make time to learn about the Advanced Theme Creation Kit and/or Zen, which might make theme creation much simpler and faster. I did mine from scratch, and it’s ok, but if I had the time I’d rather learn to make use of these tools.
• Pick up Drupal 6 Themes from Packt Publishing. It’s a great resource for learning this stuff, in spite of the absolutely awful code formatting in places, and some minor-but-annoying errors. Get it anyway. The conceptual parts of the text are fantastic, and the errors aren’t insurmountable. You can download a PDF eBook (with copy/paste enabled!) at the Packt website.
• Use the CSS reference at w3schools.

Some general development practices can’t hurt either. Use some kind of version control for your theme, like SVN, CVS, Mercurial, Bazaar, git, or whatever. This will make it easy to revert to earlier copies of your work if you blow things up, and if you can host the SVN service somewhere that’s not on your development machine, it can also serve as a backup (though you should be backing up your machines anyway).

If you have other tips, leave a comment!

addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F12%2F08%2Fdrupal-6x-themeing-tips%2F'; addthis_title = 'Drupal+6.x+Themeing+Tips'; addthis_pub = 'jonesy';

A Computer Science textbook that uses Python as its core language was released, I believe, this year. However, I CAN’T FIND IT! I searched google, amazon, bn.com, and used all the terms you’d normally think of using, and I can’t find this book anywhere. The book is black, and has a picture of a snake’s head on it. Can anyone help me out with finding this book?

addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F11%2F30%2Fwhats-the-newest-python-comp-sci-textbook%2F'; addthis_title = 'What%26%238217%3Bs+the+newest+Python+Comp.+Sci.+Textbook%3F'; addthis_pub = 'jonesy';

In the few years since Twitter’s launch, they have shrunk the number of ways you can interact with it, shrunk the number of hours in the day when you can reliably get or send messages through it, and, now, shrunk the number of useful web-based services by two, with the announcement that stikkit and I Want Sandy will be shutting down, as a result of their purchase of Values of N, creators of the two sites.

I only just started trying to use twitter again in the past week or so, and my desktop client has recorded at least 3 or 4 outages in the past week. Between that, and now this news, I think I’m just going to give up. What are they planning to do — add Sandy-like features to twitter? Why? To attract more users, insuring that their availability will sink to zero-nines? Blech.

addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F11%2F28%2Fwhere-can-i-go-that-isnt-twitter%2F'; addthis_title = 'Where+Can+I+Go+That+Isn%26%238217%3Bt+Twitter%3F'; addthis_pub = 'jonesy';

I brought my MacBook Pro in for a warranty repair yesterday around noon. Since then I’ve been using a Lenovo T61 to get basic work done, and also to see if any progress has been made in the area of Linux support for my laptop. I bought this laptop specifically because a website said that it was very well supported by Linux distributions “out of the box”, including video and wireless. I was sure to make hardware choices that didn’t require special third-party drivers… I’ve been doing this for 10 years, so I have some understanding of how to buy a laptop that I plan to put Linux on. Well, this time I apparently failed.

First, I had Ubuntu installed, and I was never able to keep the wireless card working consistently. To be honest, Ubuntu is the best distro I’ve had on this thing so far. Next, I gave OpenSUSE 11 a shot, and there’s been no end to the issues. Of course, it started with the wireless card. I have an Intel 3945ABG wireless card, according to lspci and dmesg output. In fact, here’s my lspci output right here:

00:00.0 Host bridge: Intel Corporation Mobile PM965/GM965/GL960 Memory Controller Hub (rev 0c) 00:02.0 VGA compatible controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 0c) 00:02.1 Display controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 0c) 00:19.0 Ethernet controller: Intel Corporation 82566MM Gigabit Network Connection (rev 03) 00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #4 (rev 03) 00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #5 (rev 03) 00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #2 (rev 03) 00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 03) 00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 03) 00:1c.1 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 2 (rev 03) 00:1c.2 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 3 (rev 03) 00:1c.3 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 4 (rev 03) 00:1c.4 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 5 (rev 03) 00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #1 (rev 03) 00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #2 (rev 03) 00:1d.2 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #3 (rev 03) 00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #1 (rev 03) 00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev f3) 00:1f.0 ISA bridge: Intel Corporation 82801HBM (ICH8M-E) LPC Interface Controller (rev 03) 00:1f.1 IDE interface: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) IDE Controller (rev 03) 00:1f.2 SATA controller: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) SATA AHCI Controller (rev 03) 00:1f.3 SMBus: Intel Corporation 82801H (ICH8 Family) SMBus Controller (rev 03) 03:00.0 Network controller: Intel Corporation PRO/Wireless 3945ABG Network Connection (rev 02) 15:00.0 CardBus bridge: Ricoh Co Ltd RL5c476 II (rev ba) 15:00.1 FireWire (IEEE 1394): Ricoh Co Ltd R5C832 IEEE 1394 Controller (rev 04) 15:00.2 SD Host controller: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter (rev 21) 15:00.3 System peripheral: Ricoh Co Ltd R5C843 MMC Host Controller (rev ff) 15:00.4 System peripheral: Ricoh Co Ltd R5C592 Memory Stick Bus Host Adapter (rev 11) 15:00.5 System peripheral: Ricoh Co Ltd xD-Picture Card Controller (rev 11)

I’m running the KDE4 desktop, and tried using the default NetworkManager icon that’s in the systray to get things working. From what I saw there, it appeared that my card wasn’t scanning. I put in my network details manually, and tried to connect, and it failed with no errors. In the NetworkManager log there was lots of output, but nothing particularly useful. It just said the association took to long and that it was now marking that connection as ‘invalid’. Great. So here I am, trying to use Linux on the desktop, and only 5 minutes after the very first system boot, I’m tailing log files and debugging, and basically playing sysadmin, which is exactly what I don’t want to be doing on my desktop system. Restart NetworkManager, see what dhclient is doing, reboot, check /etc/modprobe.d, lsmod…. fail. Now what?

Well, I opened kwifimanager, and it said that I had indeed associated with an access point. So… I *am* scanning? Hmm. I had no IP address, so I figured I had probably fat-fingered my WEP settings somewhere. Tailing /var/log/messages agrees, saying WEP decryption is failing. So I double-check everything, all looks normal and correct to me, I try again, and No Bueno. *sigh*.

Finally, I reverted to command-line tactics, and ran this little line:

iwconfig wlan0 essid <myssid> key <mykey>

Magically, it works, where all of the GUI nonsense had failed. Now here’s a question: how the hell do you get this to “just work” at boot time? Well, I had about 10 emails to send to clients, so I put that question off and fired up a browser and…. fail. WTF?

I had an IP address, pinged my router, pinged another host on the network, all good. Pinged an external IP I know by heart, fail. Ugh. Ran ‘cat /etc/resolv.conf’ — empty. Apparently, dhclient didn’t update the information it got from my router. It also didn’t update when I set the domain in NetworkManager to ‘home’, because it still said ’search site’. I added the proper lines in there, and tried again in the browser… fail. Now what?!?

Ran ‘netstat -rn’. I don’t have a default gateway. *sigh*…

route add default gw 192.168.1.1

And I finally have internet access.

Of course, I can’t work 24 hours a day, so I went to bed, and left my laptop running so I could get right back to work in the morning. Or not.

I had foolishly chosen to use an OpenGL screensaver. Overnight, it completely locked up the machine, rendering it useless without forcibly rebooting it. So much for getting right back to work.

Well, let’s see if I can get some of these issues fixed by updating the software, since I’m now at least connected to the internet (of course, after the forced reboot, I had to do the iwconfig->route add routine again). Ran the updater, picked some extra repositories, and it goes off to set things up. Unfortunately, it also prompts me to import probably 50 or so GPG keys. Annoying. More annoying is, after all of that, it fails to update any of my software, even though it tells me there are updates available. Why you ask? Here’s what I got…

Failed to mount cd:///?devices=/dev/sr0 on /var/adm/mount/AP_0x00000001: No medium found (mount: No medium found)

Click ok. Get same error again. Click ok. Get slightly different error…

Unexpected exception. Failed to mount cd:///?devices=/dev/sr0 on /var/adm/mount/AP_0x00000001: No medium found (mount: No medium found)

Click Ok, get another message…

Please file a bug report about this. See http://en.opensuse.org/Zypper#Troubleshooting for instructions.

I go there, the URL isn’t valid. I find the Troubleshooting page on my own, and there’s a bunch of generic troubleshooting information there. More command line sysadmin-ish stuff in there. Just the kind of stuff I don’t need to be spending otherwise billable time on. I give up and decide that I’ll just deal with it in its broken-ass state for the next 10 hours or so until I can get my beloved MacBook Pro back.

addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F11%2F25%2Flinux-on-laptop-epic-fail%2F'; addthis_title = 'Linux+on+Laptop+%3D+Epic+Fail'; addthis_pub = 'jonesy';

The iPhone’s success is somewhat in spite of itself. Sure, you can access all kinds of applications that do all kinds of things, and they’ve really gotten a few things right, like (IMHO) the phone interface, but some of the things it lacks are starting to become embarrassments.

For example, for what, exactly, is the bluetooth interface intended? Probably 98% of the people who even know what bluetooth is and care that the iPhone has this support want to use it almost exclusively so that they can use their bluetooth headsets. Where I live, the law forbids using a phone while driving unless you have a hands-free capability (a.k.a a bluetooth headset). Unfortunately, the iPhone does *not* meet the need here, because there are NO VOICE COMMANDS! Sure, there are some cars that will take on some of that responsibility, and there are headsets coming out that have the commands built into them, but having this built into the phone is, imo, the right way to do this. I shouldn’t have to be in my car, or need a particular headset to get this feature.

Also, there is no MMS support on the iPhone, and the AT&T hack to work around this limitation might actually be *worse* than just having no solution at all. MMS is used to send and receive photos (among other things, like longer text messages and even video) on your phone. Again, lots of phones have had this for several years now. What happens when someone sends a photo? Well, AT&T stores the photo somewhere, and sends you a text message with a link to a site, and a username and password to use to access the photo. The username and password are both jumbled bunches of characters, and you’re supposed to stare at this text message for an hour until you can commit it to memory, and then click the link and go log in to see the picture. You can’t *do* anything with the picture, mind you — you can look at it, and then you can close your browser. I just tell people who want to send me stuff that the iPhone doesn’t support it, because it basically doesn’t.

There are other annoyances as well, like the inability to read email in landscape view, the crippled map application (ask in the comments if you want more on that one), and the App Store closes just because you initiated an upgrade (forcing you to go back into the store manually if you, say, have more than one app to upgrade), and more!

I bought my iPhone before the 3G came out, but the 3G doesn’t address these issues either. I’ll probably get a new phone in about one year from now, and if these issues still aren’t solved, it is not likely (given the rate that competition is catching up to the other *useful* iPhone features) that I’ll get an iPhone. So if you’re thinking about developing an app for Android, do it!

addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F10%2F20%2Fwhen-will-the-iphone-get-features-other-phones-have-had-for-years%2F'; addthis_title = 'When+will+the+iPhone+get+features+other+phones+have+had+for+years%3F'; addthis_pub = 'jonesy';

So, I recently ran into one of those situations where a customer complains that his MySQL database is slow, and “it worked great until about two weeks ago”. The first thing I look at in these situations is not the queries or any code, but the indexes. Once I determined that the indexes were almost nonsensical, I had a look at the queries themselves, which proved that the indexes were, in fact, nonsensical. Running the queries as written in the code, from a mysql shell, with EXPLAIN, I was able to further confirm that the indexes (most of them, anyway) were never used.

Easy right? Just reindex the table!

NOTE: I’m going to skip the talk about all of the database design issues on this gig and just go straight to solving the immediate problem at hand. Just know that I had nothing to do with the design.

But, supposing this table has 15 million rows and is running on a machine with 2GB of RAM and only a  single (well, mirrored) drive? It will take forever to reindex that table, and the machine can’t be made completely unavailable at any time, either by driving the load up so high that the Linux OOM killer reaps the mysql process, or by putting a lock on the table for, oh, 3 days or so

The solution is to create a brand new table, which I’ll call “good” using the “SHOW CREATE TABLE” output from the bad table, which I’ll call “bad”. I do this right in the shell, actually. I run “SHOW CREATE TABLE bad”, cut and paste the output, remove the part that defines the indexes, rename the table to “good”, and bam! New, empty table.

Of course, you still have to define your new indexes, so run whatever statements are needed to do that. You might even want to populate it with some test data (say, 10000 rows from the bad table) to test your new indexes are being used as expected (or that they can be by altering the query and getting back the same results… but faster).

Once done, it’s time to start copying rows from bad to good. I’ve written a shell script to help with that part. It’s designed to run on a Linux host running MySQL.

The variables at the top of the script are pretty self-explanatory, except to note that there are separate NEWDB and OLDDB variables in case your new table also lives in a new database. The INCREMENT is the number of rows you want to copy over at a time. If you set INCREMENT to 1000, it’ll copy 1000 rows, check the load average, and if it’s under MAXLOAD, it’ll copy over another 1000 rows. It also keeps track of the number of rows in each database as it goes, since writes are still happening on the bad table while this is going on.

So here’s my nibbler script, in shell. I would’ve written it in Python, but it wasn’t my machine, and I couldn’t install the python mysql module :-/

#!/bin/bash ### ### Written by Brian K. Jones () ### ### Takes an increment, old db, and new db, and copies rows from olddb to newdb. ### Along the way, it'll check system load and sleep if it's too high. ### There's too much hard-coding right now, but it's a useful template, and ### has been tested. The script takes no CLI arguments. ### INCR=10000 NEWDB=shiny OLDDB=busted OLDTABLE=bad NEWTABLE=good MAXLOAD=3 DBUSER=mydbuser DBPASS=mydbpass rows_old=`mysql -N -D ${OLDDB} -u ${DBUSER} -p${DBPASS} -e "select count(*) from ${OLDTABLE}"` echo "rows_old is now ${rows_old}" rows_new=`mysql -N -D ${NEWDB} -u ${DBUSER} -p${DBPASS} -e "select count(*) from ${NEWTABLE}"` ## num. rows in new table echo "rows_new is now ${rows_new}" for (( rows_new=$rows_new; rows_new < $rows_old; rows_new+=$INCR )); do if [ $((rows_old - (rows_new + INCR))) -gt 0 ]; then ## Check to see if there are at least $INCR rows left to copy over. mysql -N -D ${NEWDB} -u ${DBUSER} -p${DBPASS} -e "INSERT INTO ${NEWTABLE} SELECT * FROM ${OLDDB}.${OLDTABLE} LIMIT ${rows_new},${INCR}" while [ "`awk -v max=${MAXLOAD} '$1 > max {print "TRUE"}' /proc/loadavg`" = "TRUE" ]; do echo "sleeping due to load > ${MAXLOAD}" sleep 30 done # we update rows_old because it'll be growing while this script runs. rows_old=`mysql -N -D ${OLDDB} -u ${DBUSER} -p${DBPASS} -e "select count(*) from ${OLDTABLE}" ` rows_new=`mysql -N -D ${NEWDB} -u ${DBUSER} -p${DBPASS} -e"select count(*) from ${NEWTABLE}"` time=`date +%R` echo "${time} -- rows_new = ${rows_new}, rows_old = ${rows_old}" else ## There are < $INCR rows left. Select remaining rows. remaining=$((rows_old - rows_new)) mysql -N -D ${NEWDB} -u ${DBUSER} -p${DBPASS} -e "INSERT INTO ${NEWTABLE} SELECT * FROM ${OLDDB}.${OLDTABLE} LIMIT ${rows_new},${remaining}" echo "All done!" exit fi done addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F10%2F09%2Fclone-a-table-in-mysql-without-killing-your-server%2F'; addthis_title = 'Clone+a+table+in+MySQL+without+killing+your+server'; addthis_pub = 'jonesy';

I picked up Practical Django Projects by James Bennett just the other day. The book walks through the building of a few different projects, and tries to lead you down the path of “best practices” along the way. However, the book’s publisher decided to release the first edition of this book on the heels of the Django 1.0 release, and the book is specific to pre-1.0 versions of Django.

If you’re kinda new to Django, you’re likely to be lost. I had only done a couple of really simple tutorial projects before picking up this book, and was really unnerved when I hit a seemingly impossible obstacle on page 24 — the dreaded TinyMCE integration. At that point in the book, it starts to become really obvious that this is not a 1.0-compliant Django book.

If you’re having trouble with this book and you’re using Django 1.0, I can give you two tips that are saving my bacon as I work through these bumps:

1. Go to the porting guide for 1.0, read it, and bookmark it. Better yet, just leave it open in your browser all the time.
2. For the TinyMCE issue, there’s more than one possible issue — but if your development server is giving you 404s after doing a redirect (302) trying to find the tiny_mce.js file, move the relevant line in the URLConf up. The redirect is a hint that your url didn’t match anything.

The porting guide fixes at least 3 issues in the book by page 40.

Also, if you’re a django n00b looking for help, I started a #django-newbie IRC channel on irc.freenode.net. Come join us!

addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F10%2F08%2Fpractical-django-projects-trouble%2F'; addthis_title = 'Practical+Django+Projects+Trouble%3F'; addthis_pub = 'jonesy';

So, I’ve really just had it with the #django chan on freenode. I’ve gone there several times for help only to be treated like whale turd, essentially because I’m a newbie to django I suppose. They seem to have an issue with that there. It’s not that you can’t get answers there, mind you, it’s that the answers are served up cold and begrudgingly with generous side portions of arrogance and self-importance. This isn’t a judgment of every person on the chan — it’s the general take-away I’ve personally had.

However, I really like django

So, rather than just whine and then sit idly by hoping things change, I started a chan for django newbies called, appropriately, #django-newbie on freenode. If you’re a newbie, come join in! If you’re not, come help the newbies find the docs they need to get where they want to go.

addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F10%2F08%2Ftheres-now-a-django-newbie-chan-on-freenode%2F'; addthis_title = 'There%26%238217%3Bs+now+a+%23django-newbie+chan+on+freenode'; addthis_pub = 'jonesy';

First, AddThis.com (where I was the director of IT) and Clearspring have ! A side effect of that is that I’m now (happily, on purpose, by choice) a full-time consultant! I’ll have a web site up soonish. Until then, check back here for updates. If you’re a tech firm who needs help, and don’t mind remote workers, send mail to bkjones at Google’s mail service (.com).

Some folks thought I’d passed away due to the uncharacteristic lull in posting frequency on this blog. I’m very much alive — but working for a startup and maintaining a consulting business simultaneously is hard, especially when two large projects fall into your lap at the same time. So what have I been up to?

Well, as part of the now-public merger between the company I worked for and the new company, I was doing the migration of our infrastructure to theirs. That involved rewriting some backup scripts, writing a data synchronization routine (complete with backout capability — I’ll post code samples after I clean out all the site-specific stuff — it’s python and MySQLdb!), set up a different (and kinda cool) MySQL replication scheme, a different (and also kinda cool) failover scheme, test, test, test, coordinate with everyone down in VA at the new company to make sure everything was working and in place, and then “flip the big ol’ switch”.

Now I’ll be writing even more scripts, planning even more migration of infrastructure services components, doing more testing, and retiring old AddThis assets.

I’m happy to say that the folks at Clearspring have been an awesome team to work with. The culture there is kind of like “nobody here is any better than anybody else”, and it has a rather dramatic effect on productivity compared to other places where everyone thinks they’re not allowed to be wrong and have to preserve their jobs and make other people look stupid. Nope, if you’re a junior guy or someone from another department with a good idea, and it works, that’s great! It’ll be poked and prodded at, and if it passes muster, it’ll be deployed. And why shouldn’t it be? Generally, because egos get in the way. In my book, if you don’t stand in the way of something great, you’re more of a hero than if you do. The ideas don’t all have to be yours.

And, get this, they document their stuff! And it’s easy to use and browse! I’m on a documentation project right now for another client, and it’s challenging just to get people to talk about documentation. :-/

So what’s next for me? I’m honestly not sure at the moment. I’m consulting, it’s going well, I have more than enough work, but I also have a little bit of ‘the bug’, so don’t be surprised if I come back here and tell you that I’ve joined some “social multimedia web 2.0 mindmapping in the cloud with sharing” company or something like that in the near future.

addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F09%2F30%2Fa-merger-migration-mysql-python-and-more-news%2F'; addthis_title = 'A+merger%2C+migration%2C+mysql%2C+python%2C+and+more+news'; addthis_pub = 'jonesy';

I have a small EC2 instance running with a 25GB EBS volume attached. It has a database on it that I need to manipulate by doing things like dropping indexes and creating new ones. This is on rather large (multi-GB, millions of rows) tables. After running one DROP INDEX operation that ran all day without finishing, I killed it and tried to see what was going on. Here’s the results of the first 10 minutes of testing:

-bash-3.2# dd if=/dev/zero of=/vol/128.txt bs=128k count=1000 1000+0 records in 1000+0 records out 131072000 bytes (131 MB) copied, 0.818328 seconds, 160 MB/s

This looks great. I’d love to get 160MB/s all the time. But wait! There’s more!

-bash-3.2# dd if=/dev/zero of=/vol/128.txt bs=128k count=100000 dd: writing `/vol/128.txt': No space left on device 86729+0 records in 86728+0 records out 11367641088 bytes (11 GB) copied, 268.191 seconds, 42.4 MB/s

Ok, well… that’s completely miserable. Let’s try something in between.

-bash-3.2# dd if=/dev/zero of=/vol/128.txt bs=128k count=10000 10000+0 records in 10000+0 records out 1310720000 bytes (1.3 GB) copied, 15.4684 seconds, 84.7 MB/s

So the performance gets cut in half when the number of 128k blocks is increased 10x. This kinda sucks. I’ll keep plugging along, but if anyone has hints or clues, let me know. If this is the way it’s going to be, then this is no place to run a production, IO-intensive (100,000s and maybe millions of writes per day, on top of reads) database.

addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F08%2F27%2Fsustained-io-on-ebs-no-bueno%2F'; addthis_title = 'Sustained+IO+on+EBS+%3D%3D+No+Bueno'; addthis_pub = 'jonesy';

Short Version: You can find a fantastic video here about bundling customized AMIs and registering them with Amazon so that you can launch as many instances of your new AMI as you want. The video is so good that I don’t bother writing out the steps to do the bundling (it would be pretty darn long). These are some notes about launching an AMI, customizing it, and mounting an EBS volume to it (the video linked above doesn’t cover EBS). Also, check out the ElasticFox tool which is a very good GUI for doing simple EC2 operations. Nice if you’re just getting started or doing some simple tests.

There are two ways you can go about creating a custom machine image (AMI) for use with Amazon EC2: You can create an image locally by dd’ing to a file, mounting it with “-o loop” creating a filesystem on it, and bootstrapping the whole thing yourself, or you can grab an existing AMI that will serve as a “good enough” base for you to make your customizations, then bundle the customized image.

I’ll be talking about the latter option, where you identify a “good enough” image, customize it for your needs, and save that as your AMI. Unless you’re doing some kind of highly specialized installation, or are a control freak, you shouldn’t really need to start from scratch. I was just building a test image, and wanted a CentOS 5.2 base installation.

Here’s the command you can use to browse the AMIs you have access to (they’re either public, or they’re yours):

$ ec2dim -a

If that command looks funny to you, it’s likely because you’re used to seeing the really long versions of the AWS commands. Amazon also provides shorter versions of the commands. No, really - have a look! The long version of this command is:

$ ec2-describe-images -a

Too long for my taste, but it’s nice to know it’s there.

So, rather than start from scratch, I grabbed a base image that was close enough for my needs, and customized it. It’s a 5.1 base image, pretty well stripped of things that I don’t need, and a few that I do, but that’s ok. I’d rather start with less than more.

So step one is to launch an instance of the AMI I’ve chosen to be my ‘base’. Simple enough to do:

$ ec2run ami-0459bc6d -k ec2-keypair

And that’s pretty much it. It takes a couple of minutes (literally) for the machine to actually become available. You can check to see if it’s still in “pending” state or if it’s available by running ‘ec2din’. Without arguments, that’ll show you the status of any instances you have pending or runnning. Once the instance is running, you’ll be able to glean the hostname from the information provided.

An important note at this point: Don’t confuse “image” with “instance”. For the OO types in the crowd, an “image” is an object. It does nothing by itself until you instantiate it and create an “instance” of that object. For sysadmins, the “image” is like a PXE boot image, which does nothing until you boot it, thereby creating an “instance”.

The reason I used “PXE” and “object” in the above is because of the implication it makes: you can launch as many instances of an object as you want from a single object definition. You can boot as many machines as you want from a single PXE boot image. Likewise, you can launch as many Amazon EC2 instances from an image as you want.

So, in the time it took you to read those last two paragraphs, your instance is probably running. I now grab the hostname for my instance, and ssh to it using my keypair:

$ ssh -i ec2-keypair root@<hostname>

Now that I’m in, I can customize the environment, and then “bundle” it, which will create a new AMI with all of my customizations. With the instance in question, I installed a LAMP stack, and a few other sundry tools I need to perform my testing. I also ran “yum -y upgrade” which will go off and upgrade the machine to CentOS 5.2.

One thing I want to do with this instance is test out the process for creating an EBS volume. The two pieces of information I need to do this are the size of the volume I want to create, and the “zone” I want to create it in. You can figure out which zone your instance is running in using ‘ec2din’ on your workstation (not in your instance). I took that information and created my image in the same zone using the ‘ec2addvol’ command. If you don’t have that command on your workstation, then you don’t have the latest version of the Amazon command line tools. Here’s the command I ran:

$ ec2addvol -z us-east-1b -s 25

To see how it went, run ‘ec2dvol’ by itself and it’ll show you the status of all of your volumes, as well as the unique name assigned to your volume, which you’ll need in order to attach the volume to your instance. To do the ‘attachment’, you need the name of the volume, the name of the instance (use ‘ec2din’), and you need to choose a device that you’ll tell your instance to mount. Here’s what I ran (on my workstation):

$ ec2attvol -d /dev/sdx -i i-xxxxxxxx -v vol-xxxxxxxx

Now you can go back to the shell on your instance, mount the device, create a file system, create a mount point, add it to fstab, and, as they say in the UK, “Bob’s yer uncle”. By the time I wrote this post, I had already shut down my instance, but here are the commands (caveat emptor: this is from memory):

# mkfs.ext3 /dev/sdx # mkdir /vol # mount /dev/sdx /vol

If that all works ok, you can add a line to /etc/fstab so that it’ll be mounted at boot time, but I haven’t yet figured out how to attach a volume to an instance at boot time. The mount doesn’t work if you don’t attach the volume to the instance first. You’ll get a “device doesn’t exist” error if you try it. Clues hereby solicited. I assume I could probably use ‘boto’ and some Python code to get this done, but doing the same with a shell script wrapper around the Amazon tools might also be possible — but I don’t know how reliable that would be, because you’re at the mercy of Amazon and how they decide their tools should present the data (and *if* they provide the data you need for a particular operation down the road).

So now I have an EBS volume, and an instance. The volume is attached to the instance, and I can do things with it. I’m testing some database stuff, so I copied a database over to the volume, which was now mounted, so I could just ’scp mydb.tbz root@<instance>:/vol/.’

Once my database is there, I can attach it to pretty much whatever I want, which makes it nice, because I can test the same database, and the same database code, and see how the different size Amazon instances affect the performance, which gives me more performance data to work with. For production purposes, I’ll have to look more closely at the IO metrics, play with attaching multiple volumes and spreading out the IO, and I also want to test the ’snapshot’ capabilities. It’s also nice to know that if I needed to launch this in production (there are no plans to do so, but you never know), I could upgrade the database “hardware” more or less instantly

If anyone has code or tools to help automate the management of all of this stuff, please send links! If I come up with any myself, I’ll most likely post it here.

Now that I have a customized AMI with all of my packages installed and my config changes made, I need to bundle this so that I can boot as many instances of this particular configuration as I want. An important note about bundling this *particular* image is that you MUST run ‘depmod -a; modprobe loop’ before bundling, since this process basically abstracts the manual process of bundling an image, which involves mounting a file as a volume, which requires a loopback mount.

The video I used to do the bundling is here, and if you can live through the disgustingly bad burps and chirps in the (Flash version) audio, it’s an excellent tutorial for bundling custom AMIs. While the process *is* pretty straightforward, it involves a number of steps, and the video goes through all of them, and it worked perfectly the first time through.

addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F08%2F27%2Fmore-adventures-in-amazon-ec2-and-ebs%2F'; addthis_title = 'More+Adventures+in+Amazon+EC2+%28and+EBS%29'; addthis_pub = 'jonesy';