At OthersOnline we make heavy use of memcached, memcachedb and memcacheq. While in general all of these work and work well, we're learning that many of the memcache client libraries (we're using spy) may have been developed with the assumption that the backend you are interacting with operates as a cache.

Individual messages may or may not be treated as the precious, valuable nuggets of data that no doubt they are. Individual backends may be assumed down for lengthy periods of time regardless of actual state. While in general I like memcacheq and would love to try out kestrel, the fact that both rely on a memcache client library gives me pause.
If you are going to write a queue service, for frack's sake please include a getQueueSize() method in the API. Make it fast and make it an estimate if you have to, just get 'er done.
  • documentation here largely works
  • install the udev package inside domU
  • comment out the existing getty lines in /etc/inittab and add 1:2345:respawn:/sbin/getty 38400 hvc0


If you have any sense, you've already virtualized your DC. And if you have a lack of cents, you've standardized on Xen based on a commodity Linux like Debian, Ubuntu, Centos/Red Hat or openSUSE/SLES. Solaris/ZFS still fits well in that environment as an iscsi or nfs server.

Plus, it's cool.


I'm assuming you already have a working Xen 3 install on Linux, and that you have some familiarity with creating and running Xen guests. I'm using Ubuntu 7.10 and the xen-hypervisor-3.1 package. Here's details on the hardware.

Getting down to business

  1. Download the iso from
  2. Mount the iso in your dom0: mount -o loop os200805.iso /mnt
  3. Copy the install kernel and ramdisk into your dom0 file system:
    cp /mnt/platform/i86xpv/kernel/amd64/unix /home/xen/kernels/sol-indiana/unix
    cp /mnt/boot/x86.microroot /home/xen/kernels/sol-indiana/x86.microroot
  4. Create a xen conf file for the installation. It should look something like this, and dear god please change /dev/changeme to a meaningful block device on your system. Bonus points for using a different mac address.
    name = 'solaris-indiana'
    memory = '1024'
    disk = [ 'file:/dev/changeme,0,w', 'file:/home/xen/iso/os200805.iso,6:cdrom,r' ]
    kernel = '/home/xen/kernels/sol-indiana/unix'
    ramdisk = '/home/xen/kernels/sol-indiana/x86.microroot'
    extra = '/platform/i86xpv/kernel/amd64/unix - nowin -B install_media=cdrom'
    vif = [ 'mac=00:16:3e:0c:61:50,bridge=xenbr0' ]
  5. Start the domain:
    xm create -c sol-indiana-install.cfg
  6. Select keyboard layout, then your language
  7. Login as jack/jack
  8. Type ifconfig -a to find your ip address assigned by dhcp
  9. Login remotely with ssh -X jack@[ip address]
  10. Start the install process: su root -c gui-install (when prompted the root password is 'opensolaris')
  11. Go through the install process, and at the end choose quit rather than reboot
  12. Use scp to copy boot_archive to dom0 disk:
    scp /a/platform/i86pc/amd64/boot_archive root@[dom u ip address]:/home/xen/kern els/sol-indiana/boot_archive
  13. Shutdown the solaris instance: su root -c 'halt'
  14. Create a new config file (sol-indiana.cfg):
    name = 'solaris-indiana'
    memory = '1024'
    disk = [ 'file:/dev/changme,0,w' ]
    kernel = '/home/xen/kernels/sol-indiana/unix'
    ramdisk = '/home/xen/kernels/sol-indiana/boot_archive'
    extra = '/platform/i86xpv/kernel/amd64/unix -B zfs-bootfs=rpool/27'
    vif = [ 'mac=00:16:3e:0c:61:50,bridge=xenbr0' ]
  15. Start it back up: xm create -c sol-indiana.cfg

Saving Strategy from the Strategists

| | Comments ()

I've been doing some client work lately on the RightScale platform and wanted to share my thoughts.

First, things that you should do:

  • Accept the way that RightScale does things. So yeah, maybe you might prefer nginx or lighttpd or whatever the kids are using these days for a load balancer. Too bad. Don't waste your time reproducing the work that RightScale has already done.
  • Use for dns. The first thing you'll need moving to the cloud is a dns provider with a simple api and flexibility to set low ttl values. The integration between RightScale and dnsmadeeasy works beautifully. Use it.
  • Use svn or git rather than attachments. The most important RightScript we have is one of the simplest... it installs the subversion package and does an svn checkout into /usr/local/<abc>. Later scripts and tools, etc. can rely on the presence of your code, tools and configuration in that directory.
  • Setup arrays for auto-scaling of your app servers. Arrays seem to fit best at the app tier - you probably don't want to automate new database instances, and you probably don't need to automate new load balancers.
  • Use ganglia or similar for usage metrics. You'll need good data to make good decisions. Add some custom metrics to your monitoring system and you'll have more data than you know what to do with.

Things you should not do:

  • Do everything the RightScale way. Nobody in their right mind is going to use the RightScale tomcat image with the Centos default install of tomcat 5. It sucks and it will be totally different from the tomcat install that you used for development. If you can do something better than RightScale without spending too much time on it, go ahead.
  • Count on auto scaling to save your butt. You should know that ec2 instances can take as long as 15 minutes to start up. Add in time for RightScale initialization, your code initialization, etc... and by the time you have a few more servers up and running the internets will probably be on to a competing lolcat site.
  • Rely on a 100% success rate with S3. S3 fails. That's life. And when s3 fails, RightScale can fail, causing your images to come up "stranded in booting." Yet another reason not to count on auto scaling.
  • Heavily use RightScale attachments. Your custom RightScripts can have arbitrary files associated with them. For example, you might have a script that would install ganglia and configure /etc/gmond.conf, and you might be tempted to use an attachment for /etc/gmond.conf. Attachments might be appropriate once in a while but in general they suck for version control, patching and code reuse. Our advice: use S3 for large files, svn/git for text and RightScale attachments only rarely.

Bottom line, is it worth the cost? It depends. Don't expect RightScale to displace the need for a good sys admin, but it just might save you some time every month. At $500/mo. the relevant question is whose time will it displace and how valuable is their time.

SOA with Apache Synapse

| | Comments () | TrackBacks (0)
Fast SOA with Apache Synapse. Nice. Meaning to peek into synapse when I have a chance.

Good luck getting anywhere in the market without tools tho... and we all know how much oss developers love to build nice gui tools.

Mossberg: Samsung's Instinct Doesn't Ring True As an iPhone Clone

Taking on the market leader directly rarely works. In this case, a review of the Samsung Instinct turns into a promo piece for iPhone 2.0. Sure, the Instinct may be a perfectly fine phone, but it tries to be an iPhone, and at that it is bound to fail.

AES encryption with Python

| | Comments ()

First download, build and install the Python Cryptography Toolkit. Next:

Tell me why I would use AppEngine

| | Comments ()

So, let me see... on the negative side:

  • have to use google-specific ORM code
  • no joins or other "advanced" SQL features
  • django admin interface does not work
  • very limited set of libraries available

But hey, on the positive side:

  • Dude it runs on google!

Seriously, are there really that many developers who can't figure out how to manage, deploy and scale their apps - who, by the way also have such huge scaling needs that they need to run on BigTable - that this fills any need at all? Because I don't see it.

Recent Activity

  • Sam tweeted, "@instapaper of course! there it is in the docs. somehow I always seem to miss the "just-work-please=true" option."
  • Sam tweeted, "ˆ@instapaper ah, of course. I'm using the api and not supplying a title. Would be nice if you could just parse the page to figure it out.title-ˆ@instapaper ah, of course. I'm using the a"
  • Sam tweeted, "@instapaper love the product! how about a title and a thumbnail for each item on the unread page?"
  • Sam tweeted, "From the BBC: Can the power of thought stop you ageing?"
  • Sam saved the link twotiny icons
  • Sam tweeted, "€Was thinking about using a bitmask for a mysql field and discovered the mysql SET data type. Wow, who knew?€Was thinking about using a bitmask for a mysql fie"
  • Sam tweeted, "What Do Behavioral Targeters Know About You? (via @andrew_chen)"
  • Sam tweeted, ""