Griddle Noise

HTTP Basic Auth problems that affect Zope (2 and 3) in Safari 5.1, Mac OS X Lion

2011-09-06T15:17:00.001-06:00

One very frustrating experience that I encountered after upgrading to OS X Lion (10.7) was that Safari 5.1, as included in Lion, would constantly pop up basic-authentication dialog boxes on our CMS sites, which are based on the Zope Toolkit (kind of between Zope 3.4 and Bluebream). Just about every page in our admin UI would do this. To get around this, I switched to Firefox for interacting with our admin screens, but still used Safari as my primary browser (better OS X citizen, takes advantage of Lion features which I enjoy, bookmark syncing, etc). This lead to problems as Firefox could get pushed way back in the usage stack and would be paged out, and it did NOT like to wake up after long periods of inactivity.

A couple of days ago I decided to take another look at the WebKit project as I was certain that I was not the only person having this issue. And I found a bug that had been recently closed relating to this issue. It's WebKit bug 66354 and it's resolution is in the WebKit nightly builds since at least September 2, 2011.

Apparently this only affects OS X Lion as it has to do with low level CFNetwork changes in Lion. The bug occurs not with basic auth, exactly, as I was able to use other systems behind basic auth. The bug occurs when there are redirects with Basic Auth, which our CMS uses a fair bit in its admins for basic navigation links.

As of OS X 10.7.1 and its Safari (Version 5.1 (7534.48.3)) this is broken. If you use OS X Lion and Safari and encounter HTTP Basic Auth problems, I'd recommend switching to the nightly builds.

PyFlakes and BBEdit, kind of together (at last?!)

2011-08-19T13:34:00.001-06:00

I've recently returned to BBEdit as my primary development editor after spending the past few years with the increasingly neglected TextMate. One thing that I missed from TextMate was some nice 'PyFlakes' integration. Since adding PyFlakes to my daily workflow in TextMate, either through the 'PyCheckMate' option or via the 'PyFlakes On Save' option (which popped up in a tooltip), my module imports have been cleaner and refactorings have gone more smoothly as I can catch forgotten names and imports in moved code.

The first thing I did when I moved to BBEdit was to rig up a quick script that ran PyFlakes and bound it to command-shift-V (V for Verify, or validate), based on the PyFlakes-On-Save TextMate command (which I found in a Zope bundle, I think). My script just dumped the results into a new BBEdit 'Unix Script Output' text document window. It worked, but was not particularly helpful. BBEdit has a 'check syntax' command built in that will catch SyntaxErrors in a python file, and its results go into the 'search results' window which allows you to step through the errors nicely in a dedicated window. I remember using something like this with Emacs and things that modified Make-mode to jump to error results in unit tests, making it easy to step through failures with a single key-chord.

Well today, I finally found how to get my results into that window. The script is now up on GitHub in Gist 1157742.

New Synthesizer Purchase, and near-future music ideas

2011-08-15T19:33:00.001-06:00

I bought another synth and am just waiting for it to arrive. I bought the Akai MiniAK after seeing a friend, himself quite an accomplished ‘experimental music’ composer, get quite delighted after purchasing one. I’m also exploring a new collaborative electronic music project, possibly quite club oriented, that may also do some live shows in the future. For myself, I’m interested in doing this without a computer. My laptop is getting quite aged and I don’t have the desire (or funds), yet, to replace the studio iMac with a Macbook Pro that could venture out on occasion while retaining the master keys for my suite of software. Granted, with my software suite, I could do pretty amazing things, but I’m more of a tweaker here than a performer.

I’ve been interested in getting a keyboard synth for a while now. I have a keyboard USB MIDI controller and in theory I could just attach it to my Korg MS2000BR rack mountable module. I have no power adapter for this controller (it prefers USB power and I haven’t bothered looking for an AC adapter for it), and the MS2000BR is cool, but the two pieces of gear together are a bit beefy.

I’ve considered the microKORG, partially for its kitsch value, partially because it is an interesting synth. But it’s essentially the MS2000BR in a toy-ish package, and I already have that engine in a more complete synth. Since I haven’t done much programming with the MS2000BR, I sometimes wish I had bought the microKORG instead. Then again, the programming that I have done with the MS2000BR has produced some nice results.

Recently, after revisiting some half-finished electronic experiments from the past few years, I started reconsidering getting the Alesis Micron. I like the Micron’s visual design and found its sounds interesting, although a bit too clean. The sounds can be easily dirtied up with the plethora of noise gear and MoogerFoogers and Metasonix modules I already have. Then I noticed that the Micron was disappearing from various online resellers, and appeared to have been discontinued. Still, after my friend’s MiniAK purchase and in light of my other desires, I came very close to buying a Micron…

Until I found out that the MiniAK is essentially a re-packaged Micron, with a sturdier Akai Pro body and a (silly) emphasis on the vocoder. I like my MPC500 and plan to have it at the center of whatever the hell I’m building out here.

I guess I finally have to learn (and learn to like) MIDI. I’ve got plenty of disparate pieces, just need to put them together. MS2000BR, MPC500, Electribe EMX-1, and now MiniAK. Plus all my pedals and little analog bits.

And I have my eye on getting a Dave Smith module or two. I’ve been fascinated with the Evolver for a while, and the Mo’Pho and Tetra are pretty damn cool. My big analog modular plan can wait a little while longer.

Is This Sensationalist Headline In The Form Of A Question?

2011-06-09T11:21:00.001-06:00

It depends. We'll have to wait and see. Only time will tell. And other such bullshit.

To square and back again

2011-05-19T15:11:00.001-06:00

I believe I'm ending my personal Squarespace experience. While I like Squarespace's UI and tools quite a bit, I just don't write enough to make it worth even the small amount I spend per month on their most basic package. I've copied some of my better posts from the [now old] http://eucci.squarespace.com/ blog back into this one, and give no further promises on whether I will suddenly feel like posting more often or not. But, of course, I do hope to post more. On Python, on audio, on iPads and file management... We'll see.

Needlessly adding complexity in tests in order to hide their complexity

2011-03-29T13:07:00.000-06:00

Some tweets from DHH on testing:

I respect the guys behind it and I'm all for experimentation, but the proliferation of rSpec and Cucumber makes me sad. (source)

RSpec offends me aesthetically with no discernible benefit for its added complexity over test/unit. (source)

Cucumber makes no sense to me unless you have clients reading the tests. Why would you build a test-specific parser for English? (source)

The important thing is of course that we get people testing, so tools shouldn't matter too much. But the added complexity still upsets me. (source)

I agree, and I'm glad that these kinds of tests have never really caught fire in the Python world. There are implementations of the RSpec and Cucumber ideas, but they don't seem to be as fully embraced. In my opinion, the dark side of testing in the Python world is the abuse of doc tests, thinking that they make both good test cases and good documentation, when in fact they're neither. There are good use cases for doc tests, but I think they've been horribly abused by certain sects within the Python world. However, even when they've been horribly abused, the people writing them seem to go after pretty broad code coverage, and they don't waste a lot of time trying to be cleverly concise (in fact, the verbosity of these large doc tests is what makes them so awful when they're also treated as documentation).

One of my main issues with RSpec and Cucumber as I've seen them in the wild is that there seems to be very few tests, and they're not terribly useful. They seem to be repeats of the classic "baby's first use case", which is "user logs in". Maybe their usage outside of the open-source world is different, but the few projects I've seen which use them have very few test cases that always has me going "that's it? you think you're tested?"

And as David points out, the complexity going on behind the scenes to make the tests just seems silly. Granted, 'Unit Tests' aren't always that easy to read, but they offer a finer example of API interactions. And if you want clarity, just add some comments. Take a cucumber-esque line like "verify that the file is encoded in UTF-8" and "now the file is encoded as latin-1" and put them as comments above the test/assert/verify statements that prove that line.

RSpec and Cucumber feel like the kinds of fiddly things that get in the way of doing real work - you can spend a lot of time writing all the back-end support to get a single test to read like an english Haiku. Or you can spend time writing a good battery of tests that actually get good coverage of the system.

Forget distraction free, I want file-management free!

2011-03-24T21:05:00.000-06:00

A few years back I downloaded an interesting piece of software for the Mac OS - Writeroom version 1.0. Writeroom sells itself as a "distraction free writing environment." It's main feature was that it was plain text only and could run in a full screen mode with default colors of being green text on black background. There was some fanfare around it and some skepticism. Personally, I liked the idea - get rid of all of the other windows, instant messages, dancing gifs, email notifications, etc, and just write.

This is something which I like about the iPad - only the current application has my attention. I hope that this doesn't change. I do not like what I see in Android 3.0, nor what little I've seen of the Blackberry Playbook. Their interfaces look too busy. I deal with busy interfaces all day. For me, the iPad is a relief from the world of current desktop computing.

My favorite key combination on Mac OS X is option-command-h, aka Hide Others. When the screen starts filling up with windows and more windows and more windows, Hide Others temporarily brings some order to the madness.

Back to Writeroom. My favorite feature of 1.0 was actually not this 'distraction free writing environment' nonsense. That was a useful feature, to be sure. But the feature I liked most was that it was a file management free writing environment. Writeroom 1.0 managed its own files. There was no saving, no filename choosing (the document name was taken from the first line of the document). This made it a perfect place for just jotting things down, working small ideas out, and maybe taking them into bigger things. That file management free writing environment was bliss, at least for my uses. Every time you opened the app, there were all your documents. You could close them to hide them, but they were easily accessible from a menu and a keyboard shortcut.

There were some other apps that offered similar features - the Stickies application that comes with Mac OS X (and dates back to Mac OS System 7.5) is another nice tool for temporarily holding small pieces of information without having to worry about file management; Yojimbo is a collection bucket of notes, PDFs, etc, and can be another place for notes to quickly go; and Apple Mail in Mac OS X 10.5 added Notes which could be stored in IMAP (and can finally sync to the Notes app in iOS 4) and offer a nice quick jot-down place that also takes its title from the first line.

Writeroom 2.0 killed this feature. I avoided writeroom 2.0, but when version 2.5 was on the Mac App Store at a temporary discount, I bought it. And I hate it. Now I have to think about filenames. Now I have to think about where things are being stored. Writeroom 1.0 allowed you to easily export a document out, but that kind of decision was usually made AFTER you were finished with the document. There are times when I don't want to think about that stuff when starting out. This is not true for all cases - much of my day work is spent in TextMate writing Python code, reStructuredText documents, and managing configuration files. Of course I need full file management in that situation. But when it comes to just writing down some notes, capturing some ideas, and maybe thinking through a proposal, I miss that Writeroom 1.0 functionality.

And again, this is something that the iPad (and iPhone) gets right, for most instances. Whether its OmniGraffle, Pages, Garageband, or the built in Notes app, starting a new document is damn easy. Saving is automatic. You just start using it.

Fortunately, the next version of Mac OS X, "Lion", looks like it offers many long overdue file/document management features for developers so that they can offer functionality closer to Writeroom 1.0. Again - I'll always need TextMate and some apps to do manual file management, but those uses are specialized and becoming more rare. Mac OS X Lion adds support now for auto-save, reversion, locking, and historical change tracking. In some ways, this feels like going back to some of the ideas explored in the LISA and other pre-Mac user interfaces that weren't shackled to the hardware limitations of the original Mac OS (file management in the Lisa OS was quite a bit different - everything started from the finder, including new documents). In other ways, it feels like we're catching up to what the Newton OS, Palm OS, and iOS devices have long done in their aims of imitating paper notebooks and planners - everything is just saved, automatically.

It's not perfect for every situation, but it's something I've long been wanting in more places than just calendar and contact apps.

Dead SCSI Bits

2011-02-15T13:08:00.000-07:00

Some time ago, there was a cool audio project hosted by Radiantslab called DeadSCSI. And there was also DeadSCSI 2. It all appears to be gone now, but there is at least some information available in the wayback machine. I contributed in both rounds as Eucci.

When Ergo Phizmiz recently tweeted a link to this page of failing hard drive sounds, it got me thinking about my bits of the DeadSCSI project that I had laying around on my studio drobo. I spent some of saturday doing some lightweight remastering. Early Eucci, where I abused the computer like I used to abuse tape, is pretty rough in its output quality. Levels are just all over the place. DC Offset lingers here and there. It's a mess. But there are some pretty spectrum graphs.

Last night I uploaded the tracks to my Rive collection on archive.org and cut a release as rive 058.

What is mildly interesting about this is that although all of the tracks were re-finalized on Saturday, I didn't upload them until last night. It was a process I was starting to put off, remembering that I needed to name the files properly, upload them, enter metadata, and make a web page on euc.cx. Not that big of a deal, really, but big enough to put off until tomorrow and tomorrow and tomorrow.

Last night, when I got home from work and after I had walked the dog (enjoying some unseasonal warmth before winter makes its fierce return later this week), I was thinking about what to do from that moment until dinner. I was starting to get ready to play the excellent Red Dead Redemption, but I knew I had to cut this release. Then I remembered something Merlin Mann said in the first or second episode of the Back To Work podcast. He basically said that it's easy to complain about needing time or certain circumstances in order to work on something, but that we don't complain about needing that time or certain circumstances to play video games, and we sure as hell can sit there and play a video game for an hour. It feels like a shame to have to be reminded of this ridiculousness, but it's something I apparently need to remember constantly. So instead of playing Red Dead for an hour, I spent the time preparing the release. It took a little bit longer than expected and the dog was getting impatient towards the end. But overall it was a nice use of that time.

There are many more releases to come, including some new material.

To Laptop or Not To Laptop

2011-02-13T23:36:00.000-07:00

PyCon 2011 is coming soon, and as it does, I find myself facing the question "do I bring my Macbook this year or not?"

My Macbook has spent most of the past couple of years serving as a vehicle for viewing DVDs, Hulu, or Netflix streaming from bed. It does get used occasionally as a home / travel development machine, but that use has declined significantly over the past year. I'm fortunate to be at the place in my life where I can leave work at work; but I also have been rather uninspired to do personal development work. Hence, the laptop is basically a DVD viewer.

And for DVDs, it's quite nice to watch TV series DVDs from bed. I only watch about 20-30 minutes a night before tiring out. It's a fun way to revisit personal favorites like HBO's The Wire and Deadwood. I'm currently watching Twin Peaks, seeing season 2 for the first time.

But for DVDs, it's also starting to become a liability, as I own more and more blu-ray. As soon as I decide to get a blu-ray player for upstairs (either a portable one or getting a second television), the laptop's uses will be near zero.

One potential use is for travel, at least for technical conferences. I brought the Macbook along to PyCon last year. I did the whole trip out of my day-to-day backpack, which accommodates the Macbook just fine. While at the airport and in flight, the extra weight and space did stand out, but that was only a fraction of the trip. I'm trying to remember now just how much I used it once there. I took far less notes on the device than I expected, and have done little with the notes I did take. They still sit in a folder on the Macbook's desktop.

I know I did some coding while there, but not much. I don't believe I made any major commits or pushes while at the conference. I wasn't involved in tutorials or sprints last year and won't be this year either, so there's little need to code while at the conference.

Finally, my co-worker had brought his laptop and while he did use it to maintain some situations back at the office, he told me he never used it on the conference floor. I think the most use we got out of our laptops was for watching movies on the plane. And hey, I've got an iPad for that.

So as this next tech conference comes up, I'm seriously considering leaving the laptop behind and just using good old pen and paper. Maybe my personal era of the laptop is officially over. Weird.

Sunhil Intentions for 2011

2011-01-27T17:33:00.000-07:00

As announcing one's intentions is the simplest way of hearing God laugh, I'm planning on doing the following for Eucci and AODL this year: it's time to finish cleaning out the back catalog. Whether this is a coda at the end or the beginning of the next phase for both projects is uncertain. I just know that it's time to stop letting these unreleased pieces hold me back. Yes, I'd like for some of them to be released in beautiful physical form, and some might still be so. But chances are that most of them will be released online, for free. The candidates are:

Eucci

Pachyderm (1994) - Tape collage, and really the first proper 'Eucci' recording where the early style really came together. Meant to be on tape, but it's long (45 minutes) and I could never find a satisfactory b-side.
Star City (1996-1999) - mid-era electroacoustics and musique concrete, some for radio programs.
Apt (2004) - unreleased sounds for an empty apartment in the world's last perfect spring.
Like The City, We're Bound To Last (2000-2009) - Spans some of the earliest Eucci (2000), mid-era (2004) and some late-era work (2007-2009).

AODL

Fans of Flesh and Textured Wrecks (2004-2007) - the most likely to get a proper vinyl release. Long form dark AODL, most of it in special live broadcast performances.

Maybe

Eucci, "Because 1999 Needs an Enemy" - very limited edition tape from a bad start to the year. Of interest, mostly, as it is the direct ancestor of AODL. It was the first experiments in doing feedback loops of effect pedals, and using those as the only sound source.

Keeping History

2010-11-29T14:59:00.000-07:00

Back in mid 2005 I worked on a project for customer wherein I was generating "Froogle Feed" data for the customer's e-commerce sites. For some reason, the project never went live. Suddenly, a request shows up from the customer mentioning "Froogle Project" and asking for a couple of features.

I have no memory of how I accomplished this project in the first place. I don't know if the customer does either. I just have a couple of features in an email project and zero context. Does the customer remember the experimental work we did a couple of years ago? Does he remember the UI?

The big problem is that aside from a couple of emails and basecamp messages, and a couple of generated 'feed' files, I have nothing for this project. Looking through source code history returns nothing. Looking through the current site (an old Zope 2 based site, built out of page templates, python scripts, and SQL Methods) yields nothing either. Unfortunately, there's not a lot of history in the site. And if the feature in question never went live, there's (of course) not going to be ANY history.

I dug through a big folder of printouts and such collected over the eight-plus years of history with this customer. Nothing.

I finally turned to notepads and notebooks, and did manage to find a couple of notes in a couple of different places. And digging through some basecamp messages yielded a mention of a 'froogle.py' file in a development instance that was not put into source control (CVS at the time). That development server was killed off long ago. It may have been hobbling around, but after moving offices twice in the last year or so, the number of internal servers is nearly zero. Long unused development servers tend not to get plugged back in.

Some lessons learned:

Always put a date stamp on handwritten notes. This is now a habit for me, but it apparently wasn't a habit back in 2005. I did find some dates on some pages and used that to backtrack to when this work was initially done (using date stamps on Basecamp and email messages). A side lesson to this is to always include the year when date stamping.
Always use source control, even for unfinished projects. If this one file had been checked into CVS, I would be a bit less stressed right now. I'm far less familiar with this customer's site now than I was then and trying to remember how I built multiple Froogle files and put them into a Zip file would be great knowledge right now, even if the format has changed. Since moving to Git, it's easier to keep dead branches in the central repository. CVS made branching so painful that we never would start a 'froogle-feed' branch. Even if that branch never goes live, it's damn useful to have.
Don't trust development servers. They may not be around forever, and odd little unfinished files or projects can get lost very easily over time. May not seem like a big deal, but sometimes these little projects come roaring back to life after five years. We now do most of our development on our desktop machines. When we get a new machine, all of our files roll over. Central servers work if it's a central common copy (like a master account server). I've found that development servers just lose luster over time, and tend to get very far behind in terms of hardware and core OS performance. Plus they tend not to have the kind of machine/account migration tools that desktop OS's like Mac OS X have.

iPad and Creativity, I

2010-04-09T23:54:00.000-06:00

This Post by Giles Bowkett goes into why the iPad is NOT where creativity goes to die. It's something I have been meaning to write up myself, but haven't gotten around to doing so just yet. I do hope to write more on the subject, but until then I will just add this: if the iPad was only about consumption, I would not have bought one. For months, dating back to when the iPad was just an ever growing rumor, I would show friends apps like Looptastic and Jasuto and say "imagine this on a bigger screen!"

I didn't even see the i-Electribe app coming. I have a real Korg Electribe as well. The iPad implementation is just stunning. (I also have both a real Korg MS-10 and a DS-10 cartridge for Nintendo DS. It's fun to have these real tools, and then have software representations of them in unusual places. A VST plug-in is just so boring compared to an iPad or DS surface, and both of those surfaces are far more fun to use on trains or patios than a laptop).

The iPhone has already proven itself as a fun and capable device for making music, visual art, and even for writing. It's like the universal pocket notebook, capable of capturing not just words and doodles and snapshots but also drum beats and synth sounds and singing and on and on and on. The quantity and quality of creative apps for the iPhone is part of what excites me about the platform. The iPad just adds to that excitement.

iPad, Periodicals, and Purchase Options

2010-03-05T09:10:00.000-07:00

Apple has announced an iPad ship date of Apr 3 for the wi-fi only edition, with the 3G model slated for late April. Pre-orders start next week.

I'm hoping for good magazine content, and I would be willing to pay. The web often sucks for magazine content as obtrusive advertising, annoying pagination (often in support of more obtrusive advertising), and random encounters with pay walls get in the way. And it just sucks to read on a computer screen. It's not the LCD display, but the position and layout of the screen that is frustrating.

I've had subscriptions to The Economist, The New Yorker, and Harper's in the past, and I'm starting to miss them as I've been trying to make more time for reading. The weeklies, especially, are the ones I would like to have electronically if I could pay on a month by month basis. Sometimes it's hard to keep up when life gets busy, and those weekly magazines can pile up quickly.

Finally, I'd love for the magazines to work offline. Then I think I could totally buy into the wi-fi only iPad. I'm already sold on the iPad, just can't decide which model to get. I don't leave the house or travel too often, and the iPhone is often good enough when I am out like that. But having the 3G as a pay-as-you-go option for the iPad could be handy on those occasional small trips.

Still, I think that when I take the train from SLC to DEN later next month, I will be much happier with traditional books and cameras than with a gadget, no matter how cool the gadget is. It's nice to just turn things off for a couple of days.

I think I just talked myself into the wi-fi only iPad.

Python Buildutils for local release management

2010-03-03T15:06:00.000-07:00

Earlier today, I saw a question on Twitter asking "is there some setuptools extensions for uploading a sdist to a server over scp?" I responded with a "yes - we use buildutils' publish command for that."

Buildutils is a collection of useful extensions to Distutils. It hasn't been touched in a while, but it mostly works. It adds some commands like "stats", "pyflakes" (useful, but does not work with recent versions of PyFlakes), and "publish", which allows you to upload a release via SCP or SFTP.

The "publish" command is an extension that must be configured explicitly, so we do it in our 'setup.cfg'. We have a setup.cfg template that we use for all of our packages that uploads releases into one big directory. It looks like this:

; Enable the buildutils publish command
[global]
command_packages = buildutils.publish_command

; Set the destination of the publish command
[publish]
dist_dest = scp://@internalserver/path/to/releases/

; In one command, generate a distribution and checksum,
; upload them, and then clean out the build-related turds.
[aliases]
mkrelease = sdist checksum publish clean --all

We also use zc.buildout in-house, and I've been using that to ensure that I have Buildutils and setuptools-git (uses what's tracked by Git to generate the Manifest of files to include in the distribution package). In our buildout configs, I usually have a 'devtools' section that looks like this:

[devtools]
recipe = zc.recipe.egg:scripts
interpreter = py
eggs =
    current.package [test]
    buildutils
    setuptools-git

With the above, I get a 'bin/pbu' script that has the Buildutils and setuptools-git extensions installed. 'pbu' is a convenience command-line tool from Buildutils that is basically shorthand for 'python setup.py'. Using Buildout like this, I just ensure that the tools I need to generate and publish a distribution are available, no matter what machine I'm using. It's not needed, but I just find it convenient, particularly when other developers in our company need to generate a release from their machines and may not have remembered to install something like setuptools-git.

Leaps and Pains (or - changing development/deployment and scm tools to more closely realize the component architecture dream)

2008-10-16T12:31:00.000-06:00

A year or more ago, I was really struggling with zc.buildout, a Python based tool for building out "repeatable" deployments. Buildout makes setuptools actually usable, particularly for development and deployment of web apps, although there are many other uses.

Buildout keeps everything local, allowing one app to use version 3.4.2 of one package while another app can use 3.5.2. But more than just being an 'egg' / Python package manager, it can do other tasks as well - local builds of tools (from libxml to MySQL and more), again allowing one app to build and use MySQL 5.0.x and another app to use 5.1.x; or just allowing an app to be installed onto a new box and get everything it needs, from web server to RDBMS to Memcached and beyond. We don't use all of these features (yet), but it's a nice dream.

Already it's very nice to be able to make a git clone of a customer app, run buildout, and then start it up. Buildout will put setuptools to work to ensure that proper versions of dependent components are installed (and, quite nicely, it's very easy to share both a download cache and a collection of 'installed eggs' - multiple versions living side by side, with individual buildouts picking the one they desire).

But it was not easy to get to this golden land. Prior to using Buildout, we'd check our code out of our CVS repository. Our customer apps were just another Python package, nothing special (not an application, and - more importantly - not packaged up in 'distutils' style). As we started to make more and more reusable parts, we had to do a lot of checkouts; and so I wrote a tool to help automate this checkout process. It would also check out other third party code from public Subversion repositories; all because it was easier to check out a particular tag of 'SQLAlchemy' or 'zc.table' than to try to install them into a classic-style Zope 3 'instance home'.

But it was getting harder and harder to keep up with other packages. We couldn't follow dependencies in this way, for one thing; and it required some deep knowledge of some public SVN repository layouts in order to get particular revision numbers or tags.

'Buildout' promised to change all of that, and offer us the chance to use real, honest-to-goodness distributed Python packages/eggs. But getting there was so very hard when there are deadlines beating you down.

I took a lot of my frustration out on both Setuptools (which is so goddamn woefully incomplete) and Buildout. But the fault was really in ourselves... at least, in a way. As mentioned above, it was easier to just checkout 'mypackage' into$INSTANCE_HOME/lib/python/mypackage than to figure out the install options for distutils/setuptools. As such, NONE of our code was in the Python 'distutils' style. We put some new packages into that style, but would still just check out a sub-path explicitly with CVS just like we were doing with public SVN code.

Part of the big problem that we had which made it so difficult was that we had hung onto CVS for, perhaps, too long. And doing massive file and directory restructuring with CVS is too painful to contemplate. But moving to Subversion never seemed worth the effort, and so we held on to CVS. But I knew I'd have to restructure the code someday.

Fortunately, Git arrived. Well, it had been there for a while; but it was maturing and quite fascinating and it offered us a chance to leapfrog over SVN and into proper source code management. Git is an amazing tool (perhaps made more so by being chained to CVS for so long), and it provided me with the opportunities to really restructure our code, including ripping apart single top-level packages into multiple namespaced packages (ie - instead of 'example' being the root node with 'core' and 'kickass' subpackages, I could split that into 'example.core' and 'example.kickass' as separate packages and Git repositories while keeping full histories).

For a while, I used Git with its cvsimport and cvsexportcommit tools to clean up some of our wayward branches in CVS, while starting to play with Buildout. I was still struggling to get a Zope 3 site up and running using our frameworks. And here... well, the fault was partly in ourselves for having to go through fire to get our code into acceptable 'distutils' style packages, which made learning Buildout all the more hard. But the available documentation (comprehensive, but in long doctest style documents) for some of the Zope 3 related recipes was very difficult to follow. Hell - just knowing which recipes to use was difficult!

But after many months of frustrated half-attempts, often beaten down by other pressures, I opened a few different tabs for different core Buildout recipes in my browser and furiously fought through them all... And boom! Got something working!

Unfortunately it was one of those processes where by the time I got out of the tunnel, I had no idea how exactly I had made it through. One of my big complaints as I was struggling was the lack of additional information, stories of struggle and triumph, etc. And there I was - unable to share much myself! I can't even remember when I was able to break through. It's been quite a few months. Just a couple of weeks ago we deployed our last major old customer on this new setup; and we can't imagine working any other way now.

'Git' and 'Buildout' have both been incredibly empowering. What was most difficult, for us, was that it was very difficult to make the move in small steps. Once we started having proper distutils style packages in Git, they couldn't be cloned into an instance home as a basic Python package (ie, we couldn't do the equivalent of cvs checkout -d mypackage Packages/mypackage/src/mypackage and get just that subdirectory). And we couldn't easily make distributions of our core packages and use them in a classic Zope 3 style instance home (I did come up with a solution that used virtualenv to mix and match the two worlds, but I don't think it was ever put to use in production).

So it was a long and hard road, but the payoffs were nearly immediate: we could start using more community components (and there are some terrific components/packages available for Zope 3); we could more easily use other Python packages as well (no need to have some custom trick to install ezPyCrypto, or be surprised when we deploy onto a new server and realize that we forgot some common packages). Moving customers to new server boxes was much easier, particularly for the smaller customers. And we can update customer apps to new versions with greater confidence than before when we might just try to 'cvs up' from a high location and hope everything updated OK (and who knows what versions would actually come out the other end). Now a customer deployment is a single Git package - everything else is supplied as fully packaged distributions. It's now very hard to 'break the build' as all of the components that are NOT specific to that customer have to come from a software release, which requires a very explicit action.

Giddy-up 401, File Uploads, and Safari

2008-10-01T20:36:00.000-06:00

I've recently been doing some work to support ZODB 3.8 BlobFiles in our Zope 3 based sites and applications. Doing this brought me back around to seeing some behavior I've seen in the past and probably learned to ignore: uploading a large file from Safari using a basic HTML form (with proper encoding type, POST, etc) seems to take inexplicably long. Even worse - once behind Apache, you might not get an expected response, if any. You might get a 'timed out' response, unsure if the app server has everything and will finish the request/response cycle on its own.

It turns out that Safari does not eagerly send along authentication information along with each request when logged in with Basic Auth. When it does, it seems to have a very short time window.

So say you're logged in to your application with basic auth (for better or worse). The normal pattern is that when encountering an unauthenticated situation, Zope will challenge with a 401 status code and the WWW-Authenticate header (or something like that - I'm away from all specs right now). If you're not logged in, then you get the basic auth dialog box and send along the credentials. If you are "logged in", then the browser doesn't ask for credentials again, but just sends them along.

The downside is that this causes a request to be repeated. And if you just tried uploading a 5 MB file, then that whole file has to be re-submitted to make the full request.

It's the right thing to do with the basic (ha!) information at hand - trying to post all of that data in a single request. But Safarishould recognize that it's submitting to the same server (if not the same URL!) and should automatically include the auth headers. Safari seems to do this, but only on requests in very short windows.

Firefox, on the other hand, seems to have a much longer window in which it will send the credentials along automatically on the first request, instead of waiting for the challenge.

I don't know how other browsers do it. I'm not sure what the spec says, if anything. Glancing at O'Reilly's "HTTP - The Definitive Guide" didn't seem to give any indication of whether it's better for the client to assume that it should send the authentication tokens along with each request back to the same server, or if it's better for the client to hold off on constantly sending that info along unless challenged.

Most of the time this doesn't really seem to matter - it's not something end users typically see as it goes by fast enough to rarely be noticed. Of course there are other ways of getting credentials (cookies, sessions, subdomain/remote ip mapping, etc) which we often use on the main public-facing side of our sites. But for content management and admin, Basic Auth is such an easy fallback, especially in a framework like Zope (1, 2, or 3) which has long had a strong security story and would automatically handle this crap for you way back in the days of Bobo (1996, if not earlier).

It's just an annoyance. Glad I nailed it down to this: uploading large files with Safari (I think IE is, or was, similar) to basic-auth protected sites often can time out because the browser posts once, gets the 401-Unauthorized challenge, and does the post again - this time with the auth info.

Solutions:

don't use basic auth for sites that expect moderate to heavy uploading via forms.
recommend and/or use browsers that send the auth token along more often in the first request.
provide better interfaces for uploading files; providing better communication with the uploader about status, and perhaps having a better interface into the destination web app. Fortunately there are appear to be some free and open solutions out there already.

Wheee!

New Blogging Pastures

2008-09-03T07:27:00.004-06:00

With a fair amount of success, I've pulled in the entries from the old Griddle Noise into a Squarespace account. I've long been dissatisfied with Blogger... It was good enough for a free service, I guess, but I just couldn't take it any more. It was hard to use Markdown, and it was hard to use its "wysiwyg" editor on Safari. Results were unpredictable. So I'm giving Squarespace a try.

It's really quite a nice service, and it will definitely do for now. The big issue is to either:

Make time to write
Write smaller

So head over to eucci.squarespace.com/griddle-noise. Gods and time willing, I'll get around to knocking out my tech thoughts again on topics such as:

Python
Zope 3 Component Architecture
Buildout and the realization of the component dream (almost)
Git
Object publishing
etc...

Remembering Pausch

2008-07-25T19:26:00.002-06:00

There are three things that I remember about IPC 8 (International Python Conference 8), besides it being the only Python conference which I've had the luxury of attending.

Riding from Fredericksburg to somewhere outside of DC in a major Virginia snowstorm, only the first or second such snowstorm I had seen since moving to the area in late 1997.
Meeting Tres Seaver.
The final keynote, masterfully given by Randy Pausch.

Pausch, who passed away earlier today (July 25, 2008), was charismatic, smart, and opinionated, and his keynote was funny and engaging. I must admit that I forgot Randy's name, but I'll also gladly admit that many of the points he made have remained with me and I still think of them often. There are even a couple of anecdotes which I repeat to others. Not many people can give such memorable speeches and presentations. Thanks, Randy. I'm glad to have met and heard him in person once, and wish I had been able to do so again to thank him for some great insights and inspirations.

OmniFocus Revisited

2008-07-16T12:35:00.000-06:00

I've been using OmniFocus for a while now on my Mac, and made occassional use of it's predecessor, Kinkless GTD. As an implementation of the "Getting Things Done" method, or just as a personal "what am I doing?" manager, OmniFocus 1.0 was pretty good. I, personally, liked it better than the alternatives I tried, but some of that may be due to familiarity with the OmniOutliner heritage. And OmniOutliner remains one of the greatest Mac OS X applications ever written. I believe it's one of the greatest outliners ever written. Sure, it doesn't have Mind Mapping support and it can't run a slideshow (although its data can be exported to OmniGraffle or Keynote). OmniOutliner is a joy because it's fast, beautiful, ridiculously easy to use, while still having a fair amount of power and flexibility. Kinkless GTD was actually a set of AppleScripts that made OmniOutliner act like a specialized GTD application (which OmniFocus now does natively).

But when it came to OmniFocus, I only used it at work, and even there I only used it intermittently. But with the 1.1 "Sneaky-Peek" version of the desktop client (still under development), OmniFocus supports synchronization via MobileMe or any WebDAV server. This not only enables me to share this information between the office, home, and the rarely used laptop, but it enables sharing with the awesome new OmniFocus for iPhone.

Second only to the iTunes Remote, OmniFocus for iPhone is the coolest mobile application I've seen so far on this young platform. OmniFocus for iPhone (or iPod Touch) can synchronize over Wi-Fi, Edge, or 3G. That alone makes it very useful. Cooler still is that it's location aware. "GTD" has a strong concept of contexts. A project may require picking up supplies at an office supply store, assembling at home, and mailing out items at the post office. With OmniFocus, you can add location information to any context. This location information may be based on where you currently stand (using Location Services), on a manually entered address, on an address from a Contact record, or on a business search. Then when looking at the "By Location" screen in OmniFocus, available actions get grouped by their location in relation to where you currently stand. "Grocery Store: 1 Mile; Post Office: 3 Miles..." Very Cool.

This is the first electronic implementation of GTD that actually appears usable. Even if you don't follow GTD religiously (I certainly don't), the projects/contexts combination is an effective way of organizing actions. The location-aware contexts in OmniFocus for iPhone help answer the question "what can I do based on where I am?" When used well, it should make it harder to forget those little items that you needed to do or pick up for some small project.

iPhone 2.0 and the iTunes Remote

2008-07-12T00:27:00.003-06:00

Like many others, I was bit hard by the server failure that crippled the big iPhone 2.0 update process this morning. It robbed me of said morning, and I opted to take the day off work. It's been a bit of a tough week, and last night I did my first show in months, and had company over until past 3AM afterwards. Since things are about to get crazy at work, I decided that I needed a day to catch back up on sleep while waiting for my phone to work again.

A few hours later, the phone worked.

One of the most surprising and enjoyable elements is the iTunes Remote. Full and comprehensive access to my fairly large iTunes library on the iMac: all playlists, etc, with the ability to control volume, jump around in songs, see artwork - just like the 'iPod' iPhone application! Sometimes, particularly in the morning, I might turn on Front Row and carry the nice little Apple Remote around with me to have some control over songs while getting dressed in the upstairs loft, but using the remote from up there often required reaching far over the edges and trying to point the remote in the direction of the iMac, just to be able to skip forward a track or two.

But now, I can control it all from the iPhone, without needing a line of sight! In fact, this might be what pushes me to pick up an Airport Express or two (one for speakers in said upstairs loft, maybe one to bring music out on the patio or the bathroom). Full access and control of 29 and a half days of available music playable out of real speakers and controlled by an untethered iPod size device - beautiful.

Looking to a Snow Leopard Winter.. er... Summer.

2008-06-10T14:34:00.001-06:00

I’m a bit excited about Mac OS X “Snow Leopard”. Few user-visible changes, with a focus on fine-tuning and giving developers better access to capabilities of modern hardware. It appears that Apple’s experience in making a lightweight Mac OS X “Core OS” for the iPhone will also drive this release.

One of my favorite operating system releases was OS/2 “Warp” (OS/2 3.0). OS/2 2.0 was a fascinating creature - completely divorced from Microsoft, OS/2 2.0 delivered an aggressively object-oriented runtime built on SOM (a desktop implementation of some of CORBA 1.x, I believe). It was radically different from Window 3.x. It’s hardware requirements were a bit high for the times, but it was a solid OS.

What impressed me about OS/2 3.0 “Warp” was that it’s system requirements were in some cases significantly LESS than OS/2 2.0, while performing better. I don’t know of any majoro user-visible adjustments (this was before operating system releases became the giant dog’n’pony shows that have been expected since both Windows 95 and Mac OS X).

I think that even though desktop and laptop hardware continue to get better, the rapid growth rates seen between 1995 and 2005 are slowing down. Now the pressure is on connectivity, portability, and storage storage storage for all of those mp3s and movies and photos. I think both Windows XP and Vista, along with Mac OS X 10.4 and particularly 10.5 have been a bit cavalier about their usage / expectation of resource availability without doing a good job of cleaning up afterwards. Removing a ‘TemporaryFiles’ folder used by Apple’s “Soundtrack Pro” program gave me back 25 GB of disk space. 25GB! I expect that when doing lossless audio work, I’m going to leave a lot of turds behind. But not that many. That’s an accumulation over only a few months. Now some of that may have been due to crashes brought on by the instability in Mac OS X 10.5.2’s audio subsystem (particularly in relation to some USB audio devices). But still - 25 gig! Over the course of just a couple of months!

I think that Apple is at a good place to do this. Good housekeeping is required - otherwise you end up with situations like Mac OS ‘Classic’ or even Windows Vista, where there is so much old baggage, bad hacks, outdated mentalities, etc, all in play; it makes it difficult to move the platform forward. Some companies and developers have always been mindful of this, electing to keep their products lean and fast, always (see Ableton Live - hands down, the most impressive audio application out there). Other companies don’t support that philosophy for whatever reason - backwards compatibility, rush to market, a combination of the two, etc.

This far into the Mac OS X life cycle, there’s not many new dog’n’pony features to add. The API’s have stabalized, the developer tools offer more than they ever have (Interface Builder 3 is a terrific update), the Finder and Spotlight are actually fast and usable; applications and utilities from both inside and outside of Apple are going to really shine on Mac OS X 10.5 with all that it offers to developers. A new age of PDA’s are upon us, whether it’s a device like an iPhone, an ultra-mobile Asus Eee-PC style portable, or even the Macbook Air: secondary and tertiary devices are really taking off.

I think that an underlying aspect part of the ‘Snow Leopard’ plan is to allow such devices, made by Apple (naturally), to proliferate. When it was announced that the iPhone was built on Mac OS X, I was surprised - Mac OS X has been a pretty wasteful OS - or at least, one that would consume more resources than realized (often for caching, interestingly enough). A standard install is full of crap that may be useful, but often takes up space. How many gigabytes of printer drivers now? To take the fine tuning and resource management ideas from the iPhone variation of OS X into the main system is what I think will allow for Apple to finally make the Eee PC style portable that everyone wanted the Macbook Air to be.

I’m putting my money on some kind of small device, priced around $600-$800, coming out at or around the same time as Snow Leopard. Combined with Mobile Me and Snow Leopard Server’s increasingly Exchange-like feature set (but better priced and more understandable for small organizations), the ubiquitous-data-access capability is there.

Today’s full-featured laptops (MacBooks, Inspirons, whatever) are their own entities; my aging iBook gets used rarely as I just don’t have as much data or software set up on it, and it’s sometimes too big of a pain to keep in sync.

The XO and Eee-PCs (or whatever they’re called) are also separate from the rest of one’s life; useful as a fun or educational toy, or as a geek’s favorite gadget to see what they can get running on such a little device. Most of the other developments I’ve seen in this area have centered around “how cheap and how small can we make a laptop/portable that will run (Linux/Windows XP)”. But outside of education, if this is the only focus being given, then these companies are going to be making nothing more than the next round of casual gadgets that get tossed or buried after a few months - especially if a key factor of what made Palm devices so popular (for a while) is completely neglected.

The Macbook Air is deliberately designed as a complementary computer, using the master’s optical drive even. While sexy, I think the Macbook Air misses the mark on a few items. But I think it’s an indication of things to come - laptops deliberately designed to complement your main machine. Smaller devices, from the Palm to the iPhone, have done this. And they’ll also be designed to work with your (or your company’s) data, which the Blackberry has done (and the iPhone will do when its new ‘enterprise’ support rolls out). Getting this onto other devices, without being constrained to an enterprisey system like Notes or Outlook, is where things really appear to be headed. It’s certainly something that I’d like to have. And the more I look at Snow Leopard, the more I believe that Apple is sneaking ahead of the crowd into delivering this into the hands of consumers. They’re skating to where the puck is going to be.

Granted, Windows “Live Mesh” looks to be heading in the same direction. But after Vista, Microsoft needs to reign in the Windows kernel and distribution. Windows Server 2008 and some of what has been leaked (or speculated) about “Windows 7” seem to indicate that Microsoft is aware of this. And how could they not be? But I think that even with their vast resources, Microsoft has a long ways to go to catch up - even though it appears that they’ve been playing in this area (tablet computing, ultra-mobile pc’s) for a while. A deep cleansing of the Windows core is desparately needed. And then a deep re-implementation of the UI may be needed.

Apple had a terrific luxury (and great idea) with the iPhone. While sharing the same kernel and many same APIs as the desktop (and server) Mac OS X, it has an entirely new UI that is dedicated to its intended use. Windows CE, on the other hand, tried to bring the Windows 95 look and feel to tiny devices and now I’m really not so sure it was a good idea. It allowed Microsoft to punt on some usability and design issues by falling back on the way things work on the desktop. I still see this, even in some of the newest and fanciest “iPhone killers”: some of these have a very fancy launcher app; some even have a very fancy phone and contact app that spins around in 3D and responds to gestures. But then, suddenly, you’re in the tiny-font, tiny-scrollbar, pixelated, stylus-driven world of the interior. It’s like going into a grand building like The Plaza (back when it was a hotel, at least), and finding the inside full of grey linoleum floors, flickering flourescent lights, and cinderblock walls reminiscent of an old hospital or elementary school. Quite the let-down (a lot of courthouses are like this, actually).

I also think Apple was smart to NOT have an SDK at the launch of the iPhone. I bet they would have liked one, but I think the iPhone had to launch when it did, and perhaps not-quite-everything was ready yet. If one looks back at the classic Macintosh and Palm devices and operating systems, you see systems that pulled of very clever hacks to fit within the price and size constraints of the time. The Lisa was much more than a $10,000 Macintosh - it had many features from power management to an OpenDoc style multi-tasking document based UI. But to offer those features, it was priced well out of reach. The Macintosh squeezed as much as it could down into a 128K Ram machine, and the compromises they had to make in order for that to work would end up haunting the company until its near-death. The Palm, too, took the ideas of the Newton and other tablet devices and stripped them down into a size and price point approachable by the masses. And like Apple, the design decisions that were made to make that work have crippled the Palm OS so much that even Palm sells half of its devices with Windows CE (or whatever CE is called these days). Those compromises are bad enough to deal with on your own - but when having to support third party developers and then provide some degree of backwards compatibility, it can just kill you.

By taking the time to put the SDK into beta, to polish up the OS and its APIs, I think Apple will avoid a repeat of that story. Instead of having to support every little exposed compromise that may have been made to get the iPhones out the door last June, Apple could tidy them up. By using a beta period for the SDK and next major release of the software, Apple can respond to feedback and make changes and adjustments before they become permanent.

The Hafler Trio, Dislocation

2008-04-29T00:48:00.001-06:00

I just received the Korm Plastics re-release of The Hafler Trio’s Dislocation. I’ve been waiting for this one more than any of the others thus far. The Korm Plastics re-issue series is a magnificent undertaking wherein a large chunk of Hafler Trio material that has long been unavailable or available only in “wrong” versions is being re-issued in definitive form. The packaging for each has been stunning: each release in a booklet sized “wallet” form, with full booklets, posters, post-cards, and more. The releases are all wrapped in a vellum (or vellum-like) material with additional writing. The packaging alone is of the highest grade - far higher than what these releases saw in their original forms (even ones that were also in wallet/booklet form).

I first became interested in The Hafler Trio when the Hafler Trio / Nurse With Wound collaboration was issued on CD, which I picked up in … 1995? 96? I was a big Nurse With Wound fan at the time and, thus, gobbled this one up. I was blown away. While I recognized many Nurse With Wound sounds, the overall production and flow was of a different nature. Similar, yes, in it all being (perhaps) some form of Musique:Concrete. It just sounded a bit more … polished? Subtle? In any case, it remains a personal favorite.

At this time, I was 20-21 years old, poor, bla bla bla. Most of these CDs that I liked were all import and out of my price range (when they could even be found). But one day around this time, I was in a used cassette shop, and came across the original Hafler Trio Dislocation cassette. For $4! It was a Staaltape release, in a bag (my second one - a couple of years earlier I picked up what turned out to be a fairly rare Legendary Pink Dots cassette in the same style of packaging, mostly because I didn’t yet have a CD player). Not knowing much of The Hafler Trio at the time, I knew that this was something I should get, as it was unlikely that I would easily come across such a find for years to come.

I liked the tape, but I admit that it took a while to grow on me. It was very quiet and subtle, with a lot of field recordings. It was something that I enjoyed, but I found that I had to give it more attention than similar recordings from other artists. I loved having it on tape, though. It heightened the “found sound” feeling. And I loved what was inside the bag (besides the unlabeled cassette). Inside, there was heavy paper with fragments of maps on one side and various clippings on the other. It was far more interesting than some of the cheap shock imagery (or just cheap packaging) used, again, by somewhat similar artists that interested me at that time.

However, as fascinating as it all was, that would be it for me and The Hafler Trio until 3-4 years later when, after moving to the east coast, I’d find a copy of A bag of cats at Printed Matter’s old Wooster Street location in NYC. After that find, I’d lap up whatever I could.

So Dislocation has been held in a special place, and I’d been watching that Korm re-issue series list tick along, waiting for this one to show up. Now it’s here. And it’s much more impressive than imagined. First, the booklet is a new heavy-stock foldout variation of the same maps found within the cassette version. When folded up… I can’t even describe it, really. It’s just beautiful and very different from the booklets that have accompanied the rest of the re-issue series.

But what’s really been impressive has been the audio. This time, I have no prior CD to compare; only tape. And I love tape. But now, more than any other re-issue done thus far, I can really see how well done the audio restoration has been. Remarkably well done. This album deserved (and required) it. As much as I loved the “found sound” fuzzy feeling I got from the tape, there are a lot of nuances here that I may now appreciate. Goddamn well done.

What a DVCS gives me

2008-04-07T23:28:00.004-06:00

I've seen some posts floating around about what a distributed version control system might give you. For me, these are the key elements:

Committing changes is separate from sharing. While the phrase "you can edit on a plane!" gets thrown around quite frequently, I think this is the far more important aspect of that vision. As a developer, you don't always know how a particular path of development might impact the code base. With a purely centralized system, you have to think first about where a path may be taking you as it could affect everybody else. Time and time again, I've seen developers work without any revision control safety net for days or weeks at a time because they don't want to "break the build", and they don't have time to look up the policies for branch naming, merging, etc. Not that such a thing should take a long time, but when under pressure, it's the last thing one wants to deal with. And the untracked changes keep getting bigger, and bigger. And when I say "I've seen developers..." here, I include myself.

With a DVCS, I can commit immediately. I took heavy advantage of this on a recent project where the set of work for which I was responsible was in no state to be set up on other systems. It required new configuration and possibly new tool installations and I didn't have time to help everyone else install and update their sandboxes. They didn't need my code anyways. Instead, I was able to pull in and rebase my work on updates from my co-workers while my personal branch was in development. When my code was mature, it was easy to merge into a more centralized branch. Very easy. In fact, it was just a fast-forward (in Git parlance), since I was rebasing my changes on top of those by my colleagues.

Again, this is possible with a purely centralized system, but it would have required me to realize the significance of my changes and their impact on others. My sandbox was in a "guts on the table" kind of state. Just about every commit I made was stable, but sharing them would have made it harder on other team members to do their work due to the changes made in tool and configuration dependencies.

In essence, a DVCS inverts the control back to individuals. As a developer, I can commit my changes whenever I want. With a purely centralized system, I have to think more about what I'm about to commit, since it immediately impacts all other developers.

A DVCS encourages experimentation. Being able to commit my changes whenever I want and being able to make local branches so easily makes it easier for me to start playing around with new ideas. Whether that's doing a big refactoring or restructuring of code, experimenting with a new feature of third party library, or working on updating the code to run well on the latest release of Foo, I know that I can experiment and SAVE those experiments in small chunks, without impacting anybody else. I can choose when and how I want to share my results. I can choose to throw my experiment away and not have to be reminded by its grizzly branch name for years to come.

We have an internal toolkit that bridges SQLAlchemy and Zope 3 and provides various other useful features and integrations. Late last year, I started looking into updating the code to work with SQLAlchemy 0.4 and to also clean up some ancient hacks. We were still using CVS at the time. I can't remember when I made the branch, but I knew at some point that I was heading down a potentially long path and that a branch would be required. Other priorities were coming up and I'd have to leave this work aside for a while.

Well, there's 2-3 days of feverish work, all held hostage within a single check-in. I've been wanting to pull some features out of that branch and into the current mainline branch, but since it's all in one big check-in, I can't do that easily. This is the second time I've done that (the other time was on a feature branch that lead to a dead end, but along the way I had some good ideas that I'd love to be able to extract now).

If I'd been using Git, I would have been making more commits, more frequently, and in much smaller sizes. Using Git, I would then be able to quite easily cherry pick individual commits out of that branch and apply them to the mainline.

Finally, a DVCS makes it easy to vet changes through the system. We don't have to give new employees the keys to the kingdom, particularly when their skill set is focused on a specific area. Instead, the code can go through review channels. They make commits in their local repository, and tell someone (like me) that they've made some changes. Using Git (or any other tool - but Git's named remotes makes this hella easy), I can pull in changes from their repository, verify that they're good, and push them to the canonical repository. I can merge them into other branches, if required.

Imagine this in a team situation: team members can share their repositories with each other as needed, giving each other chances to do code reviews and fixes before sharing those changes with the larger group or division; all without requiring permission to touch the central repository. Suddenly whole new workflows open up, based on the "networks of trust" inherent in all of us: a team leader collects commits from their team, and shares those changes with other team leaders. Those team leaders pull together changes from all of their teams (while sharing said changes across team lines) and push those on to a QA / Testing division. The QA / Testing division then puts their seal of approval on things by being the ones who control pushing to the "canonical" repository from which builds are based.

There's just so much more that can be done with a DVCS, and we're in an age now where there are very usable and useful tools for this job. A DVCS restores individual responsibility, encourages experimentation, enables adaptive workflows, and I believe it fits more naturally into how we humans organize our interactions. Whether this is in a rigidly defined corporate structure or a loosely connected set of worldwide open source contributors, the peer to peer nature combined with getting the whole repository enables people to step up and do bold things without having to go through channels to get any coveted "write access".

Sunday Great Openings

2008-04-06T10:49:00.000-06:00

In my last post, I revisited an earlier write-up of a personal experience involving The Hafler Trio’s so-called Trilogy in Three Parts. I’ve been on a bit of a binge lately, trying to grab up a lot of Hafler Trio releases that I let slip by over the past few years. Right now, I’ve got two thirds of the Exactly series of long-form dual disc releases revolving around the voice of Jónsi Birgisson (Sigur Ros), with one of those releases being in the form of the deluxe packaging from Simply Superior. I’ve been starting to pick up vinyl releases as well and will soon have all three evidences according to Wolf Sheep Cabbage, among others. I even managed to find a well priced vinyl copy of what has become one of my all time favorite albums: A Thirsty Fish.

But on this here Sunday morning, I find myself ill at ease. Shoulders are tense, bones are cracking, mind is dazed yet burning. To try to find a spot of calm, I first went through what I have so far of the Exactly set, but it wasn’t helping today. So I returned, again, to this initial trilogy. I haven’t listened to it much since my grand encounter four years ago. That was just a little too powerful. But up until that point, I got a fair amount of comfort out of the series. It’s still a remarkably powerful. The trick is not to get too lost in it when not wanting great and terrible visions. All I need right now is a warm, sustained, and calming presence. These can do that job rather well - enough to get me through the morning and into a much needed hot shower followed by a cold walk with the dog and some lukewarm coffee.

A Trilogy Revisited

2008-03-24T07:57:00.004-06:00

I was killing late night minutes by reading through pouring empty into the void - a collection of reviews of Hafler Trio material - when I came across a review of the entire trilogy in three parts. The writing was more personal than many of the other reviews, which were written for larger audiences (magazines, record shops, etc). And then I got to the final paragraph:

Deep into No More Twain, Of One Flesh: 11 Unequivocal Obsecrations, I was deep into trance and ripped apart by my own twain — two entities guiding and dividing my life right now — and had to fight my way out of such deep and frightening consciousness and back into the body. Scratches and cold medal were put to use to reinvigorate the breath and to bring myself down out of the super present to the merely present.

"Wow," I thought, "this guy had an experience similar to mine!"

And then I clicked on the source link - and it was, in fact, my own. April 6, 2004. What a wonderful, horrible spring.

And what the fuck has happened to my brain when I look back at many of the things produced / released at that time and wonder how the hell I could do that (and find time to write about it in fairly eloquent terms). Now I have a hard time making a single, simple, goddamn web page for a simple four track online album. How anyone's ever going to hear Like the city, we're bound to last is beyond me at this point.

Quit blaming Nader

2008-02-24T12:47:00.002-07:00

A friend of mine says, “Ralph Nader is an idiot”, just because Nader is running for president again.

I fucking HATE this line of thinking. Nothing pissed me off more than when watching the documentary on Nader recently and seeing how everyone fucking turned on him in 2004 after championing him in 2000. At least Nader has principals and stands by them. Nearly every ‘liberal’ voice that I had respected proved to me that they had no fucking principals.

Michael Moore in 2000 at a Nader rally: “If you’re voting for the lesser of two evils, you’re still voting for evil!” Four years later, Moore was on the anti-Nader bandwagon.

If we believe in democracy and having choices and voices, then anyone should be able to run for president. 2000 was not Nader’s fault. Gore, and the democrats in general, should have run a better campaign. Just like when teams complained about the Patriots running up the score on them: it’s not the Patriots job to take Tom Brady off the field and stop scoring. It’s YOUR defense’s job to get on the field and stop them (the Giants did it; Philly and Baltimore nearly did it). The democrats should have run better campaigns in 2000 and 2004. They could have stopped Nader from running in 2004 if they had picked some of his causes (just some simple ones) and championed them. But the democrats didn’t.

Honestly - whenever I hear this vitriol against Nader, a man who has championed consumer rights far more than anyone currently running or serving this country, it sounds like people wanting a fucking dictatorship. It’s like the movie ‘moon over paramour’ (or something like that) where there are two posters for the same dictator - red and blue. One guy asks the other who he’s going to vote for, “red or blue”. The other guy says, “it doesn’t matter, it’s a free dictatorship.”

I have the right to vote for whomever I want. And on principal, I will cast my vote for whomever I feel best represents what I want out of a leader. I’m not going to vote for someone just because they where an elephant or donkey lapel pin. I don’t care if my one single vote hands the election over to another eight years of Bush tyranny.

The two party system sucks big time fucking hairy nutsacks. I think South Park represented it best with the “Giant Douche versus Turd Sandwich” vote for school mascot (one of the greatest, and saddest, South Park episodes).

What sucks even more than the two party system is the electoral college being based on representative seats plus senate seats. This causes a vote in Wyoming to count four times as much as a vote in california. Now that’s not fucking fair at all.

The problem isn’t Nader. The whole system is flawed and corrupt and I feel like the anti-Nader rhetoric comes from people who would like to see it stay that way as long as they think they can rig the system to swing to their guy who may be the lesser of two evils.

It’s still evil.

And honestly - seeing the response of so many people against Nader in that film made me absolutely sick with the democratic party and the people who claim to champion its causes. They do not believe in freedom or standing for principals. They’re the selfish, self-absorbed, narcissistic pricks. “Oh, look at me, I write for the nation.” Bit whoop. Nader actually fucking fought hard for things like seatbelts in cars. Big auto wouldn’t do it. They resisted like hell. We all take this for granted now. But it took someone with actual balls to stand up and fight for something so seemingly trivial (now) but which has been a big deal when it comes to saving lives. What has Eric Alterman done for us lately?

The general democratic response to Nader shows that most of these people have no spine, no guts, and they don’t deserve any glory. They certainly don’t deserve my support.

Blaming Nader is a sign of weakness, and a sign that the general democratic party and pundits can't own up to their own fucking mistakes. If they can't do that, they can't lead.

Distributed VCS's are the Great Enablers (or: don't fear the repo)

2007-12-02T22:58:00.000-07:00

The more I play with the new breed of VCS tools, the more I appreciate them. The older generations (CVS, SVN) look increasingly archaic, supporting a computing and development model that seems unsustainable. Yet most of us lived with those tools, or something similar, for most of our development-focused lives.

When I speak of the new breed, the two standouts (to me) are Git and Mercurial. There are some other interesting ones, particularly Darcs, but Git and Mercurial seem to have the most steam and seem fairly grounded and stable. Between those two, I still find myself preferring Git. I’ve had some nasty webs to untangle and Git has provided me with the best resources to untangle them.

Those webs are actually all related to CVS and some messed up trunks and branches. Some of the code lives on in CVS, but thanks to Git, sorting out the mess and/or bringing in a huge amount of new work (done outside of version control because no one likes branching in CVS and is afraid of ‘breaking the build’) was far less traumatic than usual.

One of those messes could have been avoided had we been using Git as a company (which is planned). One of the great things these tools provide is the ability to easily do speculative development. Branching and merging is so easy. And most of those branches are private. One big problem we have with CVS is what to name a branch: how to make the name unique, informative, and communicative to others. And then we have to tag its beginnings, its breaking off points, its merge points, etc, just in case something goes wrong (or even right, in the case of multiple merges). All of those tags end up in the big cloud: long, stuffy, confusing names that outlive their usefulness. It’s one thing to deal with all of this for an important branch that everyone agrees is important. It’s another to go through all of this just for a couple of days or weeks of personal work. So no one does it. And big chunks of work are just done dangerously - nothing checked in for days at a time. And what if that big chunk of work turned out to be a failed experiment? Maybe there are a couple of good ideas in that work, and it might be worth referring to later, so maybe now one makes a branch and does a single gigantic check-in, just so that there’s a record somewhere. But now, one can’t easily untangle a couple of good ideas from the majority of failed-experiment code. “Oh!” they’ll say in the future, “I had that problem solved! It’s just all tangled up in the soft-link-experimental-branch in one big check in and I didn’t have the time to sort it out!”

I speak from personal experience on that last one. I’m still kicking myself over that scenario. The whole problem turned out to be bigger than expected, and now there’s just a big blob of crap, sitting in the CVS repository somewhere.

With a distributed VCS, I could have branched the moment that it looked like the problem was getting to be bigger than expected. Then I could keep committing in small chunks to my personal branch until I realized the experiment failed. With smaller check-ins, navigating the history to cherry-pick the couple of good usable ideas out would have been much easier, even if everything else was dicarded. I wouldn’t have to worry about ‘breaking the build’ or worry about a good name for my branch since everyone else would end up seeing it. I could manage it all myself.

This is the speculative development benefit that alone makes these tools great. It’s so easy to branch, MERGE, rebase, etc. And it can all be done without impacting anyone else.

One thing that I often hear when I start advocating distributed VCS’s is “well, I like having a central repository that I can always get to” or “is always backed up” or “is the known master copy.” There’s nothing inherant in distributed VCS’s that prevents you from having that. You can totally have a model similar to SVN/CVS in regards to a central repository with a mixture of read-only and read/write access. But unlike CVS (or SVN), what you publish out of that repository is basically the same thing that you have in a local clone. No repository is more special than any other, but that policy makes it so. You can say “all of our company’s main code is on server X under path /pub/scm/…”.

And unlike CVS (or SVN), really wild development can be done totally away from that central collection. A small team can share repositories amongst themselves, and then one person can push the changes in to the central place. Or the team may publish their repository at a new location for someone else to review and integrate. Since they all stem from the same source, comparisons and merges should all still work, even though the repositories are separate.

Imagine this in a company that has hired a new developer. Perhaps during their first three months (a typical probationary period), they do not get write access to the core repositories. With a distributed VCS, they can clone the project(s) on which they’re assigned, do their work, and then publish their results by telling their supervisor “hey, look at my changes, you can read them here …” where here may be an HTTP or just a file system path. Their supervisor can then conduct code reviews on the new guys work and make suggestions or push in changes of his own. When the new developers code is approved, the supervisor or some other higher developer is repsonsible for doing the merge. It’s all still tracked, all under version control, but the source is protected from any new-guy mistakes, and the new-guy doesn’t have to feel pressure about committing changes to a large code-base which he doesn’t yet fully grasp.

But perhaps the most killer feature of these tools is how easy it is to put anything under revision management. I sometimes have scripts that I start writing to do a small job, typically some kind of data transformation. Sometimes those scripts get changed a lot over the course of some small project, which is typically OK: they’re only going to be used once, right?

This past week, I found myself having to track down one such set of scripts again because some files had gotten overridden with new files based on WAY old formats of the data. Basically I needed to find my old transformations and run them again. Fortunately, I still had the scripts. But they didn’t work 100%, and as I looked at the code I remembered one small difference that 5% of the old old files had. Well, I didn’t remember the difference, I just remembered that they had a minor difference and I had adjusted the script appropriately to finish up that final small set of files. But now, I didn’t have the script that worked against the other 95%. When I did the work initially, it was done in such a time that I was probably using my editors UNDO/REDO buffer to move between differences if needed.

Now if I had just gone in to the directory with the scripts and done a git init; git add .; git commit sequence, I would probably have the minor differences right there. But I didn’t know such tools were available at the time. So now I had to rewrite things. This time, I put the scripts and data files under git’s control so that I had easy reference to the before and after stages of the data files, just in case this scenario ever happened again.

I didn’t have to think of a good place to put these things in our CVS repo. I just made the repository for myself and worried about where to put it for future access later. With CVS/SVN, you have to think about this up front. And when it’s just a personal little project or a personal couple of scripts, it hardly seems worth it, even if you may want some kind of history.

Actually, that is the killer feature! By making everything local, you can just do it: make a repository, make a branch, make a radical change, take a chance! If it’s worth sharing, you can think about how to do that when the time is right. With the forced-central/always-on repository structure of CVS and SVN, you have to think about those things ahead of time: where to import this code, what should I name this branch so it doesn’t interfere with others, how can I save this very experimental work safely so I can come back to it later without impacting others, is this work big enough to merit the headaches of maintaining a branch, can I commit this change and not break the build….?

As such, those systems punish speculation. I notice this behavior in myself and in my colleages: it’s preferred to just work for two weeks on something critical with no backup solution, no ability to share, no ability to backtrack, etc, than it is do deal with CVS. I once lost three days worth of work due to working like this - and it was on a project that no one else was working on or depending on! I was just doing a lot of work simultaneously and never felt comfortable committing it to CVS. And then one day, I accidentally wiped out a parent directory and lost everything.

Now, in a distributed VCS, I could have been committing and committing and could have lost everything anyways since the local repository is contained there: but I could have made my own “central” repository on my development machine or on the network to which I could push from time to time. I would have lost a lot less.

There are so many good reasons to try one of these new tools out. But I think the most important one comes down to this: just get it out of your head. Just commit the changes. Just start a local repository. Don’t create undue stress and open loops in your head about what, where, or when to import or commit something. Don’t start making copies of ‘index.html’ as ‘index1.html’, ‘index2.html’, index1-older.html’ ‘old/index.html’, ‘older/index.html’ and hope that you’ll remember their relationships to each other in the future. Just do your work, commit the changes, get that stress out of your head. Share the changes when you’re ready.

It’s a much better way of working, even if it’s only for yourself.

Broken Bulb

2007-11-20T14:44:00.000-07:00

With apologies to Johnny Cash: "Flash, I hate every inch of you."

Falling for Git

2007-11-07T21:26:00.000-07:00

You know what? Git must have come a long way in the last year. I keep reading that Git is hard to learn, has rough documentation, etc. But it’s really been quite nice in comparison to many things.

It’s especially nice once you quickly learn that the HTML man pages for Git follow a simple pattern (as I guess many online man page collections must). Just change the end of the URL from git-cvsimport.html to git-push.html or git-pull.html to look up documentation.

That I’ve been able to play around with Git quite successfully and easily just makes my frustration with some Python tools (like easy_install and zc.buildout, particularly its recipes) even more …. frustrating.

And, I’ve totally fallen in love with Git. Yes, I know there are alternatives written in Python that are quite comparable. But Git’s actually been easier to install and figure out (particularly the CVS interaction that I must currently suffer). And people who know me know that I’m no “Kernel monkey”. I’m really impressed with Git’s implementation and general behavior. Very impressed with the implementation.

By the way: if you’re having to work two ways with a CVS repository, this post has been absolutely invaluable. This collection of Git articles has been invaluable in getting some good defaults established, and some tips for building on Mac OS X (with a nice tip to download and untar the man pages directly instead of trying to build them with the asciidoc tool and its terrible dependency on troublesome XML libraries. Goddamn, how I hate XML).

AppleScripting Across the Universe

2007-11-01T01:52:00.000-06:00

After a long day at work, I wrote a long message in Basecamp about what I had accomplished, how to access it, etc. But I forgot to submit the message! Crud. I wanted to send it out before morning and didn’t want to go into the office. I couldn’t get any screen sharing connection to go between the machines. I just had a handful of SSH leaps.

AppleScript to the rescue!

This is probably the most AppleScript that I’ve ever written. Fortunately, Safari supports the command do JavaScript ... in tab. After some floundering around with a similar setup on my local machine, I finally figured out AppleScript’s interesting reference notation and was able to ferret out the window and tab containing the unsent message, add some text to the message’s textarea element, submit the form, and return the extended value.

tell application "Safari"
    set message_tab to current tab of window named "Web site > New message"

    set extended to ".... Fun fact - i wrote this before i left the office and forgot to submit it. as a result, i now know how to submit forms like this via AppleScript."
    set post_body_value to "$('post_body').value"
    set extend_value to post_body_value & " += '" & extended & "';"

    do JavaScript extend_value in message_tab
    set body_value to do JavaScript post_body_value in message_tab

    do JavaScript "document.forms[0].submit();" in message_tab

    return body_value
end tell

I pasted the above code into VIM and ran it with the command line osascript command. Worked like a champ.

And because sleep is for the weak, I decided to track down how to do the equivalent in Python. Mac OS X 10.5 provides a “Scripting Bridge” for Python and Ruby (and potentially others), which causes many frameworks and other objects to be dynamically exposed. Without the need (for better or worse) of yet-another-virtual-machine. Anyways, I cobbled the following together:

from Foundation import *
from ScriptingBridge import *
safari = SBApplication.applicationWithBundleIdentifier_('com.apple.Safari')

def find_window_named(name):
    for win in safari.windows():
        if win.name() == name:
            return win

window = find_window_named("Web site > New message")
message_tab = window.currentTab()

print safari.doJavaScript_in_("$('post_body').value", message_tab)
safari.doJavaScript_in_("document.forms[0].submit()", message_tab)

There may be a better way to do the find_window_named method, but I didn’t have the time to track it down. As it was, I was able to do do the above by playing around with everybody’s favorite Python tool, dir(), which verified my suspicion that many of the commands exposed to AppleScript were also available via the Scripting Bridge. This is evidenced by the currentTab() method of a Safari window, analogous to the current tab of window ... AppleScript. And I imagine most of these are just Objective C methods. And since AppleScript editor’s Dictionary browser told me about the do Javascript [v] in tab [t] command, it stood to reason that it would exist on the Safari object. It was there when I did pprint(dir(safari)), and I knew that I’d need to pass in a Tab object.

In any case, it’s awesome that Apple has embraced Python and Ruby and has tied them in to the Cocoa runtime. Historical note: the first Python - Objective C bindings that I know of where commissioned by a NeXT Developer who wanted to use Python and Bobo (zope.publisher) to do web work with NeXT’s Enterprise Objects Framework, without the weight and cost of WebObjects. I think that means that Python was bridged into the Objective C runtime and NeXTStep frameworks before Jython ever got going. I believe that work was done by the developer who later released Objective Everything which bridged into Perl and TCL as well as Python.

Of course, traditional MacPython from the classic Mac OS was also natively tied in to the AppleScript of that era; AppleScript has always supported other dialects (FrontierScript was a common one).

But it’s nice now to see support coming out of both Apple and Microsoft (and Sun too, I guess) for these languages. The above scripting of Safari was surprisingly easy. As was an earlier experiment to fish around my calendar store for incomplete To-Do items. Quite nice.

But what’s especially nice is that I was able to SSH into my office Mac and tell Safari to submit that form that I had neglected earlier.

Java Crybabies

2007-10-29T22:29:00.000-06:00

So Mac OS X 10.5 (Leopard) doesn’t ship with Java 6. And now Java people are all sad and mad and yelling at Apple for dropping the ball on this.

Why should Apple go out of their way to provide Java 6? After the aborted Java - Objective C bridge experiments, what else is there to do? “Native” Java applications have still never come close to feeling like a native or near native piece of the operating system. Why should Apple keep throwing engineering efforts at this system?

And whatever happened to OpenStep for Solaris by the way?

Honestly, Apple has a dynamic object runtime environment heavily tied to a C based language already. It’s done much of what Java and the .Net framework are now doing, and has been doing that since the latter half of the eighties. It’s interesting to look back to the mid nineties and at the criticisms of NeXTStep/OpenStep. “Why are you using this Objective C thing? Why not C++?”

Because what NeXT understood that others didn’t was that the runtime is what’s important. NeXTStep came closest to providing a Smalltalk-style runtime of dynamic collaborating objects without being the alien and self contained environment that Smalltalk can often be. Initially tied in to a Unix operating system, it later went multi-platform (at least to an extent - I think FoundationKit, Enterprise Objects Framework, WebObjects, and PDO (Portable Distributed Objects) ran on HP-UX and Solaris, while all of those plus AppKit ran on NT, along with D’OLE (distributing COM/OLE while Microsoft was still struggling to provide DCOM)).

In those same mid nineties, there were other attempts to provide some of the power of the NeXTStep / OpenStep platform. IBM perhaps came closes with their CORBA (Corba 1.x) based SOM, which was also to be at the heart of OpenDoc. SOM was the heart of the fascinating OS/2 2.0 and Warp. It also was used in the classic Mac OS. Of course, it was there in OpenDoc’s brief life. But beyond that, SOM was used to provide contextual menu plug-ins, interestingly enough. But it was still fairly heavy, as CORBA could be. Too much wringing and wrangling to help non-dynamic languages function in a semi-dynamic world.

And then there was Microsoft’s Cairo. Never shipped. Some of its technologies found their way into NT… But the big features? One of the big features was to be an “object oriented file system”. This resurfaced as WinFS for Vista. Twelve-plus years later, and it’s still not done.

And of course, there was Taligent. Initially, Taligent was going to be an all-new Operating System, aggressively object oriented, etc. Apple and IBM together, to make a NeXTStep for the rest of us, perhaps? Except instead of a dynamic language, they decided to go for C++. But they apparently had to pump in a lot of work to overhaul the C++ runtime and try to provide some of the dynamic loading options (of Smalltalk, NeXTStep/Objective C, etc). It was a lot of time wasted, I’m sure. And they eventually had to pull back from the all-new operating system plan. That was probably wise, considering the environment of the time. Apple never was able to complete Copland and Gershwin, and Microsoft never got Cairo finished; BeOS never found a substantial market; and even early NeXT-era Apple wasn’t able to sell the idea of a NeXTStep based Mac OS until they provided the Carbon migration path for the classic Mac APIs.

So Taligent shifted to providing, like OpenStep, differing layers that would provide these object features on top of differing host operating systems. Still never happened. Which is a bit of a bummer - they had some paradigms that would have been interesting to see.

In 1994, Jon Udell wrote a short article titled “A Taligent Update,” subtitled Will systemwide object frameworks reinvent programming?

Well, while Taligent never delivered; and OpenStep faded into WebObjects (providing the OpenStep developer tools on NT, NeXTStep Mach, and Mac OS X Server 1.x, aka Rhapsody); this seems to have actually, finally, come to pass. Cocoa is a killer framework for Mac OS X, with many fans. There are bridges to many other languages (Ruby, Perl, Python, among others). It’s not quite the pervasive system-wide framework that it was in the NeXTStep days, but in Mac OS X Leopard Cocoa looks as though it’s reclaiming its position as king of the hill. (For a while, there were many Carbon APIs that were a bitch to use from Cocoa; or at least those used to the comparative ease of Cocoa programming).

And Microsoft’s .NET framework has delivered similar in the Windows market. Of course, it doesn’t have anything like Interface Builder; but it still seems to have a much better share-and-reuse model than anything that’s come before it in Windows programming. And it’s built on: dynamic object systems. Unlike Java (which I’ll get to in a minute), the .NET framework and core languages (C#) appear to be taking cues from Objective C and purely dynamic languages/systems like Smalltalk: dynamic class extension, for example, is a new feature in C# 3.0. It’s also been possible in Objective C. This can be a dangerous feature; but also quite useful and usable. But it’s nice to see this in languages and systems that try to combine C, which has the benefits of familiarity, with the power of dynamic object-filled worlds.

And it’s much better than the heavy and strained world of COM and CORBA.

So anyways - NeXT, and now Apple, has been ahead of this game for quite some time. Granted, if it weren’t for Apple, NeXT would be another blip like Taligent. Except with a shipping product. But still - they survived. And their system wide dynamic object framework idea seems to have been vindicated.

So what of Java?

Java is the bastard child here. I’ve never been comfortable with it. It’s not cross platform: Java IS the platform. And it’s awkward. Even in its best desktop guise - Eclipse - it’s still a foreign environment on Mac OS X. Even Firefox is starting to feel more natural (and Firefox 3 looks to be trying even harder in this area). Why would we want it? Sun has never seemed to care that much about Java on Macs, except to try to showcase their “see, multi-platform!” message. But Windows has always gotten the lions share of the attention, even though Microsoft long ago stopped caring about Java.

On a side note, we’re seeing the same thing happen now with Flash. Ugh. It’s a memory hog on Mac OS X. Makes poor use of resources. If Adobe and Sun don’t seem to care enough about providing a truly killer Mac experience, is it any wonder that they’re being kicked off of the island? Apple’s got the dynamic object system language and frameworks (Objective C, Cocoa); and it’s got an increasingly impressive web environment (WebKit, with Canvas support); it has bridges into and out of AppleScript, Python, Ruby, and Perl, all included with Leopard - bridges too!

One just has to look at how huge the “Java in a Nutshell” books have become to know that Java is no small undertaking. And again, I think that Apple probably has stopped caring. They’ve tried to be good Java citizens - from the Java - Objective C bridge to the all-Java implementation of WebObjects; they’ve tried to be make the Cocoa framework appealing to Java developers and probably tried to make Java appealing to Cocoa developers. But it must not have ever happened. It’s all deprecated now.

And I don’t know what’s going on, really. I just see Apple as having other priorities. It’s not like they’re (purely) a not-invented-here company. They have, after all, built in Sun’s DTrace technology. And Apple builds on and gives back to Open Source, with projects like launchd, the new calendar server, bridge support, WebKit, etc.

But Java…? Apple has no real stake in it any more. The last Java application I ran was Eclipse, months ago, just trying to see what life in a fancy IDE would be like. It was disappointing. Desktop Java just doesn’t figure into a Mac user’s life all that much. I’d rather see Apple focus on improving their primary object language (which inspired Java), focus on improving their APIs and offering more features for programmers (all done handily in Mac OS X 10.5 - from a programmer’s standpoint, it’s extremely impressive) and providing a smooth, fast, and natural platform experience (again done handily in Mac OS X 10.5 - see the still-unequaled Interface Builder 3.0; see CoreAnimation; see a sea of new UI object offerings from Apple).

Why would they spend their time fighting uphill to support a platform whose chief aim is to be an anti-platform? At best, desktop Java on the Mac could mean “runs a windows-like-application-almost-as-good-as-windows, maybe.” Well, we now have Boot Camp and Parallels/VMWare for that. Apple wants to provide a killer alternative platform, and they’ve learned that the best way to do that is to be in control. When Mac browsers were suffering - IE on Mac OS X was slow and strange (compared to near excellent behavior in OS 9), Camino and Firefox were big and slow and non-native (Camino did a decent job, but it had different widget implementations for browser skin and in-page rendering), OmniWeb looked beautiful but its support for new HTML and Javascript was far behind, etc - Apple took control of their destiny by building Safari. And they built in on a toolkit that would let them plug in the right widgets and behavior for a native experience. Hence, all Safari users have long enjoyed having spell checking support in their TextAreas. We get that from Cocoa. As of Mac OS X 10.5, we can turn on grammar checking as well. And now, WebKit is leading much of the HTML 5 charge, recently announcing preliminary support for client side database storage, and were among the first (if not THE first) to put forward tags like canvas which may make SVG usable and make ultimately take care of many uses of Flash (remember when Flash was purely an animation tool?). Apple has all of this in their control. They don’t have to put up with lackluster players / viewers from Adobe.

And I think it’s become pretty clear that Apple’s preferred solution at this time for rich cross-platform(ish) UI and Code is - the web. HTML, CSS, Javascript. It powers Dashboard; it’s been a big selling point of the iPhone (although it is admittedly laughable that Apple said “Web 2.0 apps are your iPhone API!”, but it’s still impressive; people have built some very impressive apps with that very system).

Why does AJAX / DHTML succeed where Applets have failed? And even Flash does poorly? It’s not just because it’s “everywhere”, without a need to install and deal with a JRE. It’s because AJAX is part of the web page. It’s not a self contained rectangle that can do really cool things - within the realm of that rectangle. It’s because AJAX leverages the browser’s toolkit, so that a text field in IE behaves like a text field in IE; and a text field on a Mac behaves like a text field on a Mac. Java performance has never been that great on the Mac. Which must mean that no one cared enough to really try to make it shine, or that the technology is really heavy and inferior. I remember groaning every time I used to check the snow report page at one of the ski resorts because of the extra amount of time taken to load up Java and all that accompanies it just to render some scrolling-headlines; scrolling headlines whose text didn’t match (or even anti-alias with) the surrounding text. It’s absolutely useless and pointless. Granted - I have seen some impressive Java applets in the science arenas, and IBM had some cool chess ones in their old Deep Blue v Kasparov (?) challenge. But again, those have been few and far between.

The two desktop Java apps that I’ve used heavily at points in the past, besides my experiments with Eclipse, were with a UML tool, and then LimeWire. Both of these used on Macs. The UML tool was tolerable, but barely. Limewire was terribly slow - click and wait instead of click and point. Desktop Java is dead to me. I don’t know why you’d want to write in it. Cocoa is just an all out balls out better environment, especially for UI programming as (again) Interface Builder remains peerless. And .NET, especially with the Mono implementation, is a far more interesting playground that seems keen on taking in new dynamic, declarative, and functional features, (LINQ, F#, etc), while Java just feels like a big stack of alphabet soup and static typing and not much else. (Although I do understand that Java 6 has started to break this mold).

And as far as languages go: please. Python, Perl, and Ruby alone offer better cross platform capabilities than Java: especially on the server side where Java is supposed to shine. I didn’t have to wait for Apple’s blessing to use Python 2.5 on my desktop Mac. I didn’t have to wait for Apple’s blessing to draw native widgets with it either. I didn’t have to wait for Apple’s blessing to use Python 1.x to control other applications across the scripting bridges of Mac OS 7, 8, and 9.

Now that Java is Open Source (it is, isn’t it?), maybe the Java community can look into what it would take to provide a good Java experience on Mac OS X. I think that is its only hope. It has to become leaner. The dynamic “scripting” language crowd have all been able to find ways to take advantage of different platforms. Why is it on Apple’s head to provide Mac Java? Going back to what I asked earlier - is Sun going to try to get OpenStep going again on Solaris? Is Microsoft going to provide Java 6 for Windows? Is Apple going to provide .NET 3.5 for the Mac?

A response I’ve seen in the Java community is that Apple is arrogant for not shipping Java 6 with Leopard, and for withdrawing any development downloads and many topics related to Java on the Mac. I think that it’s arrogant of the Java community to think that they matter enough for Apple to continue to sink engineering resources into the platform. They’ve sunk a lot in over the years, and there’s never been a huge payoff. I see no reason for them to continue. They have far better alternatives.

IBM or Sun or anyone else out in the Java / Open Source community should take it upon themselves to provide a good platform if they really care. They’re obviously doing it for Linux and Windows.

Sorry, this is a long and rambling post and now it’s quite late at night when I swore I would be going to bed early. But seriously - Java has not mattered to me as a developer or Mac user for years. It’s a dead weight for Apple. Support for Mac OS “Classic” got the boot in Leopard, and even Carbon is looking like its days are numbered. If those two can be cut off, what the hell chance would Java have?

Catching Up

2007-10-12T14:50:00.000-06:00

These periods between posts keep getting longer, don’t they?

I’ve got nothing earth-shattering to talk about. Work’s been very busy, and we continue to be served well by Zope 3. I’m still royally confused by things like setuptools and eggs, mostly in regards to how they work in a Zope 3 world when you’ve already got long entrenched ways of doing software. I could not get a good answer from anyone I asked (in fact, I often got wildly competing opinions). So I’m sticking with our internal make-rake-like-ish toolkit which is primarily helpful for automating checkouts from internal and external repositories. I did have some success with zc.buildout, but I don’t yet foresee a time when I can use it to deploy whole sites/applications. I can barely see a time when I can use it on anything but small projects that are relatively stand-alone. There’s just a big gap between The Way Things Have Been Done and The Way That It Seems That Maybe Things Should Be Done In The Future.

Of course, neither setuptools nor zc.buildout seem to have “proper” releases. zc.buildout is in an endless 1.0 beta (beta-30 at this point), and setuptools is at 0.6c7. Does that mean that it’s not even at release 0.6 quality yet? None of this instills confidence in this hurried developer.

The big problem is the legacy code, which is in CVS. Some of it is being extracted out into individual packages that have the proper ‘setup.py’, ‘buildout.cfg’, etc. Finally. But I have no idea how to apply it to the bigger picture, and I’ve found very little written words that target our situation.

The biggest downside of being so busy with customer related work is that it’s very difficult to keep up with discussions, conversations, plans, etc. And I’m sure that my frustrations with lack of documentation, seemingly unfinished releases, and so on, are really the fruit of other hurried developers. I admire them for at least releasing something. It’s more than I’ve done in a long time. It’s more than I see myself being able to do for quite some time.

Anyways, the revolving door of Javascript toolkits keeps turning. I’m now deeply enamored with jQuery. “Write less, do more”. I like it. I like that it doesn’t trample all over Javascript, and thus plays well with others (especially others that play well with others, like MochiKit). MochiKit is just so big… I think I might make a stab at writing, at least for internal use, a lightweight version that brings many of its best concepts out without overlapping jQuery’s functionality. MochiKit brings many wonderful Python-ic functions and tools to the Javascript table that make general development much easier.

I’m also deeply enamored with zc.resourcelibrary which is a Zope 3 add-on that makes it much easier to manage javascript and CSS resources and their relations to each other. Among other things, it helps save resources when they’re not needed. For example::

if len(rendered_boxes) <= 3:
    return self.just_render_the_damn_boxes(rendered_boxes)
else:
    zc.resourcelibrary.need('fancy.scrolling.library')
    return self.render_the_advanced_widget(rendered_boxes)

I’ve also adjusted my coding style, returning to the underscore_separated_words style instead of the camelCasedWords style, at least for functions, attributes, and methods. This is closer in style to PEP 8 (the main style guide for Python code). The Zope style guide differs on this point, using camelCased instead. And PEP 8 does say that it’s OK, if not downright preferred, to stay true to the style around you.

But one thing I learned from looking through Rails code was that the underscore_style was easier to read, since the underscore acts like a space. And I’ve become a big fan of writing code that communicates intent; that reads like a story (somewhat). Extract Method is your friend. I’ve grown very distrustful of excessive nesting, or of having very long bodies inside of a ‘for’ or ‘if’ block.

That’s about it. Hell of an update, huh? Well, work’s really started to become work, and is quite enjoyable. I’ve got a good flow going and don’t feel I have as much need (nor place) to be an advocate or crank. As I’ve mentioned before, we’ve gotten incredible levels of code re-use by building our internal libraries and applications on top of Zope 3, and we’ve been able to grow them so much that they’re really the first level of framework. It was such a struggle to do this in Zope 2, but in Zope 3 it does fall (fairly) neatly into place. Nothing else in the Python web-framework-whatsit world comes close.

The only toolkit that’s even better? SQLAlchemy. It’s pretty much the only way I’ll interact with RDBMS systems in Python from this point out. And I don’t mean I’ll be writing every RDBMS interaction as an object-relational mapping. SQLAlchemy is great because it provides a good connection / pooling infrastructure; a good Pythonic query building infrastructure; and then a good ORM infrastructure that is capable of complex queries and mappings (as well as some pretty stone-simple ones).

Numbers

2007-08-16T07:18:00.000-06:00

I don’t know when I last used a spreadsheet for its actual spreadsheet capabilities, on a sheet I designed myself. I think it may go back to AppleWorks (on the Apple II)! Sure, I’ve used sheets like time cards and travel requests that others had made where I just had to fill in the holes. And I’ve received more than my fair share of spreadsheets used like an outliner / lightweight database / structured note list. But I don’t remember how long it’s been since I used a spreadsheet to figure out a budget, to track expenses, or any other dumb mundane thing like that. Until last night.

I was getting ready to pay my mid-month bills, and I was trying to figure out how much I could pay towards one of my credit cards and still have enough cash to cover the expenses remaining in the month. I also realized that I’ve been spending quite a bit at the iTunes Music Store and hadn’t been tracking any of it. I decided that this would be an excellent time to try out Apple’s new spreadsheet application, Numbers. I found a downloadable time trial of Apple’s iWork ‘08 suite, and immediately got to work.

Numbers is pretty damn cool. I don’t know if there are other spreadsheets that behave like this, but in modern times, it seems so obvious: instead of having the big set of cells in one large table, you work in small floating spreadsheets / tables. This is a big deal for so many reasons, with the most obvious being layout. Another great reason is that each table/spreadsheet can be more focused on its job. Already, Numbers felt a lot more intuitive than anything I had used in a long time.

When doing my simple rest-of-the-month budget, my main question was “how much can I pay on this card and still have enough cash on hand for the rest of the month?” Numbers made it easy with its slider option. For just this one cell, I was able to quickly configure it to give me a slider with a range of -700 to -500. When I got the rest of the budget entered, I could then play with the slider and watch its impact on the total-leftover cell. In previous months, I’ve generally done this calculation in my head, or compared where I stood the prior month after paying this particular bill. It was much nicer to whip up a simple spreadsheet where I could make this one particular number interactive and see the results immediately.

So I was able to get a couple of simple but nice looking spreadsheets together quickly that gave me actual data. I could easily play with this data, or just be embarrassed by it (I have spent quite a bit on the iTunes Music Store).

There are still a lot of old-style spreadsheet rules in play, at least in formulas and the like. That’s made a bit easier by being able to use header names (ie, =SUM(Total) or =MINA(Date Purchased)). I think it was Lotus’ Improv, which first appeared on NeXTStep, that worked this way. In fact, I think with Improv, it was the only way you could work: there were no A/B/C/D columns or rows. This was part of a cool feature of Improv wherein you could drag and drop header representations and regroup the data visually without impact on the calculations. I still think that was one of the most forward-thinking spreadsheet applications. But, it’s gone. I believe there’s some open source variation on the idea, possibly written just for GNUStep…?

Still, Numbers is pretty decent. I love the free-floating tables. It does make it much easier to compose complex spreadsheet pages out of multiple tables and data types. It’s pretty easy to refer to other tables as well. And it’s nice to have non-tabular data (text, graphics, etc) floating free from those numbers, making it easier to adjust layouts without impacting cells.

I’m impressed enough that I’m quite likely to buy iWork ‘08, just for Numbers alone. I have a small need for Pages and almost no need for Keynote, but I do find myself needing to get on top of my finances and similar data. Numbers is the first tool I’ve encountered that I think will let me handle my odd needs without requiring a degree or summer course in spreadsheets.

Yahoo TV Is Now Useless

2007-07-18T20:26:00.000-06:00

So, I used to use “My Excite” as my little personal portal. It had good TV listings, which is always important. But many years ago, as Excite was was burying its content under more and more ads, I switched to “My Yahoo!”. Which was even better. Great TV listings. So even now, when I don’t have cable or satellite, I found it valuable.

But then they changed. They upgraded to super fancy TV listings full of AJAX-y action.

But do you know what sucks?

THEY CAN’T GET TIMEZONES RIGHT!

It’s been quite a few months now, and they still can’t get my timezone right.

For seven or eight years, I had no problem with localized listings. Although for a few of those years I was in the blessed Eastern Time Zone. But even when I moved out west - no problem.

But now, apparently Yahoo! has invested all of their resources into flashy features but they can’t tell me when a tv show is on according to the time in my area.

Absolutely useless.

For a while, I’ve gotten by with using the TV listings on the “My Yahoo!” page. It didn’t have all of the fancy features, so somehow it managed to get the times right. But as of today, I’m told that I have to use the new fancy-ass “My Yahoo! Beta”.

First thing that happens when I visit the My Yahoo! beta?

“We don’t support Safari….”

Goddammit. Many of their other upgraded YUI based sites work fine in Safari. I can understand them saying this months ago, which was the last time I looked at the new beta. But to still be having an issue? While trying to be a shiny partner of the lovely iPhone? What’s up?

Anyways, I clicked the “live on the edge” button to see the new page. There were the TV listings. In the wrong time zone!

I submitted an email months ago about this and got the “yeah, we know, we’re working on this” response.

I know that time zones are a bitch to work with, but come on: this shit has worked for years. And now it’s been how long since the launch of the fancy new Yahoo TV section? Six months? Seven? Surely someone could have worked this out by now.

I’m so disappointed. Yahoo! was always one of the most reliable web sites. And I appreciate what they’ve given to the developer community with YUI! and other tools. But this little TV listing issue just takes the cake. It makes it absolutely useless, and now “My Yahoo!”, which has been my ‘home page’ for years, is all but useless as this became such a valued resource as their other sources stopped working.

Very frustrating.

And I’m still looking for a good TV listing site. But all I’ve come up with, so far, is pretty much bullshit. Wrong channels, inability to properly remember channels, too many ads, hard to access listings, slow display…. Augh.

Pissed.

Traits / Roles as Alternative to Abstract Base Classes

2007-05-08T14:29:00.000-06:00

While digging through the Python-3000 development list archives, trying to figure out the state of thought circling PEP’s 3119 and 3124, I came across this gem:

“Traits/roles instead of ABCs”, by Collin Winter.

With ABCs refererring to Abstract Base Classes (pep 3119).

Winter’s proposal is similar to my recent post, which is that this sort of “capability inference” should be dynamic, and not bound to the rigid nature of the class hierarchy. In my post on this subject, I showed a number of different implementations of a single interface (specification, role, whatever) - only one implementation followed the basic class-instance scenario. All others provided the exact same outward appearance, while internally they were implemented as module-level functions, class or static methods, or a dynamically composed single-use object (a brainless instance was made and had methods dynamically attached).

Winter’s roles/traits system, which refers to roles in Perl 6 and traits in Squeak, is along the same lines. I hope to hell it gains traction.

ABC may be easy as 123, but it can't beat zope.interface

2007-05-04T10:45:00.000-06:00

I guess the deadline may have come and gone for getting in PEPs for Python 3000. Guido’s already written up a PEP Parade.

Of particular interest to me has been the appearance of PEPs for Abstract Base Classes (PEP 3119) and the more exhaustive PEP 3124 which covers “Overloading, Generic Functions, Interfaces, and Adaptation.”

Both of these aim to provide ways of saying “this is file-ish”, “this is string-ish,” without requiring subclassing from a concrete “built-in” type/class. But I think they both fall short a little bit, while zope.interface (from the Zope 3 family) provides the best solution.

PEP 3119 (Abstract Base Classes) has a section covering comparisons to alternative techniques, and it specifically mentions “For now, I’ll leave it to proponents of Interfaces to explain why Interfaces are better.” So this is my ~~brief~~ attempt at explaining why.

A quote from PEP 3119 that I particularly like is “Like all other things in Python, these promises are in the nature of a gentlemen’s agreement…” The Interfaces as specified and used in Zope 3 and some other systems are the same way. They are not “bondange and discipline” Interfaces. They are not the ultra-rigid Eiffel contracts, nor are they the rigid and limited Interfaces as used by Java. They are basically a specification, and they can be used (as mentioned in PEP 3119) to provide additional metadata about a specification. There are some simple tools in zope.interface.verify to check an implementation against a specification, but those are often used in test suites; they’re not enforced hard by any system. The agreement might be “I need a seekable file”, which might mean it expects the methods/messages ‘read’, ‘seek’, and ‘tell’. If you only provide ‘read’ and ‘seek’, then it’s your fault for not living up to the agreement. That’s no different than the Python of today. What Interfaces and Abstract Base Classes aim to provide is a better clarification of what’s expected. Sometimes “file-like” in Python (today) means it just needs a ‘read’ method. Sometimes it means the full suite of file methods (read, readlines, seek, tell). Same thing with sequences: sometimes it just means “something iterable”. Other times it means “support append and extend and pop”.

Another side benefit of Interfaces as specification is that they provide a common language for, well, specifications. Many PEPs propose some sort of API, especially informational PEPs like WSGI (PEP 333) or API for Cryptographic Hash Functions (PEP 247). I’ll use PEP 247 as an example for my attempt at explaining why Zope 3’s Interfaces are Better.

A problem with Abstract Base Classes is this: they’re limited to classes. Even when PEP 3119 mentions Interfaces, it does so like this:

“Interfaces” in this context refers to a set of proposals for additional metadata elements attached to a class which are not part of the regular class hierarchy…

It then goes on to mention that such specifications (in some proposals and implementations) may be mutable; and then says that’s a problem since classes are shared state and one could mutate/violate intent. That’s a separate discussion that I’m not going to go into here.

What is important is this severely limited focus on classes. zope.interface works on objects as well, and not just normal ‘instances of a class’ object, but on classes themselves, and also modules.

There are two important verbs in zope.interface: implements and provides. provides is the most important one - it means that this Object, whatever that object may be, provides the specified interface directly.

implements is often used in class definitions. It means “instances of this class will provide the specified interface”. It can also be thought of in terms of Factories and/or Adaptation - “calling this object will give you something that provides the desired interface.”

“What does that matter?” you might ask. Well, there are all sorts of ways to compose objects in Python. A module is an object. It has members. A class is an object. An instance of a class is, of course, an object. Functions and methods are also objects in Python, but for the most part what we care about here are Modules, Classes, and Instances.

Because when it comes down to actual usage in code, it doesn’t particularly matter what an object is. In PEP 3124, the author (Phillip J Eby) shows the following interface:

class IStack(Interface):
    @abstract
    def push(self, ob)
        """Push 'ob' onto the stack"""

    @abstract
    def pop(self):
        """Pop a value and return it"""

Ignore the @abstract decorators, as they’re artifacts of the rest of his PEP and/or related to PEP 3119. What is important is the use of self.

“self” is an artifact of implementation that is invisible to use. Sure, you can write a Stack implementation like this. (Note: I’m going to use zope.interface terminology and style from here on out):

import zope.interface

class Stack(object):
    zope.interface.implements(IStack)

    def __init__(self):
        self._stack = []

    def push(self, ob):
        self._stack.append(ob)

    def pop(self):
        return self._stack.pop()

But when it’s being used, it’s used like this:

def do_something_with_a_stack(stack):
    stack.push(1)
    stack.push(2)
    # ...
    top = stack.pop()

stack_instance = Stack()
IStack.providedBy(stack_instance)
# True
IStack.providedBy(Stack)
# False

do_something_with_a_stack(stack_instance)
# works fine
do_something_with_a_stack(Stack)
# raises an exception because `Stack.push(1)` is passing `1` 
# to `self`.. unbound method, bla bla bla.

Notice that there is no ‘self’ reference visibly used when dealing with the IStack implementation. This is an extremely important detail. What are some other ways that we may provide the IStack interface.

One way is to do it with class methods and properties, effectively making a singleton. (This isn’t a good way to do it, and is just here as an example).

import zope.interface

class StackedClass(object):
    zope.interface.classProvides(IStack)

    _stack = []

    @classmethod
    def push(class_, ob):
        class_._stack.append(obj)

    @classmethod
    def pop(class_):
        return class_._stack.pop()

IStack.providedBy(StackedClass)
# True

do_something_with_a_stack(StackedClass)
# this time it works, because `StackedClass.push(1)` is a class method,
# and is passing `StackedClass` to the `class_` parameter, and `1` 
# to `ob`.

Another variation of the above is using Static Methods:

import zope.interface

class StaticStack(object):
    zope.interface.classProvides(IStack)

    _stack = []

    @staticmethod
    def push(ob):
        StaticStack._stack.append(ob)

    @staticmethod
    def pop():
        return StaticStack._stack.pop()

Again, StaticStack.push(1) and StaticStack.pop() work fine. Now lets try a third way - in a module! Let’s call this module mstack (file - mstack.py)

import zope.interface

zope.interface.moduleProvides(IStack)

_stack = []

def push(ob):
    _stack.push(ob)

def pop():
    return _stack.pop()

Then in other code:

import mstack

IStack.providedBy(mstack)
# True
mstack.push(1)
mstack.push(2)

print mstack.pop()
# 2

So whether we’re dealing with the instance in the first example (stack_instance), the classes in the second two examples (StackedClass and StaticStack), or the module in the last example (mstack), they’re all objects that live up to the IStack agreement. So having self in the Interface is pointless. self is a binding detail.

Jim Fulton, the main author of zope.interface, taught me this a long time ago. Because in Zope 2, you could also make an IStack implementation using a Folder and a pair of Python scripts. Well, those Python scripts (as used in Zope 2 “through-the-web” development) have at least 4 binding arguments. Instead of ‘self’, the initial arguments are context, container, script, traverse_subpath. Just like self is automatically taken care of by the class-instance binding machinery, the four Zope Python Script binding arguments are automatically taken care of by Zope 2’s internal machinery. You never pass those arguments in directly, you just use it like push(ob) and pop().

So there it is - many ways to provide this simple “Stack” Interface. And I believe that both [PEP 3119] and [PEP 3124] are short sighted by focusing on the class-instance relationship exclusively (or so it appears).

And since many objects, particularly instances, are mutable, one could compose an IStack implementation on the fly.

class Prototype(object):
    """ Can be anything... """

pstack = Prototype()
pstack._stack = []

def pstack_push(ob):
    pstack._stack.append(ob)

def pstack_pop():
    return pstack._stack.pop()
pstack.push = pstack_push
pstack.pop = pstack_pop

# Now we can say that this particular instance provides the IStack
# interface directly - has no impact on the `Prototype` class
zope.interface.directlyProvides(pstack, IStack)

pstack.push(1)
pstack.push(2)
print pstack.pop()
2

# We can remove support as well
del pstack.push
zope.interface.noLongerProvides(pstack, IStack)

Examples of dynamically constructed objects in the real world - a network services client, particularly one that’s in an overwraught distributed object system (CORBA, SOAP, and other things that make you cry in the night). Dynamic local ‘stub’ objects may be created at run time, but those could still be said to provide a certain interface.

So now let’s look at whether it matters that you’re dealing with a class or not:

@implementer(IStack)
def PStack():
    pstack = Prototype()
    pstack._stack = []

    def pstack_push(ob):
        pstack._stack.append(ob)

    def pstack_pop():
        return pstack._stack.pop()

    pstack.push = pstack_push
    pstack.pop = pstack_pop
    zope.interface.directlyProvides(pstack, IStack)

    return pstack

@implementer(IStack)
def StackFactory():
    # Returns a new `Stack` instance from the earlier example
    return Stack()

import mstack
import random

@implementer(IStack)
def RandomStatic():
    # chooses between the two class based versions and module
    return random.choice([StackedClass, StaticStack, mstack])

All three are factories that will return an object that provides an IStack implementation, which is exactly the same as the Stack class in the first example. That also claimed that it implements(IStack). When the class is instantiated / called, a new object is made that provides the IStack interface. In Python, another thing that doesn’t really matter is whether something is a class or function. All of the following lines of code yield a result that is the same to the consumer. The internal details of what is returned may vary, but the IStack interface works on all of them:

Stack()         # class
PStack()        # 'Prototype' dynamically constructed object
StackFactory()  # Wrapper around basic class
RandomStatic()  # Chooses one of the class/static method implementations.

And whether we’re looking at the class implementation, or any of the factory based implementations, the result should be the same:

IStack.implementedBy(Stack) # class
# True
IStack.providedBy(Stack)
# False
IStack.providedBy(Stack())
# True

IStack.implementedBy(PStack)    # Factory
# True
IStack.providedBy(PStack)
# False
IStack.providedBy(PStack())
# True

No matter which method of instantiation is used, they should all pass the verifyObject check, which checks to see whether all of the specified members are provided and that the method/function signatures match the specification

from zope.interface import verifyObject
verify_stack = partial(verifyObject, IStack)

all(verify_stack, [Stack(), PStack(), StackFactory(), RandomStatic()])
# True

Now the class-based options will fail on the implementedBy check, because it’s the Class that provides the implementation, not an instance like with Stack

IStack.implementedBy(StackedClass)
# False
IStack.providedBy(StackedClass)
# True
IStack.providedBy(StackedClass())
# False

“OK”, you might say, “but still, why does it matter? Why might we really care about whether these abstract specifications work only with classes? It seems smaller, simpler.”

The main advantage is that specification should (generally) make no assumptions about implementation. If the specification, aka “gentlemen’s agreement” is generally met, it shouldn’t matter whether it’s provided by a Class, an instance, a module, an extension module, or some dynamically constructed object. The specification language should be the same

Going back to PEP 247, the “cryptographic hash API”: there is a specification in that module about what the ‘module’ must provide, and for what the hash objects must provide. Consider also the WSGI spec, the DB-API specs, and all of the other formal and informal specs that are floating around just in the PEPs. Using zope.interface, those specifications can be spelled out in the same fashion. WSGI just cares about a particular function name signature. It can be provided by a single function in a simple module, or as a method from an object put together by a large system like the full Zope 3 application framework and server. It just wants a callable. This is a little bit ugly in zope.interface… but in reality, actually, I think it works. Here’s how it could be specified:

class IWSGIApplication(Interface):
    def __call__(environ, start_response):
        """ Document the function """
    # and/or use tagged values to set additional metadata

This just means that a WSGIApplication must be a callable object taking environ and start_response arguments. A callable object may be a function (taken from PEP 333):

def simple_app(environ, start_response):
    """Simplest possible application object"""
    status = '200 OK'
    response_headers = [('Content-type','text/plain')]
    start_response(status, response_headers)
    return ['Hello world!\n']

Or a class (the __init__ is what is callable here). Maybe the WSGI spec might also state that the result “should be iterable (support __iter__)” Maybe that’s loosely enforced, but the following example shows how the class can make separate declarations about what the class directly provides, and what its instances implement. Instead of using any decorators or magic-ish “class decorators” (the implements, classProvides calls above), we’ll make the declarations for both AppClass and simple_app in the same manner, which matches the style in PEP 3124.

class AppClass(object):
    def __init__(self, environ, start_response):
        self.environ = environ
        self.start = start_response

    def __iter__(self):
        status = '200 OK'
        response_headers = [('Content-type','text/plain')]
        self.start(status, response_headers)
        yield "Hello world!\n"

from zope.interface import directlyProvides, classImplements

# Both 'simple_app' and 'AppClass' are callable with the same arguments,
# so they both *provide* the IWSGIApplication interface

directlyProvides(simple_app, IWSGIApplication)
directlyProvides(AppClass, IWSGIApplication)

# And we can state that AppClass instances are iterable by supporting
# some phantom IIterable interface
classImplements(AppClass, IIterable)

What are the benefits of this, beyond just having a common way of spelling specifications? Instead of, or in addition to, abstract base classes, the core Python libraries can include all of these specs, even if they don’t provide any concrete implementation. Then I could have a unit test in my code that uses verifyClass or verifyObject to ensure I stay inline with the specification.

def test_verifySpec(self):
    verifyClass(ICryptoHash, MyHashClass)

Then, if the specification changes in a new version of Python or in a new version of someone elses library or framework, I can be notified.

Of if the specification undergoes a big change, a new spec could be written, such as IWSGI2Application. Then by process of adaptation (not covered in this post) or interface querying, a WSGI Server could respond appropriately to implementations of the earlier spec:

if IWSGI2Application.providedBy(app):
    # Yay! We don't have to do anything extra!
    # ... do wsgi 2 work
elif IWSGIApplication.providedBy(app):
    # We have to set up the old `start_response` object
    # ... do wsgi 1 work
else:
    raise UnsupportedOrUndeclaredImplementation(app)

Adaptation could provide a means of doing the above… (still, not going into the details.. trying not to!)

@implementer(IWSGI2Application)
@adapts(IWSGIApplication)
def wsgi1_to_wsgi2(app):
    return wsgi2wrapper(app)

# And then, replacing the `if, else` above:
wsgi_app = IWSGI2Application(app, None)
if app is None:
    raise UnsupportedOrUndeclaredImplementation(app)
# ... do wsgi2 work

When you have both specification and adaptation, then you can write your code against the spec. In the above example, the main code does IWSGI2Application(app, None) which means “for the object app, give me an object that provides IWSGI2Application, or None if there is no means of providing that interface.”

If app provides that interface directly, then app is returned directly. Otherwise an adaptation registry is found, and it’s queried for a callable object (an adapter) that will take ‘app’ as its argument and return an object that provides IWSGI2Application.

Another example: knowing that Python 3000 is going to change a lot of core specifications and implementations, such as the attributes for functions (func_code, func_defaults, etc). If an IPy2Function interface were made (and zope.interface or something like it was added to Python 2.x), then code that works with function object internals could program against their preferred spec by adding a line of code:

func = IPy2Function(func)
if my_sniffer(func.func_code):
    raise Unsafe(func)

On Python 2, you’d get the regular function straight through. In Python 3000 / 3.0, an adapter would translate __code__ into func_code, for example. I don’t expect this to happen in reality, but it’s an example of how migration paths could be made between two major software versions, allowing code to run in both.

By taking advantage of this system, my company has seen more re-use with Zope 3 than at any time in our company history. And because (most of) Zope 3 is programmed against specification, we’ve been able to plug in or completely make over the whole system by providing alternative implementations of core specs. This is very hard to do in native Zope 2 (the CMF, on which Plone is based, was probably the first Zope system that started these concepts, which Plone and others were able to take advantage of by providing new tools that matched the provided spec).

At the heart of it, again, is the gentlemen’s agreement, but brought out in full: it doesn’t matter who you are or where you came from (ie, it doesn’t matter what classes are in your family tree or if you are a simple module), as long as you get the job done. There’s a simple contract, and as long as the contract is fulfilled, then everybody is happy.

But if the gentlemen involved can only come from the class system, then there’s still a nasty aristocracy that excludes a large chunk of the populace, all of whom can potentially fulfill the contract. Let’s not cause an uprising, OK?

plispy

2007-04-30T11:32:00.000-06:00

Sometimes, it just happens...

>>> pprint(
...     sorted(
...         map(
...             linecount,
...             path('.').walkfiles('*.py')
... )))

Python's Make Rake and Bake, another and again

2007-04-24T23:33:00.000-06:00

Ian Bicking wrote a post recently titled “Python’s Makefile”. He advocates using / re-using distutils… er… setuptools. (I can’t keep them straight - they’ve both become absolute nightmares in my opinion). He then goes off about entry points, separate setup.cfg files, and other things that still go way over my head. The example he shows is convoluted, and I’m ultimately not entirely sure what he’s really advocating (besides the idea - which isn’t bad - of using the near-standard setup.py file/system instead of re-inventing).

But he mentions, earlier:

Because really people are talking about something more like rake — something where you can put together a bunch of code management tools. These aren’t commands provided by the code, these are commands used on the code.

We do have the infrastructure for this in Python, but no one is really using it. So I’m writing this to suggest people use it more: the setup.py file. So where in another environment someone does rake COMMAND, we can do python setup.py COMMAND.

For me, having an easy way to say bla bla COMMAND isn’t as important as having a good system for automating common tasks that I and/or my colleagues do frequently. As we started to depend on more and more code from internal and external repositories, due to our increased re-use when building on Zope 3, I really needed to automate checkouts and exports. Not everything was neatly packaged as an egg, or the released egg didn’t have a bugfix applied, and I still don’t understand how to make eggs work well with Zope 3 in a manner that I’m comfortable with.

I was initially excited about zc.buildout as a way to automate the monotonous but important tasks that revolve around setting up both deployment and development environments. But I didn’t like how zc.buildout specified its tasks/commands in INI format. It was relatively easy to write new ‘recipes’, so I wrote some recipes to do Subversion and CVS checkouts/exports.

But the INI format just pissed me off. It didn’t fit my needs, basically, wherein I needed more conditional control. More code control. And managing complex sets of parameters required making new top-level sections instead of nesting. Before long I was staring at a very long and very narrow file. And in the end, it was building Zope in a way that wouldn’t work for us. So I abandoned it.

I briefly looked at some tools that let you write these task files in “pure” Python. In this way, Scons appeared to be the closest thing in Python to Rake, which uses Ruby. But Scons seemed far more focused on general compilation issues (compiling C, Java, etc), but that’s never a problem that crosses my path.

I just wanted something like rake. What I liked about every Rakefile that I’ve seen is that it’s been quite readable. Rake makes common file / path commands readily available as Ruby methods, classes, and objects. Rake takes advantage of Ruby’s syntax, particularly blocks (and optional parenthesis) in a way that makes it not seem like, well, Ruby. It looks like something makefile-ish, something shell-scripting-ish, etc. That’s what I wanted; but, of course, in Python.

So I came up with a system. It’s not yet released to the world - far from finished, and there are many competing ideas out there that I don’t feel like competing with - but it’s already proven to be very useful internally. Generally, it’s been used to automate what I mentioned above: retrieving software from multiple repositories, both Subversion and CVS, and placing them in the proper directories. In particular, we try to stick with certain revisions for third party dependencies, and I got tired of trying to capture this information in READMEs and other files that we could refer to when installing certain configurations. It’s even been useful for downloading such software and applying internal patches::

patch = Command('patch')

@task('mysqldbda')
def mysqldbda():
    """ Installs mysqldbda from subversion and applies patch """
    svn = Subversion('svn://svn.zope.org/repos/main')
    svn.co('mysqldbda/tags/mysqldbda-1.0.0', target='mysqldbda')

    # patch mysqldbda
    log.info("patching mysqldbda")
    patchfile = path('fixes/mysqlda.1-5-07.patch')
    if patchfile.exists():
        print patch.read('-p1', '-i', patchfile)

@task('formencode')
def formencode():
    svn = Subversion('http://svn.colorstudy.com/FormEncode')
    svn.co('tags/0.6/formencode')

task('install', ['mysqldbda', 'formencode'])

It’s also been useful for tasks like getting MochiKit and generating all sorts of packed versions. A lot of what makes this possible is the path.py module, which provides a more object-oriented interface over os, os.path, and other Python file utilities.

ROCKFILEPATH = globals().get('ROCKFILEPATH', path('.'))
MOCHIKIT_LIB = ROCKFILEPATH/'libs'/'mochikit'
MOCHIKIT_DL = ROCKFILEPATH/'mochikit_dl'
MOCHIKIT_SRC = MOCHIKIT_DL/'MochiKit'
SCRATCH = MOCHIKIT_LIB/'_scratch.js'
mochikit = namespace('mochikit')

@mochikit.task('get')
def getmochikit():
    if MOCHIKIT_DL.exists() and bool(MOCHIKIT_DL.listdir()):
        return
    svn = Subversion('http://svn.mochikit.com/mochikit')
    svn.co('trunk', target=MOCHIKIT_DL)

@mochikit.task('clearmochilib')
def clearmochilib():
    for jscript in MOCHIKIT_LIB.files('*.js'):
        jscript.remove()

@mochikit.task('make-noexport')
def makenoexport():
    info = Subversion().info(MOCHIKIT_DL)
    src = NOEXPORT.safe_substitute(**info)
    file(MOCHIKIT_LIB/'NoExport.js','w').write(src)

@mochikit.task('build', ['get', 'clearmochilib', 'make-noexport'])
def mochi_install():
    for source in MOCHIKIT_SRC.files('*.js'):
        log.info('copy %s -> %s' % (source, MOCHIKIT_LIB))
        source.copy(MOCHIKIT_LIB)

# Javascript Packing tools (JSPack not shown - essentially it's a wrapper
# around combining and piping Javascript through Dojo's custom_rhino.jar
# to use its compression system)
def packmodules(sourcedir, modules, target):
    mods = [ (sourcedir/mod) for mod in modules ]
    log.info('Packing %s modules', path(target).name)
    JSPack(mods, target).run()

    if SCRATCH.exists():
        SCRATCH.remove()

def jsmin(sources, target):
    packmodules(MOCHIKIT_LIB, sources, MOCHIKIT_LIB/'min'/target)

@mochikit.task('minimize')
def mochiMinimize():
    """
    Generates packed versions of most individual MochiKit files, while
    combining a few core ones together.
    """
    mindir = MOCHIKIT_LIB/'min'
    for jscript in mindir.files('*.js'):
        jscript.remove()
    jsmin(['NoExport.js', 'Base.js', 'Iter.js', 'DOM.js'], 'base-iter-dom.js')
    jsmin(['Style.js', 'Signal.js'], 'style-signal.js')
    jsmin(['Async.js'], 'async.js')
    jsmin(['Color.js'], 'color.js')
    # ...

mochikit.task('install', ['build', 'minimize']).comment('INSTALL!')

I don’t think this falls under the jurisdiction of setup.py (distutils/setuptools). Nor would I want to specify these as zc.buildout recipes and have a separate configuration file to then name all of the files and directories. And, being Python, I don’t really have to deal with compilation steps so I don’t need wrappers around gcc and friends. I’m not (yet) specifying how to build large deployment scenarios. I just need to automate some development tasks, and I need to be able to write them easily. I want to write them in Python, but I want to ensure that they don’t accidentally get imported into normal projects (hence, the files above don’t have a .py extension). And as this is a specialized task, I’ll allow myself to get away with Python shortcuts that I would never touch in normal development, such as import *. In fact, it’s the import * that gives me a lot of the common commands/tools, such as the classes for interacting with Subversion and CVS, managing working directories, etc.

This really stemmed from reading this article by Martin Fowler about people wanting to replace ant with Rake with the advent of JRuby. In the post, Martin states:

The thing with build scripts is that you need both declarative and procedural qualities. The heart of a build file is defining tasks and the dependencies between them. This is the declarative part, and is where tools like ant and make excel. The trouble is that as builds get more complex these structures aren’t enough. You begin to need conditional logic; in particular you need the ability to define your own abstractions. (See my rake article for examples.)

Rake’s strength is that it gives you both of these. It provides a simple declarative syntax to define tasks and dependencies, but because this syntax is an internal DomainSpecificLanguage, you can seamlessly weave in the full power of Ruby.

At that point, I decided that this was the way to go: use Python decorators to wrap ‘task’ functions. The wrapper maintains dependency links, comments, and other things of interest to the internal system; and the wrapper allows the task name to be independent of the function name, allowing easier-to-type tasks for use from the file system. But the ‘task’ function is plain Python. Or, like some of the examples above show, task can be called without the @ symbol that makes it a decorator. Multiple callable actions can be added to a task, potentially allowing for more ‘declarative’ style:

mochikit.task('minimize').using_action(
  JSMinMap(
    {'style-signal.js': ['Style.js', 'Signal.js']},
    {'async.js': ['Async.js']},
  ))

Useful, I imagine, for very common patterns. Er. “Recipes”. In any case, it’s a very useful kind of tool. Beats setup.py, INI, or XML based automation language any day.

Cock Radio

2007-04-18T11:41:00.000-06:00

This whole Don Imus issue has confused the shit out of me. Talk radio is full of that kind of, um, talk. Anyways, it all feels like the first episode of the latest season of South Park. Sometimes, South Park can crank out a new episode in response to a very recent event, but that didn’t happen here. This episode aired weeks earlier.

As for how or why this Don Imus issue exploded in the way that it did - I just don’t understand (and now I feel like Stan Marsh at the end of that South Park episode). There are so many similar things said all the time by many radio “personalities.”

Media Matters has an excellent post up chronicling the many slurs of Glenn Beck, O’Reilly, and more: It’s not just Imus.

The response to the whole Imus situation just seems wrong: a cause celeb on which everyone can jump. The latest distraction. How the hell did it get so out of hand? Who did it really offend? Why this “nappy headed ho’s” statement? Why not “ghetto slut” (Boortz)? “Turbanned hoodlums” (Savage)?

Imus is probably far less offensive than many of these other radio people, and neither his firing nor all of this special attention is going to make anything better. Nor did it solve anything. It just provided everybody with some bullshit theater.

What about search?

2007-04-17T16:06:00.000-06:00

From my tumblog: but I don't want my search engine to be a slide show!

Google - remember that search engine of yours? How about making it better by offering some options like result filtering ("I don't feel like shopping right now, I'm trying to research")?

Reuse and non use

2007-04-17T11:55:00.000-06:00

We’ve been using Zope 3 in earnest for just over a year and a half now. I would like to report that in that year and a half our little company has achieved more re-use than at any time in our history. This is real re-use too: libraries of tools and objects that are easily shared among both horizontal and vertical markets, yet customized for each customer as needed. Benefits for one are fairly easily shared with all.

In the Zope 2 days, we tried hard to achieve this. But we were constantly having to re-invent the kind of architecture that I believe really makes this work: adaptation, which also brings dynamic view binding, dynamic UI generation (ie - registering a ‘tab’ for a particular object / interface and having it show up in the UI as necessary, etc. We had to spend a lot of time making the frameworks that would let us make frameworks.

“Frameworks for making frameworks?” - you heard right. Let’s face it: most web work is custom development. Sometimes custom development is best served by tools like Ruby on Rails or Pylons, or even by plain old PHP. But sometimes you know you’re going to have at least five customers all needing variations on the same thing in the coming months; and potentially more after that. You’re going to need to at least make a library or two.

See, Model-View-Controller isn’t just about “separating business logic from presentation”. It’s about separating it in a way that you can take business objects and logic (the ‘model’ layer; or models and services) and put more than one view on them. And by “more than one view”, I don’t mean “more than one template.” I mean putting wholly different user interfaces on it. I mean being able to take a base library and override a few select options (or many select options) as they appeal to a customer.

We tried to achieve this on some of our Zope 2 products, but it was hard to extract frameworks. We did OK, however, but I think that the most re-use we ever got was about three or four customers on one toolkit. That was over a three or four year span. We re-used patterns and snippets quite often, but it took a lot of work to extract an e-commerce toolkit from a particular customer’s site, and more work still to make it adaptable and workable for different customer requirements.

In the year and a half since using Zope 3 full time, we’ve had double that - and with far greater results. It’s not an easy system to just start using from scratch, but it can be quite worth it.

Being back at work on some legacy Zope 2 projects has made me all the more appreciative.

By the way: for a simpler Zope 3 development experience, check out Grok.

Tumbling Dirty

2007-04-16T16:19:00.000-06:00

Oh yeah: Dirty Modern. My tumblelog, generally more focused on design, music, etc.. We'll see.

I haven't posted too much in Griddle Noise because it's quite hard, sometimes, to write short entries. I always liked the tumblelog format for being explicitly simple. And Tumblr has an excellent bookmarklet for posting entries.

The Web Will Not Replace the Desktop

2007-04-13T18:52:00.000-06:00

Web 2.0 has excited us because we lowered our expectations so much. Of course web apps will get better, and one day will deliver the functionality we currently get from desktop software. They may even do more than our desktop applications one day. But isn’t it a tad strange that we think this is all a huge leap forward? - loose wire blog: It's Not the "Death" of Microsoft, it's the "Death" of Software

The author's main point is that while it's cool that people are making Mind Mapping tools in DHTML, they're still a long ways behind desktop apps like MindManager. He goes on to contend that there's just nothing exciting in the "offline" world in recent years.

While it's true that the web has made for some neat and very useful online tools, there are classes of software missed. There's a reason why we'll never get to the "every computer is just a web browser / flash player" ideal: professional software. It's a world I'm entering again as I'm finally getting my home studio together.

I'm talking about apps like Pro Tools, Reaktor, Final Cut, DVD Studio, Aperture, Lightroom, InDesign, Quark XPress, etc.

Granted, most people don't use those applications, but I think that it's a growing market. As technology grows and commodifies, we need tools to deal with it. The gap between pretty-good consumer gear and pretty-good entry level professional gear is pretty small now in many areas: digital photography, digital video, music, etc.

Now that I think about it, Apple has realized this for some time. They have a pretty good upgrade path. For those who get hooked playing with video in iMovie, there's Final Cut Express at a rather reasonable price. Those who really start to do well with that can go up to the full Final Cut Pro. For Music, there's Garage Band, Logic Express, and Logic Pro.

Within those realms, there's a huge array of plug-ins, virtual instruments, specialized sound tools and environments (Ableton Live, Reaktor 5, Max/MSP, etc). I am amazed at the sounds I get out of Reaktor, and that's only a single product in a single company's impressive set of offerings.

Perhaps the new web applications are freeing up resources on our own machines so that it no longer feels like some dreadful work environment, just at home. There are plenty of useful and usable online tools for doing quick writing, sharing, interacting, thinking, and planning. Typically they offer enough to be usable for those small (or even medium) jobs we occasionally encounter, while freeing us from having something overkill like Office for casual, personal writing. It's easier to specialize a computer for audio work by fine tuning system settings, throwing away silly applications, etc, without making that computer into an island. As long as you have a web browser, you can still check email, contribute to a planning document, etc.

But honestly, I don't think the Desktop is going to die - ever. It's great that we can do so much on the web, but I don't think the native experience is going to die, ever.

And even if you're not on a fucking plane, it does matter: when I moved into this loft, it took me a couple of months to get internet access down here. I was working on a lot of things for the office at the time, and I was able to take it home by just using my laptop: at work, I'd synchronise source code, copy stuff to my laptop and/or sync with .Mac's iDisk, and sync with .Mac for my calendar, etc; as such, even though I was offline, I could work. It was then and there, however, that I decided that although I liked Backpack, it wasn't worth paying for: I needed offline access. I needed, well, OmniOutliner and Tinderbox. My personal project files and note-taking documents are just too precious to be left online (this is why I don't and won't use Stikkit). If the occasional monster storm comes along and takes away the Internet for a few days, the worst feeling in the world would be being disconnected from my notes.

Strangely enough, these online note-takers, organizers, etc, all solve a problem that has plagued me until quite recently: how to do effective sharing of data between home, work, and laptop? How to not get out of sync? I love Tinderbox and I have a couple of big Tinderbox files that I keep on .Mac's iDisk. This means I usually have access to it. But sometimes, I forget to sync or close or save the document when I leave work or close the laptop. What about the little bits of random data, not yet filed, or not really worth filing into that larger document? How can I quickly enter, find, and share that info?

The answer didn't come from any web service, although lord knows I tried a few. The answer came when Tinderbox's developer, Eastgate, ingeniously started bundling Barebones' Yojimbo with Tinderbox. I had looked at Yojimbo in the past, but I'd gone through so many personal note taking / note capturing / note filing systems (Mac OS X has MANY). I didn't want to look at another such product and be fighting between "do I file it in Yojimbo? In Tinderbox? DEVONThink? Can I get to it from home?"

But Yojimbo has a killer feature: it syncs with .Mac! The same tool that I've used to keep calendars and contacts and Safari bookmarks transparently shared between three machines, finally someone made one of these note tools that took advantage. Now I have my enter-a-quick-note, file-it-later system that gets updated and merged automatically. No worries about having an out-of-date iDisk, about forgetting to save and sync. And best of all - it's 100% native and usable offline. And it doesn't get lost in the army of tabs since every goddamn web "app" is now just something that gets lost in a browser window (for those who wonder why I take so long to reply to mail sent to my GMail account, well, GMail sucks as an application compared to a native mail app. I just don't watch it regularly enough to stay on top of things).

So, anyways, I love a lot of the new web apps. But people need to get a grip. I've been hearing about "the death of the desktop" for eleven or so years now. Stop tricking yourself into thinking you're that high and mighty. If you don't understand the true value of native applications, professional applications, personal data, then you don't understand the desktop's power. As such, you're not going to kill it. Yes, please focus on tools that take work well on the web, tools that are simple yet useful. But don't think for a second that I want to spend every second of my computing day in a web browser, nor do I want to spend every second in an Apollo client. It's not going to replace everything, any more than Java, Netscape Constellation, or even Active Desktop did.

The Saddest Music in the World

2007-02-23T11:55:00.000-07:00

The Saddest Music In The World just joined the list of favorites-among-favorites-of-movies.

Surreal, beautiful, soft, and fuzzy, like Institute Benjamenta; but with a classic old showbiz quality and language like Sweet and Lowdown; and a bit of melancholy [even a bit of brutality] like Dogville, Shadows and Fog, and Eraserhead.

It fits in so comfortably with these other films that I love; yet stands on its own in a way that makes me terribly happy as it feels like it glues them all together. It's this one piece that makes all of the above and more (2046, City of Lost Children, etc) all the more lovable.

Isabella Rossellini's beer baroness with the platinum wig is beyond hot. Especially when she shows up with beer filled glass legs.

And Maria de Medeiros is beyond cute as the depression era dark bobbed amnesiac singer and dancer. I guess she's probably best known in the U.S. (if at all) as the wife of Bruce Willis's character in Pulp Fiction. Here, she seems straight out of a Chaplin or Keaton film, while capably leading a spectacle performance of The Song Is You mid film.

I love getting lost in spectacle.

Flex my Dojo

2007-02-23T00:51:00.000-07:00

OK. I don’t get Flex. By my understanding, Flex is the components / widgets from or for Flash. Actually, I can’t find a definition. I’m hitting the main pages, and they all treat me like I already know what Flex is and why I need it. But I don’t. From the buzz I see around the web, it smells a lot like the promises of Java Applets circa 1996. It’s the universal platform - rich internet applications! cross platform! run everywhere! It’s in the browser, so you lose yet another tab / window to some application that’s not quite a full application!

As a life long Mac user, I’m dubious of any cross platform promise. Java with some cross platform toolkit or another has never been a great experience on the Mac. Eclipse fares fairly well, but I’m still very aware that I’m in a different environment. This has something to do with look, but really it’s about feel. There’s a way that I expect my widgets to feel: ways that I can move around in a text field, responsiveness of a button, and so on. Cocoa’s text fields benefit from their NeXTStep ancestry which has provided some powerful and extensible features: basic EMacs keys (ctrl-a and ctrl-e to get to begin and end of line, ctrl-k and ctrl-y for kill and yank to a separate copy/paste buffer); spell checking; in place dictionary lookup; often, a common set of search/replace features including ‘use selection for find’; word completion; etc. Input Managers can be installed for even more control. I used an Input Manager to get word completion in early versions of Mac OS X, before it was a built in feature. Now I use “Edit in TextMate” to edit HTML TextAreas, emails (from both GMail and Mail.app), even to tweak a filename using TextMate’s editing tools. Any Cocoa text field, whether it’s a single line or a large block, can be wired this way without the application’s knowledge.

In cross platform toolkits (as well as most Carbon apps), I lose these features. XUL (Firefox/Mozilla); Java AWT, Swing, and whatever the hell Eclipse’s toolkit is called; and now: Flex. Don’t get me wrong: I like that these toolkits enable some apps to exist on the Mac at all; but I’ve seldom been as productive in them as I have been with native apps.

I took a look at a Flex Quick Start page covering “Building a simple user interface” to see basic Flex in action. As I feared, the controls followed a different set of rules than those I expect. Since the tutorial covered only basic UI, there wasn’t much that was terribly impressive. I didn’t see anything that stood out from what I had seen in rich JavaScript/DHTML toolkits like Dojo and YUI.

So I decided to do a quick comparison - how did some of the basic Dojo widgets compare to the basic Flex ones? I went to Dojo’s “See it in action” tab and looked at the FormTour, which housed some basic widgets including one of the few “rich text editors” that works in Safari. And sure enough, even though some of these widgets had some extra behavior wired in, they were still built on browser native elements. All of my usual keyboard behavior worked as expected in text fields - even in the rich editor.

I can see how Flex may be useful for fast dynamic graphical components, such as charts. Beyond that, I feel like it’s another round of the magical cross platform kool-aid; another round of trying to shoehorn a sub-universe within the sub-universe of the web browser. I think it has its uses, but some people seem to think that it’s the new holy grail of rich internet applications.

I’ve heard this song before, and I’ve been horribly disappointed almost every time. So far, all I’ve seen of Flex is a retread of the promises of Java Applets. I would like to see statements about what makes Flex different. Has it learned from Java Applets and the various “abstract” Java toolkits that promised a common UI everywhere? Does it have a focus, or will it devolve into servlets and enterprise architecture? Is Flash still an animation platform, a movie player platform, or a GUI/Form platform? Can it handle all of the above? Will Adobe be wiser than Sun and avoid big clashes with Microsoft?

My gut says not to trust it.

An update on Python SOAP toolkits

2007-02-13T08:37:00.000-07:00

Last week, I struggled with implementing a rough SOAP service in Zope 3 and Python. At the time, the state of SOAP libraries and toolkits for Python appeared to be in rough shape. The only toolkit that seemed to cover enough ground was ZSI, which also had a recent 2.0 release. ZSI, however, is a sprawling and incoherant mess. Given the SOAP, WSDL, and WS-* specs, who could expect anything more? It was a difficult system to work with as I had to do some brutal massaging of typecodes in order to get my client application to respond.

At the end of last week, TGWebServices appeared on the scene. TGWebServices brings XML-RPC and SOAP to the TurboGears web framework. It looked like it was much easier to use than what I had to cobble together.

And this week, soaplib showed signs of life with a new release. Soaplib appears to be easier to comprehend than ZSI, and it feels more modern. I wish it had been available a week ago.

SOAP. Web Services. Flash. Waaa.

2007-02-10T09:26:00.000-07:00

As a result of a peculiar chain of events, we’ve been stuck with a compiled Flash file requiring data off of a web service. The old database is gone. The old server is gone. We have been unable to get much information about the data required. We don’t have the Flash source. All we have is a single ColdFusion Component that is used a web service via WSDL and SOAP.

I’ve never used ColdFusion. Fortunately, Allaire, er, Macromedia, er, Adobe makes demos available. ColdFusion even comes with a developer version that can live beyond the typical 30 days. So using that, I was able to re-generate the WSDL. I didn’t understand it, but I had it. Then I installed a demo of Flash Professional. I made a quick document and put some web service links in it by stumbling around, and watched the HTTP/SOAP communication between Flash and Apache/ColdFusion.

I tried looking at the specs for SOAP, and recoiled. Then I naively thought I could emulate the response as a Zope Page Template emitting XML, looping over the little bit of data that the Flash client wanted. But then I realized that the SOAP style in use was RPC style, and the ‘method’ was buried in the posted XML, so I couldn’t even easily answer the incoming requests.

Since there’s no standard SOAP toolkit available for Python, Zope 3 only holds a skeleton off which one can hang a SOAP handling REQUEST/RESPONSE pair. I’m too dumb, or perhaps just too busy, to wrestle with XML parsing myself, and I quickly realized that I needed some kind of toolkit. The data that I saw in the WSDL and SOAP communications appeared far too complicated for me to read and write on my own. Again I tried going to the specs, and again I ran rapidly away as I couldn’t even figure out how the WSDL connected to the SOAP requests and responses.

Now, obviously this stuff must work out for somebody. When we first approached this problem, the advice that we got was along the lines “just put up an XML gateway.” Um, what? I then realized that the tools were probably doing everything and the developers with whom we were communicating never gave much thought to what was going on. Data format? There is no data format - just put up an XML gateway!

I can’t fault anyone for this, per se. We all get abstracted away from something. Many web developers plug along happily without knowing too many details about HTTP. Even when one knows quite a bit about the more advanced or nuanced details of HTTP, one’s not likely to know much about what’s going on at the TCP/IP layer.

While stumbling around Flash Professional, I was a wee bit impressed with how quickly I could make a web service connector, wire it to a table widget, write a quick button-click handler, and could see the results in the table. The ColdFusion Component had its WSDL generated by merely adding ?wsdl to the end of the URL. Since both tools are maintained by Macromedia, er, Adobe, the SOAP messages actually seem to work together.

But my gods, it sucks.

I have been so spoiled by HTML and Python, and by Javascript, CSS, Zope, etc. There’s no end to the value of the web browser’s “View Source” command. With Python, I can read the source of almost any library and figure out its innards if I can’t get answers from the documentation. CSS and JavaScript are equally open to public viewing. One can easily send plain text as results of HTTP calls - I think that if we had designed the system we’re wresting with right now, we would have served some kind of delimited text that any tool could read - even human eyes. This week I realized just how hard it is to make a simple SOAP request by hand, leaving me to use a TCP watcher/proxy to test requests and responses.

Using ZSI from the Python Web Services project, I was able to mash together a basic SOAP Publisher for Zope 3. It doesn’t deal with multipart messages, WS-Addressing or whatever the hell that is, “document/literal”, or even Request argument processing. Well, that’s made available on the Parsed Soap instance from ZSI that’s put on the request. The handlers - the remote procedures / “controllers” / “views” / whatever you want to call them, still have to do quite a bit of work themselves, at least with the Parsed Soap message. This is because, as far as I can tell, one can’t do much without using explicitly build custom types, generated from tools like wsdl2python. Ahhh, this takes me back to all of the crap I had to deal with in ILU/CORBA - baroque generated code that one couldn’t live without, nor was likely to understand.

Even after all of that, I had to do a lot of ZSI typecode wrangling in said generated code before I even began to get results like what was generated from ColdFusion. And these are simple calls, really - no request arguments, and the returns are just lists of dictionaries. I stand amazed at just how much work is involved with getting these little details right, even with a toolkit.

Even with that toolkit’s aid, the status line in FireFox would read “Waiting for data from…” This was puzzling, as I thought had finally mangled a seemingly proper Map and Array typecode / values together from the available tools. I started to worry - is there something at the HTTP level that SOAP is doing differently? Does it want chunked results? Is it not closing the connections? I started crawling deep in the networking guts of Zope and Twisted, trying different transfer encodings, verifying that the content-length header matched the content length in the response … nothing.

Then, on a whim, I decided to try dimensioning my array. ZSI’s array, by default, doesn’t seem to handle specifying some kind of variable length in the array type. So it was generating xsd:anyType[]. The ColdFusion results had xsd:anyType[n] with n being the length of the array. I finally got this working by dynamically altering half of a tuple on a typecode that descended from the strange Result type/typecode that I was building. I’m not even sure if this is thread safe, come to think of it. But by finally getting that dimension in there, Flash started to respond again.

Ugh. Remember, The S Stands For Simple.

That kind of development - closed, compiled, heavily dependent on tools like Flash that work on few desktops and cost a good chunk of money - is just so foreign to me. I’m no die-hard open source zealot: I love a lot of the commercial tools and applications for Mac OS X, and would rather use it than any of the “free desktops”. But when it comes to web development, I’m used to dynamic languages and source code that is often readily available or shareable without the need of heavy tools. This is (usually) code that I can read, which can teach me new techniques by real implementation and not vague “hello world” examples followed by nightmare specs. I’m used to the layers below that being either so lightweight or established that I don’t have to think about them much. I’m grateful that I still seldom have to think in XML. I’m glad that I have next to none of the endless Java acronyms memorized. Sure, I guess there’s a place for them all, but I’m happy that I’ve been able to make a living for the past ten years without all of that. And I’ll be happier still when this particular problem is behind us.

Damn Fine Coffee

2007-02-07T10:28:00.000-07:00

My David Lynch Coffee arrived today. I won't get a chance to try it until tomorrow. But damn! I'm excited.

Zope 3 drives me back to smoking; Pylons may be the sleeper hit of Python web systems

2006-11-08T15:41:00.000-07:00

So, my recent problems with Zope 3 that were making me want to resume smoking? Yeah, I’m a smoker again. Two months clean. The stress just got to be too much.

I reported my problems to the community, with long and detailed explanations and explorations and pleas for help. The responses? Very few. Far fewer than I would have hoped. I had to scrap upgrading a customer to Zope 3.3 after losing a day of work trying to upgrade their database, unsuccessfully. Back to 3.1 they go. And now maybe we’ll forget about this for another year and have an even harder time upgrading next year.

Combined with another day lost chasing an obscure bug (moral of the story - sometimes saying that you implement the interface you implement can be bad, at least in cases where people do identity checks against self when attempting to retrieve keys out of a system), I’m pretty pissed. I really want to leave it all behind. I can’t. But right now, I’d like to. Even when Zope 3 is working, there are some common activities that seem to take way too long to accomplish. I feel like I’ve been spending a vast chunk of my time in recent months focusing on building tools and classes on top of Zope 3 that could bring at least some of what I see and like in Rails and Pylons while taking advantage of Zope 3’s strengths.

I think Zope 3 can be good for CMS type applications. And even for big business and public applications, it can be a good system. The so-called “MVC” stack of the new breed of systems bugs me. I have mixed feelings on templates these days. I think that with good objects, one can generate simple HTML out of Python (or any other programming language) which can then be styled up with CSS. It’s not appropriate for all situations, but it can be a great tool.

But, I must admit, I like what I’ve seen of Pylons. I installed it yesterday and ran through a couple of the tutorials (not even getting to the Wiki tutorial). Last night, I decided to convert the Basic To-do Application that I wrote up last year to Pylons, using SQLAlchemy as the storage mechanism. Fortunately I had some recent (and inspired, if I do say so) SQLAlchemy work done for Zope 3 whose ideas I was able to borrow to get the basic Alchemy session and engine management working in my application. I had just a single table and thus a single model class (keys: ‘id’, ‘description’, and ‘done’).

I decided to go for the same minimum of functionality that I had gone for in the Basic To-do scenario: have a single page that lists completed and incomplete items, with a small form to add new items to the system.

I got this done - including the writing of the simple SQLAlchemy bridge - in about an hour or hour and a half. No time wasted writing ZCML. No time wasted turning off the ZMI. It was generally quite easy. I made a few faltering steps along the way, but not many.

I think Pylons may be the sleeper hit out of all of the Python “web frameworks”. TurboGears got to be overwhelming, immediately. I think that I’ve just never been a fan of its core components. No offense to their authors. Django is also a bit overwhelming. It’s more like Zope: more of a complete environment. That can be great for many uses, but it can be a hindrance for others. I think Django is a terrific system from what I’ve seen of it, but it’s too big for me. I have one big system in my life already.

I have a bit more admiration for the core bits of Pylons. While I’m not a big fan of its syntax, I do think that Myghty (a port of Mason) has a substantial amount to offer beyond just being a templating language: namely lots of caching options and intelligence. SkunkWeb was the only other Python web system that I’ve seen that seems to make caching such a priority. The alternate templating languages one can use in Pylons - Kid and Cheetah - are not at all appealing to me.

WebHelpers are alright as a collection of useful functions for use in templates and controllers. I’ve had a blast taking the helpers concept and implementing it in our new Zope 3 based sites. It makes a big difference. With a good base set of helpers that are easy to expand on, it becomes much more desirable to use Python expressions straight up in templates. When I saw the ERB code in many Rails projects, I was impressed - it didn’t look like the typical mess of common PHP / Classic ASP style development; nor did it have to create yet-another-language on top of its base with hard to learn bind / call semantics. One doesn’t have to translate::

<% h.text_field(name='description', size=35, maxlength=100) %>

into “what Python or template or other thing does that there code call?” compared to::

<f:text name=quest label="Question:" />

The latter coming from Spyce, from an entry on the Spyced Blog.

Actually, I was starting to want to use Spyce in my Zope projects in spaces where we had Helpers doing many common tasks (like seen above) and ZPT was way way way too verbose:

<input tal:replace="structure python:view.h.text_field(...)" />

But Spyce has grown up into a richer system, where all I really want is a good basic Python Server Pages style system. But I’ve never come across one that I’ve liked: something small, simple, fast, easy to bind, meant to work with others (ie - not meant to be standalone pages with big blocks of Python, but rather to work with a view/controller class and helpers).

Back to what I like about Pylons - the last two are Routes, which does the URL dispatch (writing maps like in Rails and Django); and the aggressive use of WSGI / Paste. These seem to be bringing the Component Architecture dream of Zope 3 closer to reality as there are additional filters and middleware one can put around one’s application. Paste provides all sorts of cool features. There seem to be a lot of options for interchangeability.

I’m hoping that Zope 3’s “eggification” will make it easy to write Zope 3 style applications without having to use Zope 3. There are a lot of great libraries and toolkits in the heart of Zope 3 whose liberation would be most welcome. Even without that, I hope to look into what might happen if Zope 3’s libraries were just added to the Python path so that a Pylons application might be written up, using Zope 3’s concept of Views (Multi-adapters) and Transactions with objects pulled from SQLAlchemy without all of Zope’s security and nonsense.

Looks like I picked the wrong week to stop ...

2006-11-03T11:28:00.000-07:00

There’s a great set of lines delivered by Lloyd Bridges in Airplane:

Looks like I picked the wrong week to quit drinking.
Looks like I picked the wrong week to quit smoking.
Looks like I picked the wrong week to quit sniffing glue.
Looks like I picked the wrong week to quit amphetamines.

Sometimes I love Zope 3. Other times, it makes me start to sound like McCroskey. I haven’t smoked now in two months, but I just smashed my head into another foolish and crappy scenario in Zope 3 (after discovering an incredibly crappy scenario earlier this week). Too stressed and confused to think clearly and engineer / hack my way around this latest situation, I just want to grab a fresh pack of Lucky Strikes (unfiltered, of course).

Waaa.

Namespace Packages in Python

2006-10-31T07:22:00.000-07:00

Ian Bicking wrote a Zope 3 Critique a couple of weeks or so ago, partially in response to a critique by Jean-Marc Orliaguet. I’ve wanted to write a response of my own, but time has not allowed it. Now, I’d like to use a few posts to post my own responses to a few items.

The first point I’d like to address comes under headline 3, The importance of standards:

I personally don’t care much about namespaces. Is zope.formlib any more “right” than zope_formlib? Is it any more hierarchical? I don’t think so. I have one namespace package, but it’s kind of a historical accident. I wouldn’t (and don’t) put any more libraries under that namespace. Each library should stand on its own, so why create these hierarchies? People read too much into those namespaces, which is happening right here.

Ugh. zope.formlib is far more “right” than zope_formlib. Granted, this is largely opinion. But man, do I love namespaces. zope_formlib looks like one of those crap-ass hacks and tricks used by the litany of languages who have no namespaces and have to fake it.

First off - I hate the underscore, and believe it should never enter package / module names. Mixed case package/module names are also evil. PEP 8 recommends this:

Modules should have short, lowercase names, without underscores. … Like modules, Python packages should have short, all-lowercase names, without underscores.

While I don’t follow PEP 8 too religiously, I’ve found my life to be generally better by trying to stay close to it. I believe that short lowercase names without underscores are a nice constraint, requiring one to think a little bit about what a module/package will entail and how it will be used. If it’s a module that will be reused frequently, it’s good to have a name that’s easy to remember and type.

At least we don’t see as much of the following any more:

from DocumentTemplate.DocumentTemplate import DocumentTemplate

ugh.

Finally, Zope 3 is a large system. I make fairly large systems using many of its pieces, some more than others. Some of those are quite nice to use from an import standpoint when one just imports the sub-package:

from zope import component, interface

from interfaces import IBar

class IFoo(interface.Interface):
    def newbar():
        """ returns 'ha!' """

class BarAdapter(object):
    interface.implements(IFoo)
    component.adapts(IBar)

    ...

That’s a trivial and nearly pointless example, but it’s a pattern I find myself using more and more. I’d be a much less happy mac if I had to do the following:

import zope_component
import zope_interface
import sqlalchemy_sql

Ugh. Why not just introduce crap like ‘require’ and ‘use’ and leave this modularism behind?

There has been one problem, historically, with Python’s packages: unlike Perl, Ruby, Java, etc, the namespaces in Python are based exclusively on filenames and file system hierarchies. That has meant that a “namespace package” like zope.* would have to include everything. It couldn’t be distributed and used as smaller chunks easily, whereas if this were Java the packages/modules would use the ‘package’ keyword:

package org.zope.component;

But now SetupTools allows one to break these pieces up and distribute them separately while still getting the organizational and conceptual benefits of big hierarchical namespaces.

Simple Nuggets - Recent Interests in Seaside, Io, and more

2006-10-30T07:33:00.000-07:00

I know that I haven’t posted much lately. I do have a couple of long draft pieces saved that I haven’t had time to finish. I don’t know if I’ll get back to them, but I’ll try.

I’ve been looking at a lot of different things recently, looking for new inspirations, tools, and/or chances to widen my understanding. I find myself growing increasingly fond of a couple of things in particular: Seaside on the web development side, and Smalltalk, Scheme/LISP, Io, and Javascript (particularly with the Prototype.js library) on the programming language side.

Seaside interests me on many levels. I’ve been staring at it with a lustful eye for some time now. There is definitely a place for heavily stateful web apps. Seaside and WebObjects make this a part of their core, where a developer can make an application in a style that is more like a desktop application: less dealing with sessions, freedom from the typical request-response cycle, etc.

But Seaside is more than continuations and callbacks. I love how you generate HTML in it: NO TEMPLATES!!! In a post titled “Turtles All The Way Down”, Ramon Leon writes:

Seaside lets me work in Smalltalk, at every level, all the way from programmatic generation of the HTML and Javascript to configuration of the application. No HTML, XML, XSLT, or SQL is necessary to build a web application, just simple, pure Smalltalk. Programming’s never been more fun!

I’ve made some in-house tools that get us closer to that goal when using Zope 3 and Python. I’ve started to hate template languages again. Well - I don’t mind using them for layout intensive work. But as I get more proficient with CSS, even that is changing.

[Edit: Moved side rant about Zope and Views and Templates out... This isn't the post for it]

It still stare lustfully at Seaside code, especially things like this very basic example:

renderContentOn: html
    html heading: count.
    html anchor callback: [self increase]; text: '++'.
    html space.
    html anchor callback: [self decrease]; text: '--'.

Ooo. Two links are made with anonymous functions that call self increase and self decrease respectively. There’s no “absolute URL”-ness going on. There’s no “how do I make it so that increase can be called through the web?”. No exposure decorator. No seven lines of XML code each for ‘increase’ and ‘decrease’ to be made into separate views. And no template. Just a solid binding between a link on a form and real code.

Seaside is a testament to the power of Smalltalk. People think that Seaside is all about continuations, and if they get just that part down, they’ve made something to compete. But it’s more than that: Seaside is a tool for building web applications in pretty much the same way one makes a desktop application. It’s not the tool or style for every job, but it definitely has its place.

In a recent bit of work we've done in Zope 3, I would have loved to have something like Seaside, especially for the checkout process. I've tried to engineer that process as well as I could, with extra preconditions and redirectors on pages to ensure that all desired information is supplied (ie - so that one can't jump from step 1 to step 4). With Seaside I could have encompassed the in a WATask component, expressing the checkout workflow as normal Python code: while shippingAddressRequired and (not order.shippingAddress): yield self.askForShippingAddressForm() (or something like that). The shopper could never accidentally (or purposefully) get past that point without supplying the right information. Doing this with preconditions, sessions, hidden form fields, cookies, etc, is all possible. But it can be a lot more code overhead.

Back to the other interests listed, really quickly: I’ve used LISP on very rare occasions (for customizing EMacs, basically, back in the day). I’ve never quite understood the appeal that LISP / Scheme have for some people until recently. While I enjoy and will probably continue to work in general purpose languages like Python, I’ve started to grow fond of these languages that are built on very simple precepts. Smalltalk has practically no syntax: it’s all messages. And I really like that Squeak is written in itself - there’s just something impressive about that. On a related topic, I checked on the progress of PyPy and was very impressed. Like Squeak, which has a small subset-of-smalltalk-to-C translator for generating the core pieces from Smalltalk itself, PyPy is Python written in Python, with translators to lower-level languages.

Regarding Scheme, I found a Scheme translator for Smalltalk and have spent a small amount of time playing with it this weekend. I came away impressed with how easy it is to parse and process Scheme, and how easy it is to implement it and its primitives in quite easily readable Smalltalk. The “everything is a list” (‘car’ and ‘cdr’) power finally came through to me as I browsed through the Smalltalk code to see how things that are keywords and operators in other languages (‘+’, ‘if’, etc) were implemented by just using car and cdr (the head and remainder of any list/pair, on which all things are based).

I’ve also become very interested in prototype-based programming. With prototype based programming, one has Objects but without classes and metaclasses and on and on. I’ve known about prototype based programming for some time, but I’ve only recently started to appreciate it. It’s radically simple. Looking around at Prototype based languages, I came across the relative newcomer Io. Io has no keywords - which, again, I find quite interesting and provocative. And powerful. It’s message based, which I like, and it has no classes. You just clone something to make something else. A simple object example from the sample code page:

Account := Object clone do(
    balance := 0
    deposit  := method(v, balance = balance + v)
    withdraw := method(v, balance = balance - v)
    show := method(writeln("Account balance: $", balance))
)

myAccount := Account clone
myAccount show
"Depositing $10\n" print
myAccount deposit(10)
myAccount show

Oh yeah - and the minimalist in me swoons over its web site.

There it is - a scattering of a portion of the thoughts that occupy my brain when I have the chance to take a breath at work.

Browser Side

2006-09-26T14:06:00.000-06:00

I’ve surprised myself by starting to enjoy doing browser-side development in CSS and JavaScript more than I enjoy the server side. Prototype, Scriptaculous, LowPro, and a comfortably growing in-house library have gone a long way towards helping with this shift, along with finally getting down into the guts of CSS and learning about the box model. Tools like FireBug and the Web Developer Toolbar for Firefox have helped out tremendously, as well as native Mac OS X tools like XyleScope and direct linking to the CSS specs from within TextMate (very helpful when wanting to learn the real details about any given property).

Iterators, Generators, Array Comprehensions, and More for JavaScript 1.7

2006-08-16T09:14:00.000-06:00

I saw a link today from Hack The Planet titled “What’s new in JavaScript 1.7: Lisp/Python!”, pointing to a “new in JavaScript 1.7” page at developer.mozilla.org.

So, of course I had to follow this and see what he meant. Sure enough, JavaScript 1.7 starts to look a bit more like Python in some places. From the “What’s new” document, this is an Array Comprehensions example:

var ten_squares = [i * i for (i in range(0, 10))];

Take out the word var, and that’s 100% valid Python (although Python can get by here with less parentheses and without the semicolon).

Iterators in JavaScript 1.7 are also very similar to Python’s, with objects providing an optional __iterator__ slot. Iterators just implement a next() method, and raise StopIteration when done. Generators turn functions into iterators by using the yield keyword.

Very interesting. For a while, I was a fan of the MochiKit JavaScript libraries. MochiKit takes a lot of inspiration from Python and some Python toolkits like Twisted. Among other things, it provides MochiKit.Iter, which brings a lot of the Python itertools functions as well as basic Iterator concepts to current JavaScript, if desired.

I must admit, however, that while I like MochiKit for its general philosophy and the fact that it’s one of the best documented ‘New Age’ JavaScript libraries/frameworks/toolkits around, I’ve gone back to using Prototype.js for most of my work. MochiKit is big. It’s powerful, but big. Yet I found myself straining to copy over some really useful little functions from Prototype, some done while trying to make the Ajax calls a little easier to work with. Ultimately, I find that my Prototype.js based code tends to be easier to read, write, and comprehend. I’m a JavaScript outsider, and Prototype.js pulls enough tricks to make programming much more familiar and convenient. While it’s not well documented, its source code is actually quite easy to read. Both toolkits are nice, but I now reserve MochiKit for heavy-lifting situations only.

Congratulations, TextMate!

2006-08-09T14:48:00.000-06:00

Congratulations to Allan Odgaard and everyone else involved with TextMate on winning an Apple Design Award for Best Mac OS X Developer Tool. TextMate has so greatly improved my development life, finally allowing me to (where possible) work on my desktops / laptop instead of on a development server as I’ve done for so many years.

A little bit of my development day today: Using Mac OS X terminal to test a new date-range formatting function. The results were clipped directly into a Circus Ponies Notebook page without leaving Terminal via Mac OS X Services. All of those clippings were then brought into Hog Bay Software’s excellent WriteRoom where I went into full screen mode to turn the test results into documentation that could run via Python’s super-cool Doctest tool. Using WriteRoom for this allowed me to focus on writing the documentation without distractions: it was like being back in the old old 70’s terminals: green text, black screen. No syntax coloring, no modes, nothing.

A couple of times while writing the documentation, I was able to use the globally installed Edit in TextMate… command that lets just about any Cocoa text field / area / etc be editable in a TextMate window. I used this during the main writing to do some big find/replace runs (some names that I had imported in the terminal were redundant in the test documentation). So, from WriteRoom, I hit Edit in TextMate… (actually, I just hit control-command-e) and a new TextMate window opens up with the contents of my WriteRoom document. I used TextMate’s nicer search/replace tools to do some mass fixing, hit ‘save’ to send the results back to WriteRoom, and closed the window - which caused a direct return to WriteRoom.

Again, this was nice for this situation since I could use a specialized no-frills tool to focus on the words I wanted to write. When I was done with the writing, I copied everything out of WriteRoom and went back to TextMate to paste the documentation in the docstring of this function. At that point I could use TextMate’s tools and powers to format the results (wrap / indent properly, apply some reStructuredText styling, etc).

What a cool little collection of tools.

And happily, when the documentation test was executed, it ran on the first pass.

Leopard, Fake Screenshot Winners, and Dreams

2006-08-03T17:06:00.000-06:00

These are absolutely amazing. Phill Ryu held a Fake Leopard Screenshot Contest in anticipation of next week’s Apple Developer Conference which will unveil the next version of Mac OS X. Some of these people put a lot of time into these designs and some of theme are quite interesting. The winning shots by Eric Patterson are quite nice, going for user experience: more views of data “your way”, nice collections of information. Some of it reminds me of the pie-in-the-sky Longhorn screen shots showcasing all of the power of Windows Vista.. well, all the power that was hoped… And Eric’s tabbed finder is about the only mock-up that I’ve seen that I would use.

Honestly - why are people so obsessed with a tabbed Finder? I like the Finder browser (metal) windows, I like Column view, but I don’t want tabs. What I do want is “Open Folder in New Browser Window” so I can branch off at any time. What I would like - a shelf. The sidebar in the Finder is nice. I use it like Tabs anyways, but it’s better. It’s so easy to throw folders on there for one time use for easy copying. And it’s useful when working on long-term projects for rapid access to documents. But it doesn’t deal with drives going away and coming back very well.

Anyways, back to the fake screenshots. The final runner-up is acknowledged as being the most out there, and the most controversial. I like this guy for - ahem - thinking differently. Stephen Sciliano wraps everything up into a single window interface, with multiple panes. I’ve seen some Unix window managers that do similar, with a heavy focus on being text-only. I think I know some people who are big fans of it. Sciliano’s mock-up reminds me a bit of OpenDoc gone wild, and reminds me of Microsoft’s new UI for Office 2007.

Another runner-up, Nathan Ziarek, focuses heavily on metadata - particularly on making metadata easy and obvious. Among other things, his shots showcase being able to associate files and other items with To Do items in iCal, and associating files with contacts so that viewing them from the contact view can link to associated files and other objects.

And that brings me back to one of my favorite Longhorn images (which I haven’t been able to find) was one that showed a Contact. Well, not a contact, but everything associated with that contact: IM chats, phone calls, files, events, etc. In one window, nice and clean. I think it was a WinFS showcase item. I haven’t seen anything like it in Longhorn / Vista screen shots over the past couple of years. That’s something I would like: ways to just group a lot of different information together without requiring a third party app.

I want this most in the home studio. I have lots of audio files and (now) a rapidly growing iPhoto library. The audio files at home that I’m concerned about are not the songs in my iTunes library, but all of the work for Eucci, aodl, etc. Most of them are raw little audio bits, not fit for a loop library or management through a dedicated audio program: they’re AIFF’s of various lengths in various states of processing. I use labels (the little color codes in the Finder) to highlight some of them, such as completed pieces or raw sources (usually the two kinds worth keeping around), but they still tend to get lost, especially if I’ve been away for a while. It would be nice if it were easier to tag those files, to group them dynamically, to associate them with a particular contact to whom I’ve promised an album, to associate them with a to-do so that I have a better sense of what I was doing if I have been away for a while. “Oh yeah, I need to just finalize the levels on that track and I can send it to …”.

Scope, Hog Bay Software, and Knowing Your Place

2006-07-27T11:54:00.000-06:00

The staggering array of personal shit management (PSM) tools for Mac OS X seems to be growing, constantly. There are Outliners, Notebooks, personal hierarchical databases, and so on.

Many of the tools may look similar on the surface. Besides downloading all of the trials or crawling through the archives of About This Particular Outliner, how can you decide which tool may be right for you? Many companies do nothing but expound on their particular feature set, but when there are many similar features between applications, it’s not always easy to filter out the two or three items that make a particular application different from substantially different from its competitors. Some companies have comparison tables between their product and one or two (usually popular) competitors. But these tables often show the main company’s product as doing more! more! more! for the same or lower price. And while that can be helpful when evaluating your options, it’s not exactly unbiased research that you’re reading.

That’s why I was impressed yesterday when I took a glance at the page for Hog Bay Software’s Mori. The lead-in paragraph and accompanying screen shot didn’t sell me on it. I have too many “digital notebooks” as it is, and the screen shot reminded me of DEVONThink. But I scrolled down the page, skimming the text and looking at screen shots, and at the bottom of the page I found a paragraph with a lead-in Is Mori the best choice for you?

Buying a note manager can be a daunting process with many choices. On one hand you have great outlining programs. If your goal is lists and outlines, try OmniOutliner. On the other hand you have powerful and complex note databases. If your goal is to create reference database of “everything” try DevonThink. But if you need something in-between, give Mori a try. We think it’s the ideal place for your day to day notes, projects, and activities.

Mori Product Information Page, Hog Bay Software, viewed July 26 2006

Wow. I think that’s refreshingly honest. It gives Mori a good scope, and it differentiates itself by saying “if you need more than lists, but less than a database of everything, we’re your app. But if you need those things, by all means, go next door and get them.”

Hog Bay Software now has my attention. Looking at their other offerings I noticed the application Clockwork, a computer timer program. You know, the buzz-me with an alarm, remind of this, etc, kind. Its page also has a “right choice for you?” paragraph. They also are the developers of WriteRoom, a “distraction free full screen writing environment.” WriteRoom is one of those applications like 37Signals’ Writeboard. Simple plain text writing, no formatting toolbars, no wysiwyg (which is still painful in most browsers), just words. 37Signals has a nice post about the difference between word processing and writing. They are different tools with different scopes. For the time in writing when the words matter more than layout and formatting, it’s wonderful to be distraction free.

And it’s refreshing to look at a software page and wonder “but why would I use that if there’s this tool and that tool and that other tool available?” only to find an answer that says “hey, that other tool might work better for you if you’re looking for something that does feature X really well.”

Typography, languages, beauty, ramble ramble ramble

2006-07-25T21:02:00.000-06:00

A post that’s been making the rounds lately, it seems, is Tim Bray’s “On Ruby”. Bray praises Ruby as being “remarkably, perhaps irresistibly, attractive,” especially for people (like Bray) who are proficient in Perl and Java. As some see Bray’s post as a pro-dynamic-language post as much as (or more than) a pro-Ruby post, I finally decided to take a read.

As I was reading, this paragraph jumped out at me:

Maybe the single biggest advantage is readability. Once you’ve got over the hump of the block/yield idiom, I find that a chunk of Ruby code shouts its meaning out louder and clearer than any other language. Anything that increases maintainability is a pearl beyond price.

While I’ve still done very little actual coding in Ruby, I have found the above to be true with most code that I’ve come across. I love reading source. Well, perhaps read is not the right term. I love to look at it, just as I love to look at screen shots when reading about any particular piece of desktop software. I have a deep love for design, and much of my own personal artwork is grounded more in design aesthetics than conventional art aesthetics, so I often take a visual approach to code. And what appeals to me at any given moment often shifts.

Ruby code tends to be quite attractive to me these days. I’m not sure why, entirely. But I think Bray hits it on the head in his next paragraph. Funny thing: when I was reading the paragraph quoted above - regarding readability - the thought “hmm, what about Python?” ran through my head. That, too, is answered in the following.

In theory, Python ought to do better, lacking all those silly end statements cluttering up the screen. But in practice, when I look at Python code I find my eyes distracted by a barrage of underscores and double-quote marks. Typography is an important component of human communication, and Ruby’s, on balance, is cleaner. Tim Bray, “On Ruby”, Jul 24 2006

Typography. Could that really be it? You know, he may be right. Phillip Eby recently had a post about Python Based DSLs. He brings up a couple of the elements that seem to give Ruby the power to make nice little domain-specific-languages in, which projects like Rails use to great advantage.

However, Ruby’s advantage in this area basically boils down to two things: being able to apply functions without parentheses, and having code blocks. The first is nice because it makes it possible to create commands or pseudo-statements, and the second is a necessity because it allows those pseudo-statements to encompass code blocks. Without at least the second of these features, Python is never going to be suitable for heavy-duty DSL construction. Phillip J Eby, “Schema Analysis and the need for Python-based DSLs”, Jul 15 2006

I think Eby captures a couple of the more common elements that make this work for Ruby. However, I think there are a couple of other elements that make Ruby look particularly good in this area. The first is Ruby’s use of Symbols. One description I found for Symbols (I forget the source) is that they’re like Strings that you’ll never show to the user. Therefor they don’t need to be printed. They also seem to work with the Ruby’s equivalent (as far as I understand it) of Python’s identity comparison (in python: obj is None, foo is marker). I’d always seen Symbols in Ruby code, but it was Rails that really made me take notice, as they’re used all over the place. Symbols in Ruby are those things that begin with a colon:

finished = Todo.find :all, :include => [:something]
render :action => 'error'

There is something about that that just looks kindof… nice. :all. In my editor, Symbols are colorized differently than strings, which helps them stand out even more. find :all. Not findAll() or find(all=True) or find('all') like one might have in Python (all of which are OK solutions, but man.. those Symbols).

Now as much as the blocks, Symbols, and optional parenthesis seems to help Ruby - it’s the meta-programming that really seems to stand out. This is some ActiveRecord code from Tracks, a GTD-focused task management application written in Rails:

class Project < ActiveRecord::Base

  has_many :todos, :dependent => true
  has_many :notes, :dependent => true, :order => "created_at DESC"
  belongs_to :user
  acts_as_list :scope => :user

  attr_protected :user

  # Project name must not be empty
  # and must be less than 255 bytes
  validates_presence_of :name, :message => "project must have a name"
  validates_length_of :name, :maximum => 255, :message => "project name must be less than %d"
  validates_uniqueness_of :name, :message => "already exists", :scope =>"user_id"

  ...

These are statements in the class level that read… easily. A project has many todos and notes and belongs to a user. It must have a unique name for an individual user that is less than 255 characters. Wow. That is a lot of information in a small space. And you don’t think of has_many as a function or method or something that’s pulling of crazy metaclass trickery: it reads like a statement. I see this done so often in Ruby that writing things like has_many are not that difficult (I’m not talking about the implementation behind it, since O-R mapping is always a beast; I’m talking about how easy it is to operate on the Project class in-line like that).

Now this is doable in Python. Zope 3 has applied similar things, albeit sparingly (and with good cause). You can’t really monkey with the Project class in Python while inside the class statement without resorting (at the least) to execution frame tricks. But one you’re comfortable with them - you can start doing some nice things. In the early days of Zope’s Interface system, declaring support for what was implemented by class instances looked like this:

class Project(Base):
    __implements__ = IProject
    __used_for__ = IUser

Now it can be this:

class Project(Base):
    implements(IProject)
    adapts(IUser)

In a SQLAlchemy based system I’ve worked on for a major project I’m involved with, I’ve applied some of these techniques to yield code like this:

class FeeScale(Base):
    act.baseTable(
        fee_scale, classify='type',
        order_by=fee_scale.c.price_floor,
    )

    @classmethod
    def getFeeForPrice(cls, price, owner, default=None):
        fee = cls.selectFirst(
            (cls.c.price_floor < price)
            & (price <= cls.c.price_ceil)
            & (cls.c.owner_id == owner.id)
        )
        if fee:
            return fee.service_fee
        return default

class RetailerFeeScale(FeeScale):
    act.whenClassified('retailer')
    implements(interfaces.IRetailerFeeScale)
    classProvides(interfaces.IQueryableFeeScale)

    owner = props.One(ILinkedFee['owner'])

class WholesalerFeeScale(FeeScale):
    act.whenClassified('wholesaler')
    implements(interfaces.IWholesalerFeeScale)
    classProvides(interfaces.IQueryableFeeScale)

    owner = props.One(ILinkedFee['owner'])

It doesn’t read quite as nice as the Ruby example above, and it’s from a different problem domain. But it’s not bad… I had to move my validation and relationship-related items into Zope Schema fields which are about specifying design and contracts. Since I use them heavily in order to do things like form generation, ILinkedFee['owner'] contains the information about the relationship, props.One is a Python descriptor that uses that field information in conjunction with SQLAlchemy’s properties to handle relationships between two mapped objects.

class ILinkedFee(Interface):
    owner = schema.HasOne(
        title=u"Owner",
        model=Ref.relative('.base.Owner'),
        options=dict(lazy=True),
    )

There are upsides and downsides to this setup. One really big downside is that there’s no single place to see everything going on with RetailerFeeScale. Earlier versions of this code had functions that played tricks inside of a class suite to spell things like hasOne(...) without assignment or using a second mechanism such as Interface schemas. That code, however, was playing too many tricks and causing too much trouble. And this setup works well with Zope 3.2 and the latticework I’ve put in place to support it.

But it seems like a lot of work, learning metaclasses and descriptors and decorators and getframe() trickery. However, looking at this mammoth project’s code tonight, I’m impressed by how concise and readable most of it is. Ruby and Smalltalk have both been major influences on me lately, even if I seldom get to actually use those languages directly. God I wish for blocks though. I think that the with statement in Python 2.5 will help. A place where Blocks seem especially powerful is their ability to construct an object, yield it to the block, let the block do things with it (set attributes, etc), and then the thing which yielded control can do more things with the object. Hmm. That’s vague. Here’s one case: explicitly saving (flushing) a SQL Alchemy object after doing a bunch of updates. It’s one of those lines that bugs me: it’s something important (it should happen), but it’s not necessarily important to the business logic. It’s small, and it often blends in with its surroundings:

fee = Fee.get(1)
fee.floor = 0.0
fee.ceil = 10.0
fee.price = 1.25
fee.save()

Small annoyance, really. But it does bug me. I believe that the with statement might help out, ensuring that the save happens immediately:

with savable(Fee.get(1)) as fee:
    fee.floor = 0.0
    fee.ceil = 10.0
    fee.price = 1.25

And with that, I put to bed this evening’s incredibly long and pointless rambling, whose beginnings I scarcely remember. Oh yeah. Ruby’s punctuation and typography is beautiful. It really is. And I envy it.

OmniPlan Approaching

2006-07-24T12:23:00.000-06:00

I’m not sure what this indicates about me, but I felt such an overwhelming air of excitement and relief that I almost - almost! - had a little tear in my eye when I saw this announcement from the OmniGroup. Omni’s new secret product that’s no longer a secret is (oh boy oh boy oh boy!): OmniPlan. Project management software.

Why am I so excited about project management software? Because there’s very little project management software for the Mac. I know this, because I recently looked for some. We were using an OmniOutliner document to try to capture all of the known things that we needed to do to get this big enormous project out the door, and to come up with some estimations on the time involved. OmniOutliner was OK for this, to some extent: it can have multiple columns, and the columns can be different types including durations. The duration column can be configured to calculate things in working days or 24 hour days, so you can say “that’s going to take 10 hours” and the document can turn that into “1 day 2 hours”. You can also have summaries, where a parent outline element can add (or average or mean) the values of the child nodes. However, with the way that we were entering data, we couldn’t take advantage of this.

As we were fleshing out the development plan, we were unable to easily spell dependencies. We got around this by adding another column, but it was just plain text. One couldn’t look at the list of things to be done and easily decide to do “that big scary thing that needs to be done so we can do these four other things.”

Turning that into a calendar so we could plan time and other commitments appropriately didn’t work out either. We did it manually, on paper, and then turned some of the big milestones into milestone entries in Basecamp. And while Basecamp excels at communication (insofar as everyone involved actually uses it), this level of task / timeline / dependency management is outside of its scope. And I’m glad for it. We seldom need this kind of MS Project style project management… Until this project came along.

So I downloaded a couple of demo versions of different Mac OS X project managers to see how easily I could turn our Plan into something that could better display time estimates, dependencies, and so on.

How easy was it? Not very. I didn’t want deep project management features. I wanted to be able to enter tasks as quickly and easily as I had with OmniOutliner and have the Plan start coming together on its own. But in everything that I tried, there were too many steps, too many dialog boxes, and so on. I don’t mind dealing with inspectors and details later, but I hate when they get in the way of doing simple data entry. And I also hate the tools whose simple data entry is too simple, so that if you enter a lot of items rapidly you then have to spend a lot of time in dialog boxes, tabs, etc, turning it into something real.

I don’t know what OmniPlan is going to look like, or cost, or anything. But Omni’s record of building good solid NeXTStep and Mac OS X applications that are fast, fun, intuitive, and flexible is nearly unbroken. I wish this application had been available a month or two ago, but better now than never. I expect we’ll see a tool that humble programmers such as myself can use without a degree in business or project management.

Desecration Sale Spectacular

2006-06-28T10:45:00.000-06:00

If you ban the burning of the American Flag because it supposedly desecrates what our soldiers have died to protect (funny - I don’t see soldiers out protecting the flag over the supermarket from lightning), shouldn’t there also be a ban on using the flag to bring everyone down to “celebrate their independence from high prices at your local car dealer and/or furniture warehouse Fourth of July Independence Day Sale Spectacular”?

How about a ban on Memorial Day sales pitches? “Remember all those who have actually died in the service of this country, often in hostile environments most of you would have a heart attack in, by getting a Free Hot Dog and a chance to win a new sofa set on us!”

I know that this kind of commercialism is here to stay. Personally, I think that it really does cheapen the meaning behind some of our holidays. Why aren’t more people up in arms about big business’s war on memorial day? Unlike the bullshit crusades against the so-called “war on Christmas”, some of these holidays actually have a potentially good national meaning that is completely - COMPLETELY - washed over. “Support our troops and celebrate your freedom by getting a new Honda at Zero Percent Financing, this weekend only!”

These proposed constitutional amendments to protect the flag and protect marriage do nothing of the sort. They’re stupid ass wedge issues. If you want to really protect marriage, do something about the divorce rate in this country; shut down the 24 Hour shotgun chapels in Vegas (does that not make more of a mockery of this “blessed institution” (cough) than anything?); boycott Budweiser for cheapening marriage with its image of a man getting married over and over in Vegas for a free case of beer; boycott television sitcoms; get “Desperate Housewives” off the air; support federal aid policies that help low income families so that financial stress is less likely to be a factor in separations and divorce; hell, why not make adultery laws with severe punishments like some countries that wobble daily between friend and foe?

Is a constitutional amendment defining marriage as being a union between a man and a woman really going to do anything to this country’s divorce rate? Is it going to suddenly put an end to single parent homes? Is it going to cut down on adultery? Murder? Foster care?

Not one goddamn bit.

And it’s the same goddamn thing with the flag burning amendment. I personally believe there are many other ways to desecrate the flag, our troops, and our history, and it’s all accepted as perfectly legitimate as long as there are free balloons for the kids.

Stupid, stupid, stupid.

Ads - Debugging Matters, and Out of the Box

2006-06-22T11:40:00.000-06:00

Terrific targeted advertising: The current ad that appears on the main page for Sci-Fi channel’s excellent new Battlestar Galactica series is for Microsoft’s Visual Studio 2005.

The ad? It just says “Cylons. Why debugging matters.”

I think that’s the first good ad I’ve seen placed by Microsoft in a long time. For those unfamiliar with Battlestar Galactica, especially the new series, the basic premise is "The Cylons were originally created by man. They evolved. They rebelled." The miniseries that brought the story back to the air (in a much more rich and intelligent flavor than the old series, which had a good idea but had poor execution and network interference) starts with the near annihilation of the human race in a massive coordinated Cylon attack. Yeah, debugging matters. Great placement. Although, of course, you could see variations on this for Terminator: "SkyNet. Why debugging matters." And for The Matrix. Which is not to say that Galactica is anything like those series (I think the nearest it could be compared to is Bladerunner). Just that someday, these robots will come back and kill us all. Sony's cute little QRIO has its own emotions and can express them in different ways, including changing eye color. You just know one night you're going to wake up and see this cute little robot at the foot of the bed staring at you with those red eyes... And then you'll never wake up again...

Sorry, off track.

While on the concept of ads, I’ve been enjoying one of the latest from Apple’s “Get a Mac” campaign. The commercial is Out of the Box and features the Mac and PC in their boxes talking about what they’re going to do. If you haven’t seen this series, they follow the sparse white backgrounds and straight-on shots of the older “Switch” commercials (and, in fact, most of Apple’s modern non-iPod commercials). Instead of the “real people” stories that were the center of “Switch”, “Get a Mac” features a humanized PC and Mac. The PC is played by John Hodgman (now a Daily Show staple), and the Mac by Justin Long (an actor in “Dodgeball”, which is a movie I unexpectedly fell in love with).

In Out of the Box, the PC and Mac are sitting in boxes - the PC’s is plain brown, the Mac’s is plain white. They introduce themselves as they do in every commercial - “Hi, I’m a Mac” “And I’m a PC”. The Mac starts getting out of his box and says “ready to get started?” to which the PC responds “well, not quite… what’s your big plan?” The Mac sits and says “well, I might make a movie, create a web site, try out my built in camera.. I can do it all right out of the box. So what about you?” The PC responds with “well first I got to download those new drivers and then erase all the trial software that came on my hard drive…”

The ad ends with the Mac getting out of the box and leaving frame while the PC says “actually I can’t go yet… the rest of me is in other boxes.”

Now this is a great ad. It’s short, funny, and quite charming. While the content of the ads is competitively aggressive, the PC and Mac are quite nice to each other (in one ad the Mac even says that the PC is great at business). Overall the series is doing a good job, in very short segments, of highlighting the cultural differences. Some ads play up to the “if you like the iPod or iTunes, we’ve got even more of that!” message, while others cover other differences. This one, however, I particularly enjoyed.

We recently purchased a new Dell machine at the office to run accounting software. A few days later, there is still a cadre of boxes in our lobby with CDs scattered everywhere. The machine is installed, and it installed fairly quickly. But what I remember most is how much my boss kept yelling at all of the trial software, and how much time he spent un-installing things. “Something new pops up every time!”

I imagine that this is how Dell and the like can offer their machines for such low prices. They stuff it with all sorts of trial software - often very annoying trial software (it bugs you more than it helps you) - which is just more advertising, basically, subsidizing the lower cost.

It’s amazing. I have a windows machine at my desk that I use on occasion (it’s actually been off for a couple of months), and I hated having to restart it because there were so many little balloons and bubbles that kept popping up for the first few minutes. “I did this” “You should upgrade to this” “Buy this” “Your machine is at risk - upgrade to the full version of bla bla antivirus for $79.95 today!” and on and on. Half of these interrupts were modal, taking away focus from what I was trying to get back to doing. Part of me is amazed that this passes as OK in the Windows world, but part of me looks at advertising today (and through time) and realizes that it’s just the norm of our society.

I always get entertained at list once during local coverage of the Utah Jazz at all of the little inserts that get slipped in, and how casually the announcer puts it in. Granted, a lot of these same games are being simulcast on radio (some games are only broadcast on radio), so perhaps it makes a bit more sense. But it’s just things like “Malone drives it up the floor and it’s knocked out of bounds. This portion of the game is brought to you by Bla Bla Bla, reminding you that it’s always a good time to Bla bla” or “brought to you by Bla Bla. Come down and see the professionals at Bla Bla, quality service guaranteed. Stockton takes it inbound…”

It’s all around us.

Anyways, I just found the Out of the Box commercial funny, having watched someone just recently go through the experience of spending his time removing all this junk software before he could set the machine up to do its actual job. It’s not like the Mac is devoid of trial software - a .Mac trial period is pimped, trials for both Apple’s iWork and Microsoft’s Office are also on board. But to my recollection, no bubbles ever popped up and interrupted my work telling me to use / install / upgrade either of those. In fact, I didn’t even notice they were on my new iBook. They didn’t hinder performance any, and - most importantly - never got in the way. Dragging them to the trash was all that was needed when I did notice they were there.

So, good ad, sells the point well. I’ve been impressed with how well all of my Macs have worked out of the box - even discovering that a new laptop had a decent charge in its battery when it arrived!

More OmniOutliner Pro Praise, and Other Tools

2006-06-20T12:06:00.000-06:00

A nice post from 43Folders came across my desk today pointing towards a post by Eric Schmidt about Using OmniOutliner Pro and Kinkless GTD in Law School. Schmidt first covers how he uses OmniOutliner Pro to take and manage notes for his law courses. One significant Pro feature in OmniOutliner is the ability to have Sections - a navigable list of top level outline elements. Kinkless GTD uses these to separate the Inbox, Projects, Actions, Archive, Templates, and Setting portions of the kGTD document. As I’ve started to work on larger outlines again in OmniOutliner, I’ve started to make use of this feature in other documents.

Now, there are numerous outliners available for the Mac OS. A glance through the archives of About This Particular Outliner shows comprehensive coverage of various outlining and mindmapping tools. I’ve used quite a few of these over the years, often just in evaluation. The only additional ones I’ve purchased and gotten any real use out of have been Aquaminds NoteTaker, DEVONThink Personal, and Tinderbox.

NoteTaker is a decent application. I got a lot of great use out of it in a school setting as a math notebook. Using EquationService, I was able to take notes in a full Notebook paradigm and enter the math formulas in LaTeX and convert them to inline PDFs, with no knowledge of LaTeX or equations being required of NoteTaker. I’ve also used NoteTaker on a couple of customer projects that had clear releases. NoteTaker’s structure has many pages, grouped by sections. A section might be “WebSite 1.6”, with a page devoted to each feature request made by the customer for that next version. NoteTaker’s pages are outlines, but it often works best with big blocks of text. No problems - that worked great for these books. But while NoteTaker is steeped in Mac OS X’s NeXTStep heritage, it has a bit of a strange feel. More NeXTStep-ish still than Mac OS-ish.

Interestingly, NoteTaker has a very very very similar competitor: Circus Ponies NoteBook. They both have the notebook metaphor - the windows look like ringed notebooks (by default), they have index tabs on the side, and the document is a collection of pages. There are differences between the two, but they both actually share a common ancestry. AquaMinds (NoteTaker) and Circus Ponies (NoteBook) were started by two different principals that had previously made a similar [product for NeXTStep][http://www.simson.net/nextworld/93.2/93.2.ApMay.Notebook.html].

DEVONThink Personal is not so much an outliner as a personal database. NoteTaker and NoteBook position themselves that way too, but DEVONThink is much more like a specialized Finder implementation than an outliner. DEVONThink has a sort-of outline “mode”, but it’s nowhere near as fast or natural as OmniOutliner or NoteTaker. Instead, DEVONThink shines as a place to collect PDFs, web pages/archives, text notes (rich or plain), etc. It has excellent cross-referencing and indexing capabilities so any document can have a “See Also” drawer which shows contextually similar documents in the system. This is great for building reference systems, especially since you can drop a new note, text, web archive (great for keeping documentation around), etc, in and then use the classification tools. Classification works similarly to the “See Also”, but shows folders instead. Folders are matched based on their contained contents. I use DEVONThink at work to collect odds and ends from various projects, and (more often) to keep and archive web pages containing documentation for tools like SQLAlchemy, Markdown, etc. On my laptop I use it to keep archives of web pages I want to read in my spare time, whether or not I have internet access.

Tinderbox is an interesting system. Part of me thinks it could be so much more if only it were modernized. It just doesn’t feel that natural on Mac OS X. It’s such a powerful and flexible system, with the ability to think and work in different modes - outline, treemaps, visual maps - with multiple windows open in the same document even! Custom attributes, agents, and rules can make for pretty powerful and personal systems. But using it is like using [Squeak][http://squeak.org/]. It’s really cool and really powerful, but it’s also its own self-contained world. I had a hard time integrating Tinderbox into my day-to-day operations as it works less and less like the Mac OS X I know and love. No Services support, changing a font requires using a traditional font menu (which is a nightmare with my sprawling font collection). Hell - you can’t even save a file with a name longer than 30 characters! I don’t think I’m in Tinderbox’s target market though. If I wrote as much as I often say i’d like to write, I’m sure it would be a great tool. I’ve wanted to use it to start mapping and managing a lot of characters and places from real life and mythic life. But I have a hard time going into Tinderbox land.

OmniOutliner, on the other hand, is tremendous. Especially as of version 3 (and even more so with version 3.5). It is so deceptively simple. A fresh document is just a basic outline. Start typing, indenting, structuring, etc. Often that’s all you need to take some notes down or think something through. But it also has excellent multi-column support. The columns can have different value types with the ability to summarize values. In evaluating whether I could afford an impromptu weekend trip, I used OmniOutliner with two columns - Topic (the main outline/text column) and Price, with Price formatted as dollars and with a “Total” summary. The top level of the outline was “San Francisco Trip”, and then inside of it I had lines for flight, hotel, tickets to an event I wanted to go to, boarding the dog, train, etc. Very quickly I added borders on rows and columns, and within seconds I had a very small and simple document that many people use spreadsheets like Excel for - but with absolutely none of the complexity or long arguments that are involved when I use Excel (or any spreadsheet, for that matter). This was just a quick “I wonder if I could…” experiment - could I make a quick list and see the estimated total? could I afford to go? (The answers were ‘yes’ and ‘no’, respectively).

But OmniOutliner does even more, and all naturally. By using Mac OS X’s text formatting system, it gets access to the Ruler (NoteTaker and NoteBook get this too). The Ruler makes it possible for just about any Rich Text input area to be at least as powerful as a tool like WordPad on Windows. Named styles, tab stops, left / right / justification alignment, paragraph indentation, spacing, and more. OmniOutliner doesn’t show the ‘Lists’ option, but NoteTaker does, so bulleted and numbered lists inside of the text are available just about anywhere.

This is what amazes me about OmniOutliner: it can be such a fast and simple basic outliner, requiring practically no training or documentation to use. It’s smooth and fast and looks great to boot. But with just a little exploration of the menus, toolbars, and inspectors, it can be unleashed. Some sample documents look closer to a full Word or Pages produced document: the outline handles and checkboxes are hidden, the indention settings are massaged, the fonts and styles applied well. Other sample documents show its usefulness as a simple spreadsheet / database system: the kind where the main text is delightfully easy to enter and formats well against the other columns (spreadsheets do not do this well at all), and the supporting columns can have intelligent value entry and handling. The date support is wonderful - “tomorrow”, “monday”, “the 3rd”, “last friday”, are all valid values. Duration columns can be set with how to measure hours - is it a work day? is the work day 8 hours? then a total of 9.5 hours immediately displays as 1 day 1.5 hr.

The section support in OmniOutliner Pro invades some of the territory enjoyed by NoteTaker and NoteBook and the like. Suddenly a single large outline can become very navigable. I exported some of my NoteTaker books out to OPML and into OmniOutliner Pro and had a system almost as nice up and running within moments (aside from some lost formatting). Nicer still is that a co-worker has started using OmniOutliner pretty heavily and it’s nice to be able to settle on a single tool and share. I’m just amazed that this tool can be so simple and useful for the quick-notes while becoming powerful enough to support a system like Kinkless GTD.

My Mac OS X "Priceless" Applications and Tools

2006-06-05T12:20:00.000-06:00

Can’t live with ‘em, can’t live without ‘em. This is about the Mac OS X applications I can’t live without, and some mentions of ones that I’ve still never found use for.

Can’t Live Without

Quicksilver

The ultimate quick launch and system control tool, Quicksilver is like the perfect marriage of graphical and command like control. Before Quicksilver I was an avid user of Launchbar. At heart, both of these products are just launchers - quick access to applications, bookmarks, etc. Launchbar allowed me to never be bothered by the Mac OS X Dock: I use the Dock to house applications and documents I use frequently, and Launchbar let me quickly get to ones that I use only on occasion. What makes both of these products stand out is that they learn. One might bring up the tool and type the letters A G L. The first hit might be a bookmark to an agile development article. But hitting the spacebar a couple of times to highlight Adobe GoLive will cause both LaunchBar and Quicksilver to increase the weight of that result. Within a couple of uses, CONTROL-SPACE A G L becomes the fastest way to launch GoLive.

Both tools offer access to more than just applications and bookmarks. Quicksilver started to differentiate itself from Launchbar (as of Launchbar version 3 - it’s now at version 4.1) as a more thorough system control tool with a lot of AppleScript support and a wide (and growing) range of plug-ins. Quicksilver could be used to control iTunes, to put the computer to sleep, and more.

I’m not sure when or why I made the switch. As Mac OS X matured, I used Launchbar less and less. When Quicksilver came on the scene, I didn’t pay much attention to it. But I kept hearing good things about it, and eventually got around to checking it out myself as it was a big hit amongst the 43Folders crowd. As I started to use Quicksilver, I found it insanely useful. I could pause and resume iTunes with a few keystrokes without having to change focus from what I was doing. I could start doing some “fire and forget until later” actions, such as emailing a quick note or to-do to a Backpack page or sending an action to my Kinkless GTD document. I could take quick advantage of Mac OS X services such as Make Sticky or Take Plain Note in DEVONThink, even in applications (like Tinderbox) that don’t support the Services system.

This is most handy when doing something very focused, such as a debugging session, when I’d notice something else I might need to come back and revisit. I don’t want to take the time to change focus to OmniOutliner or some web project tool or iCal to make the note to “come back and fix the call to …” when I’m chasing some obscure issue. Quicksilver lets me make that note almost invisibly - I don’t even have to take my eyes off of the current debugger line. Priceless.

Small feature that I absolutely love - Large Type. One of the default text actions in Quicksilver, this is damn handy when making phone calls. Apple’s Address Book, and some other applications, support this for values they have in the system. But sometimes you need to call a phone number and read off an account number that isn’t going to be in Address Book. With Quicksilver and a couple of tuning options, you can get this down to a single chord or just a couple of keystrokes, and BAM! Screen-size type. Doesn’t interfere with anything else - next keystroke or mouse click makes it go away.

TextMate

A native Mac OS X text editor, TextMate has become my new killer application. For the first time ever, I’m doing most of my Zope and what-have-you development on my desktop. Historically I’ve always done my web development on servers, using XEmacs and then VIM as my comprehensive editor of choice. Even when I’d do development on the desktop, I’d often use VIM. But TextMate has changed that.

With support for lots of languages and commands, it’s become indispensable. The killer feature was the ability to install an Edit in TextMate… command in all Cocoa applications. Now I can edit the contents of any textarea in Safari in TextMate - whether it’s GMail, a bulletin board system, Basecamp, a Zope application, a classic Zope page template, I can pop over to TextMate. Even now, I’m using MarsEdit to make this post. I’m writing it in Markdown and will convert it to HTML from within TextMate before posting it.

Over the years, I’ve found long-time Mac staple BBEdit to be too expensive, and ultimately underpowered, for my needs. BBEdit still bears too much of its classic Mac OS heritage, I feel, and some of its keyboard commands and preference screens started to feel really unnatural. Granted, I was still using an old version. But even the newer free ‘lite’ version of BBEdit, TextWrangler, felt uncomfortable. I used it for some development in the past year, but TextMate feels a lot more natural to me in most regards.

There are some features that TextWrangler / BBEdit has that are still missing from TextMate - integrated FTP support is one, split windows is another. The one feature that I can’t believe TextMate doesn’t have yet is the ability to split editing windows! But the number of plug-ins available, the “Edit in TextMate” feature, the relative ease with which new syntaxes and commands can be added, far outweigh some of the holes.

Small feature that I absolutely love - Find in Project. Any folder can be a project (without all of the project overhead imposed by IDE’s). In the Zope 3 development that I’ve been doing, this has made refactoring easy as I could rename or move a class or interface and could quickly find all uses of it. TextMate also supports many of the Cocoa find/replace features and commands (and builds on them). One handy command is “Use Selection for Find (command-E)”. Safari, Mail, MarsEdit, OmniOutliner, etc, all use this command. Select a word or phrase, hit ‘command E’ and then ‘command G’ (find again) to search for that word without bringing up the ‘Find’ dialog box, without using up the clipboard, etc. This is great with the Find in Project feature in TextMate when I’m thinking of renaming or moving a class, or just want to see where it’s used, such as when exploring other people’s code (such as the Rails or Zope frameworks themselves).

DarwinPorts

DarwinPorts has made it possible to do the kinds of development I now do on my desktops / laptop. This is where I get my Python 2.4.x with readline, Ruby 1.8.4, Docutils, subversion, etc.

OmniOutliner Pro / Kinkless GTD

OmniOutliner has been in my toolbox since version 1, back in the early Mac OS X days. I’ve gone through other outliners and information databases, but now I’m using OmniOutliner Pro pretty heavily again. Part of this is because it’s still the fastest, easiest to use, and best looking outliner on Mac OS X. A third party set of AppleScripts turns it into a powerful planning / task management system, as Kinkless GTD. I’ve tried some other systems such as GTDRules on Tinderbox, which I’ve compared before.

Kinkless GTD features a Quicksilver action called ‘Send to kGTD’ which parses text into an action and sends it to the kGTD document. If the document’s not open, it opens it. The parsing revolves around being able to specify a project and/or context to place an action in. For me, this is useful for that ‘fire and forget until later’ moment that comes up from time to time. I capture something that needs to be done, but I don’t let it consume my mind while I focus on the task at hand. When I have a moment, I can go to Kinkless GTD and file it appropriately or figure out what needs to be done for that note I scribbled off. This shortens the amount of inboxes that I have, and shortens the number of places I have to think about when I want to send myself a “deal with this later” note.

iDisk / CVS

Apple’s iDisk (part of their .Mac offerings) has a nice feature: the ability to work offline with transparent synchronization. That synchronization can be automatic or manual. This is how I keep track of my Kinkless GTD document, and any other notes and documents I want to work on without worry of location. No need to have a Lotus Notes size system just to manage a small set of personal information across three machines, especially when one of them spends half of its time offline (and one is spending ALL of that time offline these days). iDisk lets me save any document of any type, synchronize the disk to my laptop, take the laptop home, work on it there (without internet, which hopefully will get fixed soon), come back, synchronize back. I’ve thought of using USB “key drives” to house information like this, but I often fear damaging or forgetting the drive.

We still use CVS on our development servers at the office. We’re too small and busy to have time to change to subversion when CVS is doing its job (mostly) well. Some people use CVS to manage their home directories, documents, etc. I just use it to bring code home in case I want / need to work there while I’m disconnected. It’s hardly worth mentioning, except for how every work day ends with ensuring the iDisk and source directories are up to date on the laptop.

Can’t Live With

There’s really only one thing I can think of here, and it’s one thing that’s always bothered me: Virtual Desktops. I’m glad Mac OS X doesn’t support this directly. My main problem with Virtual Desktops is the ease with which important windows get lost. With Mac OS X, I use “Hide Others” and window minimization heavily when I need to focus and I have too many distractions on screen. A quick “Show All” can bring all the windows back on screen and into the dock (if minimized), and Exposé is, at that point, the best way to find a particular window when needed. With Virtual Desktops, I was always playing the game of “is it on 1? nope 2? nope 3?”. Some people establish systems - web browser always here, email always here, main editor always here - but I never was able to. I’d just lose things and get frustrated. Between Quicksilver and Mac OS X’s native app / window switching features, I have adequate control over my desktop and what populates it, I feel.

Third Time, Slow Time, Inventing, Revisiting, and Fighting

2006-05-31T17:22:00.000-06:00

A couple of months ago, we started work on a major rewrite of one of our core sites. What started as a small and specialized e-commerce service a few years ago has grown mightily over those same years. The architecture, however, has hit a limit - especially when compared to the large push for what we want to do (and can do) next. The current site works great and handles its load and duties very well. But there are small annoyances internally with some things, and there is some growth and market potential that it cannot satisfy.

The first version of the site’s primary work was done in less than a month. The main focus of that version was the visitor’s experience. The original HTML mockups drove a lot of the site, and the implementation was primarily done in Zope 2, through the web, using templates, scripts, and SQL Methods. It was a nice enough system for the time - we were very fast with it, we could tweak elements in each other’s offices, etc. It ended with an all-night debug-and-deploy session, wrought with all sorts of strange problems and angst and fun. But it worked. We continued to work on it over the next few weeks, of course, filling in some major holes that couldn’t be filled in initial development and responding to site usage.

The biggest problem with that implementation was the administration side. Since we were the only users of the administration screens, we punted on that issue. There was no form validation. All of the database CRUD statements were entered manually. Zope’s SQL Methods made this a bit easier, but their main optimization is for read-queries. Any new field required numerous code updates all over the place. The whole business was a little messy back then - requiring us to print out certain reports and delivering them to our providers on time-constrained schedules.

The site was never empty, but it got perilously close a few times during the lifetime of version 1. But there was enough interest to keep it going. With eyes on expanding and getting new providers, with providing new services (including physical goods), and with running the business side of things better, we went into version 2.

As I saw all of the HTML forms and database statements I’d have to change just to accommodate the expanding and changing data requirements, I started looking at options for easing that pain. I ultimately rolled my own system focused primarily on the common CRUD operations. There was no object-relational mapping solution at the time that I liked. A problem, at that time, was that we had many queries in place that would not work in the object-relational tools available.

What I really wanted was an architecture that:

Helped me get important / core business and administration logic out of the ZODB. We wanted to have as much of the core software on disk and under source control as possible.
Dynamic form and SQL generation for common tasks (primarily in the administration section). I wanted this so that adding new fields to the database schema required only one or two changes.
Still allowed us to use Zope’s SQL Methods for generating complex queries.
Could be introduced in a way that the public application might not even be aware: instead of a folder full of Python scripts in the ZODB, there’d be a single persistent instance of a class that had many of those scripts as methods.

The resulting framework was essentially a service layer comprised of very rough Table Data Gateways. You couldn’t load an object out and do a jeff.hair = 'bleached' and have it save. The get/create/save statements basically took a dictionary (hash table) of values to save and would flush it out to the database, after harvesting some information. It was expected that the gateways would have the data prepared before passing it off to the lowest level interface, handle_op, which would perform the requested operation.

When dealing with fairly set data, such as a form, this didn’t really matter. I cared about loading and binding the data to the widgets when a page was loaded, and then parsing/validating/converting that data on save. For that, this system worked like a champ. For some other pieces of business logic, it did OK. It wasn’t great, but it was better than what we had before - especially because it didn’t require specialized update SQL to be written for those situations where just a couple of values needed to be changed.

That site has served us fine, and now the offerings on the site are quite full (many many pages). The content providers have access to their items and reports, so we no longer have to run around town delivering reports and lists.

Both versions were deployed, in full, and then continuously massaged over time. There was no ‘beta’. Some features sneaked in after the deployment deadlines. Some specialized parts of the system were overhauled a couple of times to deal with scaling issues - ones that we could never have predicted.

We’ve kept laundry lists over the years of the things we wanted to do, but couldn’t under the first two designs. Some of these came from dealing with some hacks pulled together for a couple of special customers, some came from wishful thinking, some came from complicated set-ups required to make certain rare items available for purchase in the existing system. In a getting real sense, this was OK, but not great. It limited our ability to start working with larger and more specialized providers.

Architectures have continued to improve over the past couple of years as well. When Zope 3, version 3.1 specifically, came out, we were able to do a major rewrite of a content system for one of our oldest customers. We were even able to pick up additional related customers as a result. We’ve delivered other solutions on top of Zope 3 and the ZODB (Zope Object Database) that have been both impressive and fun to work on, although sometimes Zope 3 can still cause me to go off in a huge screaming match.

I had hoped (and still wish) that for this particular site’s version 3 implementation we could ditch the relational database completely and just use the ZODB. Instead of classic Zope ZODB usage, where scripts and templates fill up the database, Zope 3 makes it much easier to keep that stuff OUT, and then easier to keep real data in. This is one of the nicer parts about working with the Zope 3 / ZODB stack: Python is just Python is just Python. There’s no translation layer, no tricks, trying to fake inheritance. There’s no real worry about translating statements into a query language - Python is the query language (with tools like the Catalog providing application level indexes for larger queries). I had an application that I was working on in Zope 3 that I thought would be a good testbed for writing as a Rails app - until I realized that Active Record’s inheritance model could never match my object model, at least as of Rails 1.0. I also played a little bit with Python’s Turbogears and SQLObject stacks, and they had me feeling pissed and frustrated within seconds. No offense to their authors, but they just did NOT work for me.

But the decision was made that for this site, we would continue with using a relational database. Well, I knew the strengths and limitations of our current systems. And I admit, I was envious of Ruby on Rails. In fact, I even suggested that we ditch Zope for this project and use Rails instead! But it was feared that there was no time on the schedule to learn Ruby. We did, however, agree on wanting a real object-relational system, or at least something more object oriented than what we used for Version 2. Version 3 has richer business logic requirements, and I’m happiest when that code can look as clean and natural as possible. I also wanted it to be easy to define new pages and views and to have to go through less guess-and-hope work than Zope 3 (as of 3.2) typically requires.

So we’ve birthed yet another in-house framework because there’s little out there that’s satisfactory. The one thing that is satisfactory is SQLAlchemy. I built a base storage framework on top of that which allowed us to use SQL Alchemy fairly transparently from within Zope. Fields and properties now let us define and use relationships fairly transparently. Items get bound to their context, are easily traversable, and can even masquerade as Zope containers without interfering with SQLAlchemy’s on system of managing work. We also birthed a base web framework that provides some useful base classes for constructing the kinds of pages and sub-views that we need with ease. These base classes corral some core Zope 3 features and some other in-house features together to ease our web development.

It’s all pretty cool, for what it is. But it’s going so slowly that I can’t help but wonder if we would have been better off working in Rails. I still haven’t found anything in the Python world that I’d leave Zope for, but Zope 3 continues to jump between being insanely cool and powerful and flexible, and being incredibly frustrating and aggravating. That I’ve been able to pull many of the tricks I’ve already been able to pull is a testament to the better parts of its design.

Even with all of my tricks and base classes, it can sometimes still take an entire morning just to get what seems like a simple page together and rendering. On the other hand, once you get pieces in place they’re pretty sturdy. There’s low likelihood of accidentally overriding a critical method name or a view/page that applies to a different context or interface layer.

But as it’s been going so seemingly slow, this development, I start to wonder why it feels that way. I really was trying to give myself a system where I could enjoy the benefits of Zope 3’s Component Architecture with an intelligent database / object-relational mapper while also enjoying the benefits of Getting Real. “Getting Real” isn’t anything really new to me (or many people). It just helps give you the OK to say “no”. Martin Fowler’s book, Refactoring, made me realize that bad smells are OK now, so long as they’re cleaned up later. In the interest of getting something done, it’s hard to go for purity. Or in the case of Python’s “import this”:

Special cases aren’t special enough to break the rules.
Although practicality beats purity.

So, what is the problem with me then?

I think it comes down to a very special case: the major rewrite. This is not a new application for us, nor is it a moderate upgrade or maintenance release. We’ve acknowledged that the existing architecture has issues when it comes to certain growth paths. To this point, we’ve also been good in saying “no” to those growth paths. But business situations and opportunities have changed.

One problem with the rewrite is the knowledge that it’s already been done. “We already did this three years ago, why can’t you show me it working in the new system already?” There senses of worry, fear, excitement all change. This is further exaggerated by the fact that these kind of rewrites are often architectural, perhaps deeply architectural. All of those agile decisions made in earlier versions no longer apply, since many of those decisions may have led to the limitations you’re trying to overcome. Those decisions were right at the time, and have been right for the past few years. But now you realize that being able to do more promotions, discount options, special offers, are among the things you really want to do with the cart. You also need to handle more delivery options instead of the two you’ve used for the past two years. You fight and push and think and tango with the cart, just focusing on shipping, with coupons in the back of the mind. Part of you thinks “I just want to add a shipping column / attribute here and be done with it, and we’ll expand it later…. Oh wait… This IS later… Oh yeah, we need to support more options… Oh yeah, this is one of the growth areas we’re focusing on. Crap!”

It’s an interesting struggle now, those two sides in my mind. In practical terms, I just want to add an extra column or attribute and be done. But I know in a week it’ll already be pushed to its limits. Better to apologize for lack of screens and work this the hell out now. Nothing fancy - keep the interfaces and collaborators simple, and expand on them if needed when more data presents itself.

Personal File

2006-05-26T11:59:00.000-06:00

Last night I floated like a ghost past the strip club, under the freeway, over the railroad tracks, past the big electric sign and refrigeration companies and worn out corners until I got to the super-center. I needed milk, inner tubes for my bike, and dog treats. Of course, my cart started filling up with much more than just that. And in the middle of my 11pm shopping excursion I stumbled across the music/video section. As I stared (half-angrily) at the so-called “top sellers” in the country department (it’s where my cart stopped), I remembered coming across a new Johnny Cash collection on the iTunes Music Store.

So I now have Personal File. It’s a collection of recordings Johnny made (circa 1973-1980) in his studio. It’s just Johnny and his guitar: the same kind of intimate, personal, and raw sound that made the first American Recordings album such a standout. Some of these pieces are recognizable, such as When it’s Springtime in Alaska, It’s 40 Below and Jim, I Wore a Tie Today. Many songs have stories attached to them - why Johnny remembers it, why he wrote it, why he sings it, etc. It was a style seen on some of his commercial albums from that time, although those were accompanied by a larger band and sound.

The second disc of Personal File hasn’t grabbed me much. It’s fairly religious, and while I respect Johnny Cash’s religious strength, I have little interest in it. There are some nice songs on that disc though. But the first disc is the good one. It’s from a good time in Johnny’s life - settling into his life with June, his drug days behind him. The voice is powerful, most of the songs quite rich. A very enjoyable listen.

Online / Offline / Middle of the Line

2006-05-25T19:19:00.000-06:00

I’m still offline at my new home. Curiously, this hasn’t been a huge deal for me, so far. I’m getting home so late (relatively), and then still have to walk the dog (if she stayed home that day), feed her, feed myself, etc. Other factors have been eating up bits of my evenings as well: cleaning out the old place (finally done as of yesterday), getting some finishing-touch little furnishings (one at a time since I don’t drive), and yet-more organizing and layout.

The situation with Comcast has just pissed me off enough that I haven’t wanted to call back to find out what’s next. Part of this is due to the inherent stress of workdays recently. I don’t want to add to the pile with long calls and situations where no one really seems to know what’s going on. But I know I have to, soon. Tomorrow.

I have not had any major beefs with Comcast (and AT&T before them) up to this point. And the whole situation may not be their fault, but rather the fault of bad information and the odd nature of this particular move. But as someone who has been a good customer for so long, I do feel a little insulted that someone hasn’t jumped up and said “hey! you know what, you’re a valued customer and we want to keep you so I’ll figure this out personally.” Not that I expect the average customer service person to care that much, but maybe I could be passed on up to someone who does.

This is a big thing for me right now. If they don’t seem to care, why should I? And if I don’t care, why aren’t I looking for alternatives? I still don’t really want to deal with phone companies and DSL, but I know it’s an option. And as far as television goes: I’ve long known that I need to cut back. As much as I’ve loved HBO and its original programming, I haven’t even noticed that I’ve missed a couple of weeks of the Sopranos now. I get a handful of channels off my antenna. I’m starting to think that maybe an option like USDTV could work too. Dish/Satellite is out. I hate those things and don’t really have anywhere to put the damn dish anyways.

But it comes down to this: I was told what the builder was told. “When the first person in wants cable, we just have to connect it to the building. It’s there and ready to go otherwise.” But now even the landlord / builder are getting the run-around.

This service is supposedly “right there, almost ready to go”. Why can’t I just get it turned on already?

Anyways - thank goodness for Apple’s iDisk service (as part of their .Mac offerings). iDisk can be synchronized and worked with off-line. So I can at least keep my iBook up to date with my plans/actions list and a few other key documents, and I can update source code to work from home by bringing the iBook into work and using CVS. This helps with the offline mode - knowing that some information, including work stuff (useful in these busy times) can be brought with me even if there’s no connection. This is the downside to otherwise wonderful services like Backpack: except for emailing myself a copy of pages, I’m cut off from any information I may be collecting in a system like that.

This is something that I’ve been concerned about ever since Microsoft first started promoting “internet everywhere” inside of Windows. I don’t use Windows, so it didn’t concern me much (but it’s been interesting to see how their strategy effectively made the security problem they wrestle with today). But I always wondered: “what happens if there’s no internet? if the network’s down?”. The Dashboard in Mac OS X 10.4 is effectively useless on my iMac at home now. Not that I’d expect anything different: how can it update its backpack widget and all of those weather feeds with nothing to feed it? It’s just sad though, seeing that computer that is normally so great become so useless without a connection. I can’t upload photos, can’t update my web site, can’t collaborate with musicians, can’t shop for art supplies, can’t check the train schedule, can’t get new documentation, can’t get new music from the iTunes Music Store, can’t manage the Netflix queue that I’m finally going to set up… All of these things, taken for granted. And now, thanks to a stressful and confusing mess, I don’t know when I’ll be able to take them for granted again.

Helmet Head

2006-05-25T11:05:00.000-06:00

"But I don't have $100 and I can't come by with a truck for two weeks..." No piano for you! Life sucks; get a helmet.

Enjoy Forgetting

2006-05-24T10:58:00.000-06:00

I really enjoy forgetting. When I first come to a place, I notice all the little details. I notice the way the sky looks. The color of white paper. The way people walk. Doorknobs. Everything. Then I get used to the place and I don't notice those things anymore. So only by forgetting can I see the place again as it really is. - David Byrne as Narrator, True Stories (1986)

Post to MarsEdit from QuickSilver, using Markdown

2006-05-23T15:35:00.000-06:00

This is an AppleScript I cobbled together real quick to post to this weblog quickly from QuickSilver, using MarsEdit to handle the posting. I don’t claim to have any real in-depth knowledge of AppleScript, and I doubt this will win any “prettiest” or “best” code awards. I just thought it might be helpful to others.

    using terms from application "Quicksilver" 
        on process text ThisClipping
            my SendToMars(ThisClipping)
        end process text
    end using terms from

    on SendToMars(IncomingString)
        set mdcommand to "/opt/local/bin/Markdown.pl " 
        set webtext to do shell script "echo " & quoted form of IncomingString & " | " & mdcommand
        tell application "MarsEdit" 
            make new post window
            tell post window 1
                set body to webtext
                send post to weblog
            end tell
        end tell
    end SendToMars

Any text formatter could be used here – I went with Markdown for this situation as it seems to be the best fit for the tiny little text window you get with QuickSilver.

To install, I put this in Library/Application Support/Quicksilver/Actions inside my home folder under the name Post to Griddle. Any text typed can then be posted immediately. Your mileage may vary, depending on your MarsEdit configuration.

2006-05-23T15:05:00.000-06:00

This is a test, of sorts, to see how easy it is to write an applescript to send a post to GriddleNoise via MarsEdit.

It's an attempt to see how easy it might be to do some tumblelog style posts using QuickSilver as the entry pad.

Settling In

2006-05-22T16:14:00.000-06:00

I moved into my new apartment a week ago. Except for one glaring problem, it’s really quite nice. The one glaring problem is huge: no internet, yet. I’ve been a happy user of cable based internet for most of the past six or seven years, from Fredericksburg to Salt Lake. I haven’t even had a phone line since moving back to SLC.

I’ve been happy enough with cable internet, having had very few problems over the past few years. But right now, COMCAST is having troubles recognizing the existence of my new building – even though the builder remembers working with COMCAST to lay the cable. It’s there, apparently, waiting for the first tenant to place a subscription order. And hey! That’s me! I don’t think it’s any major fault of COMCAST’s though… Not really anybody’s fault. But I hope it gets resolved soon.

The apartment is small, but cozy. And it’s not even really that small. This is the third in a series of living spaces that have been open plans: the only doors (besides the entrance) in my last three apartments have been for the bathroom, storage areas, and (in the case of the last two) a/c and water heater access. What differentiates the new space is that it features an upstairs loft, letting me separate the living space from the working space while still keeping everything open. It also means that the ceiling in the working space is extremely high. Actually it slopes back as it’s an arched roof.

The building is a converted warehouse. It’s a newer warehouse than the ones I was previously in, with cinderblock walls. It was one of those high ceiling single-story warehouses with the arched roofs, which is where the room for the upstairs loft comes in. The cross beams up there remain exposed (providing fun places to try to hang / tuck things). The ceiling/roof is punctuated with clear skylights, with one in the loft area that can be cracked open for ventilation. I’ve enjoyed being able to sit / lie up there while watching clouds go by. The arched roof creates an interesting sense of coziness as it slopes down away from the living space. From my reading space in the corner, the place feels snug – almost like a cabin. I can then go over to my writing desk and peer into the workspace, enjoying the open nature. But while sitting and writing, it’s never overwhelming.

Another feature across my recent series of apartments has been hard flooring. Hardwood floors in one, painted plywood/particle board (yep) floors in another, and in the new space: brushed concrete!!! Damn that’s nice. Unlike the last two apartments though, I also have carpet in the loft – a feature that my dog greatly appreciates (thin-skinned bony greyhounds love soft surfaces).

While the kitchen and bath (on the main level) and living space (loft) feature regular lights, the workspace can be illuminated by two pairs of halogen lights. Excellent for painting, drawing, recording, everything! A large modern ceiling fan in this area circulates air and drives down the need to keep the air conditioner on constantly.

Perhaps the best feature of all is the neighborhood. I’m surrounded by metal recyclers, an old concrete plant my dad worked at when I was young (primarily empty now, although its large silos still seem to be in use). Empty warehouses, active warehouses and industry, random pockets of artists, trains, overpasses, underpasses, industrial back-alleys, a pyramid, and more. It feels good to be out just-ahead of the city. My last couple of places were a little closer to the heart – different sides of the heart, though. The area I moved out of (and where my office remains) has started to feel different over the past few months, and I don’t like the feeling. It’s not bad – in fact, I’m glad my office is still here. There is a lot of interesting growth happening and on the verge of happening here. And suddenly, it felt boring. It felt too contemporary when I wanted it to feel more creative, which is strange because I lived in the actual artist block of Artspace and had some great (and creative) neighbors. But the edge was gone.

I think my last trip to New York woke me up to this feeling. I played a show in the Red Hook neighborhood of Brooklyn and fell in love. I fell into a funk as my plane landed back in Salt Lake. Over the next couple of weeks I daydreamed of a new space to live and work. I hid in old films, images, and articles of SoHo, and scoured Flickr archives for photos from different neighborhoods and cities that captured an elusive feeling that I was wanting back. My lease came up for renewal, which put me on month-to-month until I renewed, and I decided to take a serious look around at some new options. Ideally I wanted a place I could convert on my own, with a big garage door and a layout that’d let me keep the dog(s) safe on one side while I worked on the other. I don’t really have the spare time or funds to support such an endeavor at this time. Still, I decided to look around and see what was available that would allow me to separate workspace and living space, or that I could buy. This was when a flyer showed up under my door announcing the lofts that I’ve now moved into.

As I walked down to check them out, I realized that they were in the heart of the industrial area that I loved. In fact, I had been walking through that area a couple of weeks prior and had been kicking myself for not having a camera along with me. The energy of the area was more active, fresh, raw, gritty, real, and beautiful – to me anyways. And upon seeing these new spaces and their design, I signed almost immediately. A couple of weeks later, I was moved in. It all happened so fast and naturally, just a straight-ahead instinctual thing. Compared to all of the stress I’ve felt this year, it was nice to do something big that felt just right.

Best of all – the dog loves it. In fact, she seems even more playful now.

All I need now is internet.

Tinderbox, OmniOutliner, and GTD (again)

2006-05-11T12:23:00.000-06:00

A few months ago I bit the bullet and purchased Tinderbox, an interesting outlining/mapping/linking application for Mac OS. I primarily purchased it to use Ryan Holcomb’s template for a Tinderbox based system for the Getting Things Done workflow, but I also dreamt of using it for more.

I haven’t used Tinderbox much in the past couple of months. Kinkless GTD 0.83 came out along with the OmniOutliner 3.6 beta (now in final release), and it solved many of the issues I had with earlier versions of the KGTD/OmniOutliner combination. Prior to OmniOutliner 3.6, there was just a single toolbar customization setting for OmniOutliner. In most Mac OS X Cocoa apps, you can customize the toolbar pretty easily. It’s just one toolbar – not the big mess of toolbars that the Office systems have become. Generally, it’s been a very handy feature.

As OmniOutliner grew terrific Applescript support, they allowed you to add your scripts to the toolbar. Kinkless GTD is, in fact, a handful of Applescripts that do an implementation of the Getting Things Done method of planning / management. By putting a few new buttons on the toolbar, you essentially got a new specialized application that took advantage of the terrific nature of OmniOutliner (still the easiest and most fluid outliner on Mac OS X, and looks great too). The downside was that if you wanted to work on other outlines, the GTD buttons would still be there even if they had nothing to do with the document you were working on.

With OmniOutliner 3.6, you can now have document specific custom toolbars, meaning that my KGTD outline no longer interferes with other outliner documents. A few other concerns were dealt with in the new KGTD release that made it feel much more natural, and I found myself switching back to the OmniOutliner/Kinkless combination as my “trusted system”.

There’s a big difference in styles between how OmniOutliner / KGTD work and how Tinderbox / GTDRules work.

About OmniOutliner + Kinkless GTD:

KGTD is an aggressive user of OmniOutliner’s extensive Applescript support (and in turn, OmniOutliner’s Applescript support has gotten even stronger).
KGTD’s main unit of work is a big synchronization script that moves and copies outline elements around. In the Getting Things Done method, you “plan in projects and work in contexts”, and often need at least two views of your actions – the plan list, and the actual “do this next in this place” list. Kinkless GTD manages both by copying/merging/moving.
By using AppleScript so extensively, Kinkless GTD works well with tools like Quicksilver, whose motto is “act without doing”. It’s easy to send an action into your KGTD inbox without ever switching applications or even taking your hands off the keyboard. Kinkless GTD has also done a lot of work to integrate with Apple’s iCal calendar, which in turn allows one to use various menu bar tools and Dashboard widgets to keep an eye on actions without having to bring OmniOutliner into focus. The use of applescript brings integration with other common tools.
However, KGTD requires manual execution of the synchronization command. It’s possible to end up with ghosts and duplicates since it’s doing so much copying and moving. It’s gotten a lot better when it comes to the ghosts and duplicates, but it means that an active action may have two or three copies in the same document, all separate outline rows.
OmniOutliner is a single-window application (well, single window per document). But with the new sections and navigation features in OmniOutliner 3, it’s quite easy to move around and focus on particular elements.
As a native and modern Mac OS X application, built by a shop with a long history of Cocoa/OpenStep/NeXTStep development, OmniOutliner also gets to take advantage of many advanced Mac OS X features. Services, text system plug-ins, etc. This includes automatic spell checking, in-place dictionary look-ups, and more (ie – with “Equation Service”http://www.versiontracker.com/dyn/moreinfo/macosx/14090 installed you can easily convert LaTeX source to a rendered PDF and back again, without OmniOutliner ever knowing anything about LaTeX)

About Tinderbox + GTDRules:

GTDRules is an aggressive user of Tinderbox’s system of agents and prototypes.
GTDRules main unit of work is a collection of agents that are always running. Agents in Tinderbox are live queries which can also act on the items that they match. In the document view(s), found items are displayed as children of the agent. Most of the agents in GTDRules are ‘contexts’ (again – plan in projects, work in contexts), showing the next actions for the office, for home, for errands, etc, under the agent.
Synchronization is always happening, so when an item is marked as complete, it disappears from the agent displaying it immediately.
Data is not duplicated – the agent results are treated as Aliases, which you can also make manually in Tinderbox. Aliases are like their desktop / file system counterparts: a symbolic link. This way, there are no ghosts or duplicates.
Tinderbox has a menu, customizable per document, called “Value”. You can use the Value menu to quick-stamp values on Tinderbox nodes. Since there are no checkboxes in the outline views, this is how you mark something as complete. But it’s also a way you can select many actions and set them all to be “Office” actions, in one swoop. Or put them on hold in one swoop. Customizing this menu is pretty easy.
GTDRules also takes advantage of Tinderbox’s “OnAdd” rule setting, allowing you to set default values for nodes added – ie, a work project can easily be set to have its children be ‘Office’ prototypes so that they show up in the Office context, but projects for an on-site customer could default to ‘Remote’ instead.
Tinderbox allows many windows to be open for a single document, and remembers their placement and displayed content. This allows one to take a big system like this and section it off into numerous views.
Tinderbox, however, is very much a classic style ‘Carbon’ application. It still feels like a custom classic Mac application, and does not take advantage of many modern Mac OS X features – even ones that are available to Carbon, such as Services. This isn’t a HUGE deal – at least it’s native – but it’s frustrating to use the Font menu when I have so many many fonts installed that are so easy to work with in Cocoa text areas (using the font browser in OS X).
Tinderbox has practically no Applescript support. It’s an island. Within the system there are a lot of things you can do. But you can’t easily send yourself notes from Quicksilver, from Mail rules, etc. There’s no integration with iCal or anything else. Using Tinderbox is almost like using Squeak on Mac OS X. Yeah, your mouse and keyboard and the clipboard all work, but beyond that it’s a universe unto itself.

Tinderbox’s isolation was a key figure in driving me back to Kinkless GTD / OmniOutliner. The GTDRules implementation on Tinderbox is a more “pure” implementation of the GTD workflow and easier to customize to one’s preferences (no applescript programming required). But data entry – especially of the quick “i need to remember to look at this later” variety – is not its strong suit here. There was something about it that started causing me stress when I’d look at my system. Maybe it was me getting sloppy and not cleaning / planning as well as I should. But something would just feel off.

All of that said, Tinderbox has a lot of great features that I’ve been meaning to explore. The euc.cx universe is ever growing, a great jumble of myth and fact, legend and memory. I’ve been wanting to start setting down all of the characters, locations, productions, etc, of it all and exploring the relationships. This is Tinderbox’s strong suit – links, relationships, visual maps (as well as outlines), prototypes, custom data, all should play in here. I really do need to keep myself still long enough to start learning how to do this. And I want to write again. Like really write, in hypertext, with relationships and cross references. I tried this once with ddec, a system based on an old implementation of the FTrain sitekit. But I hated writing in XML. (I just hate XML. I don’t care about it as a storage format, but I do hate writing in it, configuring in it, etc). This kind of writing is the real concept behind Eastgate as an entity. Whether anything will actually happen is hard to say at this point. But it looks like my writing desk will be in an excellent location in my new space and I hope it’ll help inspire me.

The view from where the desk will most likely be. (It’s a console table actually, just the right size for an iBook, a small lamp, a pile of notebooks, and the dog’s leash):

Studio Peak (Inspirational Peek)

2006-05-05T17:48:00.000-06:00

As I’m getting ready to move to a new loft that will give me a nicer, more dedicated working space, I’ve found myself doing a lot more painting. I was working on large canvas in front of this stack of metal cube-ish storage things. As I was taking some pictures of the work in progress, I realized I needed to get a shot of what sits up top. Here, in bite sizes, are a lot of representative elements from artists, designers, factories, and companies that are a pretty strong source of inspiration. So, to step away from the usual crankiness and software focus for a bit, I thought I’d share a bit:

SuperSampler, Lubitel, LCA, and Baby Holga (110 Format!) Cameras. The 120 Format “standard” Holga is on top of my desk and not with the rest of the family. I do love these cameras: they have a lot of personality, with the LC-A being my favorite point’n’shoot 35mm camera ever. I don’t do film as much these days. For a few years my walk home took me past a few places where I could get film developed (including the 110 and 120 formats), often dropping the 35mm off in the morning on the way to work and picking it up on the way home. I haven’t had that since I moved into my current place, and that’s taken a big bite out of me rolling over new rolls as often. Plus I do have the digital camera that’s great for taking endless pictures of the dog, of paintings, and flash conditions. But it’s a soulless camera compared to these guys, most of which are completely themed around a terrific (and fun) mantra: don’t think, just shoot.

Brushes, Brushes, Brushes. Some of these brushes came with me back from Virginia and are still in pretty good shape – probably because I don’t paint nearly as much as I used to. But when I got my current place I finally got that container they’re in, which has made access to my core set much easier than my previous contraptions.

Helmut Lang promotional cards (from one of the last seasons). Helmut Lang was my favorite designer, with a terrific design aesthetic that was applied to everything under the brand. The flagship store in New York, designed by Richard Gluckman, was a terrific piece of architecture. Completely opposite of most New York stores (crowded, impersonal, or overly flashy), the Helmut Lang space used design to usher you in to a private area with large aisles and easy access to everything, without ever telling you what tricks it was pulling. It was an amazingly comfortable space. There was a “complete experience” about Lang that I sorely miss.

Ryoji Ikeda ‘Formula’ DVD and Book. Another terrific aesthetic minimalist, Ryoji Ikeda’s installations and a live set are showcased in this excellent volume. Ikeda is another artist who, like Lang, seems adroit at knowing what to take away, what to shift, and what to tuck into a piece of art (design, clothing, composition).

New reissue of Pizzicato Five’s Romantique 96, in front of Merzbow ‘Senmaida’ card (making it look like Maki and the Dalmations are on a Merzbow promo here). Two more excellent Japanese artists. Pizzicato Five kept many Jet Age fashions, designs, and ideals alive in their music. Their album packaging was often a step above (with only a couple of the U.S. releases falling short). Credit cards, luggage tags, postcards, stickers, bookmarks were among the things one could find inside a common release. And of course, there were the pictures of Konishi and Maki. Beautiful music, and a favorite artist of mine for years. Merzbow is almost their opposite: harsh noise. Hundreds of releases across an almost equal number of labels and distributors. He carries on almost a pure dadaist spirit. The cover art and packaging quality can vary wildly. The noise is almost always top-notch and he remains one of the kings. The postcard (mostly hidden in this photo) is from Senmaida which featured art by Val Denham that I just had to have for the picture alone.

Stack of book size Hafler Trio and Pizzicato Five releases, more albums in excellent packaging. The Hafler Trio being another terrific inspiration on many many fronts. Among other things, The Hafler Trio has played with sound-as-high-art, putting out releases in very limited numbers and with many extras in the packaging or experience to make it especially worth obtaining.

Brazil!. On VHS. Ahhh, Terry Gilliam.

“I draw on anything for inspiration – a fond memory, a piece of paper, a wall in a train station” – Danger Doom/Talib Kwali

Tools

2006-05-05T11:05:00.000-06:00

I’ve never been a fan of IDE’s. I grew up on Commodore 64 BASIC – always on, just start typing in code and it works. I used IDE’s a little bit way back in my high school computer science day, but I remember it as little more than an editor / compiler combination (Lightspeed Pascal on Mac System 6). After that, I did very little programming for a few years except for casual purposes until I started using Python in 1996.

Most of my Python programming career has been focused on web development, and subsequently most of my development environments have been on development servers. For a good stretch of time, my editor/environment of choice was XEmacs. A couple of years ago I switched to VIM (lighter weight, didn’t need to bother setting up X Windows any more, and I could go into a stripped down production server running bare bones VI and not feel completely lost). A couple of months ago I started using TextMate pretty heavily, and started working on my desktops again since Mac OS X is, after all, Unix. Using DarwinPorts I could get the requirements for both Zope 3 and Ruby on Rails going pretty easily. And in this “just a plain but smart editor please” camp I remained, using the terminal to start and stop servers, to debug, to watch logs, etc.

But the other day I decided to check out RadRails, an Eclipse based IDE for Ruby/Rails development.

And holy shit, I was impressed.

Traditionally IDE’s have bugged me with their ever proliferating collection of panes and tabs. Too much noise! Too much to set up! Too much fighting to make it work my way! But with a little bit of preference tweaking (to point RadRails at the right Ruby / Rails installations on my desktop), I got RadRails running. I didn’t use it heavily (I’m not doing active Rails development, but continue to check it out from a point of envy / curiosity), but I was impressed with what it had: good knowledge of the Rails Generators, the database explorer, consoles, log watchers, etc. All things that I’m sure IDE users take for granted. I was especially impressed with how easily one could maximize any pane to be full-window size, and then go back to the many-many-many-views, using just a double click.

While it still doesn’t feel 100% native, RadRails/Eclipse actually performed and behaved rather well in Mac OS X – better than any other Java application I’ve used. Better by a long shot.

Now where’s something like this for Zope 3? Especially for my bastardized system using SQLAlchemy? :)

Snakes and Rubies in the Temple of Bondage and Discipline down in the City of Heartbreak and Needles

2006-04-13T10:29:00.000-06:00

One of the new topics making the rounds is a spate of responses to an old (2-3 years, at least) post comparing Python and Ruby. One piece of interest is this bit:

Larry Wall, the designer of Perl, has the slogan “There’s more than one way to do it”. In contrast, Bertrand Meyer, the designer of Eiffel, says “A programming language should provide one good way of performing any operation of interest; it should avoid providing two.” Ruby follows the anarchist approach of Larry Wall, while Python follows the bondage-and- discipline approach of Eiffel.

While it’s stated in the Python import this (aka “The Zen of Python”): There should be one – and preferably only one – obvious way to do it, that’s seldom the case in actuality. Ian Bicking’s response to the post linked above mentions this as well. There are well meaning attempts at consistency – but do they happen?

Python makes some interesting design trades. As many note, it’s not a so-called “Pure Object Oriented” system. You can write Python in a purely procedural style, in a Modula style, in functional style, in pure scripting style, in “pure OO” style, or in a mix of all.

The comprehensive standard library is a big mix of all of these styles. Some of it stems from certain core modules being thin layers over common C libraries. Some of it stems from different styles of different authors. Some of it stems from age.

I’ve generally been a fan of Python’s modular approach. Common modules like os and os.path provide a nice – but thin – abstraction over operating system specific behaviors, especially in regards to the file system. Python’s early development, according to legend, happened on Macs (and considering Python’s age, it should be noted that this was long before Mac OS X). Classic Mac OS, DOS/Windows, and Unix all spell paths differently. In the mid nineties, as Python was starting to really find its audience (including me), its primary rival was perceived to be Perl. Perl’s history is steeped in Unix style shell scripting. Perl scripts, then, typically had very Unix style paths and elements inside of them, just by default. While there were some good Perl implementations available for classic Mac OS, it was hard to play with other peoples code – there were just a lot of system expectations built in, especially at the time I started experimenting with the two P languages.

Python, on the other hand, had the ‘os’ and ‘os.path’ modules. These modules contain, among other things, knowledge of how the current OS spells certain file system elements, like ‘parent directory’ or path separator. Classic Mac HFS paths were separated by colons, for example, and ‘pardir’ was not ’..’ but ’::’. And, of course, Windows/DOS uses backslashes instead of forward slashes. So while you could write paths the normal way, it was (and still is) considered good design to use the os module’s tools:

    pydir = os.path.join(os.pardir, 'lib', 'python')
    appdir = os.path.join(pydir, 'example')

But! The os module’s elements for working with paths is very procedural. There’s not really a built in concept of a Path object. That may be changing in Python 2.5 if PEP 355 really finds its way in. Then the above could be more like:

    pydir = Path(Path.cwd().parent, 'lib', 'python')
    appdir = Path + 'example'

While that doesn’t look like much, it’s when one looks at some of the proposed replacements that one can see the difference:

    CURRENT (using 'os')
    --------------------
    DIR = '/usr/home/guido/bin'
    for f in os.listdir(DIR):
        if f.endswith('.py'):
            path = os.path.join(DIR, f)
            os.chmod(path, 0755)

    NEW (using 'path')
    ------------------
    for f in Path('/usr/home/guido/bin').files("*.py"):
        f.chmod(0755)

Taking that old – but useful (and usable) – procedural code and turning it into a class / objects is just beautiful, and is one of those “why didn’t they do that before?” things. The new Path class also brings a lot of functionality that’s dispersed in other modules (that people may not even be aware of) together, like ‘glob’, ‘fnmatch’, and ‘shutil’. (It looks like Ruby 1.8 picked up something similar with the new Pathname class).

The point being – this is an area where there are really quite a few ways of doing things in Python. Is it good? Is it bad? It’s a nice kind of freedom, I guess, to be able to write in both procedural/modular and OO styles. There’s definitely no hard “bondage and discipline” here.

There’s also no hard “bondage and discipline” in Python when it comes to naming. There are some good practices, but not everyone follows them (myself included). I have a personal style guide, much of it taken from practices we had at Zope corp (4 spaces indentation, must fit in 80 columns) and styles picked up over recent years (all package/module names lower case, all classes capitalized, never import * if it can be avoided, etc). And I wince when I see code that differs – mixed case package/module names being one of the biggest turn-offs to me, personally. import * has generally been offensive as well. One thing that I’ve historically liked about Python has been the ability to trace back any name I see in code. “Path… ah, it came from from path import Path! And implements.. Oh, it came from zope.interface import implements! and self.bluecheese is obviously an attribute available to the instance!” import * tends to get in the way of name chasing. I like to learn code by reading it (since most documentation tends to be flaky), so being able to find where ‘mapper’ or ‘begin’ or ‘desc’ or ‘Table’ came from is something I like. And for maintainability reasons, I’ve liked being able to find and trace names within a function/class/module without having to speculate.

But that’s just my preference. Python’s namespaces and general rules here have generally worked out for me. But as I’ve gone on, I’ve found myself much more critical of other Python code than I should be, which is contributing to this crisis of faith I’ve been going through. Whereas when I’m ready Ruby code, I have different expectations and can be a lot more forgiving.

But there’s something significant that I’ve come to realize about Ruby. It can be argued that Python’s biggest “bondage and discipline” element is its use of significant whitespace (clear structure by indentation is enforced at the language level – although there are ways around this). Ruby, on the other hand, enforces significant naming conventions. As I’ve been reading Ruby code (with most of my experience being limited to Rails and code from why the lucky stuff), I’ve grown to appreciate this:

Constants begin with a capital letter. Classes and modules are constants, and so they are always capitalized. Pathname.pwd, ActiveRecord::Base, and so on.
variables are lower case (or at least, start with a lowercase letter). This includes methods.

Understanding those two elements alone (along with learning Ruby’s symbols for self.attribute access, etc) has made it really easy to start reading and understanding Ruby code. I can understand how Rails can be a system that “prefers convention over configuration” – there seems to be a fair amount of convention enforced in the language and core library as it is.

Another thing I find interesting is that object attributes are not exposed directly, but can be exposed easily through methods. With parenthesis being optional on method calls, this makes zero-input methods (ie – ‘reader’ methods) much easier to use and work with. An historical Zope 1 and 2 problem was the case of ‘id’. Some objects would just have their ‘id’ (name within their container) set as an attribute. Others would compute it and have a callable method. In DTML and TAL, you could refer to ‘id’ and the templating language’s expression engine would figure out if it needed to be a call or not (and then translate it into ‘id()’ if needed). So sometimes, obj.id would return a string like ‘monkey’. But other times it would return (bound method id of ...). Python’s descriptors (since Python 2.2) make this a little bit better, but in my encounters with Ruby so far, it’s made code more readable and consistent and predictable. The proposed Path class for Python (mentioned above), for example, has a mix of properties and methods:

    curpath = Path.cwd()
    documents = curpath + 'Documents'
    curpath.parent
    curpath.realpath()
    curpath.mtime()
    curpath.isdir()
    str(curpath)

Compared now to Ruby’s Pathname object:

    curpath = Pathname.pwd
    documents = curpath + 'Documents'
    curpath.parent
    curpath.realpath
    curpath.mtime
    curpath.directory?
    curpath.to_s

The Ruby code just looks cleaner, more consistent – even though one could put parenthesis on all of these calls. ‘Pathname’ is obviously (or highly likely) to be a class. For the Path class proposed in Python PEP 355, one of the open issues in the PEP is this:

The name obviously has to be either “path” or “Path,” but where should it live? In its own module or in os?

Let’s look at that first part again: The name obviously has to be either ‘path’ or ‘Path’. Either? Well, a lot of core Python classes and types are lower case – although many of them weren’t really types/classes until recently. I’m talking ‘str’, ‘object’, ‘file’ (aka ‘open’), etc. Some of this has to do with the history of the language and how it has grown / changed over the years. But obviously there’s more than one way to do it if you can’t really be sure of the proper way to capitalize it.

Indentation can be significant when scanning over code (not even reading into details). So can naming and other behaviors. Python programmers may see the ‘end’ keywords for Ruby’s blocks for for loops, if statements, etc, and go “ha! Python don’t need that line noise!”. But Ruby programmers may just as easily look at parenthesis on method calls that take no arguments and go “Ha! Ruby don’t need that line noise!” They may also look at things like ‘list.sort()’ and wonder why they don’t get a sorted copy of the list back (and may also wonder why it took until Python 2.4 for Python to finally grow an in-line ‘sorted copy’ function of its own). Ruby’s use of punctuation helps distinguish between the destructive (changes the object in question) and not destructive (returns a copy of the object in question):

    Python:
    >>> alist = [3,2,7,13,8]
    >>> print alist.sort()
    None
    >>> alist
    [2, 3, 7, 8, 13]
    >>> alist = [3,2,7,13,8]
    >>> print sorted(alist)
    [2, 3, 7, 8, 13]
    >>> alist
    [3, 2, 7, 13, 8]

    Ruby:
    irb> alist = [3,2,7,13,8]
    irb> p alist.sort
    [2, 3, 7, 8, 13]
    irb> alist
    => [3, 2, 7, 13, 8]
    irb> p alist.sort!
    [2, 3, 7, 8, 13]
    irb> alist
    => [2, 3, 7, 8, 13]

Things like list.sort and having len(), sorted(), etc, being separate methods are things you get used to in Python. At the same time, it gives bad guidance for best practices (or just good practices) for writing code. If writing something like a Field object which can be bound to another object – should ‘bind(target)’ return a new copy of the Field bound to the target, or should it bind in place? In Ruby, one might write two methods, bind(target) and bind!(target), with the second one denoting that it changes the Field in place.

I don’t know how consistent other Ruby developers are in following that standard, but it’s great to have as an option. I also like the question mark used for boolean methods: alist.empty? alist.include? 7, and so on. Again – in my limited experience with Ruby, it’s been easier to understand a lot of the code I’ve seen:

    popped = alist.pop if alist.respond_to? :pop
    if popped.nil?
        puts "Bastard didn't pop" 
    end

    if hasattr(alist, 'pop'):
        popped = alist.pop()
    else:
        print "Bastard couldn't pop"

The differences are negligible here, mostly a matter of preference. How about testing for emptiness? Not just testing for truth – but testing to see if a container is empty?

    Ruby:
    if alist.empty?
        puts "It's Empty" 
    end

    Python:
    # truth testing
    if not alist:
        print "It's Empty" 
    # length testing
    if len(alist) == 0:
        print "It's Empty" 

    # Writing a custom 'empty' tester.
    def empty(obj):
        return (len(obj) == 0)

    if empty(alist):
        print "It's Empty" 

    # But 'empty' could also be destructive
    def empty(obj):
        obj[:] = []
        return obj

    if empty(alist):
        print "will always be empty because this version of empty destroys"

More than one way to do it. Of course, the truth testing is the most commonly used in Python – but there are times when ‘emptiness’ and ‘truthiness’ have different meaning / impact. alist.empty? reads easy. The punctuation in the name differentiates it from a method like empty or empty! which might clear the object out (like ‘clear’ does in Ruby arrays). So this useful feature and convention makes it possible to look at an objects list of methods and pick out the queries.

So – am I saying Ruby is better than Python? No. Nor am I saying it the other way around. There are some very different designs here. But the enforced naming conventions and other small touches – constants versus variables, leading to consistent class and method spellings; punctuation in method names; ‘optional’ method call parenthesis causing one not to have to worry about whether they need to use object.id or object.id() – have enabled me to read large framework code like that in Rails or Hobix as easily or more easily than Python, even though Python claims to be written for readability and I’ve been programming in Python for a decade now and I’ve still scarcely used Ruby at all. There really seems to be a clean and simple elegance about it that I find myself yearning for.

The communities are interesting too. Python can’t even seem to settle on a good documentation tool. pydoc is way too simple in its HTML mode (but quite nice in the interpreter when you can do help(obj)). epydoc is quite comprehensive and supports comprehensive markup options, including the ability to document instance variables, method parameters, etc…, but its output can still be a bit hard to read.. Or at least, to differentiate the wheat from the chaff (I don’t think it’s epydoc’s fault, Python modules and objects may just be exposing too many extra objects that don’t always need to be documented and those get in the way of finding useful help). The almighty effbot has added PythonDoc to the mix as well. And there’s also Pudge. Zope 3 has its own API documentation tool with some specific knowledge of additional Zope 3 elements – sadly there’s no publically available online version to use for reference (yet).

And with all of these options – there’s nothing that quite matches the discoverability and usability of the Ruby on Rails API Docs. Again, I’ve gotten far better answers out of it than on any of the Python web tools I’ve evaluated… Ever. Which is why reading the source and being able to follow names is so important to me in Python. The source is often the best documentation you can easily get to and through. The Python standard library is very well documented. But beyond that, it’s often a lot of work. Ian Bicking wrote in his post Orthogonality is Pretentious (a post I generally disagree with – orthogonality leads to a pleasant predictability, I think, in languages like Smalltalk and also in Ruby) about PHP:

One example of this design in PHP is the lack of objects in the standard library. Instead it is one very long list of functions. If you want to do something with the standard library there either is a function for doing it, or there is not a function. If you want to see what you can do, you can just scan through the index.

And that’s what I like about Ruby on Rails. In the API Docs, one of the frames lists all of the methods in the system. That alone has been useful for just looking up functionality. One big list of functions (essentially), most of which are available where you need them most. Easy list to turn to and scroll through.

Similar functionality is in the WebHelpers library for Python – but there’s no version of the docs (at this time) that have that nice “big list of functions”. It’s click / try / click / try / click / try. At least it’s documented! But still, it just takes more time to weed out pertinent information.

Jumping to api.rubyonrails.org and scrolling through that bottom list to find ‘has_many’ again to find its options is a very quick process. (Just like most Python standard library lookups – the library and module indexed reference for Python is damn handy!)

Ugh. Where was I going with all of this? Nowhere really. I still like Python and it’s still my working programming language. But there are a lot of things about Ruby and what its community has been producing over recent years (beyond just Rails) that are increasingly enticing.

Because I Could (SQLAlchemy, Zope 3, Metaclass Fun, Etc)

2006-04-07T19:04:00.000-06:00

I continue to be (generally) impressed by SQLAlchemy. After a couple of days worth of work, digging around the guts of Zope transactions and thread management along with SQLAlchemy’s guts of similar nature, I think I have a system in place that will work fine with Zope 3. Zope 3 uses a two phase commit transaction management system, whereas SQLAlchemy generally supports the basic ‘begin / commit / abort’ cycle. However, by taking advantage of SQLAlchemy’s nesting of transactions, I essentially have the following model in place:

Zope transaction begin—SQLAlchemy engine begin (starts a fresh outer scope and calls ‘begin’ on the database connection itself)
On Zope commit start (first phase of commit)—SQLAlchemy objectstore commit (sends changes managed in SQLAlchemy’s unit of work / object graph over the wire)
- This is of interest because objectstore commit also calls engine.begin() and engine.commit(). SQLAlchemy is designed for ‘reentrance’, so that multiple calls to something like engine.begin() are OK – as long as they’re matched by an equal number of finishing calls like engine.commit(). When the nesting count is back to zero, engine.commit() saves.
- In two phase commits, it’s allowable for an error to happen here. It’s probably not preferred, but it’s OK.
On Zope commit finish (final phase of commit)—SQLAlchemy engine commit. This ‘commit’ matches to the ‘begin’ at the start of the Zope transaction, and sends the ‘COMMIT’ statement to the target database.

Beyond that, I’m not doing too much work to track changes in either environment (SQLALchemy, ZODB) in this current setup. ZODB’s persistence manages the loading and unloading of its objects, and SQLAlchemy manages its own too. It’s not that I’m trying to mix them – it’s just that Zope really really really wants to run with the ZODB, so even if it’s not used, it’s probably going to be there.

If this system ends up actually doing its job, I’ll share the code. I’m trying to write it separate from the application I’m developing.

Also: for kicks, I’ve toyed with how I might get ActiveRecord style sytax (which I like) using Python and SQLAlchemy. This mostly works as it stands right now:

    from ruts.storage.base import Base
    from ruts.storage import act
    import db.tables

    class Venue(Base):
        act.fromTable(db.tables.venues)

        act.hasMany('Event', 'events', order_by='start_date')

    class Presenter(Base):
        act.fromTable(db.tables.presenters)

        act.hasMany('Event', 'events', order_by='start_date')

    class Event(Base):
        act.fromTable(db.tables.events)

        act.hasOne(Presenter)
        act.hasOne(Venue)

    >>> v = Venue.get(1)
    >>> v.events
    [<exmple.model.Event object at 0x104a930>]
    >>> v.events[0].venue.title
    'Testy McVenueVille'
    >>> v.events.append(Event(title="Jackson"))    
    >>> v.commit()
    >>> v.events
    [<example.model.Event object at 0x104a930>, <example.model.Event object at 0x2b2c090>]
    >>> e2 = Event.get(2)
    >>> e = Event.get(1)
    >>> from datetime import timedelta
    >>> e2.start_date = e.start_date + timedelta(1)
    >>> e2.end_date = e.end_date + timedelta(1)
    >>> e.end_date
    datetime.datetime(2006, 4, 8, 23, 0)
    >>> e2.end_date
    datetime.datetime(2006, 4, 9, 23, 0)
    >>> v.events[1].end_date
    datetime.datetime(2006, 4, 9, 23, 0)
    >>> v.events[1].title
    'Jackson'

SQLAlchemy

2006-04-05T17:27:00.000-06:00

Among the short list of good (or at least, personally interesting) Python projects under development today stands SQLAlchemy. It’s a powerful toolkit for working with relational databases in Python, including doing object-relational mapping. It offers a lot more power and flexibility than SQLObject (Python) and ActiveRecord (Ruby), while still staying pretty close to those in relative ease of use (ie – no Hibernate style XML mappings).

Especially interesting is that SQLAlchemy is a SQL toolkit first, and O-R mapper second. It has a lot of support for query construction, compilation, native feature support, feature-agnostic support, pre-processing, post-processing, and more, that can also be used in the mapping layer. Already it seems like it’d be free of the kind of issues that I ran into when I tried to deal with a moderately complicated Inheritance / contained object scheme with Rails and ActiveRecord.

Yay Object Database

2006-03-29T21:38:00.000-07:00

I’ll admit – I’m very tempted by Ruby on Rails. I like its “Ruby Everywhere” philosophy. I like its “beauty leads to happiness” philosophy. I like its “damn straight it’s opinionated” attitude. I like how easily it gives you an application starting point out of the box (oh, you don’t know how I’ve been struggling with this this week). At least, I like all of this in theory. I haven’t tried making any real application in it yet. I did a quick version of the Knowledge Base application, just to see some results and see how using Tags felt compared to what I’ve done in Zope 3, but it's nothing close to complete (or even to matching the Zope 3 implementation).

I’ve been working on another customer’s application in recent days and have been thinking “this would be a fun test to develop and compare!” The thought became more prominent as I’ve been working on a batch of customer requests – seemingly simple little things that are turning out to be much harder (in most cases) than I had estimated. If not harder, then just long. Now, some of this struggle is in Javascript and all of the joys of form/widget/etc design. Some of it is just data migration – extracting objects, etc. Replacing a single text attribute with a single sub-object with multiple fields. Combining a couple of field pairs on an object into two instances of a sub-object. And so on. That’s where some of the arduous part has come in – all of the ZCML and Schema required just to get a couple of simple classes registered and secured. And I had to roll a new ObjectWidget base class (I didn’t like the one that’s in Zope 3 at this time). With all of that out of the way, however, now I’m moving on… Mostly.

Anyways, I’ve started exploring how this application might work in Rails. And it didn’t take me long to find fresh love for the ZODB.

The ZODB is a full Python persistence system. It’s native. A tuple is a tuple, string is a string, a CrewContact object is a CrewContact object, and so on. When developing in Zope 3 in the (general) fashion that I’ve been developing – straight Persistent model objects, all of the views, adapters, etc, coming in off disk (looked up and bound by the component architecture, not by Zope 2 techniques like acquisition) – using the ZODB for a new application is straightforward and quite easy.

I think an object-relational mapper is good when developing against an established database. And usually there needs to be some strong mapping for that to work and match database policies against expected object behavior. Rails’ ActiveRecord is a really nice system in many respects, and it provides a lot of nice features for fairly natural expressions for establishing relationships, doing validation, etc. But it also seems to expect a fairly rigid structure in place to work. Not that this is necessarily bad. But it begs the question: if you’re going through such terrific overhead and making certain restrictions (Single Table Inheritance, which I’ll get to in a minute), why use an RDBMS at all?

Well, I know some of the answer to this, and I agree with it in part: the DBMS vendors (including open source ones) are better at the data storage, retrieval, distribution, and connectivity side of things. “Let the database management system handle that,” some say. “Let SQLObject and ActiveRecord make it feel like natural objects with a minimum of fuss.” OK. That’s admirable.

But I realized today that it’s been a long time since I’ve had to think relationally just to store my objects – objects that have meaningful relations. Specifically, I ran into this problem.

I have a class for Itinerary objects. An Itinerary is a container, which can contain many Legs. There are three types of legs, sharing a common base class/interface.
- ScheduledMeeting – Just a meeting that’s taking place in this itinerary.
- GroundTransportation – A car service, etc. Who’s providing it, any other notes (where to meet for pickup, etc).
- Flight – many fields (and in my latest implementation, sub-objects) with information concerning a flight, its crew, its passengers (crew and passengers linked lists to their respective targets), crew contact information, flight details, other notes.

I’ve been wondering how to do this in Rails. In Zope 3, I have a base interface, IItineraryLeg and specify that an IItinerary implementation contains them:

  class IItinerary(IItinerarySchema, IContainer):
      """ 
      Base itinerary information.  ItineraryLegs are associated
      with Itineraries to give trip details.
      """ 
      contains(IItineraryLeg)

And in the implementation:

    class Itinerary(BTreeContainer):
        implements(IItinerary, IItineraryContained)

        title = DublinCoreProperty('title')

        start_date = None
        end_date = None
        # Use ``Business`` as default, as is also specified in the interface
        trip_type = u"Business" 
        notes = u"" 

        @property
        def legs(self):
            return sorted(self.values(), key=startDate)

    class ItineraryLeg(Persistent, Contained):
        implements(IItineraryLeg, IItineraryLegContained)

        start_date = None
        end_date = None
        notes = u"" 

    class ScheduledMeetingLeg(ItineraryLeg):
        """ 
        'Scheduled Meetings' add nothing to the IItinerayLeg schema, really
        """ 
        implements(IScheduledMeetingLeg)

    class GroundItineraryLeg(ItineraryLeg):
        implements(IGroundItineraryLeg)

        transport_type = u"" 
        service_name = u""

See, these two legs are very simple. In ActiveRecord or any object-relational mapper using Single Table Inheritance, this would have been easy. But it’s that Flight leg that gets messy. Very messy. By my understanding of Single Table Inheritance, the flight leg, ground leg, and scheduled meeting legs data would all be mashed up in one table. If I were designing tables in an RDBMS, I would never design that way – and I’m no genius Relational designer.

The point is, there are obviously situations where my expectations of objects AND relational storage, as mapped by ActiveRecord, don’t quite mesh up. I’m not belittling the system – I just realized that (a) I have no idea how I’d want to make these tables (especially since I have some equivalent of has_one to tiny little objects, a transition made this week. Not just to tiny little objects, but a couple of them. Has two?), and (b) after I got them built, I have no idea how to map them into ActiveRecord. I may, however, be distracted things that I may or may not need, like new features Polymorphic Associations and Through Associations.

I guess, after thinking about it, I would write the three legs as separate tables/objects and write one legs action to combine them (in Rails)?

  def legs
    (ground_itinerary_legs + scheduled_meetings).sort do |a,b| 
      a.start_date <=> b.start_date
    end
  end

Well, it let me do something like that in the console, but not in my active record class.

Anyways, the point is that for many (not all, but many) tasks where a new “web application” is being created, a system like the ZODB is great. No translation. No having to bend your brain around RDBMS table creation and styles, and then back to objects, and to what can be done in between. Python data, through and through. The object database doesn’t work for every situation. But damn, it can be nice.

Too Much Effort?

2006-03-27T14:55:00.000-07:00

I’ve spent way way way way way too much of the day trying to code in a couple of simple objects into a Zope 3 application. They’re ridiculously simple, for the most part, with a couple of specialized business requirements beyond them.

Some of the slowdown has come from me back-pedaling on a couple of design decisions earlier in the day. But as I slough through just the basic ZCML to get these stupid little persistent objects registered into Zope (after the amount of work it took just to get here), I feel another headache coming on. And I’m not even to the actual hard part of the work yet. And this task is still a small part of all that I had hoped to get done today. It’s something that really should have been done a couple of hours ago, and I bet I have an hours worth of work ahead of me on this one stupid task alone.

No! Fun! Til Brooklyn!

2006-03-16T19:26:00.000-07:00

Mar 17, I'll be playing at No Fun Fest 2006, in the basement for Hospital Productions night. No Fun Fest runs Mar 17-19 at The Hook (18 Commerce Street) in Brooklyn.

Smegma, Ikue Mori, Zbigniew Karkowski, Sickness, and me, the always yodeling aodl.

Show starts around 7:30 each night, and I'm one of the first to go on. Come on down and watch me rock the trumpet tuning mic like you know I do it right.

... Update, 27 Mar 06: I realize now that this didn't get published until way too late. Blogspot was having connection issues when I rapidly tried to post this before I left town, and I forgot about it until today's update. oops.

ZCML Needs To Do Less

2006-03-13T09:20:00.000-07:00

A few months ago I defended ZCML pretty staunchly. These days, it's becoming feature #1 that makes me NOT want to use Zope.

For the uninitiated, ZCML is an XML configuration language to load up Zope components. I think that having registration separate from implementation, even in another 'language', is a good thing. But ZCML has also taken on heaps of automation capabilities. And those suck. Instead of respecting the Python programmer, it holds your hand and provides a lot of automatic features. Sometimes that's nice. A lot of times, that's wrong. You can't subclass, you can't override, you can't provide slightly different behavior. And don't even think that you can read the code behind the ZCML directives. Most of the code is full of internal handshakes in the configuration loading system. And if all you're trying to do is figure out whether you can provide javascript to a ZCML 'menuItem' directive: good luck trying to figure that one out.

So I stand firmly behind this: ZCML Needs To Do Less (although I don't agree with all of the examples shown).

Seaside, Rails, Helpers, Formlib, helping web applications through smart components, tools, and linking

2006-03-09T23:33:00.000-07:00

A web application framework that truely fascinates me is Seaside. Seaside, written in Smalltalk, has great vision and ideals for web applications by using continuations to suspend and resume code. It breaks free of the typical request/response cycle, and manages state to a fault.

This code, from an example store application for Seaside, is the go method of the WAStoreTask object. This method runs the store. Stay with me now, this is important. This runs the store. This is the flow. It’s native Smalltalk. It’s not XML, YAML, An Array or list. It doesn’t get translated into any structure like that. Upon first visit to the application, this code starts executing.

go
    | shipping billing creditCard |
    cart := WAStoreCart new.
    self isolate: [
      [self fillCart. self confirmContentsOfCart] whileFalse
    ].

    self isolate: [
        shipping := self getShippingAddress.
        billing := (self useAsBillingAddress: shipping)
                    ifFalse: [self getBillingAddress]
                    ifTrue: [shipping].
        creditCard := self getPaymentInfo.
        self shipTo: shipping billTo: billing payWith: creditCard
    ].

    self displayConfirmation.

The isolate method calls wrap a block in a WATransaction, making that transaction run over multiple requests. I’m not sure if this WATransaction maps to how I’m used to them as a Zope developer, but I do gather that it cleans extraneous objects up that were racked up in that ‘isolate’ call.

The use of continuations and a sort of invisible ‘session’ management that runs behind it makes for very brief code. Looking at my own Zope 2 e-commerce application, I see many lines spent having to look things back up: getting data out of the Session, getting data out of the Request, getting data out of the Database(s), and so on and so on and so on. With each request/response cycle, especially late into the Checkout process (which depends on all of the accumulated data of the shopping experience), the list of dependencies gets bigger. Guards have to be in place to ensure that the session and account are still active late in the checkout process, and that going to one of the checkout pages with an empty cart is disallowed… So it’s interesting to look above and see just a small handful of expressions boil down the whole experience.

An aspect of Seaside that is especially interesting is action calling, via hyperlinks or form submission. I also like how it does HTML generation, preferring to use objects/methods instead of templates. This is from a WAStoreItemView. renderContentOn: html is a common method that all the views implement. Unlike the so-called ‘MVC’ that is being popularized in Ruby on Rails, Turbogears, etc, this is actually closer to Zope 3’s views wherein it’s an object that wraps (decorates) another object.

renderContentOn: html
    html heading: item title.
    html heading: item subtitle level: 3.
    html paragraph: item description.
    html italic: item price printStringAsCents.
    html form: [
        html submitButtonWithAction: [self addToCart] text: 'Add To Cart'.
        html space.
        html submitButtonWithAction: [self answer] text: 'Done'.
    ]

Still straightforward and readable. Especially interesting are the submitButtonWithAction calls. The first argument is a block that gets executed when that button is clicked. addToCart is straightforward enough:

addToCart
    cart add: item

The page gets rendered as normal, staying on the WAStoreItemView, but each time you click ‘addToCart’ you’ll watch the cart display component fill up and fill up. [html anchorWithAction: [cart remove: anItem] text: '-'] is in that cart display to, of course, remove the item.

Back to the WAStoreItemView renderContentOn buttons, the other interesting one is the button for Done. It calls self answer. What’s it answering? A call! Specifically, this call from WAStoreFillCart. WAStoreFillCart is the component that’s responsible for browsing, searching, and displaying items. Any link to displayItem: anItem calls this:

displayItem: anItem
    main call: (WAStoreItemView new item: anItem; cart: cart)

What calls this display item, and why does it matter? Think of most web applications – there are more than a few ways to get at the data inside the application. There’s searching, there’s browsing by ‘tag’, there are dashboards (big overview pages), there are personal pages, there are direct listings. When you go from one of those listings to something more detailed, how do you go back?

In Seaside, call and answer provide that functionality, among other things. Clicking the ‘Done’ button that calls ‘self answer’ returns control back up to whatever called it. So if you found the store item on page 3 out of 5 in a batched list, you’re sent back to page 3 out of 5. If you got there through a list of links from a search result, you’re taken back to those search results. It feels natural. The developer doesn’t have to maintain information about the calling page to return back to it.

Again, looking at my own big and old Zope 2 based e-commerce application, I have a few ‘handlers’ which are thin scripts over the business logic implemented in core components. Many handlers have paths passed in about which page to report errors to, and which page to go to on success. This allows them to be used from multiple pages, within the structure of the good old scripts’n’templates system (but the scripts are really thin layers to underlying code that does validation, processing, business logic, etc). Man, wouldn’t it be nice to have that be more automatic instead of some hidden fields like ‘errorto’, ‘returnto’ with paths?

All in all, what I’ve really liked about playing around briefly with Seaside is the feeling of liberation from not only the request/response cycle, but also from the Template model. I’m suddenly very tired of templates as string insertion languages being the ‘view’ (coff!). I don’t mind using them for the broadest of layouts. But whether it’s template attribute style languages or string insertion, I’m just suddenly tired of seeing and maintaining things like this:

  <ul>
    <li tal:repeat="item view/listItems">
      <a tal:attributes="href string:${context/absolute_url}/${item/name}" 
        tal:content="item/description">Description</a>
    </li>
  </ul>

  <ul>
    <% for item in view.listItems %>
      <li><a href="./<%= item.name %>"><%= item.description %></a></li>
    <% end %>
  </ul>

My problem with them is not so much the language itself, but the casual callous way important bits like URLs and more are constructed. I’ve generally been a big fan of Zope’s TAL (Template Attribute Language, used in the first example), but sometimes it’s too hard to do simple things. Even more frustrating, lately, has been the lack of good helper functions for use in Python expressions in those templates. Zope 2 has “ZTUtils” which has a small set of useful utility and objects. But if those are in Zope 3, I haven’t found them.

I’ve generally been against the PHP style of ‘templates’ where it all becomes super muddy (how many PHP ‘apps’ do you see with print/echo statements in the code that then have HTML inside of a string?). Most of the Python PHP/ASP/JSP style systems I’ve seen have been especially bad, primarily because of Python’s block structure and lack of ‘end’ statements or closing braces inside the language, so doing the ‘for’ loop like above has varied between different implementations. It’s just not what Python is built for, at all.

My experiences with Ruby on Rails has been better. Ruby has the ‘end’ statement. And it really is a nice concise language that doesn’t look all that bad in Erb templates. But what makes Ruby on Rails especially nice is the large collection of helpers. The helpers are a big collection of functions and objects (much bigger than ZTUtils) that makes it pretty nice to work in the template. So instead of the tag-within-a-tag style shown above to generate the ‘href’ part of the link, the common idiom in rails is to use this:

  <%= link_to article.title, :action => 'show', :id => article %>
  <%= link_to 'Destroy', { :action => 'destroy', :id => article }, :confirm => 'Are you sure?' %>

This brings it closer to one of the things I like about the Seaside experience: you’re not writing into a string to make a link, you’re linking to objects and methods directly. Not only that, you’re using other options (like shown in the second example above) to do things like common Javascript events, or to do tricks to make a regular hyperlink perform an HTTP POST request. Ultimately, they make the development experience feel more like actual application programming. For the record, there’s a fairly direct Python implementation of many of Rails helpers available too.

Some of the things I’ve done for my Zope 3 development lately has been to take ideas from these systems. I’ve started building my own helpers system, implemented as a Zope adapter for the Request. Since the Request has ‘skin’ information in it, it’s possible to extend the helper with specialized functions for a particular skin, knowing that it might have certain javascript libraries or DOM identifiers / CSS stylings available. I also have a simple HTML builder system, inspired by Nevow’s Stan, but without the more advanced features like mutation, macros, or.. well, pretty much anything advanced. Applying a python getitem trick and making an object that generates any kind of Tag object by name, I can spit out pretty much anything.

With Zope 3.2 and the introduction of viewlets and content providers, I’ve been focusing on building smaller and smaller view components. I’ve stopped thinking of pages in the standard macro + content or header + content + footer sense and have gotten back into thinking of a page as a collection of connected yet independent view objects.

  html = api.htmlHelperFor(self)
  ul = T.ul(class_='menu')
  for item in self.listItems():
      ul << T.li[ html.linkTextTo(item.name, item) ]
  return unicode(ul)

  # Or, for more fun
  html = api.htmlHelperFor(self)
  return unicode(T.ul(class_='menu')[(
      T.li[html.linkTextTo(item.name, html.viewURL(item, 'delete'), confirm='Are you sure?', post=True)]
      for item in self.listItems()
  )])

Added to that, zope.formlib delivers form components with ‘Actions’ bound intelligently to methods. zope.formlib handles dispatching to the proper action instead of just having Zope’s publisher go to a method / view directly to handle forms:

    @form.action(u"Import Articles", condition=form.haveInputFields,
                 failure='handleFailure')
    def importArticles(self, action, data):
        # This gets called when the 'import articles' button is clicked, and
        # only if validation succeeds. The button only gets rendered if there
        # are input fields - a form can be generated to display data on a field
        # by field basis according to security, which can set up 'view' 
        # situations for unprivileged users - and you wouldn't want to have
        # an input button when there are no input fields, right?
        # 'data' has all of the validated form input, and can be easily
        # applied like this:
        form.applyChanges(self.context, self.form_fields, data)

    def handleFailure(self, action, data, errors):
        # Optionally, individual actions can have other methods associated
        # with them to handle action-specific features for validation,
        # validation failure, and so on.

Being actual objects, they can be referred to in the view or view’s template and filtered out if unavailable:

  controls = T.div(class_="controls")
  for action in self.actions:
      if not action.available():
          continue
      controls << action.render()
      controls << '&nbsp;&nbsp;'

I like having tools like this available. It really makes the application feel a lot stronger. ‘formlib’ is pretty simple, conceptually, while providing a lot of flexibility and features and strength on the ‘web application as actual application’ front. That’s the nice thing with Seaside and Rails’ controller/view layer and its helpers too: a lot of tools, some simple concepts, a lot of power.

Head Go Boom

2006-03-08T19:02:00.000-07:00

This was the opening line in one of those oil scam spams: Let me crave your indulgence to introduce myself to you.

My brain is now stuck in a locked groove.

Is this thing back on?

2006-03-08T13:28:00.000-07:00

Somehow, Griddle Noise got selected as a spam blog or offensive blog or something along those lines. I'm trying to decide whether I should don the dynamite hat of paranoia or not. It could be fun: add to my crankiness a fear of everyone. "They know I'm right, dammit, and they're trying to shut me up?" "Who?" "THEM!" "Who?" "Them!"

If it wouldn't be so mind-crushingly depressing (and wasn't so out of budget), I think I could use a vacation. Remember that week I spent in a shot glass in the bay? Me neither.

I actually do have a small chance coming up to get away, but it's not all stress free like I need. It's sortof work. I'll be performing at No Fun Fest 2006, March 17th, down in the basement, around 8-ish I think, somewhere in Brooklyn where the trains don't go.

ls4+2. Hearts and springs are stupid things.