27.9.05. Easy Complex Content, nth try

Throughout the years, I've been fascinated by compound document technology. I still pine for OpenDoc. Most of my work has been on the web, and every few jobs I seem to come into doing content management and every time I do, it's seldom as simple as "entering some text on a page".

I have proposals, diagrams, and outlines going back to my previous major employer. But most of my memory of getting to really deal with the problem stems from my time when I came back to Utah. I continued to use the Zope platforms through various stages of minor and self employment before landing in my current situation (where Zope is still used). Three generations of working with building compound documents come to mind.

Generation 1 - CMF Based, OpenDoc Influenced, Dynamic Structure
We had a customer that provided us with the basic structural layout of the site that they needed to manage. They needed to be able to add new pages, and those pages needed to include images, lists, etc, as well as text. I decided this was a great time to try my compound document ideas. Taking a page from OpenDoc, I decided that a document was a very basic element that was just composed of parts. In fact, the Document basically held a 'root container' part. Container parts held other parts, such as text, images, and so on. Parts could, theoretically, be quite complex. Container parts would manage sub-part creation, access, and ordering. In implementation, I believe they derived from Zope's ObjectManager base class(es). For this customer, they could create a new document and then just add parts to it. They could only edit one part at a time, but its editor would display in-place with the rest of the document. Parts could also be re-ordered. Images could be dropped in between paragraphs and told to flow on the right, left, or center. The content editor did not have to type in any code to refer to images. Navigation to other documents was also managed by another drop-in part. Most folders have index pages that contain a navigation part (listing sibling and one-folder-down content) and a text part to give a folder table of contents some meaning.

This was a pretty good system. Unfortunately, it was developed at a time when the CMF (Content Management Framework) had stalled out, it seemed, with releases. It's since picked up. But we had basically a munged version on our hands.

We were able to take advantage of much of what the CMF offered - indexing, workflow, user management, and a nice separation of UI code from content (which can sometimes be tricky with Zope 2 development). I still felt very frustrated with the experience though. It felt like I had to do too much work to get the CMF to get "out of the way" where I needed it to in order to satisfy my customer's needs. While the CMF paved the way for Zope 3 by providing some much needed new concepts to Zope development - replaceable components, service/policy objects that were easy to look up and use while also being configurable locally, the concept of "skins" to build layered and swappable user interfaces, the ability for many objects to participate on behalf of a basic content object so that a document could just be a document - it always felt quite heavy to me. Starting out with it from 'scratch' without the default content management application implementation was overwhelming. Hiding much of the default implementation was tedious. And to me - it's actually quite a nice system. It was often just too heavy for our customers budgets or our own time constraints.

But that system that I built - which was for a non profit literacy group - is still in use today and has more content than I ever envisioned. My own struggles with getting my implementation going put aside, it's not a bad system and I'm glad that it still sees use long after most of those involved with the launch of the project have since moved on. Later, when other small companies and organizations came to us with need for simple content management, I felt that a different approach was needed.

Generation 2 - Pure Zope, 'CMF Lite' in a way, still OpenDoc based, fixed structure
For the second generation it was decided that we needed something that was more directly under our control and understanding. It didn't need to solve all problems at first, but it should be flexible for growth. I had recently gotten a lot of benefit of implementing and deploying D4 (Dynamic, Declarative, Data-Driven) architectures on Zope, whereby model designs were used to dynamically build user interfaces, provide validation, and so on (things that are increasingly taken for granted today, but seemed to take a long time to come around to). Most of our D4 solutions at that point were against relational databases, with one application tied to LDAP. I wanted to take what we had learned and applying to these content documents. We had customers coming to us now that didn't have such free form content, but instead had pages that held more common structure. I wanted it to be easy to define this content and store it in the ZODB while having my code avoid direct use of the Zope 2 framework if possible. The new solution consisted of:

Parts

Still the chunks that documents were made of - parts were separate but complete small content items. A part could be text, an image, a headline, or any kind of structured record (contact information, links and descriptions about last years festival, etc).

Policies

There were policies for folders and for documents. This is where a lot of the pluggable and extensible architecture came from. Folder policies would set things like "what is the default document to show for this folder?" or "what kind of documents may be added here, and how many?". They also provided APIs for listing and organizing content. An events folder might plug in an event listing policy that would sort events on their starting time. Like CMF Tools, policies had interfaces and common names, and you could replace them at will. Unlike CMF Tools, folder policies were kept in a special containing object on folders. There were no global site settings (although I've been wanting to add this feature). Documents got their policies out of a document policy manager.

Document Schema

A document schema basically put the parts together that would make up a particular document type. A 'contact information' page might have a text part, a picture part, and an address part. The schema was just another policy on the document type. The schema policy was also responsible for getting form data out, validating it, and saving it.

Simple Events

The individual policies for a folder or document could respond to events that were passed on from the containing policy manager. The Schema policy would respond to a 'document added' event and populate the document with empty parts based on the schema definition.

Skins, and composable interfaces

Another common policy was the 'views policy', which said which template/view objects to use. These objects were given common names, like 'edit', and could be configured differently for different skins. The skins tool we used was written by Chris McDonough and was basically lifted directly out of the CMF. Skins are built into the CMF and Zope 3. A skin is composed of layers. View objects are assigned to these different layers, and basically the view object is looked up in each layer until it is found. Thus you can have both an 'admin' skin and a 'public' skin. In each case, objects are traversed to directly (this is Zope after all). But how they'll be rendered and controlled depends on the skin that's placed on top of them. It's not a great explanation, I know, but it's a pretty significant concept.

Basically, what I had managed to do was mimic a lot of functionality that I liked about Zope 3 (which was still deep in pre-release development at that time). It's a successful system - I was even able to integrate it into our e-commerce solution which is a very different styled application. It even reuses a fair amount of code from the first generation, since many of the internals (documents and parts) were CMF agnostic.

The big difference between the second generation and the first was the move away from free-form documents. The system could allow them with some other policies plugged in. But the advantage of the rigid documents is that we could design certain pages more easily because we knew exactly what slots were going to be on there, and what slots might be on there. The customer still has flexibility with the content that they can enter, but the pages are crisper than they might be if all we could to to render it is loop over the parts and tell them to draw themselves.

Generation 3 - Zope 3 based. No OpenDoc heritage. Lightweight, in less time
The main reason that I'm writing this entry is that I didn't realize until today that I had basically redone much of the work from Generation 2 in a customer application we're developing in Zope 3. The concept of parts are gone. Generation 2 schema was defined through the web. Generation 3 uses Zope 3 concepts. Instead of parts, Fields are used - even for complex objects. A page is defined through its Interface, a very strong Zope 3 concept. And that interface's schema is then used to build edit forms and has provided other exciting benefits as well. For example, I can take any of my content objects and find all of their fields that have text in them that can be used for indexing. The field in question might be simple - just a line or block of text. Or it might be complex - like a table with a title, a caption, and rows of cells. In fact - it's this table field that I wrote that made me realize that I had already made a more complicated 'part' than I had ever made for either of the previous generations. And it wasn't until this evening as I was showing progress to my boss that I realized that this set of work - much of it done just in the last three work days - pretty much blows away the 2nd generation.

Zope 3 delivers a lot of functionality that I can then build my own application stack on top of. It's very rich, like the CMF. But at the same time, it doesn't feel like it gets in the way nearly as much. It's still a rich and complex system - you can't really write a "20 minute wiki demo and video from scratch" on top of it without also having to explain the default user interface or skin configuration or URL namespaces. But at the same time, many of these concepts are (relatively) easy to deploy when setting up a new application. This one that I'm working on I've been at for less than a week now. And it's pretty much caught up to our first 'generation 2' site which took a few weeks, mostly because the Generation 2 compound document system was being written along side with it. This 3rd generation implementation is still missing some key elements that I'd like to have implemented, but I don't think they'll be that big of a concern. And the UI is much more impressive than anything we've done in the past. Thanks to MochiKit and much of the work done for the customer prior to this, we have a nice dynamic / AJAX based UI with sortable folder content listings (in javascript). Drag and drop content reordering. Inline renaming and retitling. In the past couple of days I've written a complex Table field for Zope 3 and gave it an interface that allows for dynamic Javascript editing (adding / removing rows - the column definitions are fixed). In less than an hour today I wrote a Textile field (basically a plain descendent of a text field), whose widget uses AJAX to allow for in page previewing of the Textile output.

And all of this is within an architecture that I understand. Granted - there are a lot of Zope 3 concepts that one has to learn, and there are many that I haven't learned yet. But even my own "Generation 2" system I had trouble understanding in comparison to what we've done on Zope 3. The deep dark caverns of the "Generation 2" schema system - which is quite powerful - are scary places for even me, the author, to go.

What impresses me most about Zope 3 is how easy it is to build ones own layer of toolkits / frameworks on top if it all. A common stack we have, which came together within the first week or two of using Zope 3, is this:

• Customer Specific - skins, components, utilities specific to this customers needs. May build on the next layer.
• Similar Customers - content and utility components that are common across a certain set of customers, which is the situation we're in right now with this first batch of Zope 3 work.
• Our CMS / Framework - Provides the management UI, common components, configurations, etc, that are common across many sites that we expect to build.
• zope.app - The Zope 3 application framework. Provides many of the basic components for a full Zope 3 application - containers, utilities, apis, etc. A pretty beefy layer.
• Core Zope 3 systems (zope.interface, zope.schema, zope.component, persistence, transaction, etc). These are the basic layers underneath Zope. One could, in theory, use many of these outside of 'zope.app' to build a different application server that did not depend on or provide much of what zope.app requires. There is a 'bobo' branch, for example, which builds on just a few of these core things. Using the zope.bobo, one could in theory write a 20 minute Wiki or 20 minute todo-list in a manner similar to what Rails or TurboGears shows off.

Along with all of this, there are other python libraries, including internal ones, and Zope 3 based libraries like Hurry, used as needed.

What surprises me most of all is how quickly this latest solution came together, almost without my seeing it. Nice.

26.9.05. Apple.com Through The Years

A cool photoset on flickr: "Apple's Front Pages Along The Years". Going back to 1996, it includes a couple of captures from the Amelio years. I remember that site. Apple at that time was quite large and sprawling and many sites were off on their own subdomain with their own look and feel. There was a trove of information out there on projects like the Dylan Programming Language as well as OpenDoc, Cyberdog, and a browser plug-in called "HotSauce" which could fly through MCF - Meta Content Format - data in 3D. MCF would go on to eventually become RDF.

The photoset includes NeXT's page at the time of the Apple-NeXT merger, which was a very exciting day for me. That merger literally set fire to the company. All of the excess was trimmed, and eventually Apple started the "Think Different" campaign and launched a new web site in 1997. By the time the iMac was announced (with the campaign of "Pro. Go. Whoa."), Apple's home page has stayed pretty much the same in the time since then. There's the big picture, the "hot news" scroller, and now four cells underneath (there used to be just three). It's interesting to see such consistency on a major corporate web site.

22.9.05. Some Favorite Python "Anti-Pitfalls"

I saw a link to this today - a list of Python anti-pitfalls, written back in 2003 by Richard Jones. There are a couple that I'd like to add boisterous comment to.

11. No namespace pollution (unless you really want to)
Unless you explicitly do from foo import *, all names are contained within modules / classes / objects. I love Python packages and modules. Every time I work with another language, like my rare excursions into Perl or Ruby, or my recent working with Javascript, I'm reminded of why I love this feature.

from zope.interface import Interface
import zope.schema

class IFoo(Interface):
    title = zope.schema.TextLine(...)

A very simple example. But in Ruby, for example, require 'xml/parser' is then used in code as XML::Parser - a name that is similar but different to what was in the require statement. In Python, that might look like import xml.parser and then be used as xml.parser.Parser. The last time I used Perl, I was always staring at names that I needed to use and wondering exactly where that name came from. I use neither language enough to know the difference between 'require' and 'use' (PHP suffers from this as well). Python's Modula influence is perhaps one of its strongest features. It makes large frameworks like Zope 3 and Twisted actually quite easy to understand and follow.

So with Python, every name that I see in code I can easily trace, unless the code's author uses multiple 'from foo import *' style statements or does some namespace manipulation magic. I try to avoid those these days, wherever possible.

12. The python shell (interactive interpreter)
I learn best by trial and exploration, and the excellent Python interactive interpreter is probably the number one reason that I'm a Python programmer at all. I was raised on computers like the Commodore 64 which pretty much had the BASIC language/environment loaded at all times, so I could immediately try out small or large chunks of code. Python was the first language since those early days that brought me the same joy. On a daily basis, I still drop into the interpreter to introspect objects or just to try out how something might work. For learning something new - I recently used the interpreter to figure out how to dynamically build a zip file of CSV files while never actually building a file on the file system. I was able to explore using the csv, StringIO, and zipfile modules together in the interpreter before integrating the code into my project.

Recent versions of Zope (both Zope 2 and Zope 3) make it easy to load up the Python interpreter with the ZODB and Zope system loaded. I was able to use this the other day to dig deep into some indexes to figure out why some search results weren't behaving as I expected. It turned out that I was querying the index incorrectly, but it was a behavior that didn't show itself based on the first set of data that was loaded in our ZODB database.

This interactive interpreter is available at all times. It is Python. It's one of the many features that makes working with Python without all of the weight of an IDE possible.

13. PDB
PDB is the Python debugger. This is the other feature that makes working with Python without all of the weight of an IDE possible. Sometimes, I have problems in Zope that can't be figured out with some other debugging means (logging, etc). When running Zope so that it doesn't detach from the shell, having the following statement in the Python code: import pdb; pdb.set_trace() basically puts a breakpoint in the code and when those two statements are executed (the pdb module is imported and the set_trace() function is called), Python drops to a 'pdb' shell inside of the interpreter. From here, I can step through code, inspect variables, go up and back down the call stack, and even import and use other code to help me in my inspection. When the problem is solved, deleting the 'import pdb; pdb.set_trace()' line is all that's needed to remove the breakpoint. Again - this is doable without an IDE and is available anywhere - even on the most stripped down server configuration.

There are many other anti-pitfalls in Python in Richard's list. These three (which just happened to be sequential) are the three that I'm most grateful for and are all elements of Python that I've used happily for almost ten years now, even while other new (and welcome) features and modules have come into the language since those Python 1.3 days.

14.9.05. Assorted Thoughts on Office 12, Vista, and UI Trends

Microsoft has released initial information and screen shots about Office 12. It is a complete revamp of the user interface, at least in the primary applications Word, Excel, PowerPoint, and Access. The "authoring" components of Outlook (message composition) have also been affected. So now it's time to weigh in with my own thoughts about the whole affair.

Visually, many of the applications remind me of Apple's professional applications like Final Cut Pro or even the iLife apps like iMovie. Many of the new office applications feature a brushed metal interface, similar to those seen in some Mac applications in recent years. But the similarity doesn't end there - Apple's metallic video, photo, and audio tools all do a lot of functionality within the window with sliding drawers, tabbed sections of the interface, contextual toolbars within certain frames, and so on. This is something that seems to work well in these kinds of applications. But Microsoft seems to be experimenting with taking that sort of experience to the office suite. I'm not saying that they're copying Apple (although the brushed metal AND aqua-type tabs are rather interesting). I'm saying that it's a unique time in user interface design. Office applications have been pretty much the same for the past decade, especially since Microsoft Office started its total domination with Office 97. Since then, it seems that the big changes in Office have been a large proliferation of "editions" which basically bundled different applications on top of the standard set. There have been some UI improvements. I believe it was Mac Office 98 or 2001 that first introduced the context sensitive formatting palette. This was a floating toolbar with expandable / collapsable regions that would add features based on what you were working on. This started showing up, more powerfully, in the windows version of Office, starting with Office XP, in the form of 'Task Panes'. To this point, task panes seem to be about the only major UI innovation in Office.

I should qualify all this by saying that I am not an Office user, nor a Windows user. I'm a Mac user and internet developer. Word processing and spreadsheets rarely cross my path. But I'm always interested in what's going on with major applications and user interface trends and I think that we're at an interesting time right now. Apple is making bold design decisions, some of which are hotly contested, but some of which have been wildly successful and I actually enjoy most of my days in Mac OS X. And I have to admit that Gnome is really doing an incredible job, I think, with bringing good and legitimate usability to *nix. I've even seen some Mac people joke about finding better consistency in Gnome applications than within Apple's own Mac applications. And now with the next generation of Windows coming up (Windows 2000, XP, and 2003 are all considered to be Windows NT 5.0, 5.1, and 5.2, respectively; 'Vista' is considered 6.0), Microsoft is showcasing some interesting UI design in their first beta. Looking at some screenshots of picture and video management, which is integrated into the Windows Explorer (Desktop, Finder, whatever you want to call it), I noticed that there is no menu bar. In screen shots of IE 7 Beta in Vista (Beta), I noticed that the menu and toolbar are relegated to being pushed deep within the window. The window itself is a unified titlebar and toolbar look, which has also been popular on Mac applications, and is most obvious in iTunes 5. In the IE 7 betas (which could change), the top is dominated by back/forward buttons, the location field, and a search box similar to that seen in Safari and Firefox. Beneath that are tabs for tabbed browsing, and then within that is the menu bar and a toolbar providing other IE functions (home page, print, etc). That interface is too strange, I think. It's not a problem for Macs to have the big toolbar - our menu bar is always at the top of the screen, not having to fit in a window. But on Windows, the menu bar is starting to have a new life, it seems. At least in some core system functions, and in Office 12.

Office 12 seems to do away with all menus, except for 'File'. All other functions provided by menus are now in major tabs across the top of the app (which replace top level menus, it seems), which provide big toolbars for that area of functionality.

Does it work? I don't know. I haven't actually used it and it's not likely that I ever will. At times, it looks like early WYSIWYG tools that ran full screen in DOS or on whatever machine they were on at the time that didn't have a real windowing environment. The tools seem kindof big and clumsy and put there because "where else are they going to go?" and they end up taking half the screen... But that's just what it reminds me of at times... something out of the late eighties / early nineties. At the same time, it reminds me of Apple's professional video applications. In iMovie, Final Cut, iPhoto, etc, you have one main area of content that is focuses on, and all of these other tools surrounding it that tend to stay out of the way of the content area. That's not bad. Even as I'm writing this entry in MarsEdit on the Mac, my focus remains on this window where I'm typing and I go to the toolbar on rare occasion and to the menus even less. Granted, writing this entry is much different than what Office is geared for, but I admit that I never feel this comfortable when I have to write something in Word. Word just doesn't stay out of my way when I need it to stay out of my way, and I'm certainly not a powerful enough user to deal with it effectively. So it seems that keeping the authoring area in better focus is a bigger design goal in the Office 12 UI. I like that, at least in theory.

There's a big problem with Office. The brand is so powerful that, like Photoshop, many people are trying to use these very powerful applications for very simple tasks. Many people probably end up with a copy of Office even when something like Works or a third party solution might do. There are probably people who need all of the power of Word and Excel, but I wonder what percentage of Office users they represent? And the open source contenders? OpenOffice.org seems to just be trying to be a free version of Office, which I'm not going to argue is a good or bad thing, just that it's probably even more developer driven and less user driven than even Office has been. Compare this screen shot OpenOffice.org Writer with this shot of Word 12, both of which are unreleased, I should add. Comparing the two, I'd take Word 12. The shot of Writer reminds me of the biggest beef I have with all versions of Office since at least Office 95 - way way way way way too many toolbars full of buttons that I can't differentiate from one another, let alone discern at fairly rapid glance the meaning of all of those icons. If nothing else, I'm happy to see Office finally pull away from such horrid toolbars and providing more text options. I'm also glad to see Word pull away from it's endless litany of menus, menus that I found to be too confusing back in the early 90's with Word 5.1. 5.1!!!!!! Why did it take 12 years and six versions to start changing that? Granted - Microsoft has tried some other approaches in the intervening years, such as 'smart menus' which tried to hide less used items, contextual 'smart tags', task panes (which I actually enjoyed using in my evaluation of OneNote 2003).

Anyways - Office, its clones, and other similar suites that may still exist, are so ubiquitous and overpowered that it's no wonder that there are so many grievances against them by many users. Microsoft has had a hard time selling new Office versions - not an abysmal time, but they have a hard time convincing people to upgrade. Why upgrade to something that's probably only going to be more complex? Perhaps Office 12 is an attempt to combat that perception. But here's my suggestion - learn from Apple. Office is a professional application. Market and sell it as such. Really revive Works as something like "Office Express", or make some other still-powerful-but-not-overpowered version of Office as such. Get away from all of those different editions and their confusing combination of programs. Stop selling individual programs for so much when the office suite with so many more options is not much more - either sell them for cheap, and/or provide build to order licensing so someone can get Word and Outlook for cheap. If I had Windows, I would never go near Office because the different 'editions' are confusing and often too pricey for a casual user. If you (Microsoft) cannot provide to people looking for simpler and cheaper solutions, others will step in. The Office 12 UI changes look interesting, but will they be compelling? Will the cost and licensing be worth it? Will anyone else step up to provide alternatives for more general usage? Will users notice?

If the planned Windows Vista editions (of which there are 7) are any indication, Office 12 will probably fall victim to this same thinking and users will be presented with huge product comparison grids and confusing pricing and licensing schemes. If OpenOffice.org has their stuff together, this is a strong area they should compete in. "Want a word processor? It's free. Want the word processor and the database and email? Same price". Now if only someone like OpenOffice.org, or Gnome, could start taking a look at real usability issues. Because you know - I really do like what I'm seeing from Word 12 screen shots and videos. Fortunately, on the Mac there's Pages from Apple and Nisus's Writer Express as cheap, native, and usable alternatives to Word should I ever really need them.

6.9.05. More Zope 3 Experiences

In response to my post about my recent real-world Zope 3 experiences, Martijn Faassen posts his. He covers notes on maturity. Zope 3.1 is quite mature, but there are a couple of small problem areas which we also bumped our heads against - namely the built in form machinery. The default machinery is not bad, but it does seem to be primarily targeted at basic object add/edit forms and introspection. zope.formlib, an extension developed by Zope Corporation and released to the public. We did not use zope.formlib, as it requires Python 2.4 and I thought we were having issues with Zope 3.1 and Python 2.4 on our development server. I think I was wrong about that, but I was under pressure to get this project underway so I stuck with the safe solution. I was actually trying to get zc.catalog running, and may have just not configured things properly. Oh well. Now I should have time to evaluate future plans.

I wasn't that involved with some of the heavy form development, but a co-worker was, and I think he would have benefited from zope.formlib.

A bad thing, or annoyance, about Zope 3: at this time, there doesn't seem to be any ZODB import/export functionality like there is in Zope 2. Some people who have written custom applications like bug trackers have written custom XML import/export facilities for their application, but that doesn't help for content management situations. fssync, a tool for moving Zope 3 objects between file system and ZODB, seems to be missing from the distribution. I'm not sure what to do here when content has to move between servers. I'm guessing a full copy of the ZODB's storage file is what's in order?

A good thing, though, about Zope 3: generations. A good thing with an object database behind a language like Python is that your persistent data can be pretty free-form. But as applications grow, data structures change. For our Zope 2 based RDBMS applications, we wrote a tool called 'updater' which contained methods that could be selectively executed to update RDMBS data and structure (and also could do some ZODB tweaks as well). For ZODB applications, with Zope 1 and 2 you'd often provide some persistent protocol tweaks, such as __getstate__ and __setstate__ methods that could rewrite the data on the way to/from the ZODB, and in that way they could reflect structural changes. Or you could write hooks in your code to check the existence of a new attribute, or its structure, and respond accordingly. But it wasn't the most elegant solution - how long should those hooks stay around? How are they ultimately impacting the speed of data storage and retrieval? Enter generations. With generations, you create a schema manager which basically specifies a unique key for your application or package (usually a dotted Python name, like 'zope.app.generations' or 'foo.bar.contentking'), the minimum supported 'generation' of your package, and the latest 'generation'. You also provide evolution modules with the name of their generation, like 'evolve1.py' and 'evolve8.py'. When Zope 3 starts up, it looks through these generation schema managers and will run the evolution code in order to catch up to the minimum supported generation. It provides utilities for the evolution code to locate objects matching a certain criteria or implementing a certain interface. This solves many problems that afflict the __getstate__/__setstate__ solutions, since the evolution scripts stay around forever (making it possible to upgrade really old deployments) while also executing only once. Application code shouldn't have to be performing checks like:

if item.preferredFormat.startswith('Y'):
    # Waaa, old format. Rewrite this to the new way
    ...

And with Zope 3's generations support, it doesn't have to.

...

Oh yeah, Martijn also mentions his extended catalog queries experience in Zope 3 - code I'd love to see. Querying a catalog is not all that obvious right now. but I should say that it does work quite nice for full text indexing.

5.9.05. Prototype.js and Mochikit

I've mentioned here before that I was impressed with the prototype.js library. It's a few months later and I now have a real Zope 3 project under my belt that has utilized the prototype.js library and the Scriptaculous that builds on top it. There were some great things that I was able to do with these two libraries - Drag and drop reordering? Just a couple of lines of Javascript and just a couple of lines of code in a browser view in Zope. Nicely, the data returned from the Scriptaculous reorderable functions is exactly what IOrderedContent.updateContent wanted in Zope. (Note - I did have to replace a Scriptaculous 'serialize' function, which encoded the list URI in a Rails friendly way to do so in a Zope friendly way).

So there have been some things to like about Prototype.js and friends so far:

1. Visual effects for effective communication (or gross overuse) doable with just a couple of lines of code.
2. Ajax Requests are pretty easy.
3. Referring to elements with $('someobj') is much easier to type and remember than document.getElementById('someobj'), especially since $(this.form) is nice as well. Passing DOM elements around by their ID or by the DOM object itself is fine.

Finally, it seems that the religion of Prototype.js is that most of the heavy lifting logic is still expected to be done on the server, requiring very little Javascript code on the client to accomplish some really nice and useful (or annoying) effects and communications. I don't know if that was really the design goal, but that's the feeling. It's not a bad feeling, per se, and the way that prototype.js and scriptaculous have been merged into Ruby on Rails has probably helped blur the lines between server side display logic and client side display logic even further.

But Prototype.js also seems to do some monkeying with the core Javascript object system. At least, that's the complaint made by the developers of Mochikit. There are some things that they claim it breaks, such as for(var i in obj), and I believe I've now seen this breakage in action. Using Prototype.js and friends seems to be OK, as long as you stay in their universe. MochiKit takes another stance. It provides more functional programming techniques, whereas Prototype.js provides a more object-oriented experience. MochiKit is inspired by Python (and its main developers are seasoned Python and Objective C/Cocoa developers), and provides Python style namespaces. While many of the exported functions from MochiKit are available from their direct name, there's always a full name to access them, which helps avoid name conflicts.

There are some other differences in approach as well. Prototype.js's documentation lies basically in its code and whatever examples one might be able to dig up. Strangely enough, I do find most Prototype.js and Scriptaculous code readable enough for me to figure out what to use and the basics of how to use it. The number of concepts seem quite simple, really, and that which I haven't needed to use I've been able to ignore. Mochikit is rather well documented, includes plenty of doc strings, and is also backed by unit tests. MochiKit is a much larger system and covers a few different areas of coverage. It includes not only plenty of DOM manipulation, but also iteration support, including many of the functions seen now in Python's itertools module. MochiKit covers Asynchronous programming, inspired by the event driven network programming framework for Python, Twisted - specifically its Deffered objects (which I admit make my head explode). The MochiKit.Async package covers XMLHttpRequest communication, part of the core AJAX experience.

A quick comparison of a simple Ajax setup in each. This is from a function that I wrote that calls a URL, but before making the call it shows an 'updatemsg' element. When the call is successful, the message fades out. The first half is the MochiKit.Async way. The second half, commented out, is the Prototype.js way.

foo.bar.requestWithUpdateMessage = function(url, queryString) {
  var req = MochiKit.Async.getXMLHttpRequest();
  req.open("POST", url);
  
  Element.show('updatemsg');
  // d is a MochiKit.Async.Deferred object.
  var d = MochiKit.Async.sendXMLHttpRequest(req, queryString);
  d.addBoth(function(transport) {
    Effect.Fade('updatemsg', {duration: 1.2});
  });

  /*
  new Ajax.Request(url, {
    parameters: queryString,
    onLoading: function(transport) {
      Element.show('updatemsg');
      },
    onComplete: function(transport) {
      Effect.Fade('updatemsg', {duration: 1.2});
      }
    });
    */
}

They're both pretty good ways of doing things - this is a lot easier than manually grappling with finding the correct XMLHttpRequest object in the browser and setting up the callbacks yourself. But you can see the difference in programming style.

I've also done some work with MochiKit.Base, which boasts some wonderful tools familiar to Python programmers, as well as some nice partial and binding support. Using MochiKit.Base.bind, I was able to write some AJAX client functions and turn them into methods bound to an instance, effectively creating a client stub that mirrored server side functionality. Last week, I had this code boiled down to a collection of three-line methods that were all the same except for their parameters and the target URL endpoint. Today, I now have those replaced with single line declarations:

ContentsClient = function(baseUrl) {
  this.baseURL = baseUrl;

  this.editTitleFragment = bind(updateClientWithMessage, this, 'editTitleFragment', ['name']);
  this.cancelTitle = bind(updateClientWithMessage, this, 'cancelTitle', ['name']);
  this.updateTitle = bind(updateClientWithMessage, this, 'updateTitle', ['name','title']);
}

I'm envisioning a day when I can write a server side view that queries a Zope Interface object for method signatures that can generate a JSON description on the fly to build one of these Client objects... But I don't have need for that yet.

Finally, there's MochiKit.Logging, which amongst other things gives very nice logging support - "MochiKit.Logging - we're all tired of alert()." Using log('bla bla bla') plus a simple bookmarklet makes Javascript development much nicer, especially in an increasingly event driven browser client world.

Really finally - I was able to take the MochiKit Sortable Tables demo and apply it to my CMS interface with ease - even modifying it some so that not all columns instantly become sortable, and adding integer and float support to the sort functionality. Table sorting is not a core MochiKit piece of functionality, but is instead just shown in a demo to show how the MochiKit parts can work together.

Great work!

4.9.05. Major Zope 3 Client Project Finished!

For the past few weeks, I've been working on a real Zope 3 application for a customer. Prior to this, I've just done the little never-completed apps that I've mentioned here, but they were enough to get me some general insight into how things worked.

The project that we worked on was essentially a complete rewrite of an old Zope 2 web site for one of our largest and oldest customers. This also came at a time when this customer wanted to re-do their public interface as "Pure CSS", which gave us some fun new challenges and opportunities, such as replacing the old Macromedia Fireworks generated (and then machine augmented) navigation-menu code with essentially a pure-CSS (plus 12 lines of JavaScript for Windows IE) implementation. While all of this was going on, we were also getting told of the possibility of netting sister companies web sites, each sharing a lot of common functionality, but each also having some unique requirements. Early on, I was given the task of presenting a Zope 3 based technology stack to handle the project, and was able to sell it to my coworkers.

So on to the experience. Overall, it's been really good. This customer has always edited the bulk of their content via FTP, and our software would parse out the contents and metadata and index it. In Zope 2, knowledge of FTP and catalog indexing were the responsibility of this "wrapped page" object. In Zope 3, FTP views are separate from HTTP and 'Browser' views. Content indexing is now all managed with events, interfaces, and adapters. Common metadata is handled, by default, via object annotations and an IZopeDublinCore adapter that provides easy access to that data. This means that the core class of the base 'model' object is tiny, and its collaborators are what start coming together to provide the application specific functionality.

This customer also had other managed content that they used a web interface to administer. This included structured data like press releases, event calendars, and so on. It also included some major data management regarding company operations. All of this was in a separate interface in the old application, independent of the rest of the content. With Zope 3 we were able to build a custom content management user interface that could access it all. This was done from the ground up. The CMS needs were extremely basic - access the content and edit it. There was no workflow or anything else required. The user interface to administer the content is done in one skin, while the public sees all of the content via a completely different skin. But all of the object access is the same. This is very different than Zope 2, where just about every object inherited the Zope Management Interface. Now, the ZMI is just another skin.

Another Zope 3 advantage over Zope 2 is that it's fairly easy to provide custom views for exceptions. In Zope 2, there was a template called 'standard_error_message' that you might put in a subfolder to customize, and in that template you might try to guess at what the error was or have template-driven logic to respond to a few different errors. Since this was a major web site overhaul, we needed a custom NotFound view that had logic in it that could query a special tool about redirects, and if that wasn't found, show a pretty page specific to the NotFound error. In Zope 3, this is as easy as providing any other web browser view - this one is just bound to an INotFound interface, which is the interface of a core NotFound exception that is raised by the publishing machinery. Since it's just another view, it can be bound to a skin layer and thus only behave this way on the public view of the site but not on the CMS view.

Lastly, in regards to this particular project, it was really easy to layer everything up into a nice stack. There's the core CMS, which we'll use on other projects. On top of that is a layer dealing with common functionality that this customer and its sister companies will all use some (or all) of. Then on top of that is the customer specific layer, with custom skins, views, content components, utility components, and so on. This was all quite easy to set up and build on. This is all possible in Zope 2 - we have written similar stacks - but it was easier in Zope 3. I think this is because of the separation of concerns. It's much easier to take a common component out of one layer and provide a lot of new application specific functionality for it without subclassing. It's all about adaptation.

How might this compare against other systems? I don't know of any other app servers, besides Twisted based ones, that can provide FTP access to content. Zope has provided this since Zope 2.0. I also don't know of any that provide a concept like 'skins'. Since many app servers, even content focused ones, use an RDBMS for storage, I imagine that they provide 'admin' interfaces as a completely different set of components and database queries. For content management, the skins of Zope 3 (and also Plone/CMF and many other Zope 2 based content management systems) allow objects to be dressed up very differently. In this regard, skinning is not just about putting different colors or backgrounds on things. It's about using whole different sets of presentation components for different uses. It's really quite powerful, and I really like that it's built into Zope 3. Zope 3 continues to build on the strong security that has been an integral part of the Zope story dating back to the days even before Principia. The authentication system - actually, a lot of the security system - seems to be a lot more pluggable than even before. Since Zope has been used in major media and government sites since its inception, and knows that restricting access and safely delegating control are important elements to many institutions, it has this part down.

David Heinemeier Hansson, the primary author of Ruby on Rails, was interviewed recently on the O'Reilly Network. In his interview, he talks about the differing experiences between J2EE and PHP development, and how that influenced RoR:

So I was situated at the intersection between two widely popular approaches to software development. The one represented by PHP was quick-n-dirty and the one represented by Java was slow-n-clean. So combining those influences into the ultimate goal of quick-n-clean did give me some confidence that I at least had a shot at appealing to both camps.

So how do I think Zope 3 measures up against this? Zope 3 is far from the dirty of quick'n'dirty, but it's not quite quick'n'clean like Rails. The full Zope 3 "stack", which includes the Zope 3 app server (zope.app) is a pretty exhaustive stack which provides a lot of great functionality but it can take some time to grasp. That said, I think one could build a base system build on the core zope system (zope.component, zope.interface, zope.schema, etc), the template system (if desired - zope.tal, zope.tales, zope.pagetemplate), and the storage system (again - if desired, any or all of persistent, transaction, BTrees, etc) and use those as the base for a quick'n'clean system while taking advantage of what Zope's component framework has to offer - interfaces, adaptation, etc. That adaptation is wonderful. There are some other Python systems that do a more comprehensive job, but I find them too complex to grasp.

The full Zope 3 system is not quite slow'n'clean either. It is very clean! In fact, the component architecture is simpler and cleaner in Zope 3.1 than it was in Zope 3.0! A lot of the core concepts have been cleaned up. There is XML based configuration involved. But I don't mind it. In fact, I think that it's good for any system of this size to have component registration and configuration be separate from the regular code importing/loading process. For very small web applications, it feels like overkill. But I'm very grateful for it in an application like this. I used to wage wars against ZCML (the configuration language used for Zope 3 component configuration), but I think that its syntax is now quite manageable and it's very well documented. ZCML, and the large structure of the Zope 3 application server, can make some development slower than a Rails style system, but it is still significantly faster than a J2EE system - you get to use Python (of course), and in fact, the Python code you write is much more 'normal' than comparable Zope 2 code. Concepts such as implicit acquisition are effectively gone, API's are clean... It's quite nice.

The metric that I can use to measure quickness and cleanliness is the fact that three people - only two of them developers, and only one of them having any prior experience with Zope 3 - did this project from scratch in about four weeks. Granted, there were some long days and weekend work in there, and a lot of our internal milestones slipped (or were just pranced over). But we did meet our deadline! As we start working on the solutions for the client's sister companies, the cleanliness factor will really show itself. We've come up with not only a custom solution, but a usable CMS web based interface, utilizing AJAX and providing features such as drag and drop reordering and inline renaming. All of that is pretty admirable and even normal for a few weeks worth of hard work. But to have a reusable stack that provides some major functionality in that time frame as well as a complete solution for one customer? That's hot.