28.12.05. A Glance at Zope 3.2 - Content Providers, Viewlets, and formlib

After ensuring some existing code would work under Zope 3.2 (still in beta testing at this point), I've been allowed this week to start work on a small internal (for now) project. I decided to take this opportunity to start looking at some new Zope 3.2 features. And while my experience with any of these is severely limited at this point, I want to mention them because they're all very interesting.

Content Providers, Viewlets, and Viewlet Managers

Two new top level packages in the Zope 3.2 distribution are zope.contentprovider and zope.viewlet. They're related, with the components in zope.viewlet building on zope.contentprovider.

What are these, and why are they important? In short, content providers are a simple construct that can be used to modularize web pages. Zope Page Templates have provided a decent way to modularize pages, using slots and macros to allow writing into multiple page parts more easily than a basic "insert header / body body body / insert footer" style template system. But it doesn't benefit from the flexibility that the component architecture can provide. As I've covered recently, A view in Zope is a multi-adapter, which discriminates against context (the object being viewed) and the request (HTTP request with skin information, FTP requests, and so on). A content provider adds to that, discriminating against the view as well. It's a smaller view used for that bigger view. More common terms for this are things like 'portlet'. There are some pages that you have, I'm sure, that you look at and you can see clear boxes that you are rendering in the common template, in included files, or being forced together a bit clumsily. You might be thinking to yourself "There's a lot of logic involved in rendering those navigation menus. I've got this great component architecture behind me, and I'm pulling that into a page with a 'var' and hoping it's there? Agh. And I don't want that navigation to show up like that in all of my shopping cart views... I need a different set of information over there during checkout..."

It appears to me that this is what content providers (and the viewlets built on top of them) are meant to solve. Based on JSR 168 - Portlet Specification, a content provider must implement only two methods: update() and render(). update() is the chance that the provider has to respond to the request and change state. It's always called first. Then render() is called to, um, render. On its own, it doesn't seem impressive. But its simplicity belies its power. zope.contentprovider doesn't provide an implementation, only some Interface specifications and a TALES namespace for using content provider components in Zope Page Template documents using the registered name. The component architecture will deliver the right component for the page being rendered. One has to look at zope.viewlet for base implementations.

zope.viewlet provides a common content provider in the form of a ViewletManager. The ViewletManager is probably the most common content provider one might call directly in a template. Viewlets are then registered with a ViewletManager, which will filter, sort, update, and render each Viewlet. A viewlet adds a forth discrimination - the viewlet manager. It all still builds on the component architecture - it's still multi-adapters underneath it all. So how might one use it? One example is the good old "side column" part of a page. With viewlets, one might register this as a viewlet manager with an interface name like ISideColumn or IAdditionalInformation. Then, one might provide a small viewlet for ISideColumn for all pages that shows the title, created date, modified date. Let's then say that you install a system that can find related content based on the tags or text of the current object. You could write a new viewlet that lists those pages in the ISideColumn - but only on pages that implement 'ITaggable'. The original template or viewlet manager don't have to be modified to render this, or to have any logic in them to discriminate against 'ITaggable'. The component architecture does the bulk of the work to find viewlets that can be drawn in a particular manager, and a particular viewlet manager implementation may do its own sorting and filtering of those viewlets - the base manager class provided in Zope 3.2, for example, filters out viewlets that the current user can't see.

What does this mean? What are the benefits of content providers / viewlets? I'm only speculating, based on very little personal experience so far, but I'm impressed with the simplicity of the interface. So here's a bulleted list of where I expect to benefit from this, and where I think others could too:

• Easier to specify page parts for current and future development. Build the page out of ViewletManagers, which are backed by Zope Interfaces, and then provide page parts against those. Navigation sections, promotional image displayers, menu items, footer links, sidebars, main areas. You can find these parts and formally specify them, and then provide the viewlets to flow into them.
• More intelligence in small parts. Viewlets may be simple or complex, but basically they can now have the same great (or small) intelligence that full view components have had. You know how it is - the middle left column of the page is expected to provide a search form, a navigation menu listing local pages, and a couple of small banner ads. Now all of those pieces can be more easily coded as separate components, but all in the same manner. This is possible in Zope 3.1, but content providers make this a lot easier.
• From a single code base, provide unique viewlets for the home page, section fronts, and common templates. This is another common little thing - lets say you have to have to add a hidden field to the little search form box used on every page, and the one on the front page needs a pull-down menu to allow discriminating results against different parts of the site. Normally, this little form is a small bit of code that's just managed in the templates. No big deal, there's only three - but they're all slightly different. You can't just copy and paste without paying attention to the unique requirements for each scenario. Putting that search form into a viewlet, you'd have access to the code used in each. Maybe you're using the same in each one. You can now make a specialized viewlet for the home page that adds the drop-down menu, while keeping it all tied together on the search page. You can also ensure that the 'search' sidebar widget does NOT draw on the search results page, or provide a more advanced search sidebar to filter results.
• Make a viewlet manager to handle HEAD links and easily add extra javascript library or CSS loading to pages that need them. I've commonly made page template 'slots' to allow me to load or add page-specific javascript or style sheets to individual templates. Using a viewlet manager for these, you can ensure that 'foolib.js' is there for every IFooListing based page.
• Use Page Templates, use another template language, render HTML directly - it doesn't matter. The interface is so simple, you no longer have to wonder about whether to use __call__ or __str__ or render as the best way to render a little component. 'update()' and 'render()' are all you need to worry about. Yet they can all plug into a larger page with ease, regardless of how that page is rendered. There are niceties for ZPT to make using content providers easy, but one could easily make a big "standard template" without a template language at all - just call and render a few principal ViewletManagers.
• Caching. There hasn't been a formal way of being able to cache fragments of pages. There's decent caching support (with more pluggable) in Zope 3, but no easy way of being able to apply them to parts of a page. With viewlets, it should be easier to put in caching. With more intelligence put into a viewlet, it could manage its own cached data or output. More effectively, one could put this in at the Viewlet Manager level without having to effect individual viewlets. The spec for content providers is simple enough that this could be plugged in a lot more easily and as needed.
• Based on simple standard. It's impressive to see that this is based on JSR 168. When I first heard this, some time ago, I flinched. JSR specifications are from a world that I just don't understand anymore, it seems. Yet this one is nice and simple and effective. It also should quiet some not-invented-here criticisms of Zope 3. At the same time, it has been specified and implemented in a way that fits with Zope 3. When I first looked at content providers and viewlets, I flinched then too. But as I started to use them, I realized that they didn't differ much from what I had already learned about views, widgets, and other multi-adapters. I've made 'views' with the express purpose of being rendered as part of a page, but they haven't followed a common interface. There are many other elements I see now that can fit into this system and ease use.

zope.formlib

The other addition I wanted to cover, briefly, is zope.formlib. Formlib is a general toolkit and API for doing intelligent web form development in Python. It's not a widget library - most of those are provided by the widgets in zope.app.form.browser, but could come from anywhere. It's not a validation framework - that's provided by zope.schema based fields (which work with Zope Interfaces). It does use both of those, and provides a lot of extra features to collect all of the input, apply the validation or collect errors, do invariant based validation ('end date must be later than start date'), dispatch to different responsive actions, and more.

It's a toolkit capable of building powerful base classes and individual uses for forms that are simple (just a few fields) or very complex. There were some form views provided in Zope 3.0 and 3.1 that do similar, but they weren't so easy to use and understand and didn't handle complex forms. Custom field and widget combinations required a lot of custom work, and so did dispatching to multiple actions based on success, failure, and so on. Lastly, the older (non-formlib) techniques seemed to prefer being built via the Zope 3 component configuration language when it really required Python to get the kind of flexibility that form development requires.

In wrap-up, formlib is promising because it shows a preference for using Python and some of the recent features such as decorators over giving everything over to configuration based automation. Content Providers are a simple specification for modularizing web pages into manageable components, with viewlets and viewlet managers providing a reasonable base implementation to work with. The dual layers provide a boundary between a basic spec and small set of tools (the content provider interface, a TALES expression for use in zope page templates) that have broad usage and more specialized providers, built from real-world usage and previous art and examples like JSR 168.

20.12.05. The Zope Component Architecture - One Way To Do It All

My previous post showed how Interfaces and Adapters provide the heart of the Zope Component Architecture, which beats at the center of the Zope 3 framework and application server, and is also pumping new life into Zope 2. The other chief piece of the Zope Component Architecture are Utilities. Interfaces allow coding to a specification, which can be as detailed or as general as one chooses. Adapters allow for other objects to fit that specification. Utilities are more general purpose tools that match a specification. The rest of the core 'zope' packages enhance or build on these facets, and the 'zope.app' package provides the Zope application server which is the most akin to the Zope that people know and love or hate.

Yesterday I showed how a very simple web publishing system could be made using just the basic pieces of the component architecture. Today I want to augment yesterday's example by showing how the component architecture provides "the one way to do it".

Kevin Dangoor, the primary developer behind the Turbogears project, wrote a great post today about What Turbogears Is Not. In it he discusses how choices are made about what goes into Turbogears, which prides itself on not being new technology but a combination of already offered Python toolkits. Kevin goes into coverage "one way to do it" by mentioning how the Turbogears widgets system uses Kid, the primary template system used for Turbogears. It's the default and chosen template system, so why not take advantage and say "this is how we do widgets. period," right?

Anyways, it got me thinking about what I said yesterday about the Zope Component Architecture and how everything flows through it at some point. 'Multi-adapters', especially, have become powerful. Most commonly, they're used to provide views for objects. Yesterday showed a couple of custom views built to display simple and detailed information about racing greyhounds. Today I want to show how widgets work.

First off - I did a little bit of code cleanup of the module and use zope.publisher.interfaces.IRequest and zope.publisher.browser.TestRequest instead of defining my own dummy request.

import zope.component
import zope.interface
import zope.publisher.browser
import zope.publisher.interfaces
import zope.schema

I'm not going to show the entire module again, but I want to bring back IGreyhoundData. This interface specifies what we expect any greyhound data providing object to give us:

class IGreyhoundData(zope.interface.Interface):
    """
    zope.schema provides a way of defining the properties and their bounderies
    as they make up a component. It integrates with the interface system and
    can be used for documentation, validation, data retrieval and setting, and
    more. The fields define 'get' and 'set' methods, among others, useful for
    querying and setting data on an object without having to have intimate
    knowledge of the object's structure. Useful for web forms, but also for
    database storage and retrieval.
    """
    name = zope.schema.TextLine(title=u"The Greyhound's Name")
    raceName = zope.schema.TextLine(title=u"The name on the race card.")
    fastestTime = zope.schema.Float(title=u"Fastest race time (in seconds).")
    trackLength = zope.schema.Int(title=u"The length of the track, in yards, of the fastest time.")
    retired = zope.schema.Bool(title=u"Is the greyhound still racing?")
    adopted = zope.schema.Bool(title=u"Has the greyhound been adopted?")

Lastly, I created a 'simpleConfigure' function in the top level of the module that registers the main views and utilities with the component architecture. This configure option can be run without any requirements aside from the 'zope' Python packages being in the Python path.

def basicConfigure():
    """ 
    Register the views and utilities with the component architecture. This
    could have been done in the top-level of the module, but deferring until
    now is a little bit cleaner and allows other configuration options to be
    made.
    """
    greyhounds = GreyhoundFinder()
    zope.component.provideUtility(greyhounds, name='greyhound')
    zope.component.provideAdapter(RacerView, name='view')
    zope.component.provideAdapter(DetailsView, name='detail')

All of the provided components in this setup also provide their own information about what interfaces they in turn provide and adapt. As a refresher, 'greyhounds' is an IDataFinder utility, used by the publishing system to look up the "model" object. The two adapters are multi adapters for IGreyhoundData and zope.publisher.interfaces.IRequest. They provide 'ISimpleView', which has a 'render()' method to render the view to the response stream. I want to reiterate that this has no dependencies on anything that the world thinks of as Zope. This is all in a single module that could very easily be published via CGI without ZODB, transactions, long-running-anything, security restrictions, through the web programming... None of it.. This just uses the heart.

However, showcasing the widgets used by Zope 3 does require a full Zope setup. Technically, I could configure the needed widgets myself and show how that's done, but that's beyond the point of this exercise. I chiefly want to show how the Widgets flow through the same heart. There's just one way to do it. To make a rudimentary edit form for a Greyhound in the little system I have set up, we create another class to provide the edit form view. I try to document everything that's going on. Take note of the use of zope.component.getMultiAdapter() to get the widget.

class EditForm(object):
    """ Requires zope.app.form widgets to be configured. """
    zope.interface.implements(ISimpleView)
    zope.component.adapts(IGreyhoundData, zope.publisher.interfaces.IRequest)

    def __init__(self, context, request):
        self.context = context   # aka, the racer
        self.request = request

    def render(self):
        """ 
        Goes through the schema fields and looks up the IInputWidget adapter
        configured for the field and request type. 'zope.app.form' is imported
        here to avoid placing burden on the rest of this module, since this
        code requires the widget configuration to have occurred. Just importing
        'zope' does not provide this, but running a Zope application server
        does.
        """
        from zope.component import getMultiAdapter
        from zope.app.form.interfaces import IInputWidget
        racer = self.context
        request = self.request
        out = []
        for name, field in zope.schema.getFieldsInOrder(IGreyhoundData):
            value = field.get(racer)

            # Just like getting a full 'view' to render, widgets are also
            # provided as multi adapters, adapting an object (in this case, a
            # field) and the request to an IInputWidget implementation. A
            # display-only widget would use IDisplayWidget.
            widget = getMultiAdapter((field, request), IInputWidget)
            widget.setRenderedValue(value)

            # A widget has a label, which it often gets from the field's title.
            # To render the widget, just call it.
            out.append('%s: %s' % (widget.label, widget()))

        return '\n'.join(out)

The first few lines of the render() method just help establish some shorter names, and also defer loading zope.app.form so it's not required by this whole module - although in theory it wouldn't harm anything if it were imported at the top. The zope.schema introspection machinery is used again to walk through the fields of an Interface object and get the fields. We get the object's current value by going through the field. Then we get the widget. The Zope Component Architecture comes into play once again as we try to adapt both the field and request object to an IInputWidget. The value is set on the widget based on the object's value, and we add it to the output.

How is that widget made? Does it use templates? Just write out strings in Python directly? It doesn't matter. The IInputWidget interface is all we care about here. Form submission and validation goes through the widgets and fields too, but that's not important for right now. What is important and interesting is how adapters are used here to deal with widgets, and they're completely independent from the schema. Using the 'schema' objects, one could write completely different adapters to do object-relational mapping and provide converters to read/write SQL data based on the object specification, while still using that same specification to render web forms or write out dynamic XML-RPC or JSON request responses. sqlformatter = getMultiAdapter((field, databaseConnection), ISQLFormatter) could easily be used. As a technical aside - with how multi-adapters and interfaces work, there could be generic formatters for all data types but a specific datetime formatter could be provided for a SQLite or Oracle connection. Multi-adapter lookup will match the most specific adapter based on the specifications of the objects passed in. See what I mean about everything flowing through the same heart?

It's time to put our new view into play. I added an extra configuration method that would set up the basic components and add in the edit form:

def configureWithWidgets():
    """ 
    Establishes basicConfigure and also enables the edit form which uses the
    zope.app.form system for widgets. Requires a configured Zope app server
    setup - at least with the zope.app.form widgets configured to provide
    adapters.
    """
    basicConfigure()
    zope.component.provideAdapter(EditForm, name='edit')

And with how our 'simplePublish' system is set up (the only change since yesterdays code is the use of zope.publisher.browser.TestRequest instead of the self-defined simple request), we now have edit actions:

>>> simple.configureWithWidgets()
>>> print simple.simplePublish('greyhound/edit/1')
The Greyhound's Name: <input class="textType" id="field.name" name="field.name" size="20" type="text" value="Betty Joan" />
The name on the race card.: <input class="textType" id="field.raceName" name="field.raceName" size="20" type="text" value="Beauty" />
Fastest race time (in seconds).: <input class="textType" id="field.fastestTime" name="field.fastestTime" size="10" type="text" value="31.22" />
The length of the track, in yards, of the fastest time.: <input class="textType" id="field.trackLength" name="field.trackLength" size="10" type="text" value="503" />
Is the greyhound still racing?: <input class="hiddenType" id="field.retired.used" name="field.retired.used" type="hidden" value="" /> <input class="checkboxType" checked="checked" id="field.retired" name="field.retired" type="checkbox" value="on" />
Has the greyhound been adopted?: <input class="hiddenType" id="field.adopted.used" name="field.adopted.used" type="hidden" value="" /> <input class="checkboxType" checked="checked" id="field.adopted" name="field.adopted" type="checkbox" value="on" />


The Zope Component Architecture - Interfaces, Adaptation, and Duck Typing

We've done amazing work in the past few months with Zope 3. Compared to the black magic of Zope 2 and so many other frameworks I look at, it's quite refreshing. It's not always easy, but it's been fun while also being quite trustworthy. Even without a deep understanding of all that's available, my very small company and our even smaller team of developers have achieved incredible results: a from scratch rewrite of one of our oldest and most heavily used customer's web sites, three other customers with solutions built on top of the same stack written and extracted for that earlier customer; a one week turnaround (including graphic design) of one of those other customer's web sites; sharing of benefits of refactoring of code from the customer specific layers of the stack to more general layers; an 'admin' user interface far more capable and complete than just about anything we've delivered in the past; all developed faster than ever before; and with a clean architecture despite only the lightest of planning.

What makes all this possible? The Zope Component Architecture. The Zope Component Architecture is a pretty lightweight system, especially since Zope 3.1 which simplified the component architecture that shipped with Zope 3.0 into a simple vocabulary. There are only a couple of core Python packages that provide the heart of the component architecture. And everything essentially flow through the heart.

The core packages that provide the Zope Component Architecture are zope.interface and zope.component. zope.interface provides a way of describing and querying specifications. Similar in some ways to Interface objects in Java or in systems like CORBA and ILU, zope.interface Interfaces can be used to describe objects, policies, and APIs separate from their implementation. By writing against the interface, Zope 3 takes advantage of one of the key advantages of dynamic languages like Python - Duck Typing. Duck Typing is a way of saying "if it quacks like a duck, then that's good enough for me!" Just putting the zope libraries on the Python path - configuring and setting up nothing else - here's how this works:

>>> from zope.interface import implements,  providedBy, Interface
>>> class IQuacksLikeADuck(Interface):
...     def quack():
...         """ Returns a quacking sound. """
... 
>>> class Platypus(object):
...     implements(IQuacksLikeADuck)
...  
...     def quack(self):
...         return "In my world, a platypus goes 'quack!'"
... 
>>> ben = Platypus()
>>> IQuacksLikeADuck.providedBy(ben)
True
>>> print ben.quack()
In my world, a platypus goes 'quack!'

Now, the normal and generally allowable Python way of Duck Typing is to do something like if hasattr(ben, 'quack'): ben.quack(). But that sort of 'hasattr' code only goes so far. Zope 2 is littered with skeletons like if getattr(obj, '_isPrincipiaFolderish', 0):.... As systems grow, developers forget those old names. Or objects just grow heavy with odd little attributes that say they have not only one thing about them that makes them folderish, but many. And if its folderish, then maybe we'll draw a tree for it. But if '_isPrincipiaFolderish' is really an important attribute - where does that get documented? Who knows it's there? Who knows what its values may be. Can it be callable? Zope 2 actually has attributes like this - elements that started out as simple attributes, and then grew to be callable for situations that needed to do computation. And then grew to be callable with parameters for really specific situations. But finding out about all of these combinations and their meaning was not always easy to document or even infer. Recognizing this, an Interfaces package was made for Zope 2 but was initially used only for documentation. But Interfaces were not used as core elements, and there was little impetus to keep an interface spec up to date when it wasn't actually used in code. But in Zope 3 and in new Zope 2 work, that changes. Interfaces are core members. It means that not only are most important things documented, but that it's really more important for an object providing a certain interface to just provide that interface - and not how its implemented. I don't care how it quacks, as long as it quacks. Quacking is all that I require. Interfaces allow one to formalize duck typing, without really adding too much weight to their program. In fact, with interfaces being a sort of public interface, I find it easiest now to look at a packages 'interfaces.py' module first when looking at new code. How a specific implementation of a security policy doesn't really matter to me at first - I want to know how I interact with the security policy. So interfaces become useful for documentation without having to generate or wade through documentation generated for every single function and class and method in a particular module.

Of course, interfaces on their own are still not all that great. The second part offered by zope.interface is adaptation. Sticking with the quacking example, let's add a hunter into the mix. He wants to go duck hunting. He's going to need to attract ducks by quacking like them.

>>> class Hunter(object):
...     def __init__(self, name):
...             self.name = name
... 
>>> tom = Hunter('Tom')
>>> IQuacksLikeADuck.providedBy(tom)
False

What we need here is an adapter. This is where zope.component and zope.interface really collaborate. By default, most adapters in Zope adapt from one interface to another. But you can also adapt a class to an interface, which is what we'll do here since we didn't make an 'IHunter' or 'IPerson' or 'IQuackless' interface.

>>> from zope.component import adapts
>>> class DuckCall(object):
...     implements(IQuacksLikeADuck)
...     adapts(Hunter)
...  
...     def __init__(self, hunter):
...         # Adapters are passed in their adaptee as first argument
...         self.hunter = hunter                                 
...  
...     def quack(self):
...         return self.hunter.name + ' quacks with a duck call'
... 
>>> zope.component.provideAdapter(DuckCall)

So here, we say that instances of DuckCall implement the IQuackLikeADuck interface, and that this particular class adapts Hunter objects. The next line registers the adapter with the global component architecture adapter registry. This is the easiest version of the call - a factory (in this case a class) is passed in which declares what it adapts and provides directly. Now there's a duck call available for hunters. How do we use it? There are a couple of ways. One is to use zope.component.queryAdapter or getAdapter - the only difference being that queryAdapter returns 'None' or a user supplied default if the adaptation can't happen, while getAdapter raises an exception. But the interesting way is to use the interface directly, passing in the object:

>>> IQuacksLikeADuck(tom)
<__main__.DuckCall object at 0x67cad0>

Like with 'queryAdapter', a second argument can be passed in as a default if the object cannot be adapted, otherwise an exception is raised. What this form allows for is code that works with objects by interface, and it allows for other code to provide expected interfaces for existing objects. Lets add to the wildlife. First, there's the hunters dog:

>>> class IrishWolfhound(object):
...     size = 'HUGE'
...     def __init__(self, name):
...             self.name = name
... 
>>> squeekers = IrishWolfhound('Squeekers')

And a duck. Since the duck provides 'quack' directly, we can separately say that instances of the class provide the IQuacksLikeADuck interface, or we could even say that a particular instance provides it. This shows how we can separately say that the class implements it.

>>> class Duck(object):
...     def quack(self):
...             return "The best quack is a duck quack."
... 
>>> hunted = Duck()
>>> IQuacksLikeADuck.providedBy(hunted)
False
>>> from zope.interface import classImplements
>>> classImplements(Duck, IQuacksLikeADuck)
>>> IQuacksLikeADuck.providedBy(hunted)
True

And it should be noted that if an object provides an interface itself, then the adapter lookup will return the object itself. So now that we have quite a little menagerie. Lets see what some simple tickler code might look like that would take a list of objects and try to make them quack.

>>> def tickler(*args):
...     " Goes through the provided arguments and tries to make them quack "   
...     for potentialQuacker in args:
...         quacker = IQuacksLikeADuck(potentialQuacker, None)
...         if quacker is None:
...             print "Could not quack: %r" % potentialQuacker
...         else:
...             print quacker.quack()
... 
>>> tickler(ben, tom, squeekers, hunted)
In my world, a platypus goes 'quack!'
Tom quacks with a duck call
Could not quack: <__main__.IrishWolfhound object at 0x698bd0>
The best quack is a duck quack.

Of course, this is a fiendishly simple showcase. But it's a powerful building block, and is one of the core concepts of Zope 3. Just about everything flows through interfaces and adapter registries. A powerful extension of the 'adapter' concept is the 'multi-adapter'. The multi-adapter situation most commonly used is the Browser View. A Browser View is made by adapting an object, like the Duck, and another object, like an HTTP Request, to something that gets rendered to a web browser via HTTP. Fundamentally, the Component Architecture merely provides the registries for loosely-coupled objects to work together, and it removes a lot of the magic.

In something purely theoretical, lets remove all of Zope as people know it (the object database, the publisher that traverses URLs to return objects, and so on and so on. Using the component architecture, lets do an extremely lightweight 'rails'-ish thing. This code makes a fake little database with just tuples of data, as might be returned by a basic dbapi adapter. It defines a Utility to find the 'model' from this database. A utility is a new concept. In quick, utilities are other abstract notions, used loosely. One can define and install different utilities, and the code that uses the utility has no hard dependencies on the utility object. Examples include RDB connections - 'get me an IRDBConnection with the name "racers"'; caches; and more. Sometimes all utilities for a particular interface are queried, sometimes you just want an individual one. The zope.component.provideUtility call establishes the utility. In this code we have only one and we name it 'greyhound'. That name is used later in the 'simplePublish()' function to find the utility by combination of its name and published interface. Again - this code was tested by ensuring that my Zope 3.1 installations lib/python directory was in the Python path and that's it.

import zope.component
import zope.interface
import zope.schema
class IGreyhoundData(zope.interface.Interface):
    """
    zope.schema provides a way of defining the properties and their bounderies
    as they make up a component. It integrates with the interface system and
    can be used for documentation, validation, data retrieval and setting, 
    and more. The fields define 'get' and 'set' methods, among others, useful
    for querying and setting data on an object without having to have intimate
    knowledge of the object's structure. Useful for web forms, but also for
    database storage and retrieval.
    """
    name = zope.schema.TextLine(title=u"The Greyhound's Name")
    raceName = zope.schema.TextLine(title=u"The name on the race card.")
    fastestTime = zope.schema.Float(title=u"Fastest race time (in seconds).")
    trackLength = zope.schema.Int(title=u"The length of the track, in yards, of the fastest time.")
    retired = zope.schema.Bool(title=u"False if the greyhound is still racing.")
    adopted = zope.schema.Bool(title=u"True if the greyhound has a home after the track.")

class IDataFinder(zope.interface.Interface):
    def find(oid):
        """ Returns an object that matches this object id. Raises KeyError if not found. oid should be an integer. """

# Define a fake database
greyhounddb = {
  1: ('Betty Joan', 'Beauty', 31.22, 503, True, True),
  2: ('Dazzling', 'Dazzling Pet', 32.00, 503, True, False),
}

class Phantom(object): 
    """ Phantoms can be anything... """
    pass
 
class DataFinder(object):
    zope.interface.implements(IDataFinder)
    schema = None
    factory = None
    db = None
    
    def find(self, oid):
        """ 
        Calls the fake database and uses the schema system to map fields
        from the tuple returned to a new Phantom object, and then ensures that 
        the phantom is marked to provide the schema in 'self.schema'
        """
        raw = self.db.get(oid)
        newobj = self.factory()
        for idx, name in enumerate(zope.schema.getFieldNamesInOrder(self.schema)):
            field = self.schema[name]
            value = raw[idx]
            field.set(newobj, value)
        zope.interface.directlyProvides(newobj, self.schema)
        return newobj

# Define a basic data finder
class GreyhoundFinder(DataFinder):
    schema = IGreyhoundData
    factory = Phantom
    db = greyhounddb

greyhounds = GreyhoundFinder()
zope.component.provideUtility(greyhounds, name='greyhound')

class ISimpleRequest(zope.interface.Interface):
    headers = zope.interface.Attribute('Mapping of request headers')

class SimpleRequest(object):
    zope.interface.implements(ISimpleRequest)
    def __init__(self, **headers):
        self.headers = {}
        self.headers.update(headers)

class IView(zope.interface.Interface):
    """ Simple view interface. """
    def render():
        """ Returns a rendered string of this view. """

class RacerView(object):
    zope.interface.implements(IView)
    
    def __init__(self, context, request):
        self.context = context   # aka, the racer
        self.request = request
        
    def render(self):
        return "Name: %s; Fastest Time: %0.2f" % (self.context.name, self.context.fastestTime)
zope.component.provideAdapter(RacerView, (IGreyhoundData, ISimpleRequest), name='view')

detail_template = """Name: %(name)s
Racing Name: %(raceName)s
Fastest Time: %(fastestTime)0.2f for length %(trackLength)d
Retired: %(retired)s
Adopted: %(adopted)s
"""

class DetailsView(object):
    zope.interface.implements(IView)
    
    def __init__(self, context, request):
        self.context = context   # aka, the racer
        self.request = request

    def render(self):
        racer = self.context
        mapping = dict([ 
            (name, field.get(racer)) 
            for name, field in zope.schema.getFieldsInOrder(IGreyhoundData) 
            ])
        return detail_template % mapping
zope.component.provideAdapter(DetailsView, (IGreyhoundData, ISimpleRequest), name='detail')
        
def simplePublish(path):
    """ 
    Breaks 'path' up into three components:
    
    - ``name``: used to look up a data finder utility
    - ``viewname``: used to query a multi-adapter for the found object and simple
      http request
    - ``oid``: used to look up an object out of the finder
    
    from ``name/viewname/oid``
    
    returns the rendered view.
    
    A very very simple showcase of how the zope component architecture might
    be used to register and find model accessors and views in a web publishing
    system besides the traditional Zope and 'zope.app' setup.
    """
    request = SimpleRequest(ignored='hi')
    getter, viewname, oid = path.split('/')
    oid = int(oid)
    
    # This gets an 'IDataFinder' utility with the name in the 'getter' value.
    # A greyhound getter that queries a real database could have been installed
    # instead - it doesn't impact this code.
    model = zope.component.getUtility(IDataFinder, name=getter).find(oid)
    
    # Now that we have the model object and the fake request, find something
    # that adapts both of them to a single object. Since the request is fake
    # and the views are simple, we don't really show how the request object
    # is used here. But this is basically where the web environment and model
    # environment would actually meet - only in the multi-adapter. Other than
    # that, both environments are oblivious to each other. It's only the zope
    # component architecture registries that know what's in them, and they're
    # pretty agnostic.
    view = zope.component.getMultiAdapter((model, request), name=viewname)
    return view.render()

def test():
    print "Publishing ``greyhound/view/1``"
    print simplePublish('greyhound/view/1')
    print
    print "Publishing ``greyhound/view/2``"
    print simplePublish('greyhound/view/2')
    print
    print "Publishing ``greyhound/detail/1``"
    print simplePublish('greyhound/detail/1')
    print
    print "Publishing ``greyhound/detail/2``"
    print simplePublish('greyhound/detail/2')
    
if __name__ == '__main__':
    test()

And when run:

Desktop> python zopetest.py
Publishing ``greyhound/view/1``
Name: Betty Joan; Fastest Time: 31.22

Publishing ``greyhound/view/2``
Name: Dazzling; Fastest Time: 32.00

Publishing ``greyhound/detail/1``
Name: Betty Joan
Racing Name: Beauty
Fastest Time: 31.22 for length 503
Retired: True
Adopted: True


Publishing ``greyhound/detail/2``
Name: Dazzling
Racing Name: Dazzling Pet
Fastest Time: 32.00 for length 503
Retired: True
Adopted: False

15.12.05. Agility versus Agility in Zope Development

Paul Everitt, one of the original founders of what is now Zope Corporation who is now doing a thousand different things across Europe, has a good post about 'agile content development', in reference to a CMS Watch post about a Mark Baker presentation (PDF) about this issue. Paul makes some interesting notes and reminders about some of the power of "classic Zope" through-the-web development, including a particular case where a fed-up Vignette user taught himself just enough through-the-web (TTW) Zope to be "agile" and made a replacement system for the big expensive Vignette setup in a matter of weeks.

Most seasoned Python and hardcore Zope developers cringe at the thought of managing Zope 1 and 2's through the web code. It presents many challenges for software configuration management, version control, editing, and so on. But for many users, there was a lot of flexibility in what Zope 1 and 2 offer here: free form schema management and development (using folder and document properties or ZClasses), changes instantly reflected throughout the system, ability to tweak individual scripts / handlers / templates, and so on. It turns the ZODB into more of an 'image' (in Smalltalk sense) than being just a plain database.

An interesting example of how people have learned and applied these techniques rapidly is something that a co-worker did a few years ago for one of our major customers. This customer could manage all of their free-form content via FTP, using some clever tricks we implemented. But for structured content - entries that had specialized fields - they managed that through web forms. Before Zope Page Templates, there were a couple of core dynamic content types in Zope - 'DTML Methods' and 'DTML Documents'. DTML Methods had no properties of their own - they were meant to act in the context of their container (usually a regular Zope folder). DTML Documents had their own property sheets, and the dynamic content acted in the context of the document. With 'Properties' on most Zope objects that supported them, you could add and remove property fields free-form, and marshal them to a particular data type. The default Zope property management screen would render and parse appropriately. This is long before any intelligent widget / form framework existed for Zope. It was just something that Zope did. So what my co-worker did for this customer was to use this properties system to make special objects. They were all DTML Documents, but the customer didn't know that. A property on the folder described what properties and types to add to new documents, and then they were rendered into editable fields by just asking for the properties. It's something that I wouldn't have even thought to do as a more seasoned Zope developer. But it was a plan that worked and served this customer well.

I think that maybe this is why I'm not so impressed when I see the "scaffolding in action!" screen cast for Rails or its wanna-bes. With classic Zope, even hurried and unfamiliar developers seemed to find ways to implement things like that, and without ever having to touch SQL. And for all of the faults that through-the-web development (particularly on the code side) may have, it was nice to be able to make quick fixes and patches from almost anywhere. There is something interesting about environments where objects (data) and code live in the same big ol' database. Smalltalk users and people who have used Lisp Machines often speak of their love of the seamless environment.

But let's face it - Zope 2 is great. But it's also weird and sloppy. It tries to cater to too many audiences, and there are people who have great love for certain aspects that other people loathe. ZClasses, which came the closest to allowing actual object and class development through the web, have little love from many 'power' users. Yet in the early days of Zope 3 development when features and plans were being discussed, there were many Zope 2 users who stood up in its defense: "It's so easy for me to make custom schemas through the web and make new objects for my users without having to program!" came up from a few different people. I don't even understand them! Through-the-web coding and data management in Zope 2 is a very fuzzy system. There often aren't any clear boundaries, and sustained application development in this model often gets difficult as time and size progress. The CMF (Content Management Framework), upon which Plone is built had some interesting ideas, some of which allowed for some powerful and strange customization. One of these was the 'Factory Method Content Type' (I think that was the name). Or 'Scriptable Content Type'? Anyways, with it you could take a basic base class/object type, like a Document, and really customize what happened to it at creation time. It was another way of providing this sort of strange 'agile content development' (from an implementer side) by allowing programmatic customization. Tres Seaver, one of the core CMF developers, once talked about a prototype concept, allowing any object in the Zope system to become a prototype.

An interesting concept of classic Zope, especially in the Principia (pre-Zope 2) days, was that Folders were customizable objects. They could have any number of custom attributes (properties) set on them, and have custom code (DTML Methods, initially) written for them. If you could grasp this concept, it could really be quite powerful for more than a few situations.

We still have one customer whose application is written in the old style - folders full of scripts and templates - templates, code, and configuration in the ZODB with business data in an RDBMS. We separate the scripts from the templates as much as possible, and it's a fairly manageable site. It's one that I sometimes cringe at having to deal with. But it's serving our customer just fine. While this customer is not much of a developer, he has often taught himself ways around the system when he wants to do something new (and small). We do all the major development. But it's often an interesting surprise to see how he's organically grown the parts he has access to.

Since late summer, we've shifted a lot of our focus to Zope 3. We've been building our own content management system on top of it, largely from scratch. Its initial customers are four sister companies, one with unique content management requirements, the other three with basic requirements. Zope 3 makes it easy to write and structure good frameworks and toolkits, and initialize them in a specialized order (much better than Zope 2's magical Products package initialization). We were able to structure our code into layers, from the big horizontal CMS layer to a narrower layer that provided tools that worked for all of these (and other potentially related) customers to individual packages providing direct customized components and views. By the time we got to the fourth customer's site, we were able to turn it around in a week, including graphic design and content entry. I think that's one of our fastest turn-arounds. Agile? You bet. And as the core improves, it's easy to get those updates to our customers, since it's properly managed Python code.

Now the ZODB is a database exclusively for content/data. We have no code in it anywhere. Those features aren't really there in Zope 3 right now, and I don't miss it... much. Unlike the myriad of potential audiences and uses that one could get out of a stock Zope 2 configuration, Zope 3 remains focused on developers at this point. I think that's a good thing. Even if one were to ignore the zope.app package, which provides most of the Zope application server functionality, there is a very impressive base. If someone wanted to make a version that had through-the-web classes and code, many of the base components that are necessary are there. If someone wanted to do a prototype based system, they could do it. I believe that with Zope 3 we are going to see more specialized applications released and built that can target different audiences, hopefully without having to make the core Zope systems have to cater to everyone.

Work has been done to allow Python code to exist in the ZODB as modules in Zope 3, with tools to allow checking code in and out from the file system. It's still incomplete, it seems, and I don't know if it's going anywhere. The file system checkin/checkout would allow one to use traditional tools (editors, version control) while also being able to take advantage of having code running in an image - dynamic updates, distribution (using tools like ZEO or Zope Replication Services), and so on.

Why might this be nice? Watch this QuickTime movie of Seaside halos. Seaside, a continuations based web framework written in Smalltalk, shows off many of the benefits of running code out of an image - live debugging, instant changes, and more. What's most interesting to me, however, are the "Halos". When they're toggled to be on, many elements of the web page get wrapped in boxes with an extra toolbar that allows live inspection and manipulation. Need to change the code that's rendering the shopping cart? Click on the halo button and you'll be taken directly to the method that's rendering it. All other code seems to be available for manipulation as well.

That's the closest I've been reminded of Zope 1 or 2. A dynamic running object environment on the web with live code in the database engine. The key difference is that Seaside's code editor still fits into how one might expect Smalltalk to behave, providing a web version of the System Browser. And I imagine that the Seaside Code Browser treats all code and objects the same. With Zope, file system code (regular Python code) is naturally off limits. It's also loaded and reloaded differently - updating a Python Script through the web has immediate repercussions. Updating something on the file system requires a manual product refresh or server restart.

Ugh. This is long and rambling and I'm not sure if there's really any point to it all - just some thoughts. Paul's comments reminded me of what we were getting away with in Zope back in the late nineties, before PHP had even started to make much of an impact. Zope 1 and 2 has always been a strange beast, quite unique from all other servers and frameworks. It's been very successful, yet also has a lot of people who hate it. Those of us who have used it for so many years still don't know how to explain it, and may even have two or three very different development styles for it depending on mood and the problem to be solved. Zope 3 is so refreshing - I get to take the core concepts that I've been using since 1997 but get a clean and free architecture to work with. I feel I have more freedom to make Zope 3 "my own". I could implement 'halos' for our content system. Or maybe start offering prototype based content objects. It's wide open.

6.12.05. Tinderbox - More on the Map View

Yesterday I touched on using Tinderbox as it's become a core part of my Getting Things Done workflow. In particular, I became enamored with the Map View yesterday as I had to plan out a tricky little project. While working in the map view, where I was able to think things out spatially instead of hierarchically, the items I created were still regular Tinderbox items and were wired into my GTD agents. When I went to work on this project today, I was back in hierarchical mode, checking things off as I got them done.

Ryan Holcomb's GTD template uses color to distinguish items as they move through the flow, with these colors being set automatically by the Tinderbox agents - yellow for 'on hold', orange for 'complete', bright green for 'due today', red for 'overdue' (for items marked with a due date). A benefit of this is how it appears in the map view. Even when items are children of items, they show up in a kindof thumbnail view at the current level. Thus today, checking in on the map view of my project yielded this:

This shows me that I still have one item to do for the 'Sharing View' (in the box on the right), and one item on hold in the other box. Very nice.

5.12.05. Tinderbox, Planning, and Getting Things Done

Over recent weeks, I've been trying (as I often do) to get and stay on top of things. I think I may have found a system that works. I've said this many times before, but this time may be different.

It started a couple of months ago when Kinkless GTD showed up on the scene. Kinkless GTD, also known as KGTD, is an implementation of David Allen's Getting Things Done system as an OmniOutliner document with supporting AppleScripts. It's an excellent showcase for what can be done with a well-scriptable application. OmniOutliner (and the pro version which KGTD uses) is fast, nimble, simple, but still quite powerful. It's been one of the best Mac OS X applications, and one I've owned since the first release.

Kinkless GTD had some great features that made maintaining my list of things to do easy: combination of projects and contexts (projects are a collection of actions, contexts is where those actions take place). Actions can be entered in either the project-oriented or action-oriented view and get synchronized between the two lists (different sections of the outline). Next actions are distinctly highlighted. Archival of completed actions is a single click action, easily hiding old actions and information. All of this, and more, done by taking advantages of some of the simple yet flexible features of OmniOutliner and its scripting dictionary.

At the same time, there were some aspects of KGTD that didn't quite work for me. Upgrading wasn't always easy and it's still in active development. I stopped upgrading after 0.69 (it's at 0.75 as I write this). In order to keep the lists in sync, deletion and completion required using special buttons that could be installed on the toolbar that ran applescripts to make sure the version on the other list was removed or cleared too. Changing the text of an item on one list would cause the old version to re-appear (sometimes twice) after synchronization. There's good reasons for why this happened - the script just copied items from the lists (projects, actions, and 'next actions' - the single next action for any project). It was hard for me to break habit from regular OmniOutliner usage - using native widgets to mark 'done', etc. The number columns in the document started growing rapidly, which cut back on room to dedicate to the actual text of the action. Configuring the toolbar with the new buttons caused all documents to pick up those buttons, whether they're KGTD or not. This is a Mac OS X detail. So while the toolbar made working with the KGTD document easy, it was useless for other OmniOutliner documents.

So while I was really impressed with KGTD, the shortcomings (which are admittedly minor) bothered me enough to start looking at something bigger: Tinderbox.

Tinderbox is a strange beast. I've looked at it before and have downloaded the demo often, but I couldn't quite understand it well enough to justify the cost involved. Tinderbox is really a sort of free-form hierarchical / hypertext database for notes. What attracted me this time is that it too has some strong templates for implementing a system for Getting Things Done - Ryan Holcomb's "GTD Improved" and "GTD Lite" templates, available from the Tinderbox File Exchange.

Tinderbox features prototypes, extensible attributes (that don't eat into column views), Rules and OnAdd actions for automatically setting attributes for children of certain elements, and most importantly - agents. Agents are queries and actions that can be run automatically or manually. For Holcomb's template, they do a lot of what KGTD's synchronization script does - bring items from the project lists to contextual action lists. Unlike KGTD, they're not copies - only aliases. No manual 'sync' action needs to be run - as you mark things off as completed, for example, they disappear from the action lists. Holcomb's template is not under heavy development, so there's no upgrade worries. The way that it works is done using standard Tinderbox features, and it was pretty easy for me to start customizing some behaviors and defining new agents and prototypes without even having read much of the Tinderbox manual. It didn't take long for it to start feeling like my own system.

In Kinkless GTD I stayed in project mode most of the time, in a fashion I was used to working with outliners for task lists. The contexts didn't impact me that much, even though it was KGTD that showed me why the context lists were valuable. With Tinderbox it's easier to separate planning from doing - to go through and plan out the list of next actions, their contexts, notes, etc, and then look at the '@office' agent view and start churning through the list. As that list is churned through and items are marked completed, they disappear from that list. The original item is colored orange so that when seen in other views there's a visual cue that the item is completed. The orange coloring is done by an agent. Other agents modify visual cues based on whether an item is on hold, due soon, or past due.

So this is all really cool. It's a hierarchical active database. The agents, prototypes, and ability to tweak 'OnAdd' actions bring the active parts without need for heavy scripting - just learning the query language. In some ways, this might sound like it's entering into Frontier territory. But Tinderbox is not a development environment. While it uses agents and prototypes, and can house/use AppleScript, it's a very different beast than even old versions of Frontier. Tinderbox is very much focused on maintaining projects, notes, and plans, and has many different ways of viewing and interacting with your data. And today, two weeks after finally purchasing it, I finally used one of its most powerful features: map view.

In map view, you work with visual representations of notes, appearing as boxes with the title displayed. You can add extra adornments (items only seen in map view) to group items together in a non-containment way. You can resize notes and drop other notes inside them to turn those notes into children. And perhaps most importantly, you can easily create links between items in this view. While I'm not a fan of hardcore mind-mapping, I do like the basic concepts. And sometimes it just helps to think visually. With Tinderbox, you can have many different windows open on the same data. So today, I opened one of my projects up into a map view window and started re-thinking what needed to be done in order to get things moving. I'd avoided the map view up until now, even though there's a note in Holcomb's document that says it's great for brainstorming. And boy did that prove right. Normally I think very well in outlines (and thinking rapidly in outline mode is still where OmniOutliner excels over Tinderbox), but I just couldn't do it today. Thumbing back through a small notebook I've been using on recent projects, I came across a mind-map style pair of graphs and thought "maybe that's what I need."

Working in the map view of Tinderbox enabled me to see the items I'd already thought of and I could start playing with them spatially. Using adornments to make visual groupings, I was able to start segmenting off the tasks involved. And before long, I had a document that looked something like the following. Note: The following is a quick example I made up tonight - keep your eyes off my project!

Since this was in my Getting-Things-Done document, all of the items in my new visual plan were still picking up prototypes and attributes and responding to the agents already defined. I initially worried that using map view would work best with separate documents, so I was especially happy to see it work like it did. In any of the outline views, I could still do basic prioritizing by reordering items, and this did not effect the layout on the map.

Outline View

Of course, this is a hierarchical system, and that works even in map view. Notes that contain other notes can appear like this in map view. When zoomed in on any level, you can see a faint grey outline that represents the boundaries on the parent view so you can get an idea of what the thumbnailed view will look like.

Finally, agents can be written to collect interesting items. So while working in map view, properties can be set that affect the agents and you can quickly get another view of your data. This, again, is from my example document. Holcomb's GTD template has plenty of great agents that I've found useful for working in action-mode, wherein its important to just keep an eye on just enough of the road ahead to be prepared while being comfortable knowing that all other plans are in the system and don't have to be worried about.

Agent Example

Overall, I'm impressed with Tinderbox. Being able to sort-of free think in map view and still have all of those items flow into my daily work-management system is great. The application is a bit strange, but manages to be lightweight and flexible. A blank document imposes no structural inhibitions - you can keep a document as flat and boring as you like, without having to "program down" complicated defaults. At the same time, an active user community and file exchange provide a lot of insights into the program. The price is a little steep, but after using it it's starting to feel like a true professional application - that is, an application that really helps me get my work done in a way that I want.

Finally, I'm working on using Tinderbox for managing actions and planning for both my personal and professional life. I'm working on using DEVONthink to organize and manage reference materials - artifacts that need to survive beyond the day to day morass of planning and thinking and working. So far, it's working out pretty well. In the past, I tried using DEVONthink in the planning process as well as reference, and it just didn't fit the bill so well. Separating these systems has taken away the open loop of wondering where to deal with big chunks of text or random note-taking versus where to deal with planning and acting on that plan.