6.9.11. HTTP Basic Auth problems that affect Zope (2 and 3) in Safari 5.1, Mac OS X Lion

One very frustrating experience that I encountered after upgrading to OS X Lion (10.7) was that Safari 5.1, as included in Lion, would constantly pop up basic-authentication dialog boxes on our CMS sites, which are based on the Zope Toolkit (kind of between Zope 3.4 and Bluebream). Just about every page in our admin UI would do this. To get around this, I switched to Firefox for interacting with our admin screens, but still used Safari as my primary browser (better OS X citizen, takes advantage of Lion features which I enjoy, bookmark syncing, etc). This lead to problems as Firefox could get pushed way back in the usage stack and would be paged out, and it did NOT like to wake up after long periods of inactivity.

A couple of days ago I decided to take another look at the WebKit project as I was certain that I was not the only person having this issue. And I found a bug that had been recently closed relating to this issue. It's WebKit bug 66354 and it's resolution is in the WebKit nightly builds since at least September 2, 2011.

Apparently this only affects OS X Lion as it has to do with low level CFNetwork changes in Lion. The bug occurs not with basic auth, exactly, as I was able to use other systems behind basic auth. The bug occurs when there are redirects with Basic Auth, which our CMS uses a fair bit in its admins for basic navigation links.

As of OS X 10.7.1 and its Safari (Version 5.1 (7534.48.3)) this is broken. If you use OS X Lion and Safari and encounter HTTP Basic Auth problems, I'd recommend switching to the nightly builds.

 

13.4.07. The Web Will Not Replace the Desktop

Web 2.0 has excited us because we lowered our expectations so much. Of course web apps will get better, and one day will deliver the functionality we currently get from desktop software. They may even do more than our desktop applications one day. But isn’t it a tad strange that we think this is all a huge leap forward? - loose wire blog: It's Not the "Death" of Microsoft, it's the "Death" of Software

The author's main point is that while it's cool that people are making Mind Mapping tools in DHTML, they're still a long ways behind desktop apps like MindManager. He goes on to contend that there's just nothing exciting in the "offline" world in recent years.

While it's true that the web has made for some neat and very useful online tools, there are classes of software missed. There's a reason why we'll never get to the "every computer is just a web browser / flash player" ideal: professional software. It's a world I'm entering again as I'm finally getting my home studio together.

I'm talking about apps like Pro Tools, Reaktor, Final Cut, DVD Studio, Aperture, Lightroom, InDesign, Quark XPress, etc.

Granted, most people don't use those applications, but I think that it's a growing market. As technology grows and commodifies, we need tools to deal with it. The gap between pretty-good consumer gear and pretty-good entry level professional gear is pretty small now in many areas: digital photography, digital video, music, etc.

Now that I think about it, Apple has realized this for some time. They have a pretty good upgrade path. For those who get hooked playing with video in iMovie, there's Final Cut Express at a rather reasonable price. Those who really start to do well with that can go up to the full Final Cut Pro. For Music, there's Garage Band, Logic Express, and Logic Pro.

Within those realms, there's a huge array of plug-ins, virtual instruments, specialized sound tools and environments (Ableton Live, Reaktor 5, Max/MSP, etc). I am amazed at the sounds I get out of Reaktor, and that's only a single product in a single company's impressive set of offerings.

Perhaps the new web applications are freeing up resources on our own machines so that it no longer feels like some dreadful work environment, just at home. There are plenty of useful and usable online tools for doing quick writing, sharing, interacting, thinking, and planning. Typically they offer enough to be usable for those small (or even medium) jobs we occasionally encounter, while freeing us from having something overkill like Office for casual, personal writing. It's easier to specialize a computer for audio work by fine tuning system settings, throwing away silly applications, etc, without making that computer into an island. As long as you have a web browser, you can still check email, contribute to a planning document, etc.

But honestly, I don't think the Desktop is going to die - ever. It's great that we can do so much on the web, but I don't think the native experience is going to die, ever.

And even if you're not on a fucking plane, it does matter: when I moved into this loft, it took me a couple of months to get internet access down here. I was working on a lot of things for the office at the time, and I was able to take it home by just using my laptop: at work, I'd synchronise source code, copy stuff to my laptop and/or sync with .Mac's iDisk, and sync with .Mac for my calendar, etc; as such, even though I was offline, I could work. It was then and there, however, that I decided that although I liked Backpack, it wasn't worth paying for: I needed offline access. I needed, well, OmniOutliner and Tinderbox. My personal project files and note-taking documents are just too precious to be left online (this is why I don't and won't use Stikkit). If the occasional monster storm comes along and takes away the Internet for a few days, the worst feeling in the world would be being disconnected from my notes.

Strangely enough, these online note-takers, organizers, etc, all solve a problem that has plagued me until quite recently: how to do effective sharing of data between home, work, and laptop? How to not get out of sync? I love Tinderbox and I have a couple of big Tinderbox files that I keep on .Mac's iDisk. This means I usually have access to it. But sometimes, I forget to sync or close or save the document when I leave work or close the laptop. What about the little bits of random data, not yet filed, or not really worth filing into that larger document? How can I quickly enter, find, and share that info?

The answer didn't come from any web service, although lord knows I tried a few. The answer came when Tinderbox's developer, Eastgate, ingeniously started bundling Barebones' Yojimbo with Tinderbox. I had looked at Yojimbo in the past, but I'd gone through so many personal note taking / note capturing / note filing systems (Mac OS X has MANY). I didn't want to look at another such product and be fighting between "do I file it in Yojimbo? In Tinderbox? DEVONThink? Can I get to it from home?"

But Yojimbo has a killer feature: it syncs with .Mac! The same tool that I've used to keep calendars and contacts and Safari bookmarks transparently shared between three machines, finally someone made one of these note tools that took advantage. Now I have my enter-a-quick-note, file-it-later system that gets updated and merged automatically. No worries about having an out-of-date iDisk, about forgetting to save and sync. And best of all - it's 100% native and usable offline. And it doesn't get lost in the army of tabs since every goddamn web "app" is now just something that gets lost in a browser window (for those who wonder why I take so long to reply to mail sent to my GMail account, well, GMail sucks as an application compared to a native mail app. I just don't watch it regularly enough to stay on top of things).

So, anyways, I love a lot of the new web apps. But people need to get a grip. I've been hearing about "the death of the desktop" for eleven or so years now. Stop tricking yourself into thinking you're that high and mighty. If you don't understand the true value of native applications, professional applications, personal data, then you don't understand the desktop's power. As such, you're not going to kill it. Yes, please focus on tools that take work well on the web, tools that are simple yet useful. But don't think for a second that I want to spend every second of my computing day in a web browser, nor do I want to spend every second in an Apollo client. It's not going to replace everything, any more than Java, Netscape Constellation, or even Active Desktop did.

10.2.07. SOAP. Web Services. Flash. Waaa.

As a result of a peculiar chain of events, we’ve been stuck with a compiled Flash file requiring data off of a web service. The old database is gone. The old server is gone. We have been unable to get much information about the data required. We don’t have the Flash source. All we have is a single ColdFusion Component that is used a web service via WSDL and SOAP.

I’ve never used ColdFusion. Fortunately, Allaire, er, Macromedia, er, Adobe makes demos available. ColdFusion even comes with a developer version that can live beyond the typical 30 days. So using that, I was able to re-generate the WSDL. I didn’t understand it, but I had it. Then I installed a demo of Flash Professional. I made a quick document and put some web service links in it by stumbling around, and watched the HTTP/SOAP communication between Flash and Apache/ColdFusion.

I tried looking at the specs for SOAP, and recoiled. Then I naively thought I could emulate the response as a Zope Page Template emitting XML, looping over the little bit of data that the Flash client wanted. But then I realized that the SOAP style in use was RPC style, and the ‘method’ was buried in the posted XML, so I couldn’t even easily answer the incoming requests.

Since there’s no standard SOAP toolkit available for Python, Zope 3 only holds a skeleton off which one can hang a SOAP handling REQUEST/RESPONSE pair. I’m too dumb, or perhaps just too busy, to wrestle with XML parsing myself, and I quickly realized that I needed some kind of toolkit. The data that I saw in the WSDL and SOAP communications appeared far too complicated for me to read and write on my own. Again I tried going to the specs, and again I ran rapidly away as I couldn’t even figure out how the WSDL connected to the SOAP requests and responses.

Now, obviously this stuff must work out for somebody. When we first approached this problem, the advice that we got was along the lines “just put up an XML gateway.” Um, what? I then realized that the tools were probably doing everything and the developers with whom we were communicating never gave much thought to what was going on. Data format? There is no data format - just put up an XML gateway!

I can’t fault anyone for this, per se. We all get abstracted away from something. Many web developers plug along happily without knowing too many details about HTTP. Even when one knows quite a bit about the more advanced or nuanced details of HTTP, one’s not likely to know much about what’s going on at the TCP/IP layer.

While stumbling around Flash Professional, I was a wee bit impressed with how quickly I could make a web service connector, wire it to a table widget, write a quick button-click handler, and could see the results in the table. The ColdFusion Component had its WSDL generated by merely adding ?wsdl to the end of the URL. Since both tools are maintained by Macromedia, er, Adobe, the SOAP messages actually seem to work together.

But my gods, it sucks.

I have been so spoiled by HTML and Python, and by Javascript, CSS, Zope, etc. There’s no end to the value of the web browser’s “View Source” command. With Python, I can read the source of almost any library and figure out its innards if I can’t get answers from the documentation. CSS and JavaScript are equally open to public viewing. One can easily send plain text as results of HTTP calls - I think that if we had designed the system we’re wresting with right now, we would have served some kind of delimited text that any tool could read - even human eyes. This week I realized just how hard it is to make a simple SOAP request by hand, leaving me to use a TCP watcher/proxy to test requests and responses.

Using ZSI from the Python Web Services project, I was able to mash together a basic SOAP Publisher for Zope 3. It doesn’t deal with multipart messages, WS-Addressing or whatever the hell that is, “document/literal”, or even Request argument processing. Well, that’s made available on the Parsed Soap instance from ZSI that’s put on the request. The handlers - the remote procedures / “controllers” / “views” / whatever you want to call them, still have to do quite a bit of work themselves, at least with the Parsed Soap message. This is because, as far as I can tell, one can’t do much without using explicitly build custom types, generated from tools like wsdl2python. Ahhh, this takes me back to all of the crap I had to deal with in ILU/CORBA - baroque generated code that one couldn’t live without, nor was likely to understand.

Even after all of that, I had to do a lot of ZSI typecode wrangling in said generated code before I even began to get results like what was generated from ColdFusion. And these are simple calls, really - no request arguments, and the returns are just lists of dictionaries. I stand amazed at just how much work is involved with getting these little details right, even with a toolkit.

Even with that toolkit’s aid, the status line in FireFox would read “Waiting for data from…” This was puzzling, as I thought had finally mangled a seemingly proper Map and Array typecode / values together from the available tools. I started to worry - is there something at the HTTP level that SOAP is doing differently? Does it want chunked results? Is it not closing the connections? I started crawling deep in the networking guts of Zope and Twisted, trying different transfer encodings, verifying that the content-length header matched the content length in the response … nothing.

Then, on a whim, I decided to try dimensioning my array. ZSI’s array, by default, doesn’t seem to handle specifying some kind of variable length in the array type. So it was generating xsd:anyType[]. The ColdFusion results had xsd:anyType[n] with n being the length of the array. I finally got this working by dynamically altering half of a tuple on a typecode that descended from the strange Result type/typecode that I was building. I’m not even sure if this is thread safe, come to think of it. But by finally getting that dimension in there, Flash started to respond again.

Ugh. Remember, The S Stands For Simple.

That kind of development - closed, compiled, heavily dependent on tools like Flash that work on few desktops and cost a good chunk of money - is just so foreign to me. I’m no die-hard open source zealot: I love a lot of the commercial tools and applications for Mac OS X, and would rather use it than any of the “free desktops”. But when it comes to web development, I’m used to dynamic languages and source code that is often readily available or shareable without the need of heavy tools. This is (usually) code that I can read, which can teach me new techniques by real implementation and not vague “hello world” examples followed by nightmare specs. I’m used to the layers below that being either so lightweight or established that I don’t have to think about them much. I’m grateful that I still seldom have to think in XML. I’m glad that I have next to none of the endless Java acronyms memorized. Sure, I guess there’s a place for them all, but I’m happy that I’ve been able to make a living for the past ten years without all of that. And I’ll be happier still when this particular problem is behind us.