30th November 2009

Overengineering and a lack of testing, the pitfalls of personal projects

On this weeks show 'Content Negotiation' and browser testing

Ok, so yesterday I jumped the gun somewhat with the rather over ambitious:

asp.net MVC version of http://cargowire.net up now - pretty much the same (hopefully) fleshed out notes and added link to thebarn page

http://twitter.com/cargowire/status/6182546830

Inevitably this was followed by:

@cargowire heads up for you this is what I see in Safari 4 for http://cargowire.net OK in FF3.5 http://yfrog.com/4imo3p

http://twitter.com/porkandpaws/status/6183712464

Clearly I'd made three (or at least three of my) classic 'personal project for self only' mistakes:

1. Overengineering
2. Lack of testing
3. Eagerness to get it live

I had set out to merely replace the backend of my previous .NET WebForms project (that used XSL/XML for view separation) with an implementation of asp.net/MVC. A fairly trival task, that along the way I expanded by virtue of being my own client and wanting to try out random things.

Being effectively an engine swap (I was even using the same XSL as well as javascript/css) I neglected to worry too much about browser rendering. After all, it was the same CSS that I'd tested before... surely all would be well.

The problem: Content Negotiation

My original cargowire implementation had the ability to serve different content types baked in. You could call most pages in the following ways (you still can now, try it):

• http://cargowire.net/articles (rendered html)
• http://cargowire.net/articles.html
• http://cargowire.net/articles.xml
• http://cargowire.net/articles.rss

I wanted to keep this functionality with the new asp.NET MVC version, however somewhere along the way I decided it would be a good idea to not only expose these different formats through varying file extensions but also respond to the HTTP Accept header that get sent with the request. This behaviour, similar to the respond_to method from Ruby On Rails, would allow multiple versions of the content to be requested easily.

The implementation was simple and based on my brief reading of the Http Accept header description.

            // Pseudocode of the algorithm
            format = request.extension
            if exists(format) and haveViewFor(format)
              return content as format
            else
              read in list of values from Accept header
              sort the list of accept headers by quality first and original request order second
              return content as the first format in the list that matches an available view
            if no appropriate view found
              return content as html
          

By attempting to find any possible view it allowed me to later add any format I like (json for example) just by adding the relevant view and all the plumbing would wire up.

All seemed well in Firefox, and after forgetting about this code and finishing the rest of the backend off, browser testing was irrelevant.. right?... I'd not changed any css, html or JS... I pushed live.

Why did Safari looks so funny?

During testing I'd been working with firefox, which, if you request an extensionless url, appears to send the following accept header (according to the live headers plugin):

            Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
          

So in my algorithm you'd get:

1. text/html (default q = 1) - sorted before xhtml because requested first
2. application/xhtml+xml (default q = 1)
3. application/xml;q=0.9
4. */*;q=0.8

In this scenario the code looks for a template for html, finds one and returns (ignoring the rest of the list).

When I saw the output from porkandpaws tweet it was immediately clear that he was seeing the XML output instead of html. A quick run in debug mode gave the following accept header from Safari 4:

            Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
          

It would appear that if no extension is specified Safari will prioritise XML over XHTML. With application/xml requested before application/xhtml+xml and with both being assigned the default quality level 1 my algorithm assumed the client wanted the XML view and returned it!

The quick fix was to prioritise types containing 'html' before falling back to request order. This may not be a perfect solution, but lets be pragmatic, I'd only implemented this on a whim - no-one is actually using this site other than for the html or rss view.

Conclusion

To round up then, although overengineering and random play code is almost inevitable for the curious developer when working for themselves with no timescale, it doesn't excuse full testing and you can never rule out what any change, no matter how 'isolated', might effect.

---