As the common wisdom goes, the best way to fix a badly designed system is to design it right in the first place.

I know all to well just how much truth there is in that. The challenge is in telling when a design is bad in the first place, which is more difficult than it sounds. There are all sorts of metrics that can be captured during the development process, but none of them will yield any true indication of how good (or bad) the design actually is. Experience tends to be the only thing that is reliable, and even then it can often just be a “gut feeling” that something is wrong and incredibly difficult to identify what exactly that is. When a design is mediocre, it’s even harder since there is often an equal mix of good and bad making it all that more difficult to tell one way or another.

I’ve seen a lot of interesting things, a small handful of amazing things, quite a few questionable things, and even some amazingly horrifying things done in the name of design. I’ve had to dig deep into systems and their design to either work with them or fix them (usually fix them). Through that experience I have discovered that the design of a project can be summarized with one of two simple characteristics: one for good, the other for bad…

Good Design
Good designs produce additional benefits, features, and capabilities that the original designers did not conceive or plan, but that exist as a byproduct of the design. These capabilities can be leveraged with little or no change to the original design, or extend the original design without requiring extensive refactoring or redesign. New capabilities can be found in exceptional designs with nothing more than a different perspective rather than any change to the code or design.

Bad Design
Bad designs produce workarounds, special cases, constant (and often extensive) refactoring of both the code and the design, and are typically accompanied by long, involved explanations [from the designers] on why something can’t be done.

It really is as simple as that. The hard part is that it takes experience to recognize both of these and there is no magical way to get that experience other than to work on a lot of projects. That includes working on bad projects too. They might suck, but if you pay attention to what went wrong, and take the time to figure out why, even a bad project can become a gold mine of knowledge.

A Good Example – The QNX® RTOS
One system that I have encountered that would fall into the category of good design would the the message-passing inter-process communications (IPC) in the QNX® Realtime Operating System. I have worked with 3 generations of QNX, and although each generation enhanced the IPC mechanism, the “guts” of the QNX IPC remained the same because the fundamental design was good. They were able up add capabilities to new versions without changing the fundamentals of the original design.

Each and every version of QNX has the IPC mechanism at the core of the OS, for those interested in learning more check out the System Architecture Guide online. I will will focus on a few examples of how this single design feature of the OS demonstrates the characteristic I have described in this article.

One of the first features that QNX gets “for free” as a direct result of building the IPC into the core (yes, it is in the kernel) of the OS is that of modularity. The actual OS is made up of a group of processes that work together to provide the functionality of a traditional monolithic kernel. They also extend the IPC mechanism across the network, which has the side-effect of making the network completely transparent. This is an exceedingly difficult thing to accomplish in a monolithic kernel, and thanks to the excellent design of QNX’s IPC, it comes “for free”. A network of QNX machines is a loosely-coupled multi-processing computer, something that requires special software and application awareness on other platforms to accomplish.

The IPC mechanism allows the OS itself to be built of modules, and once again we see byproducts of this design choice. Multiple filesystems modules, multiple network links, device drivers, etc. Fault protection becomes a byproduct as well, since the IPC mechanism allows the OS to be built from multiple cooperating processes using IPC, each process is protected from the other, and the failure of any process does not have a catastrophic effect on the whole OS.

The simplicity of the IPC implementation also provided synchronization of cooperating processes, something that again, is often difficult to implement with other designs. Asynchronous communication was also possible without changing the IPC mechanism, and only changing how an application made use of the IPC features.

In later versions of QNX, a graphics system was added which QNX called Photon. Photon also leveraged the IPC mechanism to simplify the implementation of graphics drivers, input devices, and “clipping” of windows as they overlap. Photon itself enjoyed its own “freebies” as a direct result of layering on top of the IPC mechanism. Remote consoles, mirrored consoles, multi-monitor support, foreign platform support (Phindows – Photon in windows & PhinX – Photon in X) all became free or trivial with the design.

These represent just a few of the additional features that just “fall out” of a single good (in this case excellent) design.

A Bad Example – Generating SQL in an OO-System
This is an all too common scenario in object-oriented development. You start out with an hierarchical object design, and then create a corresponding relational database design in which to persist the object data. The typical approach is to create a data access layer (DAL) which consists of objects that essentially serialize object data into SQL, and deserialize from SQL back into objects. To keep things simple, especially when the OO-developer is designing the database, it is common to see a “one table per object” schema design.

The problems come when you want to retrieve a “complete” object which includes all its hierarchical child objects. The most intuitive solution is to have the DAL object call other DAL objects for the child objects. This is easy to do, and easy to test, and looks fine on the surface. Of course, it isn’t fine, and the first issue to crop up is one of performance. The performance issue is caused by iterating through an object, generating additional DAL / SQL calls, which in turn do the same for each level until there are no child objects remaining. For an object with lots of child objects, or lots of levels in the object hierarchy, this quickly results a large number of SQL queries, all for a single object. The issue is compounded when retrieving data for an array of objects.

Let’s take a step back to look at this. The design isn’t very good, as it fails to resolve the issue of making an object hierarchy work well with a relational database. The design is simple, which would normally be good, but the simplicity results in iteration and an increased number of round-trips to the database server. Neither of which could be considered a beneficial byproduct of the design.

Typically this regarded as a performance, rather than a design issue and a workaround is often introduced to improve the performance. Typically the DAL layer will get a “depth” parameter added to limit the number of levels we retrieve in the hierarchy since we also discovered that we don’t always want the entire hierarchy. Every single DAL object must now be modified to use and update our depth-tracker. When we are all done, it works, but performance is still poor, because we have only reduced the iteration rather than eliminating it.

This process can continue indefinitely, workarounds usually address the symptoms of the real problem instead of the problem itself. They also tend to be very specific and targeted, increasing the likelihood that any given patch may introduce new issues and create new symptoms since the underlying issue has not been addressed. After a while, the number of patches can make identifying the root-cause more difficult.

One of the better solutions to the SQL issue outlined here is quite simple, stop thinking in OO, and start thinking in relations, which is what the database uses. Once you do this, it becomes obvious that you can take the most frequent requests, identify how many levels of objects are involved and write a single SQL query that uses join statements to get all the levels of data in one fell-swoop. This is one example where the differences between OO methodologies and relational are disconnected. What is obvious in one isn’t necessarily efficient, optimal, or even a good idea in the other, and may even be contrary to the other methodology.

Each step taken in this bad design is relatively simple, and appears to be the most intuitive at the time, making it extremely easy to wind up with a design that has issues. The symptoms often masquerade as a different problem making it difficult to realize that the design contains a flaw rather than the implementation. This is quite typical in the absence of requirements forcing an iterative cycle of partial design intermixed with coding in place of upfront design and understading of how each element of the solution will be implemented and used.

This is perhaps why this particular example (mapping object to relational) comes up so often, and why it so easy to take such a course without knowing it.

As you can see, in the bad design example, we got no beneficial byproducts from the design, and instead got a series of issues and problems requiring more and more workarounds to address.

I often hear statements in the form of “X is cheap, don’t worry about it”. This usually comes up in design sessions, software or infrastructure, it doesn’t matter, you can usually find or hear a statement in this form in either type of design session.

The most common versions would be in reference to disk space, bandwidth, and CPU.

I suppose this type of thinking is a result of the effects of Moore’s Law. One of the troubling things about this sort of thinking is that it is frequently wrong and is indicative of someone with a closed mind, an inability to see the big picture, lack of insight, short-sightedness, or any other similar phrases. In all cases, when someone makes an X is cheap statement it’s a safe bet that they haven’t really thought things through. These statements are very dangerous, particularly if you happen to be in a design session when someone says this and is of particular concern if that someone is a team leader or manager.

The most dangerous aspect of these statements is that on the surface they may appear to be accurate and correct. Unfortunately, unless we are designing as system that people will only look at (ie. the surface), rather than actually use, it is rarely correct.

I’ll make an attempt to examine the most common of these statements to show how and why they are incorrect and assumptions. The analysis will be from a business/enterprise perspective rather than a personal/consumer one. You might be surprised to see how inter-related these can be! Let’s start with:

Disk is cheap, don’t worry about size

I will have to start by conceding the fact that disk storage has gotten very cheap indeed. The average consumer can put a terabyte of disk into their PC for less than $200. So how can that NOT be cheap?

We first need to understand what is really being said here. This usually isn’t a reference to disks and storage in general, it will be in reference to data files or databases.

So lets talk about files… in the past files weren’t that big, they were text files or word documents, and although the word documents were bigger, the size was still reasonable. This is no longer true, text files are rarely used anymore, the same information is now in a PDF file, or word document, an image, or a multi-media file. Those PDF and word files usually have graphics in them too, and to ensure that they look good when they print the graphics are stored at full resolution, even if they are only displayed thumbnail-sized in the document itself. The average document size is now measured in MB instead of KB, and for media files, GB is becoming more and more common.

So, what am I trying to say? The amount of storage that is available at a reasonable cost has increased by orders of magnitude, but so has the average size of the files that we store. The net result is that we can store about the same number of files as we used to before.

Even if we can store more files there is another cost involved. The more files and directories we have on our disks, the more difficult it becomes to find a file when we need it. If a user can’t remember where they put a file on a file server and they have to search for it, that potentially ties up a lot of I/O and CPU resources as the server scans all the files. That results in slower performance for all the other users relying on that server. That same server could be hosting a database, or a VM, or even just the virtual disk of a VM, and each of those would be affected by this search. Even if you don’t search for files very often because you are so well organized… those files are getting scanned anyway a couple times a day by your anti-virus and anti-spyware software.

Now lets look at databases… Modern database servers make them easy to use, quick to access, all those good things. So where is the problem. Let’s look at how a real production database gets used. To start with, you don’t just put a production database on disk, that’s too risky, disks can fail. You put it on a raid array instead, using mirroring, or striping, or whatever. The end result is that we now have two or possibly three (depending on the level of raid) copies of that database. We need to back up the database, that’s another copy… but in today’s web-enabled global economy, we can’t take the database server offline to back it up. So we have to take a snapshot to another disk so that we can make a backup of it without it changing while we write to tape, that’s another copy. Add some developers into the mix, any system that has a database probably has some developers working on new versions of the software and they need working copies of those databases to use for testing and development. That’s another copy (per developer) for each database. What are we up to now? 5 or 6 copies? Once again, the way we use that so-called cheap disk space quickly changes the cost per MB when each MB that we use is actually stored multiple times.

Backups become important too. Bigger databases take longer to backup, which also means they take longer to restore too. That can be crucial when the database for a web-commerce site goes down and we need to restore it. The companies that chose to believe the “disk is cheap” myth will be offline waiting for the restore a lot longer than the company that chose to use their disk space more wisely.

Storage in general is no longer local in a business environment. Files get shared on file servers, as well as through the use of SAN and NAS devices. This means that accessing the data on those “cheap” disks is now over a network and now bandwidth enters into the equation. A perfect segue into…

Bandwidth is cheap, don’t worry about the size of your data

Once again, we start with trying to understand what is really being said here. This is typically the speed of the available network rather than the cost of the network, although it can easily refer to both.

The speed and cost of networks has improved dramatically over the years. They have gone from 300 baud modems to 1200, and on and on up to 14400 and 56Kbps only to be replaced by broadband connections anywhere from 128Kbps to 10Mbps. Local networks went from proprietary systems to 10Mbps Ethernet, then 100Mbps, and now 1Gbps or wireless networks that started at 11Mbps and are now 54Mbps or 108Mbps.

What was once inconceivable to transmit over a network is now commonplace. So where is the problem here?

The first misconception comes from a single user perspective of the network. This is easiest to explain with an example. A typical office will have a 100Mbps network, providing 100Mbps of network bandwidth to each user. Sounds pretty good and we don’t use hubs anymore everyone uses switches, so user A doesn’t get impacted by what user B is doing on his segment of the network. Or does he? That depends, although in most cases the answer is yes, they are affected. Those network switches only isolate traffic between two end points. So, I might have 100Mbps between me and the switch, but ultimately I don’t need anything from the switch, I need it from some resource on the network. That resource might be a file server, or a database, or a website. There is a significant chance that I’ll be competing with the other user to connect to the same end point. So while we both have 100Mbps to the switch, we are sharing the 100Mbps from the switch to the common end point, so we’ve effectively been reduced to 50% of the “perceived” bandwidth. Chances are that it won’t be just two users connecting to that common end point, but a lot more. In a small company, you are probably competing will ALL the other users, so divide that 100Mbps by 10 or 20, bringing us down to 10Mbps or 5Mbps, certainly not the amount of bandwidth that you expected to get.

I will admit that I’m not being completely fair in my calculation, its far to primitive to be accurate, everyone would have to be accessing the same server at the same time for those numbers to be correct. Statistically we will get much better performance, exactly how much better is harder to predict since it depends on what is being accessed. Of course, the bigger the files we are after, the longer we send transferring their data, the more likely we are to be affected. This is where we can start to see the impact of the “disk is cheap” mindset on other (seemingly unrelated) areas of computing.

So, I haven’t proven anything other than you won’t see 100% of your bandwidth. We’ve only been talking about an typical office LAN, lets examine networked applications.

There are two typical varieties of networked applications. Those that explicitly use the network by implementing some form of protocol, and those that indirectly use a network through file sharing, web-services, SOA, or database connections. If a protocol is designed with the “bandwidth is cheap” thinking, you’ll probably find that it sends and receives lots of data. The networks are indeed fast enough to make this appear to be a non-issue. Unless you happen to be the the sysadmin or the hosting provider. More efficient use of bandwidth translates directly into more users, which translates directly into revenue. Once you max out your bandwidth with users, you might also need to buy/build new servers to handle more customers.

The more customers you can fit into the same bandwidth, the fewer servers and infrastructure you need, which directly affects the capital investment required (which again, affects Time-To-Revenue, since you have to recoup the infrastructure cost to become profitable).

More and more often you will find servers running in VMs, which wind up aggregating network traffic from all VMs to a single physical network. Applications running in VM servers that don’t make efficient use of the network could find the VMs running out of available network bandwidth before they run out of available CPU. If we have network traffic between two VMs, it never hits the physical network, but it does wind up consuming additional CPU possibly impacting performance of the other VMs.

When we introduce wireless and cellular networks to the mix the impact is more easily observed. Sure that new 802.11n wireless is fast, but when you have an office full of users streaming video over the same shared wireless bandwidth…

So if you encounter someone that suggests “X is cheap, don’t worry about it” in your next design session or troubleshooting discussion, you will know that what they are really saying is that they probably don’t understand the issue at all.

A few weeks ago I decided that I needed to learn more about the BlackBerry from Research in Motion (RIM). So I made a call to a friend that works for one of the local cell providers and asked if he could hook me up with a demo device for a couple weeks. A few days later and I had a nice red BlackBerry curve and a SIM card to use.

To be fair, I was already certain that I wasn’t the BlackBerry personality. I’m a geek, as long as I can get to a computer, I can get to my email, I certainly didn’t need it clipped to my belt. I was wrong. I now understand the addiction of the “CrackBerry”, it will be hard to give the demo unit back, really hard. It was a Berry good experience, withdrawl is going to be rough.

Setting up email was a breeze, so was adding extra email accounts for Gmail. Facebook was already on the device, as was Yahoo messenger. There was no MSN or AIM clients, but that was quickly remedied with a download followed by a painless install. The web browser was okay, there is only so much that can be done with the screen resolution, and the browser had several viewing modes to help with this. I did try Opera Mini and it ran fine, and behaved like a desktop, but due to the screen size it too wasn’t perfect. Both were more than adequate though, and definitely better than the average cell phone. Google maps is so cool on a GPS-equipped mobile device that it was almost spooky and the navigation applications that I trialed were extremely good as well. The sensitivity of the GPS receiver is excellent, in some locations it outperformed my handheld GPSr unit. The keyboard is far more functional than I ever imagined, accessing punctuation and symbols are trivially easy and the “smart” behaviors really improve input speed rather than hinder it. It is so good that I am writing and composing this post on the BlackBerry!

Now, about the addiction that turns this from a BlackBerry to a CrackBerry. The blinkin’ light. Yup. That’s it. The device sleeps aggressively but wakes up instantly when you press a key and even in sleep mode it can receive email and instant messages. Which is where the blinkin’ light comes in. You fire off a message and set the device down, it sleeps quickly and the screen goes black making it easy to ignore knowing that you don’t need to check it. Then out of the corner of your eye, in your peripheral vision, you see the the light blink. That light makes it easy to multi-task messaging into whatever you are doing, even less obtrusively than running an instant messenger or email application on a desktop. For me, that was the hook, that little light.

So, sure as a geek, I can get to my email from just about anywhere and I don’t need it clipped to my belt. The BlackBerry let’s me know when I need to get to my email without the bother of having to check it first and it enables me to fire off a quick response if I want.

It does other things too, the camera is okay, better resolution than most camera phones, and very easy to use. It plays MP3s as well, and you can expand the memory by adding a micro SD card (although, it took me several tries to figure out how to insert the damned card). Once you have a card in there, it behaves as you’d expect, pictures and media default to the card instead of device memory, which is what most users would want. Loading up music or videos was easy enough too.

Battery life was much better than I had expected for this type of device, and charging was via a standard USB cable instead of some custom connector. If you plugged in to a USB port that was not able to provide enough power to charge the device it tells you immediately so that you don’t wind up plugging it in overnight only to find that the battery is still low in the morning.

Apple isn’t the only one that can get usability right, it would seem that RIM knows a thing or two about it as well.

Now I want one, damn it!

I have a confession, I’ve fallen victim to the power of Steve Jobs’ Reality Distortion Field. I’ve been glued to the live “blogcasts” of his keynotes from MacWorld, Apple WWDC, and the special event announcements/presentations. I don’t even own a Mac! …but now I know I want one.

Now that I have that off my chest…

I was extremely pleased as I watched the keynote from the September 5th Apple Special Event. What pleased me the most was confirmation of some of my previous musings by the announcement of the new iPod Touch. This device confirms what I suspected, the “touch” is the first of Apple’s embedded hardware platforms. When the iPhone was announced, it was the fact that this platform was running OS X, enabling Apple to leverage a self-hosted environment in Mac OS X. This has been a competitive [and strategic] advantage that QNX enjoyed in the past. The iPod touch is using the same platform and derives the same benefits.

Apple has proven they have the ability to leverage this platform, I think we are going to see some interesting products emerge from this more rapidly than anyone expected. The iPod touch was announced only 3 months after the release of the iPhone, that is an insanely short period of time between products. It also means that in order to even get the device manufactured in time for the announce availability, that the hardware had to have already been in manufacturing when the iPhone was announced.

Self-hosted development is a huge advantage that Apple now has over its competition. They can develop new applications on desktop mac computers long before any target hardware is even available. This improves time to market tremendously, a key advantage in the cut-throat consumer device markets that Apple is playing in. With an embeddable OS X as the core, handling all the hardware differences behind standardized APIs, and the ability to re-use application from previous products, this is a killer environment. Developing and debugging a WinCE device (which are the guts in many iphone competitors like the HTC TyTn, Samsung Blackjack, Motorola Q, as well as devices from Audiovox, Symbol, HP and others) is a painful experience. Much of the pain experienced with the WinCE development model is all the time spent waiting to deploy a new build of an application to the device, it’s slow, even when on a fast/powerful developer workstation. Sure there are emulators, but they are no faster when deploying or debugging an application, and networking (wired and wireless) doesn’t work at all in the emulator. A developer at Apple will be able to develop an application for the “touch” platform in easily one half or one quarter the time required for a similar app on a WinCE smartphone.

I don’t think the rest of the embedded world has much to worry about from Apple though. They aren’t likely to be sharing their hardware platform with anyone else. They might need to be worried if they are making consumer devices and Apple decides to enter their market.

I think we are going to be seeing some interesting products from Apple in the future some of which are sure to take them in directions that will be quite unexpected from the outside.

I’ll be staying turned and keeping an eye on Apple, I suggest you do the same.

I’ve been thinking about Apple recently, just like everyone else many thoughts have been provoked by the upcoming iPhone release as well as the latest beta release of Safari. I don’t own a Mac, and I’m not a Windows fan either, my background is in embedded systems and software, I run QNX, Linux, and Windows on my network.

The most interesting things that I have noticed seem to have been missed by the mainstream, or at least I haven’t seen or heard a lot of commentary on the things I’ve noticed.

Embedded Apple
The first thing that struck me about the announcement of the iPhone was the mention that it is running OS X. The iPhone is certainly not running an x86 processor, or even a PPC one, it’s closer to the iPod than it is to a Mac, and the iPod uses an ARM processor. Apple has not officially revealed which CPU is powering the iPhone, although this ars technica article is rumored to be correct.

Which means that Apple has a version of OS X for ARM. That’s a big deal. A really big deal. ARM is one of the predominate CPUs for low-power embedded systems. Apple now has OS X running on three CPU families: PPC, x86, and now ARM. To make this even more significant, is that the development environment is self-hosted. Developers (at least the one @ Apple) can code, develop, build, test, and debug their applications on their Macs, and once they have it working, and assuming they’ve allowed for the difference in pixel resolution, can recompile for the iPhone and have a running application. Of course this doesn’t have to be for the iPhone, the Apple division that is responsible for developing the iPhone is actually a consumer products group, and the iPhone is merely their first product. There are countless consumer devices that Apple can build using the ARM processor and their new embedded OS X.

Yeah, I’m making a big deal of this. Self-hosted development represents incredible productivity when developing software for embedded platforms. Not only has Apple boldly entered a new tier of consumer devices, but it has done so by equipping itself with a huge time-to-market advantage over it’s competitors.

Carrier 180°
Apple will have done something remarkable when they release the iPhone, and surprisingly, it has nothing to do with the iPhone itself. The market dynamic between wireless carrier and handset manufacturers has been distorted by the carriers for many years. Typically the carriers will dictate to the handset manufacturer which features they want in a handset and specifically how they want to be able to control those features. This is the concept of a “carrier lock” on a phone. Consumers don’t like it, but by providing the ability to control features and/or lock them to the carrier allows the carriers to subsidize the handsets since they have a reasonable guarantee of revenue stream from the device. Control of this revenue is huge factor in how much a carrier will be willing to subsidize a handset. It’s quite common for carriers to offer some handsets at ridiculously low prices or even for free with fixed-term contract.

Apple turned this dynamic around 180° on AT&T when they chose them as the provider. Steve Jobs knows that much of the success of Apple products is a direct result of the “Apple experience”. The engineering of the UI usually makes an Apple product much easier to use than the competition, and when combined with the elegant and sleek physical designs Apple is known for, the result is very high customer loyalty. That loyalty makes Apples customers quite willing to pay a higher price, a nice payback for the design effort. So, it should come as no surprise that the experience is paramount with the iPhone as well. If Apple were to allow the carriers to dictate what features could be controlled, it would negatively impact the experience. As an example, in the traditional model, many carriers might restrict the WiFi capabilities to file transfer, and prevent its use for VoIP applications to protect their revenue stream from airtime.

With the iPhone, Apple calls the shots, and leaves everything enabled. This too is a big deal, it is a complete 180° change in the dynamic between device/handset manufacturer and carrier. The effects of this change are much larger than the effect of the iPhone in the handset market. I’ll definitely be watching that to see how it turns out.

Windows Target
The last thing I want to mention is in regards to the beta release of Safari for windows. Ignoring the bugs and other issues that numerous others have found and discussed, what caught my attention was how much of the Apple GUI was present. I will openly admit that I don’t know a lot about the application development environment on the Mac, so I could be dead wrong on my interpretation here.

Jobs has talked a lot in his keynotes about core animation as a key feature in OS X, which I found interesting in the context of Safari. Now, I know that Safari has little use for core animation when rendering web sites, since web sites don’t do anything with it. When I went to a site that was password protected though, the dialog box that appear for the login credentials had some slick 3D animation as appeared and disappeared. Certainly looked like core animation to me, but I could be wrong. Others have noted that the font rendering in Safari differs from Windows. I won’t get into the debate over whether it is better to honor the font design or the pixel grid, that isn’t the important part of this. The important part is that the graphics rendering in OS X is display postscript, of which font rendering is a component. Does the presence of the Apple-style font rendering indicate that the GUI API includes Apple’s display postscript too? To me, the (apparent) presence of core animation and the Apple graphics render engine are significant. Display postscript provides an excellent hardware abstraction layer for graphics rendering, the presence of Apple’s DP in a Windows application would mean a low barrier for ALL Mac OS X graphics on Windows. I really have to wonder how much of the Mac OS X GUI / Windowing API is now available in a DLL for Windows.

As I mentioned previously, this is the one thing that I have less confidence in due to my lack of knowledge of the Mac OS X application development environment. If I am right though, the existence of core animation and the Mac OS X GUI API & render engine on Windows makes the possibility of windows as an cross-dev target quite high. Apple already has a universal binary that can carry PPC and x86 binaries simultaneously, it wouldn’t take much to extend this to also include x86/Win32 as well.

Tomorrow’s Apple
All of these things together have Apple well positioned to dramatically change the landscape for embedded device/application development as well as windows application development. The appeal of being able to produce Mac and Windows software with only a recompile would be a strong incentive to bring significant numbers of developers to the Mac platform. Could tomorrow’s Apple be the next Windows dev platform? I don’t know, but it seems to be within the realm of possibility, and that will make the future of software development very interesting indeed.

I’ve doing a lot of thinking on the topic of whether you can or can’t mix object and relational models, what I have discovered has been quite enlightening. In my previous posting, I determined that the same data hierarchies can be represented in either an object orrelational model. I also determined that you can mix the models (why not?), but that the trick is not in the mixing but in the realization that the model you have is a mixed one. The sad realization was that most people do mix the models and are completely unaware of having done so, and I needed to figure out why (and how) this was happening.

The use of IDs in objects is where the models typically mix, and also where most people fail to realize that a mix has taken place. Seems that the concept of a simple integer id incorporated into an id is a seductively powerful concept. I won’t claim to completely understand why this is, but I will share with you my thoughts on why the id has such seductive power that even the die-hard OO zealots fall victim to it’s allure.

For the older generations of developers (of which, I belong) that learned to program with the original high-level languages and who have experience with relational databases, the use of an id is quite natural. We did not typically design our data models using hierarchal data structures, we thought in terms of records, and if we designed any complex data model, we designed it in a relational database. You quickly learn to use ids to link tables together, and since we started out with more resource constrained systems, the appeal of using 3NF relational models compliments our awareness of the amount of resources our applications consume. When your code is written to deal with data as arrays of structures, it is much, much, easier to write functions (we didn’t call them methods) to manipulate the data if you could just pass an id for the record around, or perhaps a pointer rather than moving the entire structure.

The basics of writing programs typically include the use of arrays, and eventually arrays of data structures, which are usually our first forays into more advanced programming tasks involving large amounts of data. When you use an array, the index of the array becomes a “natural” id for a data record. The same is true of anyone learning an OO language, the basics still involve learning about arrays. It’s a quite easy to adapt from an array index to an id column in a relational database. The use of those ids inside a data structure that references another data structure works very well. The code itself might use a pointer, but you can’t write a pointer out to disk and then read the same data back again and still have the pointer be valid. If it’s an id, you can… they fit perfectly into the relational model when using an relation database.

Of course even a C programmer can create hierarchical data structures using pointers, linked lists, and such. In the end though, when we need to persist that data, the ids are convenient.

So what about the object world?

I believe that in a “pure” OO model, the ids do NOT belong. The represent a direct coupling of the persistence layer and the object layer, which is considered a violation of most OO methodologies. There is a problem with object models though, which is that a complete object hierarchy graph can actually represent a lot of duplicated data. The exact same reasons why we use 3NF in relational models applies here too, it takes a LOT of effort to ensure that all the copies of a child object get updated in all the in-memory object graphs. In C you’d just use pointers to “solve” this issue, but a pure object model does not expose such a low-level capability, which is where the use of an id comes into play. The id effectively becomes an abstract pointer to another object, eliminating the need to include the entire child object, while still allowing an easy way to access the data in the child object. Only populating the id of a child object becomes a convenient solution for “lazy loading” our object graph. The other issue is that there are very few object databases, so the vast majority of the time you’ll be using a relational database for the persistence layer.

So the id winds up being used as an compact, convenient alias for a child object, providing us with a convenient solution to “lazy loading” object graphs, and allowing us to pass an id as an argument to a method in place of passing the object itself. Convenience aside, this violates the pure OO methodology and introduces coupling not only between the object and the persistence layer, but depending on usage may also create tighter coupling between objects and external methods the manipulate them.

The conclusion is that the seductive quality of the id is convenience, and for most developers convenience trumps following the purer object models. In fact, this convenience is so seductive that most OO developers don’t even realize that using an id is mixing object and relational models.

Truth be told, creating a “pure” object oriented system is hard work, really hard. The relation model fits very closely with how a compiler has to actually implement the OO concepts in our HLLs, which means if you design and use your objects in a more relational way, you’ll wind up with a program that is more likely to perform much better than a pure OO system. The CPUs that eventually execute our code (which still has to occur even when we use languages that “execute” in virtual environments like JVMs and the .Net CLR) perform better with relational models than object ones.

I was once told by another developer that the design that they always used and found to work very well was to use a one to one mapping of object to database table. Truth be told, this is NOT an object model, it’s a relational one… if you can perform a 1:1 mapping from your objects to relational database tables, then you have a relational model, if you had an actual object model, it would NOT map 1:1 to a database. The reason it worked was not because they had implemented an OO system properly, and not because they managed to avoid mixing the two models. The did manage to avoid mixing the models, but they used a pure relational model, and not an object model. As I mentioned before, the pure OO model is very difficult to do properly, especially when you must provide a persistence layer that has to deal with a relational data model in the form of a database.

This performance aspect is another influence in the seductive power of ids in our objects, their use can help improve performance of a slow OO system by a significant factor.

So how would you implement something like lazy loading in a purely object model? I believe the answer is inheritance, one of the core concepts/capabilities of an OO system, and probably the most abused, misused, or neglected part of an OO language. People either go crazy with so much inheritance that you wind up with unusable code, and insanely deep / complex class hierarchies, or they ignore them all together and use inheritance as an overly complex type enum.

I think a proper object data model would have a family of objects, where the “top” levels always represent the lazy load state. This level would never contain complex objects, no arrays of child objects, and only a single level of child object (ie. the only child objects included should not contain other objects, only simple types). The various levels of “heavy” loading would be accommodated by additional objects that inherit from top level and then add their own members any additional levels of data.

Not all OO languages make this easy to do. To make it easy, you’d have to have methods that allow either the base (lazy) object or the inherited object (full object) as an argument. You can do this in C#, but you have to use casts (which is a PITA, and definitely not convenient). You don’t get this functionality unless you use an interface and inheritance, which requires a lot more up front design and planning to get into a design early enough to produce benefits.

Welcome to the dark side of Objective Relations… the id is your mistress from the relational side.

I’ve been working on a system written in .Net/C# that uses SQL Server, my database experience prior to SQL Server included Oracle, Ingress, Sybase, Postgresql, and some MySql. Each had their quirks, but I’m learning more and more of the SQL Server quirks, and at times seriously having to wonder how Microsoft gets off calling this an enterprise class product. I will fully admit that some of my complaints are with how .Net communicates with SQL Server, and these probably are not Sql Server’s fault.

Here is my short list of gripes (in no particular order):

1. No Sequences
   Yes, I know, you can use IDENTITY columns, but they are not the same, and SQL Server places some foolish restrictions on their use. Instead of an IDENTITY column being a “normal” scalar column it is special and you can never update the column, although you can insert, but only after additional SQL directives. The IDENTITY implementation is close to a sequence, but Microsoft just didn’t implement it quite right. I do wonder why they didn’t introduce proper sequences with SQL Server 2005, they could have done this quite easily, a proper sequence could peacefully co-exist with the existing IDENTITY nonsense.
2. Inconsistent Parsing
   I was creating query that needed to run properly against two versions of a table, one version had an additional column to simplify the query. Using the IF EXISTS statement, I wrote a query similiar to the following:
   IF EXISTS (
SELECT NULL
FROM sysobjects so, syscolumns sc
WHERE so.id = sc.id
AND so.name = 'table_name'
AND so.xtype = 'U'
AND sc.name = 'column_name'
) SELECT column_name FROM table_name
   When I ran it against a database that had the column it worked, when I ran it against one that didn’t have the column, I got an error of it complaining that column_name didn’t exist. I shouldn’t have even tried, that is the whole idea behind using the IF EXISTS… The following query:
   IF EXISTS (
SELECT NULL
FROM sysobjects
AND name = 'table_name'
AND so.xtype = 'U'
) SELECT * FROM table_name
   Works as expected when run against a database that does not have the table.

   So, in one case it parses the statement inside the IF EXISTS and does some checking, in another case it does not.

3. NOCHECK Constraints
   The ability to temporarily disable constraint checking can be quite convenient, particularly while bulk loading data. What I don’t like about this is that this “WITH NOCHECK” state can actually stick around indefinitely, and is included in backups of the database. The constraint appears to be present, but is not actually being enforced.
4. Escaping Reserved Words & Special Characters
   This is one of those things that looks reasonably useful until you actually take a couple moments to think about it. SQL Server will let you use special characters like spaces, periods, and even reserved words when defining your schema, all you need to to is enclose them in square brackets. The following is valid:
   CREATE TABLE [Table] (
[hard.to.use.column] varchar(20),
[also hard to use] varchar(10),
[null] varchar(10)
)
   There are good reasons why we have reserved words, and why spaces and such are not allowed in table or column names. While it is admirable that Microsoft provided such a convenient way to differentiate the real reserved words from the ones we want to use, it’s horribly misguided. No one should do this, ever, it’s just plain confusing, not to mention non-portable, foolish, and, oh yeah… stupid.
5. Inconsistent context from .Net/C#
   I will admit that I do not know if this is a SQL Server thing, or a .Net/C# thing. This is related to the debate about using parameters or not (which I won’t get into), the issue is that .Net/C# does not execute these in the same way. Here is a code sample to help demonstrate this:
   public Query1( SqlConnection cnx, int id )
   {
   cmd = cnx.CreateCommand();
   cmd.CommandText = String.Format( "SELECT * FROM myTable WHERE id = {0}", id );
   SqlDataReader r = cmd.ExecuteReader();
   if( r.Read() ) { /* read result */ }
   r.Close()
   }

   public Query2( SqlConnection cnx, int id )
   {
   cmd = cnx.CreateCommand();
   cmd.CommandText = "SELECT * FROM myTable WHERE id = @id";
   cmd.Paramaters.AddWithValue( "@id", id );
   SqlDataReader r = cmd.ExecuteReader();
   if( r.Read() ) { /* read result */ }
   r.Close()
   }

   
   Query1 will be sent to SQL Server as a normal query. If you were to build another query in the same program it would execute in the same context. In other words, you can do a transaction by issuing the raw SQL as multiple statements using a combination of cmd.ExecuteReader() and cmd.ExecuteNonQuery(). You could also issue the sql to allow IDENTITY INSERTS, and then loop through some in-memory data and create multiple INSERT statements from that data.

   Query2 should be identical, but it is not. It executes in it’s own context as a side-effect of the fact that a stored procedure is actually used to execute the actual query. The parameters are defined and their initial value set as arguments to the stored procedure, with the actual parameterized query being passed un-altered from .Net to the stored procedure as one of the arguments.

   While it seems a bit excessive to invoke a stored procedure to deal with the parameters, this is not what bothers me. What bothers me is that a parameter-less query and a parameterized query are not handled in an identical manner by .Net/C#, especially with the nice side effect of the difference in execution context of the two. Code that produces valid SQL without parameters can break without explanation simply by implementing parameters.

   Another issue related to this is that .Net/C# appears to always parse the sql before deciding how to execute it. In an attempt to reduce potential errors where you might build a query with undefined parameters, it actually makes it impossible to create a stored procedure or user defined function from C#. Since these things will use parameters in T-SQL, they conflict with how C# handles parameters, the following code snippet will produce a .Net exception:
   cmd.CommandText = "CREATE PROCEDURE foo @id int AS SELECT * FROM myTable WHERE id = @id";
cmd.ExecuteNonQuery();
   The exception will be that you have not defined @id with a cmd.Parameters.AddWithValue() statement. Of course, you didn’t do that because you are not trying to use a parameter, you are issuing DDL SQL to create a stored procedure that will use one. I have yet to find out how to tell .Net/C# to stop doing this so that this SQL can be executed. I means you can’t programatically create a database schema from within a C# application.

6. Column Aliases not Available to ORDER BY
   This one should be a bug, the fact that if you define an alias for a column you should be able to use that alias in the ORDER BY clause. You can’t. This is particularly annoying for a calculated column or one that transforms the data using CASE or by concatenating text results. You wind up having to duplicate the calculation in the ORDER BY clause. Silly.
7. Default Collation Sequence is Case-Insensitive
   When you install SQL Server (2000, MSDE, 2005, 2005 EE) the default collation sequence is case-insensitive. I’d never seen this before when working with other databases. I only discovered it by accident when looking at a client’s data and discovered that data being inserted into a column with an FK constraint was allowing lower-case when I knew the referenced table was all upper case. I was shocked. This might make some sense for some other alphabets, but it certainly produced some unexpected results for me.

   You can actually create each table and column with a different collation sequence. This might be flexible, but I suspect it breaks several rules of design and should be avoided.

So there you go. Some of the reasons why I dislike SQL Server. I’m sure I will think of some more… I’ll add some of them to the list as I think of them.

I have often heard people say that you can’t mix object models and relational models. I have to admit that it took a while for me to fully understand, and now that I do, I’m certain that most people that said it to me, didn’t. That sounds a little harsh and it probably is, but the people most likely to impart this morsel of wisdom probably have comp-sci degrees and learned how to program in an academic setting. They probably believe in “pure” programming methodologies as well, and talk about patterns, and anti-patterns, and extreme programming. Unfortunately, most are what I have previously described as fragile developers.

Anyhow, back to the topic at hand… mixing object and relational models.

There really isn’t that much difference between object and relational models. In fact, if you were to look at an entity-relationship diagram for a relational database and a object model diagram from a distance you probably wouldn’t be able to tell them apart. The truth is that I have yet to come across a data model that can’t be represented by either model.

Object models represent hierarchies, basically nested data structures that contain other data structures or collections of data structures, which in turn can be made up of similar hierarchies. You can represent the exact same hierarchies in a relational database, the main difference is that you need to add meta-data to represent the hierarchical structure, and this meta-data is visible. It usually takes the form of id columns and foreign key references. Technically speaking, the relational model is no different when you look under the covers. Those relationships appear as pointers, arrays of pointers, and linked lists, of course no OO-language worth using would ever let you see this, but it’s there and it’s the meta-data required to make the objects work. The object purists call this abstraction, and it’s a “Good Thing”, and I agree.

So where is the disconnect between object models and relational models? There really isn’t one. They can represent the same data. There are differences though. The object model is based on usage, the only reason to create an object is to use it, which is why we define objects with classes and those classes include methods, and properties for manipulating and accessing the data in the objects. A relational model doesn’t try to accomplish the same thing, it’s only goal is to store the data in an object hierarchy and provide a mechanism for retrieving that data to reconstruct the original object. Unless the object has no hierarchy, the mechanism of storing or retrieving the data will involve a sequence of steps to accomplish this task. This doesn’t make the relational model inferior to the object model, it just makes it different, remember the two models are not trying to accomplish the same goal.

Unless you are writing an application that never saves or loads any data, you’ve had to deal with object persistence (OO-speak for writing to disk). Chances are that for any application of any realistic size, that persistence involved a relational database (and a damn good chance that it’s a SQL database). This is where the first disconnect occurs. There is a mapping process involved, which will produce multiple queries to the database. The object hierarchy will be traversed and each level will produce at least one database query. It is easy to confuse this mapping between models with “mixing” the models.

I get it now, you can’t mix them, unfortunately most people DO mix them without realizing that they have done so, which is where the trouble starts. That trouble usually takes the form of id column in a database. You see there is a problem with the object model, which is that when you follow the “pure” rules, you always deal with the FULL hierarchy of objects, and those can get pretty huge. There is a tendency for object models to allow for much deeper hierarchies than the designer/developer ever intended to use. A common solution to this is to introduce the concept of a “lightly loaded” object, which in other words is a partial object hierarchy. We don’t want to load all the data for all the levels, but we don’t want to take away the ability to go from an lightly loaded object to a fully loaded one (or at least one loaded to a slightly deeper level in the hierarchy). Which means that we need to include some piece of data that can represent the deeper level and provide us with enough information to retrieve that data if and when we decided we need it. You want something you can use to reference that level, which is something that the relational model in our database is already doing for us, typically in the form of an id column.

It is far too easy to borrow the id from the database and use it in our lightly loaded object. This is where the trouble begins, since this IS mixing the two models. We’ve now taken meta-data from the relational model used to store/persist the data and we put it in our object. Our object now contains meta-data which the OO-language normally abstracts and keeps safely out of our way. Too bad we just messed that up, eh? There is a good reason that OO-languages abstract that meta-data away from your average developer, if they were allowed to see the meta-data they’d expect to be able to manipulate it, and then most of them would screw it up.

I’m not saying that you shouldn’t use that id from the database. It is really convenient, isn’t it? Very tempting. Beware though when you do, it must be done with complete knowledge that you just mixed the two models. You don’t have a pure object anymore, and all that abstraction that you created to prevent your object from having to know anything about persistence, well.. you just screwed that up, it’s too late your object not only has to have knowledge of the persistence, it’s exposing meta-data from your persistence layer. You object might even have to be “clever” to make sure that it manages these ids properly. It’s more likely though that you will create other objects that contain these ids without being part of your object hierarchy, you build peer objects that contain this data. Whups… now we are starting to represent the relational data in our object model, we’ve mixed them up even more. The trouble cascades from this point onwards.

So, you can mix the models, careful examination of most object models will contain a mix. This becomes a worst-practice when you let developer work with this model without the knowledge/realization that they are using a mixed model. They’ll run into issues (this is a classic example of what Joel Spolsky refers to as the Law of Leaky Abstractions), they’ll probably blame the relational database for some of there issues. They’ll be wrong. The only way to avoid these issues is to have full knowledge of the object model AND the relational model AND to be able to identify and understand where the two are mixed. Unfortunately it’s pretty rare to find a developer with this ability.

In the end it turns out that what you really can’t mix are developers with only partial knowledge of the two models, and that just isn’t the same as not being able to mix the models.

About a month ago, Mark Zaugg handed me a copy of this book, he’d bought it but had not yet had the time to give it a read. It was a very interesting read, quite unlike any other book I have ever read on a technical subject. The author (Stéphane Faroult) has themed the book on the classic The Art of War by Sun-Tzu, and included many quotes and examples of a historical military nature. That might seem pretty odd for a technical book, so you’ll have to trust me when I say that it works.

The book does provide some tips and techniques that can be used to tune SQL queries and to solve some common problems in some pretty non-obvious ways. After you read this tomb you will be better at analysing, understanding, and tuning queries, and you will have some new and arcane skills with SQL, but that is not the primary benefit this book delivers. Mark had told me when he loaned me the book that he had heard it would make you “turn your head sideways to look at SQL”. He won’t understand just how accurate that statement was until he’s read the book.

The author shares his knowledge of SQL and relational theory in a way that is incredibly compelling. You finish each chapter with new insight into how and (more importantly) why different queries perform differently. The examples that are provided are realistic enough to make sense and it is very easy to recognize how the information, analysis, and techniques would apply to your own projects. I now understand significantly more about relational theory, SQL, and I have indeed learned how to look at SQL with my head sideways. The knowledge the book imparts goes beyond this as well, it discusses good design of the tables in a database, and it makes you think about the design of the databases you might already be using.

I will probably pick up a copy of this book for myself, even though I have now read it, it was impressive enough that I think it should have a home on my bookshelf.

While I was fumbling around attempting to recover the few posts from my original blog, Trever managed to find that the Way Back Machine aka Internet Archive not only managed to index my site, but it had a complete copy of the original postings. I have transcribed them into the new blog, which was nice enough to let me back date them so that the appear in their original order.

Three posts have been recovered, hopefully they are still of some interest and relevance to others.

To Trev:
for( i = 0 ; i < 100 ; i++ ) printf( "Thank you!\n" );