As the common wisdom goes, the best way to fix a badly designed system is to design it right in the first place.

I know all to well just how much truth there is in that. The challenge is in telling when a design is bad in the first place, which is more difficult than it sounds. There are all sorts of metrics that can be captured during the development process, but none of them will yield any true indication of how good (or bad) the design actually is. Experience tends to be the only thing that is reliable, and even then it can often just be a “gut feeling” that something is wrong and incredibly difficult to identify what exactly that is. When a design is mediocre, it’s even harder since there is often an equal mix of good and bad making it all that more difficult to tell one way or another.

I’ve seen a lot of interesting things, a small handful of amazing things, quite a few questionable things, and even some amazingly horrifying things done in the name of design. I’ve had to dig deep into systems and their design to either work with them or fix them (usually fix them). Through that experience I have discovered that the design of a project can be summarized with one of two simple characteristics: one for good, the other for bad…

Good Design
Good designs produce additional benefits, features, and capabilities that the original designers did not conceive or plan, but that exist as a byproduct of the design. These capabilities can be leveraged with little or no change to the original design, or extend the original design without requiring extensive refactoring or redesign. New capabilities can be found in exceptional designs with nothing more than a different perspective rather than any change to the code or design.

Bad Design
Bad designs produce workarounds, special cases, constant (and often extensive) refactoring of both the code and the design, and are typically accompanied by long, involved explanations [from the designers] on why something can’t be done.

It really is as simple as that. The hard part is that it takes experience to recognize both of these and there is no magical way to get that experience other than to work on a lot of projects. That includes working on bad projects too. They might suck, but if you pay attention to what went wrong, and take the time to figure out why, even a bad project can become a gold mine of knowledge.

A Good Example – The QNX® RTOS
One system that I have encountered that would fall into the category of good design would the the message-passing inter-process communications (IPC) in the QNX® Realtime Operating System. I have worked with 3 generations of QNX, and although each generation enhanced the IPC mechanism, the “guts” of the QNX IPC remained the same because the fundamental design was good. They were able up add capabilities to new versions without changing the fundamentals of the original design.

Each and every version of QNX has the IPC mechanism at the core of the OS, for those interested in learning more check out the System Architecture Guide online. I will will focus on a few examples of how this single design feature of the OS demonstrates the characteristic I have described in this article.

One of the first features that QNX gets “for free” as a direct result of building the IPC into the core (yes, it is in the kernel) of the OS is that of modularity. The actual OS is made up of a group of processes that work together to provide the functionality of a traditional monolithic kernel. They also extend the IPC mechanism across the network, which has the side-effect of making the network completely transparent. This is an exceedingly difficult thing to accomplish in a monolithic kernel, and thanks to the excellent design of QNX’s IPC, it comes “for free”. A network of QNX machines is a loosely-coupled multi-processing computer, something that requires special software and application awareness on other platforms to accomplish.

The IPC mechanism allows the OS itself to be built of modules, and once again we see byproducts of this design choice. Multiple filesystems modules, multiple network links, device drivers, etc. Fault protection becomes a byproduct as well, since the IPC mechanism allows the OS to be built from multiple cooperating processes using IPC, each process is protected from the other, and the failure of any process does not have a catastrophic effect on the whole OS.

The simplicity of the IPC implementation also provided synchronization of cooperating processes, something that again, is often difficult to implement with other designs. Asynchronous communication was also possible without changing the IPC mechanism, and only changing how an application made use of the IPC features.

In later versions of QNX, a graphics system was added which QNX called Photon. Photon also leveraged the IPC mechanism to simplify the implementation of graphics drivers, input devices, and “clipping” of windows as they overlap. Photon itself enjoyed its own “freebies” as a direct result of layering on top of the IPC mechanism. Remote consoles, mirrored consoles, multi-monitor support, foreign platform support (Phindows – Photon in windows & PhinX – Photon in X) all became free or trivial with the design.

These represent just a few of the additional features that just “fall out” of a single good (in this case excellent) design.

A Bad Example – Generating SQL in an OO-System
This is an all too common scenario in object-oriented development. You start out with an hierarchical object design, and then create a corresponding relational database design in which to persist the object data. The typical approach is to create a data access layer (DAL) which consists of objects that essentially serialize object data into SQL, and deserialize from SQL back into objects. To keep things simple, especially when the OO-developer is designing the database, it is common to see a “one table per object” schema design.

The problems come when you want to retrieve a “complete” object which includes all its hierarchical child objects. The most intuitive solution is to have the DAL object call other DAL objects for the child objects. This is easy to do, and easy to test, and looks fine on the surface. Of course, it isn’t fine, and the first issue to crop up is one of performance. The performance issue is caused by iterating through an object, generating additional DAL / SQL calls, which in turn do the same for each level until there are no child objects remaining. For an object with lots of child objects, or lots of levels in the object hierarchy, this quickly results a large number of SQL queries, all for a single object. The issue is compounded when retrieving data for an array of objects.

Let’s take a step back to look at this. The design isn’t very good, as it fails to resolve the issue of making an object hierarchy work well with a relational database. The design is simple, which would normally be good, but the simplicity results in iteration and an increased number of round-trips to the database server. Neither of which could be considered a beneficial byproduct of the design.

Typically this regarded as a performance, rather than a design issue and a workaround is often introduced to improve the performance. Typically the DAL layer will get a “depth” parameter added to limit the number of levels we retrieve in the hierarchy since we also discovered that we don’t always want the entire hierarchy. Every single DAL object must now be modified to use and update our depth-tracker. When we are all done, it works, but performance is still poor, because we have only reduced the iteration rather than eliminating it.

This process can continue indefinitely, workarounds usually address the symptoms of the real problem instead of the problem itself. They also tend to be very specific and targeted, increasing the likelihood that any given patch may introduce new issues and create new symptoms since the underlying issue has not been addressed. After a while, the number of patches can make identifying the root-cause more difficult.

One of the better solutions to the SQL issue outlined here is quite simple, stop thinking in OO, and start thinking in relations, which is what the database uses. Once you do this, it becomes obvious that you can take the most frequent requests, identify how many levels of objects are involved and write a single SQL query that uses join statements to get all the levels of data in one fell-swoop. This is one example where the differences between OO methodologies and relational are disconnected. What is obvious in one isn’t necessarily efficient, optimal, or even a good idea in the other, and may even be contrary to the other methodology.

Each step taken in this bad design is relatively simple, and appears to be the most intuitive at the time, making it extremely easy to wind up with a design that has issues. The symptoms often masquerade as a different problem making it difficult to realize that the design contains a flaw rather than the implementation. This is quite typical in the absence of requirements forcing an iterative cycle of partial design intermixed with coding in place of upfront design and understading of how each element of the solution will be implemented and used.

This is perhaps why this particular example (mapping object to relational) comes up so often, and why it so easy to take such a course without knowing it.

As you can see, in the bad design example, we got no beneficial byproducts from the design, and instead got a series of issues and problems requiring more and more workarounds to address.

I’ve been working on a system written in .Net/C# that uses SQL Server, my database experience prior to SQL Server included Oracle, Ingress, Sybase, Postgresql, and some MySql. Each had their quirks, but I’m learning more and more of the SQL Server quirks, and at times seriously having to wonder how Microsoft gets off calling this an enterprise class product. I will fully admit that some of my complaints are with how .Net communicates with SQL Server, and these probably are not Sql Server’s fault.

Here is my short list of gripes (in no particular order):

1. No Sequences
   Yes, I know, you can use IDENTITY columns, but they are not the same, and SQL Server places some foolish restrictions on their use. Instead of an IDENTITY column being a “normal” scalar column it is special and you can never update the column, although you can insert, but only after additional SQL directives. The IDENTITY implementation is close to a sequence, but Microsoft just didn’t implement it quite right. I do wonder why they didn’t introduce proper sequences with SQL Server 2005, they could have done this quite easily, a proper sequence could peacefully co-exist with the existing IDENTITY nonsense.
2. Inconsistent Parsing
   I was creating query that needed to run properly against two versions of a table, one version had an additional column to simplify the query. Using the IF EXISTS statement, I wrote a query similiar to the following:
   IF EXISTS (
SELECT NULL
FROM sysobjects so, syscolumns sc
WHERE so.id = sc.id
AND so.name = 'table_name'
AND so.xtype = 'U'
AND sc.name = 'column_name'
) SELECT column_name FROM table_name
   When I ran it against a database that had the column it worked, when I ran it against one that didn’t have the column, I got an error of it complaining that column_name didn’t exist. I shouldn’t have even tried, that is the whole idea behind using the IF EXISTS… The following query:
   IF EXISTS (
SELECT NULL
FROM sysobjects
AND name = 'table_name'
AND so.xtype = 'U'
) SELECT * FROM table_name
   Works as expected when run against a database that does not have the table.

   So, in one case it parses the statement inside the IF EXISTS and does some checking, in another case it does not.

3. NOCHECK Constraints
   The ability to temporarily disable constraint checking can be quite convenient, particularly while bulk loading data. What I don’t like about this is that this “WITH NOCHECK” state can actually stick around indefinitely, and is included in backups of the database. The constraint appears to be present, but is not actually being enforced.
4. Escaping Reserved Words & Special Characters
   This is one of those things that looks reasonably useful until you actually take a couple moments to think about it. SQL Server will let you use special characters like spaces, periods, and even reserved words when defining your schema, all you need to to is enclose them in square brackets. The following is valid:
   CREATE TABLE [Table] (
[hard.to.use.column] varchar(20),
[also hard to use] varchar(10),
[null] varchar(10)
)
   There are good reasons why we have reserved words, and why spaces and such are not allowed in table or column names. While it is admirable that Microsoft provided such a convenient way to differentiate the real reserved words from the ones we want to use, it’s horribly misguided. No one should do this, ever, it’s just plain confusing, not to mention non-portable, foolish, and, oh yeah… stupid.
5. Inconsistent context from .Net/C#
   I will admit that I do not know if this is a SQL Server thing, or a .Net/C# thing. This is related to the debate about using parameters or not (which I won’t get into), the issue is that .Net/C# does not execute these in the same way. Here is a code sample to help demonstrate this:
   public Query1( SqlConnection cnx, int id )
   {
   cmd = cnx.CreateCommand();
   cmd.CommandText = String.Format( "SELECT * FROM myTable WHERE id = {0}", id );
   SqlDataReader r = cmd.ExecuteReader();
   if( r.Read() ) { /* read result */ }
   r.Close()
   }

   public Query2( SqlConnection cnx, int id )
   {
   cmd = cnx.CreateCommand();
   cmd.CommandText = "SELECT * FROM myTable WHERE id = @id";
   cmd.Paramaters.AddWithValue( "@id", id );
   SqlDataReader r = cmd.ExecuteReader();
   if( r.Read() ) { /* read result */ }
   r.Close()
   }

   
   Query1 will be sent to SQL Server as a normal query. If you were to build another query in the same program it would execute in the same context. In other words, you can do a transaction by issuing the raw SQL as multiple statements using a combination of cmd.ExecuteReader() and cmd.ExecuteNonQuery(). You could also issue the sql to allow IDENTITY INSERTS, and then loop through some in-memory data and create multiple INSERT statements from that data.

   Query2 should be identical, but it is not. It executes in it’s own context as a side-effect of the fact that a stored procedure is actually used to execute the actual query. The parameters are defined and their initial value set as arguments to the stored procedure, with the actual parameterized query being passed un-altered from .Net to the stored procedure as one of the arguments.

   While it seems a bit excessive to invoke a stored procedure to deal with the parameters, this is not what bothers me. What bothers me is that a parameter-less query and a parameterized query are not handled in an identical manner by .Net/C#, especially with the nice side effect of the difference in execution context of the two. Code that produces valid SQL without parameters can break without explanation simply by implementing parameters.

   Another issue related to this is that .Net/C# appears to always parse the sql before deciding how to execute it. In an attempt to reduce potential errors where you might build a query with undefined parameters, it actually makes it impossible to create a stored procedure or user defined function from C#. Since these things will use parameters in T-SQL, they conflict with how C# handles parameters, the following code snippet will produce a .Net exception:
   cmd.CommandText = "CREATE PROCEDURE foo @id int AS SELECT * FROM myTable WHERE id = @id";
cmd.ExecuteNonQuery();
   The exception will be that you have not defined @id with a cmd.Parameters.AddWithValue() statement. Of course, you didn’t do that because you are not trying to use a parameter, you are issuing DDL SQL to create a stored procedure that will use one. I have yet to find out how to tell .Net/C# to stop doing this so that this SQL can be executed. I means you can’t programatically create a database schema from within a C# application.

6. Column Aliases not Available to ORDER BY
   This one should be a bug, the fact that if you define an alias for a column you should be able to use that alias in the ORDER BY clause. You can’t. This is particularly annoying for a calculated column or one that transforms the data using CASE or by concatenating text results. You wind up having to duplicate the calculation in the ORDER BY clause. Silly.
7. Default Collation Sequence is Case-Insensitive
   When you install SQL Server (2000, MSDE, 2005, 2005 EE) the default collation sequence is case-insensitive. I’d never seen this before when working with other databases. I only discovered it by accident when looking at a client’s data and discovered that data being inserted into a column with an FK constraint was allowing lower-case when I knew the referenced table was all upper case. I was shocked. This might make some sense for some other alphabets, but it certainly produced some unexpected results for me.

   You can actually create each table and column with a different collation sequence. This might be flexible, but I suspect it breaks several rules of design and should be avoided.

So there you go. Some of the reasons why I dislike SQL Server. I’m sure I will think of some more… I’ll add some of them to the list as I think of them.

About a month ago, Mark Zaugg handed me a copy of this book, he’d bought it but had not yet had the time to give it a read. It was a very interesting read, quite unlike any other book I have ever read on a technical subject. The author (Stéphane Faroult) has themed the book on the classic The Art of War by Sun-Tzu, and included many quotes and examples of a historical military nature. That might seem pretty odd for a technical book, so you’ll have to trust me when I say that it works.

The book does provide some tips and techniques that can be used to tune SQL queries and to solve some common problems in some pretty non-obvious ways. After you read this tomb you will be better at analysing, understanding, and tuning queries, and you will have some new and arcane skills with SQL, but that is not the primary benefit this book delivers. Mark had told me when he loaned me the book that he had heard it would make you “turn your head sideways to look at SQL”. He won’t understand just how accurate that statement was until he’s read the book.

The author shares his knowledge of SQL and relational theory in a way that is incredibly compelling. You finish each chapter with new insight into how and (more importantly) why different queries perform differently. The examples that are provided are realistic enough to make sense and it is very easy to recognize how the information, analysis, and techniques would apply to your own projects. I now understand significantly more about relational theory, SQL, and I have indeed learned how to look at SQL with my head sideways. The knowledge the book imparts goes beyond this as well, it discusses good design of the tables in a database, and it makes you think about the design of the databases you might already be using.

I will probably pick up a copy of this book for myself, even though I have now read it, it was impressive enough that I think it should have a home on my bookshelf.

I’ve been writing quite a bit of SQL lately, and although I’ve been messing around with SQL for many, many years, there are still things to learn. The most recent lesson involved about 3 hours of staring at a query trying to figure out why it wasn’t working they way it was supposed to. You quickly learn that in SQL you can’t test for NULL using = (or !=) and instead you must use IS NULL and IS NOT NULL. Initially this just seems to be a quirk with SQL, but the true relevance of this hit home in my latest battle with a query.

I’ve been writing code in C for over 20 years, I’ve gotten quite familiar with, and comfortable with the uses of pointers, and in particular with null pointers. In the world of pointers, we assign null for a lot of reason… when we need to indicate that the pointer has no value, we use null, we use it in linked lists to indicate that there is no next link (or no previous link for doubly linked lists), just about all the uses though get interpreted as meaning “no value” or “nothing”. A null string, is not the same as an empty string, instead it is one that you have not created yet, it’s nothing, it indicates something that does not exist.

So, it’s easy to think of NULL in SQL the same way, as nothing. The problem is that in SQL, NULL is NOT nothing, even though the most common uses of NULL in database columns is to indicate no the absence of a value. When SQL was first created there the creators were basing many of it’s features on mathematical set theory. In set theory a null is undefined, or unknown, it is used when you don’t know the value of something, or can’t (yet) know the value of something. Set theory does not allow you to not have a value, the closest it comes to this would be an empty set. As in all mathematics though it is possible to not know the value of something while still knowing that that something does indeed exist. SQL inherits this definition of NULL from mathematics.

This is why we have to write IS NULL and IS NOT NULL instead of = NULL and != NULL. If it were nothing, then it would make sense to be able to compare nothing to nothing, and it would make sense that nothing compared to nothing would be a match. Undefined is quite different. Two unknowns are not equal, or at least we can’t know if they are equal or not since we don’t know their value. In fact it is more likely that two unknowns would be different (once we finally discover their values) than it would for them to be the same.

Okay, so I’ve explained why we can’t compare two NULL values. This isn’t new, anyone working in SQL learns this quickly enough, but they rarely understand why it can’t be done, just what the “workaround” is. Understanding this becomes crucial when you JOIN tables in SQL and the columns that you join on can contain NULLs. This is where I spent more than 3 hours scratching my head in wonder. I joined two tables, or more accurately, I joined a table with a previous version of itself in order to identify rows that existed in one table and not the other based on their data rather than their numeric id columns. I knew the id columns had changed, but that the data had not. The query didn’t work, in fact in some cases it almost looked like the query was producing the opposite result of what it should have. I was convinced that SQL Server 2000 was buggy and I’d somehow managed to confuse the SQL parser.

I was wrong. The tables had to be joined on 12 columns (yeah, I know … it was ugly), and all but one of those could contain NULLs, reality was that most of the columns did contain NULL. The problem is in how a join is specifed in SQL, here is an example:

SELECT foo, bar FROM quux INNER JOIN baz ON quux.code = baz.code

The join is specified by identifying columns that reference each other, and the join is then performed with rows from each table that contain rows with equal values. Ah! there is the problem, when NULLs are involved, nothing is equal to them, not even another NULL. So joins with columns that have nulls produce results that just plain don’t seem to work. It all makes sense when we remember that NULL does not mean nothing, but means unknown/undefined. The solution is quite simple:

SELECT foo, bar FROM quux INNER JOIN baz ON ISNULL(quux.code,0) = ISNULL(baz.code,0)

Unfortunately, this will impact performance quite significantly for large tables. Assuming there was an index on the code columns of both tables, the index would not be usable, since we are now applying the ISNULL() function, and the index was generated on the column itself, not the result of that column being fed to a function. The result is a full table scan of both tables involved.

The lesson here is that as you design a database (or refactor one) you should put some serious thought into any column that allows the use of NULL. I’d even go so far as to recommend that you try to avoid them and think of other ways to represent the same data. Often the same result can be achieved with a second table that links to a “base” table, and only contains data when a matching row in the base table requires it. When implemented in this manner the absence of data rather than the presence of a NULL is a much better, and clearer indication of nothing than using a NULL. if you do it this way, you won’t run into weird behavior from JOINs when you least expect it.

…and that’s is… NULL-Thing to see here, carry on…