I often hear statements in the form of “X is cheap, don’t worry about it”. This usually comes up in design sessions, software or infrastructure, it doesn’t matter, you can usually find or hear a statement in this form in either type of design session.

The most common versions would be in reference to disk space, bandwidth, and CPU.

I suppose this type of thinking is a result of the effects of Moore’s Law. One of the troubling things about this sort of thinking is that it is frequently wrong and is indicative of someone with a closed mind, an inability to see the big picture, lack of insight, short-sightedness, or any other similar phrases. In all cases, when someone makes an X is cheap statement it’s a safe bet that they haven’t really thought things through. These statements are very dangerous, particularly if you happen to be in a design session when someone says this and is of particular concern if that someone is a team leader or manager.

The most dangerous aspect of these statements is that on the surface they may appear to be accurate and correct. Unfortunately, unless we are designing as system that people will only look at (ie. the surface), rather than actually use, it is rarely correct.

I’ll make an attempt to examine the most common of these statements to show how and why they are incorrect and assumptions. The analysis will be from a business/enterprise perspective rather than a personal/consumer one. You might be surprised to see how inter-related these can be! Let’s start with:

Disk is cheap, don’t worry about size

I will have to start by conceding the fact that disk storage has gotten very cheap indeed. The average consumer can put a terabyte of disk into their PC for less than $200. So how can that NOT be cheap?

We first need to understand what is really being said here. This usually isn’t a reference to disks and storage in general, it will be in reference to data files or databases.

So lets talk about files… in the past files weren’t that big, they were text files or word documents, and although the word documents were bigger, the size was still reasonable. This is no longer true, text files are rarely used anymore, the same information is now in a PDF file, or word document, an image, or a multi-media file. Those PDF and word files usually have graphics in them too, and to ensure that they look good when they print the graphics are stored at full resolution, even if they are only displayed thumbnail-sized in the document itself. The average document size is now measured in MB instead of KB, and for media files, GB is becoming more and more common.

So, what am I trying to say? The amount of storage that is available at a reasonable cost has increased by orders of magnitude, but so has the average size of the files that we store. The net result is that we can store about the same number of files as we used to before.

Even if we can store more files there is another cost involved. The more files and directories we have on our disks, the more difficult it becomes to find a file when we need it. If a user can’t remember where they put a file on a file server and they have to search for it, that potentially ties up a lot of I/O and CPU resources as the server scans all the files. That results in slower performance for all the other users relying on that server. That same server could be hosting a database, or a VM, or even just the virtual disk of a VM, and each of those would be affected by this search. Even if you don’t search for files very often because you are so well organized… those files are getting scanned anyway a couple times a day by your anti-virus and anti-spyware software.

Now lets look at databases… Modern database servers make them easy to use, quick to access, all those good things. So where is the problem. Let’s look at how a real production database gets used. To start with, you don’t just put a production database on disk, that’s too risky, disks can fail. You put it on a raid array instead, using mirroring, or striping, or whatever. The end result is that we now have two or possibly three (depending on the level of raid) copies of that database. We need to back up the database, that’s another copy… but in today’s web-enabled global economy, we can’t take the database server offline to back it up. So we have to take a snapshot to another disk so that we can make a backup of it without it changing while we write to tape, that’s another copy. Add some developers into the mix, any system that has a database probably has some developers working on new versions of the software and they need working copies of those databases to use for testing and development. That’s another copy (per developer) for each database. What are we up to now? 5 or 6 copies? Once again, the way we use that so-called cheap disk space quickly changes the cost per MB when each MB that we use is actually stored multiple times.

Backups become important too. Bigger databases take longer to backup, which also means they take longer to restore too. That can be crucial when the database for a web-commerce site goes down and we need to restore it. The companies that chose to believe the “disk is cheap” myth will be offline waiting for the restore a lot longer than the company that chose to use their disk space more wisely.

Storage in general is no longer local in a business environment. Files get shared on file servers, as well as through the use of SAN and NAS devices. This means that accessing the data on those “cheap” disks is now over a network and now bandwidth enters into the equation. A perfect segue into…

Bandwidth is cheap, don’t worry about the size of your data

Once again, we start with trying to understand what is really being said here. This is typically the speed of the available network rather than the cost of the network, although it can easily refer to both.

The speed and cost of networks has improved dramatically over the years. They have gone from 300 baud modems to 1200, and on and on up to 14400 and 56Kbps only to be replaced by broadband connections anywhere from 128Kbps to 10Mbps. Local networks went from proprietary systems to 10Mbps Ethernet, then 100Mbps, and now 1Gbps or wireless networks that started at 11Mbps and are now 54Mbps or 108Mbps.

What was once inconceivable to transmit over a network is now commonplace. So where is the problem here?

The first misconception comes from a single user perspective of the network. This is easiest to explain with an example. A typical office will have a 100Mbps network, providing 100Mbps of network bandwidth to each user. Sounds pretty good and we don’t use hubs anymore everyone uses switches, so user A doesn’t get impacted by what user B is doing on his segment of the network. Or does he? That depends, although in most cases the answer is yes, they are affected. Those network switches only isolate traffic between two end points. So, I might have 100Mbps between me and the switch, but ultimately I don’t need anything from the switch, I need it from some resource on the network. That resource might be a file server, or a database, or a website. There is a significant chance that I’ll be competing with the other user to connect to the same end point. So while we both have 100Mbps to the switch, we are sharing the 100Mbps from the switch to the common end point, so we’ve effectively been reduced to 50% of the “perceived” bandwidth. Chances are that it won’t be just two users connecting to that common end point, but a lot more. In a small company, you are probably competing will ALL the other users, so divide that 100Mbps by 10 or 20, bringing us down to 10Mbps or 5Mbps, certainly not the amount of bandwidth that you expected to get.

I will admit that I’m not being completely fair in my calculation, its far to primitive to be accurate, everyone would have to be accessing the same server at the same time for those numbers to be correct. Statistically we will get much better performance, exactly how much better is harder to predict since it depends on what is being accessed. Of course, the bigger the files we are after, the longer we send transferring their data, the more likely we are to be affected. This is where we can start to see the impact of the “disk is cheap” mindset on other (seemingly unrelated) areas of computing.

So, I haven’t proven anything other than you won’t see 100% of your bandwidth. We’ve only been talking about an typical office LAN, lets examine networked applications.

There are two typical varieties of networked applications. Those that explicitly use the network by implementing some form of protocol, and those that indirectly use a network through file sharing, web-services, SOA, or database connections. If a protocol is designed with the “bandwidth is cheap” thinking, you’ll probably find that it sends and receives lots of data. The networks are indeed fast enough to make this appear to be a non-issue. Unless you happen to be the the sysadmin or the hosting provider. More efficient use of bandwidth translates directly into more users, which translates directly into revenue. Once you max out your bandwidth with users, you might also need to buy/build new servers to handle more customers.

The more customers you can fit into the same bandwidth, the fewer servers and infrastructure you need, which directly affects the capital investment required (which again, affects Time-To-Revenue, since you have to recoup the infrastructure cost to become profitable).

More and more often you will find servers running in VMs, which wind up aggregating network traffic from all VMs to a single physical network. Applications running in VM servers that don’t make efficient use of the network could find the VMs running out of available network bandwidth before they run out of available CPU. If we have network traffic between two VMs, it never hits the physical network, but it does wind up consuming additional CPU possibly impacting performance of the other VMs.

When we introduce wireless and cellular networks to the mix the impact is more easily observed. Sure that new 802.11n wireless is fast, but when you have an office full of users streaming video over the same shared wireless bandwidth…

So if you encounter someone that suggests “X is cheap, don’t worry about it” in your next design session or troubleshooting discussion, you will know that what they are really saying is that they probably don’t understand the issue at all.

I’ve been working on a system written in .Net/C# that uses SQL Server, my database experience prior to SQL Server included Oracle, Ingress, Sybase, Postgresql, and some MySql. Each had their quirks, but I’m learning more and more of the SQL Server quirks, and at times seriously having to wonder how Microsoft gets off calling this an enterprise class product. I will fully admit that some of my complaints are with how .Net communicates with SQL Server, and these probably are not Sql Server’s fault.

Here is my short list of gripes (in no particular order):

1. No Sequences
   Yes, I know, you can use IDENTITY columns, but they are not the same, and SQL Server places some foolish restrictions on their use. Instead of an IDENTITY column being a “normal” scalar column it is special and you can never update the column, although you can insert, but only after additional SQL directives. The IDENTITY implementation is close to a sequence, but Microsoft just didn’t implement it quite right. I do wonder why they didn’t introduce proper sequences with SQL Server 2005, they could have done this quite easily, a proper sequence could peacefully co-exist with the existing IDENTITY nonsense.
2. Inconsistent Parsing
   I was creating query that needed to run properly against two versions of a table, one version had an additional column to simplify the query. Using the IF EXISTS statement, I wrote a query similiar to the following:
   IF EXISTS (
SELECT NULL
FROM sysobjects so, syscolumns sc
WHERE so.id = sc.id
AND so.name = 'table_name'
AND so.xtype = 'U'
AND sc.name = 'column_name'
) SELECT column_name FROM table_name
   When I ran it against a database that had the column it worked, when I ran it against one that didn’t have the column, I got an error of it complaining that column_name didn’t exist. I shouldn’t have even tried, that is the whole idea behind using the IF EXISTS… The following query:
   IF EXISTS (
SELECT NULL
FROM sysobjects
AND name = 'table_name'
AND so.xtype = 'U'
) SELECT * FROM table_name
   Works as expected when run against a database that does not have the table.

   So, in one case it parses the statement inside the IF EXISTS and does some checking, in another case it does not.

3. NOCHECK Constraints
   The ability to temporarily disable constraint checking can be quite convenient, particularly while bulk loading data. What I don’t like about this is that this “WITH NOCHECK” state can actually stick around indefinitely, and is included in backups of the database. The constraint appears to be present, but is not actually being enforced.
4. Escaping Reserved Words & Special Characters
   This is one of those things that looks reasonably useful until you actually take a couple moments to think about it. SQL Server will let you use special characters like spaces, periods, and even reserved words when defining your schema, all you need to to is enclose them in square brackets. The following is valid:
   CREATE TABLE [Table] (
[hard.to.use.column] varchar(20),
[also hard to use] varchar(10),
[null] varchar(10)
)
   There are good reasons why we have reserved words, and why spaces and such are not allowed in table or column names. While it is admirable that Microsoft provided such a convenient way to differentiate the real reserved words from the ones we want to use, it’s horribly misguided. No one should do this, ever, it’s just plain confusing, not to mention non-portable, foolish, and, oh yeah… stupid.
5. Inconsistent context from .Net/C#
   I will admit that I do not know if this is a SQL Server thing, or a .Net/C# thing. This is related to the debate about using parameters or not (which I won’t get into), the issue is that .Net/C# does not execute these in the same way. Here is a code sample to help demonstrate this:
   public Query1( SqlConnection cnx, int id )
   {
   cmd = cnx.CreateCommand();
   cmd.CommandText = String.Format( "SELECT * FROM myTable WHERE id = {0}", id );
   SqlDataReader r = cmd.ExecuteReader();
   if( r.Read() ) { /* read result */ }
   r.Close()
   }

   public Query2( SqlConnection cnx, int id )
   {
   cmd = cnx.CreateCommand();
   cmd.CommandText = "SELECT * FROM myTable WHERE id = @id";
   cmd.Paramaters.AddWithValue( "@id", id );
   SqlDataReader r = cmd.ExecuteReader();
   if( r.Read() ) { /* read result */ }
   r.Close()
   }

   
   Query1 will be sent to SQL Server as a normal query. If you were to build another query in the same program it would execute in the same context. In other words, you can do a transaction by issuing the raw SQL as multiple statements using a combination of cmd.ExecuteReader() and cmd.ExecuteNonQuery(). You could also issue the sql to allow IDENTITY INSERTS, and then loop through some in-memory data and create multiple INSERT statements from that data.

   Query2 should be identical, but it is not. It executes in it’s own context as a side-effect of the fact that a stored procedure is actually used to execute the actual query. The parameters are defined and their initial value set as arguments to the stored procedure, with the actual parameterized query being passed un-altered from .Net to the stored procedure as one of the arguments.

   While it seems a bit excessive to invoke a stored procedure to deal with the parameters, this is not what bothers me. What bothers me is that a parameter-less query and a parameterized query are not handled in an identical manner by .Net/C#, especially with the nice side effect of the difference in execution context of the two. Code that produces valid SQL without parameters can break without explanation simply by implementing parameters.

   Another issue related to this is that .Net/C# appears to always parse the sql before deciding how to execute it. In an attempt to reduce potential errors where you might build a query with undefined parameters, it actually makes it impossible to create a stored procedure or user defined function from C#. Since these things will use parameters in T-SQL, they conflict with how C# handles parameters, the following code snippet will produce a .Net exception:
   cmd.CommandText = "CREATE PROCEDURE foo @id int AS SELECT * FROM myTable WHERE id = @id";
cmd.ExecuteNonQuery();
   The exception will be that you have not defined @id with a cmd.Parameters.AddWithValue() statement. Of course, you didn’t do that because you are not trying to use a parameter, you are issuing DDL SQL to create a stored procedure that will use one. I have yet to find out how to tell .Net/C# to stop doing this so that this SQL can be executed. I means you can’t programatically create a database schema from within a C# application.

6. Column Aliases not Available to ORDER BY
   This one should be a bug, the fact that if you define an alias for a column you should be able to use that alias in the ORDER BY clause. You can’t. This is particularly annoying for a calculated column or one that transforms the data using CASE or by concatenating text results. You wind up having to duplicate the calculation in the ORDER BY clause. Silly.
7. Default Collation Sequence is Case-Insensitive
   When you install SQL Server (2000, MSDE, 2005, 2005 EE) the default collation sequence is case-insensitive. I’d never seen this before when working with other databases. I only discovered it by accident when looking at a client’s data and discovered that data being inserted into a column with an FK constraint was allowing lower-case when I knew the referenced table was all upper case. I was shocked. This might make some sense for some other alphabets, but it certainly produced some unexpected results for me.

   You can actually create each table and column with a different collation sequence. This might be flexible, but I suspect it breaks several rules of design and should be avoided.

So there you go. Some of the reasons why I dislike SQL Server. I’m sure I will think of some more… I’ll add some of them to the list as I think of them.