Errors – I made a mistake

(25th August 2012 Signing on at 10:54)

In “The Morning Pages” yesterday, I started writing something about errors in computer systems. In this instance I’m thinking about data errors, faults that get saved to the files or the database. This is something which (perhaps unfortunately) I know something about. At the time I wrote it, I thought that it was interesting. I’m going to continue with it here. I’m not referring to what I wrote yesterday, this is written from the start.

At the beginning I suppose that it is useful to define errors for this context and then divide them into groups so that I can write about each group independently. After that (if I can be bothered, or get that far) I can compare and contrast the different groups. The world can be divided into two sorts of people: those who divide it into parts and those who do not!

What is an error? In this context it is a value in the database (let’s assume we are using databases) which is incorrect and causes some kind of problem with subsequent processing. Although for the purposes of “stream of consciousness” I don’t usually use bullets, I am going to here. There are no rules here, so I’m not breaking my non-rule, just moving away from a convention which I have followed up till now. In “error” I am not including those values where a value has been entered into the database which complies with all the rules, but which does not actually match reality. This is an interesting case, which I think I will deal with separately.

What are the possible sources of these errors?

• ·         Hardware faults: A sector on the disk is damaged, so what is written there is not what is subsequently retrieved.
• ·         Outside events: External events; power spikes or radiation produce changes to the data as it is recorded.
• ·         Software errors: A fault in a programme results in something other than what was intended being recorded.
• ·         Imports: Data has been imported which by-passed normal validation

Let’s look at the errors themselves. What is an initial classification? Some ways of looking at it is to consider how widely you will have to look in order to detect the error, at what level the error could be detected and what you could do to correct the error once it has been detected.

First of all there are errors which occur at the level of the value of a single field, in a single record. Of course all of these may repeated and be caused by any of the root cause sources above. The simplest is where the value in the storage is something which is incompatible with the definition of the field. At its most extreme this should be detected by the operating system or the database manager. One or other of these will throw some kind of exception. Rectification is going to require work at a low level.

Next level up is when the value which is stored in the field is compatible with the definition which is used by the database manager, but is not valid according to the application which is using the data. An example of this is where a database contains an “integer” field which is used to hold an enumerated value (let’s say “1,2,3,4” are valid, but anything else is not). Of course someone will say that the database should be designed with field level validation to prevent this sort of thing. Yes, it probably should (although that introduces its own problems) but many are not.

Supposing such data has got into the database, how will we find out? If we are lucky, then the application will have practiced defensive programming and when the data is encountered something will throw an error. (Pause at 11:21 resumed 11:36) If we are not so luck, then the application “will carry on regardless” and it is quite difficult to know what will happen. Let’s stick with the case that we can understand: the application, or the database or he operating system throws an error which we catch. What do we do then?    

The first thing to do, is get someone who understands the application to identify which module (term used very loosely) is throwing the error and identify what is causing the error to be thrown (if you see what I mean). The objective of this is to find a “pattern” which can be used to identify the offending data. At this stage, we are not really sure whether the problem is “data” or “program”. We could have a piece of faulty data, or we could have a program which is not handling a valid (but possibly unusual) piece of data.

Let’s assume for the time being that what we have is invalid data. This is something that the application should really be rejecting. There are now two things to be done: first, identify how widespread this problem is, and two, identify how the data came to be corrupt in the first place. Both of these objectives are addressed by finding all the records which suffer from this particular problem. This is done by using the “pattern” which describes the offending data to create a query to identify the affected records. Examining these records will tell us a number of things: is the problem widespread? Is there a pattern to the affected records? By pattern, I mean: do they all come from the same place? Are they all in the same state? Are they updated about the same time? The background information accumulated may help us to identify the root cause.

The number of records affected is a vital piece of information. How are we going to fix the values? First of all: Is it possible to infer the correct value? This can be difficult. At this stage it is well worth taking a moment to remember that with any “data correction” we are going to be “changing the record of reality”. If some action is taken which modifies data, then there should be a procedure for keeping a record of what changes were made (This often takes the form of a log in the database).If it is possible to identify “correct” values, then that is the preferred option. The next possible option is to set the data value to some kind of innocuous default, especially one which will prompt for an update the next time the record is opened. The last options are to mark the field or maybe the whole record as faulty.

So much for faults with individual fields; what about faults which need a wider scope to detect them? An example of this is where the values of two or more fields are not compatible. This is usually associated with denormalisation in the database.

Let’s take two cases: simple denormalisation (the same field is repeated in more than one record) and derived attribute (where one attribute summarises other attributes).

Take the first one first. Someone has decided to repeat the same attribute in more than one place. Let’s not argue about whether this should have been done. Someone has decided to do it! First confirm that the values are always supposed to be the same. If they are, which is what I would expect, then the pattern to use for detecting the error, is “where are they different?”

Now the pattern for handing this is pretty much the same as an error in a single field. Find out how many records are affected, examine the affected records looking for any pattern and try and identify the root cause (in my experience, a root cause which is sometimes overlooked is data which has been migrated into the system).

Now the question is “what do we do about it?” Well, obviously (?) correct the root cause. As for correcting the data, is there one of the records which can be regarded as the “master”? For denormalization, there should be. If there isn’t then this indicates a (possibly serious) fault in the application design. If there is, then update the denormalized copy to match the master. Remember that all these data changes which are done outside the application should be logged.

Now let’s look at the case of which represents a derived value. A typical example of this would be where we had a transaction which consisted of individual line items. The transaction record contains an attribute which contains the total value of individual transactions. It might also contain the number of transactions. An error would occur when the total in the “header” did not have the same numerical value as the totals in the individual “line items”.

Before we go any further, it is worth pointing out that there is a special case root cause which may be at work here. That is the possibility of “rounding” or other “arithmetic” related errors.

The pattern to use for (breaking off 12:23, resuming 12:42 after lunch) for detecting this kind of error is that the value calculated from the individual items is not equal to the value in the header record. Now there is another implicit trap here: if we use the database manager to calculate total of individual items, then we are probably not using the same code as the application, so be careful if “rounding” or similar problems are involved.

Having identified the records involved, the next steps are as before: examine the records, identify the root cause and develop a strategy for correcting the affected records. Someone is almost certain to suggest that all totals should equal the sum of their parts. I know I would consider this as a correction strategy, but it may not be correct! Why not? Consider the following two possibilities: that one transaction has been missed from the total; so the total is correct but a transaction is missing, and a record has been saved twice into the individual items table! Both these problems would be insidious, and difficult to fix reliably. Possible strategies for dealing with them involve careful examination of the records and the code doing the work in parallel.

 Now let’s cast the scope wider: are there cases where we have data errors which are even wider spread? Well, obviously we do, but (until I can think of a better way) I would handle them as variants of the “derived attribute” class. The patterns used to detect them become progressively more complicated and correction strategies become harder.

Can “human error” be detected? Possibly it can be detected, but not always reliably. What we would have to do is develop a pattern for what constituted “human error”. This might be supported by other factors which are not specifically related to the data itself like “particular operators”, “particular locations” and “particular times”.  

Now, this is where it gets interesting for me. I’ve come to the end of what I wanted to say, but I want to write more. This is where the free association comes in. The concept of data can be extended to “memory” and system to the mind or data can be extended to dna and system can be extending to living things. Does the speculation above have any validity? I think it does, moreover, in a strange metaphysical kind of way what I am trying to do at the moment is perform exercises which reshape thinking and habit. I want to get into the habit of writing a certain amount at a sitting. I’m very good at planning what I write, but I don’t always get round to doing the actual writing. Here I am forcing myself to perform the act of writing (well typing if you must) with the intent of developing the habit. Repetition is at the heart of all habit. Habit implies repetition and repetition trains habit. What are the factors which determine strength of learning? Primacy, Recency, Repetition and I think there is something else as well. Of course we have to bring in conditioning factors like reward and punishment and triggers as well.

Back to data errors (see what I mean about rambling?) Programs are data too! I can extend the thinking to other forms of data which are not exclusively record based, such as some files and, of course, programs.

Let’s look at files first. What about a word processing file, is it possible to detect (and correct) errors in a word processing file? Well of course it is! I do it all the time, and at least some of the time Word helps me. A very small portion of the time, Word hinders me, but that is another matter. Remember, right now, my objective is to train myself to write a certain amount. I’m good at planning, I’m good at editing but I want to get better at writing.

How do I detect errors in a word processing file? In different ways, at different levels: I use Word to tell me when words are spelled incorrectly. Where a word is underlined red, I can either use Word to tell me what it thinks the correct spelling is, I can look it up in the dictionary or I can completely ignore what Word is telling me. Word and I are working together. Word does the checking, but I am in control. This is similar to the “single field” class of errors which can be found in a database. Of course, we know in advance what the root cause of these errors is: my typing is a bit dodgy, as is my spelling.

A similar, in fact almost identical approach applies when we extend the scope outwards to sentences and paragraphs. This is a more complex situation which does not really have a corresponding level in the database record model. Word contains a model of English grammar. I actually have no idea how good Word’s model of grammar is. I am sure it is better than mine! Where word flags a sentence as “needing its grammar looked at” I can look at what Word suggests. Most of the time I need to make the corrections myself. These can be simple things, like adjusting the ends of sentences, but can be more complex. In either case Word is helping me to write. I am looking at the screen and concentrating on what I am writing. Writing in this way I can write much more quickly.

One of my objectives here is to reach a “Flow state”. That implies that I try and reduce the amount of interruption. After all, almost my definition, interruption interrupts flow!

What Word cannot do is help me with the sense of the sentences or especially the paragraphs. What I write is sense or non-sense because of what I think, not because of the way it is structured.

What about programs? Is it possible to detect errors in a program? Well of course it is, I do it all the time. It is what de-bugging is all about. But what about it the general sense, as data? Here I think we are into more awkward territory. Some kinds of error are detectable. For example, these days syntactic errors are detected by the parsing editors.  If they are not detected during editing, then they are detected immediately by the compiler. Data errors, references to the wrong objects are likewise detected, but what about “the sense”? I am much more doubtful about that. Sense can only be checked by comparing the results of the program with the original specification.

If I extend that idea to writing for entertainment or even technical writing, then I get the idea that there should be a “specification”. Now I suppose ultimately, there is a debate about whether the specification can be right or wrong. I guess it just “is”, in the sense that it exists. For correctness to come into play, we have to have more than one expression of the same idea. In coding this takes the form (or perhaps the forms) of: the specification, the code itself, and the tests. This is the reason why many people argue for using the tests as the specification. Apart from efficiency (why have three artefacts when you only need two?), there is the point that tests are generally expressed more precisely than specifications (at least in coding).

Hmm. I have to be careful that I don’t talk myself round in circles here. I observed something interesting just then. One was in instance where Word could not tell the word I intended to use. Even from context, it did not seem to be able to tell whether “two” or “tow” was the word that I intended to use. Of course to a human reader that sort of typographical error is fairly obvious. It is also a case where over-reliance of a word-processor spell checker can lead you astray if you allow the machine to make corrections itself.

I remember when I used to exploit the features of the terminal hardware so that I could read spelling corrections (“highlighted”) out of context of the rest of the script (which was in normal brightness). That allowed me to work very quickly and accurately. Strange that using a monochrome terminal was sometimes preferable to using a colour terminal.

When I reach the end of the 5^th page of text I’m just going to stop. Remember, this is primarily an exercise in training. I’m not deliberately trying to write nonsense but the content is much less important than the activity. There is a reason that I am keeping this stuff, and that is that I want to be able to mine the “free association” aspect in the future. I’m not sure what that will yield but it seems like a potentially interesting experiment. Having the text online means that I may be able to use it in some unexpected and unplanned (at least for the time being) way.

I am going to separate this “rambling” stuff from the “reflective” stuff. I’m not sure I care who reads the ramblings (although I may change my mind) but I feel that I care about the reflection. Today seems to have been quite successful. You may find this strange but I do not expect to read what I have written for quite some time.  

And that seems like quite at appropriate place to end. I’ve produced five pages of ramblings. I started with a topic and then went on from there. For a while I am going to take that approach, but I may develop it in different ways. I expect to experiment with all sorts of different things. One of the things I will probably steer away from is formatting.  Put simply “I don’t care”.  This is about writing, not reading! (stopped at 13:49)