Calvus Solutions – Accelerated CTRM

Accelerating The Realisation of CTRM In The Enterprise

Calvus Solutions helps its customers implement practical CTRM (Commodity Trading and Risk Management) solutions that deliver maximum realisation of benefits whilst minimising risk and disruption

Calvus Solutions has over 25 years’ experience of working with customers to deliver solutions that are a hybrid of existing software assets and new custom made software components

Calvus Solutions offers a unique perspective on delivering CTRM to organisations that want to minimise implementation costs and risks by following a gradual step-by-step framework rather than the wholesale change that is characterised by large packaged software solutions

Languages

• WhiteSpace - An entire computer language created with spaces, tabs, linefeed, etc.
• BrainF_ck - A Turing-machine like computer language designed to make development harder, rather than easier.
• Universal Turing Machine in Conway's Game of Life - A Turing machine constructed using the predictable outcomes of random-looking mating patterns. This is really very impressive.
• Klingon Language :-}

Darwinian Language Evolution

Computer Langugaes tend to follow the same principals of evolution. I find the pace of innovation amazing - we have come such a long way, technically at least. However, in general terms, we are still stumped by the most basic questions: Natural Language, Automated Parrell Exececution, etc. XXX talks of this in the 1950's in his books.

Anyway, there is a very good picture of the evolution of languages and how they related to each other... Evolution of Computer Languages, here in diagram form Evolution Image

Over the years I have tried a number of Identifier schemes:

• Atomic: String UserId; int UserId; Guid UserId;
• There is no link to the underlying domain type.
• There instances are easily confused. I.e. A page taking a Url param of "Id=123" may refer to a User, UserGroup, UserProfile, NTLM, Email, etc. This gets significantly worse in complex enterprise systems, eg. "ServiceLocationIdentifier" may be named as a local variable called "sl" and the developer needs to correctly interpret this.
• Classic Strongly Typed: class UserIdentifier { string UserID { get; set; } }
• This is significantly better, but there is the overhead of creating and maintaining all these types
• There is no scheme in place to unify them. It is possible, ofcourse, to create an class abstract BaseIdenfifier { ... }
• There is no coupling from the Identifier to the Domain Type, ie User and UserIdentifer as not linked by the scheme. Meaning you would have to build this in with every class

The code of the idea I am playing with is:

public class Id<T>
{
    public string Id { get; set;}
    public Type DomainType { get; set; }
    public override string ToString() { return Id; }
    public override int GetHasCode() { return Id.GetHashCode(); }
}

This code as the following properties (none very interesting):

• The DomainType (T) is strongly maintained.
• This means that new Id<User>("Bob") == new Id<Group>("Bob") , should throw an TypeMismatch exception
• You can treat it as a string new Id<User>("Bob") == "Bob") will evaluate as true.
• There is a string coupling between the Identifier and the Domain Type
• class User { public Id<User> UserId { get; set; } }

The fun stuff, however, happens when you introduce a 'Services' class:

public class static Services
{
    public T GetInstance(Id<T> Identifier) { ... }
    public Uri GetUri(Id<T> Identifier) { ... }
    public string GetDisplayName(Id<T> Identifier) { ... }
}

The result of which are statements like:

public void ExampleCode()
{
    Id<User> bob = new Id<User>("Bob");
    string EmailText = string.Format("Hello {0}, Please click on the following link to edit your details {1}.",
           Services.GetDisplayName(bob),
           Services.GetUri(bob));

    User bobDetails = Services.GetInstance(bob);
    SmtpHelper.SendEmail(bobDetails.Email, EmailText, "Welcome...");
}

The full version of the class is as follows:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml.Serialization;

namespace GL.General.Core
{
    /// <summary>
    /// Encapsulated the relationship between a type and its string identifier.
    /// It provides a type-strong relationship, in which for example a UserId and SaleId are NOT comparible
    /// </summary>
    /// <typeparam name="T">The domain type associated with the identifier</typeparam>
    public class Id<T>
    {
        /// <summary>
        /// Default Constructor
        /// </summary>
        public Id()
        {
            DomainType = typeof(T);
            ID = null;
        }

        /// <summary>
        /// Strong constructor
        /// </summary>
        /// <param name="id"></param>
        public Id(string id)
        {
            DomainType = typeof(T);
            ID = id;
        }

        /// <summary>
        /// Copy constructor
        /// </summary>
        /// <param name="toCopy"></param>
        public Id(Id<T> toCopy)
        {
            DomainType = typeof(T);
            ID = toCopy.ID;
        }

        /// <summary>
        /// The string Identifier. The value of the identfier
        /// </summary>
        [XmlAttribute]
        public string ID { get; set; }

        /// <summary>
        /// The type of the identifier (used for meta-data and validation only)
        /// </summary>
        [XmlIgnore]
        public Type DomainType { get; set; }

        /// <summary>
        /// This is a helper method, which provides the <see cref="DomainType"/> which is serialised in XML
        /// </summary>
        /// <remarks>
        /// This attribute, while serialised to XML, is ignored when it is read in
        /// </remarks>
        [XmlAttribute]
        public string Class
        {
            get
            {
                if (DomainType == null) return string.Empty;
                return DomainType.FullName;
            }
            set
            {
                // Ignored in deserialisation
            }
        }

        /// <summary>
        /// Provide Equality checking with Id<T> and string
        /// </summary>
        /// <param name="obj"></param>
        /// <returns></returns>
        public override bool Equals(object obj)
        {
            if (obj == null && ID == null) return true;
            if (obj != null && ID != null) return obj.ToString().Equals(ID);
            return false;
        }

        /// <summary>
        /// Provide Equality checking with Id<T> and string
        /// </summary>
        public static bool operator ==(Id<T> lhs, Id<T> rhs)
        {
            if ((object)lhs == null && (object)rhs == null) return true;
            if ((object)lhs != null && (object)rhs != null) return lhs.ID == rhs.ID;
            return false;
        }

        /// <summary>
        /// Provide Equality checking with Id<T> and string
        /// </summary>
        public static bool operator !=(Id<T> lhs, Id<T> rhs)
        {
            return !(lhs == rhs);
        }

        /// <summary>
        /// Provide Equality checking with Id<T> and string
        /// </summary>
        public static bool operator ==(Id<T> lhs, string rhs)
        {
            if ((object)lhs == null && (object)rhs == null) return true;
            if ((object)lhs != null && (object)rhs != null) return lhs.ID == rhs;
            return false;
        }

        /// <summary>
        /// Provide Equality checking with Id<T> and string
        /// </summary>
        public static bool operator !=(Id<T> lhs, string rhs)
        {
            return !(lhs == rhs);
        }

        /// <summary>
        /// Provide Equality checking with Id<T> and string
        /// </summary>
        public static bool operator ==(string lhs, Id<T> rhs)
        {
            if ((object)lhs == null && (object)rhs == null) return true;
            if ((object)lhs != null && (object)rhs != null) return lhs == rhs.ID;
            return false;
        }

        /// <summary>
        /// Provide Equality checking with Id<T> and string
        /// </summary>
        public static bool operator !=(string lhs, Id<T> rhs)
        {
            return !(lhs == rhs);
        }

        /// <summary>
        /// ToString will provide the underling string identifier
        /// </summary>
        public override string ToString()
        {
            return ID;
        }

        /// <summary>
        /// Use the underlying string as a HashCode
        /// </summary>
        /// <returns></returns>
        public override int GetHashCode()
        {
            if (ID == null) return 0;
            return ID.GetHashCode();
        }
    }
}

Twitter from C#

How do you update your status from C#. Looking at the twitter API (which is REST  based), which has a link to a blog article on how to post to twitter from c#:

[OperationContract]
public void SubmitUserStatus(string username, string tweet)
{
    // encode the username/password
    string user = Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(username + ":" + password));
    // determine what we want to upload as a status
    byte[] bytes = System.Text.Encoding.ASCII.GetBytes("status=" + tweet);
    // connect with the update page
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://twitter.com/statuses/update.xml");
    // set the method to POST
    request.Method = "POST";
    // thanks to argodev for this recent change!
    request.ServicePoint.Expect100Continue = false;
    // set the authorisation levels
    request.Headers.Add("Authorization", "Basic " + user);
    request.ContentType = "application/x-www-form-urlencoded";
    // set the length of the content
    request.ContentLength = bytes.Length;
    // set up the stream
    Stream reqStream = request.GetRequestStream();
    // write to the stream
    reqStream.Write(bytes, 0, bytes.Length);
    // close the stream
    reqStream.Close();
} 

OutOfMemoryException

The maximum allocatable size of a .NET process depends on the following factors:

• The CPU, which for the .NET universe is
• x86 which has a 4GB address range, although in practice the actual available memory is around 3.2GB as the BIOS and video share the available address range.
• x64 which has an address range of 18.45 exabytes which is practically unlimited (although we said that about 386, x86 and y2k)
• Italium , which is beyond the scope of this article.
• The Windows operating system
• Windows 98/ME
• Windows NT/XP/2000 (x86 and x64)
• Vista (x86 and x64)
• Windows Server 2003 (x86 and x64)
• The headers PE signature of the OS Process /LARGEADDRESSAWARE
• Available swap space

The x64 platform (x64 CPU, x64 OS) will yield a maximum process size in excess of the available system RAM. For example, my 8GB Windows 7 machine was able to yield a process of 8,656 MB, although at that point the system was swapping so much it was unusable.

The x86 platform (x86/x64 CPU, x86 OS) will yield 1.5GB as the maximum process size be ##DEFAULT##. Most people assume this will be 2GB, however the garbage collector needs its own space and does not allocate the maximum.

It is possible to allocated over 2GB on a .NET x86 process by marking the assembly or EXE with /LARGEADDRESSAWARE. This cannot be done via Visual Studio, although you can create a Post Build Event (see attached solution for an example). Execute the following command-line tool EDITBIN.EXE /LARGEADDRESSAWARE MyApp.exe. This command can normally be found in C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin. In order to confirm that the binary headers have been correctly updated run the following command DUMPBIN.EXE /HEADERS MyApp.exe, which should give:

Microsoft (R) COFF/PE Dumper Version 9.00.30729.01
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file MaxMemory-x86-LargeAddressAware.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
             14C machine (x86)
               3 number of sections
        49FDF23D time date stamp Sun May 03 20:36:29 2009
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             122 characteristics
                   Executable
                   Application can handle large (>2GB) addresses
                   32 bit word machine

Strangely, it easier to test this on an x64 system which has lots of available memory. In tests (see test utility described later) I have been able to allocate 2'803MB.

It is more complex on a x86 systems as the operating system must have the /3GB switch in the c:\boot.ini file. For example, marking an .NET application with /LARGEADDRESSAWARE, but running it on a Windows XP system will yield 1,500MB max -- even if the machine has the maximum physical RAM of 4GB/3.2GB free! You need to have both /LARGEADDRESSAWARE on the specific application, and the operating system configured with /3GB.

.NET Maximum Memory Test Tool for x86, x64

I have found it tricky to isolate the different factors and to marry up the theory with actual hands-on results. To this end I created a utility allocate increasing amounts of memory (in 64KB chunks) to determine the actual maximum allocatable memory for a particular system (CPU, OS, RAM, swap space, and configuration /3GB, /PAE)

Full source (GPL) and compiled .EXE are available for download from the footer of this article. No installation is required.

The utility comes in three flavours:

• x64 -- called "MaxMemory-x64.exe"
• x86 -- called "MaxMemory.exe"
• x86 + /LARGEADDRESSAWARE -- called "MaxMemory-x86-LargeAddressAware.exe"

Fig. 004 -- MaxMemory.exe screenshot

Table of Findings

• I am able to allocate 8,656MB using "MaxMemory-x64.exe" on x86, Windows 7, 8GM RAM
• I am able to allocate 2,803MB using "MaxMemory-x86-LargeAddressAware.exe" on x86, Windows 7, 8GM RAM
• I am able to allocate 1,411MB using "MaxMemory.exe" on x86, Windows 7, 8GM RAM
• I am able to allocate 2,191MB using "MaxMemory-x86-LargeAddressAware.exe" on x86, Windows XP + /3GB, 2'496MB RAM, VMWare
• I am able to allocate 2,178MB using "MaxMemory-x86-LargeAddressAware.exe" on Xeon, Windows Server 2003 SE + /3GB, 2GB RAM, VMWare

I am sure the tweaking the allocations and system configuration by yield better results, and that the actual number may also vary from attempt to attempt.

On the x86 platform, allocations above the 1,500MB mark seem to be very slow. I do not know why this is the case.

A very good article, showing various approaches to .NET Reliablity and related defensive techniques.

-- Comments and refinements are welcome, Author.

See SokoSolve Part One

The last release of SokoSolve was in Jan 2008 -- now over a year later it is time to revisit SokoSolve and take stock. For the initial version 1.3.45-beta, most of the development work was done during an intense period of my life while my mom was very sick with cancer. Since then I have not put much effort into the code, nor have I been monitoring or promoting the project as I did in the initial phases. The result being a period of stability during which the project has stood on it own feet.

After spending a couple days reviewing the stats, the site, the code, and its 'google presence', I have the following thoughts:

• I forgot just how much effort I put into the initial version
• The current version is stable and is fairly well polished. The decision to use a lot of in-line HTML in the GUI creates a good, polished look.
• The solver GUI is overly complex and is hidden away from the user
• Given the number of downloads, there is been little or no feedback from the user-group. In fact, I have no clear idea of the number of active users.
• While the solver can solve some fairly hard puzzles, it is not as effective as I had originally hoped. The problem domain it seems continues to be rich and deceptively hard to make iterative progress. This is however, gut feel, the solver sub-system lacks a clear benchmark library which makes extremely hard to set a baseline and show progress over time.
• The solver GUI is somewhat over-whelming, clearly good for internal development, but probability a blocker for advanced users or would-be developers for using it. It also has a number of bugs/failings that hail the user experience.

As on April 2009 SokoSolve is a stable project with typical statistics:

• 11706 web site hits (by way of the page counter)
• SourceForge rank around 7000-8000
• Averages 3-10 downloads per day, with 2,616 download in the last 12 months
• Tellingly, no forum activity in 10 months

Distribution Notes

• Added SokoSolve to Libregamewiki
• Requested a review/link in freegamer.blogspot.com

TODO

• Replace the default cell images (kill the brown!)

A simple C# code snippet to encode and decode a string using TripleDES.

public string Encode(string Key, string PlainText)
{
    TripleDESCryptoServiceProvider tdes = new TripleDESCryptoServiceProvider();

    //mode of operation. there are other 4 modes. We choose ECB(Electronic code Book)
    tdes.Key = GenerateKey(Key);

    //padding mode(if any extra byte added)
    tdes.Mode = CipherMode.ECB;
    tdes.Padding = PaddingMode.PKCS7;

    //transform the specified region of bytes array to resultArray
    ICryptoTransform cTransform = tdes.CreateEncryptor();

    byte[] toEncryptArray = UTF8Encoding.UTF8.GetBytes(PlainText);
    byte[] resultArray = cTransform.TransformFinalBlock(toEncryptArray, 0, toEncryptArray.Length);
    //Release resources held by TripleDes Encryptor

    tdes.Clear();
    //Return the encrypted data into unreadable string format

    return Convert.ToBase64String(resultArray, 0, resultArray.Length);
}

public string Decode(string Key, string CypherText)
{
    TripleDESCryptoServiceProvider tdes = new TripleDESCryptoServiceProvider();
    //set the secret key for the tripleDES algorithm

    tdes.Key = GenerateKey(Key);
    //mode of operation. there are other 4 modes. We choose ECB(Electronic code Book)

    byte[] toEncryptArray = Convert.FromBase64String(CypherText);

    tdes.Mode = CipherMode.ECB;
    //padding mode(if any extra byte added)

    tdes.Padding = PaddingMode.PKCS7;

    ICryptoTransform cTransform = tdes.CreateDecryptor();
    byte[] resultArray = cTransform.TransformFinalBlock(toEncryptArray, 0, toEncryptArray.Length);
    //Release resources held by TripleDes Encryptor                

    tdes.Clear();
    //return the Clear decrypted TEXT

    return UTF8Encoding.UTF8.GetString(resultArray);
}

NOTE: This code will only work on small strings (not large files) as the data is read into memory and not streamed/buffered.

Building my a previous article where I encrypt/decrypt a string, I wanted to go further and ZIP it. So started a rather long and painful wander into the world in compression.

Knowing the GZIP is included in the .NET framework I naturally started there.

However, after crafting a basic unit test that did a basic round trip SOURCE->GZIP->RESULT and comparing that SOURCE==RESULT -- which worked, I discovered that the compressed size was LARGER than the SOURCE. Some articles on the web led my to believe that my implementation was incorrect giving a larger result.

However, this did not resolve my issue. It turns out that I had selected a .JPG file as my source binary test data. The file format itself if of course heavily compressed already resulting in the file size growing my almost 100%!

This means an implementation of GZIP needs to check the resulting file size, then if it is larger discard the compression and use the raw file source instead.

This leave a last problem, how do you know which method to use. We now find ourselves moving back to a formal file format like .ZIP, .7Z, etc. My basic solution is to always add a 4-byte prefix to the binary stream 0x0000 means RAW, while 0x0001 means GZIP (the rest of the padding if future-proofing). Nutz.

This deep-delve into compression lead me to the 7Zip SDK which as a complete implementation on C# amongst others. Wow! Clearly, I would need to implement this...

Peter Bromberg made a great start. In the end all I did was to simplify what he had build and create a clean implementation I called 7Zip.SDK.Clean ( [7Zip.SDK.Clean.zip])

For the pleasure of the code. After five or six complete rewrites, and a whole bunch of totally ludicrous over-engineering my sokoban pet project is finally in good alpha state, perhaps even beta.

I also made a wikipedia entry. The project is hosted by SourceForge.

Download the SokoSolve Game, which has advanced mouse movement (crate dragging). Give it a try and let me know if it is any good; or better yet how I could make it better.

So far the thing I am most proud of is the solver which after a week of effort is starting to take shape and yield good results.

Check it out:

• www.sokosolve.net
• SokoSolve @ Sourceforge

Introduction

Creating a blog from scratch is not a clever thing to do. There are too many out there already (DasBlog , Blogger , WordPress ). Blog is the new Hello World. However, having played with a couple of them over the years, I found myself frustrated - some being too simple, some too complex. In the end I wanted a blog specific to my evolving requirements. I also wanted an online note-taking system that would allow me to categorise my personal notes in a hierarchical/journal/wiki structure; and then to move a select few into a public blog format.

History

• 02-Apr-2009 > Initial Version
• Sourced my layout from Ars Technica -- thanks :-)
• Sourced my CCS -- thanks :-)
• Created my own internal Article XML format
• Created my own breed of wiki-syntax from wikipedia
• Integrated a magnificent JavaScript Syntax Highlighter from alexgorbatchev.com
• 03-Apr-2009
• Added Attachments for code, images
  
  Fig. 001 -- Test Image from ARS Tech
  , zip [SourceHighlighter.zip], See Content Disposition
• 04-Apr-2009 > Alpha Version
• Content-Disposition for Attachment Downloads
• Browse By Tag
• Indented Lists prefix with oo and ooo
• Article Single View
• Push blog-roll to DomainRoot
• Login & Security with Cookies.
• I tried to use Response.Cookie.Add(new HttpCookie(...)) but failed while setting it directly worked Response.Cookies["XXX"] = "YYY";
• Deployment Scripts and Cleanup
• Options for INLINE images { Frame }
• Inline Code Snippets
• Google analytics
• URL Rewriting also for Articles IDs; Article.aspx?ID=XXX -> /Article/XXX
• 9 to 14-Apr-2009
• Refactor Visibility to Channels, updated security appropriately
• Admin Features
• Edit Article
• Rename Article files by ArticleID
• RSS syndication, with the aid of the open source library rss-net
• 15, 16-Apr-2009 to present
• Encrypt/7Zip local files
• Added google.com verification
• Add a favicon
• Filter pages for each page (Default, Browse*, Article, RSS)
• Limit the number of items on the home page to 20
• Google search optimisations
• Niggling bugs (missing images, attachments missing, etc)
• Article Comments
• Logoff
• Journal Features
• Journal View (Twitter-style, but integrated to blog and wiki)
• Added Twitter REST integration, a post to the journal of type=Twitter will automatically update twitter
• Added Search functionality

Nice to haves

• Multiple Renderers
• HTML
• Text
• WPF
• Summary View

• Web Services for ArticleRepository
• Security
• Support for large data, many files
• OpenID 
• Wiki Functionality
• Differentiate Wiki URI markup vs blog. Ie a wiki link will not contain the | found in normal links
• Wiki-links that do not exist can be created by clicking on them
• Wiki/Blog are stored seperately, can articles can be converted easily. They share the same XML format.
• When rendering a blog article, wiki-links are ignored.

Google Search Engine Optimisations

• Added link directly
• Added google.com verification
• Added: Add google.com sitemap
• Added URL Rewrites to capture the old blog URL from google. TODO: Test after deployed live
• Created a google analytics account
• Updated by Blogger account to point back to the new site
• Blogger used weblogs.com to ping, so I registered for that - the site says that I fail RSS validation.

Notes & Findings

• HOWTO make a UL-tag list display horizontally
• CSS padding vs. margin
  
  Fig. 003 -- CSS margin vs. padding
• C# DateTime Formats
• HOWTO encode HTML with a static function? HttpUtility.HtmlEncode("Encode This String");
• Added Google WebMaster Tools
• RSS Specification is maintained by the RSS Avisory Board, they also have an online RSS validation tool, which I currently fail
• I always forget how to escape/encode a System.String to the appropriate HTML-friendly string. Answer: Use HttpServerUtility.HtmlEncode the only proble being this is not a static method. You can retrieve it with {{{HttpContext.Current.Server.HtmlEncode}} but this will only work with a web app.
• Complete list of HTML escape characters
• URL Rewriting did not work. See
• Getting URL Rewriting working was not easy.
• Simple rewriting worked with the ASP.NET simple development web server, but not IIS6 when deployed. This is because only .ASPX (and a few others) are mapped to the ASP.NET ISAPI dll.
• http://weblogs.asp.net/scottgu/archive/2007/02/26/tip-trick-url-rewriting-with-asp-net.aspx
• How to Configure wildcard url rewriting in IIS6, IIS7
• Open Source URL Rewriter for .NET

Concept: Journal

• I want to be able to make twitter like updates (which may get posted to twitter automatically)
• I want to be able to jot down very fast notes (which may later become blog or wiki posts)
• I want to create a list of books, ordered by time using the book:// or book-name:// notation
• I want to create a link dump of interresting read (this should have the ability to retrieve the TITLE of the URL automatically)
• The site (new/changes to pages) should be published as news in the Journal

I started out with the idea of writing an acticle on Software Engineering and Architecture prerequisites. However, that is too large a topic and I don't feel qualified to give a definitive set. Instead, here are the concepts/theory that I consider important: Software Engineering and Architecture has, in my opinion, the following prerequisite's:

Software Engineering and Architecture

• UML - Unified Modeling Language
• RUP - Rational Unified Process
• XP - eXtreme Programming
• OOP - Object Oriented Programming
• Functional Programming (misunderstood)
• Design Patterns - Gang Of Four
• CMM - Capability Maturity Model - A pragmatic approach to chart an organization's development ability in terms of its stability/ability to reproduce success/maturity/development process. Most of the world sits at CMM-0, while Microsoft/IBM sit at CMM-4, and god sits at CMM-5.
• SOA - Service Oriented Architecture - The impact of web services, and automated services across organisational boundaries.
• AOP - Aspect Oriented Programming - I don't know of a short, or cute way of explaining this -- which is a good cute to say that I do not understand it
• Refactoring - To rework an existing chunk of code/design/architecture, in such a way as to increase it's elegance/speed/quality without increasing its functionality.
• TDD - Test Driven Development 
• Agile Development - SCRUMM 

In the past I have used this list as a basis for interviews.

Pragmatics

• Bi-Polar development world (Open Source & Java Soup vs. Microsoft Stack vs. The Old Stuff)

Concepts: "Tao or Philosophy of Software Engineering"

• Concept: Software Economics (Lack of Scarcity, Quality-Performance-Price triangle)
• Physical Law: Heisenberg's Uncertainty Principal
• Physical Law: 2nd Law of Thermodynamic
• Maths: Godels Proof: Deterministic World Flaw (Proof?)
• Physics: Occams Razor
• Concept: Emergence
• Maths: Chaos Theory and Emergence, as applied to Software Engineering
• Concept: Quality before Performance
• Science: Scientific Method, as applied to Software Engineering

I plan to return to these themes over time, and expand or explain the theme's theory and my personal philosophy of (Software Engineering, Software Architecture, hacking, etc ).

Visual studio (and its related tools) tends to leave a lot of temp files around. Mostly this is not a problem, however every now and again they are a nuisance:

• When you want to distribute your source code
• When you want to add the code to external source control (SubVersion)
• When you need the free space
• When you need to archive the project

Fig. 0 -- Screenshot

The basic rules for clean-up are:

• All directories (recursive): obj/*
• All directories (recursive): bin/*
• All directories (recursive): TestResults/*
• All files: *.sou
• All files: *.resharper
• All files: *.resharper.user
• All files: *.gpState

Oh, and I have been playing with WPF (Windows Presentation Foundation), which required .NET version 3.5.

Q: How to get the default value of a type? return default(myObj.GetType()) does not work.

A: Forums Say...

My Solution

public static object DefaultValueOfType(Type t)
{
    return t.IsValueType ? Activator.CreateInstance(t) : null;
}

[Source Slashdot] jexrand recommends an interview with John De Goes in which he argues: "The tools market is dead. Open source killed it." The software developer turned president of N-BRAIN explains the effect that open source has had on the developer tools market, and how this forced the company to release the personal edition of UNA free of charge. According to De Goes, selling a source-code editor, even a very good one, is all but impossible in the post-open source era, especially given that, "Some developers would rather quit their job than be forced to use a new editor or IDE." N-BRAIN's decision is but one in a string of similar announcements from tools companies announcing the free release of their previously commercial development tools.

Open-source is here to stay, and it is going to change the business model for a great many products. I regularly uses tools from both sides (NUnit, NCover, Reflector) and (Resharper). Both open source and commercial product have to be innovative -- but they tend to innovate in differenet ways and for different reasons.

Here is a list of development tools and project that I keep an eye on:

Operational

• log4net 
• NUnit  Alternatively Microsoft Quality Framework  from TFS 
• NHibernate  Alteratively LINQ to SQL  Entity Framework 
• Winsor Castle 
• ASP.NET MVC 

Interesting

• Web Platform Installer 

ReSharper is a .NET IDE plug-in to give more sophisticated editing, navigation, refactoring, and code generations.

Refactoring Definition

"Refactoring is the exercise of taking functional code, and while not increasing its functionality, increasing its quality" -- Debugging is therefore the act of altering the code to fix a specific bug, while not intrinsically increasing the quality of the code around it.

Oh, ReSharper, how do I love thee, let me count the ways: Refactoring should be a key skill of a software engineer. Once formal conception design is complete, there are always grey-areas or bad implementations to correct, or continuous improvement opportunities as we get a more complete understanding of how the software should be constructed.

The key concept, or rather precept, in refactoring is quality. To understand refactoring we therefore need to understand code quality (See article What is quality code?)

Refactoring Mantra

• 1st Mantra Of Refactoring -- If you are reading code, you should be planning how to refactor it.
• 2nd Mantra Of Refactoring -- Compiler-refactors, or automation-assisted refactors are King.
• 3rd Mantra Of Refactoring -- The single bane of refactoring is the introduced error.
• 3rd Mantras Corollary -- Unit tests and Refactoring is a match made in Heaven.
• 4th Mantra Of Refactoring -- If a bug is logged against a class, that class should be refactored.

Books

Refactoring is well covered as a key discipline/skill in print:

• Refactoring: Improving the Design of Existing Code
• Refactoring to Patterns
• Design Patterns: Elements of Reusable Object-Oriented Software a.k.a. Gang-Of-Four book!

Rules Of Thumb

• Refectoring should be explicitly listed on the project work breakdown.
• Refactoring should account for 10-30% of the effort during the construction and maintenance phases.
• Large-scale (non-compiler/automated) refactoring candidates should be listed, and then planned for project deadtime.
• Refactoring should be used as the key principal in Code Review sessions, altering code to discuss ideas and better quality implementations.
• All developers should engage in refactoring work this includes Architects, Leads, Senior and Junior developers.
• In general, a refactor that reduces total amount code, is normally successfull.
• The first tool of basic refactoring is the 'helper method'. As my mentor (Anthony Steele) once said "Forget about OO, must people don't understand procedural programming"... Meaning, that most developers do not leverage the power of basic procedural programming to reduce or hide complexity.
• Refactoring should, in general, reduce the 'contract' the code has with the outside world. That is to say, a piece of code, or a class, or a method, should expose the smallest amount of complexity to the outside world.

Refactoring is a team skill

Refactoring is a colaborative dialogue -- if the team is constanly refactoring, but without cohesion the results will reflect this, the code will be a patchwork of divergent ideas. In my code world view consistency is more important the correctness(within bounds of reason). It is better to all do the wrong thing the same way, that to sometimes to the right thing in many diferent ways. If you have exprience in larger project with more than one strong personality -- you will recognise the truth of it. In part this is true, because you can organise a large-scale refactor of a first case (consistently wrong), but not the second case (partially right).

Back to the key point: Refactoring is a colaborative dialogue. The team should be constantly discussing posibilities for refactoring, in the short, medium and long term. Easy, medium and extreme refactors. The ongoing refactoring dialog has many advantages:

• It keeps things fun and interresting
• It serves to establish a common lingo/terminology and common design patterns (and anti-patterns!)
• It introduces new ideas for design and implementation, and provides a basis for evaluating them (What would the new refactor give us? What is the cost? What is the Risk?).

Evaluating a Refactor

In very time terms, before we start a refactor we always ask the same questions:

• What would the new refactor give us? This is the expected code quality increase. We evaluate the current and future benefits.
• What is the cost? Essentially, man hours; but also includes the effort needed to motivated the required time from management and the rest of the team. It is important to include the oppurtunity cost of more features/functionalty what could have been implemented.
• What is the risk? Knock-on bugs!

Link Dump -- Article and Links of Interrest, but as yet unfiled.

Unfiled

• Distributed Apps for Azure
• Garbage Collection Types in .NET
• MSDN Architecture Journal
• Cisco Unified Comms Primer
• wikip://ASP.NET_MVC_Framework
• wikip://Representational_State_Transfer
• Static Code Analysis
• SOA
• Services QA
• Entity Framework
• Type Inference
• Software Design
• Test Tool
• Code Metrics
• MSMQ Out Of Resources FAQ
• Concurrency in .NET
• Free Certs
• How to get trace/logging for WF Rules
• An operating system written in C#, called "Cosmos"
• http://invalidlogic.com/software/blogging-apps-are-the-new-hello-world/ 
• http://blogs.conchango.com/anthonysteele/archive/2009/03/29/social-web-beyond-walled-gardens.aspx  
• http://nortools.blogspot.com/2007/12/progressive-enhancement.html  
• http://www.sitepoint.com/blogs/2009/04/06/web-users-are-not-more-productive/  
• Construct your own touch table
• Periodic Table of Operators
• C# to JavaScript research
• http://alistair.cockburn.us/Software+development+as+a+cooperative+game

Misc

• http://groups.msn.com/windowsscript/utilities.msnw
• http://www.ebuyer.com/product/155584

The difference between Engineers and Programmers

An IT software developer and a mechanical engineer are given the same problem to solve in software, with the same deadlines. On the day of the final deadline, the software developer walks in with his cup of coffee, looking a little tired. He delivers the software along with the standard speech of "maintenance phases". He concludes by saying that he will be back on Monday to tackle the bugs.

Just after the software developer leaves; a haggard, drained-looking fellow stumbles through the door. It is the mechanical engineer. He delivers the software, and makes to turn around and leave and get some sleep. He is immediately stopped and asked: "What about the bugs?" A confused look moves across the engineer’s face. "What bugs? I assumed that you would not accept any defects in the software." He stumbles out of the office…

The game has a *very* simple set of laws, from these laws amazing amounts of complexity can be generated -- and not just noise. Life can be used to create a computer within itself, which may in turn be used to create a computer within itself, which may be used in turn...

• Introduction
• Advanced
• Holy Gail is a Turing Machine  encoded as a Turing Game Of Life Pattern
• Space Ships