OutOfMemoryException in long running NANT build (AKA StringBuilder is broken)

On my current project we have experienced an interesting problem with our build. There's nothing particularly strange about our setup:

Technology: .NET (C#)

Build: NANT 0.90 + msbuild

Test libs: Nunit & Selenium 2 (webdriver API)

Our build does a fairly usual set of tasks:

- compile projects

- run unit tests

- deploy to IIS

- run integration tests

- run acceptance tests

The project has been running for a long time (4+ years) and so has a large number of tests and a codebase destributed over 30+ Visual Studio Projects. Half way through the current release we encountered a problem where our build would throw an OutOfMemory exception.

After a LOT of investigation we discovered this post which explains exactly what we were seeing. Basically parts of .NET use the StringBuilder for creating and manipulating strings and the problem occurs when the StringBuilder is used with large strings (large = big enough to be put on the Large Object Heap). The best part is that there is no (easy) workaround/fix.

In the end the only way we could get this to work was by splitting our build into two separate builds, we now first run a build that compiles and runs unit tests. Then the second build uses the build artefacts of the first build to run integration and acceptance tests.