Back in May of 2007, Linus Torvolds did a talk at Google about git. Specifically he tended to reiterate the point on how your workflow changes dramatically when you can perform a merge or get new code from other people in under a second. Merging multiple branches into your own branch without remembering revision numbers and knowing that your history is in tact, being able to view the entire history of a project without an internet connection, and just being able to work with your version control system in sub-second time can change how you work with your code. You become willing to make branches because it's so cheap and easy. You can try out new things quickly and have an easy way to toss them if it doesn't work out, or merge back parts if it does. I think that the philosophy of really fast version control and its effects on your productivity are quite real, and very important. I also think that the same mentality can and should be applied to your build process.
[caption id="" align="aligncenter" width="200" caption="An example of how development might work via a revision control system such as git. I just needed a picture. Image provided by wikimedia commons."][/caption]
Of course the first thing you might think is "There is no way I'm going to get my build down to under a second." And for anything non-trivial, you'd be right. However, I think having a goal of a build plus unit tests in under 20 seconds is not an unreasonable demand. Even better if it's under 10 seconds. But projects can get really large and involve thousands of files and a similar set of thousands of unit tests. How can you build all of that in under 20 seconds? The answer is: you don't. Do I sound contradictory? I'm not, but you might have to change how you think about your project.
The main idea behind getting a build under 20 seconds is that it will allow you to compile what you're working on and run all of your unit tests so that you can test your changes and see if anything breaks. If your build takes several minutes you're not going to want to build and run your unit tests until your done. This makes doing Test Driven Development a lot more difficult because if you don't run your tests constantly, you're probably not designing around your tests passing. You might write less tests, have a less elegant design, and ultimately produce poorer quality code. But still, a large project with hundreds of thousands of lines of code isn't going to even compile, much less run all of the unit tests in under 20 seconds, so what do you do?
As with any development problem, you break it down into a smaller and more manageable chunk. Instead of writing a giant application with thousands of different files and unit tests, you write more small libraries and services that hold maybe a few dozen or hundred files and tests that contain decoupled and independent structures. These smaller libraries and services can have their own separate build process as they are all separate small projects in and of themselves. That way you can write just the unit tests relating to those small projects and cut your build times down to under 20 seconds.
Now granted that's only the build time for one module, but all of them together can still take several minutes. But that's ok. Most bugs and development that you're working on will only touch one or maybe two modules at a time, causing you to run the build process for just those modules to fix the problem or add the feature. There's no need to compile and run unit tests for a lot of other systems that are completely unrelated to what you're working on. As for if your change will break things in other packages, that's what integration tests are for.
Besides, if you separate your code into smaller libraries and services, you'll be providing a natural way of decoupling systems and making them no longer become dependent on each other. If you write good strong unit tests that mock any external entities that are called or would call your service or library, those tests enforce the contract that your other systems will require. This will ensure that changes you make will not break other systems. If the changes you make do break the systems, you just create or modify your unit tests to fail due to this breakage so that you know why your system should output what it does or call who it does.
By driving down your build times by having sections be in separate projects, you will have faster turnaround and feedback on the work that you are doing right now. You can run a build right after changing a single line of code just to make sure, and 20 seconds isn't too long of a wait to get a response. It's almost like a small break, but no quite long enough that you start trolling slashdot and forget you're building something.