The other day, I came across a very interesting paper on continuous testing during development. In it, the authors found that program correctness for a group of students could be predicted based on whether the students used continuous or manual testing. Continuous testing is running your compilation and project tests as you save files locally, as opposed to continuous integration which usually works as you check in code. Manual testing is having the student run the test suite manually. Those students who used continuous testing were a few times more likely to finish the project and had less mistakes.
The key is that after saving a file, using the built-in Emacs hooks, I launch a compile process. The hook only does so if the file being saved is located in the project directory by looking for a string in the filename, "tmp" in this case.
Here is a screencast of the behaviour with the code in the left side of the split. In the right split, I am editing a script (/tmp/runtest) which represents the test suite. The "project" is located in /tmp. The video shows me saving the test suite file twice. Once with no errors and the second time with an error. In the first case, the compilation buffer goes away once the test suite has run, which keeps things tidy. In the second case, the compilation buffer stays around because an error occurred in the "test suite".
This specific setup works best with a compilation and test phase which runs relatively fast. To make it work for longer test suites, you'd need to probably modify the test-command code to kill the compilation first. You'd probably also want to modify the after-save-hook to use some kind of timer after which you start the compile.
There are lots of things which I'd like to work better, but it works OK for me now.
Let me know how it works out for you if you try it out. The code is here.