Thursday, 27 December 2007

My thoughts about FiveAM - Common Lisp testing framework

I spent all day today adding some tests as I needed to do some refactoring. I wanted to choose a CL testing framework and came across Phil Gregory's great post: Common Lisp Testing Frameworks. I read through his review and as you can tell by the title, I chose to go with FiveAM. I won't go over what Phil covered in his post except to say that FiveAM has what I expect in a testing framework and more.

You can jump to the the bottom line.

Simple tests are defined simply:

(5am:test my-test-case
(5am::is (= 2 (1+ 1)))

FiveAM has test dependencies:

(5am:test (my-other-test-case :depends-on my-test-case)
(5am::is (= 3 (1+ (1+ 1)))))

FiveAM allows you to group tests using test suites:

(def-suite arith-tests :description "Arithmetic tests")
(in-suite arith-tests)
(... above tests ...)

However, the grouping is only useful for selecting which tests to run. You can't (for example) make one set of tests dependent on another set of tests. This makes the feature only useful for organizational purposes. It isn't a deal-killer especially since you can write a function to work around this limitation like the one below:

(defun add-test-dependency (a b)
"Make test-suite a depend on test-suite b by making every test in a depend
on every test in b"
(let ((suite-a (safe-get-test a))
(suite-b (safe-get-test b))
suite-a-tests suite-b-tests)
(maphash #'(lambda (sym obj)
(declare (ignore obj))
(push sym suite-b-tests))
(5am::tests suite-b))
(maphash #'(lambda (sym obj)
(declare (ignore sym))
(push obj suite-a-tests))
(5am::tests suite-a))
(loop for test-name in suite-a-tests
(let* ((test (safe-get-test test-name))
(depends-on (5am::depends-on test)))
(let ((new-depends-on
(if depends-on
`(and ,depends-on ,suite-b-tests)
`(and ,@suite-b-tests))))
(print test)
(print new-depends-on)
(setf (5am::depends-on test) new-depends-on))))))

In hindsight, a smarter way to do this would be to introduce a pseudo-test in b that depended on every test in b and then every test in a would depend on this pseudo-test. Ah well.

But my favourite feature is the fact that FiveAM lets you generate samples from a distribution of inputs and feed them into your functions to test. Here is one stupid example:

(test encode-password
"Passwords are encoded as ALGO$SALT$HASHED-PASSWORD. This code tests
that the structure is correct and that the components of the encoding
pass sanity checks. In the case of encode-password the salt is randomly
(for-all ((raw-password (gen-string
:length (gen-integer :min 5 :max 10)
:elements (gen-character :code-limit (char-code #\~)
:alphanumericp #'alphanumericp
:code (gen-integer :min (char-code #\Space)
:max (char-code #\~))))))
(let ((encoded-pw (myapp::encode-password raw-password)))
(validate-encoded-password encoded-pw))))

The for-all macro takes a list of generators and iterates through a set of samples for all the represented values. This is done through the use of generators (gen-string and friends.) In this case, I am iterating through a distribution of strings that generates a string between 5 and 10 characters long the contents of which are in the "interesting" ASCII character range. The body of the for-all macro is dedicated to encoding the password and validating that the encoding is sane. Although it isn't important, validate-encoded-password looks like:

(defun validate-encoded-password (encoded-pw)
(destructuring-bind (algo salt hash)
(split-sequence:split-sequence #\$ encoded-pw)
(is (string= "md5" algo))
(is (= 5 (length salt)))
(is (= 32 (length hash)))))

The Bottom Line

5am is very suitable for testing in CL but test groups should really have the ability to take part in dependencies.


Anonymous said...

I also wrote about FiveAM a year ago:

Sohail Somani said...

Thanks! Personally, I would prefer that FiveAM not compile every time. I always just get asdf to recompile whatever I've changed and its dependents anyway.

Anonymous said...

Thanks very much for this, found your add-test-dependency code very useful! (just needed to change 'safe-get-test' to 'get-test')