Testing a Side-Effecting Haskell Monad

In this post I’m going to continue discussing testing CHP, this time focusing on the CHP monad. The monad is a bit complex (and indeed, there is a subtle bug in the current version of CHP to do with poison-handling) because it has several semantic aspects:

• it is an error-handling/short-circuiting monad (because it must obey the poison semantics),
• it has some extra semantics to do with choice that treats the leading action differently,
• underneath it all, there is the IO monad, which must be lifted correctly.

CHP is a monad with side-effects: quite aside from the communication with parallel processes, the presence of lifted IO means that it can have arbitrary side-effects. This means, if my terminology is correct, that we need a check for operational equality rather than denotational. Let’s explore this by considering testing an error-handling monad.

Testing An Error-Handling Monad

The Either type can be used as an error-handling monad by using the Left constructor to hold an error, and the Right constructor to hold a successful return. One property that should hold is that Left e >>= k == Left e. That’s an actual Haskell equality; we can test that property for any function k, and use actual equality on the result.

It’s also possible to use a similar error-handling monad transformer. The transformers library offers an ErrorT e m a type that allows you to wrap a monad “m”, using error type “e”, with successful returns of type “a” (with a function throwError :: e -> ErrorT e m a for introducing errors). We can test this using the Identity monad as type “m”, which would make sure the code was pure. But the semantics of this error transformer involve what actions occur in the error monad. For example, runErrorT (m >> throwError e >> n) should be equivalent to m >> return (Left e), but usually m (which is a monadic action) cannot be directly compared for equality.

Setting this in the IO monad makes it all much clearer: runErrorT (lift (putStrLn "Hello") >> throwError e >> lift (putStrLn "Goodbye")) should print Hello on the screen, and then finish with an error — and not print Goodbye. Testing this is difficult because we need to have a harness that can check for which side-effects occurred, making sure the right ones (e.g. printing Hello) did happen, and the wrong ones (e.g. printing Goodbye) didn’t happen.

This is exactly the situation that the CHP monad is in; we always have IO beneath the CHP monad (which we can’t swap out for a more easily testable monad!), so we must test equality of code by checking which side effects occurred. We need to check that liftIO_CHP (putStrLn "Hello") >> throwPoison >> liftIO_CHP (putStrLn "Goodbye") prints Hello then throws poison, and doesn’t print Goodbye. Printing text is a difficult side-effect to track in a test harness, but the nice thing about IO is that it has a whole host of side-effects to choose from! So I test by observing alterations to a Software Transactional Memory (STM) variable (TVar) — therefore my test inputs are of type TVar String -> CHP a.

Testing CHP’s Error-Handling

I use these two functions as a harness to execute the CHP code:

runCHPEx :: CHP a -> IO (Either SomeException a)
runCHPEx m = (Right <$> runCHP m) `C.catches`
    [Handler (\(e::UncaughtPoisonException) -> return $ Left $ toException e)
    ,Handler (\(e::ErrorCall) -> return $ Left $ toException e)]


runCode :: (TVar String -> CHP a) -> IO (String, Either SomeException a)
runCode m = do
  tv <- newTVarIO ""
  r <- runCHPEx (m tv)
  s <- atomically (readTVar tv)
  return (s, r)

The runCode function gives back (in the IO monad) the String that was the final value of the TVar, and the result of running the CHP computation — either an uncaught poison/error exception or an actual return value. (One planned change to CHP semantics in version 3.0.0 is that uncaught poison gets translated into an exception, rather than giving runCHP a Maybe-wrapped return.) Here’s some example HUnit tests (with some useful helper functions):

obsTestPoison :: Test
obsTestPoison = TestList
  [("ab", Right 'b') ==* \tv -> tv!'a' >> tv!'b'
  ,("", Left UncaughtPoisonException) ==* const throwPoison
  ,("a", Left UncaughtPoisonException) ==* \tv -> tv!'a' >> throwPoison
  ]
  where
    (!) :: TVar String -> Char -> CHP Char
    (!) tv c = c <$ (liftIO_CHP $ atomically $
                       readTVar tv >>= writeTVar tv . (++[c]))

    (==*) :: (String, Either UncaughtPoisonException Char)
          -> (TVar String -> CHP Char) -> Test
    (==*) x m = TestCase $ runCode m >>= assertEqual "" (onLeftSnd Just x)
                  . onLeftSnd fromException
      where
        onLeftSnd :: (b -> c) -> (a, Either b d) -> (a, Either c d)
        onLeftSnd f (x, Left y) = (x, Left $ f y)
        onLeftSnd f (x, Right y) = (x, Right y)

The left-hand side of the starred equality is the expected result. The right-hand side is the code (which is of type TVar String -> CHP Char) which should produce that result.

So by observing deliberately-placed side-effects in the code, we can check for equality in a side-effecting monad. These unit tests aren’t the only way I’m testing my monad though — in my next post, I’ll build on this to form a more advanced property-based testing technique.