DevOps Zone is brought to you in partnership with:

Mr. Lott has been involved in over 70 software development projects in a career that spans 30 years. He has worked in the capacity of internet strategist, software architect, project leader, DBA, programmer. Since 1993 he has been focused on data warehousing and the associated e-business architectures that make the right data available to the right people to support their business decision-making. Steven is a DZone MVB and is not an employee of DZone and has posted 136 posts at DZone. You can read more from them at their website. View Full User Profile

"Strict" Unit Testing -- Everything In Isolation Is Too Much Work

09.24.2011
| 11859 views |
  • submit to reddit
Folks like to claim that unit testing absolutely requires each class be tested in isolation using mocks for all dependencies.  This is a noble aspiration, but doesn't work out perfectly well in Python.


First, "unit" is intentionally vague.  It could be a class, a function, a module or a package.  It's "unit" of code.  Anything could be considered a "unit".


Second--and more important--the extensive mocking isn't fully appropriate for Python programming.  Mocks are very helpful in statically-typed languages where you must be very fussy about assuring that all of the interface definitions are carefully matched up properly. 


In Python, duck typing allows a mock to be defined quite trivially.  A mock library isn't terribly helpful, since it doesn't reduce the code volume or complexity in any meaningful way.


Dependencies without Injection


The larger issue with trying to unit test in Python with mock objects is the impact of change.


We have some class with an interface.



class AppFeature( object ):     def app_method( self, anotherObject ):         etc.
class AnotherClass( object ):     def another_method( self ):         etc.

We've properly used dependency injection to make AppFeature depend on an instance of AnotherClass.  This means that we're supposed to create a mock of AnotherClass to test the AppFeature.


class MockAnotherClass( object ):     def another_method( self ):         etc.

In Python, this mock isn't a best practice.  It can be helpful.  But adding a mock can also be confusing and misleading.


Refactoring Scenario

Consider the situation where we're refactoring and change the interface to AnotherClass.  We modify another_method to take an additional argument, for example.


How many mocks do we have?  How many need to be changed?  What happens when we miss one of the mocks and have the mysterious Isolated Test Failure?  

While we can use a naming convention and grep to locate the mocks, this can (and does) get murky when we've got a mock that replaces a complex cluster of objects with a simple Facade for testing purposes.  Now, we've got a mock that doesn't trivially replace the mocked class.

Alternative: Less Strict Mocking

In Python--and other duck typing languages--a less mock-heavy approach seems more productive.  The goal of testing every class in isolation surrounded by mocks needs to be relaxed.  A more helpful approach is to work up through the layers.

  1. Test the "low-level" classes--those with few or no dependencies--in isolation.  This is easy because they're already isolated by design.
  2. The classes which depend on these low-level classes can simply use the low-level classes without shame or embarrassment.  The low-level classes work.  Higher-level classes can depend on them.  It's okay.
  3. In some cases, mocks are required for particularly complex or difficult classes.  Nothing is wrong with mocks.  But fussy overuse of mocks does create additional work.
The benefit of this is

  • The layered architecture is tested the way it's actually used.  The low-level classes are tested in isolation as well as being tested in conjunction with the classes that depend on them.
  • It's easier to refactor.  The design changes aren't propagated into mocks.
  • Layer boundaries can be more strictly enforced.  Circularities are exposed in a more useful way through the dependencies and layered testing.
We need to still work out proper dependency injection.  If we try to mock every dependency, we are forced to confront every dependency in glorious detail.  If we don't mock every single dependency, we can slide by without properly isolating our design.
References
Published at DZone with permission of Steven Lott, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Andrew Spencer replied on Sun, 2011/09/25 - 3:14am

Your arguments are persuasive.  To my mind, the key to a unit test - for whatever definition of "unit" - is that if the test fails it should fail due to an error inside and not outside the unit.  Tests are a tool not only for detecting that an error exists, but also for locating it. I wrote about this (from a Java perspective) a little while ago here on DZone.

A corollary to your suggested "building-an-onion" style of testing is that, if a unit depends on another unit, its test should depend on that other unit's test. In the event of failure in a low-level class, the build would skip tests of dependent classes, which might give false positives (is a test failure a "positive" or a "negative"?).

This would avoid the distracting situation where you make 1 mistake in a low-level class and suddenly you get 5 or 10 test failures in completely different layers (not to mention in the integration tests).

On your other point, it's funny but it had never occurred to me that mock generators were an artefact of using a statically-typed language...

Attila Magyar replied on Sun, 2011/09/25 - 1:27pm

I don't agree that mocking is not very useful in the dynamically typed world. Mocks are used to verify the interaction between objects, not just stubs out return values. For example if you want to check whether a method on a mock was called maximum 3 times with an argument which contains the "abc" substring, then your hand written mock won't be very concise and readable. Readability affects the maintainability which is one of your main concern in this article. 
For example in groovy (with spock) you can write this:

(1..3) * m.someMethod( {it.contains("abc")} )

But I think what really can reduce the maintainability is applying OO principles like single responsibility, and writing small and cohesive classes. Classes which are focused are easy to test, because they have one role, and only a few dependencies. Nevertheless the overall design of the software will be better.

Mladen Girazovski replied on Mon, 2011/09/26 - 1:37am

How many mocks do we have?  How many need to be changed?  What happens when we miss one of the mocks and have the mysterious Isolated Test Failure?   

Don't know about Python,

but in Java, you'd need to structure your tests, avoid redundancy and not depend on unimportant implementation details, ie if 30 tests are using a constructor directly without actually testing it, guess what happens if you change the constructor?

Also, Mocks in Java should not in every test be created from scratch, a change to the interface/behaviour would break many test otherwise even without directly depdending on it.

Use factory methods or Builders for your mocks, only modify the behaviour where necessary.

Gerard Meszaros calls the pattern "Configurable Test Double"

http://xunitpatterns.com/Configurable%20Test%20Double.html

 

Nabeel Manara replied on Fri, 2012/01/27 - 10:16am

You need to mock out to a level which
- allows you to not have any side effects when executing tests
- allows you to immediately understand which is the affected "unit" when a test fails without necessarily have to fire up the debugger etc...

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.