ruby-core

On Sat, 14 Jan 2006, mathew wrote:

> Consider '+'. Requirements are that it add any two things which constitute
> numbers. Trouble is, you can't even test every possible pair of floating point
> numbers, let alone the integers and rationals. Any test of N pairs that you do
> will be statistically insignificant.

You are not testing number theory, you are testing software.  All
you need to show is that all parts of the code operate correcly.
Sometimes code will fail as a result of input from another part,
hence the doubling in testing for every if...else...end.  Your
argument sounds to me like "A car cannot be said to work correctly
until it has driven on overy road." Yes, you have to test all the
gears. Yes, you must test the brakes.  Yes you must calibrate the
speedometer.  Software, except the Monte Carlo sort, is not
statistical in nature.

> 
> > So, the acceptance tests used by customers to confirm that
> > requirements are met may be written in a documentation tool from which
> > test code is generated.
> 
> Right. Unit tests are not the requirements; they are acceptance tests used to
> confirm that the requirements have been met. There's a subtle but very
> important difference.

Unit test, TTBOMK are not what is conventionally known as acceptance
tests.  Acceptance tests operate on the package as a whole. They are
to determine whether the customer will accept the whole package. Unit
tests operate on single methods.
> 
> Similarly, requirements are not specifications. The specifications are
> (hopefully) derived from the requirements, but they typically describe a
> superset of the functionality strictly required, and almost always describe a
> superset of the functionality tested.

Which is fair enough.  Could a word processor be tested on all
documents?
> 
> For example:
> 
> Requirements: method to add two vectors of Floats.
> 
> Specification (as implemented): method adds two vectors of objects which
> support '+'.

Depending on the language, that could be wrong: overgeneralization
is as much an error as undergeneralisation.  Example vividly
recalled from being taught to program: The Christmas execise was to
write a stock control program in Pascal.  Many people went to a lot
of trouble to handle common input errors.  We were explicitly told
this was wrong, because it was not in the specification.  I don't
remmeber being told to "fail early" at the time, though.


> Unit tests: take a sample file of vectors of floats, add them in pairs,
> compare against the file of correct output.
> 
> All perfectly reasonable. However, the unit tests do not tell you whether
> anything other than floats will work with the method. That's perfectly
> reasonable, because working with something other than floats wasn't a
> requirement. However, it's also perfectly reasonable that the first
> implementation happens to work with integers, rationals, and so on--because

It may work, but it is an unsupported feature.

> the power of Ruby is that generic programming doesn't take extra effort. In
> fact, it would take extra effort to prevent the code from working with other
> types.
> 
> So, if you only provide unit tests and no documentation, users have to guess.
> They might look at the unit tests, and guess that the method only works with
> floats. End result: code bloat because they go and unnecessarily find or
> implement an integer vector add.

If one is expected to write tests for all types a generic method
will support, then it is making a rod for one's own back.  There
isn't time.  Besides, once they see what the unit tests say about
the method, the source will be more comprehensible to them, and if
they read it they can see it is generic.
> 
> Or, they might look at the code, and see that it works with Complex. Then they
> go and write a ton of scientific visualization software that relies on the
> fact. Then you want to refactor the method to use CBLAS for speed, and it
> breaks NASA, and there are howls of anguish.

Yes.  It happens.  You have conflicting requirements which are
unknown.  That is reality.  That is why having a requirements
document, having a specification document, all developed at a phase
in the beginning of a project is disliked by those who do Agile
development.  Often people don't know what they want until they've
tried some version(s).. That even happens with languages.  K&R C??
Remember that? :-)  I have read something which I can't find on the
web about the authors of awk being surprised about the size of
programs it was used for.  And we are told Rite is in gestation.
> 
> Hopefully I'm getting the problem across better this time around. I am *not*
> discussing unit tests vs documentation, or unit tests vs contracts, or unit
> tests vs anything else, or documentation vs anything else. At no point have I
> said that unit tests are anything other than essential. My point is simply
> that a documented API is *also* essential, and at a minimum it should describe
> the functionality that is *supported*, as opposed to that which is tested or

but in your example case Floats were the only supported API, and the
unit tests only documented that.

> that which happens to work. Leaving people to guess the supported API from the
> unit tests (or worse, the code) is a flawed approach which has led to problems
> refactoring.
> 
> In a nutshell:
> 
> documentation = what is *supported* or *supposed to work*
> unit tests = what is *actually tested*

it may be argued that untested cases are not deemed to be supported.
Usually people are grateful when someome submits a bunch of tests,
some that fail.  It helps find unknown bugs, which documentation
doesn't. :-)  Documentation is necessary of course.  Unit tests are
not light reading.  I've never seen a book of them at an airport,
anyway.
> code = what *happens to work*
> 
> What is *tested* != what is *supported*, because you can't test everything. I

the point is that it should be sufficiently close, surely?
And you can test for failure cases as well.

> can't feasibly test every possible RSS feed against my RSS parser, for
> example.

The RSS grammar is not infinite, even if it can produce an infinite
variety of productions.
> 
> What is *supported* != what *happens to work*, not least because Ruby code
> tends to be generic by default. Even ignoring the generic issue, though, just
> because my RSS parser accepts a particular malformed feed today, doesn't mean
> I'm necessarily going support that malformed feed after I refactor the parser.
> 
> And when I'm using libraries, though I do also care about what's tested, my
> primary concern is what's supported. I care about knowing that something I use
> today won't change in an unexpected way and break my application. e.g. Rake,
> Yaml, ...

That would break regression tests, some of which will be unit tests.
> 
> 
> mathew
> [ And as usual, as soon as I get into a discussion, the mailing list software
> stops sending me mail. ]
> 
        Hugh

Thread

Prev Next

In This Thread

Prev Next