ruby-core

*Hugh Sasse *<hgs dmu.ac.uk>:
> On Sat, 14 Jan 2006, mathew wrote:
>
> > Consider '+'. Requirements are that it add any two things which constitute
> > numbers. Trouble is, you can't even test every possible pair of floating point
> > numbers, let alone the integers and rationals. Any test of N pairs that you do
> > will be statistically insignificant.
>
> You are not testing number theory, you are testing software.  All
> you need to show is that all parts of the code operate correcly.
>   
Yes, and how do you do that?

The world was littered with pieces of date code that had been tested and 
shown to operate correctly.

Until the year exceeded 1999, that is.

> > For example:
> > 
> > Requirements: method to add two vectors of Floats.
> > 
> > Specification (as implemented): method adds two vectors of objects which
> > support '+'.
>
> Depending on the language, that could be wrong: overgeneralization
> is as much an error as undergeneralisation.
Yes, but we're talking about Ruby here, so let's not go off on a tangent.

> However, it's also perfectly reasonable that the first
> > implementation happens to work with integers, rationals, and so on--because
>
> It may work, but it is an unsupported feature.
Right. And my entire point is that without documentation, you don't know 
which features are truly unsupported, as opposed to merely 
insufficiently unit tested. And without knowing that, you can't refactor 
effectively.

> > Or, they might look at the code, and see that it works with Complex. Then they
> > go and write a ton of scientific visualization software that relies on the
> > fact. Then you want to refactor the method to use CBLAS for speed, and it
> > breaks NASA, and there are howls of anguish.
>
> Yes.  It happens.  You have conflicting requirements which are
> unknown.  That is reality.  That is why having a requirements
> document, having a specification document, all developed at a phase
> in the beginning of a project is disliked by those who do Agile
> development.
Which is another irrelevant tangent. I'm a strong *proponent* of agile 
development. I'm not suggesting anyone should write the spec or 
requirements before they write the code.

> but in your example case Floats were the only supported API, and the
> unit tests only documented that.
No, the unit tests did not document that. The fact that there is no unit 
test which tests set-of-conditions X, does not mean that X is not a 
supported set of conditions. It cannot, because it is infeasible to unit 
test every possible set of conditions a piece of code might be presented 
with.

Think of Y2K. Lots of code didn't unit test with years over 2000. Did 
that mean years over 2000 were not supported dates? No, it meant the 
unit tests were inadequate and the code was buggy and needed to be 
fixed. We didn't go back and tell people "tough luck, you should have 
noticed that the year 2000 wasn't tested, therefore it isn't supported, 
go find another date library".

> it may be argued that untested cases are not deemed to be supported.
>   
Which isn't a viable argument, as you can't test (for example) every 
possible case of adding two floating point numbers.

> > What is *tested* != what is *supported*, because you can't test everything. I
>
> the point is that it should be sufficiently close, surely?
>   
It would be nice if it could be, but in reality it's not. Perhaps its my 
bias as a developer doing primarily web and systems integration work; in 
my world, unpredictable data variations happen all the time. You learn 
to write code which is as flexible and tolerant of errors as possible, 
and accept that it will be used with data that doesn't look like the 
data in your tests.

And that's assuming a developer who's conscientious enough to write 
hundreds or even thousands of test cases, in order to get sufficient 
coverage.

> I care about knowing that something I use
> > today won't change in an unexpected way and break my application. e.g. Rake,
> > Yaml, ...
>
> That would break regression tests, some of which will be unit tests.
So why weren't the problems with Rake and Yaml picked up in time?


mathew

Thread

Prev Next

In This Thread

Prev Next