42 points alpaylan 19 hours ago 13 comments

jruohonen 2 hours ago | parent

I think the alleged dictum between functional and non-functional is the biggest conceptual, theoretical, or even philosophical issue.

weinzierl 1 hour ago | parent

This was also my first thought when reading OP. It is not restricted to testing even. The whole development process has this distinction (e.g. functional vs non-functional requirements) and it hurts more than it helps.

gls2ro 1 hour ago | parent

I think this is an important distinction that is also real when thinking about verification: verifying function can be boolean or lets say limited to a simple set while verifying non functional properties means mapping to a more continous interval and so it is a qualitative decision.

gzell 1 hour ago | parent

Always enjoyable for us aged s/w engineers to see the new generations relearn the fundamentals.

wreath 40 minutes ago | parent

I dont think this will last much longer. Looking around, i see most engineers not being interested in anything other than AI related stuff. Certainly true for the junior ones

andsoitis 27 minutes ago | parent

AI opens up whole new ways of validating that systems behave correctly or to find issues.

gls2ro 22 minutes ago | parent

Maybe or maybe now we can do proper testing.

Most of the testing techniques are baed on groups/set and graph theories while test data is based on sampling.

So now we really have the chance to model verification and validation even better

ozim 1 hour ago | parent

I would like "non-functional testing" to be called "cross-functional testing".

g-b-r 1 hour ago | parent

I recall finding different definitions in every book about testing I read.

I think most of the confusion came from people suddenly deciding that a unit test can't test less than a whole object, in OOP.

smitty1e 29 minutes ago | parent

Mad Men meme:

"Testing, but applied to terminology across publications."

gfalcao 30 minutes ago | parent

great infographic

sfeizi 18 minutes ago | parent

does vocabulary matter anymore as coding agents can make interpretations?

jerf 9 minutes ago | parent

I think the problem with testing vocabulary is that the field is just too continuous. We have all sorts of situations in the world where there is no sharp line, yet we have terminology for various chunks of the possibility space even so because there are still some sort of clump in the space that we can identify. For example, there isn't a true sharp line between any of the primary colors, but we do not generally have a problem labelling things as "red" or "green" or whatever even so. There is a certain personality type frequently encountered online that this bothers no end, who can't really deal with the idea of words that have fuzzy boundaries around them and struggle with the concept of a "fuzzy line", but most of us get on even so.

However, when there is no natural clumping, there isn't even a place to put the fuzzy boundary necessarily. I think a lot of testing has this problem. I've disliked "unit" versus "integration" terminology for at least a decade now, because to me there's just a continuum. Yes, obviously, tests can test more or less of the code base at a time, and can draw in fewer or more dependencies in the process, and arguably there is an important distinction between 0 other dependencies included and 1, but beyond that I've thought for a long time there aren't any other distinctions worth putting a label on.

There is value to tests that mock out all the other dependencies. There is value to writing stubs. There is value to bringing up those dependencies and mocking or stubbing out their dependencies. There is value to having a full staging environment and running acceptance tests by hand, and that staging environment itself can have any number of mocks or stubs in it itself. And there is no rule that says you rigidly must make this decision based on the exact number of dependencies away something is from your code, to the extent that that is even a well-defined concept, nor are "mock" and "stub" themselves some sort of bright shining line as one can mix and match them freely, nor is that the complete list of all possibilities. The result is a fairly smooth continuum of possible tests. In my opinion, any attempt to put labels on very particular bits of it raises the risk of overprivileging certain parts of the space just because they get a name and creating the streetlight effect on the named bits and causing the others to be missed.

Even "performance testing" versus "correctness testing" gets awfully fuzzy; again, there is arguably some value in saying this test has performance concerns and this other test is just related to correctness, but even "performance testing" still has to worry about a lot of other things too and I don't find it anywhere near a binary distinction. Anyone who has spent any time with performance testing has certainly learned the risk of worrying about "performance" without also worrying about all the other things we test for at the same time, or to put it another way, I can make my performance testing look really good if I let the code it is testing be incorrect.

It is unreasonable to expect a universal nomenclature catalog of such a smooth space. We can't even as a species completely agree on color delineations in the spectrum, which is a simpler problem in most ways.