[Cuis] (Rescued) Re: Fwd: Re: DIRECT version number
Peter van Rooijen
peter at aba-instituut.nl
Sat Aug 1 06:57:13 CDT 2015
Hi Hannes,
You asked for an example of a Feature Test. I assume, but may be wrong,
that you want this to understand better what the intention of such a test
is. Of course I can only answer form my own imagination.
1 It should be something that provides a definite answer to a question
about the image/system, but where there is not necessarily a good or bad
outcome (as there is with a unit test; pass=good, fail or error=bad). So
for instance it could test if there is any class loaded that implements
AnsiDateAndTime protocol (or at least part of it and how much). Or it could
test if the number of classes is less than the number of instance
variables. Or it could test how many classes have a class comment. It could
test how many literals may be in a method. Still, it seems logical to want
to define Features in such a way that the presence of the feature is a good
situation (better or no worse than the absence).
2 An essential feature of a Feature Test is that it should be able to load
and run in in any image. Its no use of course if you want to figure
something out and you can't load and run the test that determines what you
want to know. So The Feature test must be able to compile and run source
code (it could conceivably interpret syntax trees but that seems rather
contrived given that the compiler is normally available in any image). And
while doing that it should be able to suppress any problems.
3 There should be some way to (build Feature Test Suites suites and/or) run
all the loaded Feature tests together, silently, and receive feedback only
at the end. Most simply in the form of a document/text that shows how it
went. Optionally using some kind of GUI, viz. SUnitBrowser. The default
FeatureTestSuite could simply run all loaded Feature Tests, and you could
subclass it if you wanted to define specific Suites.
I think it would be great to have a FeatureTest class that you could
subclass to run tests on how much of the Ansi standard is implemented in
the image and how accurately it follows the semantics prescribed. The
original Camp Smalltalk attempts at such a test suite were based on SUnit
as the test framework, but that didn't work as it is essentially unsuited
to loading tests about code that may not be laoded.
Cheers, Peter
On Fri, Jul 31, 2015 at 8:29 AM, H. Hirzel <hannes.hirzel at gmail.com> wrote:
> Could we do examples of such Feature Tests?
>
> Class String is a good candidate to start with
>
> Reasons
> a) Is used everywhere
> b) Interface is non-trivial
> String selectors size 166 (in Cuis)
> 338 (in Pharo)
> 331 (in Squeak)
> --> there are issues when porting.
>
> We might want to have a StringExtensions package
>
> --Hannes
>
> On 7/22/15, Phil (list) <pbpublist at gmail.com> wrote:
> > On Wed, 2015-07-22 at 13:29 +0200, Peter van Rooijen wrote:
> >> I'm thinking about some features (pun not intentional) of this Feature
> >> Test framework:
> >>
> >>
> >> 1. It's reasonable to assume that many tests will depend on something
> >> else working, but that cannot be counted on, and
> >> we would not want to repeat testing for that and also would not run
> >> into it failing all the time and that filling our feedback.
> >>
> >
> > Why not? I agree that these would be a different category of test in
> > that the dependencies would be more complex and often dependent on more
> > than one package, but why would their functioning be considered
> > optional? If they fail, shouldn't they either be addressed right away
> > or explicitly deprecated? If you make the tests easy to ignore/forget
> > about, they will be. If the functionality they are testing can't be
> > counted on, it won't be used.
> >
> > If your thinking is that these would be tests that are part of package X
> > but might rely on package Y which might not be loaded yet, why not
> > instead just make the tests part of package Z which depends on package X
> > and Y? The whole idea is that these are not unit tests in that sense...
> > have them live where ever it is appropriate.
> >
> >>
> >> 1a. So it would make sense to add a #precondition method to each
> >> Feature Test class.
> >>
> >>
> >> FeatureAnsiArray
> >> class
> >> precondition
> >>
> >>
> >> self run: 'Array' "i.e. the global Array must be present"
> >>
> >>
> >> then only if the precondition for the class is satisfied, will the
> >> test methods be executed. so if most of them start with
> >>
> >>
> >> 'Array new …' then they would all fail anyway so we don't need to test
> >> them.
> >>
> >>
> >> 2. You would want to be able to assure that in a particular object you
> >> would be able to access a particular variable.
> >>
> >>
> >> so in the test method you would write:
> >>
> >>
> >> FeatureTest1
> >> class
> >> test1
> >>
> >> self setContext: OrderdCollection new
> >>
> >>
> >> self run: 'elements' "determine if the inst var elements is
> >> present in a new OrderedCollection"
> >>
> >>
> >> self run: 'elements == nil' expect: false
> >>
> >>
> >> self run: 'elements isOrderedCollection' expect: true
> >>
> >>
> >> let's say the test runner would continue testing even if something
> >> failed, e.g. the array is called array, not elements. then we already
> >> know that the following expressions will fail
> >>
> >>
> >> so we might instead write:
> >>
> >>
> >> self run: 'elements' ifFail: [^self]
> >>
> >>
> >>
> >> 3. Instead of implicitly testing for a global using run:
> >> 'NameOfTheGlobal' or for a class var using setContext: and then
> >> run'NameOfTheClassVar' there could be convenience methods for
> >>
> >>
> >> self expectGlobal: 'NameOfTheGlobal' "argument may be given as
> >> a Symbol as well"
> >>
> >>
> >> self expectClass: 'NameOfTheClass' "additionally verified that
> >> the global holds a class"
> >>
> >>
> >> self expectSharedVariable: 'Foo' inClass: 'Bar'
> >>
> >>
> >> this would make for nicer feedback since the expectation is made
> >> clearer
> >
> > I went the other way when I did the ApiFile tests in that it didn't seem
> > terribly important to use most of the testing framework capabilities
> > (other than the overall pass/fail aspect to keep the initial PoC as
> > simple as possible) So they are simply small snippets of code that
> > performed the desired task but didn't care where it failed (if it
> > failed): the failure to successfully complete the task would be the
> > indicator that there was a problem and we would know that either
> > something being depended on had broken and needed to be fixed or that
> > the test needed to be updated/overhauled to represent the new way of
> > accomplishing the task.
> >
> > My thinking was that as we started to build up a number of these, we
> > might start to see common breakage patterns and then we could decide
> > whether or not to handle that them more explicitly (whether using the
> > existing test framework capabilities, creating a new one, etc.) Trying
> > to engineer it up front didn't seem like a great idea not knowing what
> > common failure states would look like yet.
> >
> >>
> >>
> >> Okay just 2 more cents!
> >>
> >
> > Mine as well. This is a worthwhile discussion/exercise IMO as we need
> > to get to a common understanding of what we are doing here.
> >
> >>
> >> Cheers, Peter
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Jul 22, 2015 at 12:57 PM, Peter van Rooijen
> >> <peter at aba-instituut.nl> wrote:
> >> Hi Ken,
> >>
> >> On Wed, Jul 22, 2015 at 12:33 AM, Ken.Dickey
> >> <Ken.Dickey at whidbey.com> wrote:
> >> On Tue, 21 Jul 2015 07:59:47 -0700
> >> Peter van Rooijen <peter at aba-instituut.nl> wrote:
> >>
> >> >> I was thinking: "What should a Feature Test be?".
> >>
> >> I tend to think of a hierarchy of requirements.
> >> Perhaps something like:
> >>
> >> Feature requireAll: #( <feature name>.. ).
> >> Classes requireAll: #( <class name>.. ).
> >> Methods requireAll: #( <selector name>.. ).
> >> MethodsForKind class: <class name> requireAll:
> >> #( <selectorName>.. ).
> >> Tests requireAllPass: #( <unit test name> ).
> >>
> >>
> >> Yeah, that's not at all what I'm thinking :). I'm thinking of
> >> something that is automatically runnable, like a unit test. It
> >> tests something, like a un test. But if the test does not
> >> pass, it is NOT a bug, unlike with a unit test. It's just that
> >> we would like to know about it. Also, with nit tests there is
> >> the assumption that the code that represents the test is
> >> always compilable, with feature tests that cannot be assumed,
> >> so there would need to be protection against that. Of course
> >> we want to be able to load the feature tests in any condition,
> >> so putting it in the form of source text and compiling that is
> >> a possibility. The fact that that makes it slower than unit
> >> tests is not a problem, because unlike unit tests, we don't
> >> have to run them continuously.
> >>
> >> The Feature class lets us require named (macro)
> >> Features with a version check. I prefer that
> >> requirements at this level actually load the packages
> >> required and only report failure if that is not
> >> possible, although we could have a "preflight" verson
> >> which just checks without loading any featured
> >> packages.
> >>
> >>
> >> I see. The thing I was thinking about merely reports about the
> >> state of a system (of code), it does not try to configure that
> >> in any way.
> >>
> >>
> >> API's are basically "protocols", which in the absence
> >> of symbolic execution means checking that classes and
> >> specific method selectors exist, or more specifically,
> >> method selectors are applicable to specific KindOf:
> >> classes.
> >>
> >>
> >> Well, in my mind some semantics could be expected (and tested
> >> for). For instance I might be interested if there is a
> >> DateTime class in the image and if it implements the ANSI
> >> DateAndTime protocol (not sure if there is one named that). Or
> >> perhaps another class that does that. These tests can test
> >> some actual semantics no problem.
> >>
> >>
> >> Perhaps some of you remember that Camp Smalltalk started with
> >> Ralph Johnson's desire to build an ANSI test suite. The way
> >> people went about it (extension methods to SUnit? I don't
> >> really remember) was wrong and could not possibly work (in my
> >> opinion anyway), but I could not convince a lot of people and
> >> such a test suite was never written. But with Feature Tests I
> >> think we could come a long way.
> >>
> >> Further, some Unit Tests may be required to pass to
> >> ensure compliance with some specification.
> >>
> >>
> >> Well, except that the tests would not be unit tests in the
> >> strictest sense. But semantics, not merely interface, can be
> >> tested for sure.
> >>
> >> We should be able to automate at least some of this
> >>
> >>
> >> Automate the running of the feature tests? Of course.
> >>
> >> including a first pass of generating the test sets,
> >> which could then be pruned by hand as required.
> >>
> >>
> >> That I don't see happening. You test what YOU think is
> >> important to be sure of. No machine can decide/calculate that
> >> for you. Perhaps I'm misunderstanding you.
> >>
> >>
> >> Cheers, Peter
> >>
> >>
> >> $0.02,
> >> -KenD
> >>
> >>
> >>
> >> _______________________________________________
> >> Cuis mailing list
> >> Cuis at jvuletich.org
> >> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
> >>
> >>
> >>
> >> _______________________________________________
> >> Cuis mailing list
> >> Cuis at jvuletich.org
> >> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
> >
> >
> >
> > _______________________________________________
> > Cuis mailing list
> > Cuis at jvuletich.org
> > http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
> >
>
> _______________________________________________
> Cuis mailing list
> Cuis at jvuletich.org
> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://jvuletich.org/pipermail/cuis_jvuletich.org/attachments/20150801/63b2b971/attachment-0003.html>
More information about the Cuis
mailing list