Re: [BLACKBOX] Software testing

From: [at]} <Gunnar>
Date: Tue, 19 Jan 2010 20:13:44 +0100

----boundary-LibPST-iamunique-320709160_-_-
Content-type: text/plain

Chris Burrows wrote:
> [...]
> You can get away with that much of the time when programming in Oberon
> as it
> encourages good programming practices and is excellent at letting you
> know
> when you do something stupid. However, I would not recommend using a
> trial
> and error / hacking approach to software development using more lax
> languages (e.g. C).
>
I certainly agree that the language Component Pascal helps to programmer
by sorting out obvious mistakes. Compared to a real scripting language
like VbScript or Python you do not have to run a programm in order to
check it for errors like misspelled variables (VbScript has "option
explicit", but it's not commenly used). You needn't worry about pointer
semantics in as many cases (there's a lot more danger than sense in
incrementing the pointer to an array rather than an element of it) as in
a C language variant... The list goes on.

The real problems are complexity and market pressure. Imagine the
following: You have to find a bug in 800 lines of code (probably not you
own). Your supervisor estimated this to take no more than half a day
(since it is presumed to be a simple error). Before this you dealt with
a completely different module/topic. Afterwards you are expected to
continue developement of even another module right were debugging
interrupted you a few days ago.
My point is, that under these circumstances most programmers will take
the first half-way plausible solution they can demonstrate to work. This
means of course, that they may not fix the error for all cases, in which
in can occur but mainly the reported one.

The downside of this approach certainly is the possibility of building
on top of a weak base as Bob noted.

> Many software developers appear to follow the model used in other
> industries
> today i.e. let customers do the work of finding and reporting faults
> rather
> than doing inhouse testing and quality control. Easy internet
> distribution
> and automatic updates only serve to encourage these practices. While this
> approach might appear cheaper in the short term it often ends up
> considerably more expensive in the long term.
I wouldn't be so strict here. One obviously important point is the "time
to market". Combine this with the fact, that selection of a software
seems to depend on features, which may become usefull later on, you get
a powerful incensive to start marketing with premature software (what,
your software isn't capable of 3D and has no SAP interface?!).
This software has to be good enough to survive demonstrations but
needn't have all the bugs fixed which show up when you use it
intensively. This works well, as long as you accompany those who try it
or use it because you fix bugs "on the fly" (especially if your software
is highly adaptable and has to be taylored to the customers needs anyway).

To be honest though, this differs greatly from my private needs. When I
try a software which looks promising but turns out to be buggy I usually
deinstall it and try another.

> [...]
> How much testing is *necessary* (not just 'good enough') depends on
> the job
> at hand. When we are developing 'one-off' software to use ourselves we do
> minimal testing - any problems can be fixed as and when we encounter
> them.
> On the other hand when we are working on products to be potentially
> deployed
> to 100,000's of remote customers, testing is much more exhaustive. For
> example, our CD burning product was used in house for a couple of
> years and
> was rewritten twice before it was considered robust enough to be let out
> into the wild. Testing of Armaide continued for a couple of months
> after it
> was judged to be 'good enough' and before a release candidate was finally
> distributed.
This is exceptionally good practise for developing software!

In my company modules are only redesigned and rewritten on very rare
occasions.
For my private programming I have not had to rewritte larger potions of
software yet.

> Bad first impressions and an avalanche of technical support
> problems can kill a product.
Definitely. But don't take too much time writing it, because most
companies don't replace one existing software with another just because
it is more stable.

{([at]})nowhere.xy
new software of fix another bug. The lucky and skilled ones may climb a
tree that arches over it, the others just move in order not to drown in
it. ;-)

regards
gunnar

----
To unsubscribe, send a message with body "SIGNOFF BLACKBOX" to LISTSERV{([at]})nowhere.xy----boundary-LibPST-iamunique-320709160_-_-
Content-type: application/rtf
Content-transfer-encoding: base64
Content-Disposition: attachment; filename="rtf-body.rtf"
e1xydGYxXGFuc2lcYW5zaWNwZzEyNTJcZnJvbXRleHQgXGRlZmYwe1xmb250dGJsDQp7XGYwXGZz
d2lzcyBBcmlhbDt9DQp7XGYxXGZtb2Rlcm4gQ291cmllciBOZXc7fQ0Ke1xmMlxmbmlsXGZjaGFy
c2V0MiBTeW1ib2w7fQ0Ke1xmM1xmbW9kZXJuXGZjaGFyc2V0MCBDb3VyaWVyIE5ldzt9fQ0Ke1xj
b2xvcnRibFxyZWQwXGdyZWVuMFxibHVlMDtccmVkMFxncmVlbjBcYmx1ZTI1NTt9DQpcdWMxXHBh
cmRccGxhaW5cZGVmdGFiMzYwIFxmMFxmczIwIENocmlzIEJ1cnJvd3Mgd3JvdGU6XHBhcg0KPiBb
Li4uXVxwYXINCj4gWW91IGNhbiBnZXQgYXdheSB3aXRoIHRoYXQgbXVjaCBvZiB0aGUgdGltZSB3
aGVuIHByb2dyYW1taW5nIGluIE9iZXJvbiBccGFyDQo+IGFzIGl0XHBhcg0KPiBlbmNvdXJhZ2Vz
IGdvb2QgcHJvZ3JhbW1pbmcgcHJhY3RpY2VzIGFuZCBpcyBleGNlbGxlbnQgYXQgbGV0dGluZyB5
b3UgXHBhcg0KPiBrbm93XHBhcg0KPiB3aGVuIHlvdSBkbyBzb21ldGhpbmcgc3R1cGlkLiBIb3dl
dmVyLCBJIHdvdWxkIG5vdCByZWNvbW1lbmQgdXNpbmcgYSBccGFyDQo+IHRyaWFsXHBhcg0KPiBh
bmQgZXJyb3IgLyBoYWNraW5nIGFwcHJvYWNoIHRvIHNvZnR3YXJlIGRldmVsb3BtZW50IHVzaW5n
IG1vcmUgbGF4XHBhcg0KPiBsYW5ndWFnZXMgKGUuZy4gQykuXHBhcg0KPiAgIFxwYXINCkkgY2Vy
dGFpbmx5IGFncmVlIHRoYXQgdGhlIGxhbmd1YWdlIENvbXBvbmVudCBQYXNjYWwgaGVscHMgdG8g
cHJvZ3JhbW1lciBccGFyDQpieSBzb3J0aW5nIG91dCBvYnZpb3VzIG1pc3Rha2VzLiBDb21wYXJl
ZCB0byBhIHJlYWwgc2NyaXB0aW5nIGxhbmd1YWdlIFxwYXINCmxpa2UgVmJTY3JpcHQgb3IgUHl0
aG9uIHlvdSBkbyBub3QgaGF2ZSB0byBydW4gYSBwcm9ncmFtbSBpbiBvcmRlciB0byBccGFyDQpj
aGVjayBpdCBmb3IgZXJyb3JzIGxpa2UgbWlzc3BlbGxlZCB2YXJpYWJsZXMgKFZiU2NyaXB0IGhh
cyAib3B0aW9uIFxwYXINCmV4cGxpY2l0IiwgYnV0IGl0J3Mgbm90IGNvbW1lbmx5IHVzZWQpLiBZ
b3UgbmVlZG4ndCB3b3JyeSBhYm91dCBwb2ludGVyIFxwYXINCnNlbWFudGljcyBpbiBhcyBtYW55
IGNhc2VzICh0aGVyZSdzIGEgbG90IG1vcmUgZGFuZ2VyIHRoYW4gc2Vuc2UgaW4gXHBhcg0KaW5j
cmVtZW50aW5nIHRoZSBwb2ludGVyIHRvIGFuIGFycmF5IHJhdGhlciB0aGFuIGFuIGVsZW1lbnQg
b2YgaXQpIGFzIGluIFxwYXINCmEgQyBsYW5ndWFnZSB2YXJpYW50Li4uIFRoZSBsaXN0IGdvZXMg
b24uXHBhcg0KXHBhcg0KVGhlIHJlYWwgcHJvYmxlbXMgYXJlIGNvbXBsZXhpdHkgYW5kIG1hcmtl
dCBwcmVzc3VyZS4gSW1hZ2luZSB0aGUgXHBhcg0KZm9sbG93aW5nOiBZb3UgaGF2ZSB0byBmaW5k
IGEgYnVnIGluIDgwMCBsaW5lcyBvZiBjb2RlIChwcm9iYWJseSBub3QgeW91IFxwYXINCm93biku
IFlvdXIgc3VwZXJ2aXNvciBlc3RpbWF0ZWQgdGhpcyB0byB0YWtlIG5vIG1vcmUgdGhhbiBoYWxm
IGEgZGF5IFxwYXINCihzaW5jZSBpdCBpcyBwcmVzdW1lZCB0byBiZSBhIHNpbXBsZSBlcnJvciku
IEJlZm9yZSB0aGlzIHlvdSBkZWFsdCB3aXRoIFxwYXINCmEgY29tcGxldGVseSBkaWZmZXJlbnQg
bW9kdWxlL3RvcGljLiBBZnRlcndhcmRzIHlvdSBhcmUgZXhwZWN0ZWQgdG8gXHBhcg0KY29udGlu
dWUgZGV2ZWxvcGVtZW50IG9mIGV2ZW4gYW5vdGhlciBtb2R1bGUgcmlnaHQgd2VyZSBkZWJ1Z2dp
bmcgXHBhcg0KaW50ZXJydXB0ZWQgeW91IGEgZmV3IGRheXMgYWdvLlxwYXINCk15IHBvaW50IGlz
LCB0aGF0IHVuZGVyIHRoZXNlIGNpcmN1bXN0YW5jZXMgbW9zdCBwcm9ncmFtbWVycyB3aWxsIHRh
a2UgXHBhcg0KdGhlIGZpcnN0IGhhbGYtd2F5IHBsYXVzaWJsZSBzb2x1dGlvbiB0aGV5IGNhbiBk
ZW1vbnN0cmF0ZSB0byB3b3JrLiBUaGlzIFxwYXINCm1lYW5zIG9mIGNvdXJzZSwgdGhhdCB0aGV5
IG1heSBub3QgZml4IHRoZSBlcnJvciBmb3IgYWxsIGNhc2VzLCBpbiB3aGljaCBccGFyDQppbiBj
YW4gb2NjdXIgYnV0IG1haW5seSB0aGUgcmVwb3J0ZWQgb25lLlxwYXINClxwYXINClRoZSBkb3du
c2lkZSBvZiB0aGlzIGFwcHJvYWNoIGNlcnRhaW5seSBpcyB0aGUgcG9zc2liaWxpdHkgb2YgYnVp
bGRpbmcgXHBhcg0Kb24gdG9wIG9mIGEgd2VhayBiYXNlIGFzIEJvYiBub3RlZC5ccGFyDQpccGFy
DQo+IE1hbnkgc29mdHdhcmUgZGV2ZWxvcGVycyBhcHBlYXIgdG8gZm9sbG93IHRoZSBtb2RlbCB1
c2VkIGluIG90aGVyIFxwYXINCj4gaW5kdXN0cmllc1xwYXINCj4gdG9kYXkgaS5lLiBsZXQgY3Vz
dG9tZXJzIGRvIHRoZSB3b3JrIG9mIGZpbmRpbmcgYW5kIHJlcG9ydGluZyBmYXVsdHMgXHBhcg0K
PiByYXRoZXJccGFyDQo+IHRoYW4gZG9pbmcgaW5ob3VzZSB0ZXN0aW5nIGFuZCBxdWFsaXR5IGNv
bnRyb2wuIEVhc3kgaW50ZXJuZXQgXHBhcg0KPiBkaXN0cmlidXRpb25ccGFyDQo+IGFuZCBhdXRv
bWF0aWMgdXBkYXRlcyBvbmx5IHNlcnZlIHRvIGVuY291cmFnZSB0aGVzZSBwcmFjdGljZXMuIFdo
aWxlIHRoaXNccGFyDQo+IGFwcHJvYWNoIG1pZ2h0IGFwcGVhciBjaGVhcGVyIGluIHRoZSBzaG9y
dCB0ZXJtIGl0IG9mdGVuIGVuZHMgdXBccGFyDQo+IGNvbnNpZGVyYWJseSBtb3JlIGV4cGVuc2l2
ZSBpbiB0aGUgbG9uZyB0ZXJtLiAgIFxwYXINCkkgd291bGRuJ3QgYmUgc28gc3RyaWN0IGhlcmUu
IE9uZSBvYnZpb3VzbHkgaW1wb3J0YW50IHBvaW50IGlzIHRoZSAidGltZSBccGFyDQp0byBtYXJr
ZXQiLiBDb21iaW5lIHRoaXMgd2l0aCB0aGUgZmFjdCwgdGhhdCBzZWxlY3Rpb24gb2YgYSBzb2Z0
d2FyZSBccGFyDQpzZWVtcyB0byBkZXBlbmQgb24gZmVhdHVyZXMsIHdoaWNoIG1heSBiZWNvbWUg
dXNlZnVsbCBsYXRlciBvbiwgeW91IGdldCBccGFyDQphIHBvd2VyZnVsIGluY2Vuc2l2ZSB0byBz
dGFydCBtYXJrZXRpbmcgd2l0aCBwcmVtYXR1cmUgc29mdHdhcmUgKHdoYXQsIFxwYXINCnlvdXIg
c29mdHdhcmUgaXNuJ3QgY2FwYWJsZSBvZiAzRCBhbmQgaGFzIG5vIFNBUCBpbnRlcmZhY2U/ISku
XHBhcg0KVGhpcyBzb2Z0d2FyZSBoYXMgdG8gYmUgZ29vZCBlbm91Z2ggdG8gc3Vydml2ZSBkZW1v
bnN0cmF0aW9ucyBidXQgXHBhcg0KbmVlZG4ndCBoYXZlIGFsbCB0aGUgYnVncyBmaXhlZCB3aGlj
aCBzaG93IHVwIHdoZW4geW91IHVzZSBpdCBccGFyDQppbnRlbnNpdmVseS4gVGhpcyB3b3JrcyB3
ZWxsLCBhcyBsb25nIGFzIHlvdSBhY2NvbXBhbnkgdGhvc2Ugd2hvIHRyeSBpdCBccGFyDQpvciB1
c2UgaXQgYmVjYXVzZSB5b3UgZml4IGJ1Z3MgIm9uIHRoZSBmbHkiIChlc3BlY2lhbGx5IGlmIHlv
dXIgc29mdHdhcmUgXHBhcg0KaXMgaGlnaGx5IGFkYXB0YWJsZSBhbmQgaGFzIHRvIGJlIHRheWxv
cmVkIHRvIHRoZSBjdXN0b21lcnMgbmVlZHMgYW55d2F5KS5ccGFyDQpccGFyDQpUbyBiZSBob25l
c3QgdGhvdWdoLCB0aGlzIGRpZmZlcnMgZ3JlYXRseSBmcm9tIG15IHByaXZhdGUgbmVlZHMuIFdo
ZW4gSSBccGFyDQp0cnkgYSBzb2Z0d2FyZSB3aGljaCBsb29rcyBwcm9taXNpbmcgYnV0IHR1cm5z
IG91dCB0byBiZSBidWdneSBJIHVzdWFsbHkgXHBhcg0KZGVpbnN0YWxsIGl0IGFuZCB0cnkgYW5v
dGhlci5ccGFyDQpccGFyDQo+IFsuLi5dXHBhcg0KPiBIb3cgbXVjaCB0ZXN0aW5nIGlzICpuZWNl
c3NhcnkqIChub3QganVzdCAnZ29vZCBlbm91Z2gnKSBkZXBlbmRzIG9uIFxwYXINCj4gdGhlIGpv
YlxwYXINCj4gYXQgaGFuZC4gV2hlbiB3ZSBhcmUgZGV2ZWxvcGluZyAnb25lLW9mZicgc29mdHdh
cmUgdG8gdXNlIG91cnNlbHZlcyB3ZSBkb1xwYXINCj4gbWluaW1hbCB0ZXN0aW5nIC0gYW55IHBy
b2JsZW1zIGNhbiBiZSBmaXhlZCBhcyBhbmQgd2hlbiB3ZSBlbmNvdW50ZXIgXHBhcg0KPiB0aGVt
LlxwYXINCj4gT24gdGhlIG90aGVyIGhhbmQgd2hlbiB3ZSBhcmUgd29ya2luZyBvbiBwcm9kdWN0
cyB0byBiZSBwb3RlbnRpYWxseSBccGFyDQo+IGRlcGxveWVkXHBhcg0KPiB0byAxMDAsMDAwJ3Mg
b2YgcmVtb3RlIGN1c3RvbWVycywgdGVzdGluZyBpcyBtdWNoIG1vcmUgZXhoYXVzdGl2ZS4gRm9y
XHBhcg0KPiBleGFtcGxlLCBvdXIgQ0QgYnVybmluZyBwcm9kdWN0IHdhcyB1c2VkIGluIGhvdXNl
IGZvciBhIGNvdXBsZSBvZiBccGFyDQo+IHllYXJzIGFuZFxwYXINCj4gd2FzIHJld3JpdHRlbiB0
d2ljZSBiZWZvcmUgaXQgd2FzIGNvbnNpZGVyZWQgcm9idXN0IGVub3VnaCB0byBiZSBsZXQgb3V0
XHBhcg0KPiBpbnRvIHRoZSB3aWxkLiBUZXN0aW5nIG9mIEFybWFpZGUgY29udGludWVkIGZvciBh
IGNvdXBsZSBvZiBtb250aHMgXHBhcg0KPiBhZnRlciBpdFxwYXINCj4gd2FzIGp1ZGdlZCB0byBi
ZSAnZ29vZCBlbm91Z2gnIGFuZCBiZWZvcmUgYSByZWxlYXNlIGNhbmRpZGF0ZSB3YXMgZmluYWxs
eVxwYXINCj4gZGlzdHJpYnV0ZWQuIFxwYXINClRoaXMgaXMgZXhjZXB0aW9uYWxseSBnb29kIHBy
YWN0aXNlIGZvciBkZXZlbG9waW5nIHNvZnR3YXJlIVxwYXINClxwYXINCkluIG15IGNvbXBhbnkg
bW9kdWxlcyBhcmUgb25seSByZWRlc2lnbmVkIGFuZCByZXdyaXR0ZW4gb24gdmVyeSByYXJlIFxw
YXINCm9jY2FzaW9ucy5ccGFyDQpGb3IgbXkgcHJpdmF0ZSBwcm9ncmFtbWluZyBJIGhhdmUgbm90
IGhhZCB0byByZXdyaXR0ZSBsYXJnZXIgcG90aW9ucyBvZiBccGFyDQpzb2Z0d2FyZSB5ZXQuXHBh
cg0KXHBhcg0KPiBCYWQgZmlyc3QgaW1wcmVzc2lvbnMgYW5kIGFuIGF2YWxhbmNoZSBvZiB0ZWNo
bmljYWwgc3VwcG9ydFxwYXINCj4gcHJvYmxlbXMgY2FuIGtpbGwgYSBwcm9kdWN0LiBccGFyDQpE
ZWZpbml0ZWx5LiBCdXQgZG9uJ3QgdGFrZSB0b28gbXVjaCB0aW1lIHdyaXRpbmcgaXQsIGJlY2F1
c2UgbW9zdCBccGFyDQpjb21wYW5pZXMgZG9uJ3QgcmVwbGFjZSBvbmUgZXhpc3Rpbmcgc29mdHdh
cmUgd2l0aCBhbm90aGVyIGp1c3QgYmVjYXVzZSBccGFyDQppdCBpcyBtb3JlIHN0YWJsZS5ccGFy
DQpccGFyDQpARG91ZzogV2UgYWxsIHNlZW0gdG8gc3RhcnQgZnJvbSB3aXRoaW4gbXVja3kgd2F0
ZXJzIGV2ZXJ5IHRpbWUgd2Ugd3JpdGUgXHBhcg0KbmV3IHNvZnR3YXJlIG9mIGZpeCBhbm90aGVy
IGJ1Zy4gVGhlIGx1Y2t5IGFuZCBza2lsbGVkIG9uZXMgbWF5IGNsaW1iIGEgXHBhcg0KdHJlZSB0
aGF0IGFyY2hlcyBvdmVyIGl0LCB0aGUgb3RoZXJzIGp1c3QgbW92ZSBpbiBvcmRlciBub3QgdG8g
ZHJvd24gaW4gXHBhcg0KaXQuIDstKVxwYXINClxwYXINCnJlZ2FyZHNccGFyDQpndW5uYXJccGFy
DQpccGFyDQpccGFyDQotLS0tXHBhcg0KVG8gdW5zdWJzY3JpYmUsIHNlbmQgYSBtZXNzYWdlIHdp
dGggYm9keSAiU0lHTk9GRiBCTEFDS0JPWCIgdG8gTElTVFNFUlZATElTVFMuT0JFUk9OLkNIfX0A
lyApYQ==
----boundary-LibPST-iamunique-320709160_-_---

Received on Tue Jan 19 2010 - 20:13:44 UTC