Skip to content
Sign in
Artwork for Software Testing Unleashed - QA, DevEx & Quality Engineering
Software Testing Unleashed - QA, DevEx & Quality Engineering · Thursday · 18 min

Dark Side of Testing: The Failures We Cause Ourselves - Bart Knaack

A CD full of production data, snapped in half, and what that says about test data Testing is supposed to prevent failures in production, yet sometimes the test itself is what breaks things. Bart Knaack calls this the dark side of testing, and we talk about test data leaking out of poorly protected environments, test systems that give attackers a look at your code, and why testing in production needs a risk assessment of its own. Bart would rather ask questions than hand out checklists, because a ticked box stops people thinking. He also tells the story of a team with 10,000 end-to-end test cases that ended up needing only 50. "If you can't bear the consequence of your test, don't test it." - Bart Knaack Bart Knaack is an IT-professional, working in the field for over 25 years. He has had many different roles, mostly in QA-Testing. He has been working for many big corporates and is a frequently seen speaker at (international) conferences. Next to that he is also a PhD candidate at the Open University on the subject of process based assessment in software education. With a solid background in IT for many years he has in depth knowledge of the different challenges the industry is facing at the moment. Highlights: Testing itself can cause failures: the dark side of testing covers leaked test data, exposed test environments and failing tests that touch production. Non-depersonalized production data on a test system or a CD is ungoverned data; Bart Knaack broke such a CD rather than process it. A test environment with weaker protection than production lets attackers study the code and reuse its flaws against the live system. Testing in production needs its own risk assessment, ideally with a network engineer in the room who knows which environments are connected. Ten thousand end-to-end test cases shrank to 50 after the team moved checks such as address validation into unit testing. 📘 Tool migration ahead? Teamscale playbook: migrate only 20% of tests, keep nearly all bug detection. Download for free

0:00-18:37
Play in
Transcript file

transcript

No transcript — this publisher did not publish one.

show notes

A CD full of production data, snapped in half, and what that says about test data

Testing is supposed to prevent failures in production, yet sometimes the test itself is what breaks things. Bart Knaack calls this the dark side of testing, and we talk about test data leaking out of poorly protected environments, test systems that give attackers a look at your code, and why testing in production needs a risk assessment of its own. Bart would rather ask questions than hand out checklists, because a ticked box stops people thinking. He also tells the story of a team with 10,000 end-to-end test cases that ended up needing only 50.

"If you can't bear the consequence of your test, don't test it." - Bart Knaack

Bart Knaack is an IT-professional, working in the field for over 25 years. He has had many different roles, mostly in QA-Testing. He has been working for many big corporates and is a frequently seen speaker at (international) conferences. Next to that he is also a PhD candidate at the Open University on the subject of process based assessment in software education.

With a solid background in IT for many years he has in depth knowledge of the different challenges the industry is facing at the moment.

Highlights:

  • Testing itself can cause failures: the dark side of testing covers leaked test data, exposed test environments and failing tests that touch production.
  • Non-depersonalized production data on a test system or a CD is ungoverned data; Bart Knaack broke such a CD rather than process it.
  • A test environment with weaker protection than production lets attackers study the code and reuse its flaws against the live system.
  • Testing in production needs its own risk assessment, ideally with a network engineer in the room who knows which environments are connected.
  • Ten thousand end-to-end test cases shrank to 50 after the team moved checks such as address validation into unit testing.

📘 Tool migration ahead? Teamscale playbook: migrate only 20% of tests, keep nearly all bug detection. Download for free

links2

more episodes

All episodes