1. Tina Fletcher| Targeting Quality| Oct 9, 2015 Why Getting Good Test Data is Hard, and What to Do About It

2.  Currently a Test Strategist; have also been a Test Lead, Team Lead, and Test Specialist  But really… someone who cares about doing thoughtful, responsible testing that (hopefully) helps lead to better quality releases  I’m also fletcher.tina.m@gmail.com and @fletchertinamfletcher.tina.m@gmail.com Who am I?

3.  We don’t talk about it enough – Not glamorous  avoided – Sounds easy  underestimated – Not “testing”  forgotten  In my experience: – Not glamorous  necessary prerequisite – Sounds easy  surprisingly challenging – Not “testing”  forgetting is expensive Why am I talking about test data? Software Testing Test Data

4.  Have made enough mistakes to start offering advice?  What the rest of this talk will look like: – 10 useful questions to ask when starting a new software project  Example applications to help solidify concepts: – Media player – Email client – Learning management system for online courses Why am I talking about test data? (…cont.)

5. 10 Useful Test-Data Related Questions to Ask When Starting a New Software Project

6.  General definition – Research shows: “test data is the data you test with” – Tina’s best attempt: “something a user might have, enter, or do while using your feature that you care about testing before release” 1. What is test data, in our context?

7.  For your project – Define a test data unit  Media files, emails, online courses, students  Rows of database entries, JSON objects – Examples help to identify scope and variables  <1 sec 128kbps mp3 file  Email sent to 2 recipients, 3 image attachments, HTML table in body, AS 12.1 provider  Course with >1500 students, 6 quizzes and heavy group discussion usage  User registered as a student in 3 courses and a teaching assistant in another 1. What is test data, in our context? (…cont.)

8.  General reasons – Need test data in order to execute tests – Test under conditions that resemble production – Useful for many team members (developer, tester, UX designer, product owner) 2. Why does test data matter to us?

9.  Project-specific examples – We want to be confident that users can play all common media file types without encountering any errors – We want to be confident that users will be able to read and interact with any email message they receive, regardless of its properties – We want to be confident that the online learning system’s performance is not impacted by large classes with many concurrent user sessions – We want to be confident that permissions in the online learning system are properly restricted according to user roles 2. Why does test data matter to us? (…cont.)

10.  Random vs controlled – Mailing list subscription or crafting specific emails  Realistic vs simplified – Testing combinations of things or one at a time  Static vs changing – Same set of files or rotating/expanding collection  Able to tweak on demand as interesting things are observed – Something weird happens after ~2 minutes of playback 3. What should our test data look like?

11.  Understand functionality and performance expectations – Support claims for media file types – Load time before playback starts (large files)  Prioritize risk areas and how things might fail – Emails that have been forwarded across multiple providers  Decide what is in and out of scope – Top 25 email providers  Understand your customers – Common configurations of permission settings 3. What should our test data look like? (…cont.)

12.  Complete test data management suites (generation, analysis, cleaning): – IBM, Informatica, Grid Tools  Copy from existing source: – Test data repository – Production  Lightweight/freeware (rows of data): – http://www.generatedata.com, http://www.randomuser.me, http://www.mockaroo.com http://www.generatedata.comhttp://www.randomuser.mehttp://www.mockaroo.com  Mock data (APIs): – www.getsandbox.com, www.mockable.io www.getsandbox.comwww.mockable.io 4. How can we get test data?

13.  Use your team/company to generate data (“dogfooding”) – Internal training courses  Build your own tool to generate data – Email spammer (volume)  Use automated tests to generate data – Simulate student activities in a course  Generate it manually – Sometimes the only way, or most realistic way 4. How can we get test data? (…cont.)

14.  Using multiple methods is usually best – More variation – Each has strengths/limitations 4. How can we get test data? (…cont.)

15.  Inconsistencies in generated data  Large number of variations to obtain  Copyright  Privacy of “real” data – Email sharing – Masking – reduced usefulness? 5. What test data challenges might we encounter?

16.  “Old” data – Database migration – Time stamps  Cost – Paid accounts – Paid tools – Hosting multiple test environments – Time!! … 5. What test data challenges might we encounter? (…cont)

17.  Copy from a central location – Security/privacy concerns – Ability to partition data on demand – DRM restrictions  Stored in shared accounts or test environments – Return to known state / avoid irreversible change – Conflicts during simultaneous usage  Back up your test data! 6. Where are we going to keep test data? 7. How are we going to move/share test data?

18.  Are you finding bugs? – Are they valid? Important?  Are people outside of your team finding bugs? – e.g. beta, sales, other dev teams?  Are you able to test any scenarios that you randomly think of during testing? – Note: save/share additions and improvements to the data set 8. How will we know if our test data is effective?

19.  “Full coverage” is a myth; important to understand what’s missing  Ideas for mitigating actions: – Beta or crowd source testing – More unit tests – Review usage patterns (priority) – Communicate  New ideas from unexpected sources  Prepare: support, ops 9. What test data were we not able to get?

20.  Future you, or other teams, probably need to make changes in future – Avoid time cost for “reinventing the wheel”  Documentation: – What you have – What you don’t have – Where to find it – How to use it – How to add/tweak data 10. What test data legacy will we leave behind?

21.  Post mortem notes: – How you arrived at the current set of test data (and why) – Key things that need to be repeated – Ideas that didn’t pan out – Things that could have gone better – Things you didn’t get to – Ideas and recommendations for future 10. What test data legacy will we leave behind? (…cont.)

22.  What other questions would you add to this list?  What challenges have you encountered related to test data?  What creative test data solutions are you proud of?  What did I miss?  … or anything else! Over to you…

23.  Why is getting good test data hard? – Lots of variables and data characteristics to consider – Choosing sources and generation methods – Balancing data realism, privacy, $$, time – Can also be difficult to move, store, share, and tweak  What can you do about it? – Plan ahead – Ask yourself the 10 questions – Keep improving Conclusion

24. Thank you!