Experimentation in Software Engineering An Introduction

It is my belief that software engineers not only need to know software engineering methods and processes, but that they also should know how to assess them. Conse­ quently, I have taught principles of experimentation and empirical studies as part of the software engineering curriculum. Until now, this meant selecting a text from another discipline, usually psychology, and augmenting it with journal or confer­ ence papers that provide students with software engineering examples of experi­ ments and empirical studies. This book fills an important gap in the software engineering literature: it pro­ vides a concise, comprehensive look at an important aspect of software engineer­ ing: experimental analysis of how well software engineering methods, methodologies, and processes work. Since all of these change so rapidly in our field, it is important to know how to evaluate new ones. This book teaches how to go about doing this and thus is valuable not only for the software engineering stu­ dent, but also for the practicing software engineering professional who will be able to • Evaluate software engineering techniques. • Determine the value (or lack thereof) of claims made about a software engineer­ ing method or process in published studies. Finally, this book serves as a valuable resource for the software engineering researcher.

1 Introduction.- 1.1 Software engineering context.- 1.2 Science and software engineering.- 2 Empirical strategies.- 2.1 Overview of empirical strategies.- 2.2 Surveys.- 2.3 Case studies.- 2.4 Experiments.- 2.5 Empirical strategies comparison.- 2.6 Empiricism in a software engineering context.- 3 Measurement.- 3.1 Basic concepts.- 3.2 Measurements in software engineering.- 4 Experiment process.- 4.1 Variables, treatments, objects and subjects.- 4.2 Process.- 5 Definition.- 5.1 Define experiment.- 5.2 Example.- 5.3 Summary.- 6 Planning.- 6.1 Context selection.- 6.2 Hypothesis formulation.- 6.3 Variables selection.- 6.4 Selection of subjects.- 6.5 Experiment design.- 6.6 Instrumentation.- 6.7 Validity evaluation.- 6.8 Detailed description of validity threats.- 6.9 Priority among types of validity threats.- 7 Operation.- 7.1 Preparation.- 7.2 Execution.- 7.3 Data validation.- 8 Analysis and interpretation.- 8.1 Descriptive statistics.- 8.2 Data set reduction.- 8.3 Hypothesis testing.- 9 Presentation and package.- 9.1 An experiment report outline.- 10 Literature survey.- 10.1 Inspection experiments.- 10.2 Other experiments in Software Engineering.- 10.3 Resources.- 11 Example: Experiment process.- 11.1 Definition.- 11.2 Planning.- 11.3 Operation.- 11.4 Analysis and interpretation.- 11.5 Summary and conclusions.- 12 Example: C versus C++.- 12.1 Introduction and problem statement.- 12.2 Experiment planning.- 12.3 Analysis and interpretation.- 12.4 Conclusions and further work.- 13 Exercises.- 13.1 Understanding.- 13.2 Training.- 13.3 Reviewing.- 13.4 Assignments.- Appendix A: Statistical tables.- Appendix B: Experiment process overview.- References.- About the authors.