Appendix C: Sources and further reading

The practices in this book were worked out by people who faced the same problem, a fast and fallible producer of work, long before that producer was a language model. This appendix names the debts and points to the originals. Attributions were checked against primary sources before printing; where a claim could not be fully confirmed, the entry says so. Links point to the persistent identifier where one exists, otherwise to the canonical home of the work; entries without links are in print and easily found.

Tests that must be seen to fail

• Richard J. Lipton, Fault Diagnosis of Computer Programs, student term paper, Carnegie Mellon University (1971). The origin of mutation testing.
• Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward, Hints on Test Data Selection: Help for the Practicing Programmer, IEEE Computer (1978). Established mutation testing as a discipline: seed small faults, then measure whether the test suite detects them. doi.org/10.1109/C-M.1978.218136
• Timothy Budd, Mutation Analysis of Program Test Data, PhD thesis, Yale University (1980). The first implemented mutation-testing system. gwern.net/doc/cs/algorithm/1980-budd.pdf
• Gerald M. Weinberg, The Psychology of Computer Programming (1971). Introduced bebugging, seeding known defects to estimate how many unknown ones remain; also the first sustained study of programming as human work.
• Kent Beck, Test-Driven Development: By Example (2002). Codified the cycle in which a test is watched failing before the code that passes it is written.
• Yury Izrailevsky and Ariel Tseitlin, The Netflix Simian Army, Netflix Tech Blog (2011). Extended deliberate failure injection to production systems. techblog.netflix.com/2011/07/netflix-simian-army.html

Oracles and disagreement

• Edsger W. Dijkstra, Notes on Structured Programming, EWD249, written in 1969. The source of the principle that testing shows the presence of defects, never their absence. www.cs.utexas.edu/~EWD/transcriptions/EWD02xx/EWD249/EWD249.html
• Elaine J. Weyuker, On Testing Non-testable Programs, The Computer Journal (1982). Defined the case where no oracle exists to judge whether output is correct, and what can be done about it. doi.org/10.1093/comjnl/25.4.465
• William E. Howden, Theoretical and Empirical Studies of Program Testing, IEEE Transactions on Software Engineering (1978). Early formal analysis of what a test result can establish; the origin of the test-oracle concept. doi.org/10.1109/TSE.1978.231514
• William M. McKeeman, Differential Testing for Software, Digital Technical Journal (1998). Named and demonstrated differential testing: independent implementations compared on generated inputs. www.semanticscholar.org/paper/Differential-Testing-for-Software-McKeeman
• Xuejun Yang, Yang Chen, Eric Eide, and John Regehr, Finding and Understanding Bugs in C Compilers, PLDI (2011). Demonstrated differential testing at scale; the Csmith generator exposed hundreds of defects in production compilers. users.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf
• Koen Claessen and John Hughes, QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs, ICFP (2000). Introduced property-based testing: stated properties checked against generated inputs. doi.org/10.1145/351240.351266
• Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo, The Oracle Problem in Software Testing: A Survey, IEEE Transactions on Software Engineering (2015). The comprehensive account of the oracle problem and its partial solutions. doi.org/10.1109/TSE.2014.2372785

Generated proof

• Donald E. Knuth, Literate Programming, The Computer Journal (1984). Program and documentation produced from a single source, so neither can drift alone. doi.org/10.1093/comjnl/27.2.97
• Ken Thompson, Reflections on Trusting Trust, Turing Award lecture, Communications of the ACM (1984). Demonstrated that trusting a binary requires trusting its entire build lineage, the strongest form of the distinction between provenance and verification. doi.org/10.1145/358198.358210
• Tim Peters, the doctest module, Python standard library (1999). Made documentation examples executable tests. docs.python.org/3/library/doctest.html
• Jérémie Bobbio and the Debian Reproducible Builds project (2013 onward). Established bit-for-bit reproducibility as a verifiable property of a build system. reproducible-builds.org/docs/history

Boundaries and design

• David L. Parnas, On the Criteria To Be Used in Decomposing Systems into Modules, Communications of the ACM (1972). Established information hiding as the criterion for decomposition: a module should hide a decision likely to change. doi.org/10.1145/361598.361623
• Michael E. Fagan, Design and Code Inspections to Reduce Errors in Program Development, IBM Systems Journal (1976). Introduced formal inspection with measured defect-removal rates, applied before implementation is complete. doi.org/10.1147/sj.153.0182
• Michael C. Feathers, Working Effectively with Legacy Code (2004). Defined the boundary concept this book uses throughout under other names: a place where behavior can be altered and observed without editing the code in place.
• Barry W. Boehm, Software Engineering Economics (1981). Quantified how the cost of correcting a defect grows with the phase in which it is found; the underlying analysis appears in Software Engineering, IEEE Transactions on Computers (1976). selab.netlab.uky.edu/homepage/boehm-sw-eng-paper.pdf

Small batches and integration

• Kent Beck, Extreme Programming Explained: Embrace Change (1999). Argued small, continuously integrated changes as the core of development practice rather than a tactic.
• Martin Fowler and Matthew Foemmel, Continuous Integration, martinfowler.co, (2000). Defined the practice of continuous integration as it is now understood. martinfowler.com/articles/originalContinuousIntegration.html
• Grady Booch, Object-Oriented Analysis and Design with Applications, second edition (1994). Documents internal releases forming a sort of continuous integration; the phrase is often dated to the 1991 first edition, but the confirmed citation is the second.
• Donald G. Reinertsen, The Principles of Product Development Flow (2009). The economics of batch size and queues that underlie chapter 9.

Checklists, scars, and honest failure

• The Boeing Model 299 crash at Wright Field, October 30, 1935, and the pilot’s checklist created in response. The founding case: a machine too complex to operate from memory alone. www.nationalmuseum.af.mil/Visit/Museum-Exhibits/Fact-Sheets/Display/Article/610002/model-299-crash
• Peter Pronovost and the Michigan Keystone ICU project, New England Journal of Medicine (2006). A five-item checklist produced a large, sustained reduction in catheter-related infections in intensive care. doi.org/10.1056/NEJMoa061115
• Atul Gawande, “The Checklist Manifesto: How to Get Things Right, (2009). Generalized the checklist as a safety instrument across medicine, aviation, and construction. atulgawande.com/book/the-checklist-manifesto
• The NASA Aviation Safety Reporting System (1976 onward). A confidential, non-punitive incident reporting system; the model for blameless failure reporting. asrs.arc.nasa.gov
• Sidney Dekker, Just Culture: Balancing Safety and Accountability (2007). Argued that punishing error reporting suppresses the information safety depends on.
• John Allspaw, Blameless PostMortems and a Just Culture, Etsy Code as Craf, (2012). Brought blameless postmortem practice to web operations. www.etsy.com/codeascraft/blameless-postmortems
• John Lunney and Sue Lueder, Postmortem Culture: Learning from Failure, Site Reliability Engineering, chapter 1, (2016). Documented the practice as standard operations at scale. sre.google/sre-book/postmortem-culture

The division of labor

• Charles Babbage, “On the Economy of Machinery and Manufactures, (1832). Analyzed the division of mental labor in computation a century before electronic computers. archive.org/details/oneconomyofmachi00babb
• Ada Lovelace, Note G to her translation of Menabrea’s Sketch of the Analytical Engine (1843). Distinguished what the engine could do from what remains the operator’s judgment. www.cs.yale.edu/homes/tap/Files/ada-lovelace-notes.html
• J. C. R. Licklider, Man-Computer Symbiosis, IRE Transactions on Human Factors in Electronics (1960). Proposed the human-computer partnership in which each side contributes what it does best. groups.csail.mit.edu/medg/people/psz/Licklider.html
• Laurie Williams, Robert Kessler, Ward Cunningham, and Ron Jeffries, Strengthening the Case for Pair Programming, IEEE Software (2000). Measured the effect of two people sharing one problem, neither certifying their own work. doi.org/10.1109/52.854064

If you read only three: Parnas (1972), Dijkstra’s EWD249, and Licklider (1960). Everything in this book is downstream of somebody, and mostly of them.