Skip to main content
×
×
Home

Dynamic witnesses for static type errors (or, Ill-Typed Programs Usually Go Wrong)

  • ERIC L. SEIDEL (a1), RANJIT JHALA (a1) and WESTLEY WEIMER (a2)
Abstract

Static type errors are a common stumbling block for newcomers to typed functional languages. We present a dynamic approach to explaining type errors by generating counterexample witness inputs that illustrate how an ill-typed program goes wrong. First, given an ill-typed function, we symbolically execute the body to synthesize witness values that make the program go wrong. We prove that our procedure synthesizes general witnesses in that if a witness is found, then for all inhabited input types, there exist values that can make the function go wrong. Second, we show how to extend this procedure to produce a reduction graph that can be used to interactively visualize and debug witness executions. Third, we evaluate the coverage of our approach on two data sets comprising over 4,500 ill-typed student programs. Our technique is able to generate witnesses for around 85% of the programs, our reduction graph yields small counterexamples for over 80% of the witnesses, and a simple heuristic allows us to use witnesses to locate the source of type errors with around 70% accuracy. Finally, we evaluate whether our witnesses help students understand and fix type errors, and find that students presented with our witnesses show a greater understanding of type errors than those presented with a standard error message.

Copyright
References
Hide All
Bayne, M., Cook, R. & Ernst, M. D. (2011) Always-available static and dynamic feedback. In Proceedings of the 33rd International Conference on Software Engineering. ICSE '11. New York, NY, USA: ACM, pp. 521–530.
Cadar, C., Dunbar, D. & Engler, D. (2008) KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation. OSDI'08. Berkeley, CA, USA: USENIX Association, pp. 209–224.
Chamarthi, H. R., Dillinger, P. C., Kaufmann, M. & Manolios, P. (2011) Integrating testing and interactive theorem proving. In Proceedings of the 10th International Workshop on the ACL2 Theorem Prover and its Applications. ACL2, pp. 4–19.
Charguéraud, A. (2014) Improving type error messages in ocaml. In Proceedings of the ML Family/OCaml Users and Developers Workshops. Electronic Proceedings in Theoretical Computer Science, vol. 198. Open Publishing Association, pp. 80–97.
Chen, S. & Erwig, M. (2014) Counter-factual typing for debugging type errors. In Proceedings of the 41st ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages. POPL. New York, NY, USA: ACM, pp. 583–594.
Christiansen, D. R. (2014) Reflect on your mistakes! lightweight domain-specific error messages. In Proceedings of the 15th Symposium on Trends in Functional Programming.
Claessen, K. & Hughes, J. (2000) QuickCheck: A lightweight tool for random testing of haskell programs. In Proceedings of the 5th ACM SIGPLAN International Conference on Functional Programming. New York, NY, USA: ACM, pp. 268–279.
Clements, J., Flatt, M. & Felleisen, M. (2001) Modeling an algebraic stepper. In Programming Languages and Systems. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer, pp. 320–334.
Csallner, C. & Smaragdakis, Y. (2004) JCrasher: An automatic robustness tester for java. Softw. Pract. Exp. 34 (11), 10251050.
Damas, L & Milner, R. (1982) Principal type-schemes for functional programs. In Proceedings of the 9th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages. New York, NY, USA: ACM, pp. 207–212.
Felleisen, M., Findler, R. B. & Flatt, M. (2009) Semantics Engineering with PLT Redex. 1st ed. The MIT Press.
Flanagan, C., Flatt, M., Krishnamurthi, S., Weirich, S. & Felleisen, M. (1996) Catching bugs in the web of program invariants. In Proceedings of the ACM SIGPLAN 1996 Conference on Programming Language Design and Implementation. New York, NY, USA: ACM, pp. 23–32.
Fleiss, J. L. (1971) Measuring nominal scale agreement among many raters. Psychol. Bull. 76 (5), 378.
Gast, H. (2004) Explaining ML type errors by data flows. In Implementation and Application of Functional Languages. Lecture Notes in Computer Science. Berlin Heidelberg: Springer, pp. 7289.
Godefroid, P., Klarlund, N. & Sen, K. (2005) DART: Directed automated random testing. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation. New York, NY, USA: ACM, pp. 213–223.
Guo, P. J. (2013) Online Python Tutor: Embeddable web-based program visualization for CS education. In Proceedings of the 44th ACM Technical Symposium on Computer Science Education. SIGCSE '13. New York, NY, USA: ACM, pp. 579–584.
Haack, C. & Wells, J. B. (2003) Type error slicing in implicitly typed Higher-Order languages. In Programming Languages and Systems. Lecture Notes in Computer Science. Berlin Heidelberg: Springer, pp. 284301.
Hage, J. & Heeren, B. (2006) Heuristics for type error discovery and recovery. In Implementation and Application of Functional Languages. Lecture Notes in Computer Science. Berlin Heidelberg: Springer, pp. 199216.
Hage, J. & Heeren, B. (2009) Strategies for solving constraints in type and effect systems. Electron. Notes Theor. Comput. Sci. 236 (2), 163183.
Heeren, B., Hage, J. & Swierstra, S. D. (2003) Scripting the type inference process. In Proceedings of the 8th ACM SIGPLAN International Conference on Functional Programming, vol. 38. ACM, pp. 3–13.
Krippendorff, K. (2012) Content Analysis: An Introduction to Its Methodology. SAGE Publications.
Landis, J. R & Koch, G. G. (1977) The measurement of observer agreement for categorical data. Biometrics 33 (1), 159174.
Lee, O. & Yi, K. (1998) Proofs about a folklore let-polymorphic type inference algorithm. ACM Trans. Program. Lang. Syst. 20 (4), 707723.
Lempsink, E. (2009) Generic Type-Safe Diff and Patch for Families of Datatypes. M.Phil. thesis, Universiteit Utrecht.
Lerner, B., Grossman, D. & Chambers, C. (2006) Seminal: Searching for ML type-error messages. In Proceedings of the 2006 Workshop on ML. New York, NY, USA: ACM, pp. 63–73.
Lerner, B. S., Flower, M., Grossman, D. & Chambers, C. (2007) Searching for type-error messages. In Proceedings of the 28th ACM SIGPLAN Conference on Programming Language Design and Implementation. New York, NY, USA: ACM, pp. 425–434.
Lindblad, F. (2007) Property directed generation of first-order test data. In Proceedings of the Eighth Symposium on Trends in Functional Programming. Morazán, M. T. (ed), vol. 8, pp. 105–123.
Loncaric, C., Chandra, S., Schlesinger, C. & Sridharan, M. (2016) A practical framework for type inference error explanation. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications. ACM, pp. 781–799.
Mann, H. B. & Whitney, D. R. (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat. 18 (1), 5060.
Marceau, G., Fisler, K. & Krishnamurthi, S. (2011a) Measuring the effectiveness of error messages designed for novice programmers. In Proceedings of the 42Nd ACM Technical Symposium on Computer Science Education. New York, NY, USA: ACM, pp. 499–504.
Marceau, G., Fisler, K. & Krishnamurthi, S. (2011b) Mind your language: On novices' interactions with error messages. In Proceedings of the 10th SIGPLAN Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software. Onward! 2011. New York, NY, USA: ACM, pp. 3–18.
McAdam, B. J. (1998) On the unification of substitutions in type inference. In Implementation of Functional Languages, Hammond, K., Davie, T. & Clack, C. (eds), Lecture Notes in Computer Science. Berlin Heidelberg: Springer, pp. 137152.
Naylor, M. & Runciman, C. (2007) Finding inputs that reach a target expression. In Proceedings of the 7th IEEE International Working Conference on Source Code Analysis and Manipulation. pp. 133–142.
Nelson, G. & Oppen, D. C. (1979) Simplification by cooperating decision procedures. ACM Trans. Program. Lang. Syst. 1 (2), 245257.
Neubauer, M. & Thiemann, P. (2003) Discriminative sum types locate the source of type errors. In Proceedings of the 8th ACM SIGPLAN International Conference on Functional Programming. New York, NY, USA: ACM, pp. 15–26.
Nguyen, P. C, & Van Horn, D. (2015) Relatively complete counterexamples for higher-order programs. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation. New York, NY, USA: ACM, pp. 446–456.
Pacheco, C., Lahiri, S. K, Ernst, M. D. & Ball, T. (2007) Feedback-Directed random test generation. In Proceedings of the 29th International Conference on Software Engineering. ICSE '07, pp. 75–84.
Pavlinovic, Z., King, T. & Wies, T. (2014) Finding minimum type error sources. In Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & Applications. New York, NY, USA: ACM, pp. 525–542.
Pavlinovic, Z., King, T. & Wies, T. (2015) Practical SMT-based type error localization. In Proceedings of the 20th ACM SIGPLAN International Conference on Functional Programming. New York, NY, USA: ACM, pp. 412–423.
Perera, R., Acar, U. A., Cheney, J. & Levy, P. B. (2012) Functional programs that explain their work. In Proceedings of the 17th ACM SIGPLAN International Conference on Functional Programming. New York, NY, USA: ACM, pp. 365–376.
Rahli, V., Wells, J., Pirie, J. & Kamareddine, F. (2015) Skalpel: A type error slicer for standard ML. Electron. Notes Theor. Comput. Sci. 312 (24), 197213.
Runciman, C., Naylor, M. & Lindblad, F. (2008) Smallcheck and lazy smallcheck: Automatic exhaustive testing for small values. In Proceedings of the 1st ACM SIGPLAN Symposium on Haskell. New York, NY, USA: ACM, pp. 37–48.
Sagonas, K., Silva, J. & Tamarit, S. (2013) Precise explanation of success typing errors. In Proceedings of the ACM SIGPLAN 2013 Workshop on Partial Evaluation and Program Manipulation. New York, NY, USA: ACM, pp. 33–42.
Schilling, T. (2011) Constraint-free type error slicing. In Trends in Functional Programming. Lecture Notes in Computer Science. Berlin Heidelberg: Springer, pp. 116.
Seidel, E. L., Jhala, R. & Weimer, W. (2016b June) Dynamic Witnesses for Static Type Errors.
Seidel, E. L., Jhala, R. & Weimer, W. (2016a) Dynamic witnesses for static type errors (or, ill-typed programs usually go wrong) In Proceedings of the 21st ACM SIGPLAN International Conference on Functional Programming. ACM, pp. 228–242.
Seidel, E. L., Vazou, N. & Jhala, R. (2015) Type targeted testing. In Proceedings of the 24th European Symposium on Programming on Programming Languages and Systems. ESOP '15. New York, NY, USA: Springer-Verlag New York, Inc, pp. 812–836.
Serrano, A. & Hage, J. (2016) Type error diagnosis for embedded DSLs by Two-Stage specialized type rules. In Programming Languages and Systems. Lecture Notes in Computer Science. Berlin Heidelberg: Springer, pp. 672698.
Seven, D. (2014 17 Apr.) Knightmare: A DevOps Cautionary Tale. https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/. Accessed: 2017-4-24.
Tillmann, N. & de Halleux, J. (2008) Pex–White box test generation for .NET. In Tests and Proofs, Beckert, B. & Hähnle, R. (eds), Lecture Notes in Computer Science. Berlin Heidelberg: Springer, pp. 134153.
Vytiniotis, D., Peyton Jones, S. & Magalhães, J. P. (2012) Equality proofs and deferred type errors: A compiler pearl. In Proceedings of the 17th ACM SIGPLAN International Conference on Functional Programming. ICFP '12. New York, NY, USA: ACM, pp. 341–352.
Wheeler, D. A. (2014 23 Nov.) The apple goto fail vulnerability: lessons learned. https://www.dwheeler.com/essays/apple-goto-fail.html. Accessed: 2017-4-24.
Zhang, D. & Myers, A. C. (2014) Toward general diagnosis of static errors. In Proceedings of the 41st ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages. POPL '14. New York, NY, USA: ACM, pp. 569–581.
Zhang, D., Myers, A. C., Vytiniotis, D. & Peyton-Jones, S. (2015) Diagnosing type errors with class. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation. PLDI 2015. New York, NY, USA: ACM, pp. 12–21.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of Functional Programming
  • ISSN: 0956-7968
  • EISSN: 1469-7653
  • URL: /core/journals/journal-of-functional-programming
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 20 *
Loading metrics...

Abstract views

Total abstract views: 287 *
Loading metrics...

* Views captured on Cambridge Core between 21st May 2018 - 15th August 2018. This data will be updated every 24 hours.

Dynamic witnesses for static type errors (or, Ill-Typed Programs Usually Go Wrong)

  • ERIC L. SEIDEL (a1), RANJIT JHALA (a1) and WESTLEY WEIMER (a2)
Submit a response

Discussions

No Discussions have been published for this article.

×

Reply to: Submit a response


Your details


Conflicting interests

Do you have any conflicting interests? *