Skip to main content Accessibility help
×
×
Home

How to evaluate the performance of gradual type systems

  • BEN GREENMAN (a1), ASUMU TAKIKAWA (a2), MAX S. NEW (a3), DANIEL FELTEY (a4), ROBERT BRUCE FINDLER (a5), JAN VITEK (a6) and MATTHIAS FELLEISEN (a7)...
Abstract

A sound gradual type system ensures that untyped components of a program can never break the guarantees of statically typed components. This assurance relies on runtime checks, which in turn impose performance overhead in proportion to the frequency and nature of interaction between typed and untyped components. The literature on gradual typing lacks rigorous descriptions of methods for measuring the performance of gradual type systems. This gap has consequences for the implementors of gradual type systems and developers who use such systems. Without systematic evaluation of mixed-typed programs, implementors cannot precisely determine how improvements to a gradual type system affect performance. Developers cannot predict whether adding types to part of a program will significantly degrade (or improve) its performance. This paper presents the first method for evaluating the performance of sound gradual type systems. The method quantifies both the absolute performance of a gradual type system and the relative performance of two implementations of the same gradual type system. To validate the method, the paper reports on its application to 20 programs and 3 implementations of Typed Racket.

Copyright
References
Hide All
Aiken, A., Wimmers, E. L. & Lakshman, T. K. (1994) Soft Typing with conditional types. In Proceedings of ACM Symposium on Principles of Programming Languages, pp. 163173.
Allende, E., Callaú, O., Fabry, J., Tanter, É. & Denker, M. (2013) Gradual typing for Smalltalk. Sci. Comput. Program.96(1), 5269.
Anderson, C., Giannini, P. & Drossopoulou, S. (2005) Toward type inference for JavaScript. In Proceedings of European Conference Object-Oriented Programming, pp. 428452.
Bauman, S., Bolz, C. F., Hirschfield, R., Kirilichev, V., Pape, T., Siek, J. G. & Tobin-Hochstadt, S. (2015) Pycket: A tracing JIT for a functional language. In Proceedings of ACM International Conference on Functional Programming, pp. 2234.
Bauman, S., Tobin-Hochstadt, S., Siek, J. G. & Bolz-Tereick, J. G. (2017) Sound gradual typing: Only mostly dead. In Proceedings of the ACM on Programming Languages. 1(OOPSLA), pp. 54:154:24.
Bloom, B., Field, J., Nystrom, N., Östlund, J., Richards, G., Strniša, R., Vitek, J. & Wrigstad, T. (2009) Thorn: Robust, concurrent, extensible scripting on the JVM. In Proceedings of ACM Conference on Object-Oriented Programming, Systems, Languages and Applications, pp. 117136.
Cartwright, R. & Fagan, M. (1991) Soft typing. In Proceedings of ACM Conference on Programming Language Design and Implementation, pp. 278292.
Consel, C. (1988) New insights into partial evaluation: The SCHISM experiment. In Proceedings of European Symposium on Programming, pp. 236246.
Curtsinger, C. & Berger, E. (2013) Stabilizer: Statistically sound performance evaluation. In Proceedings of ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 219228.
Dimoulas, C., Tobin-Hochstadt, S. & Felleisen, M. (2012) Complete monitors for behavioral contracts. In Proceedings of European Symposium on Programming, pp. 214233.
Fieller, E. C. (1957) Some problems in interval estimation. J. R. Stat. Soc.16(2), 175185.
Flanagan, C., Flatt, M., Krishnamurthi, S., Weirich, S. & Felleisen, M. (1996) Catching bugs in the web of program invariants. In Proceedings of ACM Conference on Programming Language Design and Implementation, pp. 2332.
Flatt, M. (2016) Bindings as sets of scopes. In Proceedings of ACM Symposium on Principles of Programming Languages, pp. 705717.
Franz, Volker H. (2007) Ratios: A short guide to confidence limits and proper use. Unpublished Manuscript. https://arxiv.org/abs/0710.2024.
Furr, M., (David) An, J.-H., Foster, J. S. & Hicks, M. (2009) Static type inference for Ruby. In Proceedings of Symposium on Applied Computing, pp. 18591866.
Gallesio, E. & Serrano, M. (1995) Bigloo: A portable and optimizing compiler for strict functional languages. In Proceedings of International Static Analysis Symposium, pp. 366381.
Garcia, R. (2013) Calculating threesomes, with blame. In Proceedings of ACM International Conference on Functional Programming, pp. 417428.
Garcia, R. & Cimini, M. (2015) Principal type schemes for gradual programs. In Proceedings of ACM Symposium on Principles of Programming Languages, pp. 303315.
Garcia, R., Clark, A. M. & Tanter, É. (2016) Abstracting gradual typing. In Proceedings of ACM Symposium on Principles of Programming Languages, pp. 429442.
Greenberg, M. (2015) Space-efficient manifest contracts. In Proceedings of ACM Symposium on Principles of Programming Languages, pp. 181194.
Greenberg, M. (2016) Space-efficient latent contracts. arXiv:1604.02474v4. Accessed 2018-12-03.
Greenman, B. & Felleisen, M. (2018) A spectrum of type soundness and performance. In Proceedings of the ACM on Programming Languages. 2(ICFP), pp. 71:171:32.
Greenman, B. & Migeed, Z. (2018) On the cost of type-tag soundness. In Proceedings of ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation, pp. 3039.
Gu, D., Verbrugge, C. & Gagnon, E. (2005) Code layout as a source of noise in JVM performance. Stud. Inform. Univers.4(1), 8399.
Henglein, F. (1992) Global tagging optimization by type inference. In Proceedings of ACM Symposium on LISP and Functional Programming, pp. 205215.
Henglein, F. & Rehof, J. (1995) Safe polymorphic type inference for a dynamically typed language: Translating scheme to ML. In Proceedings of ACM International Conference on Functional Programming Languages and Computer Architecture, pp. 192203.
Herman, D., Tomb, A. & Flanagan, C. (2010) Space-efficient gradual typing. Higher Order Symb. Comput. 23(2), 167189.
Jagannathan, S. & Wright, A. K. (1995) Effective flow analysis for avoiding run-time checks. In Proceedings of International Static Analysis Symposium, pp. 207224.
Knowles, K., Tomb, A., Gronski, J., Freund, S. N. & Flanagan, C. (2007) Sage: Unified Hybrid Checking for First-Class Types, General Refinement Types, and Dynamic (Extended Report).
Lemonnier, E. (2006) Pluto: Or how to make Perl juggle with billions. Forum on Free and Open Source Software (FREENIX). Available at: http://erwan.lemonnier.se/talks/pluto.html. Accessed 2018-12-03.
Matthews, J. & Findler, R. B. (2009) Operational semantics for multi-language programs. Trans. Program. Lang. Syst. 31(3), 12:112:44.
Moon, D. A. (1974) MACLISP Reference Manual. Massachusetts Institute of Technology.
Mytkowicz, T., Diwan, A., Hauswirth, M. & Sweeney, P. F. (2009) Producing wrong data without doing anything obviously wrong. In Proceedings of ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 265276.
Neyman, J. (1937) Outline of a theory of statistical estimation based on the classical theory of probability. Philos. Trans. R. Soc. Lond.236(767), 333380.
Nguyen, L. C. (2014) Tough Behavior in Repeated Bargaining Game. A Computer Simulation Study. Master in Economics dissertation, University of Trento.
Nguyen, P., Tobin-Hochstadt, S., & Van Horn, D. (2014) Soft Contract Verification. In Proceedings of ACM International Conference on Functional Programming, pp. 139152.
Rastogi, A., Swamy, N., Fournet, C., Bierman, G. & Vekris, P. (2015) Safe & efficient gradual typing for TypeScript. In Proceedings of ACM Symposium on Principles of Programming Languages, pp. 167180.
Ren, B. M. & Foster, J. S. (2016) Just-in-time static type checking for dynamic languages. In Proceedings of ACM Conference on Programming Language Design and Implementation, pp. 462476.
Richards, G., Nardelli, F. Z. & Vitek, J. (2015) Concrete types for TypeScript. In Proceedings of European Conference on Object-Oriented Programming, pp. 76100.
Siek, J. G. & Taha, W. (2006) Gradual typing for functional languages. In Proceedings of Scheme and Functional Programming Workshop, University of Chicago, TR-2006-06, 2006.
Siek, J. G., Thiemann, P. & Wadler, P. (2015a) Blame and coercion: Together again for the first time. In Proceedings ACM Conference on Programming Language Design and Implementation, pp. 425435.
Siek, J. G. & Vachharajani, M. (2008) Gradual typing with unification-based inference. In Proceedings of Dynamic Languages Symposium, Article 7.
Siek, J. G., Vitousek, M. M., Cimini, M., Tobin-Hochstadt, S. & Garcia, R. (2015b) Monotonic references for efficient gradual typing. In Proceedings of European Symposium on Programming, pp. 432456.
Siek, J. G. & Wadler, P. (2010) Threesomes, with and without blame. In Proceedings of ACM Symposium on Principles of Programming Languages, pp. 365376.
St-Amour, V., Andersen, L. & Felleisen, M. (2015) Feature-specific profiling. In Proceedings of International Conference on Compiler Construction, pp. 4968.
Steele, G. L. (1990) Common Lisp the Language, 2nd ed. Digital Press.
Strickland, T. S., Tobin-Hochstadt, S. & Felleisen, M. (2009) Practical variable-arity polymorphism. In Proceedings of European Symposium on Programming, pp. 3246.
Strickland, T. S., Tobin-Hochstadt, S., Findler, R. B. & Flatt, M. (2012) Chaperones and impersonators: Run-time support for reasonable interposition. In Proceedings of ACM Conference on Object-Oriented Programming, Systems, Languages and Applications, pp. 943962.
Suzuki, N. (1981) Inferring types in smalltalk. In Proceedings of ACM Symposium on Principles of Programming Languages, pp. 187199.
Takikawa, A., Feltey, D., Dean, E., Findler, R. B., Flatt, M., Tobin-Hochstadt, S. & Felleisen, M. (2015) Towards practical gradual typing. In Proceedings of European Conference on Object-Oriented Programming, pp. 427.
Takikawa, A., Feltey, D., Greenman, B., New, M. S., Vitek, J. & Felleisen, M. (2016) Is sound gradual typing dead? In Proceedings of ACM Symposium on Principles of Programming Languages, pp. 456468.
Takikawa, A., Strickland, T. S., Dimoulas, C., Tobin-Hochstadt, S. & Felleisen, M. (2012) Gradual typing for first-class classes. In Proceedings of ACM Conference on Object-Oriented Programming, Systems, Languages and Applications, pp. 793810.
Takikawa, A., Strickland, T. S. & Tobin-Hochstadt, S. (2013) Constraining delimited control with contracts. In Proceedings of European Symposium on Programming, pp. 229248.
Tobin-Hochstadt, S. & Felleisen, M. (2006) Interlanguage migration: From scripts to programs. In Proceedings of Dynamic Languages Symposium, pp. 964974.
Tobin-Hochstadt, S. & Felleisen, M. (2008) The design and implementation of typed scheme. In Proceedings of ACM Symposium on Principles of Programming Languages, pp. 395406.
Tobin-Hochstadt, S., Felleisen, M., Findler, R. B., Flatt, M., Greenman, B., Kent, A. M., St-Amour, V., Strickland, T. S. & Takikawa, A. (2017) Migratory typing: Ten years later. In Proceedings of Summit on Advances in Programming Languages, pp. 17:117:17.
Vitousek, M. M., Kent, A., Siek, J. G. & Baker, J. (2014) Design and evaluation of gradual typing for Python. In Proceedings of Dynamic Languages Symposium, pp. 4556.
Vitousek, M. M., Swords, C. & Siek, J. G. (2017) Big types in little runtime: Open-world soundness and collaborative blame for gradual type systems. In Proceedings of ACM Symposium on Principles of Programming Languages, pp. 762774.
Wolff, R., Garcia, R., Tanter, É & Aldrich, J. (2011) Gradual typestate. In Proceedins of European Conference on Object-Oriented Programming, pp. 459483.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of Functional Programming
  • ISSN: 0956-7968
  • EISSN: 1469-7653
  • URL: /core/journals/journal-of-functional-programming
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed

How to evaluate the performance of gradual type systems

  • BEN GREENMAN (a1), ASUMU TAKIKAWA (a2), MAX S. NEW (a3), DANIEL FELTEY (a4), ROBERT BRUCE FINDLER (a5), JAN VITEK (a6) and MATTHIAS FELLEISEN (a7)...
Submit a response

Discussions

No Discussions have been published for this article.

×

Reply to: Submit a response


Your details


Conflicting interests

Do you have any conflicting interests? *