No CrossRef data available.
Article contents
Reservoir computing and the Sooner-is-Better bottleneck
Published online by Cambridge University Press: 02 June 2016
Abstract
Prior language input is not lost but integrated with the current input. This principle is demonstrated by “reservoir computing”: Untrained recurrent neural networks project input sequences onto a random point in high-dimensional state space. Earlier inputs can be retrieved from this projection, albeit less reliably so as more input is received. The bottleneck is therefore not “Now-or-Never” but “Sooner-is-Better.”
Information
- Type
- Open Peer Commentary
- Information
- Copyright
- Copyright © Cambridge University Press 2016
References
Bi, G. & Poo, M. (2001) Synaptic modification of correlated activity: Hebb's postulate revisited. Annual Review of Neuroscience
24:139–66.Google Scholar
Buonomano, D. V. & Maass, W. (2009) State-dependent computations: Spatiotemporal processing in cortical networks. Nature Reviews Neuroscience
10:113–25.Google Scholar
Christiansen, M. H. & Chater, N. (1999) Toward a connectionist model of recursion in human linguistic performance. Cognitive Science
23:157–205.Google Scholar
Dominey, P. F., Hoen, M., Blanc, J.-M. & Lelekov-Boissard, T. (2003) Neurological basis of language and sequential cognition: Evidence from simulation, aphasia and ERP studies. Brain and Language
86:207–25.CrossRefGoogle ScholarPubMed
Fitz, H. (2011) A liquid-state model of variability effects in learning nonadjacent dependencies. In: Proceedings of the 33rd Annual Conference of the Cognitive Science Society, Boston, MA, July 2011, ed. Carlson, L., Hölscher, C. & Shipley, T., pp. 897–902. Cognitive Science Society.Google Scholar
Frank, S. L. & Bod, R. (2011) Insensitivity of the human sentence-processing system to hierarchical structure. Psychological Science
22:829–34.CrossRefGoogle ScholarPubMed
Gerstner, W., Kistler, W. M., Naud, R. & Paninski, L. (2014) Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press.CrossRefGoogle Scholar
Hinaut, X. & Dominey, P. F. (2013) Real-time parallel processing of grammatical structure in the fronto-striatal system: A recurrent network simulation study using reservoir computing. PLOS ONE
8(2):e52946.CrossRefGoogle ScholarPubMed
Jaeger, H. & Haas, H. (2004) Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science
304:78–80.Google Scholar
Lukoševičius, M. & Jaeger, H. (2009) Reservoir computing approaches to recurrent neural network training. Computer Science Review
3:127–49.CrossRefGoogle Scholar
Maass, W., Natschläger, T. & Markram, H. (2002) Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation
14:2531–60.Google Scholar
Mongillo, G., Barak, O. & Tsodyks, M. (2008) Synaptic theory of working memory. Science
319:1543–46.Google Scholar
Petersson, K. M. & Hagoort, P. (2012) The neurobiology of syntax: Beyond string sets. Philosophical Transactions of the Royal Society B
367:1971–83.CrossRefGoogle ScholarPubMed
Rabinovich, M., Huerta, R. & Laurent, G. (2008) Transient dynamics for neural processing. Science
321:48–50.CrossRefGoogle ScholarPubMed
Rigotti, M., Barak, O., Warden, M. R., Wang, X. -J., Daw, N. D., Miller, E. K. & Fusi, S. (2013) The importance of mixed selectivity in complex cognitive tasks. Nature
497:585–90.Google Scholar
Singer, W. (2013) Cortical dynamics revisited. Trends in Cognitive Sciences
17:616–26.Google Scholar
Target article
The Now-or-Never bottleneck: A fundamental constraint on language
Related commentaries (28)
Better late than Now-or-Never: The case of interactive repair phenomena
Conceptual short-term memory (CSTM) supports core claims of Christiansen and Chater
Consequences of the Now-or-Never bottleneck for signed versus spoken languages
Exploring some edges: Chunk-and-Pass processing at the very beginning, across representations, and on to action
Gestalt-like representations hijack Chunk-and-Pass processing
How long is now? The multiple timescales of language processing
Is Now-or-Never language processing good enough?
Language acquisition is model-based rather than model-free
Language processing is not a race against time
Linguistic representations and memory architectures: The devil is in the details
Linguistic structure emerges through the interaction of memory constraints and communicative pressures
Linguistics, cognitive psychology, and the Now-or-Never bottleneck
Many important language universals are not reducible to processing or cognition
Mechanisms for interaction: Syntax as procedures for online interactive meaning building
Memory limitations and chunking are variable and cannot explain language structure
Natural language processing and the Now-or-Never bottleneck
Neural constraints and flexibility in language processing
Now or … later: Perceptual data are not immediately forgotten during language processing
On the generalizability of the Chunk-and-Pass processing approach: Perspectives from language acquisition and music
Pro and con: Internal speech and the evolution of complex language
Processing cost and its consequences
Realizing the Now-or-Never bottleneck and Chunk-and-Pass processing with Item-Order-Rank working memories and masking field chunking networks
Reservoir computing and the Sooner-is-Better bottleneck
Socio-demographic influences on language structure and change: Not all learners are the same
The bottleneck may be the solution, not the problem
The ideomotor recycling theory for language
What gets passed in “Chunk-and-Pass” processing? A predictive processing solution to the Now-or-Never bottleneck
“Process and perish” or multiple buffers with push-down stacks?
Author response
Squeezing through the Now-or-Never bottleneck: Reconnecting language processing, acquisition, change, and structure