CNS*2013 Workshop on
Methods of Information Theory in Computational Neuroscience
Wednesday and Thursday, July 17 and 18, 2013
Paris, France
Overview
Methods originally developed in Information Theory have found wide applicability in computational neuroscience. Beyond these original methods there is a need to develop novel tools and approaches that are driven by problems arising in neuroscience.
A number of researchers in computational/systems neuroscience and in information/communication theory are investigating problems of information representation and processing. While the goals are often the same, these researchers bring different perspectives and points of view to a common set of neuroscience problems. Often they participate in different fora and their interaction is limited.
The goal of the workshop is to bring some of these researchers together to discuss challenges posed by neuroscience and to exchange ideas and present their latest work.
The workshop is targeted towards computational and systems neuroscientists with interest in methods of information theory as well as information/communication theorists with interest in neuroscience.
References
- C.E. Shannon, A Mathematical Theory of Communication, Bell System Technical Journal, vol. 27, pp. 379-423 and 623-656, 1948.
- Milenkovic, O., Alterovitz, G., Battail, G., Coleman, T. P., et al., Eds., Special Issue on Molecular Biology and Neuroscience, IEEE Transactions on Information Theory, Volume 56, Number 2, February, 2010.
- Dimitrov, A.G., Lazar, A.A. and Victor, J.D., Information Theory in Neuroscience, Journal of Computational Neuroscience, Vol. 30, No. 1, February 2011, pp. 1-5, Special Issue on Methods of Information Theory.
Standing Committee
- Alex G. Dimitrov, Department of Mathematics, Washington State University - Vancouver.
- Aurel A. Lazar, Department of Electrical Engineering, Columbia University.
Program Committee
- Michael C. Gastpar, Laboratory for Information in Networked Systems, EPFL and UC Berkeley.
- Conor Houghton, Department of Mathematics, Trinity College Dublin.
- Simon R. Schultz, Department of Bioengineering, Imperial College.
- Tatyana O. Sharpee, The Computational Neurobiology Laboratory, Salk Institute.
Invited Speakers
What Does a Neuron Do? An Online Learning View of Neuronal Computation
Dmitri B. Chklovskii, HHMI Janelia Farm, Ashburn, VA.
Sparse Sampling: Sensing Brain Activity at Infinite Resolution
Pier Luigi Dragotti, Imperial College.
The problem of reconstructing or estimating partially observed or sampled signals is an important one that finds application in many areas of signal processing. Traditional acquisition and reconstruction approaches are heavily influences by classical Shannon sampling theory which gives an exact sampling and interpolation formula for bandlimited signals. Recently, the emerging theory of sparse sampling has challenged the way we think about signal acquisition and has demonstrated that, by using more sophisticated signal models, it is possible to break away from the need to sample signals at the Nyquist rate. The insight that sub-Nyquist sampling can, under some circumstances, allow perfect reconstruction is revolutionizing signal processing, communications and inverse problems.
The main aim of this talk is to give an overview of these new exciting findings in sampling theory. The new fundamental theoretical results of sparse sampling will be reviewed and constructive algorithms will be presented. We also discuss the effect of noise on the sampling and reconstruction of sparse signals. In this context, a variation of an iterative algorithm due to Cadzow is proposed and shown to perform close to optimal over a wide range of signal to noise ratios. To emphasize the relevance of these new theories, a set of applications in neuroscience will be presented. In particular, we will present a new fast algorithm for spike detection from two-photon calcium imaging and will show how to perform spike sorting at sub-Nyquist rates.
Resonating Vector Strength: How to Determine Periodicity in a Sequence of Events
J. Leo van Hammen, Technical University of Munich
Can periodicity of discrete events be measured and, if so, how? In fact, the two questions are identical and our goal is to understand why and to see how uncertainty modifies the answer. Typically, we find responses as events at times {t1, t2 , … , tn}. Say, for a given periodic stimulus of angular frequency ωo = 2π/To . Then the question is, given a bunch of times {t1, t2, … , tn} on the real axis, how periodic are they? And do they repeat in "some" sense in accordance with the stimulus period To ? The question and the answer are at least as old as a classical paper due to von Mises and dating back to 1918. The key idea is simply this. We map the events tj onto the unit circle through tj → exp (iωtj) and focus on their center of gravity ρ(ω), a complex number in the unit disk. Its absolute value |ρ(ωo)| with ω ≡ ωo is what von Mises studied and is now called the vector strength. We prove that the nearer |ρ(ωo)| is to 1 the more periodic the events tj are w.r.t. To , a simple topological criterion. Furthermore, we also show why it is useful to study ρ(ω) as a function of ω so as to obtain a 'resonating' vector strength (RVS), an idea strongly deviating from the classical characteristic function. Finally, we discuss how noise as a means of quantifying our uncertainty regarding the tj modifies ρ(ω) as a measure of periodicity.
Lubomir Kostal, Academy of Sciences of the Cech Republic.
Simon Laughlin, Department of Zoology, University of Cambridge.
Aurel A. Lazar, Department of Electrical Engineering, Columbia University.
Categorical Perception: from Coding Efficiency to Reaction Times
Jean-Pierre Nadal, Laboratoire de Physique Statistique, CNRS UMR8550, Ecole Normale Supérieure, Paris.
We address issues specific to the perception of categories (e.g., vowels, familiar faces, etc.) making a clear distinction between identifying a category (an element of a discrete set) and estimating a continuous parameter (such as a direction). With the neural decision making process as main focus, we consider discrete (typically binary) choice tasks, implying the identification of the stimulus as an exemplar of a category.
First, we exhibit a link between optimal Bayesian decoding (identification) and coding efficiency, the latter being measured by the mutual information between the discrete category set and the neural activity. Focusing on the high signal-to-noise ratio limit with a large population of stimulus-specific encoding cells, we then obtain an analytical expression for the mutual information. We deduce the properties of the most efficient codes. One main outcome is to find that, in this high signal-to-noise ratio limit, the Fisher information at the population level should be the greatest between categories, which is achieved by having many cells with the stimulus-discriminating parts (steepest slope) of their tuning curves placed in the transition regions between categories in stimulus space. At the behavioral level, this leads to the main features that are characteristic of categorical perception (see, e.g., S. Harnard, Editor: “Categorical Perception: The Groundwork of Cognition”, Cambridge University Press; 1987), thus appearing as a byproduct of optimal coding.
Next, we characterize the properties of the best estimator of the likelihood of the category, when this estimator takes its inputs from the neural coding layer. This allows to study the reaction-times in a perceptual identification task, for a given (not necessarily optimized) coding layer. Adopting the diffusion-to-bound approach to model the decisional process, we relate analytically the bias and variance of the diffusion process underlying decision making to macroscopic quantities that are behaviorally measurable. A major consequence is the existence of a quantitative link between reaction times and discrimination accuracy. The results account for empirical facts, both qualitatively (e.g., more time is needed to identify a category from a stimulus at the boundary compared to a stimulus lying within a category), and quantitatively (working on published experimental data on phoneme identification tasks).
Joint work with Laurent Bonnasse-Gahot, Centre d’Analyse et de Mathématique Sociales, CNRS UMR8557, Ecole des Hautes Etudes en Sciences Sociales, Paris.
References
- Bonnasse-Gahot L, Nadal JP: Neural coding of categories: information efficiency and optimal population codes. Journal of computational neuroscience 2008, 25(1): 169-187.
- Bonnasse-Gahot L, Nadal JP: Perception of categories: from coding efficiency to reaction times. Brain Research 2012, 1434: 47-61.
Israel Nelken, Hebrew University.
Alex Pouget, University of Geneva.
Mark van Rossum, University of Edinburgh.
Simon R. Schultz, Department of Bioengineering, Imperial College.
Characterizing Neural Feature Selectivity and Invariance Using Natural Stimuli
Tatyana O. Sharpee, The Computational Neurobiology Laboratory, Salk Institute.
In this talk I will describe a set of computational tools for characterizing responses of high level sensory neurons. The goal is to describe in as simple as possible ways how the responses of these neurons signal the appearance of conjunctions of different features in the environment. The focus will be on computational methods that are designed to work with stimuli derived from the natural sensory environment. Some of the new methods that I will discuss characterize neural feature selectivity while assuming that the neural responses exhibit a certain type of invariance, such as position invariance for visual neurons. Other methods do not require one to make an assumption of invariance, and instead can determine the type of invariance by analyzing relationship between the multiple stimulus features that affect the neural responses. I will discuss the relative advantages and limitations of these computational tools and illustrate their performance using model neurons as well as recordings from the visual system.
Information transmission across the retinogeniculate synapse
Lawrence C. Sincich, University of Alabama.
Taking a single connected pair of neurons as an example, I will present a case study of how information is transmitted from one neuron to another in vivo. Derived from recordings of neurons in the lateral geniculate nucleus of a primate, where inputs from single retinal ganglion cells can be simultaneously recorded, I will show how retinogeniculate transfer of information can occasionally be so efficient as to be lossless. In our experiments, the stimuli consisted of a small spot of light modulated with naturalistic temporal frequencies, and thus the transfer dynamics are very close to what would be expected when primates (including humans!) view natural scenes. I will discuss the empirical pitfalls and opportunities created by physiological recordings of pairs of connected neurons in the living animal.
Naftali Tishby, Hebrew University.
The Simultaneous Silence of Neurons Explains Higher-order Interactions in Ensemble Spiking activity
Taro Toyoizumi, Riken Brain Sciences Institute.
Collective spiking activity of neurons is the basis of information processing in the brain. Sparse neuronal activity in a population of neurons limits possible spiking patterns and, thereby, influences the information content conveyed by each pattern. However, because of the combinatorial explosion of the number of parameters required to describe higher-order interactions (HOIs), the characterization of neuronal interactions has been mostly limited to lower-order interactions, such as pairwise interactions.
Here, we propose a new model that characterizes population-spiking activity by adding a single parameter to the previously proposed pairwise interaction model. This parameter describes the fraction of time a group of neurons is simultaneously silent, which can be alternatively expressed as a specific combination of HOIs. We apply our model to groups of neighboring neurons that are simultaneously recorded from spontaneously active slice cultures from the hippocampal CA3 area. Most groups of neurons that are not adequately explained by the pairwise interaction model exhibit significantly longer periods of simultaneous silence than the chance level expected from firing rates and pairwise correlations, demonstrating that simultaneous silence is a common property coded by HOIs.
To confirm that the simultaneous silence is also a major property, we systematically obtained a one-dimensional data-driven HOI term that is asymptotically optimal when added to a pairwise-interaction model. This analysis exhibited the structured HOIs expected from the simultaneous silence of neurons, i.e., positive pairwise interactions are followed by negative triple-wise interactions, and then positive quadruple-wise interactions. These results suggest that seemingly complex HOIs can be explained by simultaneous silence of multiple neurons. We discuss the implication of simultaneous silence for our understanding of the underlying circuit architecture and information coding.
Overview
Methods originally developed in Information Theory have found wide applicability in computational neuroscience. Beyond these original methods there is a need to develop novel tools and approaches that are driven by problems arising in neuroscience.
A number of researchers in computational/systems neuroscience and in information/communication theory are investigating problems of information representation and processing. While the goals are often the same, these researchers bring different perspectives and points of view to a common set of neuroscience problems. Often they participate in different fora and their interaction is limited.
The goal of the workshop is to bring some of these researchers together to discuss challenges posed by neuroscience and to exchange ideas and present their latest work.
The workshop is targeted towards computational and systems neuroscientists with interest in methods of information theory as well as information/communication theorists with interest in neuroscience.
References
- C.E. Shannon, A Mathematical Theory of Communication, Bell System Technical Journal, vol. 27, pp. 379-423 and 623-656, 1948.
- Milenkovic, O., Alterovitz, G., Battail, G., Coleman, T. P., et al., Eds., Special Issue on Molecular Biology and Neuroscience, IEEE Transactions on Information Theory, Volume 56, Number 2, February, 2010.
- Dimitrov, A.G., Lazar, A.A. and Victor, J.D., Information Theory in Neuroscience, Journal of Computational Neuroscience, Vol. 30, No. 1, February 2011, pp. 1-5, Special Issue on Methods of Information Theory.
Standing Committee
- Alex G. Dimitrov, Department of Mathematics, Washington State University - Vancouver.
- Aurel A. Lazar, Department of Electrical Engineering, Columbia University.
Program Committee
- Michael C. Gastpar, Laboratory for Information in Networked Systems, EPFL and UC Berkeley.
- Conor Houghton, Department of Mathematics, Trinity College Dublin.
- Simon R. Schultz, Department of Bioengineering, Imperial College.
- Tatyana O. Sharpee, The Computational Neurobiology Laboratory, Salk Institute.
Invited Speakers
What Does a Neuron Do? An Online Learning View of Neuronal Computation
Dmitri B. Chklovskii, HHMI Janelia Farm, Ashburn, VA.
Sparse Sampling: Sensing Brain Activity at Infinite Resolution
Pier Luigi Dragotti, Imperial College.
The problem of reconstructing or estimating partially observed or sampled signals is an important one that finds application in many areas of signal processing. Traditional acquisition and reconstruction approaches are heavily influences by classical Shannon sampling theory which gives an exact sampling and interpolation formula for bandlimited signals. Recently, the emerging theory of sparse sampling has challenged the way we think about signal acquisition and has demonstrated that, by using more sophisticated signal models, it is possible to break away from the need to sample signals at the Nyquist rate. The insight that sub-Nyquist sampling can, under some circumstances, allow perfect reconstruction is revolutionizing signal processing, communications and inverse problems.
The main aim of this talk is to give an overview of these new exciting findings in sampling theory. The new fundamental theoretical results of sparse sampling will be reviewed and constructive algorithms will be presented. We also discuss the effect of noise on the sampling and reconstruction of sparse signals. In this context, a variation of an iterative algorithm due to Cadzow is proposed and shown to perform close to optimal over a wide range of signal to noise ratios. To emphasize the relevance of these new theories, a set of applications in neuroscience will be presented. In particular, we will present a new fast algorithm for spike detection from two-photon calcium imaging and will show how to perform spike sorting at sub-Nyquist rates.
Resonating Vector Strength: How to Determine Periodicity in a Sequence of Events
J. Leo van Hammen, Technical University of Munich
Can periodicity of discrete events be measured and, if so, how? In fact, the two questions are identical and our goal is to understand why and to see how uncertainty modifies the answer. Typically, we find responses as events at times {t1, t2 , … , tn}. Say, for a given periodic stimulus of angular frequency ωo = 2π/To . Then the question is, given a bunch of times {t1, t2, … , tn} on the real axis, how periodic are they? And do they repeat in "some" sense in accordance with the stimulus period To ? The question and the answer are at least as old as a classical paper due to von Mises and dating back to 1918. The key idea is simply this. We map the events tj onto the unit circle through tj → exp (iωtj) and focus on their center of gravity ρ(ω), a complex number in the unit disk. Its absolute value |ρ(ωo)| with ω ≡ ωo is what von Mises studied and is now called the vector strength. We prove that the nearer |ρ(ωo)| is to 1 the more periodic the events tj are w.r.t. To , a simple topological criterion. Furthermore, we also show why it is useful to study ρ(ω) as a function of ω so as to obtain a 'resonating' vector strength (RVS), an idea strongly deviating from the classical characteristic function. Finally, we discuss how noise as a means of quantifying our uncertainty regarding the tj modifies ρ(ω) as a measure of periodicity.
Lubomir Kostal, Academy of Sciences of the Cech Republic.
Simon Laughlin, Department of Zoology, University of Cambridge.
Aurel A. Lazar, Department of Electrical Engineering, Columbia University.
Categorical Perception: from Coding Efficiency to Reaction Times
Jean-Pierre Nadal, Laboratoire de Physique Statistique, CNRS UMR8550, Ecole Normale Supérieure, Paris.
We address issues specific to the perception of categories (e.g., vowels, familiar faces, etc.) making a clear distinction between identifying a category (an element of a discrete set) and estimating a continuous parameter (such as a direction). With the neural decision making process as main focus, we consider discrete (typically binary) choice tasks, implying the identification of the stimulus as an exemplar of a category.
First, we exhibit a link between optimal Bayesian decoding (identification) and coding efficiency, the latter being measured by the mutual information between the discrete category set and the neural activity. Focusing on the high signal-to-noise ratio limit with a large population of stimulus-specific encoding cells, we then obtain an analytical expression for the mutual information. We deduce the properties of the most efficient codes. One main outcome is to find that, in this high signal-to-noise ratio limit, the Fisher information at the population level should be the greatest between categories, which is achieved by having many cells with the stimulus-discriminating parts (steepest slope) of their tuning curves placed in the transition regions between categories in stimulus space. At the behavioral level, this leads to the main features that are characteristic of categorical perception (see, e.g., S. Harnard, Editor: “Categorical Perception: The Groundwork of Cognition”, Cambridge University Press; 1987), thus appearing as a byproduct of optimal coding.
Next, we characterize the properties of the best estimator of the likelihood of the category, when this estimator takes its inputs from the neural coding layer. This allows to study the reaction-times in a perceptual identification task, for a given (not necessarily optimized) coding layer. Adopting the diffusion-to-bound approach to model the decisional process, we relate analytically the bias and variance of the diffusion process underlying decision making to macroscopic quantities that are behaviorally measurable. A major consequence is the existence of a quantitative link between reaction times and discrimination accuracy. The results account for empirical facts, both qualitatively (e.g., more time is needed to identify a category from a stimulus at the boundary compared to a stimulus lying within a category), and quantitatively (working on published experimental data on phoneme identification tasks).
Joint work with Laurent Bonnasse-Gahot, Centre d’Analyse et de Mathématique Sociales, CNRS UMR8557, Ecole des Hautes Etudes en Sciences Sociales, Paris.
References
- Bonnasse-Gahot L, Nadal JP: Neural coding of categories: information efficiency and optimal population codes. Journal of computational neuroscience 2008, 25(1): 169-187.
- Bonnasse-Gahot L, Nadal JP: Perception of categories: from coding efficiency to reaction times. Brain Research 2012, 1434: 47-61.
Israel Nelken, Hebrew University.
Alex Pouget, University of Geneva.
Mark van Rossum, University of Edinburgh.
Simon R. Schultz, Department of Bioengineering, Imperial College.
Characterizing Neural Feature Selectivity and Invariance Using Natural Stimuli
Tatyana O. Sharpee, The Computational Neurobiology Laboratory, Salk Institute.
In this talk I will describe a set of computational tools for characterizing responses of high level sensory neurons. The goal is to describe in as simple as possible ways how the responses of these neurons signal the appearance of conjunctions of different features in the environment. The focus will be on computational methods that are designed to work with stimuli derived from the natural sensory environment. Some of the new methods that I will discuss characterize neural feature selectivity while assuming that the neural responses exhibit a certain type of invariance, such as position invariance for visual neurons. Other methods do not require one to make an assumption of invariance, and instead can determine the type of invariance by analyzing relationship between the multiple stimulus features that affect the neural responses. I will discuss the relative advantages and limitations of these computational tools and illustrate their performance using model neurons as well as recordings from the visual system.
Information transmission across the retinogeniculate synapse
Lawrence C. Sincich, University of Alabama.
Taking a single connected pair of neurons as an example, I will present a case study of how information is transmitted from one neuron to another in vivo. Derived from recordings of neurons in the lateral geniculate nucleus of a primate, where inputs from single retinal ganglion cells can be simultaneously recorded, I will show how retinogeniculate transfer of information can occasionally be so efficient as to be lossless. In our experiments, the stimuli consisted of a small spot of light modulated with naturalistic temporal frequencies, and thus the transfer dynamics are very close to what would be expected when primates (including humans!) view natural scenes. I will discuss the empirical pitfalls and opportunities created by physiological recordings of pairs of connected neurons in the living animal.
Naftali Tishby, Hebrew University.
The Simultaneous Silence of Neurons Explains Higher-order Interactions in Ensemble Spiking activity
Taro Toyoizumi, Riken Brain Sciences Institute.
Collective spiking activity of neurons is the basis of information processing in the brain. Sparse neuronal activity in a population of neurons limits possible spiking patterns and, thereby, influences the information content conveyed by each pattern. However, because of the combinatorial explosion of the number of parameters required to describe higher-order interactions (HOIs), the characterization of neuronal interactions has been mostly limited to lower-order interactions, such as pairwise interactions.
Here, we propose a new model that characterizes population-spiking activity by adding a single parameter to the previously proposed pairwise interaction model. This parameter describes the fraction of time a group of neurons is simultaneously silent, which can be alternatively expressed as a specific combination of HOIs. We apply our model to groups of neighboring neurons that are simultaneously recorded from spontaneously active slice cultures from the hippocampal CA3 area. Most groups of neurons that are not adequately explained by the pairwise interaction model exhibit significantly longer periods of simultaneous silence than the chance level expected from firing rates and pairwise correlations, demonstrating that simultaneous silence is a common property coded by HOIs.
To confirm that the simultaneous silence is also a major property, we systematically obtained a one-dimensional data-driven HOI term that is asymptotically optimal when added to a pairwise-interaction model. This analysis exhibited the structured HOIs expected from the simultaneous silence of neurons, i.e., positive pairwise interactions are followed by negative triple-wise interactions, and then positive quadruple-wise interactions. These results suggest that seemingly complex HOIs can be explained by simultaneous silence of multiple neurons. We discuss the implication of simultaneous silence for our understanding of the underlying circuit architecture and information coding.