Autoencoding Pixies: Amortised Variational Inference with Graph Convolutions for Functional Distributional Semantics

02 February 2021, Version 1
This content is an early or alternative research output and has not been peer-reviewed by Cambridge University Press at the time of posting.

Abstract

The aim of distributional semantics is to learn the meanings of words from a corpus of text. The aim of formal semantics is to develop mathematical models of meaning. Functional Distributional Semantics provides a framework for distributional semantics which is interpretable in formal semantic terms, by representing the meaning of a word as a truth-conditional function (a binary classifier). However, the model introduces a large number of latent variables, which means that inference is computationally expensive, and training a model is therefore slow to converge. In this work, I introduce the Pixie Autoencoder, which augments the generative model of Functional Distributional Semantics with a graph-convolutional neural network to perform amortised variational inference. This allows the model to be trained more effectively, achieving better results on semantic similarity in context, and outperforming BERT, a large pre-trained language model.

Keywords

distributional semantics
formal semantics
neural networks
probabilistic graphical models
approximate inference
logical inference

Comments

Comments are not moderated before they are posted, but they can be removed by the site moderators if they are found to be in contravention of our Commenting and Discussion Policy [opens in a new tab] - please read this policy before you post. Comments should be used for scholarly discussion of the content in question. You can find more information about how to use the commenting feature here [opens in a new tab] .
This site is protected by reCAPTCHA and the Google Privacy Policy [opens in a new tab] and Terms of Service [opens in a new tab] apply.