Improvements in hardware and data collection have galvanized progress in NLP across many benchmark tasks. Impressive performance has been achieved in language modeling [@radford2019language; @Zellers2019; @keskar2019ctrl] and span-selection question answering [@devlin:naacl19; @yang2019xlnet; @lan2019albert] through massive data and massive models. With models exceeding human performance on such tasks, now is an excellent time to reflect on a key question:

In this paper, we consider how the data and world a language learner is exposed to define and constrains the scope of that learner’s semantics. Meaning does not arise from the statistical distribution of words, but from their use by people to communicate. Many of the assumptions and understandings on which communication relies lie outside of text. We must consider what is missing from models trained solely on text corpora, even when those corpora are meticulously annotated or Internet-scale.

You can’t learn language from the radio. Nearly every NLP course will at some point make this claim. The futility of learning language from linguistic signal alone is intuitive, and mirrors the belief that humans lean deeply on non-linguistic knowledge [@chomsky1965aspects; @chomsky1980language]. However, as a field we attempt this futility: trying to learn language from the Internet, which stands in as the modern radio to deliver limitless language. In this piece, we argue that the need for language to attach to “extralinguistic events" [@ErvinTripp1973] and the requirement for social context [@Baldwin1996] should guide our research.

Drawing inspiration from previous work in NLP, Cognitive Science, and Linguistics, we propose the notion of a World Scope (WS) as a lens through which to audit progress in NLP. We describe five WSs, and note that most trending work in NLP operates in the second (Internet-scale data).

We define five levels of World Scope:

1. WS1. Corpus (our past)

2. WS2. Internet (most of current NLP)

3. WS3. Perception (multimodal NLP)

4. WS4. Embodiment

5. WS5. Social

These World Scopes go beyond text to consider the contextual foundations of language: grounding, embodiment, and social interaction. We describe a brief history and ongoing progression of how contextual information can factor into representations and tasks. We conclude with a discussion of how this integration can move the field forward. We believe this World Scope framing serves as a roadmap for truly contextual language understanding.

1 WS1: Corpora and Representations↩︎

The story of data-driven language research begins with the corpus. The Penn Treebank [@Marcus1993] is the canonical example of a clean subset of naturally generated language, processed and annotated for the purpose of studying representations. Such corpora and the model representations built from them exemplify WS1. Community energy was initially directed at finding formal linguistic structure, such as recovering syntax trees. Recent success on downstream tasks has not required such explicitly annotated signal, leaning instead on unstructured fuzzy representations. These representations span from dense word vectors [@skipgram] to contextualized pretrained representations [@peters-etal-2018-deep; @devlin:naacl19].

Word representations have a long history predating the recent success of deep learning methods. Outside of NLP, philosophy [@austin1975things] and linguistics [@Lakoff1973; @coleman-kay-1981] recognized that meaning is flexible yet structured. Early experiments on neural networks trained with sequences of words [@Elman1990; @bengio2003] suggested that vector representations could capture both syntax and semantics. Subsequent experiments with larger models, documents, and corpora have demonstrated that representations learned from text capture a great deal of information about meaning in and out of context [@Collobert2008AUA; @turian-etal-2010-word; @skipgram; @mccann2017learned].

The intuition of such embedding representations, that context lends meaning, has long been acknowledged [@firth1957; @turney2010frequency]. Earlier on, discrete, hierarchical representations, such as agglomerative clustering guided by mutual information [@Brown1992], were constructed with some innate interpretability. A word’s position in such a hierarchy captures semantic and syntactic distinctions. When the Baum–Welch algorithm [@Welch2003] is applied to unsupervised Hidden Markov Models, it assigns a class distribution to every word, and that distribution is a partial representation of a word’s “meaning.” If the set of classes is small, syntax-like classes are induced; if the set is large, classes become more semantic. These representations are powerful in that they capture linguistic intuitions without supervision, but they are constrained by the structure they impose with respect to the number of classes chosen.

The intuition that meaning requires a large context, that “You shall know a word by the company it keeps." – [@firth1957], manifested early via Latent Semantic Indexing/Analysis [@LSI; @LSI2; @dumais2004] and later in the generative framework of Latent Dirichlet Allocation [@LDA]. LDA represents a document as a bag-of-words conditioned on latent topics, while LSI/A use singular value decomposition to project a co-occurrence matrix to a low dimensional word vector that preserves locality. These methods discard sentence structure in favor of the document.

Representing words through other words is a comfortable proposition, as it provides the illusion of definitions by implicit analogy to thesauri and related words in a dictionary definition. However, the recent trends in deep learning approaches to language modeling favor representing meaning in fixed-length vectors with no obvious interpretation. The question of where meaning resides in “connectionist” systems like Deep Neural Networks is an old one [@pollack1987; @james:nips95]. Are concepts distributed through edges or local to units in an artificial neural network?

“... there has been a long and unresolved debate between those who favor localist representations in which each processing element corresponds to a meaningful concept and those who favor distributed representations.” [@HINTON19901]
Special Issue on Connectionist Symbol Processing**

In connectionism, words were no longer defined over interpretable dimensions or symbols, which were perceived as having intrinsic meaning. The tension of modeling symbols and distributed representations is articulated by [@Smolensky1990], and alternative representations [@kohonen1984; @hinton-mcclelland-rumehart; @barlow1989] and approaches to structure and composition [@erk-pado-2008-structured; @Socher12Semantic] span decades of research.

The Brown Corpus [@browncorpus] and Penn Treebank [@Marcus1993] defined context and structure in NLP for decades. Only relatively recently [@baroni2009wacky] has the cost of annotations decreased enough, and have large-scale web-crawls become viable, to enable the introduction of more complex text-based tasks. This transition to larger, unstructured context (WS2) induced a richer semantics than was previously believed possible under the distributional hypothesis.

2 WS2: The Written World↩︎

Corpora in NLP have broadened to include large web-crawls. The use of unstructured, unlabeled, multi-domain, and multilingual data broadens our world scope, in the limit, to everything humanity has ever written.¹ We are no longer constrained to a single author or source, and the temptation for NLP is to believe everything that needs knowing can be learned from the written world. But, a large and noisy text corpus is still a text corpus.

This move towards using large scale raw data has led to substantial advances in performance on existing and novel community benchmarks [@devlin:naacl19; @brown2020fewshot]. Scale in data and modeling has demonstrated that a single representation can discover both rich syntax and semantics without our help [@tenney-etal-2019-bert]. This change is perhaps best seen in transfer learning enabled by representations in deep models. Traditionally, transfer learning relied on our understanding of model classes, such as English grammar. Domain adaptation simply required sufficient data to capture lexical variation, by assuming most higher-level structure would remain the same. Unsupervised representations today capture deep associations across multiple domains, and can be used successfully transfer knowledge into surprisingly diverse contexts [@brown2020fewshot].

These representations require scale in terms of both data and parameters. Concretely, [@skipgram] trained on 1.6 billion tokens, while [@pennington-etal-2014-glove] scaled up to 840 billion tokens from Common Crawl. Recent approaches have made progress by substantially increasing the number of model parameters to better consume these vast quantities of data. Where [@peters-etal-2018-deep] introduced ELMo with \(\sim\!\!10^8\) parameters, Transformer models [@Vaswani2017] have continued to scale by orders of magnitude between papers [@devlin:naacl19; @radford2019language; @Zellers2019] to \(\sim\!\!10^{11}\) [@brown2020fewshot].

Current models are the next (impressive) step in language modeling which started with [@Good1953], the weights of [@Kneser1995; @Chen1996], and the power-law distributions of [@Teh2006a]. Modern approaches to learning dense representations allow us to better estimate these distributions from massive corpora. However, modeling lexical co-occurrence, no matter the scale, is still modeling the written world. Models constructed this way blindly search for symbolic co-occurences void of meaning.

How can models yield both “impressive results” and “diminishing returns”? Language modeling—the modern workhorse of neural NLP systems—is a canonical example. Recent pretraining literature has produced results that few could have predicted, crowding leaderboards with “super-human" accuracy [@rajpurkar2018know]. However, there are diminishing returns. For example, on the LAMBADA dataset [@paperno2016lambada], designed to capture human intuition, GPT2 [@radford2019language] (1.5B), Megatron-LM [@shoeybi2019megatron] (8.3B), and TuringNLG [@rosset_2020] (17B) perform within a few points of each other and very far from perfect (<68%). When adding another order of magnitude of parameters (175B) [@brown2020fewshot] gain 8 percentage-points, impressive but still leaving 25% unsolved. Continuing to expand hardware, data sizes, and financial compute cost by orders of magnitude will yield further gains, but the slope of the increase is quickly decreasing.

The aforementioned approaches for learning transferable representations demonstrate that sentence and document context provide powerful signals for learning aspects of meaning, especially semantic relations among words [@fu2014learning] and inferential relationships among sentences [@wang2018glue]. The extent to which they capture deeper notions of contextual meaning remains an open question. Past work has found that pretrained word and sentence representations fail to capture many grounded features of words [@lucy2017distributional] and sentences, and current NLU systems fail on the thick tail of experience-informed inferences, such as hard coreference problems [@peng2015solving]. “I parked my car in the compact parking space because it looked (big/small) enough.” still presents problems for text-only learners.

As text pretraining schemes seem to be reaching the point of diminishing returns, even for some syntactic phenomena [@vanschijndel2019quantity], we posit that other forms of supervision, such as multimodal perception [@ilharco-etal-2019-large], are necessary to learn the remaining aspects of meaning in context. Learning by observation should not be a purely linguistic process, since leveraging and combining the patterns of multimodal perception can combinatorially boost the amount of signal in data through cross-referencing and synthesis.

3 WS3: The World of Sights and Sounds↩︎

Language learning needs perception, because perception forms the basis for many of our semantic axioms. Learned, physical heuristics, such as the fact that a falling cat will land quietly, are generalized and abstracted into language metaphors like as nimble as a cat [@lakoff1980]. World knowledge forms the basis for how people make entailment and reasoning decisions, commonly driven by mental simulation and analogy [@hofstadter2013surfaces]. Perception is the foremost source of reporting bias. The assumption that we all see and hear the same things informs not just what we name, but what we choose to assume and leave unwritten. Further, there exists strong evidence that children require grounded sensory perception, not just speech, to learn language [@sachs1981language; @ogrady2005; @vigliocco2014language].

Perception includes auditory, tactile, and visual input. Even restricted to purely linguistic signals, sarcasm, stress, and meaning can be implied through prosody. Further, tactile senses lend meaning, both physical [@sinapov:icra14; @Thomason2016] and abstract, to concepts like heavy and soft. Visual perception is a rich signal for modeling a vastness of experiences in the world that cannot be documented by text alone [@harnad:phys90].

For example, frames and scripts [@schank+abelson77; @Charniak1977; @Dejong1981; @Mooney1985] require understanding often unstated sets of pre- and post-conditions about the world. To borrow from [@Charniak1977], how should we learn the meaning, method, and implications of painting? A web crawl of knowledge from an exponential number of possible how-to, text-only guides and manuals [@Bisk2020] is misdirected without some fundamental referents to which to ground symbols. Models must be able to watch and recognize objects, people, and activities to understand the language describing them [@hake; @krishnavisualgenome; @yatskar2016; @perlis:aaai16] and access fine-grained notions of causality, physics, and social interactions.

While the NLP community has played an important role in the history of grounding [@mooney:aaai08], recently remarkable progress has taken place in the Computer Vision community.

It is tempting to assume that vision models trained to identify 1,000 ImageNet classes [@ILSVRC15]² are limited to extracting a bag of visual words. In reality, Computer Vision has been making in-roads into complex visual, physical, and social phenomena, while providing reusable infrastructure.³ The stability of these architectures allows for new research into more challenging world modeling. [@Mottaghi2016] predicts the effects of forces on objects in images. [@Bakhtin2019] extends this physical reasoning to complex puzzles of cause and effect. [@Sun2019-video; @Sun2019b] models scripts and actions, and alternative unsupervised training regimes [@bachman2019amdim] open up research towards automatic concept formation.

Advances in computer vision have enabled building semantic representations rich enough to interact with natural language. In the last decade of work descendant from image captioning [@Farhadi2010; @daume12midge], a myriad of tasks on visual question answering [@Antol2015; @das2018embodied; @yagcioglu-etal-2018-recipeqa], natural language and visual reasoning [@suhr-etal-2019-corpus], visual commonsense [@Zellers2019a], and multilingual captioning/translation via video [@Wang_2019_ICCV] have emerged. These combined text and vision benchmarks are rich enough to train large-scale, multimodal transformers [@Li2019; @Lu2019; @Zhou2019a] without language pretraining (e.g. via conceptual captions [@sharma-etal-2018-conceptual]) or further broadened to include audio [@tsai-etal-2019-multimodal]. Vision can also help ground speech signals [@Tejas2020; @harwath2019] to facilitate discovery of linguistic concepts [@harwath2019learning].

At the same time, NLP resources contributed to the success of these vision backbones. Hierarchical semantic representations emerge from ImageNet classification pretraining partially due to class hypernyms owed to that dataset’s WordNet origins. For example, the person class sub-divides into many professions and hobbies, like firefighter, gymnast, and doctor. To differentiate such sibling classes, learned vectors can also encode lower-level characteristics like clothing, hair, and typical surrounding scenes. These representations allow for pixel level masks and skeletal modeling, and can be extended to zero-shot settings targeting all 20K ImageNet categories [@chaozeroshot; @changpinyozeroshot]. Modern architectures also learn to differentiate instances within a general class, such as face. For example, facial recognition benchmarks require distinguishing over 10K unique faces [@liu2015faceattributes]. While vision is by no means “solved,” benchmarks have led to off-the-shelf tools for building representations rich enough to identify tens of thousands of objects, scenes, and individuals.

A WS3 agent, having access to potentially endless hours of video data showing the intricate details of daily comings and goings, procedures, and events, reduces susceptibility to the reporting bias of WS2. An ideal WS3 agent will exhibit better long-tail generalization and understanding than any language-only system could. This generalization should manifest in existing benchmarks, but would be most prominent in a test of zero-shot circumstances, such as “Will this car fit through that tunnel?,” and rarely documented behaviors as examined in script learning. Yet the WS3 agent will likely fail to answer, "Would a ceramic or paper plate make a better frisbee?" The agent has not tried to throw various objects and understand how their velocity and shape interact with the atmosphere to create lift. The agent cannot test novel hypotheses by intervention and action in the world.

4 WS4: Embodiment and Action↩︎

In human development, interactive multimodal sensory experience forms the basis of action-oriented categories [@thelen:mit96] as children learn how to manipulate their perception by manipulating their environment. Language grounding enables an agent to connect words to these action-oriented categories for communication [@smith:mit05], but requires action to fully discover such connections. Embodiment—situated action taking—is therefore a natural next broader context.

An embodied agent, whether in a virtual world, such as a 2D Maze [@macmahon:aaai06], a grid world [@chevalier+al-ICLR2019], a simulated house [@Anderson2017a; @Thomason2019a; @shridhar:cvpr2020], or the real world [@Tellex2011; @matuszek2018IJCAI; @thomason:jair20; @tellex:arcras:20] must translate from language to action. Control and action taking open several new dimensions to understanding and actively learning about the world. Queries can be resolved via dialog-based exploration with a human interlocutor [@liu:aaai15], even as new object properties, like texture and weight [@thomason:corl17], or feedback, like muscle activations [@moro-kennington-2018-multimodal], become available. We see the need for embodied language with complex meaning when thinking deeply about even the most innocuous of questions:

Is an orange more like a baseball or more like a banana?

WS1 is likely not to have an answer beyond that the objects are common nouns that can both be held. WS2 may capture that oranges and baseballs both roll, but is not the deformation strength, surface texture, or relative sizes of these objects [@elazar2019large]. WS3 may realize the relative deformability of these objects, but is likely to confuse how much force is necessary given that baseballs are used much more roughly than oranges. WS4 can appreciate the nuances of the question—the orange and baseball afford similar manipulation because they have similar texture and weight, while the orange and banana both contain peels, deform, and are edible. People can reason over rich representations of common objects that these words evoke.

Planning is where people first learn abstraction and simple examples of post-conditions through trial and error. The most basic scripts humans learn start with moving our own bodies and achieving simple goals as children, such as stacking blocks. In this space, we have unlimited supervision from the environment and can learn to generalize across plans and actions. In general, simple worlds do not entail simple concepts: even in a block world concepts like “mirroring” appear [@Bisk2018]. Humans generalize and apply physical phenomena to abstract concepts with ease.

In addition to learning basic physical properties of the world from interaction, WS4 also allows the agent to construct rich pre-linguistic representations from which to generalize. [@hespos2004] show pre-linguistic category formation within children that are then later codified by social constructs. Mounting evidence seems to indicate that children have trouble transferring knowledge from the 2D world of books [@bar2013] and iPads [@Lin2017] to the physical 3D world. So while we might choose to believe that we can encode parameters [@chomsky1981] more effectively and efficiently than evolution provided us, developmental experiments indicate doing so without 3D interaction may prove difficult.

Part of the problem is that much of the knowledge humans hold about the world is intuitive, possibly incommunicable by language, but still required to understand language. Much of this knowledge revolves around physical realities that real-world agents will encounter. Consider how many explicit and implicit metaphors are based on the idea that far-away things have little influence on manipulating local space: “a distant concern” and “we’ll cross that bridge when we come to it.”

Robotics and embodiment are not available in the same off-the-shelf manner as computer vision models. However, there is rapid progress in simulators and commercial robotics, and as language researchers we should match these advances at every step. As action spaces grow, we can study complex language instructions in simulated homes [@shridhar:cvpr2020] or map language to physical robot control [@blukis2019; @chai:ijcai18]. The last few years have seen massive advances in both high fidelity simulators for robotics [@todorov2012mujoco; @pybullet; @isaac; @xiang:cvpr20] and the cost and availability of commodity hardware [@baxter; @campeau2019kinova; @murali2019pyrobot].

As computers transition from desktops to pervasive mobile and edge devices, we must make and meet the expectation that NLP can be deployed in any of these contexts. Current representations have very limited utility in even the most basic robotic settings [@Scalise2019], making collaborative robotics [@rosenthal2010effective] largely a domain of custom engineering rather than science.

5 WS5: The Social World↩︎

Interpersonal communication is the foundational use case of natural language [@dunbar1993coevolution]. The physical world gives meaning to metaphors and instructions, but utterances come from a source with a purpose. Take J.L. Austin’s classic example of “BULL” being written on the side of a fence in a large field [@austin1975things]. It is a fundamentally social inference to realize that this word indicates the presence of a dangerous creature, and that the word is written on the opposite side of the fence from where that creature lives.

Interpersonal dialogue as a grand test for AI is older than the term “artificial intelligence,” beginning at least with [@turing2009computing]’s Imitation Game. Turing was careful to show how easily a naïve tester could be tricked. Framing, such as suggesting that a chatbot speaks English as a second language [@sample2014scientists], can create the appearance of genuine content where there is none [@weizenbaum1966eliza]. This phenomenon has been noted countless times, from criticisms of Speech Recognition as “deceit and glamour” [@pierce1969whither] to complaints of humanity’s “gullibility gap” [@marcus2019rebooting]. We instead focus on why the social world is vital to language learning.

5.0.0.1 Language that Does Something

Work in the philosophy of language has long suggested that function is the source of meaning, as famously illustrated through Wittgenstein’s “language games” [@wittgenstein1953philosophical; @wittgenstein1958blue]. In linguistics, the usage-based theory of language acquisition suggests that constructions that are useful are the building blocks for everything else [@langacker1987foundations; @langacker1991foundations]. The economy of this notion of use has been the subject of much inquiry and debate [@grice1975logic]. In recent years, these threads have begun to shed light on what use-cases language presents in both acquisition and its initial origins in our species [@tomasello2009constructing; @barsalou2008grounded], indicating the fundamental role of the social world.

WS1, WS2, WS3, and WS4 expand the factorizations of information available to linguistic meaning. allows language to be a cause instead of just a source of data. This is the ultimate goal for a language learner: to generate language that does something to the world.

Passive creation and evaluation of generated language separates generated utterances from their effects on other people, and while the latter is a rich learning signal it is inherently difficult to annotate. In order to learn the effects language has on the world, an agent must participate in linguistic activity, such as negotiation [@yang-etal-2019-lets; @he2018decoupling; @lewis2017deal], collaboration [@Chai_Fang_Liu_She_2017], visual disambiguation [@Anderson2017a; @lazaridou2016multi; @liu:aaai15], or providing emotional support [@rashkin-etal-2019-towards]. These activities require inferring mental states and social outcomes—a key area of interest in itself [@Zadeh_2019_CVPR].

What “lame” means in terms of discriminative information is always at question: it can be defined as “undesirable,” but what it tells one about the processes operating in the environment requires social context to determine [@bloom2002children]. It is the toddler’s social experimentation with “You’re so lame!” that gives the word weight and definite intent [@ornaghi2011role]. In other words, the discriminative signal for the most foundational part of a word’s meaning can only be observed by its effect on the world, and active experimentation is key to learning that effect. Active experimentation with language starkly contrasts with the disembodied chat bots that are the focus of the current dialogue community [@roller:arxiv20; @adiwardana2020towards; @zhou2020design; @chen2018gunrock; @serban2017deep], which often do not learn from individual experiences and whose environments are not persistent enough to learn the effects of actions.

5.0.0.2 Theory of Mind

When attempting to get what we want, we confront people who have their own desires and identities. The ability to consider the feelings and knowledge of others is now commonly referred to as the “Theory of Mind” [@nematzadeh2018evaluating]. This paradigm has also been described under the “Speaker-Listener” model [@stephens2010speaker], and a rich theory to describe this computationally is being actively developed under the Rational Speech Act Model [@frank2012predicting; @bergen2016pragmatic].

A series of challenges that attempt to address this fundamental aspect of communication have been introduced [@nematzadeh2018evaluating; @sap-etal-2019-social]. These works are a great start towards deeper understanding, but static datasets can be problematic due to the risk of embedding spurious patterns and bias [@devries:arxiv20; @le-etal-2019-revisiting; @gururangan2018annotation; @glockner2018breaking], especially because examples where annotators cannot agree (which are usually thrown out before the dataset is released) still occur in real use cases. More flexible, dynamic evaluation [@zellers2020evaluating; @dinan2019build] are a partial solution, but true persistence of identity and adaption to change are both necessary and still a long way off.

Training data in WS1-4, complex and large as it can be, does not offer the discriminatory signals that make the hypothesizing of consistent identity or mental states an efficient path towards lowering perplexity or raising accuracy [@liu2016not; @devault2006societal]. First, there is a lack of inductive bias [@martin2018event]. Models learn what they need to discriminate between potential labels, and it is unlikely that universal function approximators such as neural networks would ever reliably posit that people, events, and causality exist without being biased towards such solutions [@mitchell1980need]. Second, current cross entropy training losses actively discourage learning the tail of the distribution properly, as statistically infrequent events are drowned out [@pennington-etal-2014-glove; @holtzman2019curious]. Meanwhile, it is precisely human’s ability to draw on past experience and make zero-shot decisions that AI aims to emulate.

5.0.0.3 Language in a Social Context

Whenever language is used between people, it exists in a concrete social context: status, role, intention, and countless other variables intersect at a specific point [@wardhaugh2011introduction]. These complexities are overlooked through selecting labels on which crowd workers agree. Current notions of ground truth in dataset construction are based on crowd consensus bereft of social context. We posit that ecologically valid evaluation of generative models will require the construction of situations where artificial agents are considered to have enough identity to be granted social standing for these interactions.

Social interaction is a precious signal, but initial studies have been strained by the training-validation-test set scenario and reference-backed evaluations. Collecting data about rich natural situations is often impossible. To address this gap, learning by participation, where users can freely interact with an agent, is a necessary step to the ultimately social venture of communication. By exhibiting different attributes and sending varying signals, the sociolinguistic construction of identity [@ochs1993constructing] could be examined more deeply. Such experimentation in social intelligence is simply not possible with a fixed corpus. Once models are expected to be interacted with when tested, probing their decision boundaries for simplifications of reality and a lack of commonsense knowledge as in [@Gardner:2020; @Kaushik2020Learning] will become natural.

6 Self-Evaluation↩︎

We use the notion of World Scopes to make the following concrete claims:

You can’t learn language ...↩︎

6.0.0.1 ... from the radio (Internet). WS2 \(\subset\) WS3

A task learner cannot be said to be in WS3 if it can succeed without perception (e.g., visual, auditory).

6.0.0.2 ... from a television. WS3 \(\subset\) WS4

A task learner cannot be said to be in WS4 if the space of its world actions and consequences can be enumerated.

6.0.0.3 ... by yourself. WS4 \(\subset\) WS5

A task learner cannot be said to be in WS5 unless achieving its goals requires cooperating with a human in the loop.

By these definitions, most of NLP research still resides in WS2. This fact does not invalidate the utility or need for any of the research within NLP, but it is to say that much of that existing research targets a different goal than language learning.

6.0.0.4 Where Should We Start?

Many in our community are already examining phenomena in WSs 3-5. Note that research can explore higher WS phenomena without a resultant learner being in a higher WS. For example, a chatbot can investigate principles of the social world, but still lack the underlying social standing required for WS5. Next we describe four language use contexts which we believe are both research questions to be tackled and help illustrate the need to move beyond WS2.

Second language acquisition when visiting a foreign country leverages a shared, social world model that allows pointing to referent objects and miming internal states like hunger. The interlingua is physical and experiential. Such a rich internal world model should also be the goal for MT models: starting with images [@huang-etal-2020-unsupervised], moving through simulation, and then to the real world.

Coreference and WSD leverage a shared scene and theory of mind. To what extent are current coreference resolution issues resolved if an agent models the listener’s desires and experiences explicitly rather than looking solely for adjacent lexical items? This setting is easiest to explore in embodied environments, but is not exclusive to them (e.g., TextWorld [@cote18textworld]).

Novel word learning from tactile knowledge and use: What is the instrument that you wear like a guitar but play like a piano? Objects can be described with both gestures and words about appearance and function. Such knowledge could begin to tackle physical metaphors that current NLP systems struggle with.

Personally charged language: How should a dialogue agent learn what is hurtful to a specific person? To someone who is sensitive about their grades because they had a period of struggle in school, the sentiment of “Don’t be a fool!” can be hurtful, while for others it may seem playful. Social knowledge is requisite for realistic understanding of sentiment in situated human contexts.

6.0.0.5 Relevant recent work

The move from WS2 to WS3 requires rethinking existing tasks and investigating where their semantics can be expanded and grounded. This idea is not new [@chen:icml08; @feng-lapata-2010-topic; @bruni2014; @lazaridou-etal-2016-red] and has accelerated in the last few years. [@multi30k] reframes machine translation with visual observations, a trend extended into videos [@Wang_2019_ICCV]. [@tacos:regnerietal:tacl] introduce a foundational dataset aligning text descriptions and semantic annotations of actions with videos. Vision can even inform core tasks like syntax [@shi-etal-2019-visually] and language modeling [@ororbia-etal-2019-like]. Careful design is key, as visually augmented tasks can fail to require sensory perception [@thomason:naacl19].

Language-guided, embodied agents invoke many of the challenges of WS4. Language-based navigation [@Anderson2017a] and task completion [@shridhar:cvpr2020] in simulation environments ground language to actions, but even complex simulation action spaces can be discretized and enumerated. Real world, language-guided robots for task completion [@tellex:rss14] and learning [@she:sigdial14] face challenging, continuous perception and control [@tellex:arcras:20]. Consequently, research in this space is often restricted to small grammars [@paul:ijrr18; @walter:rss13] or controlled dialog responses [@thomason:jair20]. These efforts to translate language instructions to actions build towards using language for end-to-end, continuous control (WS4).

Collaborative games have long served as a testbed for studying language [@Werner1991] and emergent communication [@schlangen2019grounded; @ref-games-2018; @chaabouni2020compositionality]. [@suhr-etal-2019-executing] introduced an environment for evaluating language understanding in the service of a shared goal, and [@andreas-klein-2016-reasoning] use a visual paradigm for studying pragmatics. Such efforts help us examine how inductive biases and environmental pressures build towards socialization (WS5), even if full social context is still too difficult and expensive to be practical.

Most of these works provide resources such as data, code, simulators and methodology for evaluating the multimodal content of linguistic representations [@schlangen2019language; @silberer-lapata-2014-learning; @bruni-etal-2012-distributional]. Moving forward, we encourage a broad re-examination of how NLP frames the relationship between meaning and context [@benderclimbing] and how pretraining obfuscates our ability to measure generalization [@linzencan].

7 Conclusions↩︎

Our World Scopes are steep steps. WS5 implies a persistent agent experiencing time and a personalized set of experiences. With few exceptions [@carlson2010toward], machine learning models have been confined to IID datasets that lack the structure in time from which humans draw correlations about long-range causal dependencies. What if a machine was allowed to participate consistently? This is difficult to test under current evaluation paradigms for generalization. Yet, this is the structure of generalization in human development: drawing analogies to episodic memories and gathering new data through non-independent experiments.

As with many who have analyzed the history of NLP, its trends [@church2007], its maturation toward a science [@steedman-2008-last], and its major challenges [@Hirschberg2015; @McClelland:2019], we hope to provide momentum for a direction many are already heading. We call for and embrace the incremental, but purposeful, contextualization of language in human experience. With all that we have learned about what words can tell us and what they keep implicit, now is the time to ask: What tasks, representations, and inductive-biases will fill the gaps?

Computer vision and speech recognition are mature enough for investigation of broader linguistic contexts (WS3). The robotics industry is rapidly developing commodity hardware and sophisticated software that both facilitate new research and expect to incorporate language technologies (WS4). Simulators and videogames provide potential environments for social language learners (WS5). Our call to action is to encourage the community to lean in to trends prioritizing grounding and agency, and explicitly aim to broaden the corresponding World Scopes available to our models.

Acknowledgements↩︎

Thanks to Raymond Mooney for suggestions, Paul Smolensky for disagreements, Catriona Silvey for developmental psychology help, and to a superset of: Emily Bender, Ryan Cotterel, Jesse Dunietz, Edward Grefenstette, Dirk Hovy, Casey Kennington, Ajay Divakaran, David Schlangend, Diyi Yang, and Semih Yagcioglu for pointers and suggestions.

---

1. A parallel discussion would focus on the hardware required to enable advances to higher World Scopes. Playstations [@pinto2009high] and then GPUs [@NIPS2012_4824] made many WS2 advances possible. Perception, interaction, and robotics leverage other new hardware.↩︎

2. Or the 1,600 classes of [@Anderson2017].↩︎

3. Torchvision/Detectron2 include dozens of trained models.↩︎