OASIS 4

OASIS: Thanks for sitting down to talk with us. What's changed for you in your field since you started? ‍

I think right now the field has really achieved what G.W. Bush wanted, which was that the NIH decided that everything has to have a purpose and it had to address some practical problem. And it had to provide a concrete solution. It makes it harder to justify projects that are focused more on understanding the nature of the cognitive and linguistic systems.

‍

‍OASIS: Who are your professional heroes?

PW: I can think of one person, Geoff Hinton. He's been this person that has been like the second and third author on important papers, seemingly in the background. And you start to see over the length of a career that he may have been the secret sauce to a lot of other people's shining moment. That modesty, that humility, impresses me, and I'm really happy he won the Nobel.

‍

‍OASIS: Oh yeah, I saw some physicists complaining that it wasn't physics.

PW: Yeah, it wasn't! He's someone who kept pursuing what he thought was the right answer and  no one was with him for many years. So he had that confidence and certainty. It led at the very least to a provocative set of outcomes in the form of these large language models.

‍‍

OASIS: What would you like to tell students or early career researchers?

PW: There's going to be this tension. You want to make your unique mark, something you can say is yours. But at the same time, the problems have gotten pretty difficult at this point for single players to make a dent. And you have to find a partner that's doing more of the modeling side of things. And if you're a modeler, you definitely need someone who understands the content area. I'd be on the lookout for that person that could complement you because I think at this point it's too challenging, especially if you want funding.

‍‍

OASIS: What is the strangest hill you would die on?

PW: People flirt with the idea of the end of theory. The problems are going to be solved with modeling methods that are driven by data, and the outcomes will be those we don't really understand, and we need to let go of the idea that we need to understand it. This approach is contrasted with the one where people come up with a theory and then, if you will, look for the confirmation empirical data. Now, in that situation prediction was dependent on theory. But we don't need that anymore.

But I'm not going to give up on theory. So the hill is, we should pursue grand theories even of cognition and of language and in philosophy, even though we now have the capability to achieve predictions without understanding how. We can re-express what was achieved in a data-driven way in terms that look like a theory. I still believe there's a calculus of meaning composition!

So I think the data-driven approaches are awesome and we ought to pursue them. Because they will help us develop those theories. And I think especially for cognition and for language. One reasonably might ask, well, why do you need theory, really, other than maybe it's just a habit of training. It's just in my experience that the reason it's important is because if you use a data-driven approach, the project ends, if you will, at the point when you do pretty good prediction. And then there's nothing more to do almost.

‍‍

OASIS: Aren't these models called "generative"? Shouldn't they generate what's next?

PW: Yes, but in terms of our own efforts, like what do you do next? Why do you do it? Theory is pretty key to filling in or explaining why you should do it. It's maybe more for us, to help guide our efforts. But with only a black box, you have a system that does the prediction, a calculator that does the prediction, what do you do next?

‍‍

OASIS: What do you feel optimistic about?

PW: A long-term motivation or question was how do you get mind out of matter? Like cognition or emotion or certain behaviors, out of physical processes in neurons? That has always struck many people as really beyond what we'll ever achieve, but now I think we're coming closer to a solution. These larger systems are giving someone like me more confidence that this is something that might be solvable. So it's moved from being a total mystery to being a potential project.

Another thing that's kind of cool is that we have systems now that can do a trillion operations per second, and the human brain is estimated around 60 trillion connections given the number of neurons. But computers' compute capabilities are approaching or maybe have already met the human compute potential. And that's a pretty big landmark. And that's happened roughly around now, probably. So I'm excited about that.

And if that's true, then the limitation, like being able to understand the system is no longer a problem. We have the compute, we just don't have the science yet. And I think that I'm excited by that. We are in a new period. We have enough processors to test our theories, but we don't have the science yet.

We know these systems are not learning like people learn. So what could we add to them that would help make that difference? To give an example, I'm intrigued with how the language system is laid out.in both sort of at a theory level as well as sort of like at the cortical level. If we look at it in terms of cortex, it looks as if let's say areas relevant to verb processing are close to areas relevant to the processing of motion and spatial relations.

That observation is very compatible with linguistic proposals about what many verbs mean.And so it raises, I think, some really cool ideas or possibilities that the human system is really set up to take advantage of this non-linguistic hardware. You might even call it parasitic. But that's a kind of negative! But it like it's glomming on to structures that were there prior to language.

And those structures are imposing pressures on the learning system. That maybe could accelerate the learning process or enable it. And so, the future looks exciting in this regard, because right now the large language models are not designed with this kind of architectural constraint built in. And if they were, we could see really different kinds of performance.

‍‍

OASIS: Thanks for that optimistic outlook. We're looking forward to hearing your talk at OASIS 4.

PW: Thanks for the invitation, I'm looking forward to it too.

OASIS: Thanks for sitting down with us. What advice would you give linguists presenting their work to philosophers?

MS: I'm answering this question from the perspective of a linguist who spent a year in a philosophy department (Durham) as the only linguist. This was very formative for me, as it's really good to present to people who have different assumptions, and defend our basic assumptions inherited through education and supervision. That's not something we have to do (usually) when we present to each other within the discipline.

One thing about philosophers is that they're so good at arguing and they're focused on the structure of an argument: Is this coherent? Why do people assume that? What if we assume something else? is it really true that all languages are like that? Linguists are often a little less focused on the implications of arguments and more interested in data. Being around philosophers helped me become more balanced in this respect, I think.

I first felt a little affronted! But after a while, I felt this kind of constant questioning was really useful, made me stronger as a scholar, and actually made me think of a lot of new research questions.

If you're a linguist talking to philosophers, be prepared to be questioned a lot and be on your toes to defend your assumptions and your framework, as well as your data and your specific arguments - and of these, being able to defend the framework is probably the most important

Another thing is: think about how best to present your data. We can often become obsessed with small contrasts, but ultimately we have to connect to bigger questions which are more interesting (or tangible) to philosophers.

I also think it’s important not to simplify the data, though. Languages are complex! Depending on their background, philosophers may have a reluctance to accept the extreme complexity of grammar and meaning cross-linguistically, but we do have to stick to our guns and push back on simplifications.

‍

‍OASIS: Tell us about an interdisciplinary event that you enjoyed because of its structure.

MS: When I was at Durham, we organized a really big conference called "The past and future of universal grammar." We invited people from really diverse fields - not just generative grammar but also construction grammar, sociolinguistics, computational linguists, historical linguistics, even archaeologists, and obviously philosophers as well. This was the first time I saw people really having heated academic arguments—a bit disturbing as a polite English person—there were some tense moments where people were actually shouting at each other!

But the reason I enjoyed it is because it really clarified for me the reason why it's worth having such meetings. It's just not possible to read everything - if you're in construction grammar, and you want to critique generative grammar, you will read maybe Chomsky, but can’t possibly read that widely so you end up with a skewed vision of what generative grammar is about. And the same is true in the opposite direction of course: construction grammar isn't one monolithic thing’ it’s a huge group endeavour. The only way to really engage across frameworks is to get people in a room and make them listen to each other.

This event made me think about teaching as well. In classes I might set students one paper on a large topic to read, e.g., here's a paper on signed languages—and they really find it hard. It made me realize how important it is to write those papers, like handbook articles, to reach out. I think they're really valuable. Even writing for the public, to distill things and make them digestible. Some people do that really well. Collaboration is also clearly an important part of this.

‍

‍OASIS: How did you get into linguistics?

MS: I did my degree in modern languages because I love learning languages. And I was a little dismayed when I got to university and found all the courses were literary analysis rather than language. So I tried to seek out whatever wasn't literature, so that was linguistics. I loved it because it answered the why questions I had, for instance, why people in the real world had different pronunciations from what we were taught. This kind of thing may seem a little trivial now, but as a kid you ask your teacher, and they don't know the answer. And in fact they don't even know that anyone knows the answer!

‍

‍OASIS: Wow, that's really striking.

MS: Yeah, there's really a lack of linguistics in other industries that work with languages, including language teaching. We started a program to address this, where we work with schoolteachers and develop materials to help them understand and teach linguistics, and be prepared to answer questions from their students. So that's quite a big part of what I'm doing right now.

Young people have so many questions; they need to know that there are answers to those questions. We've really failed to convey the major findings of our discipline to everyone else— other academic fields but also language teachers, exam boards, government agencies—for example, people simply not understanding non-standard dialects are not indicatove of a language deficit. We need to do better because people are really excited about linguistics when they hear about it. So I hope it's changing now and linguists are able to reach out a bit more beyond the ivory towers.

‍

OASIS: Thanks for doing this interview! We're looking forward to your talk at OASIS 4.

MS: Thanks, I'm looking forward to it too. See you there!

OASIS: Thanks for sitting down with us. What's your favorite ontological entity and why?

DA: I had a big dinnertime argument about this with Peter Svenonius and Artemis Alexiadou and a bunch of other people after a conference up in Tromsø once. And I just thought I was saying normal things about what syntactic objects are, and it turns out that they both disagreed with what I was saying. They were insistent that essentially syntactic objects are symbols, and they're computed over with a computational device, while I thought of them like objects, built by a device. A symbol needs to represent something else, right? And I couldn't get out of them what they thought it represented.

I know there are philosophers out there, for example, the philosopher Georges Rey, who thinks that linguistic entities are definitely symbols and represent things. But I don’t think those linguistic entities are symbolic or representational.

‍

‍OASIS: So we're talking about syntactic features.

DA: Syntactic objects, really. Not the symbols in the theory that represent those objects, but the objects themselves. Those things,  some people think of them as symbols. Which I don't understand. Philosophers like Rey think they represent some kind of abstract object. I don't think linguists believe this when they really think about it. I don't think Chomsky believes it, for example.

I don't think the notion of representation is what's going on with the structure for John walks.

That doesn't represent anything. Sure, it has interpretations, but it's not by virtue of it being a symbol. It's just a structure with no inherent representational nature to it. It only gets its connection to other things via some system that says, ah, I'm going to make you mean this.

‍

‍OASIS: Yeah, I mean, compositional semantics, right?

DA: Yeah, semantics, or phonology or sign phonology or whatever - just a system that can interpret the syntactic object. But as far as the actual things that the theory is about, they are, I think, non-symbolic. And I think that means that the theory of them therefore has to be a constructive theory, that tells you how to construct structures.

‍

‍OASIS: If I were to use the word instructions, are they instructions? I mean, are features instructions?

DA: Well, there are only instructions inasmuch as there is a device which can interpret them as instructions. They're just a kind of thing, I think. So ontologically, they're just the kind of thing that can be manipulated by internal systems to create larger things, and they can be interpreted by systems external to that internal system, but still mind-internal, which might interpret them as instructions.

But imagine there was no such system. Imagine that there was no phonology and imagine there was no semantics. I still think that you could still have a syntax of bits that would just put the bits together. I always just understood that this is what we were doing. We're just building structures that are accessed by systems of form and meaning.

‍

‍OASIS: What hangs on this question?

DA: There was an interview with Chomsky where he had said something like "well, you know, we just use sets to build syntactic structures with, because we understand sets, even if they're abstract entities and they're probably not in the human mind and don’t really make sense". And I shared the intuition that it seems unlikely we have set like objects in our heads. So what do we have in nature? We know that there are fundamental bits that can be built into structures. These structures don't have the properties of a set, it's a molecule or something like that, with relationships between the elements in the molecule. So can we use this to think of something that's more realistic as a mental object?

And so that's what took me on to thinking of, this notion of parthood as a fundamental relationship between elements, so that when you build something complex, you build a part-whole construction, rather than building an abstract set theoretic construction.

If you look at the history of logic, early on people were really unhappy with sets as a way of building maths, because of the paradoxes. There are solutions to those, but one way people went in the early days (Lesniewsky, and Goodman and Leonard) was by building mereological logics, based on the part whole relationship rather than the set membership  relationship. And of course mereological ideas have been really big in semantics and very useful for us. The reason no one ever thought of it for syntax is because it gives you an undifferentiated structure. But say we differentiate the structure in a minimal way, what can we get out of that?

And so the system that I built up [in my upcoming book, MIT Press] is one that does that and it gives you essentially an extended projection complement relation as one way of being a part. And what you might think of as a specifier (though it's not quite a specifier), as another way of being a part, so you have differentiated part whole structures. That seemed to me to be intuitively less weird than sets for syntactic objects. It makes them look awfully like trees. You can build them in a similar way to how Chomsky's Merge works, you take two things and get a part whole relationship between them rather than a new object.  And then you can get Internal Merge exactly the same way that Chomsky did, applying that operation to two things, one of which is part of another.

So yeah, fundamentally I think that part of the motivation for me for my new approach was sort of ontological. It was like, sets are not great for syntax.

‍

‍OASIS: So you've also answered my next question, "What is everyone wrong about?"

DA: Once when I was chatting with Chris Collins about the notion of copy in syntactic theory he was like, you must remember, David, we're doing formal language theory. And I thought, not really! We're building a constructive system that licenses an infinite number of objects, rather than doing formal language theory, and that makes a material difference in how we  think about the properties of those objects. In the case at hand it was about what copies are, which I think gets muddled when we think of them as computational symbols as opposed to actual objects.

‍

‍OASIS: It feels like asking mathematicians if numbers really exist.

DA: Yeah, a little bit. But I take these things to exist as part of our minds. So the way I like to think of the syntactic object that the semantic and phonological system can interpret as instructions,  is like a gesture, like this [wiggles his fingers]. I've just made a bunch of gestures with my hand. The potentiality is in my hand. It's limited by the anatomy of my hand. And obviously it's not quite like a generative system because it has that limit but I think a sentence, or rather the structure underlying one,  is like a gesture, but of the mind.

‍

‍OASIS: That's a nice metaphor. To finish up, did you have any thoughts about the significance of LLMs?

DA: One of the really fascinating things about LLMs is about how different they are to humans in doing things that are so similar to what humans do. I think I talked about this in my 2019 popsci book [Language Unlimited] —I was interested in a newspaper story that had come out a few years before where there were computers who were set to essentially communicate with each other via a language-like system. So a little bit like an LLM generating something with English input, without human interference. And so they kind of started to communicate the things they needed to communicate to each other qua computational units. And what they did started off looking quite like English because that was what they were fed. But their posts had little errors in them which then fed into that process and quite  quickly  you got things that looked totally non-human!  And it's really fascinating because, of course, their interests and the tasks they were doing are quite non-human, since they can do things in ways that humans are really very bad at, right? So they just developed their own set of conventions and it looked all very weird.

If you look at far more recent LLMs, I mean, it's really fascinating in so many ways, but we still see something a bit similar. One of the things I think is really interesting about LLMs is that the generalizations they make are good ones—well, we don't really know what those are, but they appear to deliver the same output as humans—but they are getting to those generalizations in a way that’s quite different to what we do.

So there's that really interesting work by Tal Linzen and others where they look at attraction errors,  you know, like the keys to the cabinet are versus the keys to the cabinet is. And they show that a particular kind of LLM—LSTMs, I think they are—give rise to these attraction errors in a way that's not massively dissimilar to humans, but what conditions the errors appears to be quite distinct. The LSTMs are sensitive to the linear distance between the NP that triggers the error and the NP that should trigger the right version, whereas humans seem to be sensitive to structural distance. So if you embed the NP inside a PP, for example, you get a different outcome for the two, right? You get these different effects, even though you have the same actual output.

And if we take that further, you can imagine that LLMs are going to be way better than we are at lots of language tasks, right? I mean, Google NotebookLM, you can just stick a paper in there and it'll immediately tell you what it's about and even generate a podcast for it. We couldn't do that with such efficiency—LLMs are literally better at that language related task than we are. So that's quite interesting. What are the differences between what language models are capable of doing via generalization over massive data sets and what we are capable of doing?

And also connected to that is the fact that the requirements that humans have to learn a language, and the requirements that a LLM has, are quite different. They really need billions of bytes of data, whereas we need a lot less than that to learn a language as a kid.

I think a lot of this debate around  "oh, linguistics is now rubbish because we can do everything with LLMs" misses a difference in perspective. People who say this are  seeing language as E-language, a set of texts over which you can make generalizations, not as I-language, the internal cognitive system. LLM technology  has done  something really interesting with E-language,  way more interesting than I ever thought you'd see,  and way more useful than I ever thought you'd see. So that's absolutely something that's worth paying attention to. But I think it's doing something really pretty  different from what we're doing.

‍

‍OASIS: If semantics is about the relationships humans make between language and the world—well, there's no world with LLMs.

DA: There's a fascinating thing about humans that’s come to prominence more recently, which is that much of our world has become an online world. And then there is almost no actual world. Truth is social in a kind of  Trumpian fashion, right? Suddenly the world, for many people, is not their interaction with the physical, but with the digital, so it’s a vast network of propositions not experiences. And that’s why AIs hallucinate, right? That’s all they have access to too.

‍

‍OASIS:Yeah, except we're not in the matrix, right? We're not looking at the falling green lines, we're having experiences.

Yeah. And even totally online people still come with human psychology. I mean, you're right. Obviously, we live in the world, I'm drinking tea, we're chatting, right? But my drinking tea? I kind of know what I'm doing. Are you an AI?

‍

‍OASIS: I don't know, maybe we're actually brains in vats? But we have to soldier on as if we're not!

DA: So I think that the way forward with that is a maintenance of multiple critical perspectives on whatever you're doing, right? We can probably get a better view by being critical about all of these kinds of things and figuring out for ourselves rather than just going down that kind of rather barren skepticism about reality.

‍

‍OASIS: Let's assume we're not brains in vats and we'll see you in York in January, looking forward!

DA: Thanks, I'm looking forward to it too.