Natural Language Processing
Professor Julia Hockenmaier explains how natural language processing will impact construction and other fields.
Natural language processing represents an emerging opportunity for AI researchers looking to improve the construction industry — and industry in general. We talked to Computer Science Professor Julia Hockenmaier, an NLP expert and Grainger Engineer since 2007, about the future of the field. Her comments have been condensed and edited.
Exciting Fundamental Problems
It’s becoming more and more interesting to think about how we can apply our fundamental research ideas to actual industry applications. I didn’t know anything about the construction industry — what are the problems? What are the issues, so on? — until Mani [Golparvar] approached me. Then I realized: Oh, this is actually really, really interesting. There’s a lot of exciting fundamental problems that all have a sort of practical impact.
In particular, it turns out to schedule the construction of a big building of any sort, there are a lot of documents — schedules, some of which get physically adjusted on the fly, lots of regulation that everyone needs to understand so what you’re building is in compliance. And then it’s these BIMs, these 3D or 4D models of what you’re building. The model of what your building itself and the model of how it develops over time. And now we can think about how to link the textual descriptions of the schedule and the plans and so on to this 4D model. That’s something that I’m really excited about getting started on. There’s a lot of really really interesting problems in this space.
Think about how does this huge, complex building get put together? What are the different steps involved? When do you dig the foundation? When do the electricians come? When do the plumbers come? What’s the right sequence in which things have to happen? Some things have to happen before other things can get started, right? You’d like to make sure that the model gets constructed as efficiently as possible. So can you take a plan of a building and ask questions like...Can you come up with a detailed work schedule from the BIM? Or if you have the schedule listing different steps, can you verify that this is the right order? Or can you come up with an ordering of these steps? If you think about the final [textual] products, it’s hard linking what’s actually happening on the construction site.
Working With Robots
A big problem going forward is going to be climate change, building maintenance. Surfside. Could you use AI to automatically do building inspections or make them more efficient or better? If you’re going to have to inspect all the coastal condos, you’re not going to be able to do those manually. And then you get into the importance of regulations. Each municipality is going to have its own [text-based] regulations. Understanding if something is or isn’t up to code is going to be important and it’s not going to be easy. That requires physical observations and natural language processing. You might not have a model. So you’re going to need sensory input.
This sort of dangerous inspection and maintenance work may be done with autonomous vehicles or robots, so we’re thinking a lot about robots and how to interact with them. There’s this longstanding problem: Can you use natural language to communicate with robots. This is something we’re starting to work on. There are a lot of practical problems — where do you get the data? The real world is changing as you’re interacting with the robot. How do you adapt to that?
Is This Really What You Mean?
A lot of actions can only be done if certain preconditions are met. You can only drill a hole in a wall if the wall exists already. Perhaps, you have to plaster the wall before you paint it. There are going to be preconditions that have to be met for certain actions to be executable
If someone tells you to do something, do you know what needs to be the case for you to execute that? Are the entities needed actually there? Can you identify them? Do they have to be in a certain state? Do you know what the state should be? Do you know if it’s in that state? Primarily it’s a question of what’s called world knowledge. That also requires a relationship with computer vision. What is the current state of the world and does this meet the preconditions? There are real safety issues there.
The other question that would come up in this context is: If someone told you to do something, can you precompute what would happen? Is it safe to actually execute those actions? Let’s say someone tells you to remove a post. If you think it’s feasible, do you think it will cause a collapse? Can you identify this and do you know when to ask for further clarification? What are the consequences of your actions and do you need to ask ‘Is this really what you mean?’
General Lessons
In working with industry people, we’re constantly asking what would be a useful application? What’s a useful task, what’s a feasible task, and how do we actually get data to attack those?
I think the construction industry is particularly interesting because there are so many tangible problems, because it is so physical. But there are a lot of industries or problems that have similar issues. So I’d like to learn general lessons that could be applied to other areas that could really leave a mark. Whatever we’re learning can be applied to other areas as well, even if they have their own specific tasks and problems. And as educators, we want to train enough people to be able to work in the industry.
You need practitioners who can understand these technologies. Engineers who can use these tools. And computer scientists who understand the industry well enough to work in construction or apply these things to other industries.