Sasan Tavakkol, Software Engineer at Google AI, elaborates the rǝ platform and explains how spatial data analysis and AI could be employed for a more intelligent city and tomorrow's mobility systems.
1. Hey, Sasan! Thanks for joining us today. Please tell us about your projects from 2018 to the present:
Let me introduce myself briefly, and then we will get to the questions. My name is Sasan Tavakkol, and I'm a software engineer at Google Research New York, or Google AI, as it is known externally. Before that, I was a Ph.D. student at the University of Southern California. I started with Google about two years ago. Let me tell you a bit about my projects. My first project at Google was an open-source project called Kartta Labs. Kartta Labs mission is to create a 4D map of the world. We want to have the 3D buildings, the 3D model of buildings, plus a time dimension — think of Google Earth with a time-slider. Such that you have the 3D world, but you also have the time dimension; you can go back in time in Kartta Labs and see Manhattan, say 100 years ago.
2. About the rǝ platform; the platform that you and your google AI team members initiated. Please tell us about that project:
So Kartta Labs, as I said, is an open-source project and an open beta project, which aims to create a 3D map of the world with a time dimension. We have different modules, different components that I'll explain. One module — our major module — is Kartta Labs map services. So to have the 3D buildings, we need first to have the map. We want to know where that building is located, right? So we want to have a map server — think of, again, Google Maps — with a time-slider. So you could go back in time and see the map of the 1940s of, say, New York or Los Angeles. This is our first module. The second module is the 3D reconstruction part, where some machine learning algorithms would look at a photo and try to make sense of it and create and reconstruct that photo in 3D.
Sub-modules of the Kartta Labs software system.
Image Credits: https://arxiv.org/pdf/2010.06536.pdf
3. Could you please elaborate on the crowdsourcing platform of rǝ city?
In Kartta Labs, we relied on crowdsourcing and machine learning to achieve our goal. There are two parts; the maps and the 3D reconstruction. In the maps module, we relied on people; we relied on crowdsourcing to collect these maps, upload maps to georectify them, and vectorize them. By georectifying I mean that we have this historical map, somebody uploaded that map, but we need to know where that map is located. And we need a correspondence between every point on historical maps and contemporary maps. So we want to know if this point on this map corresponds to this specific point in Manhattan. That's called rectification or geo-referencing. We have some automated algorithms to roughly guess the map’s location to help our users get started. After you upload the map, our website will roughly guess the location of the map; it will say that “okay, this looks like downtown Manhattan.” And then, the user will start putting points on the historical map and the corresponding point on a contemporary map. Then, based on these, as we call them to control points, the historical map would be warped or will be georectified such that each point of it corresponds to a point on contemporary maps. And the second step, we relied on our users to vectorize. This map vectorization means that we see that there's a building here; we see the footprint of this building. But that's an image, it's not very useful, we cannot search it, we cannot provide that data to our users in a useful way. So we want to know exactly what are the boundaries of this footprint. And that's called vectorization. So users will choose an image to rectify, and then they will start tracing artifacts on the map, tracing the building, tracing the roads, the rivers, and that's called vectorization. After this step, this data is stored in our backend map services and is available, open data, open-source.
4. Could you please elaborate on the temporal map server of rǝ city?
So that was all about the map platform. But a subtle but important difference between our map and the ordinary maps was this temporal aspect. When users trace a map or upload a map, they will say, “okay, this map is from the 1940s”. Or when they trace the footprint, they will try to add some time aspect; if that information is available on the map, they will say “this building was built in 1935, and it was standing in 1980” if that information is available. Of course, if it's not available, they will take their best guess they will say, “okay, I see that there's a building here in the 1940s. I don't see the same building on a map from the 1970s. So that means that probably that building was torn down”. And I will put the end date for the 1970s. So that's about the temporal aspect of our map.
5. Could you please elaborate on the 3D experience platform of rǝ city?
As I said, our primary goal was to provide a 3D experience, as we call it, a collaborative time travel experience for our users. We want to take our users back in time, and we want to help them see, say, their childhood neighborhood in 3D to achieve this goal. We had the 2D maps, so we know where those buildings are, we have the footprint of the buildings, and we needed a bit of extra information to create the 3D model for the buildings. If we didn't have much information, we rely on the height of that building if that information was available. Or maybe the number of floors of the building. And then, we will just extrude the rough 3D geometry of the building. But if we had a photo of that building, a user — users as I said, they could upload historical photos to our system — they could say that “okay, this photo belongs to this building”, then our deep learning algorithm would look at this photo and do a segmentation job on it to recognize; where are the windows, where are the doors, and we'll find all those instances. And we will try to recreate — the algorithm will create a 3D model based on that information, and also the footprint of the building from the maps module. So we will have a rough geometry of the building based on the footprint and the artifacts on each facade, like the windows, the doors, etc., and then we will create the 3D building and put it on the map.
So Kartta Labs, as I said, our major goal is to create this for the map of the world. But the applications of Kartta Labs are endless. We want Kartta Labs to act as a platform where users can make any application on top of it. They could add their data or add another layer of information on top of our data. And that's kind of our primary goal. And we are excited to see what people will come up with, what applications they will come up with this kind of historical data. One possible use case is studying land use and how it changes over time. Because as I said, we have this information, we have the maps of the different cities over time. So a researcher can put this together and study how land use has changed over time. And they can augment this data with data from elsewhere, like population data or transportation data. They can augment this data on our platform and then make a better conclusion about whatever they are interested in. Users can get creative with this kind of data. And we hope to see these kinds of applications. It can have a lot of applications for transportation, especially if users can augment transportation data on our system. And they can study over time how adding streets or maybe blocking the street has changed the transportation pattern. And that can definitely help researchers to make informed decisions about what they should do next.
Recreating Historical Streetscapes Using Deep Learning and Crowdsourcing: Image Credits: https://ai.googleblog.com/2020/10/recreating-historical-streetscapes.html
6. Could your platform be employed to forecast the land use and city development around NY and NJ to find the required transportation infrastructure? And What type of data should be employed?
One interesting pattern that I think researchers can get out of our historical map data is urbanization. I think if we study how cities are getting larger and suburbs are getting smaller, we can figure out how people — especially if you augment this information with the population information from US Census — have moved from suburbs and from rural areas, to the cities and to downtown. We can learn a lot from this pattern.
7. How important is it for transport authorities to forecast land use patterns and visualize these predictions to glean insights from them?
But something that pandemic showed us, urbanization didn't really help when we have a pandemic. So the cities were overcrowded, and we saw how a disease like COVID-19 could be spread in the cities very quickly. But it also showed us something else moving forward —and I see that it will be an option — a lot of tech companies already gave their employees this option to work remotely. This may cause another migration of population from urban areas to suburbs and it has already started, but this will probably be accelerated, when we have true autonomous cars when the mass of the population can have access to these kinds of autonomous cars. I can see that people — even those people that have to work in urban areas — may decide to work in suburbs and commute to their work every day. Because first, they are not driving. Second, they can be productive in their car, they can start checking their emails, maybe answering some phone calls. And I also expect that autonomous cars will hopefully reduce traffic jams and will help with congestion. If this is true, then we will be able to live in the suburb but work in downtown; I think by studying the pattern of urbanization, say on our platform re.city, we can get some insight about the reverse thing that is going to happen, and that's that people are actually going to move back to the suburb.
8. For people in the civil engineering and transportation industry who like to follow your journey to become a Software Engineer, what steps you suggest:
This is a great question. I started my journey as a civil engineer; I got a Bachelor's and a Master's in Civil Engineering from Tehran Polytechnic. And then, I earned my Ph.D. in computational hydrodynamics, which falls somewhere between civil engineering and computer science. But today, I work for Google AI as a software engineer in research. I know that many people like to take this path, and I will be happy to discuss how I did it. I ended up doing a formal education in computer science. I got a Master's from USC while I was working toward my Ph.D. But I will say that's not required. The cool thing about the software engineering industry and the computer science industry is that you don't need to have a degree in computer science to work. If you know what you are supposed to know, you will pass the interviews, and the companies will happily hire you. And you won't really feel like you are different. The companies will be happy to hire you if you know what you're supposed to know, even if you don't have the degree. And that's pretty good about CS and software engineering.
To give some pointers about how you want to do the switch, I will say the first thing that you need to think about is whether you should do it or not. Don't just follow the crowd. Don't just say that, “yeah, I want to be a software engineering because that's like the hot thing today.” Make sure that you love this, make sure that this job is what you like — I love it. I know a lot of people who love it. But I also know a lot of people just don't like sitting behind the computer and coding; it’s not for everyone. But if you like solving problems, if you like having a large impact, a scalable impact, this is a very cool industry. And there's a lot of job opportunities in computer science. So if you have a love for solving math problems and serious coding problems, this is absolutely a good decision for you.
And how should you do the switch? Well, you have the internet today, and you can find anything on it. You can find all the online courses; you can find all the resources that you need to get prepared for an interview. Just go on YouTube and search “how to become a computer scientist” or “how to become a software engineer,” you'll find a lot of resources. I'm not going to introduce specific resources, but I will say that there are a few things that you should focus on. You want to know the fundamentals, you want to know the algorithms, you want to know data structures. And don't just skip them, don't think that the cool thing is actually coding. The algorithm and data structures are what we use every day. And you cannot become a good software engineer without knowing about the basics. So algorithms, data structures, and you definitely need to know, coding. You need to know how to code in a language of your choice; language doesn't really matter that much. Better to choose an object-oriented language because that's also important for where you want to work and also what you want to do for the interviews. To count it again: data structure, algorithms, object-oriented programming, and a language of your choice.
It might be easy to get lost. But try just to focus on one aspect of software engineering, then move to the next one. Once you are ready, just prepare a resume and start applying to different companies. Because if you don't have the educational background or job experience, you will have a hard time getting the interviews. It helps create some artifacts; it helps to have a GitHub account where you work on open source projects or contribute to other people's open-source projects. Or maybe create your own thing. These artifacts will help you get the interviews, but to pass the interviews, you need to code. Usually, the interviews are primarily technical, and they will ask you tough programming questions, and you will usually code it on the spot. So that's all about it. So thanks so much for having me.
Sources:
launch: http://re.city
Kartta Labs: Collaborative Time Travel: https://arxiv.org/abs/2010.06536
IEEE Journal Paper: https://ieeexplore.ieee.org/abstract/document/9378093
Recreating Historical Streetscapes Using Deep Learning and Crowdsourcing: https://ai.googleblog.com/2020/10/recreating-historical-streetscapes.html