Why This Stanford Biophysicist Says Humans Hallucinate More Than Robots

a different one from from you know ⁓ zoom and everything let me just double check your setup do you hear an echo

It's pretty good.

You're okay, Okay, we're all good. All right.

Do you want plain background or do you want more lab background?

Great. ⁓ I think has some background looks nice. Looks more, yeah, more, more, more enriched context about who you are. All right, let's get started. Hello everyone. Welcome back to TwoSet AI. Today I'm talking with Jen Lippard. He's the founder and CEO of Open Mind. They're the company building what they call the Android for robots. So Jen is also the professor of bioengineering at Stanford, which makes this conversation very interesting. of biophysicists building robotic operating systems. Gents, thanks for joining.

pleasure. Good morning.

Good morning. Tell us a little bit about yourself.

⁓ sure. Well, let's see. ⁓ I started out as a physics professor at UC Berkeley, and then I got interested in healthcare and computing on sensitive data. And that is what I do on campus. But a few years ago, I got fascinated by large language models like many other people. And I was curious to see if large language models can be used to not only help my students with their homework, but can also be used to control physical hardware. So think of giving a chat GPT arms, legs, and a head and allowing it to explore freely outside in the park and talk to people. And so I was curious to see what the implications were of large language models. for robotics and that's a little bit what led to OpenMind.

When did this happen?

This was about two years ago.

I see. We know that two years ago, 2023, 2004 were, you know, the large language models and co-pilots. And last year was the year of agents. And I, right now I keep hearing 2026 is the year of embodied AI and robotics. So, and you're actually building in the space. Do you feel that's real? It's a fair take.

Well. There's still disagreement on what that exactly means. Some people are building what they call physical AI and other people care about embodied AI and other people are building world models or foundation models or some combination of those things. So it's not completely clear to me what those terms mean, but from a purely technical perspective, it's certainly true. that there's now a clear path to making robots much more interactive and quote unquote much smarter than they've ever been. If you think about the old days when people use ROS2 for example to program Roomba vacuum cleaners, we were used to robots navigating our living room and trying to vacuum. But those robots were very simple. They executed preprogrammed paths and performed one or two functions. But now with large language models and other computer tools, you can add incredible functionality to robots. They can now speak and listen, make you laugh, make you cry, crack jokes, win the Math Olympics, and also navigate your home. They can take people to different places in your house. They know what a living room is, and they can recognize it based on the things inside those spaces. So it certainly seems like the software for general purpose robots is advancing very quickly. for me, as an educator and a parent, it's fascinating to watch that happen.

Is there anything special about this year that excites you most? We just started, I know.

I know, well, come on, we're not even in February. oh, well, on the hardware side, for the first time it's possible to buy a humanoid that is relatively robust and also more and more cost effective. The cost of the humanoid is probably going to hit about $20,000 this year.

Yeah.

which is much more reasonable. It's still super expensive, but at least it's somewhat more reasonable. And the software is also advancing quickly. So this is going to be the year where a lot of people in the US, for the first time, will get to meet a humanoid robot and talk to it and have this robot help them, not only in like a factory, but more and more in workplaces and hospitals and schools and your house. And I think that will be really interesting for a lot of people here in the US. People outside of the US, for example, in China and other places have a little bit more experience with being surrounded by robots, but at least here in the US, it's still kind of a new thing.

That's interesting. So are we lagging behind? Can I ask that?

well, let's see. That's a lot to unpack right there. When it comes to the supply chain for humanoid robots, that is mostly in Asia, either in China or in Korea. And so just from a hardware supply chain perspective, It's certainly true that the US is lagging. From a software perspective, we're probably a tiny little bit ahead. But that is also changing very quickly. And we're seeing more and more state of the art large language models ⁓ come out of China. So on the hardware and software side, ⁓ the US is definitely facing much more competition, at least in robotics, than it's ever had.

Does that worry you in any ways? Or?

well. ⁓ it depends because I wear a lot of different hats. So as a parent, I have a particular set of concerns as a teacher or professor. I have a particular set of concerns as someone who is fascinated by technology. I have a perspective and. I spend very little of my time worrying about which country is ahead. I just don't spend a lot of time on that. And as a professor, my main goal has always been to help ⁓ smart students who want to learn to do amazing things. And I've never spent any time thinking about, you know, where were they born or where do they come from? So that mindset is a little bit foreign to me. ⁓ From a US manufacturing perspective, and this is true of every country, it is certainly beneficial for countries to have a diversified, robust supply chain that they can draw on for essential things. ⁓ the cars, the washing machines, the medications, the computers. And it's probably true for every country to have many choices when it comes to who can supply vital things from ⁓ metals to magnets to books to robots. So that's not specific to the US. That's just a general recommendation to all countries.

Yeah, yeah, we've experienced this shortage of just simply masks.

Correct. The crazy story there, this is a personal story, remember the early days of COVID. And all of us were running around here in the US trying to get masks and gloves and even like disinfectant was hard to get. And despite also being part of the Stanford School of Medicine, I couldn't get any of those supplies for my home and my kids. Remarkably, I got a shipment of masks and protective gear from China. And yeah, it was remarkable to me that just based on research collaborations I had with universities in China, they just said, hey, Jan, we'll send you some boxes. And they showed up my door. And that just goes to show how important it is or what the benefits are of having local supply chain and manufacturing. That was a great lesson in what happens when global shipping comes to a grinding halt. And that shows the benefits of being able to make stuff that's important, ideally close to where you live.

That's a hard demand though, right? Do you see robotics one day will become like, you know, household livelihood related, just like masks or hint sanitizers?

Yo. Well, so there's two fascinating questions there. One fascinating question you alluded to was, what are implications of robotics and automation for making stuff near us, near where we live? In the old days, a few years ago, you could probably make a strong argument for ⁓ vertical integration and concentrating manufacturing to a small number of places. And that allows you to be hyper efficient. With increased automation, you can imagine that changing a little bit, where it starts to be more and more possible to make lots of different things very close to us. For example, here in the company, we have just a room filled with 3D printers. And that makes it super convenient for us to print like new robot heads. Different people want different designs, and we can generate a new one in just a day. So there's ⁓ more and more people thinking about implications for manufacturing of increased automation. And for example, there's this notion of having smaller factories closer to big cities so that stuff cities need can be manufactured in ⁓ proximity. With respect to the other part of your question, will humanoids become Like, will they become a household essential? You know, like milk, eggs, your coffee machine, and your humanoid. I think that really depends on your family. ⁓ For example, if you're 86 and want to live at home alone and ⁓ require extra help, ⁓ then a health companion robot ⁓ may be vital for you.

Yeah. Yup. Yep.

it may be the thing that allows you to age in your house as opposed to a assisted living facility. So for that kind of person, humanoid may be essential. And for many other people, it may be a luxury. And for other people, it may not be useful at all. So I think it really depends on your family and on what your health is like, ⁓ how safe your home is, if your kids want help with their math homework, or whatever else the case may be.

Hmm. Which industries are actually adopting these, you robotics humanoids right now? Where is this happening right now? Not the future.

Well, the human is right now the biggest market, at least in the US is still education and research and many companies that build software are currently purchasing humanoids because it's an obvious like new form factor that requires a compute and has a really interesting capabilities. The main market for robots in the US are manufacturing. So think about car plants and the gigafactory for batteries. ⁓ Amazon has deployed 1.1 million robots for warehouse operations. And a lot of people simply are not aware of this. They've never been inside a gigafactory. They've never been inside a modern Amazon warehouse. So they don't actually know what that looks like. There's also a trend to ⁓ minimize the interactions of quote unquote normal people with robots. For example, in some hospitals, the robots that move the laundry to the big hospital washing machine in the basement, those frequently operate in the middle of the night. And that prevents like hospital visitors from even ever seeing the robots. So ⁓ that's a situation where robots are deployed, but they're used at times to minimize people running into them and colliding them and seeing that and things like that. When it comes to humanoids in a typical workplace or household, that is still...

Mm, I feel like that's I wouldn't. Yeah, that's why I wouldn't do it Yeah.

extremely rare. There are some crazy people that have humanoids in their home, like in my house, but I wouldn't consider that to be mainstream or generic early in 2026. Humanoids will become more more available for normal consumers and developers and educators and so forth in the middle of this year. with broader availability at the end of this year. But even those are still going to have known deficiencies. They'll still be loud. ⁓ They'll still make thunking noises when they walk through your home. The cooling fans will run too much. There'll still be lots of rough edges that need to be sorted. For example, the current generation of human noise typically doesn't charge itself. and you're expected to like turn it off and take the battery out and charge it. But the next generation of humanoids ⁓ coming this summer and this fall generally will come with self charging. So as soon as the battery is low, your humanoid will walk to the charging chair and sit down, charge, and after something like 30 minutes ⁓ will be charged. And that just will give you a much, better user experience because it's going to be much more like a companion or colleague as opposed to something you're constantly plugging batteries in and out of.

Right. I mean, my grandma needs help and the help cannot ask grandma to, you know, put putting batteries and, you know, charge it in the meantime. It's like, ⁓ now I need to take a break. You charge me.

Absolutely. Exactly, exactly. No, no, no. That's got to be totally seamless and it's got to be super reliable. It's got to work for years. And that leads us to a really interesting question, which is who's actually going to make humanoids?

Cool.

Well, I think in the past, the notion was the idea was there'd be like a Silicon Valley startup that would ⁓ start making humanoids. But I think the reality is that there's already really big companies that have decades of experience making millions of things that are super reliable. And most obviously it's the car industry, right?

Yeah.

When you go outside in the morning to drive to school, you expect your car to start. You expect your car to be... Convenient, comfortable, quiet, reliable, repairable, and so forth. And for many robotics companies, they don't necessarily have large-scale manufacturing in their DNA. So my suspicion is that car companies will emerge as the real suppliers of hundreds of thousands or millions of robots. And some car companies are very overt about this. The most, the best example is probably Optimus and Tesla. There's a very, very deep synergy of a large car company with an AI company, with a robotics effort. There's a lot of synergy there. Chinese car companies like BYD and Xiaomi, course, and also Korean companies like Hyundai are very interested in humanoids and they're almost certainly very well positioned to ship them on scales of millions of units.

Yeah, that seems to be a natural extension for what, you know, they've already been doing. How big are these humanoids? Can I expect somebody who's as tall as me?

well, I haven't seen you in person, so I can't answer that specifically. ⁓ well, there's really a bifurcation right now. In the old days, the idea was that a humanoid was tall and strong. So we think about the optimists. right, you know, and people they've seen.

I'm like 5'3", so I'm short. I'm 5'3", so I'm short.

Robocop movies and space alien movies and things like that. And so for the most part, like the, in the really old days of humanoids, like a year or two ago, the form factor was more like an optimist form factor. So call it like a meter 60 or a meter 65. The slightly newer generation of humanoids was then a little bit shorter, typified by like the Unitree G1, which is like 1 meter 20, 1 meter 25. But there's a whole new generation of kid-sized humanoids, typified by something like the Booster. And the Booster is about a meter in height. It weighs 24 kilograms. And the benefit of that humanoid is that most Americans can pick it up with one hand, just like you can pick up your little kid, and then you can plop it in the backseat. So for ⁓ many, many use cases where the humanoid doesn't have to be physically strong, my suspicion is that the kid's size humanoid form factor will be the definitive one. Kids love it. It's not scary. Parents love it because they can pick it up and plop it in the car. And no one starts crying or runs away from it because it really has this more kid form factor.

Yeah, it's like a household electronics, you know, if I can pick it up and charge it.

Exactly, you you have to be able to pick up your vacuum cleaner and just in the same way, if you've ever tried to manhandle a 70 kilogram large humanoid into the front seat of your car, you'll know that it's super awkward and super difficult. There's arms going all which ways and legs and the head is flopping back and the thing's super heavy and it's just a major pain. So that smaller kid form factor, about a meter, I think is gonna be the breakthrough form factor, at least for like education, reception, health companion, ⁓ security type use cases in the US.

But why humanoid? You know, we've had robotic arms in, like she said, manufacturing surgery for many years, right? Why not just like better arms?

well, there's something absolutely fascinating about a humanoid. When a little wheeled robot goes by and delivers your grocery or something like that, people are already starting to tune that out. It doesn't really surprise them. I think there's some deep fascination of people, of humans, with machines. that kind of look like them. So when a humanoid walks through the park in San Francisco, the little kids come running and the dogs come running and the teachers come running and random people come running and the cars stop and things like that. And the large part of that, I think has to do with the form factor. The other aspect of the form factor is that homes and cars and chairs and electrical switches and stairs, all of that is built for humans.

Mm-hmm.

So if your robot has a humanoid form factor, it's guaranteed to be compatible with the vast amount of infrastructure that's been built for people. For example, the booster robot, when we put the booster in the car, we just use a car seat we got from Walmart. And it works really well because the humanoid has a similar size to like a six-year-old. booster seat we buy from Walmart for the booster humanoid works just fine.

When do you think these robots will human humanoid will actually enter our homes?

Well, again, I really think that depends on so many different factors. If you're a intrepid ⁓ technologist and want to experience what that looks like, you can buy them today. There's no barriers. There's software like ours. There's the Nvidia Thor chip.

I'm ready.

which is inside more and more of the robots. And so don't be afraid. If you want to experience that today, go for it. Just email us or lots of other different people and we can help you get started. But that's still the sort of developer, engineer, hacker community. Presumably what you're referring to is like

So don't be afraid.

Like my mother going to the store and picking out her humanoid or something like that. That's going to start to be possible this year in the US, but the functionality of those humanoids will still be very limited. And generally for liability and insurance and safety reasons, those humanoids will almost all be the light. weak and small form factor. There's a real safety benefit to the humanoid not being very strong, then by definition if it's not very strong there's many things it can't do that you might not want it to do.

Hmm, yeah. ⁓ if I, mean, I don't know, as an average person, if I get one in my house this year, what do think I can do with it? Most likely.

Well, so there's good news. And the good news is that some of the software that's being built for humanoids is deliberately open and contains things like an app store that developers can build apps for. So I may have guesses and preferences about what I want humanoids to do in my home, but your answer may be totally different.

Thank you.

And that means that with the right software, you can go to ⁓ GitHub. You can snap together different models and different prompts in such a way to bring about a completely new functionality that you may want for your home. And then if you want to, as a developer, you can make it available on an app store. So that's something we're doing at OpenMind. We just announced a app store for Humanoid. And basically, this allows people to build apps or skill chips that add particular functionality to the robot. So my hope is that this future will be very open and there will be an infinite number of capabilities that people add to their humanoids and then share with others. But in terms of the initial apps that are on the App Store, There's really three categories. There's one category that is education. And a typical scenario there would be ⁓ like math education or languages. And that is based on the robot being able to walk up to your child, ask questions, and also observe, for example, the book the kid is reading and the material in the problem set. that your kid brings home from middle school or wherever, and then being able to ⁓ talk about that, ask questions, and try to teach material. So that's very heavily vision-voice-based interaction. There isn't a big mechanical component there, except for the ability of the robot head to move and track and look at the child and also show interest, emotion, and other things through a display. Other use cases are really safety at home. So for example, your robot can go to different rooms, it can look around, and if it detects anything that is not safe, it can notify you. And the third set of capabilities is health companionship. For example, there's going to be an app soon on the App Store that ⁓ tells the robot to find you if it doesn't see you. And if it finds you and you're, for example, lying on the floor, then it will ask you, hey, do you need help? And if you don't respond, then a human nurse will be able to connect through a tele-ops back end, observe the camera data, look at the chat transcript. then if the human nurse concludes, oh, this person actually needs help, then he or she can call 911. So those are the major things we're building right now. But the thing we've been most surprised by is that every single person we talk to wants their humanoid to do something different. It's really crazy, and I never would have imagined. Here's one use case. Someone wants a humanoid to be the ultimate alarm clock. So they want the humanoid to come to the bed and say, hey.

I bet. You

It's late, wake up and then there's no snooze button because they want the humanoid to run away. And then just from far away, keep saying, Hey, you really have to get up now so that they physically have to get up and then go chase the humanoid and then they're awake. So, um, uh, and I never would have imagined that people want, you know, they, they want a small form factor humanoid in their house to like run around and make sure they wake up at the right time.

They really have a hard time getting up, I guess.

Yeah, they must. ⁓ It's ⁓ I don't know. You know, it's an anonymous developer. I don't know who this person is, but they must be very technically savvy and have a horrible time getting up in the morning.

Yep, yep, yep. What about you? You said you have one at home. What do you use it for?

Well, the whole math education thing is really based on my 13 year old son who was in middle school at the time. You know, he was sitting at the kitchen table doing his math homework and then he just started asking a quadruped dog for help with his math homework. We've all grown up with the joke of the dog ate my homework, but I was shocked to see that picture being reversed where all of a sudden the kid is just having a conversation and the dog is actually doing the math for Ben. I hope the Pula Alto school district is not listening to this and I hope his math teacher is also not listening to this. But so then of course what you have to do as a parent is change the prompts that control the robot to make sure that the robot

you

teachers as opposed to just doing the homework for him.

Yeah, yeah, there's a dog that's actually helping, right? Yeah, no excuse, you have to do your homework.

Yes. Exactly. And if you had also the wake up functionality in the dog, ⁓ you would have the perfect scenario where the dog can now go wake up the kids and make sure they're not late for school. And then of course, if they have questions about the homework, then you have the entire power of chat GPT 5.2 at your beck and call.

Yeah, that's amazing. You know, my husband calls the kids to get up in the morning. He always says, like, wake up, wake up before you put on the makeup. And then my daughter recently, right, she's nine now and she's getting so tired of the sound. And I think she'll be more okay with it if it's a robot.

Thank Yeah. Uh-huh. Yeah, well, know, kids rapidly adjust to almost everything. The secret way I was able to get my teenager to wake up was simply ask Ben to go wake up his older brother. And Ben would just run. He'd open the door and he'd...

Bye.

jump onto ⁓ Sam at which point Sam would absolutely have to get up. So I don't think we, so there's something special about the younger brother waking up the older brother. But ⁓ I wouldn't be surprised if kids like rapidly get used to whatever new piece of technology we put into their lives.

Yeah, that's amazing. Do you feel there's any safety concern just by using these robots at home itself?

That's a great question. And there's like thousands of safety concerns. ⁓ There's privacy concerns, like do you really want a camera running around your house? ⁓ There are more general data security concerns. ⁓ Where are the data going? Where are the voice data going? Where are the pictures going? Who's looking at those pictures? Who's saving those pictures? ⁓ Then there are physical hazards, like it's a big battery on legs. There are pinch and fall hazards. For example, the quadruped dogs, their ⁓ knee joints are, have like a little gap where a little kid can stick their finger into the knee joints. And then if the robot dog gets up, that could have terrible consequences ⁓ for like a little finger that fits into that joint. So There's a incredible amount of stuff that needs to be considered from battery charging, fire safety, pinch hazards for little fingers, just the weight of the quadruped dog. This is not specific to quadruped dogs. If you have a large dog at home, if that large dog jumps and plops down on your baby, that's obviously a real concern as well. But in addition to that, you have all the data security and privacy concerns.

Do you think we're putting in how far away from all these concerns being somehow safeguarded or removed so that it can be put into use?

It's going to be a moving approximation that's kind of like asking, when is the internet going to be perfectly safe? Well, the answer is never. So as soon as you fix one bug, then you just get attacked in a different way. And as soon as you think your data are secure, then it shows up somewhere else. So that's going to be a problem that's going to be with us for the rest of time. And it's not specific to humanoid robots. ⁓ It's also a problem for phones and vacuum cleaners in your car and everything you do when you interact either with physical objects or computers or the internet.

minutes. Good point, we are used to it, we just figure it out and we just tell the kids that don't do this, don't do that.

Exactly. Most people are surprisingly ⁓ undisciplined when it comes to privacy. A great example would be that most of us use Gmail. Right? Super convenient, it's free. But you may have wondered why Gmail is super convenient and free. Or you may have wondered about the business model of Instagram. Perhaps you wondered about the business model of Facebook back in the day. And most of us ⁓ turn out to be quite comfortable trading privacy for convenient, fun goods and services. And that's a choice, of course, that most of us are making. And I use Gmail too because it's awesome and convenient, but I full well know that it may not be the best thing to do. So I wouldn't be surprised if most people really are quite blase about privacy and security. And I wish people were at least a little bit more sensitive and informed about some of those challenges.

Yeah, it's a trade off, right? So we're all dealing with, I feel like as consumers, maybe we have a weaker ground in a sense, because it would be, first of all, we're not as aware of where this is going. We were showing some benefits first.

Yeah, well, ⁓ I like to think that everyone, and especially everyone listening to this podcast, ⁓ go do something about this. Learn more. Read open source code. And just ⁓ devote a little bit of attention to this. ⁓ You have a lot of agency to ⁓ also drive this discussion. One example would be California has a unusual set of privacy focus legislation in place that most states don't have. The Europeans have also more infrastructure in place as it relates to privacy and data. so ⁓ many people listening to this podcast are voters. Go talk to your legislators. ⁓ make sure that you bring these issues to the front.

I remember you mentioned last time when we chat you used Asimov's three laws of robotics. you have a comment on that? Like do not harm human should be implemented in the robotics in the cold.

⁓ Absolutely. I would like to think that all responsible companies have a clearly articulated set of behaviors or guidelines they expect their chat bots or their robots to stick to. And it's interesting to me that they are not many good public examples of best practices. So as a company, we started with Asimov's Three Laws of Robotics, which is right out of a science fiction book. But that seemed like a really good place to start. Like, don't hurt people. It seems glaringly obvious. But as Asimov has pointed out, what appears to be an obvious and simple rule sometimes can struggle to accommodate many different edge cases. Imagine, for example, you're driving a school bus. filled with kids. And imagine you see an accident unfolding ahead of you. What is your main goal in life? Is it as the driver of the school bus to protect all the little kids in the bus? Or is it maybe avoiding a collision in front of you? And so even an apparently simple question like, don't hurt humans. ⁓ Turns out to be ⁓ much more nuanced than ⁓ people might at first imagine.

Is this something that you're building in your system?

Well, we have a lot of infrastructure devoted to ⁓ where those kind of guardrails and rules can be stored, how other people can verify that they exist and what they are, and how they can be changed. We unfortunately do not have the legal or the philosophical acumen ⁓ to ⁓ really be able to optimize what those rule sets could look like. So we've stayed with Asimov's laws now for the last year and a half, but it would be absolutely awesome if there's lawyers or historians or philosophers listening to this who say, here's a better take on what those rule sets could look like for humanoids. It's a major gap. And I haven't seen a lot of attention put to this.

It seems to be like experiments are needed and then when experts in these areas can see, right, and then they can maybe improve the interpretation of the steps potentially.

Right, and almost certainly that's going to happen very naturally as more people encounter the technology, then these questions will become more relevant very quickly. What are the rules that these machines are following? And then hopefully many people will devote more attention to this.

Yeah. Let's talk a little bit about what you're building. mean, you, you don't seem based on what you're telling me. You don't seem to be building anything physical. You're not a hardware company. You're basically the software and API layer for building robots. Am I correct?

Thank you. Yes, we're a software company and our main ⁓ effort is a open source software stack for thinking machines called OM1. So if you go to GitHub forward slash open mind, you will find the OM1 software stack. That software stack focuses on decision making and data fusion. So the problem the software stack solves is If you're a humanoid in a complex dynamic environment, what is the best thing for you to do in the next moment? Once you've made those decisions, then our software stack uses many other types of models to convert a decision to a physical action. Maybe the best thing to do in this moment is to pick up an apple. In that case, ⁓ motion policy or a vision ⁓ language action or a action model may be suitable. But our software lives one layer above most robotics models because we focus on this decision-making problem. ⁓ Yeah, so it's open source. If you want to learn or contribute, ⁓ just read the code and ⁓ start ⁓ fixing stuff and adding stuff. And however, you asked, are we software or hardware? However, ⁓ we spend a lot of our time also on the hardware because ⁓ everything matters to a humanoid. The mass matters, how tall it is, battery lifetime, thermal. ⁓

Yep.

where the robot is with respect to people and pets, where the robot is in a particular space, what kind of sensors it's using, what the firmware is on those sensors, what the power envelope is. You can't just add more and more and more computers to humanoid. At some point, it can't even get up from the chair. So when you're writing software for humanoids, you end up spending an incredible amount of time dealing with the physical world, whether it's batteries or temperatures or ⁓ spatial constraints or movement. So that's why when you go visit Open Mind in San Francisco, you'll see lots of computers, but you'll see piles and piles of heads and cameras and soldering irons and battery chargers and calibration targets for spatial navigation and things like that.

Yeah, so that's outside of this chart. This is just OM1. So this is the system architecture for the operating system.

Correct. This is a highly simplified architectural diagram for a multi-model stack for social-focused humanoid and quadruped robots. Yes.

It's ⁓ a lot of ⁓ quantification of what it is. I was looking at this because I feel this is very easy to understand. As you said, it's simple. So I find this interesting about like the natural language data bus. And you have it here too, right? At the bottom between your hardware layer and with all the software side. So I was curious, does this mean that everything becomes text? So you have the vision model and you do see the pictures and you have the sound, you have the text. Is everything converted to text?

Yes. And that has a few massive advantages and a few massive disadvantages. A lot of people, especially if you're a computer scientist, think this might be ridiculous. Because if you've ever worked with video data, one thing you might want to do is you might want to push raw video data straight into some sophisticated model. And maybe you want to push raw audio data into this sophisticated model.

Tell me.

⁓ However, when you try to do that, you quickly run into either hardware or software constraints, or you end up with a system that's almost impossible to debug or almost impossible to understand what's going on or why certain things happened. And one of the benefits of having models literally talk to one another using natural language is that you can eavesdrop on what all those models are saying very efficiently and very clearly and very well. And if you find problems, you can very easily also apply natural language guardrails to different parts of the system. So you augment or prevent certain behaviors. So in our mind, all of engineering is a trade-off. And the trade-off we're making in our software stack is we only deal with very high level decision making. And we're happy with the robot making big decisions every second or two, which for some robot use tasks, use cases, is way too slow. Imagine, for example, you're building a drone to target a tank. Of course, in the last few hundred milliseconds, that targeting loop needs to be much, much, much faster. Or imagine you're building a ballerina humanoid to balance on his or her big toe. In that case, your loops, your stability and movement control loops need to be operating not at 0.5 Hertz, but maybe 500 Hertz. And the decision we've made as a company is We don't do drone targeting, don't do ballerina humanoids, we don't do onion chopping. We focus exclusively on use cases where a one or two second decision cycle is more than good enough to keep people safe, teach people things, keep your home safe. take things from a car to your front door or whatever the case may be. So that's the big limitation of this kind of software architecture. And because we're okay with focusing on some use cases, this kind of use of natural language to interconnect 10 to 15 different specialized models works out really well and then has the advantages of you can just see what all those different models are doing. You can fix things. You can update and improve one small part of the system as opposed to having to redo everything. And so there's some disadvantages, but many advantages, at least in our mind.

I see. Yeah. So you sacrifice a little bit on speed potentially, but you have the flexibility of making it work and have have the performance that's close to what a human wanted instead of, you know,

Yes. Yeah, we certainly have a preference for ⁓ having the humanoid do useful things as opposed to some idealized goal of ⁓ perfection in five or 10 years from today. Yeah, so we're kind of impatient and we kind of care about high level cognitive tasks much more than we do about wet wiping your floor or chopping onions.

Yeah, we have to deal with hallucination, right? We are using a lot of these LLMs behind the scenes anyway. It's all text. You must be facing this problem as well. How do you handle that?

Well, I have a horrible secret. I've been grading undergraduate homework for 15 years. And let me tell you that if you've ever submitted a problem set at three in the morning and you wake up and you don't remember what you wrote, let me tell you that the teachers and faculty grading your problem set

Mm-hmm.

have lost all illusion about hallucinations. And hallucinations are not a computer thing. It's something all of us do almost daily. And certainly, undergraduates that submit problem sets in three in the morning ⁓ have said the most bizarre things. I like to think that hallucination is a much bigger problem than it is about humanoids. Humans have developed a court system and juries and judges and so forth to try to figure out what people should and shouldn't do or if they've done something wrong and so forth. so in my experience, humans are highly imperfect and many of the challenges that you have to deal with when you build ⁓ smart humanoids just directly reflect that the models have been trained on human behavior. So they've seen all the crazy bad stuff people do and they know about that. And so one way to make those models a lot better is if all of us behaved a heck of a whole lot better. So we wouldn't need juries in jails. And then if we trained our robots on the same corpus, they would also be extremely well behaved. this whole question about like robot behavior and hallucination and so forth, ⁓ my recommendation would be first to look at the decisions all of us make daily before we get all worried or before we somehow say that robots are particular in their ability to make bad decisions.

Yeah. Yeah, it's more, it's more the actions that they're going to take. Instead of like saying something bad, maybe, you know, writing something on paper at 3 a.m. is more tolerable than if you just, you know, just wake me up at 3 a.m., jump in my bed as my robotic companion, right? Don't do that.

Right. Well, any parent listening to this, I'm sure you've also had to navigate situations where your five-year-old has done things that maybe you also thought necessarily wasn't that awesome. But sure, and there's no simple answer here. One thing we found to be very useful is to have decisions made. by a committee. So we totally have one LLM running in the middle. That's the core LLM. But then we have another LLM, which we call the mother LLM, or the referee or mentor. And what that LLM is doing, it's observing the robot interacting with people in front of it, and then every 30 seconds writing a critique. Stand up straight. Look at people ahead of you. ⁓ preface everything you say with ⁓ or ⁓ and things like that. And part of the role of the mentor LLM or the mother LLM is to ⁓ bias the system towards good behavior. So that's one thing we see work, but even that has limitations. And the other thing that we're very proud of is the extent to which we are integrating humans in our software. We spent a lot of time on a good tele ops and observability portal. And that allows people like human nurses, teachers and retired police officers to observe data and observe behavior of the robot and counteracting that.

Mm-hmm.

or helping deal with edge cases. This is probably true of most robotics companies, Waymo and everyone else. It is highly beneficial to have a awesome smart human in the loop to deal with edge cases and make sure that the system is doing quote unquote the right thing. And for a lot of these home deployments or schools or hospitals, We're looking at easily a period of the next five years where many of these systems will have an awesome, smart human ⁓ very close to the robots, figuratively speaking.

That's interesting. beyond OM1, you also have something called fabric, which my understanding is, you you mentioned that these robots can share skills. I find that fascinating. So one robot learns, let's say Chinese, and then the other ones all learn it instantly. And that reminds me of Carl Jung's ⁓ collective unconsciousness. How do you feel about this? This feels like this is going to evolve really fast, right? Doesn't that scare you as a parent of two kids?

Sure. Well, all of us have seen the horror movies where the Borg collective learns a new skill. And that kind of capability has been used in the military for decades. For example, if one, in the context of electronic warfare, where if one ⁓ radio receiver hears a new type of radio frequency emission,

Right. Yeah.

then you want to have all other radio receivers be aware of that new pattern or new transmitter. And that allows you to respond to a new threat ⁓ very efficiently. And robots, of course, already have this property of being very easy to upgrade. For example, every single time your Tesla upgrades, all Teslas receiving that upgrade will have new capabilities. And it's already true that motion policies learned by some robots can be immediately transmitted to all other similar robots, giving them new capabilities. For example, the booster robots in our lab. A booster engineer came by last week and applied a firmware upgrade to improve the dancing. And that's something, of course, if you're a teacher listening to this, you know that all of us, we start from zero. And then we learn numbers, and then we learn to count, and we learn to spell. I'm so bad at spelling, but most other people learn how to spell. And that, of course, that's why we all go to school and university, and that takes decades. But what's really interesting, you're absolutely right about robots is that if one robot learns a new skill and gets really good at something, it can immediately laterally transmit that skill to all others. And that brings with it incredible opportunity, but also incredible peril. And I spend a lot of my time watching science fiction movies and reading science fiction books simply because a lot of the

Yeah.

The that we're anticipating on the engineering side have already been imagined in sci-fi novels and movies. And so we typically ⁓ learn a lot, whether it's from The Matrix or Black Mirror or Space Odyssey or many other relevant movies.

Yeah, so it's happening. You know, last time I clipped our conversation a little bit, you call this the time of maximum uncertainty and also the time of least preparation. was so good. I have to clip it and put it on LinkedIn and Twitter. You now have to answer that. What do you mean by that?

Well, one thing that is important is not what the technology does, but how quickly it appears in our lives. With other pieces of technology like airplanes, it took something like 40 or 50 years from the Wright brothers with their first flight to the point where most Americans had been in an airplane. That was 40 years. And then it took another 20 years before it became routine. So that's 60 years worth of experience with what does the technology mean? What does it do? How does it affect my life? And 60 years is a reasonable time scale. That's like almost two generations. That's a reasonable time scale for ⁓ human society to adapt. But what we're dealing with here is a technology that is advancing its capabilities very, very rapidly. And that means just the speed itself makes it hard for all of us to know what's going on, stay current, and to adapt to it. So when I look around and I see universities and schools and regulators and politicians and Just all the people around us, for all of us, the speed at which things are advancing, I think is the real challenge. In terms of that quote, the period of maximum uncertainty and least preparation, that's exactly true. In 10 years from now, we'll have a first assessment for better for worse on what the technology actually does. to lawyers and to doctors and journalists and movie directors and teachers. the only exception to this is probably self-driving because already in San Francisco today, it's kind of routine data Waymo. And that used to be science fiction two years ago. But now more and more people have had that experience and it took Waymo 10 years to get the system to the point where it's as awesome as it is today. I mean, it's eight times safer than a human driving a car. So I, as a parent, I'm much more comfortable having my kids be in a Waymo because I know they're safer compared to like a human driving them. Exactly. And I'm a horrible driver, by the way. Like I'm always thinking about random things or looking at

Albert, yeah.

the horizon or having a coffee or something like that. my kids are certainly better off in a Waymo compared to me driving them somewhere. yeah, so it's this moment where we understand relatively little about the technology and what its capabilities are on limitations. And we're also least prepared.

Yeah, 10 years is a long development time and two years, you know, people will still have time to adapt to Waymo, but it's a lot faster right now. What do you think we should be, you know, ⁓ preparing for and what are the skills that's going to actually matter?

Yeah. Unfortunately, I don't know. And my fortune telling or future telling abilities are limited. But what I've been telling my kids is this old notion that you go to college, you learn a skill, and that's what you do for the rest of your life, as far as I'm concerned, is completely dead. And one of the more obvious implications is that every one of us for the rest of our life ⁓ should spend a moment every day learning something new. So ⁓ this notion that Stanford also has been articulating of lifelong learning, that is more and more true. As the world around us changes quickly, we have to change quickly too. We simply can't say, you know, I got a physics degree 20 years ago or whenever it was. That's good enough for me. No. All of us ⁓ will need to be able to reinvent ourselves in terms of our skills and what we spend our time on more and more frequently. So this property of being nimble ⁓ seems incredibly important. So try to pay attention what's going on around you.

Mm-hmm.

⁓ learn new stuff, try stuff out. The only reason open mind was started was because, ⁓ I opened a box with a humanoid in my living room and I was super underwhelmed with what that humanoid was able to do. It could like get up and shake hand, but that was it, which is incredibly dumb. Like that's not very interesting. And so that then led to building software to add new capabilities. So like. Try stuff out, pay attention what's going on, and you should anticipate learning new things every day for the rest of your life.

universities dead.

Many of them are, but some of them will become even more important. So what you're going to see is a bifurcation where a university that offers a generic product and a generic brand is almost certainly dead. And then there will be some universities which are already famous and they will become even more famous and even more important. So you will see this bifurcation where offering a generic product and generic service no longer cuts it.

I asked you last time if you could recommend any books that you like to maybe impart more wisdom onto me and my audience. But I remember you told me that books feel too slow for you now. What would you recommend to help shape the minds adapting to this new era?

Hahaha! Well, I love books, but these days when I'm asked to write a book, say, ⁓ like, don't be silly. I'm still building. I'm not in my golden years. not sitting there contemplating the sunset yet. I still have many things to build. if you're a book editor, please don't ask me to write a book for you. Please ask me in like 30 years. But well.

No. ⁓

I think this depends on every single person. I learn best by doing things with my hands or practically trying stuff out. So the way I learned physics is not necessarily through a book, but I like build stuff and see what happens and then it doesn't work. And then I have to figure out why it doesn't work. And then I'll write some equations and then I'll ballpark it. so it's just like iterative process of trying to build things and figure out why they don't work and then ⁓ I end up learning stuff. So I'd recommend people to experiment. But that's a very personal thing.

Go out and play. Yeah, yeah, yeah, me too. Why is OpenMind.org is a dot, dot org.

⁓ It's a dot org for two reasons. The first reason is when we started out, we didn't have a lot of money and ⁓ the dot com was 2 million and the dot org was infinitesimally a smaller amount. So that led to the dot org. And ⁓ the more important reason for the dot org is we're open source software. And I do not want to live in a future where a humanoid shows up at your front door, knocks on the door and says, Hey, I'm your new humanoid, I come pre-configured and my software is secret and magic. And that to my mind would be horrifically sad. It would be very depressing because this technology would appear like complete magic that parachutes from the sky. And I want this software to be open. I want this technology to be open. I want children to be able to look inside and regulators and

Thank

teachers and developers everywhere to be able to peer into the brain of these robots and say, you know, I kind of get what's going on. And here's how I can fix it. And here's how I can add capability to these systems. So the reason we're a .org is because ⁓ we want this technology to be open rather than ⁓ magic coming from some secret lab somewhere.

Awesome. love that closing. it's open mind is truly open, not like the other open something. No, I said it. said it. By the way, I'm sorry. I think I said your name wrong. You go by Yen. It's not as young. Okay, young. Like Y-A-N-N.

Yes, you said it, not me. No, it's a yawn, very similar to Mandarin for sheep.

⁓ Young. Okay. Okay. It's like just like my last name. My last name is Young. So maybe I should change to your spelling. Yeah. Okay. Yeah. Let me me let me let me correct my opening once. Once I ⁓ today I'm sitting down with Young Lippard. All right. So that's that part. So anyway, so Young, thank you so much for the conversation. I really I personally really enjoyed it. I thought I you know, I come into this conversation. I thought I'm pretty far away from robots and robotics.

All good. Thank

because I'm a LLM engineer. But I learned that actually maybe I'm much closer than I expected. ⁓

You have been working on robotic software the whole time. You just didn't know it.

I didn't realize that it's a lot of text and large-language models. could be something that, you know, maybe I guess a developer familiar with Python and LLMs, vision models can take a stab at. Right?

That's exactly right. ⁓ so, and you have ⁓ unique experiences and skills based on previous work you've done, your family, your family history, your languages, where you live, and an aggregate that puts you in an awesome position to start building software to make the robots around you maximal useful for you and your family.

I, or maybe for other people. And this is true for my audience too. And thanks for your guidance, right? Cause you, you make it real for all of us that this is possible. So I really appreciate you opening up that horizon for all of us. And also thank you for sharing your honest take about where robotics is going and you know, the uncertainty concerns about humanity that's happening right now.

Yes, well, thank you for the great questions and anything I can do, just email me. It's jan at j-a-n at openmind.org. And anything I can do, let me know.

By the way, you hiring? Are you hiring? I'll post it if you're hiring.

⁓ Yes, absolutely. ⁓ If you love robots and if you love making them ⁓ safe and useful then ⁓ come drop by. almost certainly have an interesting thing for you to do.

Awesome. Yeah.