77: Automate All the Things!? A UX Reality Check on AI (ft. Dr Stefanie Hutka)
Show Notes Intentional Design Leadership Circle Are you a director-plus design leader looking to connect with others and grow your capabilities? Join Jesse and Peter for the fall cohort of the Intentional Design Leadership Circle, taking place on Wednesdays from October 7th through November 10th. Only 20 seats available for this live online program. Learn more and register. About this Episode Dr. Stefanie Hutka, design researcher, Berkeley educator, and author of What Your Machines Should Do, joins us to look to the history of industrial automation for lessons on AI transformation. From Toyota’s jidoka and the andon cord to Taylorism’s lingering grip, we explore why AI amplifies whatever conditions it finds, and why strategy and values must come before the tools. Dr. Stefanie Hutka: https://sendfull.com/, LinkedIn profile Jesse James Garrett: https://jessejamesgarrett.com/ Peter Merholz: https://petermerholz.com/ Transcript Hey everybody, before we jump into the show, I wanted to share with you something that Peter and I have been working on. It’s kind of special. We have poured all of the thinking and all of the wisdom that’s come out of the last several years of this show into a six-week leadership skills development and support program that we call the Intentional Design Leadership Circle. It’s happening October 7th through November 10th, and there are only 20 seats for this live online program facilitated by Peter and myself. Find out more information and get your seat at findingourway.design. Now, on with the show. Jesse: I’m Jesse James Garrett, Peter: and I’m Peter Merholz. Jesse: And we’re finding our way, Peter: navigating the opportunities Jesse: and challenges Peter: of design and design leadership. Jesse: On today’s show, we’re joined by Dr. Stefanie Hutka, design professor, researcher, and author of the book What Your Machines Should Do: The Science and Strategy of Human-Centered Automation. She’ll share lessons from the history of industrial automation to observe models to follow and models to avoid as organizations move towards AI transformation. From the humble loom to the factory floor to the values that underpin our processes, let’s go exploring with Stephanie Hutka. Peter: Stefanie, thank you so much for joining us. Stefanie: Glad to be here, Peter and Jesse. Peter: To kick off, I think it would be good just to get a sense of how you introduce yourself. I know you teach, and you write, and you speak, and you work, and so, who are you? Who is Stefanie Hutka in the world? Stefanie: Yeah. So I’ve been introducing myself by way of hats, that I’ve been wearing a couple of different hats. Hat number one, and arguably the largest hat, the top hat, if you will, is as a researcher. So I have even more sub-hats under that field. I go back, to early days, my PhD’s in cognitive neuroscience. I was very much an academic researcher. I worked as a research scientist, found my way into design research, and that is primarily the type of work I do today at my consultancy called Sendfull, that I’ve been running for about three or so years now. So hat number one, researcher, specifically design researcher today. Then hat number two is as educator. So I’ve been teaching at UC Berkeley, primarily at the School of Information, but also at Jacobs, the design school. I teach Introduction to User Experience Design. I just started my fifth year of teaching that to professional master’s students. I also have designed some new courses, including UX for AI and Designing Future Systems, which dove into some systems thinking mindsets and toolkits, as well as some strategic foresight. And then the third hat is as author. So I’ve written various things over the years, scientific articles, industry articles. But most recently, I have a new book that is coming out, my first book: What Your Machine Should Do: The Science and Strategy of Human-Centered Automation with Rosenfeld Media. That will be launching in November and currently on pre-order. What’s worth automating? Peter: Wow. Jesse and I have been exploring the subject of AI and design, primarily design leadership, for the last almost two years now. And when we found out about what you were writing about we’re intrigued. And so, unpack a little bit what the book is about for an audience of design and user experience leaders who maybe are still kind of grappling with concepts of automation. Stefanie: For sure. So the book is a field guide for product and design decision makers to navigate this question of how do we automate and closing what I’ve called the automation strategy gap, where you have these dreams of fully automated futures and running after let’s automate everything, versus the reality of what’s actually valuable to our users. How does this fit within their ecosystem? What is our organizational readiness? What is the actual capability of the tools at this time compared to that fully automated future? And so the book shares different frameworks and tools, some of which are derived from my own practice, some of which are derived from a synthesis of literature that I like to say is: it’s intended to be durable; it’s evergreen stuff. This is not a book about how to use Claude Design or whatever tool’s gonna shift by the time this podcast finishes recording. I’m really looking at what can we learn across human factors, human computer interaction, management theory, cognitive science, and acknowledge that we are in a very new territory. There’s many ways in which generative AI is unprecedented. Those three ways that I usually refer to that is: the unprecedented pace of development; the breadth of use cases that it can cover and what that means for human cognition and how much we can offload; and then the stochasticity of the output. So it is definitely, you know… it has unprecedented elements. That said, we have been automating stuff, we have been building autonomous systems, for quite some time, so what can we learn from those principles so it doesn’t feel like we’re completely, you know, flailing as if there’s no prior, priors plural. So the book is about those things. How much of a burger is this? Jesse: One of the interesting debates about this current wave of AI technology recently is the question of, well, how radical a change is this? And there is definitely this school of thought that holds that AI is, quote-unquote, “normal technology.” It’s just another wave of technological change. It’s not very different from the things that have gone before. And then there’s this opposing camp, which is very much on the side that this is so fundamentally different that there’s not a lot that we can learn from the past, and I wonder where you fall on that continuum. Stefanie: Yes. A little bit of column A or column B, or for my fellow design researchers listening, it depends. I think, you know, how are we, maybe how are we, defining AI, assuming we’re talking about generative AI, ’cause there are many types of AI. But yes, like the AI moment that we are in, I’ll start with maybe column B of what might be similar. So there’s something from one of your past episodes. I took notes on this ’cause I, I was excited about this conversation. Jesse: She’s a researcher. Stefanie: Exactly. I was doing research, secondary research. This is what I do. The Paul Ford episode from a couple months back, you talk about compilers as an example of, now you have the sort of translation layer of intent going into the ones and zeros, and, we’ve had moments like this where we can articulate our intent in different ways that speed up our processes. So from a technological standpoint, there have been big advances. We’ve had disruptive innovation. We’ve had mobile. We have had cloud. And those were definitely major events. We’ve had various tech bubbles before. We have navigated those. So we have, I think, from a disruptive innovation standpoint, we have some priors in terms of designing automated systems, like moving from human-in-the-loop to human-on-the-loop. I mean, there are planes flying over us right now with autopilot systems that we’re presumably pretty comfortable. We’re not too worried about, how will the pilot lock back in with situational awareness when they need to? And we have figured some of these things out, so maybe we can learn from some of that. So that’s what I’m talking about with some of those priors. Now, where I think AI is different right now is what I’ve called 10X cognitive offloading, and what I mean by that is, cognitive offloading is this just phenomena of delegating our thinking tasks to some sort of external device and often technology. The classic example is Google Maps. I give this example. I bike to campus, to South Hall. I’ve been doing that route for five years. I’ve got that locked in. But the moment someone invites me, say, to Cheeseboard, which if you live in the East Bay, both of you know this, it’s not far. It’s like a mile from South Hall, but I don’t do that trip very often, and I am entirely reliant on Google Maps. So I’ve offloaded my navigation capabilities to this digital map. And typically, when we’ve offloaded to technology, it’s been relatively finite in terms of our tasks, and I think the way, at least by default, a lot of these generative AI systems have been designed, like the phenomena of the prompt bar, it really invites kind of wholesale offloading, and we’re delegating the entire observe, orient, decision, act loop increasingly with agents such that if we are offloading, quote-unquote, “the wrong thing,” we can very quickly start to de-skill. If we haven’t learned the thing in the first place, there’s gonna be a gap in our learning. And I think that getting to those three things I mentioned at the top of the podcast around one of those being the breadth of use cases to which you can apply this technology, I think that is where it is different, and combined with the pace and the inability for sort of slower moving systems, governance layer, for example. We can’t build guardrails as fast as the tech is moving. I think that’s where things get unique. Grappling with the pace of change Jesse: It’s interesting to think about that pace challenge for organizations as they are trying to adapt themselves and adopt these technologies and actually get some meaningful value out of them, in that the technology is changing so fast that organizations hardly have the chance to orient toward one approach before the state-of-the-art changes. So if we put this in the context of a larger history of industrial automation, right? Where we’ve had increasingly more and more machines doing things that humans used to do, if we can define automation that broadly. We are now in a place where there is this kind of… this shearing force where the technology’s moving so much faster than the organization that the organization fundamentally breaks because of it. And I wonder about, in your research, from your perspective, what we can learn from the pressures that have been applied to organizations in the past, from past waves of innovation that might teach us something about what’s gonna make an organization resilient now in the face of what’s happening. Stefanie: Sure. So what immediately comes to mind is a contrast that I draw, in a chapter where I get into systems mapping, systems thinking, is contrasting Toyota production system with General Motors’ approach. So Toyota production system, one of the key sort of tenets of this is automation with a human touch, jidoka, comes from the Toyota founder, as a young boy, saw his mother working at a loom and saw that she constantly had to be monitoring this loom and invented this mechanism such that when the thread ran out or got stuck, the loom knew when to stop. And this was ultimately like, fast-forward, Toyota moves from textiles into auto manufacturing. Japan devastated after World War II. American automakers very, very resource rich and you get to Toyota saying, “Okay, how do we move forward given our limited resources?” And you have this approach to automation that’s very complementary to people. You have this idea of kind of the andon cord, you pull it if there’s an issue on the line and everyone comes rushing. And I think that’s interesting because it speaks to kind of the values that the company sort of organized around. Certainly there’s contextual factors, there’s different geographical, region, and cultural values, so on and so forth. And there’s a psychological safety when you’re pulling that cord that you feel like, “Okay, I can pull that cord.” So, that’s an example of maybe automation done arguably well versus a much more sort of Taylorist approach that I think we see with, General Motors as a contrast, but we also see in a lot of our thinking now. So in contrast, you have this mindset from like the early 1900s, if we can atomize work and, we can easily swap someone out to do it, and I think there’s an underlying mental model that there’s limits to our human capabilities and, if only we can sort of automate the human error out of things, we will be more productive. If you’re driving on the I-80, if you saw any of those Artisan billboards in the last couple of years, like “Stop hiring humans” with hiring intentionally misspelt to imply that we are error-prone fleshbags, I think you see a lot of that. Yeah, I just threw in fleshbags to wake up our listeners. You see some of that same sort of Taylorist mindset that’s very efficiency oriented, so you get this sort of trade-off between efficiency and effectiveness that comes down to the values in which the automation approach is grounded. So I think the shearing forces are coming from maybe this sort of Taylorist orientation, and this is challenging because this is like the paradigm arguably, like the underlying mental models that may be examined or unexamined that are influencing the values on which the organization moves. Learning from looms Jesse: You use this term in there, jidoka, which is a concept that I’m familiar with, but I think a lot of our listeners might not be, that is closely related to how this philosophy of automation was developed at Toyota, and I wonder if you can expand on that. Stefanie: The Loom example is the one I go to, like, the idea of, you still have humans in the loop, but you don’t need to have them constantly. There’s an element of there are things that are safe for machines, but there is a point at which the machine capabilities stop and, you need to bring the human back in. It’s not necessarily any sort of value judgment, just like this is what the machine can do, and there’s going to be a point where the machine ends its capabilities and the human needs to take over. Jesse: Yeah, I guess what I take from that is the idea that you build into the system the awareness of its own limitations and the awareness of when the system needs to stop and turn control back over to a human operator or a human intervention of some kind. Stefanie: Yes. I was asked this similar question. I gave a talk yesterday morning. It was a Agentic UX summit, and one of the Q&A was like, “How do you know when the machine should stop?” It’s like, well, I’ve got a historical anecdote for you. Like back to your earlier question about what’s different this time, maybe the capability for adaptation of the system as you’re giving it feedback, like loops are all the rage right now, and we’re building agents, and this idea of there’s still a human sort of building that loop. But if you’re optimizing the loop each time, the machine is doing incrementally more and you still need a human somewhere in there. But that dividing line isn’t static by any means. So I think that adds a new layer of challenge. Peter: As you were talking about the Toyota manufacturing process I was thinking and I was doing a little research during that. There was also the work of W. Edwards Deming, right? It wasn’t just one person figuring it out, right? It was, like, this interesting confluence of thoughts and philosophies that led to things like lean manufacturing and the Toyota production process. And the thing about W. Edwards Deming is funny is he’s an American who taught the Japanese how to manufacture cars because the Americans wouldn’t listen to him. He got no traction in the United States. Goes to Japan, teaches them, and then Japan eats our lunch, in the ’70s and ’80s. Cultural contexts that encourage or enable a certain embrace of these tools and in maybe healthier ways versus less healthy ways, right? ‘Cause you also mentioned Taylorism, which is a bad thing, um, largely. Like, I mean, you can make arguments for aspects of Taylorism, and efficiency’s not a bad thing in and of itself, but scientific management, when you go to those extremes, it becomes problematic. What are you witnessing in terms of organizations’ ability to embrace automation in a healthy, good way versus perhaps in a toxic, damaging way? The experimentation mindset Stefanie: Sure. So I see an experimentation mindset, like what I’ve coined because, gotta coin stuff when you write books. Jesse: I don’t know anything about that. Stefanie: Yeah, it’s like you’re building with intention, experimenting with intention is something that seems to be a hallmark of organizations that are embracing this change. And I think there’s something that doesn’t get enough airtime, which is a psychological safety aspect of leadership saying “Hey, y’all can explore in intentional ways. We’re going to make space for this.” One of the most interesting examples that I saw in researching this is Canva’s approach. They’ve had a AI discovery week two years in a row. So the fact that they did it again this year suggests that perhaps it was valuable last year, and it’s basically like the whole company shuts down for a week and just focuses. The first year they did it in 2025, an educational aspect, and then like a two-day hackathon, and you have everyone from like the chefs to the engineers to the marketing folks to the legal folks involved in building this, understanding of capabilities together and building intuition around those capabilities. And then this year it was a little bit more agentic focused and people building out parts of their workflow. And again, the fact that we’re gonna get everyone together, we are going to just get hands-on, and you can develop the sort of taste for: the machines suck at doing this, but they’re really good at doing that. Okay, how might we work together with this? And so we can debate about how does that translate to product development and customer quality. But there is an element of the fact that that was sanctioned and that everyone is just like trying stuff out together and learning together. I feel like there’s something that echoes the andon cord phenomena where we can pull the andon cord, we all rush to see what’s happening. I think according to the literature of this, the first thing that people do to andon cord puller in the Toyota scenario is thank the cord puller for highlighting the issue. And so, I think that’s perhaps a parallel around, yeah, psychological safety, experimentation culture. Yeah. Peter: You mentioned psychological safety. So I’ve been doing a fair bit of thinking around how do you maintain quality in the face of AI? One of the common concerns that I’m hearing from literally every design leader is, “We’re producing more stuff, but it’s not any better,” right? And it’s either them, their own teams are producing more stuff, or ,more often these tools are enabling product managers and engineers and marketers and others to essentially produce slop that now the design team has to hold at bay. And one of the things that I’ve been noodling on is what I’m calling the creating the conditions for quality work. So quality, you have a set of standards, but standards don’t live on their own. You need to have a set of conditions to uphold and enact those standards. And toppest most of the poppest most of those standards is psychological safety. Psychological safety predates AI, you know, at least in design work. It’s been shown just to be a core value of any effective work team. There was Project Aristotle from Google, pointed that out, that the thing that mattered more than anything else in how do you get a group of people to do good work together is psychological safety. Not talent, not skills. Those were way down the list compared to do these people have that freedom to experiment, to try stuff, to fail, to push back on one another, as long as it’s in the interest of making the work better. So that’s one of the thoughts. AI amplifies and accelerates Peter: On my whiteboard behind me, I have the phrase, “AI amplifies.” Stefanie: Oh, yes. Peter: I’m curious what your research has pointed out. My hypothesis, or my postulate actually, would be AI doesn’t meaningfully change anything. AI takes an existing condition and amplifies it. So if those existing conditions are healthy and positive and good, now those people are just generating awesomeness. And if those conditions are toxic and negative and problematic it only gets more so quicker. Are you seeing that in your work? Stefanie: 100%. For better or for worse, yes, I am still on, Substack. And I had one viral note, and then wrote a piece following up on it, which was basically, AI is an accelerant. What are you feeding it? And I think it’s– we can say accelerant or amplifier but this is 100% it. The joke I make is, you know, AI, you bolt it on, it’ll either accelerate your dysfunction or also strategy. And I think when people are like, “Oh, we can build and find out,” yes, build and find out or ship and find out can work if you have a clear strategy, you have hypotheses, you have metrics that you’re going to measure, like was this hypothesis supported or not? And presumably, you will only ship a few times and you will learn something, and you will adjust. But what I am seeing more often is, it is ship and find out, and it’s the same dysfunction as before, but now you’re burning out your customers with basically an endless A/B test. So at some point, they’re gonna wise up to the fact that you’re just, like, you’re building a product strategy by did this work?, ’cause we can vibe code it and ship it now. So I think all that said, l ike, all the goodness of AI: oh, you can work fast with small teams, and you have a strong strategy it is going to amplify that. You can build faster and get that out faster and iterate faster. If you have dysfunction, it will also accelerate that. And I think same thing with inequity, too. Thinking about people who have access to these tools or education systems or if you have curriculum where students are being taught how to use AI effectively versus there’s already existing dysfunction, like student to teacher ratios, and students are just like using AI to write their essays ’cause they’re, like, time scarcity or don’t get enough attention, it will accelerate that inequity as well. So yes, it’s a bolt-on accelerant amplifier. Choose your word here. Jesse: Mm-hmm, It’s interesting the way in which it does amplify both the positives and the negatives within an organization, right? And then in some ways for leaders, what that does is that makes it ever more glaring, the rocks that have been just underneath the water that maybe they didn’t even see, and it asks leaders to maybe let go of some dogma, maybe let go of some ways of doing things that have served them really well up to this point. And I find myself wondering about adaptation and what it takes for leaders to adapt their own thinking to create space for some of the things that you’re talking about, the experimentation and the safety, the ability for the team to take chances, and how the leader might need to change what they do in order to make that possible. From AI-first strategy to Strategy-first AI Stefanie: Yes. I’m connecting this to this concept that I talk about in the book, which is going from AI-first strategy to strategy-first AI. And we can argue that strategy, as defined by Martin and Lafley, the series of choices that you make in effort to pursue some sort of winning aspiration. Like, where do you wanna play and how are you gonna win, and how are you using AI to support that? So I think there’s a connection point between, okay, you gotta have a strategy and know where you wanna go, but also stuff is changing so quickly that is Horizon three even still a thing, or has it just been smushed down? So I think there’s like this inherent tension of how fast things are moving and how that leads to what is our strategy, or how can we see around the corner, and there’s all these other, like, macro factors impacting where we wanna go. Jesse: It feels like there’s something there in terms of how leaders balance their time and attention. So if you’re in a leadership role in a product development organization, it used to be that your job was basically to keep the process buttoned up, and to make sure that everything went according to some predictable cascade that was something that was repeatable and you could be held accountable for as a leader. And now it feels like it’s much more… Peter and I have used the surfing metaphor in the past, where it’s much more about your ability to flexibly respond to the changes that are going on rather than your ability to set an expectation, and then drive a team through a process in a way that fulfills that expectation. So when I think about, yeah, when I think about the skill set, I guess that’s where I wonder, like, how do leaders need to think differently about the work that they do? AI as white water rafting Stefanie: Yes. I love that the surfing metaphor, my editor was getting sick of all of my white water rafting metaphors. I happened to have written chapter six, which is about strategy and AI, in Chile in Futaleufu, which is known for some of the best white water in the world. So yes, this flexible adaptation and I think maybe this metaphor of steering that, like, Terry Winograd coined that many decades ago. But how do you steer in the rapids? And I think that mindset, there is definitely an embrace of ambiguity. This goes back to the psychological safety piece, trusting that you have certain values. I think being very, like, value-centric ’cause again, to go back to my overused river analogy, when you are in the current, when you’re in white water, it’s like you can have the best laid plans, but it’s really, what are the sort of embodied values in that case of your procedural memory. But here when we’re in those river— like, what do we stand for as a company? Do we, do we value quality? ‘Cause that’s the thing that’s going to be how you’re gonna navigate around the rock. So I know I had this great back and forth with one of my tech reviewers ’cause I was talking about vision versus values. ‘Cause vision is dependent on seeing the future and if we’re not sure if we can confidently see too far out into the future, that falls apart a little bit. But if you have values, that can guide behavior when you are surfing/steering/white water rafting. Peter: You used the word stochasticity. That is an easy word to say. And I wonder if the stochasticity is related to this because, you’re not quite clear. You’re not certain. It is not determinist what is coming out of this system, so you have to be willing to kind of bob and weave with it and nudge and course correct it. It’s very cybernetic. There’s always a dialogue as opposed to you do a thing and then it’s done. Is that a fair connection? Stefanie: That is a fair connection. I love that because I was thinking of it primarily at the… what is a tool and the quality of our tools being stochastic has all of these bidirectional influences on our ways of working on systems, like macro systems, so on and so forth. So I think that’s a very fair connection and I love that, and that’s something that I will be mulling beyond this call. Peter: When you were talking about the Jidoka and the Toyota manufacturing process, and part of its reason for being is it came out of a resource-constrained reality. And I think, likely the reason the United States did not embrace these approaches is because we had the resources. We’re not constrained. We don’t understand an approach that assumes resource constraint, whereas the Japanese that was where they were starting. And I’m wondering, what is the analogy today with generative AI tooling in these environments and resource constraint? I’m asking in part because I’m working with a bunch of massive legacy enterprises, hundreds of thousands of employees. They are not in any way resource constrained. I mean, the designers always feel like there’s not enough of them to do the work given the amount of work to be done or whatever. But in any legitimate context, there are plenty of resources. That’s not the problem. And I wonder if that’s actually why a lot of those companies are struggling to embrace AI, not just because of bureaucracy and legacy ways of working, but a philosophy or a need that just doesn’t exist for them. Whereas startups or earlier phase companies or entities that are struggling, there’s some, I don’t know, to overuse perhaps now a supposed Japanese philosophy of beginner’s mind, right? Like there’s some willingness to really embrace this approach for what it is meant to be, as opposed to slapping it on or bolting on because you’ve been told, but it’s not really suitable given your context. Jesse: Right Making sure AI is fit-for-purpose Peter: But those Japanese manufacturing processes or practices were born of resource constraint. What do we learn from that when trying to apply automation tools in an environment that maybe isn’t resource constrained? Does it just mean you can do more or is that a misfit? Is there something actually ill-suited to a resource-full environment and these kinds of approaches? Stefanie: Ooh. Yes. So I’m thinking about what is the incentive structure to bring in that technology if you have unlimited resources. I think it’s a lot of the maybe incentive structures, our old processes and maybe reluctance to adopt new processes that is creating resistance, and maybe it’s just, like, how information flows. It’s like the double-edged sword. If you have a big organization, there’s more information flows, it’s harder to institute processes. So I think there may be an element of what we have seen historically, like when you brought in electricity instead of a steam engine into a factory, you know, decades and decades ago. It took 40 years for the ways of working to adapt to that because you didn’t redesign the factory. And so is that related to constraints? I think it’s maybe getting very excited over what are the technical capabilities, but not realizing that the people side of it needs to change. We are sociotechnical systems, and you gotta give the socio part of the sociotechnical some love as well, and there needs to be this joint optimization. And so I think that’s, if anything, what I’ve seen more in the research for this book, which is the over-pivot on we can automate all of the things, but not realizing that that also materially changes how we work and how we collaborate. If roles are blurring, okay, we need a new process. We need maybe different team formations. We need maybe to bring in maybe different types of expertise. And so that’s more so how I’ve seen this show up. Jesse: When I think about these large-scale enterprises, honestly, what it comes back to for me is they know too much, they’ve seen too much, they’ve been through too much. They have accumulated so much latent wisdom inside these organizations about what it takes to deliver whatever it is that they’re there to create. And it could be 3M, it could be General Motors, it could be General Electric, it could be Verizon. Like, It could be any of these very large-scale organizations where, yeah, you know what? We’ve got decades of experience that tells us what it takes to deliver at scale. And this stuff flies in the face of that. And for you as a manager to stand up among your peers and say, “We’re gonna ignore the 120 years of background and experience that we have in what it takes to run a railroad or an electrical grid or whatever the large scale thing is that you’re doing,” there’s a risk. There’s a political risk that comes with that, and it means standing up for a value prop that may or may not even really be there. And so I find myself wondering about sort of the chilling effect inside organizations that potentially limits leaders from stepping into the capabilities of these things because it means flying in the face of decades of wisdom Stefanie: Yes. I’ve seen some of this in my interviews, like primary research for the book around folks who have successfully navigated this just around small AI pilots just saying, “Okay, we’re going to start small. We’re not going to necessarily redesign the entire factory, but let’s use this sort of like little cupcake. We’re gonna do the cupcake of the cake in this area. And this is the runway I need. This is what we’re measuring. And if that goes well, that can be a sort of safe way to introduce that if you have someone who’s going out.” I’m sure you’ve seen, some of this as well. But I think, yeah, absolutely that, is true. And then of course, no one is speaking up because they’re worried that that won’t go over well, and all of a sudden you have the company continuing to operate for steam when it needs to be adapting to electricity. Peter: I think the risk frame is interesting because of the stochastic nature of AI. Businesses love certainty and dependability. And so you might think that they would fear AI because what it produces is often uncertain and out of their control. But for some reason, businesses have just decided to massively embrace AI before really understanding the implications of doing so. I think with the promise of automation and frankly firing a lot of people and cost savings. And so it’s identifying this internal contradiction going on within these businesses. Like, there’s risk aversion to a point, but I’m willing to take that risk if the amount of money to be saved is just orders of magnitude. You also mentioned values earlier, that there was some underlying value that maybe we hadn’t yet tapped into that is now being expressed when they’re behaving like, “Oh yeah, actually risk has been fine. Look at how we’re embracing AI.” That means it’s not that risk is fine, but firing a lot of people is even finer or something. Leadership v. Builders Stefanie: Yes. Oh my gosh, yes. I have so much. I have, I have… Our listeners cannot see my little squiggles, but there are two squiggles on top of each other, and they represent leadership and builders talking past one another, And this gets to one of the keystone frameworks from this book, this autonomy decision matrix, that ultimately the axes are could you do it and should you do it? And it’s designed to be the sort of common language where you have, okay, we wanna automate X, we wanna automate customer service. What does that even mean? And so that you can ideally, if you have both parties or representatives from both parties in the room, you can hash out what are you actually talking about, ’cause I think we have dreams of fully automated futures, we need to report to shareholders at the top layer, and oh my gosh, AI can do all of these things. And then to the stochasticity point, something that was coming up in the book is that even builders, like folks who are very technically skilled, they don’t even sometimes know the full capabilities of this tool. They’re like, “We’re still figuring it out.” So when leadership is like, “Let’s automate customer service,” and builder’s like, “Oh, wait a sec, no, that’s definitely…” There are a lot more humans in the loop than you think, and probably will be for a much longer time, and they are talking past one another, but then there’s top-down mandates happening, and you get all sorts of weird stuff being like, “Here’s our spreadsheet of AI features,” and “Let’s start jamming those in according to a roadmap,” and just, you start getting this sort of breakdown from that initial talking past. So one of the tools from the book, it is an attempt to be a common map so that you’re looking at how confident are we that we could do it from a capabilities and readiness perspective, and then from the should do it, what evidence do we have around, customer needs, et cetera, et cetera. And we can actually plot this thing on a two by two, this thing being what we’re talking about automating, and starting to get precise. Okay, when you say customer service, what tasks in customer service? What percentage of things are we trying to automate? Is it 80% of routine requests? Okay, what, how do we define a routine request? Can we go back into the data and define it? And all of a sudden you’re like, “Okay, maybe we can do 40% of that now, but we need to build a triage system today to go back to humans.” I think is related to maybe some of the uncertainty at the builders level, and that is creating tension when you’re going back up to the leadership. Peter: Well, right, ’cause the leaders expect this to behave like any other determinant technology, and the builders realize that’s not how this works, right? Stefanie: Yes, exactly. Jesse: When I think about these legacy companies and what the history of those companies has taught them about what it means to be successful, it feels like there’s a place where the technology starts to hold a different kind of meaning here because of its unpredictability, because of its stochasticity, because of the way in which we’re all now sort of, when I was leading AI work for Capital One back in the day, I used to say that this work is, it’s not about being a dog trainer, it’s about being a lion tamer, in the sense that the dog is eventually trained, the lion is never ever tame. And so the work continues on an ongoing basis to manage and shape the unpredictability of it. And the decision-making cultures in organizations when nobody can know for sure what is actually gonna go down. And the implications for how decisions get made, what inputs come into those decisions, and what voices, to your point about the builders, what voices are involved in those decisions, whether there is actually a change that needs to happen in decision-making culture in order to actually leverage this technology to its fullest potential. Stefanie: I love that. On my little blue piece of paper, I have a very rough and scrappy little stakeholder map with bubbles, and I have builders, I have leaders, and I wrote investors. We could put shareholders there if you want to. And the only reason I put them on the map and brought them into the mix here, I’m actually not betting against larger organizations, that, like, small organizations that are AI first are going to win out. But I think that investor perception plays into this dreams of automated future thing, ’cause if you’re like, “Oh, there are these big companies,” they might be trying their darnedest to integrate AI in meaningful ways. But there’s this idea that because you can automate all of the things so quickly, you can cut out that middle layer of management, and you can just fire a bunch of people, and so therefore, smaller companies inherently will be able to do the same thing as larger companies. Like the hill that I die on is there is a layer of tacit knowledge and situated action that like generative AI with large language models, there is an absolute ceiling. Sure, capabilities are getting better every single day. We can automate more than we are doing now, but there is a ceiling, and that information flow, you will still need those large organizations. But the investor perception that we can automate all the things adds to the sort of preconditions that leads to the dreams of automated futures. And so anyhow, that is worthwhile to mention in like the pressures those larger organizations might be feeling and reacting to, which may not actually be true of even what is happening in the building. Jesse: Well, it’s interesting to think about, what is the kind of the upper limit of all of this? Stefanie: Oh, yes. Jesse: Like just in the digital space where you don’t have factory floors to maintain, there are no chip fabs or assembly plants or shipping or logistics or any of that kind of stuff. It’s just data. It’s just ones and zeros being produced. It leads to a question like, well, how few people could you run Facebook with? Like, how small a team could potentially enable a global offering? And in that case, what are the humans on that team doing and what are the robots doing, right? Stefanie: Yes. So yes, how small, I won’t give you like a absolute number, but I’m thinking about the subject matter expertise of someone in, I don’t know, someone in legal, someone in software engineering, like that deep well of knowledge. I think there are certain things that are not machine readable. No matter how much you, hook up your Slack or your email your, whatever’s in your knowledge base, the meeting that happened after the meeting that actually informed the decisions, probably not in that knowledge base. The sort of embodiment aspect of just our being is not in that knowledge base. This is why I’m a lot more interested in, say, world models or even like biological models, There’s a lab in Singapore, very sci-fi, building this. But like there is a, there is a limit of how much of that knowledge you can make machine readable. So at some point, if you’re building something that is more complex, maybe you’ll need fewer people, but you will still need some expertise steering these machines. So will you have a fully super dark factory? Like a, like level three automation where you have a super dark factory that’s basically like a child or like you’re someone who’s just autonomously navigating and forming their own goals. There is no example of that yet, but I’d argue that we cannot get to that unless you’re literally just recreating people. And so will organizations be smaller? Yes, but they will not be as small as we might think. Jesse: I love imagining the future headline, “Rogue automotive plant starts manufacturing kitchen appliances,” right? Stefanie: Yes. No. It, it does raise the, yes, it has, it has formed its own goals, and it is now… Jesse: Mm-hmm. It’s decided this is the business move. Stefanie: Yes, exactly. That is… Is that AI first strategy? Very literally. Peter: You guess? Jesse: Yeah. AI, the Double Diamond, and New Roles Peter: Very, Very much so. Not wholly unrelated, but behind me on my whiteboard is the double diamond. And it’s up there because when I talk to design leaders, I find that it’s a way to talk about AI and design and user experience in the modern workplace. And given that you are teaching the next generation of folks who are going to be practicing this, I’m curious how what I’m about to say lands. And with the double diamond there are two vertical lines in the middle of each diamond. And the idea is when you’re on the left of the front diamond, the people doing that kind of work in strategy, in ideation and concept development, fucking love AI. I did this listening tour of design executives and what I’m saying comes from conversations with one in particular, but others have agreed as well. Strategists love th-these tools. They love being able to produce visions of near futures, six to twelve months out. Put real information, real data in those. Put those in front of real people, get real responses, iterate on them in a safe prototypy way and start even building the thing, right? The tools are affording those abilities. So you have this strategy team that’s embracing AI because of the just more rapid sense-making and you don’t need to coordinate as many other people. You can do a lot more on your own in terms of figuring out some strategic work. And then on the back half of the second diamond, the kind of delivery and ship parts of the process, that group is loving this because they’re now in Git, they’re now making pull requests, they’re now shipping code. And that chasm that used to exist between even a design technologist and an engineer has just, in many organizations, shrunk or super blurred. There’s not a chasm anymore. You’re just walking from thing to thing. But what’s happened is the folks in the middle of the double diamond are the ones who are struggling because partly they’re feeling it on both ends, right? If the strategists are able to do good enough UX work, and I’ll call the middle of the double diamond, the end of the first diamond and beginning of the second , the UX-iest part of this. The strategists are able to not just produce prototypes, but they can then turn that into something that looks and works pretty well. And the design technologists to front-end developers or the UI designers are able to kind of back up their work in using design systems and whatnot, kind of assemble a perfectly good experience. And so the people who tend to operate in the middle of the diamond are getting squeezed. At least this is what I’m hearing from some folks. I’m wondering how that framing lands for you, how you are thinking about the roles in user experience and design work as you’re having to be very intentional about it in the teaching mode that you sometimes operate in. Stefanie: Yes, I have. I think being in that middle, like, when we talk about UX roles and I think we’re saying that UX as a discipline remains important. However, the roles certainly are changing in how they’re defined, and I feel like being in the middle of the double diamond is an artifact of what has happened to UX, like post-Agile, post-commodification with design boot camps, et cetera, and so on and so forth. And how I am teaching design, I think, in many ways goes back to, like, the earliest conceptions of UX like, you know, sort of Adaptive Path, early user experience architect, and, going even farther back into the history of human factors of HCI and then service design feeding into that. A much more expansive definition where it is like, no, understand how both diamonds work, maybe have a deep well of knowledge on one part of this are you drawn to? But I don’t want you to go out into the world and just be operating at that middle layer, which is what many students who come into the classroom think they’re about to do. Like I have Jesse’s Elements of UX diagram. I think it’s in our second lecture, and the assumption for many of the students are that UX is the top layer. It is the visual maybe design layer. It’s very execution. Going Deeper in the Elements Peter: I’m sure Jesse wants to comment, but I’m not gonna let him. Because you mentioned his elements. So on my drawing of the double diamond, it might not be obvious, but I drew the five elements, Stefanie: Okay. Peter: And I talk about how those top two elements are what AI does really well, right? So, so the visual and like the top el- elements of UI design, AI can do those. And so when I think about the middle of the double diamond and the work that was UX, what I see is still an opportunity for those who might not find themselves at either end, or just an opportunity generally, is to go deeper into the elements of user experience. Because for so long, digital product design has been so focused on UI that what all the tools are trained on is that, and they don’t know how to deliver quality structure, right? They don’t understand IA, they don’t understand real flows and conditionals and all that kind of stuff that leads to a robust software experience, right? Everything got very superficial over the last 10 years in terms of these digital tools. If it’s more than you tap a screen, do a thing, and then tap back out, like these systems tend to be too hard to use. And so part of me then is wondering, and I’m wondering if this comes up in your class, right, is there a renewed focus on the deeper elements of user experience given that AI is able to automate those top parts? Stefanie: Yes. Oh my gosh. And I have a anecdote, so not to super flex. I get great teaching evaluations, but there was one critical piece of feedback maybe two years ago from a student, which was like, “This was too much of a UX research class.” And I think they’re attributing it to me being a design researcher. I’ve made great pains in subsequent years to highlight that the bottom of the diagram is user needs, and that is how we determine the goals. If you are making your text larger, it is ’cause you’re moving the user spotlight of attention ’cause it is related to their goals. Or if you’re using color to draw attention, like, it’s all connected. And I think that is– I know as a researcher, obviously I’m very excited about focusing on that, but I do wanna emphasize that to students throughout that process you’re constantly doing research. You’re tying it back to who is in this experience and trying to build a very, I think, holistic understanding of what is context. I think that is something that is lost, like the ethnographic nature of design research. I think that is quite new to folks in many cases if they’re not coming from a social science background. So I certainly do emphasize that throughout the process and remind folks not only that there is a foundation to that diagram, but that they all ladder upon one another. You can pull that thread. Jesse: So over the course of this conversation and indeed your book as well there’s been this recurring theme of connecting back to what’s gone before to learn from the history of automation and the history of organizational change related to it in order to make better choices going forward I wonder about the parts of it that we can’t predict, the parts of it that are genuinely new. And for you, from your perspective, I wonder, what are the big unknowns that you’re interested in seeing the answers to over these next few years? Stefanie: Sure. One of the big unknowns is to what extent will we be conducting, quote-unquote, user research on machines? This came up in several of my interviews around what is the output from machines? What are the gaps in the output? Can we do a log analysis of what’s coming out of our machines so we can understand either bias patterns or kind of understanding almost like the machine psychology. I think that’s kind of interesting. I brought in a friend from IDEO who wrote a paper on synthetic users to do a Q&A with my class, and then got into a whole back and forth on LinkedIn with another practitioner, in a good way about the role of synthetic data in innovation in the earlier… in discovery. And right now, I think that common wisdom is that these are regression to the mean machines, don’t use synthetic users, you know, LLM personas, like, random machine’s idea of a 23-year-old’s attitudes about iced coffee. Like that’s not gonna be predictive of actual behavior, so don’t make decisions based off of that. But there are, if you look at, like, large data sets around okay, if we have a lot of historical user behavior and we’re essentially treating a synthetic model as a database that we are querying, maybe there’s actually some utility. So like this gets to, to what extent can we bring in synthetic data earlier on in the process, I think is really interesting. And like right now, I can share if you want in the show notes, there’s a paper talking about like predicting how well prototypes will resonate. It was actually in a like scent study of all things, like different fragrances. Peter: Oh, not even information scent… Stefanie: Exactly, not even information scent, actual scent. And it was like quite accurate, but, you know, it was very bounded. And there was a lot of historical data. So I think that’s interesting. Something that was brought up from a guest speaker from Microsoft in my UX for AI course, he is designing computer use agents and I think my students’ minds were blown apart a little bit by the idea of having a persona or two personas, one of whom is a person and one of whom is an agent. And that’s really kind of mind-bending, I think in many, many ways. And do we accept that? And is that, is that responsible? Should we be doing that? Is it just a reality? So that’s something that I’m really curious about. And then I think maybe a fourth thing is designing for physical AI. I know we’ve talked a lot about, presumably, software based AI in this conversation, agentic systems and assistants, so on and so forth. But where I’m very curious and a lot of my background in industry when I’ve been in-house has been working on spatial computing products, so augmented reality, virtual reality, a lot of the same technologies that make that work apply to robotic systems like visual inertial odometry and like spatial intelligence and all that sort of stuff. And I’m really curious about the role, especially of industrial design coming back into the fore and designing interaction patterns for physical AI. I’m curious about that deep understanding of context. I think we will need, by necessity, to double down on ethnography if we’re designing household robots like Nome from Frog, I think is a really nice example of this, of a companion robot that goes around and picks up your kids Lego pieces or, stuff on the floor, and it looks kind of like a cute little coat rack. It’s not a humanoid. Of the things I mentioned, I think that’s what I’m most excited about. I see that really requiring sort of that holistic UX and design mindset and skill set to design what’s coming after the more software based AI situation that we’re in. Jesse: Dr. Stefanie Hutka, thank you so much for being with us. Stefanie: Thank you so much, Jesse. Thanks, Peter. Peter: Oh, this has been fantastic. Thank you for joining us. Stefanie: My pleasure. Jesse: If people wanna find you on the internet, where can they find you? Stefanie: Yes. You can find me first and foremost on LinkedIn. So if you look up just my name, Stefanie Hutka, feel free to connect with me there. I also publish a biweekly Substack that is Sendfull, S-E-N-D-F-U-L-L, and you can check out my book, What Your Machines Should Do, which is now available on pre-order and coming out in November Peter: And tell our listeners how important it is to pre-order books. Stefanie: It is incredibly important to pre-order books. That is all. But yes please buy books. Please leave Amazon reviews. Yes. Peter: Awesome Jesse: Fantastic. Thank you so much Stefanie: Thank you both. Jesse: For more Finding Our Way, visit findingourway.design for past episodes and transcripts, or follow the show on LinkedIn. Visit petermerholz.com to find Peter’s newsletter, The Merholz Agenda, as well as Design Org Dimensions featuring his latest thinking and the actual tools he uses with clients. If you’re looking for help with AI transformation or you just need a private advisor to help you solve your hardest leadership problems, visit my website at jessejamesgarrett.com to book your free one hour consultation. If you’ve found value in something you’ve heard today, we hope you’ll pass this episode along to someone else who can use it. Thanks for everything you do for others, and thanks so much for listening.
