79 – A Pragmatic Look at Leading Design + AI (ft. Emily Campbell)
Show Notes Design leader Emily Campbell joins Peter and Jesse to share the pragmatic details of her experience as a VP of Design for an organization embracing AI as a medium, material, and means of development. Emily Campbell: https://emilycampbell.co/ Jesse James Garrett: https://jessejamesgarrett.com/ Peter Merholz: https://petermerholz.com/ Transcript Jesse: I’m Jesse James Garrett, Peter: and I’m Peter Merholz. Jesse: And we’re finding our way, Peter: navigating the opportunities Jesse: and challenges Peter: of design and design leadership. Jesse: On today’s show, we’re joined by design leader Emily Campbell. She’ll share with us her journey and perspective on building AI-empowered design teams. We’ll look at the implications of AI for hiring, metrics, team roles, cross-functional collaboration, leadership, and the value proposition of design itself. There’s a whole lot in here, so here’s our conversation with Emily Campbell. Peter: Emily, thank you so much for joining us. Emily: Thanks for having me. I’m excited Peter: We’re excited too. Before we get started, for our listeners who might not know you, how do you introduce yourself? Emily: Yeah. So I’m Emily Campbell. I’m currently the VP of design at HackerRank, which is a technical skills and hiring platform. Jesse: AI has been a strong thread in your thinking and your writing as a design leader for some time now. And I’m curious about how AI became a part of how you approach design and design leadership. Emily: Absolutely. Yeah. I find it fascinating, and I’ll tell you why. I’ve been working with machine learning products for quite some time, but when ChatGPT dropped in, what, 2022? The company where I was leading design and research, Vendr, was very early to bring it in, and so we had to start to figure out what is this new material that our users and our customers are gonna be able to play with themselves? What does it look like to create those interactions on our end, and how do we define what it means to have a good user experience inside of this product and platform we were building? Developing a language of AI interaction Emily: And so I started to need to develop my own vocabulary for AI interaction, whether you are interacting with the model directly in these conversational experiences or working with places where AI would bring in, like, ambient insights or help to summarize and make sense of the work that you were doing inside of our products. So that was the reason why I started to get into it. And then in January of ’24, I dropped a side project called The Shape of AI, which I continue to manage, and it’s a catalog of vocabulary I was developing to make sense of and then use to help my teams make sense of this material. What is it? What’s actually happening behind the scenes? I think it’s very easy for us to get very fixated on what exists today, but this whole AI UX, agentic UX, everything that we’re working with right now is evolving so fast that the truth or the source of truth is evolving along with it. And so, yeah, it’s been this project of just trying to understand this moving target and then be intentional about working with it. Jesse: One thing that struck me about The Shape of AI project was the way in which it was really designed to facilitate conversations. And it’s interesting to think about designers as facilitators of conversation and that as being part of what we do in design as a function. And I’m, curious about, I guess, your thoughts on how AI impacts how we think about design as a function. The shifting responsibility of Design Emily: I think that what you’re describing manifests actually on a few different levels. So I’ll, tell you what I’m seeing across them. So the first is it just brings us all together, us being designers, product managers, engineers, business analysts, whoever, into kind of a common working group, because what used to stand in the way of us collaborating was the ability to write code and to assess data, make decisions based on it. So the first thing I’d see is that AI has flattened this PED, or whatever you might call it, organization and really made it so that we can all contribute ideas and contribute directly even to the code base regardless of our position or our role or our background. So that’s one of just like, our job as designers is not to show people the picture of what should be, it’s actually increasingly to play around with what is or turn this prototyping concept into a, more sort of realistic source of truth so that we can have a conversation together as opposed to having to have this kind of linear shifting of whatever the source of truth is over the course of the growth and fidelity of this thing we’re creating and then ultimately shipping. So that’s one. I think it allows us to be a lot more intertwined in how we work. The second is that we are no longer building the thing people interact with. Increasingly we’re building the thing that determines the thing that people interact with, right? And that’s where this whole agent design world comes in because it used to be that we would determine, okay, well, someone comes in and they see this screen and these buttons and this information, and maybe there’s some differences depending on what their role is inside of the organization or the content inside of their algorithm, but we had a lot of control over the, bounding box. And so that’s where we’d use like usability testing to say, “Did we build this right? Can people use this the way we expected to do this thing we think they’re trying to do?” Now we’re moving into this world where the humans on the other side are the ones determining their experience, and we’re just shaping the ecosystem. And so there, I think it’s just the role of the design, and the thing that we’re creating is more abstracted away from like a screen that we are drawing a picture of. And so again, it really forces us to change the conversation about what are we trying to build? How do we know if it’s good? What are the risks? What are the different scenarios somebody might be in? And it just forces a change of vocabulary and a change of focus as a group. So I’d say that’s a kind of high level summary of some of the changes I’m experiencing. Is Adaptive UI actually a thing? Peter: Something that’s been addressed a fair bit in the last year or two when it comes to design and AI is the opportunity for even more personalized experiences, I guess, where every user can have some user experience that is distinct to them thanks to the AI having done some work. I kind of want to call bullshit on that just because it’s been tried before, right? The idea of personalization and purpose-fit interfaces for individuals is not new because of AI. Don’t get me ranting about how we tend to forget the prior 30 years of HCI when we do these things. But, in any attempt to creating interfaces that evolve tend to get rejected because it turns out people like stability and fixity in their tools. That’s a mental model they bring from life and they want to have in software. And I’m still waiting for the examples of those interfaces. You know, It’s been a couple years now that we’ve been playing with these tools, and I still don’t know of an interface out there that is AI-driven, where it re- forms itself for me, and my experience would be different than yours and different than Jesse’s based on my prior use of it. At least nothing that isn’t just like what you already get when you shop on Amazon, right? And, what I see on Amazon is different than what you guys see on Amazon because of my prior history. So I’m like what am I missing? Or am I missing something? Emily: No, I think you’re totally right. And I mean, the reality is we’ve had some degree of generative experiences because we’ve been living in this algorithmic world since, you know, 2006 when Facebook launched. Since these algorithmic platforms started to take over our life. And I think we need to recognize that the people driving the discourse right now tend to be the people who build software professionally, where building something for yourself is just what we do. And so we can very easily adapt our mindset to, oh I’m going on a trip and I need an app that can let me plan with my friends. Well, let me just build that for myself, and then we can all jump into it. So, that is certainly an extreme example that we will see, but the likelihood of that being something that catches on for the reasons that you’re describing, Peter, I mean, like, it’s one thing to create something for yourself. As soon as you have another person using it, the complications of management and permissions and information and data goes up exponentially. So, when we talk about generative AI or generative UI, like one of the examples where I do see it showing up is more at like the artifact level. So I was researching a local issue here in Moab where I live, and I was surprised when ChatGPT actually threw a supply and demand curve artifact into the conversation that I didn’t ask for. But it was actually incredibly helpful because it was interactive, and it was giving me some information about a local budgetary question. But I actually could play around with the parameters that it was talking about. I think that’s a great example of where generative UI can actually be very beneficial. Or say you’re talking to chat support, and you run into a situation where you’ve asked for something a couple of times and the agent hasn’t been able to fulfill your needs. Well, rather than direct you to a new page or ask you to go talk to somebody else, it might just give you the UI inside of the chat interface to say, “Hey, do you wanna just enter this yourself?” Or you know, any other type of interactivity that lets you get back on track. So I do think that there is a world where it exists, but it exists more in the situation or the context of the engagement as opposed to like we’re all gonna have our own version of the digital web existing on our own devices. And then the last thing I’d say is around these algorithmic experiences. Right now they are so opaque. So I spent some time over the weekend building myself a plugin that I could put on LinkedIn that would hide certain types of posts and replace them with artwork or vocabulary or poetry. And it’s actually quite depressing. I’ll send you a picture , but let’s just say my for you page on LinkedIn is entirely cleared out by blocked posts. But it made me realize the only reason I’m not engaging with that platform anymore is because the content is no longer relevant to me. If the incentive structure of LinkedIn were to get my attention back and have me engage with it, giving me some degree of control over what I see would actually be really beneficial. And we’re actually seeing that happen on X right now where they’ve been playing with their algorithm. I would hope that we will move eventually to a world where that type of control is also available, where it’s not so much AI-driven, it’s a little bit more kind of playing with the machine learning knobs. But that’s where personalization, it’s like the incentive doesn’t change for the company, but the way I reach it is more delivered through my priorities or interests, which might be different from yours. Designing when you don’t know exactly how it will look Peter: So some of this reminds me of when Jesse and I were little web design babies, and we… like, I worked for a graphic design firm in 1990–, make sure I get the timing right, 1996, so as it was embracing the web. I was a web developer there. And one of the biggest frustrations I had is helping graphic designers understand that they could not detailed architect every pixel on the screen for their users, right? That they needed to lean into this web medium, understand HTML, understand that typography’s gonna change, and depending on browsers and all that kind of stuff. And there was some success in doing that. Some designers realized that, like, you could only go so far in nailing down what the user experience was. And then we reverted, I think, when it came to starting with mobile and then the web kind of picking up on that again where we were, thanks to CSS and all kinds of things, able to pretty highly specify what’s going on the other end of the screen. And designers love that, right? Designers love to dictate what the experience is that their users are going to have. Or let me say many designers love that. This is where I think you start hearing about craft again. Oh, craft. Delicious, delightful craft. But what you’re talking about raises the question how do I judge the goodness of a design, right? If I’m a designer creating some experience for someone, but that experience can emerge based on all these contextual inputs, most of which I don’t have control over as the designer, what does it mean to do design and to do design well in that context? Emily: Yeah. Again I, think that’s where we’re starting to see some separation depending on who and what you’re designing for. And again, like a lot of the conversation is being dictated by a specific type of designer, which is more oriented towards brand design or emotional design, things that are meant to catch the senses but maybe not help you do something else. And so a lot of conversation around craft right now that I’m observing, at least in terms of like the general discourse, what you’re seeing on the platforms, is oriented towards how do you capture someone’s attention? How do you stand out? How do you appear authentic? How do you help people connect with something that they wanna interact with? I mean, brand is increasingly becoming like community-oriented and I would say brand designers are more trying to capture that vibe shift towards the brand you’re representing as opposed to what like a product designer would be doing or, someone who’s working more in that medium. So how we define good starts with what we’re trying to do in the first place and being deliberate about that. But I’ll speak to my world which is a little more to this like product design side. One of the things I’ve had my team start to do is actually write specs for the more probabilistic parts of the experience. And a spec really just meaning defining what good means upfront so that we have something to baseline off of when we get deeper into the process of actually seeing how the design itself is emerging and then also how people are using it. So for example, HackerRank right now where I am has an AI interviewer product that is replacing what has come to be known as like a really nasty part of the hiring process for engineers where you get sent some take-home tests that you have to take and then send it back and now companies are very concerned about cheating and integrity so it’s led to some decay in the user experience. So we’ve been playing around with, well, what if instead you could do the work and then talk through it as opposed to having somebody think maybe you were, cheating on the work itself. So what does it look like to be interviewed by an AI agent? I mean, that’s a really distinct and kind of icky experience if it’s not done with care. So we’ve had to think about, like, if a human was interviewing you, what would that experience look like in a way that would be good? Especially since interviews at scale, whether it’s a human or a machine, it’s just generally a little bit inhuman. It’s meant to be a repeatable process. So we forced ourselves to say what does good mean? What are the different scenarios where things can go wrong? How would we expect a proctor or an interviewer to handle that situation? And then as we start to design the experience with this AI agent, we have this real benchmark that we can look at the actual conversations for example that the AI is having with the users and say, “Oh, interesting. Here a user asked a question and we would expect the AI to respond one way, it responded a different way.” That’s now an observed error and so we can improve the prompt or we can improve the harness or the interface context and so on to try and get a better outcome. So I think a lot of what does good mean and what does it mean to design starts with being intentional, which has always been the case. Starts with defining what is a positive experience through the lens of the person experiencing it, and then the material itself, of how that experience is actually rendered, is the thing that’s evolving. But I think we just can get very easily stuck in that when we’re still in the world of fundamentals. So that’s some of the things that we’ve been playing with and I think we’re gonna keep seeing evolution there, ’cause there’s also all sorts of learning that’s starting to come out about like effective evals. It’s easy to use them in a way that actually gets the AI or the agent to behave in a certain way that’s almost like over-fitting to the situations you’ve observed, so we have to think about situations we haven’t observed. So there’s a lot of really interesting discovery and research that goes into making sure that you’re casting a wide net as well. AI as material and medium Jesse: It’s interesting the way in which this brings, as you touched on, the probabilistic into the design conversation. And in the course of robbing designers of their auteurship, we open up potentially these new areas to play in creatively. When I think about probabilistic design, I think about games. I think about video games or board games or these other kinds of complex interlocking systems that are built around, in some cases, probabilistic processes, the roll of the dice as the core. And I find myself curious about how many designers can really make that leap to that probabilistic way of thinking? Because as Peter pointed out, the history of design tradition carries within it a degree of deterministic thinking. And those of us who’ve worked in digital design for the last 25 years, have had to cope with the variability of the digital environment in ways that design traditions historically have not had to. And now this feels like an even further step away from… I’m reminded of a talk I gave many years ago that I think is on the web under the title Design for Engagement, although it went under a number of titles over the years, where I talk about design practice having historically been grounded in mastery of a medium. And so graphic design has its dogma and its ways of thinking about problems that are rooted in the medium of paper fundamentally. And so a graphic designer can tell you what paper has taught them about things, basically. And an industrial designer can tell you what plastic and rubber and metal have been able to teach them about what it means to make something. And I’m curious about how those people can find a way to embrace a new way of thinking when their identities have been so tied to feeling like you have mastered something. How do you bring people along into this new world and help them see the creative opportunities there when their ideas about themselves and their practices and their value are really rooted in this mastery of a medium? Emily: I came out into design about 2008, ‘ 9, ’10, when we were starting to see mobile design start to take up and we’re… Myself, we as designers were still working with code. Like, I wrote the CSS of the first platform that I designed for. So I think that for my generation of people in this industry, this idea of kind of like tinkering and playing around with the material or the medium itself comes very naturally. Then you have a generation of designers who came out of art schools and simultaneously out of boot camps, where both were not fully capturing the other side of the world that each was seeing. So I think it’s a multifaceted challenge, I guess, that hits us all a little bit differently. I’ll tell you this. The people that I’m seeing thrive right now are maybe not the folks who are traditionally working on the interfaces, but people who are service designers, conversational designers, more traditional UX designers, and design thinkers. Those are folks who are, at least in the world that I’m seeing, the teams that I’m building, really excelling. In terms of connecting this more probabilistic world to the traditional or more classic design work, I don’t know that I have an answer for that, honestly. While still respecting the fact that everybody connects to their work and to why they do this particular work in a different way, I often find myself wondering if we rotate a little bit too far towards what design looks like at any time as a definition of what it means to be a designer versus kind of what are we trying to achieve, which in our world is helping people navigate information effectively, have command over their experiences, their data, their networks, and maintain their own sense of agency as they increasingly live their lives in this digital space, which might require us to be comfortable changing the surface that we are working within while still coming back to our roots. Because the fundamentals are not that different than they used to be, at least as it relates to understanding bias, understanding how people cognitively process information, the upper limits of our ability to process information at scale as humans, and how we use these different technologies in an assistive way. That’s what I look to when I look for meaning and purpose and coach other people on as well. But I do think that, we are at a moment where people do need to just touch into themselves and say, “What drives me? What feeds me? Why am I here?” And I do wanna just recognize, like I think it’s a very difficult moment for lot of folks who are having to go through that personal transformation at the same time that the work and the things that they are working on are transforming as well. Evolving as a Design Leader these past few years Peter: You’ve been VP of Design at HackerRank for about two and a half years, which is an interesting timeframe when we look at kind of the moment when these tools really started affecting kind of how design work happens. And I’m wondering, in these two and a half years that you’ve been there, how you’ve changed as a leader. Like, was there certain set of practices or assumptions that you entered with that now, two and a half years later, you realize like, “Oh, I need to show up in different ways for my team, for others in the business”? Like, how have you evolved in these two and a half years? Emily: Oh my gosh, it’s night and day. It really is. I’d say every six months or so, I’ve had to completely reinvent my job over the last two and a half or three years. So when I started, the team that I inherited was very disempowered. They were still operating in a world where a designer would wait for a product manager to give them a spec that they were supposed to be working on, and then they would work on it and be told by all number of stakeholders whether they were on the right track or not. And trying to get information about how people intended to use this product, what they were trying to do, more traditional discovery was sometimes hard to come by. It was a pretty desparate situation, honestly. And so the first six months was really just focused on how do we build a team of empowered people who have the resources, information, and time that they need to do good work? And then what does it look like to raise that bar for the folks who are practitioners so they know what they are capable of achieving and that they will be rewarded by opportunity and by empowerment and so on on the other side. So the first six months was really trying to pull this team into just a higher mature way of working. And then something started to happen where the team started to spread out in a bell curve. And as this new technology began to come into our work life and also our personal life, some people naturally found themselves tinkering with it, attracted by it. Some people, for maybe intimidation reasons or because their work wasn’t as closely associated with bringing this in, were a lot slower to move. And so we went through about a year where the team was really kind of all over the place. And I found myself and the leaders that I work with kind of pulling people up, but also meeting the frontline folks where they were and trying to stay relevant and current to what they were seeing, ’cause they were pushing everyone, including me, forward. And then in the last six to nine months, particularly after last Christmas was around the time that we saw this Claude Code breakout and everybody could suddenly do all these great things with coding, the team has really come back together to the point where everyone is now actively using these in different ways, but actively pushing on this material and really reinventing themselves. And if anything, I’d say it’s the most empowered the team has been, even if we’re doing less classical design work, where executives, product managers, engineers are coming to the design team and saying, “How should we be thinking about this? What are the different ways we might approach this? Hey, I made a prototype. Is this good? How do I know if it’s good? What does good design mean?” And now we have designers who aren’t just at the table, but they’re actually driving a lot of the strategic thinking and a lot of the ideas around the user experience where they were completely left out of that two years prior. And I’d like to say that I had a large part of that, but it’s also just the individuals, because now they don’t have to wait for me to create the permission structure. They don’t have to wait for some PM to say, “Okay, here’s some data. Here’s what we can work on.” They now can go and raise a PR. They can go and make a change to the point that we have this bot that lives in our Slack where if you have an idea of something you want to improve, you can just describe it and it’ll raise a PR. The designers have been working with the engineers to actually improve how that bot interacts with them, so it now will actually create your own private node of this new idea. It’ll give you pictures of the before and after. Like, you can touch this material directly from within Slack, and that’s where I’ve seen the biggest change. So from a leadership perspective, my world has definitely moved more into facilitating where information lives, trying to unblock people. I feel like I’m doing a lot more operational work as a leader, and then trying to find for myself and the team at the same time what does it mean to grow as a designer as we’re trying to reimagine what that even means. So it’s, it’s been quite a journey over the last couple of years, and I’m sure I’m not the only person who’s feeling that exact same trip. Creating alignment when everything is shifting Jesse: Yeah. So much of it comes back to how you manage changing expectations both within the team as well as with your cross-functional partners, as well as with the executives. And the burden of all of that expectation management falls on the design leader. And I’m curious about your read on, like, what are the biggest tactical challenges that you saw in creating that alignment when the expectations themselves might be a moving target, right? Emily: I think it starts with understanding what our expectations are for ourselves and for our team because any expectation we’re being fed is completely under-informed because nobody really knows what’s going on right now. So the perfect example is the executive who says, “Hey, can you do this tomorrow? You have AI,” with some magical thinking that AI is going to be able to fully replace our jobs overnight and why are we even here, but it’s a good thing ’cause we can give you this pretty picture or whatever you’re looking for. In some way I think pulling people into the process has been the most effective guard of the process. When you have people creating their own prototypes, playing with the design, exploring the design system and our rules and so on, it becomes pretty aware to them quite quickly that this is more complicated than they realize. So for example, we had an engineer who vibe coded an entire new way of thinking about one of our products and then realized pretty quickly that the information architecture was completely broken. He didn’t know it was information architecture. To him, it was just, “This is confusing. I don’t understand why this doesn’t just work.” And so in that situation, the designer worked very closely with this engineer to actually, like, go through, what do we want someone to do here? What are the different situations you had in mind? And it allowed us to have a conversation about the reality that your ability to have that picture in your head it doesn’t account for all of those different edge cases, situations, circumstances, lack of context and so on that we would maybe be thinking about proactively as designers. So just getting people realizing that design is not, quote unquote, solved, has allowed us to have a better hand at creating shared expectations about what we can do. I think that’s probably the biggest place where we’ve seen friction is just, people thinking this is a magic box. I don’t even… I’m struggling to find my words here, for the reason that when I think about expectations, I don’t even know what to expect in six months. I don’t know what to expect in six weeks. So even just thinking about my expectations for myself as a leader or my team and how I’m coaching them to, to evolve and where I’m seeing growth, there’s a large degree of humility and trying to just be very comfortable with not always having the answers. That’s also been a really difficult part of this moment that again I’m sure I’m not the only design leader or designer who’s feeling that tension and that uncertainty. Peter: So earlier you talked about how when you rejoined HackerRank, I see two and a half years ago, the team that you inherited was somewhat disempowered, and now it sounds like the team that you’re leading is not only empowered, they themselves are leading internally. You’ve explained some of the mechanisms, but I’m curious to what you would ascribe this to, either, did your team evolve to fit better within the context in which HackerRank operates and builds, or did the people outside the team, engineers, product, maybe marketing and executives, others as well, did they develop new awarenesses and understandings such that they could allow design to meet them? I’m trying to figure out, like, where did the change happen that allowed for design to be perceived, to better integrate with the more strategic and, to have greater agency within the business? Emily: Yeah. It was a combination of all of the above. Certainly the skill sets that we’re hiring for and developing have changed. We have a much higher bar for technical fluency. Just curiosity. Curiosity is a skill that I explicitly look for when I’m hiring people and that I’ve been trying to help develop in the folks on the team to be comfortable asking questions not just about themselves or what they’re building, to understand the worlds that other people are building within, especially as we’re working more within these like code bases and shared spaces, and also just a curiosity about these models, this more probabilistic aspect of these things we’re building. So I think that’s part of it. Like we’re definitely intentionally changing the shape of the team. That also has been met by the individuals themselves just leaning in and creating their own futures. And so I’ve seen my job just creating that space for them to grow and then letting them fill it. And, you know, obviously when you’re managing a team, there’s different strengths and places where it’s not as easy for the individual. But that’s been another focus of just creating the space and then developing the skills that’ll help them grow into it. But we are seeing folks lean in from the outside, and I don’t know how much of that to take credit in terms of just, like, joining as a more empowered leader myself and allowing others to see more of what we’re doing and challenging them to understand. Like I had a conversation with an engineer last December who told me he had not had a chance to work with a design team like this, and he was starting to have a better understanding of why we pushed back and when we did, and he was feeling a lot more excited about working with us. So I think it’s like a reinforcement loop. We’re growing the agency, the skill set and the expectations of the individuals, but in return we’re seeing the organization meet us and wanna pull more out of that, wanna be more part of that. So that’s been not without its challenges, but I think it’s been two-sided. And again, I think it also comes just down to this fact that we’re all working differently. To be successful building products, you have to get out of your own discipline moreso than ever because success is no longer drawing a picture. It’s getting something to a form that can actually be interacted with ’cause it’s the only way that you can evaluate the probabilistic aspects to it and the data that’s going through it. We have to be, at least as product designers it’s increasingly important that we’re actually working in this more interactive material. We’re still doing more traditional work. We’re still working in the canvas. We’re still doing exploration, but that drift between the two is a lot more comfortable and more frequent. Leading when practice is disparate Jesse: One of the patterns that I’ve heard people talk about is that when you have designers who are all on their own journeys with the AI tooling they develop their own quirks, their own dialects, their own rhythms and approaches to the prompt craft. And I find myself wondering about how you as a leader, especially from an operational perspective, but just, just in terms of having a common language, having a common way of doing things, how do you create the space for a bunch of different variant approaches to design and still feel like you can make apples to apples comparisons in terms of performance management or even just be able to set a cultural standard for quality or anything else? Emily: I have had to let go quite a bit of things that I maybe had as like a security blanket previously, of being able to see the work evolve and always be aware of where things were going well or going not well. That’s been a big learning because increasingly, and this is a problem that I hope somebody will solve soon, but we’ll have five or six different versions of thought expressed through some artifact live at any given time, and none of them are the truth at any given time either. And so as a design leader, in terms of holding that quality bar, I’m spending a lot more time in the more operational area, making sure that… Like for example one of the designers on my team has been completely redeveloping our design system component structure into a skills-based structure. So we actually now have an like agentic first definition of our components that can be linted by an agent that capture the semantic intent of how this thing fits into the system and how it’s supposed to act, visual aspects and so on. That is one of the few ways that I can control for the inputs into the work. So that’s one. We still spend a lot of time, I spend a lot of time one-on-one. Managers underneath me spend a lot of time working with me and then working with their teams on still getting back to these like fundamentals. So, hey, let’s put the screen to the side. What does this experience mean for the user? Where might they get tripped up? What are the things that are higher risk that we need to consider? And again, that’s where like speccing it out and describing good beforehand. It’s like this barbell effect, where we’re spending a lot more time on the inputs and on defining the experience, defining the service layer and so on. And then I’m having to just let go of the work itself. And then on the backside, and this is where more of this like evaluation work comes in, we have to make sure that not just the properties of the things we’re building, like the pixels, but also the agents and the unknowns that are inside of this experience are operating as expected as well. And so I’ve pushed very hard to get designers involved in that, building rubrics, building evaluation loops, watching, like, actually watching and reviewing the trace data coming out of the system and then saying, “Hey, does this match our intent? And do we need to change this at the product level, at the prompt level?” So it’s a lot of like system dynamics. That I love. I mean, my brain enjoys it but it’s an entire remapping. And then in terms of like career development, again, there’s a combination of fundamentals that still matter. So the ability to translate in a language that is either clear to other humans or increasingly to these models, what do I intend to build? Is it intentional? What’s informing it? How much effort does it take to get to a good version of that? That kind of captures this question of, do I have that strength in the fundamental property that we would look for? But then there’s this new category, which is like, how well can you work with this medium and orchestrate it and steer it and bring others into the process? That’s a lot harder to capture. And so increasingly, individual performance is a mapping of these fundamental skills, but also the ability to work with this system and get better outcomes out it collectively. We’re early in this. Six months ago, we were doing more traditional leveling and I’m remapping it right now, but I think it’s gonna keep evolving. Tolerating failure Jesse: Mm-hmm. Earlier you touched on giving people the resources, giving people the time and the space and the permission to experiment. It seems to me that going along with that permission, there has to be a certain cultural tolerance for failure that has to be encouraged because you want to try to get people to jump into the unknown and see what’s on the other side, and sometimes what’s on the other side is not very good. How are you managing tolerance for failure? Emily: I’ll say this, the people who are struggling, or to the extent we’ve seen folks struggle, been more a reflection of a hesitancy to leaning versus a reflection of what they did when they actually were playing around in this new world, if that makes any sense. So there definitely was some early resistance. Like, there’s a lot of reasons to be resistant to what’s happening. Like, there’s, ethical and ecological reasons to be concerned. There’s a lot of concern at different levels of, like, what does this mean for me and my growth path? So I think it’s worth, acknowledging that. So when I say resistance, it’s more like, not wanting to even try to see what’s on the other side, to have an informed opinion, even if that informed opinion ends up including things that you feel aren’t working well or could change. So that’s probably where I’ve seen, the most tension. Outside of that, there definitely is a degree to which I’ve been focusing on creating this, like, risk tolerance pool, but it’s not only happening in design, and I think that’s the critical thing that it’s everybody who’s seeing their jobs evolve right now. We’re seeing our markets evolve. We’re seeing the processes, the things that we could fall back on and unconsciously rely on. “Oh, well, and I do this, and then somebody else would do this next thing.” So I think it really needs to be something that’s solved at the organizational level. It’s not something that at the individual department or individual business unit we can manage by ourselves. So I spend a lot of time with my peers at the leadership level talking about what does it mean to have PMs and designers raising PRs? What do we need from engineers in order for people to feel comfortable leaning in? What do people need from us? So I think it’s a larger reconstitution of what is a team that develops products or develops experiences or services together that’s happening. I, I have… I keep ending my sentences with these, like, dot dot dots because what I feel confident about right now, I’m certain I will not be as confident about some part of that in six weeks. New ways of getting to understanding Peter: But let’s take it directionally though. And a conversation that we’ve had with actually our prior two guests is around teams and teaming and roles and who does what, right? In the before times, in 2022 let’s say, you had your designer, you had your product manager, you had your engineer. Maybe you had your researcher and content designer, and you maybe had your business analyst and quality assurance, right? People’s roles defined their work. And in the new times, that’s getting all… that snow globe is getting shook up quite vigorously. And I’m wondering what you’re seeing then in terms of people coming together to do this work and what are the mechanisms that predispose that group of people coming together to do the work, to do it well, and maybe better than others who are not engaging in those mechanisms? What are maybe some early things that you’re witnessing that suggest a trend towards some new ways of behaving, particularly around collaboration and in teams? Emily: Yeah. Yeah. So the way I’ve described it when I’ve talked to folks about this is in the before times, which I love, that’s great, 2022, such an era ago. But back then the teams that won were the ones that were less wrong faster or less wrong more often. And by that, I mean the entire SDLC/PED team process was intended around the idea of we have to guess what this person downstream of us wants to do and what they need in order to do that well, and then build that thing accurately and as quickly as possible so we can get it in their hands and find out how wrong we were. And you’re either wrong because you missed the mark in the first place, or you’re wrong because you were right, but it actually opened up even more problems to solve or the actual problem to solve, and you have to repeat it. And so a lot of our processes were around, well, how do we get information faster, which is where you get like move fast and break things and so on. And so like the rise of product discovery as this core concept is what was feeding that engine. Well, the more information we have up front the more likely we’re gonna get to that faster, better answer. So, we’re still in that world, but a few things are changing. Like, first of all, we can run more of this in parallel. Like we can be talking to customers, building five different prototypes at the same time. It doesn’t cost, certainly the engineering time and even less design time now to get to different ideas that are actually somewhat realistic. And we can bring those back fairly quickly to our customers or people that represent our users, or even just internally for those conversations. So one thing I’d say is that we’re spending less time talking and more time doing, but not assuming the thing that we’re doing is the thing we’re ultimately going to ship. So, that then has the effect of making it less important to have someone who’s like really good at building interactive prototypes, and more important to find somebody who can think and maintain their understanding of their system while they’re working in different, whether it’s in Figma or in code or documentation and so on. And then where I think we’re seeing things slow down is more at that decision-making moment because we can build faster, so we have to actually make that call of what are we going to ship. And that’s where I think we have the most tension because who’s gonna ultimately make that decision becomes harder to determine In our case, our founder has been a lot more involved as the CEO, helping us unblock decision-making because we’re actually not operating in a world where a product manager wrote a PRD and that’s the source of truth and we ship it when every single test case underneath that PRD is accepted. Instead, we’re having to kind of vibe build our way into what do we think our customers want, ’cause realize, the people we’re building for, they’re transforming too. Like at HackerRank, for example until a few months ago, the primary people we were building for were technical recruiters because those were the people at the front line to help find engineers to bring into the hiring process that would then later on be talking to a director of engineering. But now that the directors of engineering are saying, “The people we need to hire are totally different than the people we needed six months ago,” they are the ones increasingly in the loop, which is then causing us to build more technical products ’cause we’re now building for the actual end-end user, not this person in the middle. So that’s an example of how complicated this gets. Everything is changing all at once, but I would say the thing that is definitely evolving, I think in a very positive way, is we are not spending time waiting for somebody else to tell us the answer. We are collectively having to determine that for ourselves and make sure at any given time, as we get new information, everyone’s generally aligned on what it is we wanna ship, which has been, I think, what teams aspired to be back in the before time. Those really fast, fast-moving, highly empowered teams were the ones who could ship quickly and get to decision-making faster. We’re just not hindered by the same constraints that we once had, so more teams have that available to them. That felt like a word sandwich and I don’t, like as, as I say this it is this change. It’s like change management isn’t something that can be done for you. Like we’re all in the process of going through change management collectively, which means as a leader, we need to be thinking about at any given time what is truth, what do we know isn’t truth, and what are the questions we’re managing, and how do we feel comfortable either managing that risk or at least coming up with a bet that allows other people to move forward. So a lot more of that like shift left mentality is hitting us as well. What does discovery look like now? Jesse: You touched earlier on the notion of continuous discovery and Teresa Torres’ work around that subject. And in the AI transformation consulting work that I’ve been doing this has been a topic that comes up so often because what I’ve been discussing with leaders is the need to develop internal continuous discovery capabilities because you’ve got teams that are running off on explorations in every direction, and you’re getting different results in different places. And there’s an element in this of taking that, as you described it, taking that lens that we have historically turned on users and turning it on ourselves to some extent and developing new internal capabilities to do that. And I wonder how that process has been for you of building out discovery as not just a philosophy or a good idea, but an actual practice that’s going to inform the evolution of the team. And what other kinds of practices you’ve had to build out or institute to facilitate this transformation? Emily: So I mentioned building specs for agents, but it’s not just for agents, it’s for these like probabilistic experiences. So that’s been, I think, one of the most powerful anchors that we’ve put in place, which is, we don’t have a PRD representing a truth that we’re shipping downstream. We have almost a kind of declaration of what we think people will experience from the system or should experience, and that is the thing that’s evolving as we get new information. So, it’s almost like a continuous truth. Maybe that’s the continuous discovery part, but it’s being contributed to by everybody, and it’s objective. It’s not someone’s role to produce. And it also determines then, as I mentioned, like how we evaluate the parts that we’re not directly touching, which is the AI aspects of it. So that’s been a really helpful tool, and not just a tool for ourselves, but also a tool for evaluating the platform. We have more data available to us. We don’t need to wait for somebody to give us permission to talk to somebody else. Like, for example, Fullstory has recently released an MCP. We can be working on something inside of our worlds, whatever harness you’re using to connect with some agent, and you can actually say, like, “Hey, if I were to build this, how many people who have touched this part of the product in the last month would be likely to see it? What are they doing right now?” So we can almost ask the questions proactively, preemptively, which we always could do, but it was harder to get the data. And so even though it was available to us, it wasn’t always something that made sense or something that was feasible in the moment. So that’s another thing of just starting to realize the things that were blockers previously might not be a blocker anymore has also been really helpful. And so one of the things we do when we’re working on a new question or so on is, like, what data do we already have? Versus that idea of like shipping something downstream to get the information. Well, now, a lot of times we have the ability to ask those questions, and that means figuring out what questions to ask becomes the really important thing because the data is more available to us. So that’s more of just yeah, building those parameters and then actually trying to find answers in the data that exists in ways we might not have mapped those patterns previously. That’s been another tool that we have found. And then finally, yeah, the ability to just create many different things in parallel. We have a number of large customers, we’re talking about a new part of our interview tooling that we’re talking to them about. We have different versions of our prototype that we’re showing. Now here, the delicacy becomes, well, if you show person A something, and you show person B something different, do you end up building both A and B, or who makes that decision about what this final thing should look like? And that’s another place where having clear intent of who we’re building for and why, some founder-driven or determined decision-making able, whether that’s me or somebody above me, and then also maybe there’s some generative aspect, back to the start of our conversation. Like, maybe there are places where we start to say somebody might be able to have experience A or experience B, and maybe that’s okay. And that’s something, we’re starting to grapple with as well. I think what discovery has to start with though, as it always has, is that intent. Like, we have to ask the questions that we plan to answer. We have to have some idea at any given time of what do we think truth is. To the extent we are spending time talking, it’s more in that realm. It’s having those conversations and looking at the system together and saying, “What do we know? What don’t we know?” And using that to make decisions as opposed to debating, “I like this, I don’t like this, you’re wrong, this is hard to build,” and so on, which is what was blocking a lot of teams previously. Peter: You stated earlier how your role has shifted, if I understood right, away from as much maybe active design direction and more, you talked about operations and kind of enablement of the team to help them do their best work. And I’m wondering if you were to work with some form of partner or mentor, what are the questions, like, what are your highest order questions right now? What are the challenges that you’re trying to solve that you would welcome some insight into going forward as you’ve shifted into this new vein? Emily: That’s a long list. I have more questions than answers even at this point. Oh, gosh. So my team is still rather small. I mean, we’re somewhere around 12, 12 folks that I directly oversee, and then obviously providing leadership to a larger organization alongside my peers. But I imagine that the world I’m in and describing is very different than somebody who’s leading a team of 300 designers. So I am not sure I feel confident that the really scrappy way we’re working scales effectively without adding additional layers of management, and that’s a question I have is, like, what does management turn into as we start to have these more empowered teams, as work is going faster, and as we’re increasingly working in these kind of like fuzzy boundaries? What does the form of a design manager need to evolve into in a way that is valued and valuable? I find a lot of conviction that it is valuable, but I’m not quite sure I know at scale what that looks like. That’s a question I’m curious about and talking to people I admire and trust. And then similarly, and you alluded to this earlier, within a smaller group it’s a lot easier to create new career paths, new leveling that can be digestible and that people trust. But when you are working with a larger group of, for example, new talent and also senior talent, how is that evolving and how should I be thinking about coaching, direction decision-making, and so on… Rewarding? I mean, that’s another one… of trying to solve. And of course, the labor market is a lot more fluid than it has been in the past for really top players. And so, what are those… Yeah, maybe there’s another one of, like, what signals are other people seeing as early signs of growth, of acceleration at the individual level or at the team level that might be more visible at scale or might be more visible in different scenarios. That’s a third. Of just, I can generally sense that from the people I work with, from the closeness that I have to them or to their managers. But I’m not sure we have determined what younger talent needs to look like. And even then, you have, I don’t even mean younger in age. Newer talent might be people coming out of their own explorations, like, autodidactical folks. It might be this kind of new form of education and boot camps that’s starting to emerge. And then of course, traditional education. Like, I’m seeing a lot of signs that early talent is actually surprisingly competent right now because our tools can do a lot of the stuff that we learned as, like, the more rigorous parts. “Oh, I don’t know how this tool works.” Doesn’t matter. I can now have Claude just do it. So how do we find signals of strength in those fundamentals, and how do we close that gap? These are just some of the questions that float around my head and give me anxiety on a week-over-week basis. The big one is, leveling, individual growth. It shouldn’t feel like every man for himself or woman for herself. I hope that we continue to feel like we can build teams intentionally and support people and remain human-centered throughout this process and not just, hey, who can talk to the robots better are the winners. I don’t think that serves us, our companies, our users, our teams, and so on in the long run. But it really is the Wild West right now. Jesse: Emily Campbell, thank you so much for being with us Emily: Thank you for having me. This was so great. Peter: This was fantastic. Thank you. Jesse: If people wanna catch up with you on the internet and find out what’s going on in your brain, how can they do that? Emily: Yeah, absolutely. So I’m on X @elou, E-L-O-U, and increasingly trying to be active there. Now that I have my plugin that hides all the junk, I’m on LinkedIn as well, and sometimes active there. And emilycampbell.co is my personal site, and you can always find me there as well. Jesse: Thanks, Emily. Emily: All right. Thanks so much. Cheers Jesse: For more Finding Our Way, visit findingourway.design for past episodes and transcripts, or follow the show on LinkedIn. Visit petermerholz.com to find Peter’s newsletter, The Merholz Agenda, as well as Design Org Dimensions featuring his latest thinking and the actual tools he uses with clients. If you’re looking for help with AI transformation or you just need a private advisor to help you solve your hardest leadership problems, visit my website at jessejamesgarrett.com to book your free one hour consultation. If you’ve found value in something you’ve heard today, we hope you’ll pass this episode along to someone else who can use it. Thanks for everything you do for others, and thanks so much for listening.
