#192 - Slowing Things Down to Speed Up Research with Jared Forney of Okta

So my strategy with working with legal is like, I try to come as early in the process as I can, even if I'm just coming with scraps. It's like, here is something I can see coming. And here are the things that we anticipate might become problems in three to six months time. Can you help me piece this together? And that builds so much goodwill because it's like, hey, it's fine that you're bringing pieces. That's more than most people come to be with, but you're coming so early that it gives us time to work collaboratively together. Whereas, another partner like more like maybe a product management partner or a technical partner might come more looking for final finished goods, right? You know, they're like, I want to be able to take something and run with it. And so like how those stakeholders come to me at what point in the process, what their kind of idea of a deliverable looks like, that kind of shapes the relationships I have with each of my stakeholders that I work with and how I interface with them. But it all comes down to building that trust over time because that's when when you need to communicate change because change management is a big part of our job too. When you have to communicate change, particularly big change, that's where you that trust becomes the capital that you use to help drive that change. Hey, this is Erin May and this is Carol Gast. And this is awkward silence is. awkward silence is brought to you by user interviews the fastest way to recruit targeted high quality participants for any kind of research. Hello everybody and welcome back to awkward challenges. Today we're here with Jared Forney. Jared is the principal research operations person at Octa has written and talked extensively about research operations and on the cutting edge of what is next, which is what we're going to talk about. We're going to talk about the changing role of research and research ops, which are always changing, but that change is happening real fast right now and in real time. So we're going to talk about the latest and greatest Jared. Thank you so much for being here with us today. Yeah, Erin. Thanks so much. It's a pleasure to be here. Awesome. All right. So let's stick right in and kind of orient the conversation around. There's a lot changing. There's a lot to think about. Of course, we're going to talk about AI, but what are some of the most important areas for research and research ops to think about when it comes to upskilling and evolving right now? Yeah, there's a lot there. So it's a great question to start on. I think, you know, the things that I know I'm thinking about a lot in my team's thinking a lot about our two kind of key functions. You mentioned scale. So scale is really like one of the first ones is how do we grow not only researches impact, but the scope of our work and the scope of the people that we impact. And I think the other one that we really focus on is rigor, you know, especially those two things sometimes feel diametrically opposed because Ed, there's an increasing there's always an increasing demand for research and research insight, which is a good problem to have. But at the same time, they're especially with the all of the new tooling and of course, AI enhancements that are coming with access to research tooling. There's an increasing demand for people outside of traditional research roles who have a desire to do research and research like activities. Kate Cousie famously refer these people as powders, people who do research, PWDRs. The thing that's really most prevalent with that is it's the upskilling is coming from more than just the research team, right? We're having to educate and help guide a new legion of these folks who are doing research activities, right? Who may have had limited exposure to research methodologies before who are not used to conducting research of their own at scale in a certain degree. So that increasingly is the challenge of not only researchers performing the day-to-day work of research itself, you know, whether it's generational, exploratory, the messy stickiness of people in general, but also like helping to enable and I will use the D word early here, democratize research outside of our traditional roles because as we've seen with the AI revolution in general, everyone's roles are blurring together, right? Product roles are blurring with design. Design is blurring with engineering research is blurring with both those fields. So like increasingly we're all being asked to upskill in different ways. Okay, yeah, great. So that's great place to start. You mentioned scale and rigor and I can imagine a sort of two by two, you can imagine there's like an inverse relationship between these. How do you think about them? Like, is it possible to have her cake and eat it too? Can we get more scale and more speed and more rigor or are they always opposing each other? How do you think about the trade-offs there? Yeah, I mean, it is a balance, right? I mean, there's, I think my answer to that would be like, it depends on how big your cake is. I think it really, the balance comes from understanding and having a really fundamental grasp on what your goals and objectives are. I think that's what something that like, is ironic is it sometimes I feel like that gets lost amongst higher levels of stakeholders of like, there's this frenetic energy that comes with all of these AI adoption models where it's like, well, we're already behind if we're not adapting all these tools and doing this and doing that. But sometimes we lose track of the why and to what end, right? And so really what I've thought a lot about at Octan, what like my team has thought a lot about is like getting back to like, what are our underlying goals and purposes for wanting to adopt and use these tools to begin with? Because we're being given an unprecedented amount of access and tooling. That is like something that's like a sea change from even where we were two years ago. The level of access has gone up tremendously. But I think what has been a little bit more of a fast follow and what has been something that's been more of a work in progress organizationally is thinking about, okay, like, what is this going to enable us to do? What objectives is this going to accomplish? So before we even think about how do we make sure we get the quality while scaling at the same time, we have to like fundamentally go back to what our goals are. So how is this going to help us reach out and effectively serve our customers better in what ways, you know, is it about being able to reach out to more participants more effectively? Is it about being able to get more insight per participant because we all know recruiting is very difficult and still in this day at age as a frequent model neck. Is it about being able to streamline the delivery process, right? Being able to put insights in context with where people are, where they're working, those sorts of things. Like, thinking about those underlying goals and working backwards into the tooling stack is really how I think about it. Okay. Yeah. Great. And is that that sort of meta question of why are we even doing this in the first place? Is that something you've been able to, I guess, democratize? And, you know, does that question sit with research with research ops or is that sort of, you know, a behavior that can can be taught? Yeah. I mean, it's something it's, I think it works best, right? When it's a collective understanding, right? Like, when everyone's on board with that, I think most people would agree. When everyone's on board with, why are we doing this? Everything tends to go better. But I do think it's what if I'm speaking particularly to research and research operations, it's a unique time in our field where we have a lot, we have an outsized impact on that decision, right? Because we're so close to the tooling stacks that people are so focused on. But and like internally at, at doctors, to use this as an example, like, we have an entirely new product development life cycle that's really built around the fundamentally the AI enhancements. It is it undergirds all of it. But more than that, it really, organizationally, we stopped and realized that we need more gates in some cases. Because this is moving so quickly in as much as we're automating and allowing for more things to be systematized, more agentic interactions, whether it's like, you know, building product requirement docs or reviews or that sort of thing. At the same time, we realized we need in some cases more human checks than we did before. And so really, that's where the opportunity lies collectively across the product organization is it's a good time to almost stop and pause for a minute. Yeah. Really think about what your process looks like today before you just kind of pour gas, pour gas on the fire. Yeah, slow down the speed up. And that can be a difficult, you've got to be careful, you message that to you, right? Yes. Yeah. And I know that's something you've talked about a lot as well is rethinking kind of who are the stakeholders and departments that you need to be working with in these changing environments in this democratized environment. Can you tell us a little bit more about that? Yeah. I think one of the more fundamental things about research ops that I tell people is it's a currency of trust, right? Like that is really our part and parcel business is about building trust and confidence in the systems that underlie not just research, but product development as a whole, right? And so that is that currency is something that's a little bit hard. It's not as fungible as maybe something like an insight or or something that's like more concrete and quantifiable. But as long as organizations are composed of people and relationships with those people, that is going to be our business, right? And really a lot of what I think about is like how as a research operations person, can I help drive those conversations in directions that will help build trust internally? So you know, when I'm thinking about developing a tool or building confidence between stakeholders, a lot of times when I'm building those relationships, whether it's with legal, whether it's with our product managers or designers, it's always in the back of my mind. I'm like, I'm trying to increase the trust that those stakeholders and those partners have. with not just research jobs, but the trust that they have in the process. So, with legal, legal gets bombarded from every direction, right? I have my heart goes out till a lot of people in legal, because a lot of times when most people I would imagine are interfacing with legal, it's because something's already on fire. Or about to be. Something's about to combust. And so a lot of times, everything is reactionary for them. And so my strategy with working with legal is like, I try to come as early in the process as I can. Sometimes I'm just coming with scraps. That's more than most people come to be with. But you're coming so early that it gives us time to work collaboratively together, right? Whereas another partner like more like maybe a product management partner or a technical partner might come more looking for final finished goods, right? Yeah. And trust, of course, has always been so important with stakeholder management. You know, what have my big learnings doing this podcast for all these years has been the research you do internally is as important, if not more important than their research you do externally, right? So you're always getting to know these folks and what makes them tick. What are their motivations and this might change over time? I'm curious, you know, have any of these observations changed for you with AI, with this speeding up of product development, with democratization? You know, is it, you can imagine there's a temptation, for example, to forego getting ahead of the legal early because we're in such a hurry, but is it perhaps even more important to take the time to do that? Yeah. Yeah. I mean, it's a big change, right? And that's what's like, perhaps been kind of most arresting about like how fast this is moving. It's like even parties and individuals who had normally have been more cautious or more conservative in their deployment. Everything is sped up drastically. It's in the words, kind of surprising. And it takes a little bit of recalibration, like, oh, you're on a completely different gear than you were before because we're all feeling that pressure. Right? I think part of what comes with that in that in recalibrating and recontextualizing that relationship is one like, again, to build that trust where it's like, hey, I know you're under a lot of time pressure. I'm so am I. We both are in the same boat together. And knowing that it's like when you need to move fast on something, again, let's get to the underlying reason behind that. And do we really need to move that fast? Because that's another aspect of this where making especially higher level stakeholders aware of like here are the things that feel urgent, but it's not as urgent as we think it is. And there are of course like, you know, like nobody likes to hear about like, well, the what ifs, right? The consequences of this, right? They want to focus on the outcome, the potential outcome rather than all necessarily all the risks. But I think at the same time, if you help people realize that there are really like to, you know, like as you said, are in like going slow to go fast, right? And it doesn't necessarily mean everything grinding to a halt either. It's just like very considered like in our product development life cycle, very concerted checks at certain points in the process where there's concentrated amounts of human oversight that allow for all the other AI enhancements to happen, right? It's like we're not, we're not substituting one for another. It's an and and really what that brings to the table is this opportunity for not only to slow things down through those gates, but it helps people, it helps people feel validated, right? Like that because that's the obviously the other existential concern is like, the AI is going to take my job, right? And but it's like, no, your role becomes more important than ever as we apply fewer and fewer like as we automate more and more, those concentrated checks where your expertise comes into account becomes exponentially more important. And so it's like to help people understand that and help our stakeholders understand that we're concentrating the human attention and effort now much more than we were before. It was maybe more diffuse and spread throughout the process. Now it's like really focused. Yeah, I feel like it was at last year. Everyone was sort of building these right, AI agent or charts. Like, what was the people and now, you know, we've everyone's using the AI. We're seeing a lot of AI slop and now we're seeing human and loop here and here, right? And really mapping that out. And so you see these kind of corrections adoption correction and this kind of healthy back and forth, I think happening as as AI adoption scales. You know, we're seeing pullbacks in various ways, right? I mean, we were just talking a little bit before this like the first major pullback that I'm sure a lot of organizations are experiencing them like holy cow tokens are expensive. Yeah. We're seeing like, oh, now we have, we all have token budgets and oh my gosh, this is more expensive than people. You know, so there's a lot of like in the broader market, like a lot of pullback first fiscally, right? With that, I think we'll also come a realization that like a collective realization by the market of like there are things that are AI is good at and things that it's really not good at. And that's where again, the specialization and focus of human attention and human action becomes all more important. And it feels like research ops has a very unique role to play in that, right? Because, you know, I'm imagining and this is an oversimplification, but let's say five years ago, research ops were making a lot of templates and documents and revisiting them as processes and methods change, but they have a shelf life of a while. Whereas now, right, it's what we're building AI native systems or but the technology is changing what we're learning about where to put that human in the loop. So how has that changed, I guess, that process of setting these systems and setting these templates and figuring out how to democratize research? Yeah. And you know, what's interesting about that, Aaron, is like in as much as like, yeah, I mean, you know, five years ago, I didn't even say it was two years ago. Sure. It's like, it's really, it's really underscored like in the last 18 months, things are just fundamentally different. Yeah. You know, what's really interesting about that is some of the core motions aren't really all that different. I think that in the other thing I think about when I think about and frame agentic AI in particular is it's really just another stakeholder. It's another user. The only difference is is they're just not human and can operate at scale orders of magnitude more in terms of actions than a single person can take. But the same guard rails, the same types of provisioning actions, the same types of policy and governance guidelines still apply. So really what a lot of what it's done is it's not so much of like, oh, we're not doing these types of activities anymore. We're focused all on these instead. Really it's just a reshaping of the actions we take. So maybe things that might have started as templates and recruitment guides now become prompts, right? It's taking the same skill set and applying it in a different context. I know we've also been experimenting with synthetic users a lot, which I know that's also a term that is very, it's a very hot topic right now and myself included in terms of the skepticism. But it's really pretty incredible to see how you can apply for certain types of like very early stage work, you know, even before you would typically bring something in like an evaluative study where you'd be bringing actual participants in the room, doing it almost like as a pre test sort of thing. It's been a really interesting exercise and understanding not just what the capabilities are, but like how do our current templates are personas and user profiles in our past study work? How can that be leveraged and utilized in this new context and like what are the pros and cons of that? And like again, like how is it serving that process in our product development lifecycle to get us to that end goal of connecting better with our customers? Yeah, this is a great example with we just Nick just shared in the chat. We did a recent report on synthetic users and what's clear is that in many cases the hype is ahead of actual usage, the folks who are ahead in usage are some of the forefront, but then you also have maybe some of those powder who don't have a lot of guardrails who could be maybe not using it in the best way. And so research research shops really do have this unique role with synthetic users with AI technology in general to share best practices, guardrails and keep those up to date by getting hands on with them, right? This is not an ivory tower thing. This is where we're learning too, but we're going to share what we're learning. Yeah, I mean, I think that that's the other really unique opportunity we have. And you mentioned also like doing research internally. That's something that like I come from a research background. So I kind of a lot of people go the other way. They start as research operations and then go into research. I went the other direction. Yeah. I do miss talking to customers every day. Like that is one aspect of my job. I do miss, but like what I found to be fulfillment in that is actually talking internally. And I think it's something that we all don't do often enough because we have stakeholders too, right? We have our customer. our users are our folks internally. And there's just so much we can learn from how our colleagues work in what they care about. And like as you mentioned, just like having this co-learning opportunity, because it's one of these very rare paradigm shifts in the way we work that like everyone has something to contribute, right? And it's kind of a great reset and equalizer in that way, that it allows us all to have kind of an egalitarian voice on certain parts of this experience. And we all can bring different types of expertise to the table. And that's where I think the role of research operations in particular is so central to this, because it's about learning an insight fundamentally, and how again, how we can improve our products and better fulfill our users needs. But it's also an opportunity to all learn a lot more about ourselves and how we work as an organization, because of course, the products that go out in the world are a direct reflection of the internal corporate culture that builds that product. Yeah, but I want to ask just a couple more questions, and then we'll move to the Q&A portion. And so, you know, you mentioned that you went from research to research ops, right? And a lot of fields you're seeing a lot of compression that's got to be one of the words of the year, right? Of different roles, right? A given function can handle more breadth because of AI, because of the need for speed, whatever it might be. How are you seeing that play out in research research ops? Are you seeing these roles blend together? Are you seeing more specialization? Are you seeing them cover into maybe some of the design and product management areas? And what are you seeing there? Yeah, I know just speaking personally, it's got a lot more technical at a hurry. I don't know about you, but I have my first GitHub repo. Oh, yeah, like, I mean, I also like one of the things that I just on that on that note in particular, like I would say, like I wouldn't consider myself get a expert by any means. But like one thing that's what thing that I did early on was I really early on started doing personal projects, right? To get comfortable with the repository and get work. So I had dabbled as a hobbyist in the past, but part of what I find most engaging about what AI is capable of love is like what I think many are calling like the personal software, like building products for users or one of one. I think that's also really a really great way to get used to the workflows and things so that when I went into work, I was already familiar with some of the nomenclature and that really helped. But all that being said, had to get very technical very quickly, adopting a lot more of the basic engineering structure and language, not just to communicate with our engineering teams, but even just to like do the work of automation and engage with these tools because as much as they're accelerating into a consumer-minded direction, these are still fundamentally very technical tools, right? And we're all kind of, especially as we dip in to more and more of this and kind of seed more of the control to these systems. There's a lot that we're executing on that we don't fully understand, right? As as end users of it. So being able to up-level your technical skill, I think is something that's like increased, been increasingly important. I spent more time configuring APIs and things that I ever did previously. But at the same time, I think the other thing that I've had to really sharpen my skills on more is written communication. It's something that you're constantly working on. But I think a lot of people have gotten more sensitive and rightly so to AI-generated written communications. We've all learned it detected. We know what M-Dash-itis looks like. I'm used to use them and know I don't. It's been a plus for me. Right. There's places where it's appropriate. There's places and uses where most people are like, "Okay, it's like a routine systemic communication. It doesn't need to be necessarily hand-tailored." But frequently, we're starting to already see evidence of this of what is deemed a valuable interaction by people is one that is inherently human. One that flaws and all is inherently a person to person interaction. It only underscores why that skill set is so important to be able to clearly communicate an idea, to tell a story, to be able to resonate with people because, again, as long as we are building, as long as people are building software for other people, people will be an essential part of that process. Those are the skills that I think about. Even though they seem kind of diametrically opposed to one another, it is something that's both highly technical and highly human at the same time. Those two things I feel like really go hand in hand in terms of what I've been doing more lately. Yeah, it's really the HCI has never been more HCI. It's like on exponentially so. But then the last thing I wanted to ask you was, you've talked a lot about reconsidering how insights are collected, stored, accessed, and shared. Tell us about that. Yeah. That's how I got my foot in the door in research operations was through knowledge management. I think for a lot of people, that's kind of a common front door. Knowledge management and repositories are always an evergreen topic because we have to store that knowledge, that insight, that collective corporate brain somewhere. I think what's what I'm finding in the evolution of that, we've gone through two different repository systems in the time where dovetail users currently. But I think really what's changing about that is where we might have had single systems or record before, a repository in this monolithic sense of here's the place where insights live and everybody goes to it. With the event of MCP, being able to agentically connect to disparate sources of information to then be very quickly amalgamated into a hole, a repository almost becomes another hook. Another source of information, maybe you've got your product telemetry over here, you've got your repository over here, you've got support tickets here, and then you've got maybe a Slack channel that is coming from AE contacts with your customers over here. All of these things can be united by something like MCP and then it becomes less about the individual sources themselves and more about orchestration or orchestration has been the hot topic in the last couple months. I think what's exciting about that is that's one, that's the stuff I'm really into. That's kind of like why I got into this role is for workflows because the thing I tell every vendor I work with and I'm a broken record about this. So if anyone's heard me talk before you hear this phrase, I always tell them don't sell me a platform, sell me glue. Because what more and more we're moving away from in my view is this platform of theization, these monolithic stores of vast quantities of data that become a single access point and instead it's about how nicely do you play with others. It's more than just about data portability, right? It's about how are you contributing to an organization's workflow as a whole, right? So how well do you integrate? Is it APIs, the MCP, is it data piping in one way or another, web hooks, what have you? That type of orchestration becomes exponentially important now in the age of AI because it's only accelerating. And now when we look at vendors, a lot of it, the first question is, is like, well, do you do these things? Because if you don't, it becomes a harder proposition to be able to like, because the first thing that goes to your mind is manual work. So that's just like, I think the paradigm shift that we're seeing with that too. Yeah, but Jared, how am I going to charge you 100K if I don't call myself a platform? Well, I mean, a lot of times it's like, you know, and I mean, of course, like budgets are always another aspect of this and there are lots of ways to provide value, right? I mean, and there are certain aspects and centralization of things. There's a place for hubs, right? But it's like hub and spoke, where a lot of like, you know, it's not everything we expect to be, you know, floating in the ether, because there are problems with that too, right? If it's too disconnected and too floaty, right, it becomes hard to identify centralization points, like even in our workflow, we have two core pillars, right? We have like our intake and management flow, and then we have our data, data store repository and distribution, right? So like, you know, it's not all just this massive web of interconnectedness, but you know, like the more integrations you have, of course, the more points of failure you have to. So a lot of what you have to balance is like identifying how many sources do you want? And again, what underlying goals are you supporting, right? Like if your organization works better in a single hub, that may work great for you, you know, I mean, your organization, depending on what their needs and goals are. So it's just something I'm observing more broadly, but to your point, like there is a time and place for centralization of certain aspects, certainly with participant management as one. -Oquard interruption. This episode of awkward silences, like every episode of awkward silences, is brought to you by user interviews. We know that finding participants for research is hard. User interviews is the fastest way to recruit targeted, high quality participants for any kind of research. We're not a testing platform. Instead, we're fully focused on making sure you can get just in time insights for your product development, business strategy, marketing, and more. Go to userinterviews.com/offward to get your first three participants free. -I think it'll be interesting, too, to just see how the bundling and unbundling and rebundling will play out with all of this, too, right? Because we're in, I guess you would call it an unbundling of data sources sort of plugging in, right? And being portable, but. -The season of SKUs. -The season of SKUs. But anyway, yeah, I'm curious how much you mentioned orchestration. The other word that comes to mind for me is sort of like algorithm, building and what I mean by that, you've got 20 sources of truth for information now, which gets primacy, who builds that and where it's access. Is that the kind of work you're doing and is that very different than the work you used to do when you think about managing knowledge? Yeah. I mean, it's definitely a challenge because it's like, how do you decide what is canonical, right? Because all, you know, a key aspect of knowledge management is what I've heard affectability called data gardening, right? We have to make sure that things are pruned and neat and old studies are deprecated and like news, you know, new studies are balanced with work that has like long tail significance, right? So a lot of that comes with like a re-examination of like what is kind of the effective half life of certain methodologies, right? We look at things that are canonical generative studies, things that are like foundational research, things that inform like our personas, for example, our buyer and customer personas, there's more, maybe more foundational work tends to change a little bit more slowly. There's something like a usability study, which sometimes is more of a flash in the pan and like, you know, has immediate value, but then it dissipates quickly as things are iterated on. So, you know, there's a, there's a balance, a rebalancing that happens with that, especially when some of these systems through MCP don't always have the ability to differentiate between that. They're not able to make that distinction necessarily. And so what becomes important is not only what we're putting into our repositories and these data stores, but also like you said, what primacy and value we're assigning to it is something that like is actively being discussed right now. There's, there's ways we're talking about doing that through taxonomy. There's ways that that are a little bit more blunt for like blunt for it's like is it doesn't even get included at all, right? Is it something that lives outside of a repository simply because its half life is so short? So that's something that's very top of mind right now. Yeah. Okay. Yeah. Great. All right. Let's take some audience questions. We'll start with one here from your friend and mine, Caleb Lucifer. What was the single best tactic and argument that successfully got your team to slow down and invest in governance? Yes. Thank you, Caleb, for the question. And thank you to everyone who submitted questions. I'm looking forward to answering more of these. Ah, slowing down. The biggest thing I think is is really that focus on rigour and quality, right? That's something that's like a collective discussion that's happening, particularly as we scale. There's an understanding that more research faster doesn't mean it's good research, right? There doesn't mean it's it's contributing in a way that necessarily is helping us move forward. And so a lot of what we're doing to encourage that governance. And then this is like where our research team kind of led by example, like I'm really proud of how our team did this when, when this AI adoption really first started to accelerate internally within our company, we realized like we need kind of a central statement, a set of principles and a way to plan our flag in the ground knowing we're planting it on shifting sand, but like we got to start somewhere and especially before it's decided for us. And so a lot of it came down to like, here's how we define quality. Here's how we disclose AI usage. Here's when when we do adopt it, this is how we use it. And this is how we don't use it. And I think by setting those principles and socializing them, that made the case for, ooh, I never thought about that. Like from an organizational standpoint, it's like, that's a really good point about we should necessarily take things at face value or, you know, we should narrow our scope and maybe focus on small language models that are only citing our research rather than using broader LLM's that are calling outside of our organization and using outside sources that maybe are a little bit more of a black box. The way we kind of impose that or suggested that sense of governance wasn't even necessarily by saying slow down. It's just like, here's how we're moving, right? We're like really intentionally showing like, yes, we're crawling, walking and running, but we're really over disclosing. We're being really transparent in how we're taking each of those steps as we accelerate. Right. And like you were sort of mentioning before, aggregate, everything is without a doubt moving faster, but not everything needs to move out of breakneck pace for that to be true and just being really intentional about that. And I also love that you took the time to define what quality means because I think that's one of those things that can be so hand-wavy right? It's like, we have to be rigorous. We have to have quality, whatever it might be, but if you actually take the time to socialize what we mean by that, people will be like, well, yeah, of course we want to do that. Yeah. I mean, it's like, yeah, share definitions are so important because there's so much flying around that like just even taking the time to come up with a universally set agreed upon set of definitions, just that exercise alone. I feel like it's so healthy for organizations to have. I can't even imagine like the ability to just like, I can't say we had a unifying workshop in that way, but looking back, it's something I maybe wish I would have done. It's like, get all these people to room for like an hour or two and just like come out with a agreed set upon definitions. We did it more incrementally, but like we got there all the same and I think that's what's really helped everybody start to like, even as things are flying around, there's chaos sometimes, we can always we have the set of uniform definitions and checks we can return to. Yeah, that's great. Okay, we've got a zoomed out question here from Natasha Duncan at Boots user researcher. The question is, any tips for getting buy-in for research operations in general and getting the time to do the tasks? Yeah. So I'm going to make maybe an assumption here. So Natasha, I'm sorry if I'm making an assumption that maybe you're on a smaller team or possibly a team of one, but if you're not, that's great, but this, hopefully this will still help you. I think that the argument that I was in this boat too, it's like, how do you make the case for doing research operations activities when there's so many other pressures, so much other research to be done? The argument that I used and when I realized I couldn't do both jobs well at the same time is making your organization aware of the fact that this work is going to get done either way. Like the operations work has to be done. And the question is, who's doing it and what is it preventing them from doing instead? So when there's not a dedicated operations person in place, the first sign that you need one is that your researchers are doing it, right? So they're doing those operational tasks, whether it's participant coordination, incentive distribution, knowledge management, procurement, all those tasks, and they're not doing research, right? They're not talking to customers. They're not distributing and sharing insight. They're not helping inform product decisions in that way. That operations work is going to happen, whether or not you have a dedicated person. And the case I made to my organization is like, what do you want our researchers doing more? And it was a pretty easy answer for them. We want them doing more research, of course. I'm like, well, then like this is the case of where if you have someone who's dedicated at those tasks, a couple of things happen. One, obviously they do, they have more time to focus on the customer experience. And then the second thing that happens is it unlocks all of these potential process improvements that acts as a flywheel on the rest of the research process, right? And then you can start talking at things about like now we're very fortunate in the fact that we made what started as a research ops team of one, just me. We now are like getting ready to hire our fourth person. And what that's what's really exciting about that is one, it shows confidence in the organization that this is a role that's valuable. But the other thing is it's allowing our ops people to start to specialize. So we have someone who's just focused on democratization and socialization or excuse me, democratization in general and then research enablement. We have someone who's just focused on our beta programs, which is like turning into this huge aspect of our research experience and product development lifecycle. And we have another person who's like focusing more just on the participant side. And so I think really what that specialization starts to allow you to do is like, it builds the case for being able to further scale and you talk about force multiplier for being able to expand your research operation and your and really just the ability to learn as an organization. Yeah. Yeah. So researchers are going to get to do more of what they're supposed to do research. The ops stuff they have to do, they know all you have to do, but you're also going to make that stuff faster and more efficient, which researchers of course don't have time to do to think about those systems, right? It's blocking and tackling and then you can specialize and force multiply all that stuff. I do think it's the golden arrow for operations of all kinds with particularly because of AI and the need to democratize and the technology and all the systems. I'm curious at, you know, I'm marketing landers. We've got GTM engineers. We have content engineers. Now everyone's an engineer. Has this been to research yet? Or are we still? I think we're getting there, right? I mean, it's like this is what I always like say that like so so much of research operations borrows fundamentals from design operations with borrowed fundamentals from DevOps, right? So like we're all inheriting language and rituals and things from organizations or from functions that came before us and are still exist today, obviously. But what's different is how we're leveraging those traditions, right? And how we're putting our unique flavor on some of those rituals because it is a different function, right? You know, I've seen some talk from peers in the industry of like, yeah, I'm seeing a blurring between design, design ops and research ops and like, how do I make the case that this is a separate function? And it really comes down to, again, like your organization size can have an impact here. Like I could see a world in which sometimes someone might be like kind of forced to straddle a little bit, but it really comes down to like, who is your primary audience that you're enabling first and foremost, right? So for like a design operations person, it's designers, right? It's in the name dev ops, it's developers, right? You know, with research operations, first and foremost, my core stakeholders on my research team, those are the people who I am first and foremost responsible to. And then we start talking about spheres of influence outside of that and really like being able to make the case for that separate role really comes with that maturity curve of your organization. Like where do you, that's a great place to, that great question to ask yourself if you're trying to make the case for research ops 2 is where do you think your organization is on that, on that research maturity curve, right? And just because you're not far along, if you find you're like in the earlier stages of that, that doesn't mean that there's not still an opportunity for research operations, it's just your framing is different. It's more about maybe it's more tactical in nature, right? Then it is strategic. The goal, I think for most people and where I know I want to be, is more of a strategic function, right? Because it's about thinking for the long term. But like there's still a lot of pragmatic stuff I do every day. I joke this role is like at equal times 30,000 feet and three inches. So there could be a lot of thrash that comes with that, but it's a really important part of the function is being able to think at that high strategic level and also work on the nitty gritty in the same day. Awesome. All right. Our next question comes from Aina who is a UXR. How can Reop's teams enable more faster research when timelines are more compressed than ever? Yeah. This is the seminal question of our time, right? You know, I think it really comes down to like redefining what fast means and like redefining what delivery means in that context because and this just comes again, I know I keep cycling back to like what are your goals, right? What is your organization's goals with that research? What's driving that need for quick insight? Is it a go-to-market launch? Is it some external pressure? Is it a competitor? Is it something about an opportunity that we're seeing in the market? You know, there's all sorts of underlying causes that drive that frenetic need for speed and by starting to understand that, that's where I start to work backwards to like methodologies and then to and then detooling and and processes and things like that from there because there's a lot of value that you can get obviously from like, you know, there's a million usability tools out there that'll promise results in an hour, right? You can get something that resembles insight or research in that short period of time, but it often turns to be kind of fluff, right? It's something that may not and a fluff at best and at worst it's dangerous in the sense that it is something that masquerades is insight that could be then taken and run with and applied to the company's detriment. That's the real that that's one of the dangers of AI, right? Is it still convincing whether it's through hallucinations and and through the like faux detail it provides? It could be really dangerous for things in the name of speed to take and run with. So a lot of what I try to focus on with teams like that when they're asking when they're facing pressures for speed is like, I have a philosophy, I think about a of 150, 15 and really it's like, what is 100% of this process implemented look like? Okay, now cut that in half. Okay, now cut that to 15% of what it was. You know, it's like the classic MVP model, but applied to your own research process and methodology. Use is like, what is that 15% version of a methodology? Like what does a 15% generative study look like? You know what I mean? If you're trimmed down, if you have to boil it to down the essentials, what does it look like? Are there opportunities where okay, maybe you have to make trade offs with participants. Maybe you're doing a little bit more internal research than external. If you're talking about participants, maybe it's you're doing slightly smaller sample sizes. Maybe you're using AI tooling to help speed up the synthesis where it would have been a little more hands on and more tax on only focus before and there are trade offs with all of that. But the idea being so much of what our operations function is in the service of speed and efficiency, it's about building trust. And how can I use my methodologies and process and tooling to not only help enable our teams to move faster, but to deliver trust, even in smaller pieces, that'll give my stakeholders the confidence to give us more time and space to move from that 15 to that 50 to that 100% implementation of that process. All right. I think we have time for two more questions if that's all right with you. Okay. Here we go. That is a few sentences here. So bear with me. This is from Kittest and Kittest is. I don't know if I have your job here. You are a former UX. Okay. We often hear how AI agents can help streamline our workflows and save us time on the most minute of research tests. But I want to know what have been some of the challenges and limitations of using AI in your work. What are the things we should know and set up guard rails for? Yeah. That's a great question. I think the limitations, right? I think that the first and foremost, when we hear the most is hallucination, right? It's the overconfidence that comes with being able to take something out of context, a very AI system that takes something out of context, over-prescribed weight, being able to generate really convincing looking, but function factually inaccurate insight, right? And so it's like that is one of the risks that we have that we deal with is maintaining that quality. I think another aspect that we have to be careful of in that same vein, especially as we talk about agent to AI, is so much of it gets upleveled early in the problem space defining process. So like, think about like a product requirements stock, right? Like fundamentally, that's a template, right? Like we do product requirements, stocks over like PM and other parts of product, they write those docs over and over again. And they follow the generally the same format. It's a template. And so templates are a great avenue for agentic generation because we can start and spin up something as a framework to work off of from there. But as, and we've even started to see some evidence of this too, it's like, oh, we can do these PR docs, generations, and then we can start pulling in sources from what we know already and injecting them into this document to then push out something that's like, hey, this is ready for review. Let's do it. And part of the risk that we see with that is, okay, it's great that you're like starting in the template and you're plugging in from all these sources to help inform this document. But if nobody's checking where those agents, those agent hooks are coming from, what prompts they're using and it's just being dropped in and then taken verbatim as truth, that's where the risk is. And so a big part of like the automation and templatization, a big part of what we do and what our responsibility is is to make sure there's good fundamentals in those templates before, right? So that's the prompts themselves, the MCP hooks that we have into the tooling, how those hooks and prompts are designed. So when it's ingesting the tools of doing it within a certain set of rules and frameworks. And then also, I think part of our job is education too. I often think about, you know, the importance of building good fundamentals. I think that so many people are in this rush to learn new things quickly. There's a lot of fear that people are going to be left behind. And so we just adopt things and run with them because it's like, we don't want to be the person who it wants to take the time to learn something more thoroughly because we're just in such a hurry to demonstrate results. And so I think really where we have an opportunity operationally as someone who's like watching all the pieces fit together and including when they don't fit together well, we have the opportunity to raise our hands and say like, we all need to take time to explore and learn together and make mistakes together. Right? Like, that's something that I think we really need to be able to have the space to do more of as we experiment is we need to learn the we have the space and opportunity to make mistakes in a controlled way. And so as researchers and things, that's where the guard rails, that's what I'm really trying to consciously put more guard rails around is like creating that space for mistakes to happen in a safe and controlled way. Because when we were able to learn from those and demonstrate like, hey, look at this, look how crazy this, look at this crazy thing that happened in this wall garden. Like what has that got out there? That'd be bad, right? Like it gives everybody a chance to slow down and give themselves the permission to be a little bit more thoughtful, to take the time to like learn and apply better prompts, better systems and, you know, ultimately better processes. That's great. And I think that's so needed. And, you know, there are so many mandates to learn AI and use AI and just having those support systems, whether it be dedicated AI, helper operations, help, but providing that safety that time, those tools. So folks can learn because learning isn't supposed to be fast, right? It's messy and hard and full of failure and all those things and absolutely. So that's great to hear. All right. Last question. Any tips on getting into the UXR reop space? Has the new camera with minimal experience? This is from Rituzia. Yeah. So I think this is just something that like if you're coming in as like a new grad, you know, I think it's one like I have my hard goes out to you. It's a tough world out there. It's a really tough time to be in the market right now and you know even just as someone who is also part of the hiring tracks It's unprecedented right I would I I feel like I would struggle in this new world that we're in so by Like hard goes out to anyone who's in this role right now and I I talk to and mentor a lot of people in this space too and I think the things that I would say if you're not coming from a research background I think a lot of people are kind of fret that it's like oh, I just don't have this background I'm worried that like it won't be applicable and like I the thing I tell people is that your life experience Every experience you have can apply to this field I want to example I'll talk to talk about briefly as I I talked to the guy who was looking to get into the research base and he didn't have any design Experiencer background in design and he was trying to like you know move and take that next step and I'm like Well, what did you do before this and he's like well? I was like a metal worker and like you know and I did this that and the other and I'm like What do you mean you don't have design experience like you work with your hands and a physical medium with all sorts of constraints and techniques and all that stuff is absolutely applicable You just have to recontextualize it in this new world that you're in and I think that's a really important message That I have for a lot of people who are looking to go into research and even research operations is that you have more Experience than you think you do the important part of what the and the hard work comes from Translating that and convincing others that that experience really counts for something and that you bring a unique perspective to the table Because like I didn't start in research operations I actually started as a designer researcher hybrid. I've gone through three Career changes in my time here at Octa been here nine years and I've changed roles three times and so like each of those jumps I had to convince my manager or my manager's manager that like this was something that was not only important for my career development But was also going to be a force multiplier for the organization, right? It's like by helping me make this transition in this role here is what I'm going to be able to do and That case is something that takes time to make and you have to you know have the right You know stakeholders and things to help you through that process, but ultimately When it comes to moving into research operations a big part of what I say is like if not me who? Right and what are we giving up by not by it by having somebody else do this? That's not centralized that doesn't have the context that's distributed throughout the organization that's scattered in siloed Right That's how I often tell people to frame that conversation if they're starting to move into research Ops or make a case for that first tire Yeah, I'll tell you what to and to your point. There's a difficult environment no doubt about that, but I think Experience arguably counted more than it does right now where everyone's having to reinvent themselves And it's really that growth mindset that willingness to learn that passion for the craft That's always counted, but I think that'll take you a long way now too as well as demonstrating Learning some of these AI skills and things like that that not all the people who've been around do have or have taken that time to learn Right, and like even the AI stuff that we're learning now in 18 months who knows if it's even going to be relevant anymore Like the world master Right, I mean, it's like to your point Aaron like really it's like the passion for learning and development and that that that counts for a lot Right like being adaptive and being able to like shift gears and being well being leveled to you know dive headfirst into ambiguity And a new space and being comfortable with being uncomfortable. Yeah, it's a really important skill set and something that is Sometimes shows up as an intangible, you know on a resume But you know, it's something that you really I think when you can speak to it and tie it to experience and of course tie it to impact because of the you know in the In the process that's ultimately what everybody it like is looking for when they're doing hiring at least at the top level But when you get in a room and I've seen this like with with people I've mentored when you get in a room the ability to tie that story into that Willing this in passion to learn and grow is the thing that Ultimately gets you to where you need to be Well, awesome. Jared that takes us just about this time just to close work and focus find you or are you active on LinkedIn or extra? Yeah, so I'm on LinkedIn feel free to connect with me on LinkedIn I also have my own personal site at jerd4ny.com where I do Riding's and musings every once in a while Those are the part of the two main places you can find me I'm always happy to try it where I can like chat with folks I love talking people in the industry. I was just at the Reops rally around re at research ops last week, which is a really great conference Hopefully we'll see some some things come out of that So I try to do some networking around but please come say hi to me Connect to me. I linked in and I'm you know If I have the opportunity and time I love to chat with people in the industry because it's just a really exciting time to To be in our field And I think it's there's no better time like the president for research operations professionals in particular Yeah, you're here. Well, thank you Jared so much for being here. Thanks to our audience for joining Anyone if you can get to your question will try to respond in our follow up Thank you everyone have a great rest of your day. Thank you You Thanks for listening to awkward silence is brought to you by user interviews theme music by fragile gang Hey there awkward silence is listener. Thanks for listening If you like what you heard we always appreciate a rating or review on your podcast app with choice We'd also love to hear from you with feedback guest topics or ideas so that we can improve your podcast listening experience We're running a quick survey so you can share your thoughts on what you like about the show Which episodes you like best which subjects you like to hear more about which stuff you're sick of and more just about you The fans have kept us on the air for the past five years We know surveys usually suck seat episode 21 with Erica Hall for more on that But this one's quick and useful we promise Thanks for helping us make this the best podcast it can be You can find the survey link in the episode description of any episode or head on over to use our interviews calm slash awkward survey