Ep 95: When Docs Becomes a Liability

I'm sitting upstairs. I hear the sound. [humming] I have no. Nothing. Nothing. You're. For the podcast listeners, he's doing gestures and it's not helping at all. Is it macarena? It's. It's. It could be macarena. I'm trying to hum the Gabby Dollhouse, part of the Gabby Dollhouse theme song. See? You know that one too. You just blocked your mouth. I hate. Thank you for bringing back trauma. I hate that show. Just cannot stand. Not saying left and a pinch on the right. Oh, Penny's hands and hold on tight. First off, cats are objectively a spawn of hell. Demons with fur that walk on four legs. I guess we're putting it out in the cat lovers. DJ and I are dog people. Yeah, sorry, cat lovers. Just gonna have to agree to disagree on this one. That means that it's already run to a deficit from the get-go. And then it just goes downhill from there in terms of everything else. It's come to mind that I feel like the difference between dog people and cat people are sort of the same between hamburger lovers versus chicken sandwich lovers. Really? I feel like it. Okay. There's like a very nuanced niche chicken sandwich loving group. Okay. Okay. That's where I buy it. I don't like the hamburgers. I don't like the beef. Make sure. Loaded. Sure. I am a hamburger. Kind of sour. Smash burger. Large hamburgers. Maybe multiple burgers. Maybe multiple buns. At once or. Sure. Damn. Okay. More buns. Okay. I mean, I. I don't have skin in the game in the hamburger versus chicken burger. All right. I just don't like cats. They sometimes they call me the big Mac of the size of something. You know, in certain circumstances. They do. I've heard people call you that. All in terribly, not prompted. Yes. It's the strangest thing calling someone a burger. I don't get it. But, hey. Welcome back to the size of it's about soures where you have burger lovers, dog lovers, and somehow cat lovers. And some. Let me preface also that I feel like I'm insanely allergic. I'm allergic to a dander and it's gotten worse as I've gotten older. So if I do go to a house and folks are with their cats that are just roaming freely and don't lock a vacuum or a lint roller, I am not permitted to stay in that house. My wife's the same way. She's allergic to cats. I'm allergic, but just more mentally allergic to cats. Well, there's folks allergic to cats and then there's folks allergic to writing, right, PJ? That's true. There's a lot more people allergic to writing than allergic to cats. That's for sure. So one topic that we always love to talk about is writing. And either writing for this documentation sake, writing blogs, documenting decisions. I think that's along the lines of where we like to try to capture this. And it makes me think of when we were working on the Disney, Disney+ side system, or maybe even say like the Disney+ templates components, whatever, we ended up calling that. UIKit, CapA, lowercaseD design system, I don't feel we did any documentation, which is kind of wild. A lot of the documentation from that sense lived in Gira. So it was a different flavor of documentation. It was more spectraven documentation, right? Not usage guidelines. There's -- Yeah. So it's documentation for the person implementing it to consume it once and then it goes away. Exactly. Right. It's not documentation for everyone to consume. Yeah. So the audience of that is much smaller. Well, that's important also. One thing that I think sort of brought this topic to mind is ad-alacian, especially -- or any system that's been around for a while, maybe the say three plus years. There's documentation on the top side and on the wikis that's evidently become stale. One thing that I've noticed is we have done an excellent job at documenting things. So while we release features, we have usually an internal blog post that is like a work in progress. This is where we're going. We have comms through each phase. So whether it's like an early access or beta or release of the general public, those blog posts and those wiki pages simply don't get come back to. So we're really good at generating this stuff, but we do a really shit job at maintaining the docs. And one thing that's sort of like the pain of my existence at work is while I've been tuning an agent to handle design wiki documentation and a lot of more design-observated things, I can control what knowledge is available, but even beyond that, any designer was just using an agent to crawl and do like a simple search. It doesn't have any sort of sense on quality of documentation, staleness. And like you would mention at some point, at this point right now, it becomes a liability because folks are getting older documentation that may have more reactions, more views, and the LLM doesn't know what is valid or not. It doesn't know anything. That's part of the problem, right? So if a human being can look at the documentation and be able to tell, okay, wait, these things are not the same anymore. This is way off and start to use common sense to know, okay, given how stale this is, it probably isn't something that I can trust 100%, or at least need to start asking some questions. Agents don't necessarily know that. And so you're putting yourself at risk because it's going to take that documentation seriously, which it should, and it can't necessarily determine whether it should be skeptical of what it's given. So that's the problem. I mean, theoretically, you could add to the prompt, you know, check when the date that this documentation was updated compared to the component that it refers to. If the date is, you know, if there's the delta is more than X months, you don't read the documentation, like prompt for help. I don't know what that next step is, but have something where it's at least taking a pause before it starts moving a million miles per hour based off of documentation that may not be up to date. Yeah, one thing that I saw on Meta's Wiki implementation and Meta, it was much more scattered. And I think you could probably illustrate what this was on different organizations too well. There was a Wiki that mainly was developer docs. They would publish stuff on there. Design documentation. For whatever reason, we started in Wiki and then we never disobeyed it through, like it was sort of fizzles away. And then there's the internal design system documentation. So those are like the three sort of places that things I get put in. The Wiki had a freshness score very similar to, where did we last see this? Was it, I want to say dig, but it wasn't dig. I just saw something from Kevin Rose, so I thought dig. But there was another app back in the day that had freshness score, staleness, that sort of thing. Maybe those are app abstract or quick or not. But I think that would be extremely useful if there was just a marker. And on the Atlassian, like Confluence Pages, there is a, like a widget that you could put that is called the document control. You could slap onto your Confluence Pages and then at timestamps it says, oh, this was human verified at this date. And then it runs on like a 90 day cycle to be human verified again. That's one of the mechanisms I want to add. But with human intervention, there is no automation for me to add these in pages. I have to go while I'm going through these design Wiki pages. I'm adding this controller and then also validating the fork that. So I think as much as we're putting our eggs in AI crawling data, I think this human sort of oversight of this is still extremely important. There was an AI tool that I used at InstaCart, which would essentially index all of your internal information, your Wiki docs and whatnot, Google docs, all that sort of information. And if you submitted it, if you added it to the system, you would have to verify it. It would prompt you to re-verify things over a certain amount of time. So you could verify this for six months, verify this for 90 days. And then once that time runs out, you have to manually go through and verify those things again. I think while that's more laborious than just a staleness, I think that makes a lot more sense because information isn't bred. And it doesn't necessarily just get stale because it got older like bread does. It might be just as valid five years from now as it was from the day it was made. And so I think using time is probably decent for many documents, but not for all. Your color system may not change all that much over time. And it may stay the same whereas a button component that's changing every release cycle may not necessarily all that same pattern. So that's what gets difficult about using something like time to judge whether something is still valid. I do think it requires someone to make a judgment on whether this, whether documentation is up to date and just year nay. This is stale or it's not stale. So yeah, obviously this is Davy suggesting fix the thons again. One of the things that I think I was trying to do for this this quarter is rally folks around a like a docs fix a dollar or bug bash and I think obviously for something that is more dead related. I don't think that I would easily get a commitment for like a week of people's time. So I was trying to figure out let me write a plan to see if I could get one day's time. And then I've even focused that even like more magnet because if it's one day's time and there's some of us in the US, there's some of us in Australia. I can't really oversee and make sure that folks are getting what they need done. So potentially even just a one hour 90 minute bug bash session where we go and a lot of the stuff you do is just it could just be like our development docs for instance. I've graded them based on staleness and then based on whether like the person that wrote it is still here. That's like another mechanism. So that may influence whether something has changed. But I have no insight on the development runbooks or playbooks that we have whether they're valid. We need someone to go in and take a few minutes to to scrub through it and say yeah, you're name. And you know, there's ways to circumvent some of those situations such as you could theoretically manually write the API documentation for a component, you know, all the props and the accepted values, whether it's required or not, or you can have that documentation generated from the code and and have that be the source of truth that's constantly being updated when the API changes. So some of these things can be automated and I encourage us to do that whenever possible because it reduces the staleness of oh, great. We updated the component but we forgot to update the docs. Now we're providing bank information in the documentation. That's practices. It gets a little bit harder to do. But I do think I think for some things you can automate that and that helps blunt the code or reduce the lift for keeping these things up to date. Yeah, another popular thing that I think that I've done now in numerous systems is as we implement new theming going back to the site and then just re-alputing the images like obviously that is should be easy enough right if the component is themed then just to go back to the file then to export. But another thing is also docsite specifically may not have a specific owner or a pointer contact that owns that and for the long run. Where do those images or those staged is it like if I'm updating a label component and I then also updating the docs in my label component. Sigma file or is there a canonical library source that is for the docsite like a publishing kit. We ideally you're rendering out the code if it's web-based. Once it's mobile it becomes a whole other all of wax. One other approach that we've noodled with but never got the chance to implement was to have a canonical Sigma file that has visuals for things that can't be rendered in code that adopt the current version of the design system. You keep that file up to date and then essentially use the rest API to extract images from all those visuals to code. So you're making requests for the latest version of that image assuming that you're keeping your update in the library for that or hell you could even just have a B in your Figma library that the public library that all the designers use so it's always up to date. You just have a section for documentation visuals as those components update the visuals update because they're inheriting those components and then the rest API is extracting those images saving it to disk on the server and then using those for the latest build. That would be great because then once component updates in the library it updates on the documentation. What I'm extracting from that is you could also then have special backgrounds and frames and nice rounded corners or whatever that you liked to put the component on as long as the component in Figma is instance this thing could run a script right then say I need this to be extracted. That's a great idea. I love it. You know theoretically there's a GitHub action or something where when it when it merge when a PR has been merged and the documentation updates assuming the documentation updates on a PR merge. One of the actions is that it says okay Figma file. I'm going to request all these images by the rest API save those to disk somewhere could be an s3 bucket could be whatever whatever you used to render out images and then and then you're making sure that you're not just automating the documentation based off the latest code you're also automating the updating of assets. I'm sure others feel the same way. I hate updating images for the docs site. Well for my use cases unique where like I told you my docs site is on the monorepo so updating images is a huge like pain in the pain in the bum. And I hate having to go in having to then look for my look for that that docs sites kit place it in there make sure that whatever image is on the whatever overrides are on the docs sites trying to replicate it so it doesn't look too different and then having to export it and all that sort of stuff that just seems like it's like 2002 and I'm. It's a pain in the butt empty being this thing it's a pain in the butt and I think anything anything to remove manual tasks to keep things up to date is ideal in my in my opinion at least so that's that's what I'm always aiming for. If and when visuals need to be made is there is a tool chain that we can use to just have it be pulled straight from the source. That brings up like other pieces of documentation so I think we were very healthy indexed on. The design system documentation so for us are ours is publicly available but there's other pieces that tell more of a comprehensive story that I found has been really good for finding out information on like nuanced decisions for instance or the why should I care that this thing was updated and we tend to write a lot of internal posts the just as as comms and most of the time like when something is released it's usually like what's changed I have a before and after why should I care what breaking changes may be available so it's more or less like a very in-depth release note with a lot of visuals I could see this automation working for those images as well so like having like a blog post like kit and then being able to place place things on there and then through the course of my project if I'm releasing an article to tell people that I'm starting to work you could just sort of have this this job run and export me new images as well. It'd be great if if there were not just saved versions within Figma but actual Seembird versions which you could then theoretically with the rest API request images for nodes on a specific version because then you could easily do it before after here was 1.63 of the button here's 1.64 on the button you can show visual diffs between it that is a it's easy for me to say that building that is a substantial feature for something like Figma but I that would be the ideal I mean you theoretically could do that with if you you based off of the name of the save version you could save a history of the images and then basically reference a prior version of that image asset to show the difference but that would that's a pretty sophisticated process in and of itself as well it just depends on how important that is and how automated you want that to be versus just manually grinding through all right here's the old version of it manually creating image for that and here's the latest version of it. Yeah I mean the the comms around like what we're doing and how we're getting design decisions like I had thought that also the other piece of documentation that I haven't seen really done that often is more of a like design language visual design change log or more of a like decision log I don't remember if I've mentioned that like our change log is typically this changed a pj this is changed yeah baby why is it changed oh you should go ask so and so but we should ideally have a decision log for when we decide to modify colors to make them lighter or darker whether we make things more rounder it seems like that's the trend now we're making shit pills now right rounded completely yeah why when did we do this who decided this and when what presentation or blog or additional documentation could we link this to so I think that's important just to to be able to go into the way back machine and explain why something happened I don't know if a lot of companies would would rush at the idea of building out a whole process specifically for that but where I do think it could be valuable is not just within the context of a design system but in terms of experimentation where man I cannot tell you how many times at numerous companies someone comes in understandably so and says oh I I know I know how to fix everything we just need to do this like yeah yeah we've tried that like 12 friggin times we've already tried it right but there's no documentation of that it's just word of mouth right and so it might not be 12 times it might be once it may be 120 times having that log of you know this this thing change for this reason because of xyz and as much context as possible theoretically something like an agent can you know someone can come in and say I want to run this experiment for this purpose and an agent could say these things have been done that are related to it here's what succeeded here's what hasn't here's prior decisions based off of of what we've learned which just is very hard to find if if it exists at all at a lot of companies other than word of mouth if that person's still there helpful for a design system but I think also just helpful for companies that are very you know that are experimentation driven and and just you know growth focus just want to try anything and see what works a lot of times trying all those things aren't documented so you're repeating yourself over and over and over again which makes no sense yeah I'm also of the minds that with all the tech these days so obviously like where I'm drinking the Kool-Aid I record our meetings with loom but outside of loom you could record it with zoom also that that's available if your info sec team approves and there's no reason in my mind why team meetings or any meeting that you're discussing design decisions is not recorded and rockable so that's another mechanism like I know the decision log is like quite a bit to maintain but I'm just thinking of a world where I have a brief and then I also have updates like weekly updates that I that I provide if there's a specific point of a meeting that I want other design managers to make sure to see and head nod for me I'll post that snippet and if it's recorded you could type stamp it obviously yeah all the that's one area where AI has been pretty damn effective of being able to transcribe meetings summarized decently in every once well there's a hallucination no doubt but I would argue people hallucinate as well they remember things in that meeting that never happened right and so so I think it's it's a little bit it's a little bit it's a little murky that we're arguing that I found a lot of value and something like that so and especially if there's if if a company is experiencing a lot of attrition a lot of turnover then all of a sudden you're not losing that information as it as people move on so that you know when someone does say well I want to try this experiment and there's no one there to say no we've tried that a bazillion times and so it can theoretically address you know kind of blunt that issue before it happens yeah there's I mean there's like this sort of this back by mind thing that's just someone's going to ask me about design decision and I need some mechanism to say oh we talked about it in this stand up or talked about it this week and it was related to this thing and it's a lot of times this is trying to satisfy the that's the the query and it's like okay seems good seems like it's yeah it's thought yeah but you're right if someone like especially someone like I'm I've only been here for about seven months and there's times where I ask about have you thought about doing this and then you know designers would say oh we've tried this xyz times and they will give me reasons why it's why they decided a certain way which is which is good but those reasons aren't set in stone anywhere those aren't in on a wiki those aren't on a page so it's it's all it's sort of one of those things that's at the new person as someone tells me decisions I want to make sure to capture that in a doc to say like oh this this person gave this commentary let's see if we could dig into some user research that that was around that time or check on our help channels or dig into it on future future research so it's it's the what is that like the cover your ass right CYA I would be really interested if a company would be would measure the time it takes to capture those decisions and document it right the the full process of okay decisions been made we document it we publish it versus the time it takes for someone to try to uncover that information through word of mouth and how many people they have to talk to and what the what the general delay is in in getting an answer what's the delta there is it is it faster just to ask someone I'm gonna bet the farm that it's not that there's a lot of hidden cost associated with you know you're really focused on something Davy you're in the zone I I bother you on slack break your concentration you don't even know what the answer is okay I go talk to someone else they don't know talk to this person now you know four people deep theoretically interrupted four people's focus to get an answer does that net is that a net positive versus the time it takes to document and have people access I don't know I would again bet the farm that it would be much more efficient just to take the time pro proactively to document those things but I think that's what it would take for companies to really support something like this because it seems like there's a an allergic reaction to spending time on procedure and documentation for a lot of these sorts of things yeah it's also like a co-location thing to like where obviously I'm or or remote most of my team is not here they're in a different time zone so we have to get smart about how we document decisions and we've typically been okay at at them but there's still a lot of discussion that happens in like crits that aren't documented in like I think the crits weren't even recorded before I joined but then I wanted to watch them async and follow up on that so those yeah those need to be centralized and maybe there's obviously like if you run an agent and you put those meeting notes into the the knowledge maybe that might be to verbose for like a general agent but maybe you have just like a decision like a crick agent where you try to look at design decisions and when people have talked about things and the thing that I found I've done this thing to like where I've asked about I have documentation on dashboards and like analytics and and and figure out right now the dashboard broke there's no documentation on how to maintain the dashboards I go around like you said I asked the first person then I asked the second person and their knowledge of the situation varies so then I'm thinking like oh well I could check option A I could check option C and this happens to be that I mentioned it in a standup and then the third person it's at all it's involved with this there's a here's this issue ticket that is around this situation so it could be anywhere it could be in a ticket it could be I hate from stuff to be in slack threads but at least slack threads are linkable too so that's a benefit it's better than nothing well I mean to your point okay so you you message on slack it some people I think this this is heavily dependent on people's personality and relationship with slack some people are able to just kind of let slack activity wash over them and and they're not constantly checking it other people need that need a clean slack right there's no unread messages constantly going back so every time so let's say you post let's say you post to a channel with two or three hundred people at a large company that is completely possible let's just say you flip a coin some people ignore slack other people need to keep it updated so then if it's 200 people 100 you you theoretically have created a situation where a hundred people are going to check that message determine if it if it makes sense for them and then go back to their workflow so just that as opposed to like I'm just gonna read the docs myself or have an area where I can access it right don't need to ping a big channel you theoretically caused a lot of over cognitive overhead for a lot of people which is a simple question right so that's where these things start to be a big deal it's not at the small companies where there's a couple dozen people it's when you're talking thousands of people across multiple time zones that's where it just it gets gnarlier the order of magnitude for a decision becomes significant yeah and then I think this is this also is a nod to just so slack is as a piece of documentation but there there's been I think in certain teams there's been some questioning on whether our our help uh designs is some channels are public or not a benefit of things that our public is that not only can we search it but a designer or an engineer can also search it and one thing that I was tinkering around with a few weeks ago was slack has mcp tools now that allow you to query in a much more automated and in-depth fashion so that's I think it it does favor you well to have these threaded conversations that are in slack although I do want to grab them and take them and put them more in a neutral place at least we have the tooling to now traverse that right well and there's nothing to say that you can't at some point so one of the things that we you know have had to really evangelize at companies is hey please don't DM a system designer ask the question in the in the slack channel that case has been before it was just hey let's you know the more people that can see it the more people that can can get context the more people that can answer it or and also the more people can learn for every one person to ask a question there's probably 10 people that have the same question that doesn't have an had chance to ask yet but now nowadays there's the added benefit of okay you know theoretically you could have a uh an LM scrub your slack channel every day give a summary of it in terms of trends common questions common components that are frequently discussed requests and that can be something that can be tracked over time okay what was the one what was the what were the top three components that were confusing for designers and engineers this quarter based off of slack activity being being being being are those correct you definitely want to verify but at least you have something to start from and can retrace before it was just you're buried in slack messages you never are leveraging that for something more that's what now theoretically slack can become a vector for documentation because there are tools to parse that summarize it and act as almost like a checklist for ways to improve the documentation it's great we love docs it's they're only going to become more important in fact i i do think there's a point there could be a point if if LM's continue to improve and they become a more integrated part of the process of for design and engineering to where it's documentation driven product development where you're describing what you want the thing to be the requirements you start with that i mean theoretically some people did that do that already with prds but i think a lot of people a lot of people outside the industry will probably be shocked at how much work isn't done with the purity or very very loose purity and i think a lot of people that have been in the business understand like oh yeah like sometimes it's just hey i want to do this thing and like there's nothing there's very loose requirements at best for making that and that and i don't know if that's going to work if an agent is being is being utilized heavily to do the work when that's saying i'll wrap that up with is that unless you put your name on something and you put yourself as a contributor either like on a like a racie like table or what have you the LM will not know that you participate in the sanctity so that's like another thing that's been a little bit confusing there's been a lot of interest in using the agent that i've been making to figure out like who's worked on what who's the last person to work on dark mode and it gave an answer we don't know which is i appreciate that which is good because at least it didn't make it up right so i have guardrails and i give shitty answers but then people are upset when it gives you like a non-answer i i think it's great because you're not just making making stuff up but that's because there wasn't a specific brief that was linked to dark mode and then had someone's name on it so it's important get your credit it is it is important and i think you know once there's a name associated with with something i think it also provides the opportunity that when that person moves on it's a moment in time to say okay who's taking this over people are gonna have questions who's gonna be the point person for that for that thing right more docs doesn't have to be in gira or components is whatever you use it doesn't have to be perfect i have no skin on what you use yeah use google docs use obsidian use any thing yeah use anything yeah use anything use a shit you could use a notebook and then have a i have one of those like overhead cameras for like document like a document camera almost i'm doing that you get an overhead projector and you just like you just show that over projector and and yeah and record a video listeners know won't know what you're talking about that's okay that's okay i'm just gonna i'm not i'm gonna provide any more description that's it shout out shout out to the oldies out there shout out to classroom technology from the 90s yeah loosest loosest reference to technology for sure well thanks for your that's a good one thank you yeah good one thanks man see you