You are listening to episode number 17 of Rexbirds, a recommender systems experts where we discuss the application and research in recommender systems talking to experts from industry and academia. I am Marcel Karowski, data scientist and Rex's enthusiast and I'm your show host. I hope you enjoyed this show and I'll be happy if you support it. Doing this is very easy and straightforward. Subscribe to Rexbirds on your favorite podcast player, leave a rating and follow me on LinkedIn. This is the easiest way to support me and this project. Thank you. If you seek more details on our discussions check out the episode show notes. There you can find a rich compilation of papers, blog posts, guest information, talks and more for each episode along with the episode transcript. If you have questions, a recommendation for an interesting expert you want to have a my show or any other suggestions reach out to me on LinkedIn or Twitter or contact me via email at
[email protected] that is M-A-R-C-E-L-R-E-C-S-P-E-R-T-S.com and now off to my conversation with Miguel Fieri from Microsoft. A lot of companies they claim that they have a recommended system and then they start scratching in the service and they don't. I don't think there are many technological solutions that have these super extreme high ROI. You know, when you have a lot of talented people kind of, a lot of times people have strong opinions on how the code should be done or whether we should use a class or a function. There are some intense discussions about that, right? We found a solution to that problem and I think that was like the main driver of the success of recommenders because the result is that we had a lot of talented people that could work together on the same direction, a constitution. So that's what we created. We created a lot. Our role was that all of us, all the contributors of recommenders had to behave in a certain way. In our view, the best way is when you have something that works and then adapt it to your problem, right? So that's why we have so many examples, right? Like almost no matter what problem you have in recommendation system, you're going to find there five to ten different notebooks that hopefully what you can do is you know you can run the notebook, see if it works and then just remove the dataset that you see there and put your dataset and then start it right away. Hello and welcome to this new episode of RECSPRITZ, a recommender systems experts. Today I have invited Miguel Fierro, as my guest and we will be talking about Microsoft Recommenders and also a bit about most recent developments in generative AI and what Miguel's view is in terms of these most recent developments and especially about Microsoft Recommenders. To give you brief introduction, Miguel Fierro is a principal data scientist manager working at Microsoft where he is leading the personalization team and this is not his only duty or work but he is also an adjunct professor at the i.e. university. Miguel Fierro has obtained a master's degree in robotics and automation which is actually the topic where he obtained his PhD which he has gotten from the King's College in London also in robotics and he has done several publications at for example KDD, dub dub dub and the Nurebs. So hello and welcome to the show Miguel. Hi hi Marcel, nice to see you and yeah happy to have a conversation about recommendation systems. Yeah I'm happy that you that you joined the crowd because I see you very often posting very interesting insights that you share on linked and so you are very active there sharing your thoughts, giving advice to people, also how to keep up with the most recent developments and yeah I really like to to see your thoughts and what you are sharing there. Can you share with us actually who are you and how did you get into recommender systems? Yeah well I'll say I'm a very curious person, probably that's a very good way of defining me. I studied engineering, my undergrad was electrical engineering and when I was finishing that degree I realized I didn't like it so kind of wasted some years of my life and then I was kind of at the end of the degree everybody had to do a final degree project right and typically the normal thing is people do the final degree project in their out of expertise that for me would be electricity that I thought electricity was not really very interesting for me so I was finding other possibilities and I went through the department of robotics and I kind of fell in love with robotics and AI and that's that's where I did my master PC and then I did a couple of startups and I later I ended up at Microsoft and it was kind of when I was doing the transition between startup and Microsoft that I started working on on recommendation systems. The startup I had was V12 recommender so we had a SaaS solution that you could take a picture of a government on the street and it would recommend what is the most similar garment in a catalog of our retainer. That was kind of my first experience with recommendations systems and then when I got to Microsoft I also started working on on recommendations system. One of the things we did is recommender repository that is you know it's got a lot of a good number of star big attention and a lot of contributions from really talented people and yeah I would say that's that's kind of my path to the area. Oh okay I see so you basically started out with fashion recommendations one could say and then delved into other areas of recommender systems. I mean Microsoft as a whole is engaged in so many different areas where you can personalize your products with the help and support of recommender systems. So which other domains and areas have you encountered during your path besides that very start on fashion recommendations. It's interesting because during my time Microsoft I've been working in I could say two big areas one is you could call it technical sales and technical sales is that we go to customers typically large large customers and we held them to build solutions for Nashra and and my team precisely we held them build recommendation systems for Nashra and we worked from media companies to big retailers some of the big retailers also gaming in the gaming industry as well. Yeah I would say this three is the three that we made more recommendations and particularly in retail probably where we had the biggest impact and then also then I moved to people called engineering which is basically you create internal products for Microsoft and then I worked on some internal products related to e-commerce so retail as well and also gaming so actually you know these three probably are the strongest. So quite a broad set of very different domains and also with all their specifics I mean recommenders are not always only the generic set of algorithms that you have but you also need to incorporate a lot of domain expertise domain knowledge to kind of tailor them to the corresponding domain. So what is it actually that that made you so fascinated about recommender systems because I do get somehow that you were in the very beginning very enthusiastic about robotics kind of driven from being dissatisfied with electrical engineering in itself and then at some point there was some kind of another switch was AI kind of the bridge leading you from let's say robotics into the field of recommender systems and then you switched only let's say one part or what is it that made you hey personalization that's it. Yeah so I would say like you know first of all robotics is super broad right like you have the mechanics of robotics of electronics and also you also you have the AI right which is the kind of how the robot moves or behaves etc right when I was
Even when I was doing my busy, I was more focused on the AI part. So maybe I would say 80% or 85% was AI versus multibit electronics and almost nothing related to mechanics. So I already was leaning more towards AI, but I particularly like this applicability. And usability, right? Like the nice thing of a robot is you have sub programming, and at the end you have robot moves, right? Also in AI, all the time you have that, right? Like you create our recommendation system and you can provide better experience to the users, right? Another thing that was very interesting, also I have my entrepreneurship background. So these, these ability to kind of, I will say that in AI, you can do two things. You can either in terms of economic value. You can either increase revenue or you can reduce cost, right? A lot of the time many of the AI solutions are kind of reducing cost. Like for example, you know, Chad's, you know, you had a company, you had a call center, and you had a lot of people doing a repetitive task, of calling, etc. Then you can have a bot that you can do it all the time. And then these people maybe they can go to high level jobs, right? In that situation you are reducing costs, right? And a lot of people, a lot of things related to computer vision, to forecasting, to NLP. A lot of them are related to reducing costs. And reducing costs is something that, I mean, the finance people, they is not the best thing, right? Because, you know, reducing costs, you always have a limit. And it's very problematic. The best thing you can always have is increased revenue. That's the best thing. Because the sky is a limit. Kind of, right? I mean, the limit is the human population, right? But I mean, the limit is very, very high. So our recommendation, our recommendation solution is actually one of the few AI systems that can increase revenue. And another thing that is very, very important and this is one of the biggest hurdles that we had in my team and in general, in Tamigatins, is what we call the dependency. So dependencies are, when you create an ML solution, sometimes you are dependent on some other technology. Just an example. So let's say you create a churn solution. You want to identify churn, or you want to segmentate customers in different groups, right? Now, what you want to do is you want to reduce churn, right? So you create your machine learning solution that creates some segments of people. And then you need another system. So you have another dependency, which is the email marketing software that sends these emails, right? And so you are dependent not only on the technology, but also on what you put in this email, right? The problem is that there is no direct path from the churn identification that you build with value, right? Whereas in RECO, you have a direct path, right? So you have a marketplace. You basically put different items for each individual. And then the person buys or doesn't buy. So it's completely direct. It gets a bit the difference between the, let's say, predictive nature of certain ML output versus the prescriptive nature of what we see with recommender systems, where we directly intervene with the experience that users make. Would you say it like that? Yeah, exactly. And I think that was one of the reasons why I got attracted, because suddenly, a lot of the success, the majority of the success relies on my team. I don't have to rely on different things. That's a massive abundance. And the other thing is that I remember those report by Mackenzie around 10 years ago talking about the Amazon recommender. He's kind of famous. And they mentioned that 35% of the revenue in the Amazon.com accounts for recommendations. That's-- I mean, 35% of the revenue-- I remember that I did the math, because-- 35% of the people think it's a lot. But so 35, when it was published, it was around 20, something 25 or 27 billion. Now it's around 100 billion. And the very interesting thing is that there's a massive ROI, because you don't need tens of thousands of engineers to create these 100 billion. So I don't think there are many technological solutions that have these super extreme high ROI. That's very interesting. Basically, you don't need the people to scale it. You need basically a good team of engineers, scientists that you work with, of people that are able to integrate and innovate. But then to scale it, basically, given your algorithms, your systems are actually designed in a scalable way, then it's about the compute and the storage that you need to throw it in. It's funny that you are bringing up that study by McKinsey, because I've also seen several times studies there in terms of personalization, the value of personalization that people are demanding it. And I'm always having difficulties to finding the real root source of that 35%. When you brought it up, I actually had to consider that fortune article that was somehow citing internals by Amazon, where they claimed to make a 29% sales increase within just one year by integrating recommendations in almost every part of the purchasing and buying process. So yeah, definitely numbers that are in there, let's say, Wallpark. And it might convince you that there's really a growth. And that actually really likes the perspective that you are bringing up versus top line versus bottom line. So it's not that-- let's say it's providing a value of reducing costs, but the opportunity is somehow limited, whereas it is not really as limited when it comes to increasing revenue. OK, cool. Yeah, it's our main topic for this episode. And I'm really glad that you joined, because as far as I have understood, you're the main person in charge of driving the Microsoft recommender repository, which is a great collection of many algorithms. Can you give us some background of how it came to that point and how you basically developed that repository and why it gained that much attraction? Yeah, so the background, the reason why we build this is because when we were working in-- my team was working in customer sales sales sales, the way we work is that one or two of us will go to a customer and build a solution. And then it was very clear at the beginning that we weren't reinventing the wheel at the time. So we will go to customer A, and then we will do our recommender first class. And we won't share any code. We'll go to customer B, something. And then a couple of us will say, I don't know, guys, we should do some library so we can share instead of taking six months or nine months to the project. It should take us three or whatever, right? And then around that time, in my team, we were doing a lot of blogs and a lot of DR and technical marketing. So we said, OK, can we publish this on open source? And our leadership said, yeah, do it. And what we did is at the beginning, we wanted this to be a big effort, not a effort of marketing because our team was small, right? The core contributors to recommenders, I don't think over the years, we've been more than between five to eight, something like that. But we need a more people. So first of all, the first thing we did is we partnered with Microsoft Research. They have a very, very good team in Beijing. It's the team of Shishia. He might have been one of the top researchers in recommendation systems. So we partnered with them. And also, we partnered with some other open source contributors. One that I think was the first person to agree to me to kind of contribute external personal to Microsoft. Is Nikolás Hague. He had a very famous at the time, recommend a framework called Surprise. Nikolás now is a meta. And he is actually a contributor to psychic learn. And I think he also contributed to Pytor. So I mean, the guys is really good. And I remember he was the first person to actually, I'll tell Microsoft to contribute to recommenders. He did a notebook with Surprise, actually. It is a friend of that we see. He's very good. And that was kind of the origin. But I think there is one very, very important thing, which is kind of the strategy that we use to build recommenders. And we spent a lot of time on the contribution guidelines. And how we contribute? Because in many technical teams, and I think this happens in all companies, in very complex Google, Microsoft, MetaO, Amazon. When you have a lot of talented people, kind of a lot of times people have strong opinions on how the code should be done, or whether we should use a class or a function. There are some intense discussions about that. So we found a solution to that problem. And I think that was the main driver of this access of recommenders, because the result is that we had a lot of talented people that could work together on the same direction. So the thing is, this came from a podcast that I was reading about the Roman Empire. The Roman and Paisen have worked on that for a couple of years.
is a very interesting time empire because it kind of rose and died. And the historian, the guy who was doing the podcast, he said that there were three reasons why the Roman Empire was so successful. The first one is talent. So they had the best warriors. And if you think about the big tech companies, it's kind of easy to attract talent. So all the companies, they have these fees. The other, the second thing in the Roman Empire is that they have like a mission. So they, they actually wanted to kind of Latinize Europe and and expand and share the culture, etc. All right. So that's kind of a common mission. All right. And again, in all these companies, it's kind of easy to have a common mission. Like depending on the company, sometimes it's bought and out. Sometimes it's stopped down. But it's kind of easy, right? So these two pieces are there. What is the one that is missing? So the one that was missing, then so on in the Roman Empire that surprised me is that they had the law. So in the Roman Empire law was very, very important. Like everybody had to abide to the law. And then I thought that was the piece that was missing. We had, we needed a law. We needed a constitution. So that's why we created, we created a law. Our law was that all of us, all the contributors, recommenders had to behave in a certain way. And this is something that we all choose. We all agree on this way of working, right? So that it completely changed the way we were because at the beginning we were having a lot of discussions and a lot of meetings like, you know, where we were not making advance. And the moment we had this law and we agreed on how to behave and how to review poor requests between us and how to think about making decisions, suddenly we were able to go super, super fast. So that was super, super important. And that's something that I was employed in many of my teams in different teams. And I think that was super, super important for us as a super commander. Okay. So this was kind of let's say, they're not really the turning point, but the point where it really accelerated since you were actually providing alignment with in your team. So somehow internally at Microsoft work for the people working on that ripple that you kind of derived from the demands of where you put into production, recommender systems, seeing that there was a demand for making things more generic. And thereby, of course, decreasing the effort that you needed to put into each and every time where there was another demand. So that kind of alignment allowed you to speed up, but also I would say to manage the complexity of the development process to much better or more efficient degree than it was before. And then was it that you shared or made public that law, these guidelines with your open source community or how did you actually enable this? Yeah, yeah. The guideline is there is available. And I think a lot of teams, the way they operate is what some people call the benevolent dictator. The goal of the dictator, typical is kind of the more senior engineer is kind of the person who kind of makes the final decision. That's how Python was about. That's how Linux was about. Now the problem is that that's no routine. That's a leader that's, well, I'm not that's a leader, but that's kind of the dictator that actually tells all the people what to do. In the approach that we took, we didn't have any leader, like all of us were equal, right? That we all respected each other and trusted each other. So we were able to have, we really enjoyed the project because we were really thinking that all the contributors, I think, believe that it was their project. It's not like probably, yeah, the contributors to the leader of the contributors to Python, I don't know to what degree they believe is their project, right? So what I wanted to achieve with this way of working is that it's not my project or it's not the project of the four people that we started recommend this. It's more, everybody is welcome to participate. Everybody will follow the rules and no matter the experience, no matter the same unity, everybody follow the same study. Okay, okay, I see. These rules, so how strict are they? What kind of level do they have to guide people to behave or write code or design algorithms in a certain way in order to contribute to that repository? So they are not that strong, I would say, for example, one very important thing that a lot of people are not doing is how to make technical decisions, right? For example, one decision that was very tricky. We had in recommenders, we had code in Spark, in PySpar, and we had code in Python, right? Now, for those that the not a difference, if you look at like the typical Python code is quite different to typical PySpar code. The PySpar code looks a lot like Java. Okay, so if you are if you're a Python poorest, right? If you're a Python poorest, you'll say, no, no, we're going to do everything as Python, right? Because this is a Python library and we shouldn't do classes for everything, you know? And the way we decided is, okay, so instead of saying we're going to do a vote, which is what most people do, it's like, okay, so there are 10 people or 11 to 12 people, let's raise their hand. What we say is, okay, who are we serving here? And we call these every-dum-based design, okay? The thing is, who are we serving? Well, we are serving the people that go into recommenders and are the users of recommenders. They want to build recommendations. So we want to make the experience fantastic, right? So we stop thinking about what I legal personally like, right? And say, okay, what a person that start with recommenders will like? And then we thought, okay, I guess if you're a Piesparc developer, you are used to this specific Java/Python way of doing things, everything's a class, etc. So if you come to recommenders and everything is like Python, it's going to be really weird for you, right? So then we said, okay, then what we do is all the tools, all the functions and classes related to Piesparc, they follow the Piesparc naming and the Piesparc structure. Whereas the rest, they follow Python, right? So the thing is, when you have a team, and obviously depending on the members' routine, right? But you completely changed the discussion from what I prefer to what your customer prefers, what the person you're serving prefers. And then it's like, it's all about bringing evidence on what the customer wants. And then the discussion is completely switched, right? It's like, okay, and then people will come and say, okay, I've seen you know, these customers doing this and these are the customers doing this. And then basically the only thing we do is just follow what they want. We give them what they want. That's a good example of something that we did. And I think it's completely different to what other people do. Okay, okay. So I've just gone through a bit of it and you are actually providing also a couple of quick start, Jupyter notebooks where you are quickly guided through an algorithm. It's fundamentals and applying it to some certain data sets. So for example, the Amazon reviews, all like people always know and recommend a system, the MovieLens data set. And then you are applying it to it, evaluating its performance in terms of several retrieval metrics. And there I already have seen that some of them require you to be having access to a Spark session or to fire up a Spark session and work in Spark with of course, the use of Pi Spark. But some other notebooks don't require this and have other dependencies and run in different, let's say, environments or under different assumptions. So in there, it's not like that you provide, let's say each environment for each algorithm, but rather those which you kind of assessed are the most commonly used context for using a certain algorithm. Would it be a way of how to put it? Yeah, yeah, yeah. Actually, the original recommenders, it wasn't a people library, right? It was just all the notebooks. And the reason why we wanted to have notebooks is again, because we were all the time thinking about our user and we were thinking, okay, what is the best way for these people to start using recommendations? So in our view, the best way is when you have something that works and then adapt it to your problem, right? Like almost no matter what problem you have in recommendations system, you're going to find there five to ten different notebooks that hopefully what you can do is you know, you can run the notebook see if it works and then just remove the data set that you see there and put your data set and then start iterating. And that is something that I think that's another of the reasons why it was so successful because people start using it that way, right? Like they don't know recommenders and that's exactly what they did. And actually some of our customers, they kind of like that person, I'm sure they say, hey, that's exactly, you know, that's what we did. We had you had this data set that that it was kind of a toy data set and then we put our data set and we press run and then the team run and and they said, wow, now the next thing is how do we productionize that and you know, they were We're able to very, very, very well.
quickly iterate and create a solution that works. >> Okay. I see. To provide our listeners a bit more with what it really looks like. So can you maybe quickly guide us through the structure of the repository and what's maybe contained in it? >> Yeah. I will say that we have three big pieces of content. Then the first one is the examples, which are all the notebooks. Then we have, I think around, 30 different algorithms with different data sets. Sometimes is the typical user item interaction data set. Sometimes is what you could find in any commerce where you have a user, you have a product and details for the product or user and details for the user. Sometimes you have examples for a text recommendation like news recommendations. You can use text and yeah. So if people want to start recommending this, that's the first place that I will go. Then the first one is the library. So all the functions are not in the notebooks, they are in a library. So basically you can do peak install recommenders and you get the library. That's where if you are a developer, that's where you put your algorithms and your utilities. Then the last one is the test. Again, the test is another reason why recommenders has been so successful. We have around 900 tests that run almost every day. The reason why we have a very, very sophisticated pipeline for MLLs and for testing. The reason we wanted to have this really, really strong and really complex way of testing is because we wanted to make sure that every time when people download the library, it worked. That's what we wanted to make sure. And yeah, that was having a very strong MLLs pipeline. So one of the most difficult problems in ML development and development, which is the maintenance. The maintenance is very, very difficult because you need constant map power. So if you have a strong, a very, very strong test infrastructure, it's like, in my mind, it's like a wall, right? It's protecting you for bugs and it's protecting you to make sure that everything works. Yeah, makes your job far easier and also provides reliability for the users that take advantage from using recommenders. Yeah, actually in terms of the users, so do you have some insights of who besides Microsoft internally is actually using recommenders, whether for, let's say, developing ideation or also actually in production? It's very funny because a lot of people from top companies, even some people that are competitors like Cloud providers, they kind of forked our repo. And it's funny because every now and then we go like, "Oh, look, these guys are fucking, so maybe they are using it." Yeah, but I mean, like the thing is, this is an open source project and we don't have any telemetry in size, we don't really know who is using the only thing we can know is the information that is provided by GitHub. We don't really know who is using it. I know, I know internally, there are many things on Microsoft that are using it. I know customers that I directly work with them that I know they are using it, but you know, every now and then, it's funny, every now and then, somebody, you know, right to me on LinkedIn or is like, "Amigur, I'm working with a commander, saying, "Can you help me with this thing? Who are you?" Yeah, we've been working on this, we've been using this for two years in life. Wow, okay. Yeah, so it's very interesting. Okay, that definitely sounds great. I mean, if you're sometimes very surprised about where it has made its way at and where it's being used and, yeah. So maybe the other side, which is sometimes also the same side as the users, but are talking a bit more about the contributors. So as far as I get from what you are saying, as there are a couple of people at Microsoft who are actually working out there, five to eight main contributors that you mentioned, how is actually the reception in the community overall? So who is actually contributing to it? What is kind of the structure? Is it just a main core of people contributing to it? But is it more commonly shared between non-microsoft people and Microsoft people or what does it look like? Yeah, so right now we actually divide people into two groups. One is what we call the maintainers, and the other group is what we call the contributors. So the maintainers are seven right now. And these are the people that can accept PRs. Everybody can accept or review PRs or any contributor, but these seven people are the people that actually are looking up all the PRs. And then we have a contributors from outside Microsoft, for example, we have people from different universities. We have people from people that actually, they were at Microsoft and now they are on other companies, even some competitors or us. Yeah, I would say that to like a lot of people from universities and then several people from all the big tech and not that big tech, just technical companies. OK. In terms of organizing the contributor community, so how do you decide whether you want to go with a new algorithm? So I guess the last time I did my account, I ended up counting 33 different algorithms and where you basically reference the source and more of it in the notes on the main readme. But how actually do you decide which new algorithms you want to onboard into the repository? Yeah, so I guess almost every time it was either somebody in our team that they said, OK, these looks super interesting. Let's add it. Or somebody external that kind of proposed something. The way it works is that anybody can provide any algorithm as long as it's not the same content. If you want to have another implementation of the same thing, then somehow you need to, for example, improve what was there or maybe remove. Remove what was there and add the new contributors. So the only reason is because we want to make it simple for people. But yeah, anyone cannot do the algorithm they want. We already touched on, we're rather implicitly touched on the scalability to a certain degree because as we discussed, there are some of these algorithms kind of tied to using Spark. But I've also seen that some of them are, let's say, GPU ready or something like that. So actually, how do you perceive the scalability across the board of algorithms? Would you say that all of them are ready to use and scale to let say millions of customers on your user base? I mean, it's hard to say because it also depends on their amount of data that these customers generate, the window, length that you look back. But what is your perception on the scalability of those algorithms and the library in general? Yeah, that's a very good question. It's very interesting because intuitively, people think that recommendation system is a problem where you have a lot of data, right? So it's either Spark or GPU, this sort of training or many multiple deep use, multi-node or something like that, right? In reality, what we found is that there are not that many customers, at least in our experience, right? There are not that many customers with huge amounts of data, very, very few. We had some customers that they had a big amount of data and then it becomes challenging. Data engineering or machine learning engineer, like how to deploy this thing, how to train with an algorithm that is fast enough to accommodate this amount. But most of the time with a small Spark cluster or even a CPU machine or even a GPU machine with a very, very simply big GPU, maybe just one GPU or maybe one of these clusters that have four GPUs, that's enough. Yeah, talking about, let's say, a data intensive machine learning algorithms. I mean, with Microsoft, along with OpenAI, you are at the forefront of a bunch of most recent innovative breakthroughs, especially tied to generative AI with ChatGPT and large growth of large language models. What is your perception on the impact that large language models?
and all these most recent developments are having on recommender systems but also on models in specific. So what do you think is going to change or how do things need to change to adapt? Yeah that's very interesting because there are people around MIT and kind of close people. They're different different views right and maybe I can comment on both views. There is one view that and I've seen some papers where you have something like GPD4 and then you transform the recommendation problem to text and something as simple as Miguel who has this profile, this is what this item who is this product with this price with you know like it's kind of you take the data set and just put it together right and that's the information that you give to GPD4 to the LLM and then you say recommend similar products and actually you know is that you need to do some tricks in terms of memory actually you know right now GPD4 has in that I think 30,000 or 30,000 token space right so it's a huge number of tokens so you can actually add a lot of information there and that's a recommended right like and I've seen some papers where data approach looks and works very well there's another very similar approach there's a NLP network by google that's called t5 and some researchers they did a version apply to recommendation system then I think they call it B5 and it's kind of the idea is come the same basically you take the data of a recommender and kind of somehow put it in text form and just use the LLM to to predict to recommend back and that is the approach and I think we're going to see more and more this approach right like just use pure LLM as a recommender and then we have the other point of view that they say that is not enough actually some of the arguably the best researcher in might in the team in recommender team actually yes I asked in this question and he was not very convinced of this approach of the just LLM approach and he believes that it's going to be more like a combination of the backbone of the LLM's like the attention mechanism plus maybe the GNNs the graphical neural networks graphical neural networks for record is very very interesting and maybe the extra information through knowledge graph actually we have some networks that are knowledge graph aware so basically you can get extra information of entity purse and you can add that information to your network so he believes that that's kind of the route so but again in my opinion both routes are very interesting but yeah like personally yeah just to give you my personal view I would like to explore the GPT4 LLM path I mean they share some common ground which is that yeah we should definitely see it as a component towards building better recommender systems whatever better means I mean on the downside if you for example would turn each user that you want to provide recommendations for in that kind of text that you elaborated on for showing us the first approach I mean that might also drive you into into scalability issues for example or also there might be some arguments saying okay but then how do you actually optimize for things on a global scale for example if you want to balance relevance of recommendations with let's say the diversity of recommendations or something like that so this is something that you might get into trouble there but I guess the the path is not at all at its end so we are still on our way to find out how we can make use of these systems to build a better recommender systems but I actually like the set you brought up the GNN perspective because it was if I remember correctly Max Welling who was providing a keynote at Rexels 2021 that was all about GNNs and their relationship to recommender systems actually Max Welling Microsoft over here position and now he's a Microsoft he's a he's he's really really trying yeah he's one of the pioneers in graph commercial networks another one another researcher that I like a lot is Petra Belikovic from Big Mind he's doing super interesting work in in you know some kind of neural algorithm so how to how to mix neural network with algorithms with the algorithms but yeah general there are many many ideas and many parts that are very interesting another one for example is reinforcement learning the you know a recommender system is is naturally a closed loop system right I'm in team you know you can do a very first learning algorithm but the problem of of recommendation system is that it's actually very very difficult to do this close look and and for me you know I come from robotics and in robotics everything is closed look right you have your model the model prediction then you have a sensor and then feed right like this close loop uh you actually potentially could create the same in a in a website right but the problem is that you have so many dependencies right like you need to have your machine learning model you need to do the prediction and then in real time you need to track all the information of the user and and kind of mix it with the with the model so it's very very difficult I've seen just a few companies doing very reinforcement learning in record very you know very well and you know it's very difficult yeah I guess there are some very great publications also coming from from Google in terms of reinforcement learning for recommender systems so also a topic that people have been thinking about already quite some time so I I remember some some youtube papers there that were dealing with reinforcement learning for rexers and then also several deep reinforcement learning techniques however what you said I would challenge it a bit and so I would agree with saying that close loops are quite the standard in recommender systems but actually that they are also posing some kind of the problem because then you might enter several different biases that yeah you result from okay users can just interact what the recommender system decided to show to them and then it's always a bit hard to see really the ground truth of user intention and what users really like where you then need to embed more organic signals from users that were not under the influence of recommender systems yeah absolutely and I think that problem actually is very very big in media companies right like Netflix for example because if you compare for example something like a media company like Netflix to something like I mean commerce when you enter to Netflix you have all of your pages is a many many recommenders like many many different recommenders I also accept the hero role the top role which is typically it's kind of done manually everything that is is a recommendation so that that that's where you have these bias the the selection bias right because you don't you don't have that many people searching and and then in commerce I will say it's easier because in the commerce people search more right like you go to having a commerce and then it's like I want some specific food or some specific garment right and then you get all the all the search so then in that case it's kind of easy to get that and bias information but yeah that's one of the biggest problems yeah true point true point yeah actually you really do like these two different perspectives on the effect of lm's and all that part that is associated with them on recommender systems talking less about the system the methods and the approaches but more about the people what is your perception because I also see I mean you are very active on LinkedIn providing advice providing technical insights and your ideas which is very beneficial for the community I think what is it that you would provide people who are very active in the field maybe also getting a bit anxious about what they need to learn where they need to skill up what is it that you would provide Rex's researchers or Rex's practitioners as an advice from what you see and how you experience that changes so particularly for people in research and I will say I would say the continuation is typically more for what I call apply AI rather than research right because in research your objective is to create performing machine learning I agree just just your only objective right apply AI is a little bit more broad right like your objective is not just to make a good recommender system is more like okay so how can you help the business with recommendations so for these people one idea that I share a lot is this T-shaped professional profile so this shape professional is somebody that is it has a very deep expertise in one or two areas so let's say recommendations and then it has you know general
knowledge of all the areas related to your business. So for example, maybe you are the best or the best on your team recommendation system, but hopefully you also know a little bit about energy, a little bit about compare vision, also how the business side of machine learning works. Maybe also you know, emelloks, right? Maybe you are not the super expert, but the thing is, from your perspective as an individual, I think it's really, really powerful, right? Because you are, first of all, you are the go-to person for one area. In this case, let's say recommendation. So if someone in the team has a deed about recommendation system, this person is going to ask you. So that gives you a lot of leverage. But also you are not isolated on just your RF expertise. You know all the areas, right? Like maybe, for example, for recommendations system, it's also interesting to understand how a website works. And what are the key metrics in a website like any commerce, right? How important is the usability in the business? So what are the key metrics like click through rate or conversion rate or the turn, like how do you recommend there can affect all these metrics, right? But that's the T-Shade Professional. And then from the company point of view, if you have a team of T-Shade Professional, you have the best team in the world. Because it's like you have like special forces team, right? So imagine, it's like you have the mega expert in record, mega expert in computer vision, mega expert in NLP, mega expert in MLOPS, right? This is suddenly you have an amazing team of people that can solve any problem, right? So then this T-Shade Professional, in my view, is one most companies want. Because it's like not just the super good researcher that only knows about their area is more like A. So how can you provide value with what you're super expert? Yeah, definitely. I mean, it also facilitates communication a lot, because even though the Rex's expert is actually not a computer vision expert on MLOPS expert, if they know they're basics, they are much more versatile when communicating with the MLOPS or with the computer vision experts and sharing thought there, because there is some kind of a shared knowledge or let's say skill base, which acts as a bridge between these two people to then solve problems much more quickly. I have to think about in a technological manner as well. So I mean, these data scientists who are working in a business context, and for example, only concentrating on Jupyter notebooks or something like that, this is not really providing the best value, because you also need to think about how, for example, to create a package, how to work with Docker, how to adhere to certain MLOPS principles, CI, CD, and so on and so forth, to really be able to then work along with data engineers, ML engineers, and so on and so forth, and not kind of the old image of throwing their Jupyter notebooks over the fence and leaves the stuff to other people. That's very common, actually. Yeah. So I mean, still a lot of room for the development there for people, but also for their different areas that we are talking about. Yeah, in terms of the more broader picture of recommender systems or more general personalization, which challenges do you see for the future that we ought to solve or to address? Well, I see still a challenge of adoption. A challenge and an opportunity there, right? Because it's very interesting. Recommendation system is something that is being there for many years, is not really new. And something very interesting is that a lot of companies, they claim that they have a recommender system. Then you start scratching in the surface and they don't. Actually, to the point that I have some meetings with customers, and I remember one specific, like a big retailer in Europe that we have to help them with our reconciliation, right? And then the PM, my PM came and said, Miguel, yeah, I don't think these guys need our solution because they already have a recommender. And I said, OK, yeah, well, let's talk with them and see if we can have them so hot, right? And then actually, it's only that with the head of their hands is a company. They actually recognize that they didn't have anything. But when we talked with the PM, the PM was not able to go super deep into the ditches and into the algorithms. And at the end, it's like, you actually need something that we have to offer. It's very interesting that not a lot of people are taking advantage of this solution. So I think that's an opportunity. And again, a challenge, because I mean, why people are not adopting. That's one thing. The other thing that I think is very, very important is the ethical part in the recommendation system. So a recommendation system can be used to provide a better experience to users that it views. And in a bad way, it can also be used to influence the behavior of people, right? And we've seen, for example, we've seen this kind of camera-sumolithic that kind of a recommender system was used to target specific groups of people to vote for some specific candidate, right? That is savvy. And I think it's the way it's happening right now, with GBD4, TADGPT, right? Like, who's to blame? Who should be responsible? Actually, I had a-- Sam Alvman, the CEO of OpenAI, was doing a tour in Europe. And actually, I think the reason he was in Germany, last week or so, he went to Spain. And I was in the town with him. And yeah, it's interesting because he went to the Congress of the US to ask for regulation. But then the EU, the Europe is doing some regulation. And then he mentioned that, no, that's too much regulation. The real thing was like, yeah, yeah, yeah. I decided, you want to regulate-- actually, you guys haven't seen his certification in the Congress. The lawyers and the politicians, they were kind of saying, well, it's the first time in history that a company comes to ask you to regulate, right? A lot of people, they have done their careers going after the companies to regulate. And the company said, no. You are the first one that came to us, which is, wow. And then a week later, Europe said, OK, yeah, this is that. And they said, nope. I don't know. It's great to see. At first side, it seems a bit contradicting because on one side, he's saying this, on the other one, he's saying this. On the other side, one might also argue that, for example, there's too few regulation in the states and too much regulation in the EU. So what is kind of the common ground? But that's also easy to say, but maybe hard to define, and then also to bring into reality. So yeah, but I followed that and was also getting a bit confused. It was funny, though. But I think in general, we need to be careful with recommendation systems in the world. I guess in any technology, it can be used to, the way it is, it's a search engine and superpowers. It's like a technical solution that saves me time. When I go to an anti-comers, instead of spending half an hour, I want to spend, particularly I don't like to buy a lot. I don't want to spend all the money. So for me, it's like five minutes, and I get exactly why I want. That's what does. Yeah. In recommended systems, it's very domain or context dependent of how critical that ethical side could be. So one might say that for, let's say, retail recommendation, there's not such a great problem. But on the other side, if you, for example, are operating a marketplace and don't provide enough opportunity to be shown for your smaller businesses or something like that, then you again have ethical concerns. Whereas, let's say in a social media or news recommendations, the ethical concerns are much more apparent since you might, let's say, create division or spread hate or something like that. And then, of course, they need to be addressed much more urgently or importantly. So maybe just as there are short reference there. So for people who want to know more about fairness and recommendation systems, I'll look up the last episode with Michael Extrand, where we actually talked a lot about fairness, which is one of the many ethical concerns one might come up in recommended systems. And yeah, I like what you say. I mean, it's kind of with every technology that it should be used with care, such it is with recommended systems. Yeah, Miguel. Thanks for sharing all these great thoughts and impressions that you have from your work and also on the recommenders library and all of that stuff. I mean, also some great hints for people to think about how they want to position themselves. So for example, bringing up your T-shaped data scientist or T-shaped data person, I guess, not only holds true for data scientists, but also for generally for people working in software IT or something.
something like that. And also about the challenges regarding adoption or more broader adoption of recommender systems and the ethical concerns. You have already done to a certain degree what I always ask my guests towards the end of the episode, which is if they want me to reach out to a certain person to have them as my guest or invite them to be my guest. Besides the people that we already mentioned, is there some specific person that you are having in mind that you would like me to feature or invite to the show and have on rexperts? I mean, yeah Nicolas, Nicolas Hag is a super nice guy. Definitely really, really good. And Nicolas is very handsome, right? That's it's a great, great, great developer. And you know, people and on Microsoft research, and it's not so easy that is, but you know, there are a couple of like one of the containers of recommenders is Jenschen Lien. He's really, really good at this one best researchers, I know. And his manager is Shinshia. Shing is amazing. Yeah, this idea is really not sure how easy it will be to get them, but I mean, these are fantastic. Okay, great. So I will do my best and reach out to have them or provide them the opportunity to also get them bored of this podcast. Yeah, so Miguel, thank you for your time and thank you for sharing your thoughts with the recommender systems community. It was a great talk and great to the horizon. Yeah, thank you myself. Super nice interview. Thanks. So have a nice day. Bye. Thank you so much for listening to this episode of rexperts, recommender systems experts, the podcast that brings you the experts in recommender systems. If you enjoy this podcast, please subscribe to it on your favorite podcast player and please share it with anybody you think might benefit from it. If you have questions or recommendations for an interesting expert, you want to have a my show or any other suggestions, drop me a message on Twitter or send me an email to
[email protected]. Thank you again for listening and sharing and make sure not to miss the next episode because people who listen to this also listen to the next episode. Goodbye.
, bye.
PodcastTranscript AI