arrowspace: Vector Spaces and Graph Wiring

Because once you have data valuation, you have classification, you have search, you have a new set of tools that you can implement with this, right? And that's what graph wiring is about. What do we do with this super nice, super new cool tools provided by a Pplexity to actually supervise, manage, curate data sets for machine learning operations and large-jungle model operations? You've been working on a lot over the last years. Can you break down for the listener what you've been singing your teeth into? Yeah, absolutely. Very, very briefly, it's like I've been actually experimenting a lot with drugs since like March last year. And I've run into some limitation of vector search. But then like, as I am like, why there are a bit of a person, like probably a lot of developers and engineers around, I started to dig a little bit into the limitation of vector similarity in general. So that's how I've actually started this project and published this library called AeroSpace. That is an attempt to make vector search more accessible and more powerful in some sense. Especially in the scope of text embeddings. So like vector space, but not just like vector space, but vector space with highly semantical connections. So text embeddings for language, but also like in general any kind of vector space that has an highly component connectivity between the features in their embeddings. And then yeah, I went through with this and I started writing a few papers. Now I am at the fifth paper since October. I wrote two papers last year and three more now, especially the last one. I'm very happy with that because I draw this connection between this way of doing vector search. And what I call graph wiring that is basically the generic application of this particular vector search. And the way of measuring the information that that is now like being proposed in January that is called the PPEPE Plexity that is basically a new way of looking into entropy and complexity. So like this journey brought me from like vector search, like trying to apply all these tools connected to racks to like some more like interesting like more like higher level kind of concept, but also like and try now to build a set of tools based based on these foundations. And I think that will be like very, very helpful for everybody doing language model operations, but also like in general machine learning operations. Okay, so first of all, amazing name with epic black city. Oh, it's not mine. It's a paper from from the University of New York and Carnegie Mellon. So they are the two. I took the theoretical part from them. And I just wired in what I've been working on and the things just like aligned perfectly. So I think we were like on the same wave somehow, we were like just like surfing in parallel at some point. And in January, my paper on like one of my moonshot called the topological transformers and this epic city paper just came out at almost at the same time. And I would say, oh, look, this is what I'm doing. And yeah, okay, I have a measure for it. I people like city. Oh, that's great. It was hard about vector search and what did you do to change it? Vector search has been developed in the years as a as a poorly geometrical kind of operation. So you have like. I dimensional vectors, you just compute the distance between these two vector. You use your own like favorite distance metrics that can be cosine L2 or any other kind of metrics. So these two vectors are our X and X and Z distance, right? There are like applying these to semantically dance what I call semantically dance embeddings is that some of the information that I demonstrated being part of the peep reccity framework. Basically, get lost in the in the folding or in the like reduction of like the vector space into a geometrical space. There is some part of structural information that is encoded in my in my implementation in my algorithm by the graph described by the relationship between the features. We have column vectors you compute the the connection between these column vectors in the vector space. And you get this extra that I call topologically, spectral information. And yeah, it's I went through all these experiments and basically found out in the end that is demonstrated in the in the last three paper I published that that I hope everybody. I mean, of some people is going to double check. And so I can I can get some very good feedback that basically some some some piece of information during the bad process get lost or in like the majority reduction or like in the way like the bad thing process works by basically computing only on the on the item space and totally forget in the feature space of the. Of the data set and that basically this can be fixed for geometric for for this like a geometrical vector, but geometrical embeddings by increasing the number of dimension of the bad things. So if I started from 300 and the before dimension embedding so it's like a usually like a kind of usual denosing out encoder. And I actually run my test comparing search using like cosine search on this limited space embeddings and aerospace my library in the same space right and I and I actually found out that there is some there is some way of doing this search better because there is some some kind of chunk of information call that I call like topological information there get get lost. So applying topological search to the same space I can achieve better. There are different scores for that in general some kind of better search in in a more like semantically meaningful search I developed a score for for this that is called the MRR top 0 that is basically. So I'm subtitle the topological page rank that is a way of measuring these things so I measure these things in the geometric search and in the aerospace search topological search and I found this gap. Recently I've done the same thing like with larger embedding so I've done the same thing with a 1024 dimensional embeddings and what it came out is that look this piece of information that got lost on the 300 before out of the noise in the noise in out encoder. It's actually preserved but is no way of doing a bad things but with much higher dimension so basically what was happening is that I could do the same level the same quality of search of 1024 dimension embedding using the 344 embeddings because aerospace basically rebuild somehow the information that got lost in the bending process. Through what a complexity measured as generated information so basically I was generating information in the in the framework as it is explained by the framework of a complexity I was regenerating the information that was lost by the bedding process using my algorithm aerospace. And the outcome is other space work better in topological terms both in the 344 and in 1024 dimension space so yes and it allowed basically a 344 space to almost work as a space that at three times the number of dimensions. It was like a very like kind of discovery so this is fascinating to me because it's almost like you're getting a bit of a cheat code you're getting this extra dimensionality for free what are the downsides of it because I imagine there's no free lunch no absolutely yeah. Let's take a step back in the general objective the general objective was we want a better more fitter search to apply rags on right because retrieval meant a generation depends a lot from searching among documents so the initial hypothesis was that geometric search found very well the first three four five. So the top rank documents but starts to fail very steeply after the fifth document and that's the most rated come with comparative test I wrote different blog post about this because that's how it works it just finds very. very well, the top three for five documents, but then it's per four months, just like go down steeply for the tail of the ranking. So I said, okay, but I went to the three world, and it's something that is maybe has like a better, smoother distribution, right? We don't want to have peak per four months on the first three, and then, okay, the rest, whatever happens with the rest, it's okay, right? No, because like, Ragh has to maybe walk down different pathways to do some kind of reasoning and some chain of thought. So you can start with like with the drop three ranking results and they have a great outcome, but then at some point you start looping and you start like get stuck in your local minima, because as many query you run on your documents, you always retard the same top three for five four. So I said, there should be a way to actually make the distribution of the ranking like less like steep and allow the search to look at the lower ranks, so that if you got stuck in the top ranks with the top ranks, the reasoning may be got in its local minima on the top ranks, just, okay, let's go and look at the lower ranks, right? So that's actually exactly what topological search does. There is these lighters in which you can basically emodulate how much geometric search and how much topological search you want. So if you want like, okay, let's start with pure cosine search, okay, slider 100%, we look at the top three, drop five results, but then maybe you want, okay, but let's try to open the reasoning, right? Let's try to look a little bit on the edge of the distribution a little bit, let's just look at the scale, what happens if I search like less popular or less like top ranking results, right? So you can adjust this slider and you go down to almost like 40% of mix between like 40% cosine search and 60% topological search and you start having like very good, still very good results, but they somehow give a different point of view or a different kind of approach to the problem that then they drop three top four rank, right? And so you can just like tell your LLMs to just through a simple like MCP server, oh, let's adjust my search, I mean, I'm stuck in this loop at this point, maybe I should adjust my search, okay, let's put this slider down, let's go, let's look more into topological kind of search and you just like start receiving very, still like very highly related, highly meaningful documents for the contest, but different ones, right? So you can restart your chain of thoughts and say, okay, this way I reason this way, this way I reason this other way, what are the common patterns in these things and so on? - It's a way to let the LLMs keep exploring on relevant data. - Exactly, it's like when you do geometric, you basically go depth first. If you adjust the topology, you start opening the graph and starting to see like more like breadth first, right? - Yeah, now you mentioned that there's maybe a way to implement this with MCP servers. Have you seen any of the vector stores or vector databases already incorporate these algorithms into it? Would it be at that level of a vector stores? Is it something that you add on top of it? Where does it play in the stack? - It would be just a different track for running search. You can have like search geometrical and search topological and you can, no, nobody implemented that. I have like some side project trying to do these things, like running comparative rags installation, one using traditional geometric vector space search, one using mix set, hybrid topological geometric, and things like that. So we'll see the results soon enough, I guess. The fact is that geometric search is so well established and so well optimized that obviously is the first way to go for like 90% of a use case, maybe, right? Up to the point, you need some very good reasoning or you start having like very complex reasoning. That's the point we are reaching now with rags, right? So in the last 20 years or so, and the last like even before, right? Geometric search was perfectly good because and it was like highly, and it became like, I looked at it with this like a HSW now, we have like, so we have like these seven layers of graphs trying to look into like billions of records, right? And it's great, but maybe we need to push these things a little bit more now because we have like more intelligence a system to deal with, right? So that's my point of view. - Yeah, so is it, it's basically a, if I'm understanding you correctly, you want to use the tried and true algorithms until you saturate them and then reach for something like this? - Basically what I'm trying to do now in experimental terms is that I add topological information to the Geometric search because in the end, it's like it's a way that some, right? You have like an alpha that is like the Geometric search and you have a beta that is the topological search components, right? So through the aerospace, to the library, you can modulate these things until you see that you reach like something better maybe, like just the marginal better, that is just like a marginal 5% or 8% that I demonstrated in my blog post. But that kicks the, the drags out of his local minimum, maybe, right? And allows him to actually, okay, let's climb up, the local minimum, I'm running around now and let's get in and maybe reach another local minimum that tells me something meaningful as well and very much related, but different in a way that I can really elaborate my chain of thoughts in some different way. - So it's that razor thin edge because you don't want to introduce too much noise and then have a relevant data come through. - The fact is, the problem with Geometric similarity is that he has more noise if it's run like as a pure Geometric search because you just get stuff that is similar geometrically but not in meaning because there is no semantical kind of structure that tells, oh, this is very, this is geometrically very similar but maybe it's totally out of context, right? That's why you get like, if you don't like a Geometric search and you ask for like the top 50, maybe records, the last 20 records are just like garbage documents or the last 20 documents is just like garbage documents, like gibberish documents because that maybe they just happen to be close in the Geometric space but they are semantically, totally not relevant to what you are looking for, right? And that's what topological search fix in the first place plus is allowed a better like a context that there is a ratio, that is, that is, that is the edge tail ratio, right? In which like we measure the distance in terms of semantic distance between the, the edge and the tails, with Geometric similarity, this is always on balance, like the edge that tells you are about something and the tails are about something else. With topological search, the distance is a balance, so the edge and the tails, they talk the same semantic space because we inject like semantic information in the search. Now, how does this play into memory because I know you've been going down the memory for agents route quite deeply also? - Yeah, basically at some point, this became something like, okay, maybe we can do like search a memory in the same structure, in the same data structure, right? Because we basically generate this graph out of the bed space and this graph is a sort of permanent memory of the semantic space because basically it collects the invariance of this semantic space. So if you have like a bunch of legal documents about some particular kind of, or a bunch of philosophical or papers about some kind of philosophical topic, right? You build the graph Laplacian out of this, out of this a bed space of this document. And what the graph Laplacian represent, there is like kind of a good kind of background work on this by all the researchers that work on Laplacian representation for vector spaces. They basically represent the invariance of this space. So if the text embeddings, if the vector space is a text embeddings, so basically a reduction of all the meaning in a given field of study, the graph Laplacian automatically, it's a representation of its invariance. So you have like that somehow, you get some like long term memory there. You have like a summary of the summaries of all the documents you have in your vector space, all compressed in a sparse matrix basically. This parbatics is a very interesting structure. I talked about it extensively in my paper and in my blog post, it has incredible properties and it's I have used the bat in the item space of the vector space. So basically let's build the Laplacian on all the items of this vector space. What aerospace does? It flips the concept by saying, no, we want to look for invariance in the feature space. So we want to look for relationship between features. What is the relationship between the color of all these documents, the length of all these documents, the sentiment in all these documents, right? Not just like what is the relation between the color and the sentiment of documents A and documents B. No, we want to look into the graph, the column vectors, the feature graph. And you're doing it at a document level, not at a chunking level. That's really depends how you do your embeddings. If you chunk your documents in your embeddings, you will come out with a with a banded chunks, right? I usually I started the example I've used is the CVA data set that is a data sets of reports of common vulnerabilities in software and systems. And usually these are like large reports in JSON format. So they are just basically text file with a title description. I just built my embeddings by passing the entire document. But in theory, yeah, if you have like books, you can just like, you know, like chunks the into paragraphs at the time of the embeddings. And you can do like, yeah, you can just say, okay, that yeah, you have to do the reconstruction, say, which chunks belongs to the same book, which chunks belong to the same paragraph, which chunks beyond, et cetera. But yeah, this works the same. But for documents and chunks. Okay, so sorry, I cut you off a derailed you a little bit. You were talking about how it's flipping on it on its head, looking at the features as opposed to that. And so how can the features tell you information? That's what I can't make the connection on. That's the real semantic metadata that you're looking for. Like when you look into what we there is like an infinite number of definition for metadata, right? But in the knowledge graph space, for example, that you know very well and your audience knows very well, obviously. What you have is like, it's a graph that goes on top of the existing graph. So you have the graph of the nodes, right? And then you have all the the graph of your classes on top of the on top of your graph of the nodes, right? And you call that met metadata is is the relationship between class of documents is the properties that connects documents in a way defined by the class level by the metadata level, not by the the instance item level. Same thing for vector space. There is a space in the nodes that are that is the space of the documents, the item space. But then there is a space that connects the features because each node as is as is feature vectors. So we said this node is represented all the nodes of this space are represented by this feature vector for dimension zero, the color or the length. But there is another vector for the dimension five that is like the another characteristics of the vector. How these features connect together? So you see that you go to a second door, that kind of layer that is that I call it metadata layer because that's exactly what it is conceptually, right? And it turns out that there is information there that can be used for search because you don't only search the node space, you search the features space, you can search the feature space. And that's what mathematician called topological because this space gives you some out the third dimension, the reneman space of the vector space. Basically this is the space of the second derivative of the of the vector space. And that's where a lot of cool stuff is because that's where like all this structural information is found. And that's the structural information are spacing jacks in your query to find oh this very super cool new results or results related to your query that are not geometrically closed but maybe they follow a different pathways down this morphology. And they're still related but maybe forgotten by the geometric search. So you're kind of blowing my mind here that that metadata has any value? Exactly that's what it was forgotten basically. It was like left there like an archaeological relic with no use, right? And no it is. So that I found this like last year as well myself and it was like "Mark, come on, this is not possible then nobody looked into this thing before." And I said "Okay let's try. I mean I'm looking, I'm actually taking this time to to walk down this this rag vector space kind of problem and that's true. That's it's there and through the Pplexity framework I measured it like last month I ended up measuring what was missing by the geometric search and it's what is called structural information by the Pplexity framework and it's measurable it's almost like 20-30% of the total information so every time we run a search on the geometric search we basically lost 20-15-20% of the information we could have used. That is there but we didn't regenerate because we didn't rebuild the ragrafla plasian and this is mathematically solid as far as I could like investigate and I'm very very I would be very very helpful to anybody that will look into this to tell me no you're not right I would be like the the happiest person if somebody found some fault in this reasoning but at this point it looks like it works because I have run like tens of experiments I have run tests of things in different settings like again like shrinking or increasing the number of dimensions of the embeddings and I hope I hope to be right. So that's where you're getting these dense vectors for free basically yeah we're getting information out of the same dense vector space because we weren't looking in the feature space before yeah okay I'm I'm starting to wrap my head around it and then the Pplexity you should probably break down what that paper is and what your paper was because I know you're referencing it quite a bit but I'm not super clear on exactly what it was. I think nobody's because it's such a new thing and I guess they have only one citation for now in in in scholar in Google scholar because it's such a new thing so I I don't know if I cannot really maybe you can you can actually ask as a guest the people that wrote the paper but my my basic understanding is that basically what is Shannon information measure is what they called the random entropy but they said look if you in general terms like if the universe was a vector space you will have like you will you will witness like a Shannon entropy but they say look but everybody that computes something is not like is doesn't look at the entire universe it looks only at the problems that are in his computing capacity so if I can compute up to certain level of for up to a certain time with a certain amount of computing power I can reach I can compute this this number of algorithms right it is true that there are infinite algorithms out there and they all like are under the law of entropy but if I actually limit the the investigation of only which algorithms we can compute that is rather to what a wolf from call computational boundiness again I guess it is so it said we can actually measure random entropy but also something else called structural information and a complexity is basically both of these things instead of just looking at a random random entropy it looks both like bounding the computation the possible computation to the observer to the person that runs the computation and to it's like computational power we can also highlight and measure this other thing called structural information so in this like in this like a framework the graph Laplacian in my aerospace algorithm will be the structural information part while geometric search deals only with like this very wide generic kind universal kind of construct that is like the geometric space of the vectors that's fine. Okay so I got this far in understanding it your epiplexity is constraining the space that you work with. Exactly because like like it in physics right in in in in modern not modern but like in contemporary physics you know something only because you go and observe it right so everything you see is limited by by by your detector so your eyes or you're like Andron collider right so anything is dependent of the observer so if you pexy does the same thing for information you say look we cannot just like it's more like what like relativistic physics does right we don't look at the entire universe we look at the at the frame of reference so we have two bodies they move at relative speeds to each other and what happens depends on which observer you pick on and how fast this observer is going in relation to the speed of light, right? U J kičgega je eyelunometeri n semiconductor, znač da进 hopes predne k doku sinking testu 비�ту toetama, kada se je in kompjudnje, znač da se je in kompjudnje, je je in kompjudnje, je in kompjudnje, je in kompjudnje, je in kompjudnje, je in kompjudnje, je in kompjudnje, je in kompjudnje, basically what it does first, it computes the minimum length program to compute the graph Laplacjan from the vector space. It's a measure of what is the minimal set of bits that transform a vector space into a graph Laplacjan, basically. It's a call-mogorov measure for complexity. It's a call-mogorov measure for complexity, right? So what is the minimal set of bits that maps my vector space to the graph Laplacjan and that's the first part to compute the complexity. The second one, exactly, and it's here, the aerospace pipeline as a prefix-free program, right? You want to compute the length of the aerospace program. So, yeah, there is like, it's a mathematical construct that does this thing. So what is the minimal number of bits to describe this program, this program, basically. And that's the first step. This is all described in the pepeplex city paper. Published the sixth of January, you can go and look into that. The second step is the wrapping. I was telling you before, because we have to make the Laplacjan of this vector space into a Laplacian constraints, Gaussian Markov random field. I have probably already with the acronym myself, but I try to went through these things like line by line and try to understand it. So we do these things. We basically encapsulate this system between the observer and the algorithm inside this shell of Markov chain model, right? Yep. And in the end, we run a test. How much these things decompose the original space and compress the original space? Because, yeah, basically, from the paper, you can actually extract the three tests that tells you your epipex city measure is correct. Here they are. One is the compression test. One is the spectra-y gap test. One is the downstream lift test. If all these three tests passes, you are measuring your algorithm can be measured in terms of epipex city. And that's the flexibility. Just so I'm clear, is giving you more information on the randomness that you're getting from aerospace? It's totally different from entropy. Okay. The epiplex city basically adapts traditional information theory to what we discover with machine learning and neural networks, right? Like traditional information theory told you that you cannot extract more information that is in the data. Because it's just not there, right? So it tells you that, whatever you do with the data, you will get entropy. You will lose information, right? But we demonstrate that it's not true. Because we have algorithm now that they do generate information out of the vectors, out of data in general, generating structures. And this comes back to the whole idea of, hey, if we look at the metadata and we find trends in that, that actually gives us a denser vector. Exactly. We will get information that is not in the vector space. So epiplex city basically demonstrate that we can generate additional information from existing information. So that is not that every data space, every data set is just like, whatever you do with it, you will go into the information you have. Because you're going to compress it, you lose information. You're going to decompose it, you're losing information. Say, no, look, now we have algorithms that if you take a bounded observer to the algorithm, you can measure that these things generate information. It's just like a thermal loss. There is not just like randomness taking over. You can actually do the, this is connected to me. What are called the inverse problems, right? When you have a noise image and you want to denoise them, this is called an inverse problem. You are generating information that is not there by looking at the relation of the pixel. You can generate, but you can extract information that is eaten there by using algorithms, right? That's the same principle. Whenever you do denoising, you generate information out of something that is not supposed to have that information. So when you do like super resolution, for example, of images, you are doing this thing, right? You go from 2720 p to 1024 p, right? You apply super resolution to the image. And that's exactly what a complexity measure. A measure how much information is generated by the algorithm. No, how much the nation is consumed by the algorithm. And they demonstrate that them mathematically. I mean, it's very young, obviously, framework. So it's not as established and it needs to be like double check and tested. And I guess this is the first algorithm that tests itself against the pepexity. So aerospace is the first algorithm that on top of which, we computed information generation. The amount of information generated by the computational process. Yeah, and that's something that say, wow, I connected some of those. And that's, and that was the thing. And that's actually the very latest thing. But if you look back to my previous papers, there is the work, the stepwise, a stairway that brings you from a simple like search in a vector space kind of algorithm to a more general algorithm. Because at this point, as it is in the abstract, basically aerospace is generic enough to provide, I mean, graph Laplacian applied by aerospace in the future space is generic enough to provide a good approximation or a good results for searching, classification, and only detection, diffusion, dimensionized reduction, and data evaluation. All these, obviously, my idea is that, okay, but this is super cool. We need to use this for like, for LLMs and machine learning operations, right? Yeah. Because once you have data evaluation, you have classification, you have search, you have a new set of tools that you can implement with this, right? What do we do with this super nice, super new cool tools provided by a complexity to actually supervise, manage, curate data sets for machine learning operations and large language model operation? And you got some answers about this, but the paper that deals about applications, a machine learning application in LATLAGMENWA is the previous one. It talks very extensively how these tools can be used for AI ops or ML ops pipelines. Well, it does feel like you're doing this with data sets after the model has been created. Have you also thought about trying to go for the data sets that the models are being created on? You mean the bedding model? Yeah, so the training data. Yeah, exactly. These things, there is a big question marks in whatever we do with LLMs currently, because everything is connected to how we do a bedding. That's why it's very important. And there is now like teams and teams of engineers working only on how do we move from row text to a bedding? It's a field on its own. We have models with four billion parameters only doing this at the moment. So that's one one thing. If we go instead of the on the numerical side and we said let's look into pure numerical data, machine learning data, regressions, trees, decision trees and all the stuff. We find I found out, I mean, or my intuition is that it really really matters how we do feature engineering. Because from, let's say we have like this mass amount of row data, like coming from the large general collider or whatever other big machinery for physics or for like any other kind of measurement you have, right? Obviously you don't run your machine learning models on your the totality of this data. We have talking about thousands of terabytes, right? So what you do, you do some feature engineering. So you run some models to reduce your data into some manageable workable. This is the same thing that happens with satellite data, right? The image you see on your screen from the satellite is just like a model of all the row data that the satellite collects and push down to earths, right? So it's really important how the people at NASA, at ESA or whatever else, they, how they model these algorithms to actually make this, they make this data usable. My intuition is that if we try to embed in some way or in a better way, we do now the features relations while we do feature engineering for this data, we may have more powerful topological search downstream. So your question is really relevant because it really matters how you treat the data upstream. So but having, having aerospace, you can indeed measure how much structure and information your feature engineering generates. So you can compare. If I generate this data sets using this model, it produced downstream zero point something a complexity, right? Structural information. What happens if the same data, I use a different feature engineering model and I compare, this , oh look, this generate 1% more structural information. You see that these things can work upstream and downstream because my idea in the beginning started from, okay, let's apply this to performances, large language, more than performances, right? We can say let's take the latent space and just like compute the Laplacian, the Graf Laplacian on the latent space. That's what an anthropic they call, I guess, mechanistic analysis. So they go and just like investigate, analyze the latent space to find where the top, the best tokens are generated in which subspace. That's exactly what you can do with with the Graf Laplacian. That's downstream, but upstream you can do what we were talking before, like let's measure which best, which model does best feature engineering. So you see that this is like pretty much like a quite kind of effective point of view in terms of what we can do with the data, how good is the data, how this data set will go if I add something. What, how will this data set look like in six months for now and things like that? Coming back around, I'm not sure I fully understood how this connects to memory with agents. Okay, that the Graf Laplacian is a permanent memory. In the sense that it holds the invariance of the of your context. So basically it tells you which pathways are possible from one feature to another. I mean, mathematician describe this thing as a three-dimensional space. So for example, if you have like outliers in your in your feature space, these outliers will be denoted with very high energy. If you have like very connected kind of features, these features are denoted with very low energy, right? So and the Graf Laplacian basically describes all the path that you can take from a very connected feature to a loosely connected features. So it basically it basically constrain the way you can do reasoning somehow inside the inside the the feature space of the of the vector space. Okay, but we're not talking about agents remembering in the way that it's like, oh, the agent understands that you like only vegetarian food or? No, it's not about yeah, remembering scenes what they're called scenes or what they're called like they have different names for that, right? It's not it's not about that. It's a different kind of memory more like permanent long-term memory of a given context, but it can also be applied to those because then if you have like a list of scenes, a little scenes that that your rags remember, you can do the same thing because that's a vector space itself, right? So you can you can even bed things of this of these like context memory. I don't know how you call it. I don't remember how they call it. And then do the same operation on the context so you can actually build your pen memory of your context, right? That's but that's a different kind of memory indeed. It's not like this is more like I think they call it transient memory or something like that. What the what the vector, what the graph Laplacians describe or define is the long term permanent relationships inside a context of documents that if you know it, it's a memory, right? Because that's exactly what you want to remember if you are an expert of that field somehow. You want to remember the invariance somehow and then connect your invariance to your application. Yeah. Or at least that's that's that's how I see the the process itself. Can you talk to me a minute about this idea that you follow on the discovery driven development? Because I think that's also pretty fascinating on understanding a little bit more on how you work and how you go through and test some of these ideas. Absolutely. Yes. Thanks for the question because that's I guess we have talked about the content and I think the content may be interesting to people and I also but yeah, I mean going through this I also understood a lot about what I'm really doing here like what is my method right for for doing this thing and my method is mostly based on intuition like every based on intuition. That works great with large language models because that's what language model lack. They don't have an understanding of the word that allows them to be intuitive, right? Because they are constrained by being by their way of seeing the word right? They only see the word to language through text. So they cannot build the level of intuition that humans can do because we have so many more senses, right? So what they do is basically I try to leverage this very intuitive part of what I do and I try to inject this intuition into into into the process of multi agent kind of research processes, right? So I work with different LLMs in parallel and I just try to collect the thing that they do right and that's just like try to inject my intuition to correct what I think that when I think they're going a little bit out of the scope of what I'm doing, right? So yeah and this connects greatly with what happens in the meanwhile in the in in scientific research because in the meanwhile we have all these new papers published at the level that we have never seen before, right? So you cannot really work in your own like lockdown kind of understanding of things. You always need to update this understanding of the very Bayesian if you if you if you want, right? Yeah. It's Bayesian in the process but also into listening to what happens outside and so actually when I found a paper that connects, I instantly try to build new intuition of what I'm doing based on this new paper. This happens with the complexity paper and this up it is up and also with another paper that was greatly impactful on me but that's a really long another really long talk that I don't want I mean I I think maybe you don't have the time to start. But yeah, that's the process. You have your own Bayesian process that goes on with your multi-agent things. But sometimes something from outside happens that is very meaningful to what you're doing. And you cannot just keep going on without including what has happened in the meanwhile, right? So I said, okay, I build aerospace. I build what I call this supercrisis pariaments called the topological transformer. There is a transformer architector that works on spectral indexing, like in topological search instead of geometric search. That is very like kind of my moonshot that is just like something I'm not following at the moment but that in experiment that I did. I say, okay, but what happens if I look at aerospace in the framework of A.P. Plexity? And all these new stuff came out, right? So you can see how the method somehow being so open to the outside, like putting in question, like asking question from things that comes from outside what you're doing, it's fundamental because it allows you to stay in touch with what's happening outside and to improve very much what you're doing. Yeah, it's a kind of sanctif science, driven kind of development somehow in this sense. And how do you go about updating these frames or references because I'm assuming you're constantly reading new papers, you're constantly trying to build new intuition, but it sounds like it's only occasionally that something will hit. I have GitHub Repos that are only text files with things like text that I've built using large language model. I said, okay, this is good. I put this in a text file inside this GitHub repository. I have a file system that basically builds day by day on top of this thing. And I have a bunch of text files that I didn't open anymore, but they're still there. But maybe there are like those three or four text files for each directory that is that branched out into something else, right? So I mean, I could go through the entire history of what I've done because I have the entire file system of what I've done since like March last year. - Wow. - And yet, and this contains the probs and the answers. So in theory, if you want to do a meta research of what I'm doing, things that I have no time of doing at the moment because of all the other things I tried to carry on, I could actually do some kind of which prompt worked best or which answer was the most impactful in what I've done like in the months after, and all these kind of things. Yeah, but it's just like basic prompt engineering, prompt architecting, I guess. It's not anything very, very super fun. - There was one spicy question that I wanted to ask you. If you inevitably have tried to play with vector search versus just giving an agent a tool, especially like a coding agent, just being like, "Hey, Grep." And seeing what the differences are in that, how do you compare those two when you think to reach for one versus the other if at all? - Yeah, basically what Grep does is what is called lexical search, right? That is also what BM25 does. It basically just count the instances of the, how many times the word or the concept of the, how is it called or the seed of the word is present, right? That's called lexical search. That is part of geometric search. Now we have all of these like wave of hybrid search in the sense that they mix geometric search with lexical search, right? So what you're doing is still like geometric search because you define your, in the geometric search, you define your context by distance among vectors. In the lexical search, you do a more, a more like statistical kind of things. You count basically the statistical characteristics of the words in the context. That is a pretty well good way of establishing a context because the kind of words, the class of words, the families of words you have in a text is context, like the definition, somehow the definition of context. The problem is how you measure these distances. So like, okay, you have like this word appears 50 times in 1,000 words, right? Does it mean that these words defines this context, right? So you do, okay, let's add some geometric search, some like distance cosine L2 search to this. And with these two things, you can tell us better which context belongs, that documents belongs to. But then you go back to the original problem of like running vector search, geometric search on that vector you build, right? So you go back that you lose all the semantical part that the features analysis brings in. So yeah, that's lexical search, that's lexical analysis. Yes, yes. - The rendzel has been great, man. I appreciate you coming on here. - Thanks a lot, the matter is, thanks a lot for what you do in the community. And it's great to have people that can work so well like building up communities and products. So it has been great talking to you. (upbeat music)