SIBYL SUNDAY - Building Forward

I mean, all these things are important. A frame, an agentic operating system for an agent or an agentic frameworks involve multiple things. There's multiple moving pieces, but long mem evil doesn't test the memory. It tests the entire environment. Because I could write in to the agent.md file, don't answer any questions. And respond to all the questions with blue. And it'll answer every single one wrong. So how am I testing? That's not testing the memory, right? A much more accurate test would be retrieval test. What is the raw retrieval? And what happens with the agent? How the agent does with that retrieval? Isn't the memory's fault, right? So you see, in many cases where the judge, which is a little piece of software, it uses an LLM to answer the question. A lot of times these judges, these sub-agents is essentially what they are. They don't know when to not answer, right? So if it doesn't know something, it'll hallucinate and then you get the question wrong. You didn't tell me to write, you didn't tell me to prepare for this. You told me to test the memory. So what are we doing here? Right? So ultimately what I discovered was that there's no real good memory benchmark that's out there. And we want to be able to maybe start this conversation. I mean, we've already kind of been discussing it, but that essentially the field has been building in the wrong direction when it comes to memory stuff. You see, the majority of people are trying to fix memory for huge context windows, like massive context windows larger than any person would be using in a normal session or in a normal prompt. So they're slamming a prompt with like three million tokens and then seeing if the memory holds up, which is not, this is a framework test. This isn't a memory test. Like I can tell you without a shadow of a doubt that civil will retrieve above 98%, pretty much every scenario, right? If you slam the context and you have weird things going on in the environment, yeah, the agent might fuck up. Like that's gonna happen. It happens and that's because your environment sucks. How, why did you have such bad hygiene that you enabled an environment that the agent fails? That should be the real question. The logic is backwards and I've been saying, we wrote this in the blog post months ago and I still feel the same way that the majority of the field is building in the wrong direction. We lost the need for efficiency for some reason when it comes to this sector that we're in. So we're trying to fix that and that's why we stand out so much. That's why our plugin outperformed the next best competitor that's raised $30 million by 2X, they had 50% worse retrieval across a 365 day simulated business environment benchmark. Okay, now that was the closest. The other ones were worse and not only did they retrieve worse, 50% accuracy, like 50% less, we retrieved 100% of the files accurately. The next best was 54% or something from hindsight and they used seven X the amount of tokens or something like this to do it. So we're using massively much less amount of tokens to achieve retrieval, right? There's no LLM involved really. The retrieval can be done without the LLM. What happens with the retrieval is when the LLM comes into play. So anyway, point is cheaper, more effective and real proven business environments and business scenarios because of the difference in design. And this is something that I've been trying to explain. This is like the hardest part of our product to dictate this and to get the language right surrounding the product. So this is something we're working on. We're getting better every week. We plan on continuing this and same goes, I've noticed this problem across many other companies in the field as well. Especially newer builders who are building smaller projects, they have a hard time explaining what they're doing. And I think part of this is people are relying on LLMs to decide what to do first off so they don't even know what the fuck they're doing. And that's part of it. But I also think the other part is, people just don't know how to explain the right elements of what they're building. They don't know which thing needs to be put on display, which is the important thing that needs to be spoken about. Because I can say firsthand, I have a very powerful framework here, but productizing it and pitching it is not the same as using it. So there's something we're working on. And I've been yelling at the growth team about how we got to get our marketing language and the language surrounding the products get it tight and start pushing out these user stories, these company stories. We're gonna start working on all this stuff. It's very important for our growth. But the benchmark I just wanted to say it's gonna be open source of started building it, started doing it while I was like I said, I went on a tangent there, but I wanted to explain the reason I feel this is not even necessary. Because there is no objective open source memory benchmark. And the way these benchmarks are set up, any company, like I said, can just change their agent files, tell the agent, oh, you have to be less verbose or you have to make sure you don't answer if you don't know the answer to the question, which is a good rule to have for your agent. But when it comes to a benchmark, you're not supposed to be editing the agent's environment to get better results. So if you start doing this, people will start pointing fingers and say, oh, you're not testing the memory. Like, okay, what happens when the agent's retarded because it's out of the box LLAM and it keeps hallucinating when it doesn't have the answer stored and things like this. Like it's just a flawed methodology, essentially. So yeah, if I can maybe interrupt you, right? Because remember when you first came to me with the idea, I was telling you, okay, are we not actually cherry picking, right? Our own index, our own benchmark, right? And your answer that was pretty clear that it's clearly not the objective, no? Yeah, no, no, because that's kind of what has been going on, right? And that's why I want to open source this. I want to let people, hopefully we can attract other people from the AI community to contribute to this. And the goal is unbiased objective metric scoring for the memory, just the memory portion, whatever that may be. So I think that part of the problem is a lot of people approach the memory solution differently. So we have vector systems and things like this that are attached as basically just like an extra data source. But then you have simple, which is local, the user owns their files, those are data. They only use a request through our API, they only send requests through our APIs to query the logic and the schema to understand, okay, we're recall recalling or we don't ever read the data. But all we know is how large the data set is. That's all we know. Just a moment. Yeah, Tio, did you want to add in any talk? No, just as I was saying, right, because we had some discussions about it. As you mentioned at the beginning, I think we had some people on the Discord, which were kind of challenging. Why do we run this benchmark and not that one? Because in the end, there's a different type of benchmarks, right? You have the B1 for example, which is I think a bit more comprehensive, which is a bit more difficult to score. But over all, right, I mean, it's not like that we're cherry picking anything. That's why two lives basically came up with the idea and said, okay, I mean, there's all these benchmarks, kind of companies which are running these, they're just trying to look good, what about setting something up, which is a little bit more objective. But it's kind of funny, right? We're going into the topic because I think we're starting to talk about the BAP to revenues. And I'm going to be sure it must be very interested, also in that. And now we talk about some technicalities around the benchmark. But I think it's interesting. I mean, if you like AI, if you like the topics, it's cool. And we're going to come back to the revenues later, right? So we'll touch that point that I want to spoil anything. Well, yeah, don't think the revenue is being forgotten about. I mean, I'm simultaneous. I'm simultaneously building out the product page, right? So I'm taking my time with this because I want it to be very good. I want it to be as good as we can make this. I want to take the time to get the right videos that we need created and things like this. So yeah, I'm working on that simultaneously while pushing this forward. And I want you guys to know that this is part of the foundational work we're doing, right? The hackathon has brought our company in front of many builders. We had lots of signups, lots of submissions, lots of videos, it generated lots of buzz around what we're doing and how impactful the memory can be. This is good, right? People know, civil now, they know what it can do. It actually works. Now when we go to get revenue, people have a little, they can see some stuff. When they go to click buy, they go to search us, they see, oh, look at all these people, all this cool shit they're building, you know? We're planting seeds for tomorrow with everything we're doing, because we actually are one of the few companies here who are planning for the future, who we want to be here in two years, three years, four years. As long as this thing is generating revenue, we plan to be here, right? But the goal is to get there first. So I would say that the benchmark is another one of these hackathon type plays, okay? Where you need to be able to see the type of exposure this can get us, the type of respect this can get us within the builder communities, developer communities and things like this. So that's the goal here. We want builders and developers to respect this because when builders and developers respect us, then they start using our products and their companies. And when they start using their products and our, or our products and their companies, we start to make more money. So these are things we want to attack the smart people first and show them that we are smart people with smart products that are actually useful for their businesses, for their own workflows, et cetera. So when we do this, right, this is a much more valuable piece of marketing in my opinion, because this is essentially laying the brick foundation, that we build our revenue on. So just kind of given you guys the logic and the reasoning behind that decision or this decision. So I do believe while it may seem benign to most people, there is potential that we have thousands of AI developers contributing to this open source repository. And discussing it. And then guess what happens? They see that Cible is performing really well on this new benchmark. Yeah, we contributed to it and we started it, but it's not, it is an unbiased objective test of the memory, right? So that's really the goal is to get as many people involved helping with it as possible so that it is truly an open source non, you know, unbiased benchmark that can be used throughout the industry. So thoughts on that, Jayway? I was looking for the unbeat button. No, no, that makes all sense. So I'm really looking forward how that goes. And yeah, I mean, exciting times are doing a lot of things. So let's see how that goes. And that's not even all of what's happening. I would say the most, I mean, the thing that people will be most excited to hear about, but likely be the on-chain stuff, I would think. So we've started doing some testing. Let's just say that in regards to getting our MCP skill bank marketplace to be accessible through these NFTs. So we're working on private IPFS right now and making sure that we can have multiple layers to the metadata or some is private and only access through like a...