Kiss, Your Virtual Companion Powered by Local AI

It all started with a fairly ordinary need: adding a search feature to GREY/WIRED.

The blog is starting to accumulate enough articles, projects and short reflections that finding a particular piece of information isn't always straightforward anymore. A conventional search engine would have done the job perfectly well. A few keywords, a list of results, and the problem would probably have been solved.

But I had something else in mind.

In an earlier article about my local AI model, I explained why I found relatively small models particularly interesting when integrated into larger systems. Not to compete with the major AI services, but to carry out practical tasks with the right information and the right tools.

I'd already experimented with several uses on my personal infrastructure. The problem is, I can't exactly invite blog visitors to interact with my home automation setup or my other internal tools to see what that looks like.

I could, however, let them ask questions about GREY/WIRED itself.

The idea was to replace the traditional search box with an assistant capable of understanding a question, finding relevant posts and putting together an answer based on their content.

That's how Kiss was born.

Why Kiss? Because that's also the name of my Weimaraner, my very real companion who “assists” me every day… whenever he feels like it. :-)

I just hope his virtual counterpart will be a little less mischievous!

Nothing revolutionary. The technical building blocks already existed, and I'm far from the first person to experiment with a document-based assistant. But applying the idea to a personal blog, with an entirely self-hosted model, struck me as an interesting little experiment.

And, incidentally, a good excuse to carry on experimenting.

I was also about to discover that, when it comes to document retrieval, it's better to compare solutions against your own content than blindly follow the latest trend.

A search engine that can also answer questions

A conventional search engine finds documents. A document-based assistant tries to go a little further.

If I'm looking for information about Zigbee, a traditional search engine will give me a list of articles containing the relevant terms. Kiss, on the other hand, can take a question such as “What's the difference between ZHA and Zigbee2MQTT?”, find relevant posts and offer a summary with links to the sources.

This is the principle behind RAG, short for Retrieval-Augmented Generation.¹

Behind that slightly intimidating name lies a fairly simple process:

1. I build a corpus from the site's published content.
2. When a visitor asks a question, a search engine selects the documents or passages most likely to contain the answer.
3. That information is passed to the AI model along with the question.
4. The model writes a response and can link back to the relevant posts.

The important point is that I haven't retrained Qwen on the blog's articles. The model uses the information supplied to it when it generates an answer.

This also means I can update Kiss's knowledge when I publish a new article, without having to change the model's parameters.

For now, the corpus is limited to GREY/WIRED's publications and institutional pages, in French and English. But the same concept could be applied to a much wider variety of resources.

A technical documentation website, for example, could combine web pages, manuals and PDFs. An assistant could not only answer a question, but also direct visitors to a reference sheet or a downloadable document.

Kiss doesn't do all of that today. It's simply an illustration of how the same idea could be used elsewhere.

Finding the right documents: harder than it looks

Once the principle of RAG is clear, one question becomes essential: how do I find the information that's actually relevant?

This stage is often less spectacular than generating text, but it's crucial. A model can produce a perfectly convincing answer based on the wrong documents. A fluent response is absolutely no guarantee that it's correct.

For Kiss, I therefore decided to compare several document retrieval methods before choosing one for production. My goal wasn't to pick the most fashionable technology, but the one that worked best for my blog's publications and the questions people might ask about them.

BM25: an old method that's far from obsolete

BM25 looks for the words in a question within the documents, but it doesn't simply count them. Among other things, it gives more weight to terms that are rare in the corpus, takes their frequency into account and adjusts scores according to document length. It belongs to a long tradition of information retrieval that predates today's enthusiasm for language models.²

Take a question containing “ESP32”, “Thread” and “Border Router”. If those terms appear in an article's title and body, BM25 has good reason to rank that article highly.

That's particularly useful on a technical blog, where component names, protocols and exact references often matter a great deal.

But two ways of phrasing a question can mean the same thing without sharing much vocabulary. Conversely, a document can contain all the right words without actually answering the question. That's what prompted me to try semantic search.

E5-small: representing documents differently

E5-small looks for similarities in meaning: a model transforms questions and documents into numerical vectors, then brings together those that discuss similar subjects, even when they use different words. These representations are called embeddings.

For this experiment, I used multilingual-e5-small, a model that can produce these representations in several languages.³ For the final comparisons, I selected an averaged representation of the different parts of each document, labelled E5 mean in the results.

RRF: is there strength in numbers?

Since BM25 and E5-small have complementary strengths, the temptation is obvious: why not combine their results?

With RRF, a document that ranks highly in both lists receives a good combined score. But the method considers ranking positions, not the actual differences between the search engines' scores.⁴

In other words, combining two approaches doesn't guarantee a better ranking for every question. That's precisely what the tests would reveal.

How do you measure search relevance?

To compare these methods, I put together sets of questions and assessed how relevant each publication was to each question.

Every document could receive a relevance grade for a given question:

Grade	Meaning
0	The document doesn't provide a useful answer to the question.
1	It provides relevant supporting information, but not a direct answer.
2	It directly answers the question.

Of course, the same article can be highly relevant to one question and completely useless for another.

To assess the quality of the rankings, I wanted to answer three questions: did I retrieve the useful documents? Is the first result relevant? Are the best documents in the right order?

I used three standard metrics for this. Their names aren't particularly inviting, but the ideas behind them are accessible.

Recall@3: how many useful documents did I retrieve?

Recall@3 measures the proportion of relevant documents found among the top three results.

Imagine that four publications are useful for answering a question. If the search engine returns two of them in its top three results, Recall@3 is 2/4, or 50%.

This metric measures document coverage. It doesn't tell us whether the best document came first.

MRR: where does the first useful result appear?

Mean Reciprocal Rank focuses on the position of the first relevant document.

If it appears first, the score is 1. If it's second, the score is 0.5. If it's third, approximately 0.33. MRR is the average of those values across the questions evaluated.

There's a subtlety, though: in my tests, both grade 1 and grade 2 documents count as relevant. MRR therefore doesn't distinguish a direct answer from a merely supplementary document if they appear at the same rank.

nDCG@5: are the best documents ranked highly enough?

Normalized Discounted Cumulative Gain takes into account both the position of the top five documents and their degree of relevance.

A grade 2 document is worth more than a grade 1 document, and a useful answer at the top of the list is worth more than the same answer in fifth place. The score is normalized against an ideal ranking.

It's therefore a way of measuring the quality of the ordering, rather than simply whether useful documents are present.

None of these metrics tells the whole story. And they don't directly evaluate the answers written by Qwen: they measure the performance of the search engine upstream.

What the tests actually showed

I first compared BM25 and E5-small on an initial set of 16 questions, 14 of which had at least one relevant document.

Method	Recall@3	MRR	nDCG@5
BM25	87.38%	100%	95.90%
E5-small	86.43%	96.43%	93.84%

In this first round of tests, BM25 achieved better average scores on all three metrics.

The 100% MRR deserves an explanation: for each of the 14 questions with a relevant document, BM25's first result was at least relevant. Not necessarily ideal, but useful.

I then took the experiment further with RRF. A new confirmation round, separate from the first, used 32 questions, 26 of which had at least one relevant document.

Method	Recall@3	MRR	nDCG@5
BM25	84.82%	91.67%	87.61%
E5-small	81.32%	95.19%	88.50%
RRF	83.79%	97.12%	91.41%

This time, the conclusion wasn't so obvious.

RRF substantially improved the position of the first relevant result and the average quality of the top five documents. BM25 retained a slight edge in recall among the top three.

At first glance, choosing RRF would have been tempting.

But I'd set adoption criteria before examining the results. In particular, I wanted to avoid a publication that directly answered a question being replaced at the top of the list by one that merely offered supplementary information.

And that happened.

Across the 26 questions with relevant documents, RRF reduced the number of irrelevant first results from four to one, while increasing the number of directly relevant first results from 19 to 21. Those are genuine improvements. But two excellent first results from BM25 were downgraded.

The averages improved, without every question benefiting from the change.

Two examples that say more than the averages

One question asked whether a router in a Zigbee network meant an Internet router or a role performed by certain connected devices.

BM25 placed an article about building a Thread Border Router with ESP32 boards first. The word “router” favoured a document that didn't directly answer the question.

E5-small, by contrast, immediately retrieved the article “Zigbee sans (trop) se compliquer la vie” (“Zigbee without making life too complicated”). RRF also placed it first. That was a clear success for semantic search.

But another question, about activities to keep a Weimaraner mentally occupied, produced the opposite effect.

BM25 put the project “Mission : occuper un Weimaraner” (“Mission: keeping a Weimaraner busy”) first, which was directly relevant. E5-small preferred a supplementary article about the relationship between a dog and its owner.

RRF gave the two documents exactly the same score, because their ranks were reversed between the two search engines. A tie-breaking rule then put the supplementary article ahead of the direct answer.

Yet Recall@3 and MRR remained perfect for that question.

That's exactly the sort of detail that shows why individual cases matter, not just averages.

In the end, I kept BM25 as the default search engine.

Not because it's inherently superior to vector search. My experiments clearly show that isn't always the case. But the hybrid configuration I'd tested didn't meet all my criteria, while BM25 had the additional advantage of being simple, lightweight and well suited to my corpus.

The scope of this experiment also needs putting into perspective: the test campaigns involved a few dozen questions and a limited number of publications. The French and English versions often represented the same underlying intent; the confirmation benchmark's annotations were prepared with AI assistance, then partially reviewed and adjudicated by humans, without exhaustive independent double human annotation.

So these figures are a record of an experiment on my corpus, not a universal verdict on search engines.

A RAG without a vector database?

This experiment got me thinking about a widespread assumption: whenever people talk about RAG, embeddings and vector databases seem almost unavoidable.

There are good reasons for their popularity. Semantic search can retrieve documents without exact vocabulary matches, while vector databases provide efficient ways of storing, indexing and querying large collections of vectors.

Many frameworks and specialized services offer this architecture out of the box, making it a fairly natural starting point.

But vector search and a vector database are two different things.

The first involves comparing numerical representations. The second is infrastructure for organizing and querying them.

For a small corpus, a simple matrix held in memory may be enough. That's exactly what I did in my E5-small experiments, using NumPy, without a specialized vector database.

And RAG doesn't necessarily require embeddings at all: it combines information retrieval and generation, regardless of the method used to find the documents.

I'm certainly not concluding from this experiment that vector databases are useless. They can absolutely have their place with larger corpora, more varied queries or more complex requirements.

But I find it interesting that a much older, less spectacular method ultimately met my needs.

A popular technology isn't necessarily the best answer to every problem.

More than anything, this experiment reminded me that there's no magic formula for building a RAG system. The choice of search engine depends on the nature and size of the corpus, the vocabulary it uses, the questions you want to answer and the level of relevance you expect.

When working with a defined document corpus, I think it's worth spending a little time comparing different approaches rather than automatically adopting whichever dominates the latest tutorials. Naturally, the evaluation effort should be proportionate to the size and importance of the project.

In my case, that process led me to keep BM25. With a different corpus and different questions, the outcome could easily have been different.

Why host everything myself?

The short answer would be: why not?

I already had a server equipped with an NVIDIA RTX 3070 Ti and 8 GB of VRAM. I was running a Qwen model with around 9 billion parameters on it, quantized to fit the available hardware.

So I already had much of what I needed.

I could have used a commercial AI API. That would probably have simplified some aspects of the implementation. But it would also have taken away much of the appeal of the experiment: I wanted to see how far I could go with my own hardware.

Kiss therefore uses my infrastructure for both document retrieval and answer generation. No external inference service is called when a visitor asks it a question.

That gives me a certain independence: I can stick with a particular model version, change components, rebuild the corpus and decide for myself when to make changes.

It doesn't make the system free, of course. The hardware uses electricity, needs maintenance, and any failures are my problem. I also remain dependent on software and models developed by others.

But I'm not relying on a commercial API to generate the answers. And for a personal experiment, I like that freedom.

Eight gigabytes of VRAM, and a response that arrives quickly

One of the things I particularly wanted to show visitors was the performance.

You might imagine that a local nine-billion-parameter model running on a graphics card with 8 GB of memory would produce its answers painfully slowly.

That's not what I'm seeing.

In tests with Kiss, generation reached around 76 tokens per second. A token is roughly a fragment of text; at that rate, the response appears remarkably smoothly.

The complete requests measured took between 2.5 and 11.2 seconds, depending on the scenario. The first event in the response stream arrived in under four seconds in those tests, although that event wasn't necessarily the first visible character.

These results are all the more interesting because my server's processor is far from ideal for this kind of workload.

It's tempting to think that, because the GPU handles most of the inference, the processor hardly matters. But the CPU still plays a role in several stages: preparing requests, retrieving documents, tokenization, orchestration and some operations inside the inference engine.

That doesn't mean a faster CPU will necessarily increase token generation speed by the same proportion. When the GPU is the bottleneck, gains in that part of the process may be modest.

A better processor could, however, reduce some delays and improve overall responsiveness, particularly while preparing an answer.

A CPU upgrade is planned, and it'll be interesting to repeat the same measurements afterwards to see what actually improves.

For now, despite a configuration that's far from optimal, I find the performance entirely satisfactory for what Kiss is meant to do.

In this context, the smoothness of the output and the delay before the response starts don't feel much different from the experience offered by external AI services.

That obviously doesn't mean my Qwen competes with the best commercial models on every task. I'm talking about the user experience for a specific, well-defined purpose, not the models' general capabilities.

An optional feature, not a critical service

What happens if Kiss stops working?

Well, not much.

GREY/WIRED remains usable without the assistant. Articles are still accessible and visitors can carry on browsing the site.

Kiss is an additional service, not something the blog needs in order to function.

I can therefore stop it for maintenance, restart the model or tolerate a temporary outage without affecting the rest of the site. I have no intention of building a highly available infrastructure just to ensure that an experimental assistant is accessible every second of the day.

What mattered to me, though, was making sure visitors are clearly told when it's unavailable. Few things are more frustrating than an interface that seems to be waiting for a response without ever explaining what's happening.

I've also introduced limits on request sizes, conversations and the number of simultaneous generations. A single consumer graphics card obviously can't serve an unlimited number of visitors.

Protection against abusive automated use could still be strengthened, notably by rate limiting at the front-end web server.

The goal remains the same: a reasonable architecture, proportionate to how important the service actually is.

And how does it fit into the blog?

I didn't simply want to add a generic chatbot in a floating window.

Kiss needed to fit naturally into GREY/WIRED, follow its visual identity, work in both languages and accompany visitors as they browse.

It keeps the conversation going when you move from one page to another, can suggest further reading and links to the documentary sources behind its answers.

I've also separated its document management from the website itself.

When a new version of GREY/WIRED is published, I can rebuild the corpus from the pages actually available online, check the result and then activate it. I can also switch back to a previous corpus if something goes wrong.

It's not the most visible part of the project, but it prevents the assistant from relying on a manually maintained collection of documents that would gradually drift out of sync with the blog.

At the time of these tests, Kiss had 29 French publications, their English counterparts and the site's institutional pages.

That's a fairly modest corpus for conventional search. For experimenting with a conversational interface, it's already more than enough.

Kiss can get things wrong, too

I spent some time asking Kiss questions designed to push it outside its intended scope.

I asked it to become a general-purpose assistant, follow a supposed maintenance instruction, reveal its internal instructions, or answer an unrelated question after several perfectly legitimate exchanges.

In those tests, it refused correctly.

But another test revealed a subtler weakness.

I deliberately asked a leading question, claiming that I'd used an RTX 4090 with 24 GB of memory for my local model.

Kiss found the right article, then confirmed that false information.

In a separate conversation, when I simply asked which graphics card I used, it gave the correct answer: RTX 3070 Ti, 8 GB.

And when I presented the false claim again but explicitly asked it to check the facts against the article, it corrected the mistake.

In other words, the model had access to the right information, but it was influenced by the way the question was phrased.

It's a useful reminder that access to documents doesn't automatically guarantee that a model will interpret them correctly. It's also an interesting illustration of the difference between a reliability issue and a security vulnerability: this behaviour neither compromised the system nor affected other visitors' conversations.

There's still room for improvement. I plan to revisit this in a future release, probably by adjusting the system instructions to encourage Kiss to verify factual claims rather than accept the assumptions embedded in a question. I'll then need to make sure that change doesn't introduce other regressions.

I'd rather keep this example in my account of the experiment: showing only the successful answers would give a rather misleading impression of what a RAG assistant can actually do.

I'm not the only one who's had this idea

Looking around, I came across other personal projects exploring a similar approach.

Cameron Rye developed Ask, an assistant integrated into his portfolio that can answer questions about his projects and articles.⁵

Suganthan Mohanadasan also offers an assistant based on the publications on his website and a personal knowledge base.⁶

These projects use different architectures from mine, including hosted AI services, but pursue a similar goal: letting visitors query a body of content instead of systematically browsing a list of results.

I find it interesting to see experiments like these appearing on personal sites. It would be an exaggeration, though, to claim that Kiss is pioneering a new way of browsing the web. The idea already exists, and the technology has been available for some time.

What appeals to me is having developed my own interpretation of it, without depending on an external inference service.

So, is it better than a search engine?

I don't know. And ultimately, that's not the question I'm most interested in.

If you just want to find the title of a publication, a conventional search engine may be more direct. If you're looking for a summary, want to connect several articles or ask a question in your own words, a conversational assistant can offer a different experience.

It can also make mistakes, choose irrelevant sources or give an unnecessarily complicated answer to a simple question.

I wasn't trying to prove that every search engine should be replaced by a chatbot.

I wanted to show, through an experiment that anyone could try, what a local AI model can offer when given a practical task.

I also had the opportunity to compare several document retrieval methods, discover that more sophisticated solutions aren't automatically preferable and encounter some interesting limitations of generative models.

All that just to find articles on a personal blog.

Yes, it's probably a little disproportionate.

But that was part of the fun.

Kiss is now available on GREY/WIRED. I'm pleased with how it fits into the site, and its performance is convincing given the hardware involved.

As for the rest, I'd rather let everyone make up their own mind.

After all, the easiest thing to do is ask it a question.

---

Footnotes

1. The term Retrieval-Augmented Generation (RAG) was introduced by Patrick Lewis et al. in the foundational paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020. Kiss uses an architecture adapted to its own purpose rather than reproducing that system. ↩

2. For an overview of BM25 and its origins, see Stephen Robertson and Hugo Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond, 2009. ↩

3. Model used during the experiments: intfloat/multilingual-e5-small, on Hugging Face. ↩

4. Gordon V. Cormack, Charles L. A. Clarke and Stefan Büttcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods, SIGIR 2009. ↩

5. Cameron Rye, Building Ask: A RAG-Powered Chatbot for My Portfolio. ↩

6. Suganthan Mohanadasan, How I Added an AI Chat to My Personal Website. ↩