Semantic search is not the silver bullet - even more, you can’t treat your embedding model as the silver bullet.Depending on the use case, full-text search can be simple, reliable and efficient. Here’s the flow for semantic search:question - model -> embedding - vector search -> relevant docsAnd here’s what it looks like without semantic search:question - tokenizer or model -> extracted terms/ property filters - full-text search -> relevant docsBoth work, and I am not convinced one is universally better than the other.Fact Check: Semantic Search Is Better Than Full-Text SearchTotally wrong. You can use semantic or full-text search, or even run some regular filters to get those documents. The core of this chatbot is very simple - for each user question, it searches a large number of documents, finds relevant information and feeds the documents to OpenAI’s Chat Completions API. In other words, it doesn’t matter too much how you search the data. In fact, we did this with our very own chatbot, SQrL, which utilizes both semantic and full-text search. This indeed is orthogonal to full-text versus semantic search. It focuses on understanding the intent and meaning behind the query, rather than relying solely on specific keywords.Fact Check: ChatGPT-Style Chatbots with Domain-Specific Knowledge Require Semantic SearchWrong. Full-text search typically treats each keyword independently, without considering the relationships between them or the overall context.Semantic search, however, takes into account the context, synonyms, related terms and overall meaning of the query to retrieve more relevant results. On the other hand, semantic search aims to understand the users’ intent behind queries by analyzing its semantics, context and relationships between words or concepts. It retrieves documents or web pages that contain the exact keywords specified in the query. I wanted to share my insights, and fact-check some widely held beliefs about semantic and full-text search.First, Some Basics…In full-text search, the emphasis is primarily on keyword matching. As a query engine engineer at SingleStore, I was fortunate enough to have both developed SingleStore’s full-text search engine, and initiated the vector/semantic search effort.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |