Users cannot find what they are looking for: why site search returns the wrong things

Site search usually works in the technical sense and fails in every other sense, and the cause is almost always one of three things. The query goes to the database as LIKE '%term%', so there is no ranking at all. What the user typed does not match the catalogue text character for character (a typo, a plural, a missing accent) and the result set comes back empty. And nobody reads the search log, so nobody knows which queries return nothing. The first two are usually fixable without replacing your search engine. Skip the third and the engine you pay for will return the same bad results, faster.

Your first move is logging queries, not comparing engines

Most teams start improving search by drawing up a vendor comparison. The right starting point is what your own users type. Record five fields on every search: the query text, how many results came back, the position of the result the user clicked (and the fact that they clicked nothing, when that happens), which filters were applied, and what the session did next.

Four metrics fall out of that. Zero-result rate is the famous one and also the most misleading, because a search that returns forty irrelevant products counts as a success. No-click rate is a better signal. Search exit rate is the harshest one. The fourth is comparing conversion for sessions that used search against sessions that did not. On a healthy site, people who search convert better because their intent is explicit. If yours converts worse, the search box is costing you money.

Query distributions are long-tailed almost everywhere: a few hundred queries carry most of the volume and thousands of distinct queries make up the tail. That is good news, because reading the top 200 queries by hand is an afternoon of work with an outsized payoff. Baymard Institute's ecommerce search benchmark puts 56 percent of the sites they review at "mediocre or worse" search UX, so this is still an area where doing it properly separates you from the field.

People do not all search the same way

The most common design mistake is assuming users type part of a product name and expect that product. Baymard's research on query types identifies eight distinct behaviours, and the share of sites that handle each one badly varies enormously:

• Exact searches (product names, model numbers, text pasted from elsewhere): 12 percent of sites have issues.
• Product type searches ("sandals", "laptops"): 20 percent.
• Feature searches ("leather jacket", "5 litre bucket"): 39 percent.
• Symptom searches ("sore throat", "stained carpet"): 37 percent.
• Use case searches ("wedding gift", "gaming laptop"): 43 percent.
• Compatibility searches ("Dell laptop charger", "iPhone 15 case"): 44 percent.
• Abbreviation and symbol searches ("13in laptop", "sleeping bag -5 degrees"): 54 percent.
• Non-product searches ("return policy", "when will it ship"): 66 percent.

That last line is the one teams forget. If your search index contains only products, someone typing "returns" gets an empty results page while the page they wanted sits on your site. Index your content pages, FAQ entries and blog posts too, and show them as a separate block in the results. Compatibility and abbreviation queries are a data problem instead: if the catalogue says "13.3 inch", you have to teach the index that 13", 13 inch and 13in mean the same thing.

Why LIKE '%term%' hits a wall

The leading wildcard makes that query useless to an ordinary B-tree index, so cost grows linearly with the table. Worse, the result is boolean. A row either matches or it does not, there is no relevance score to sort by, and results end up ordered by name or by id. Substring semantics also bite from unexpected angles: searching for "art" surfaces cartridges, quarts and smartphones.

Postgres offers a partial remedy. A GIN index using gin_trgm_ops from the pg_trgm extension has been able to accelerate LIKE and ILIKE since PostgreSQL 9.1, and regular-expression matches since 9.3. So the scan gets faster. Ranking, word endings and synonyms are still entirely absent. We covered what actually makes queries slow and how index choices play out in the database bottlenecks article.

How far Postgres full-text search gets you

Full-text search is built into Postgres and it is more than enough for most internal tools, admin panels and mid-sized catalogues. The setup pattern is short: a stored generated tsvector column (PostgreSQL 12 and later) combining the fields you want searchable, with a GIN index on top. Because a generated column expression has to be immutable, you must call to_tsvector in its two-argument form with the text search configuration spelled out.

Turn user input into a query with websearch_to_tsquery. The documentation is explicit that this function never raises syntax errors, which means you can hand it raw user text: quoted text becomes a phrase search, the word OR becomes a disjunction, and a leading dash becomes negation. Building a to_tsquery string by concatenation instead gives you an endpoint that throws on the first stray parenthesis a user types.

The limits are documented and worth knowing before you commit. ts_rank uses no global information, so it cannot weight rare terms higher and its scores cannot be fairly normalised to a percentage. Ranking requires reading the tsvector of every matching row, which makes it I/O bound and slow on broad result sets. There is no typo tolerance. Synonym dictionaries, facet counts and autocomplete suggestions are all your own code.

Typo tolerance comes from two different places

If you are staying in Postgres, pg_trgm works well as a second layer. The similarity threshold defaults to 0.3, the % operator tests against that threshold, <% compares a word against a longer string, and <-> returns a distance so you can use it in ORDER BY. The practical pattern is to run full-text search first and fall back to a trigram query when it returns nothing. That is how somebody typing "thermostast" still finds the thermostat.

Dedicated engines give you this as default behaviour. Meilisearch tolerates one typo in terms of five characters or more and two typos from nine characters up. Typesense sets num_typos to 2 by default and caps it there, because the cost of typo tolerance grows quickly with the number of edits allowed.

The trap is assuming more tolerance is always better. On model codes and part numbers it actively causes harm: "A15" and "A16" differ by one character, and the customer orders the wrong spare. Both engines let you configure this per field. Turn tolerance off on identifier fields.

Language-specific pitfalls, with Turkish as the worked example

Anything beyond plain English text has traps that only show up in production. Turkish is a good illustration because it hits three of them at once, and the same three appear in German, Polish, Arabic and Finnish in different combinations.

Case folding is locale dependent. In Turkish, the lowercase of I is ı (dotless) and the lowercase of İ is i. Under English rules you get i in both cases. If your database or index lowercases with the wrong locale, "IŞIK" is indexed as "işik" and a user searching "ışık" finds nothing. Elasticsearch's built-in turkish analyzer includes a turkish_lowercase filter for exactly this reason. In Postgres the behaviour depends on your database collation, so test it with one query rather than assuming.

Users skip the diacritics. People type "ogrenci" instead of "öğrenci", "resume" instead of "résumé". The fix is unaccent or ICU normalisation, and the part that matters is applying the identical normalisation at index time and at query time. Do it on one side only and you make things worse than before.

Morphology. Turkish is agglutinative, so "arabalarımızdan" is a single word carrying four suffixes. Postgres ships a Turkish Snowball stemmer among its default dictionaries and Elasticsearch has a Turkish analyzer, but stemmers are blunt instruments and will occasionally collapse unrelated words to the same root. Protect brand and product names from stemming with a keyword marker; that is what the turkish_keywords filter in Elasticsearch's Turkish analyzer is for.

Before touching any of this, pull 100 real queries from your log and write down the top three results you expect for each. Run that set after every configuration change. Otherwise you find out that fixing one query broke three others when a customer tells you. If you run one catalogue across several languages, the multilingual architecture article covers how the index and the URL structure should line up.

Relevance is not a setting, it is a list of decisions

"Improve relevance" is not a work item. Write the ranking down as explicit rules. An exact match on a model number or SKU always comes first. Then the text score. Then business rules: in stock ahead of backordered, promoted items up, discontinued down. Popularity comes last and with a bounded weight, because the moment global popularity starts outranking a query-specific exact match, your search has become a worse version of your category page.

The synonym list is content, not code, and the product team should own it. "Notebook" and "laptop", common brand misspellings, unit variants, industry jargon against consumer wording. That list grows out of the query log and it is never finished. The zero-result query list is the single most valuable document you can hand the product team: demand exists, supply does not.

What search quietly leaks

A search index is usually a copy of your main database, and authorisation rules do not travel with the copy. The classic accident is drafts, unpublished products, customer-specific price lists or another tenant's records sitting in the index, visible to anyone who types the right word. Filter in the query itself rather than in the UI: write tenant and permission fields into the index and attach them as a mandatory filter on every search. How to model those permissions in the first place is covered in the authorisation article.

Building autocomplete suggestions from user queries opens a second door. People type order numbers, phone numbers and email addresses into search boxes. Generate suggestions only from cleaned, reviewed queries above a frequency threshold, never from the raw stream.

When to move to a dedicated engine

Staying in Postgres makes sense when the catalogue is under a few hundred thousand records, search is not the primary navigation path, you do not need facet counts or instant as-you-type results, and the team does not want another system to operate. Moving out makes sense when search is the main revenue path, the catalogue is multilingual, ranking rules change weekly, or you are dealing with millions of records or high concurrent query volume.

The real cost of that decision is not the engine, it is index synchronisation. There are three approaches. Dual writes are easiest and will drift eventually. Periodic full reindexing is safe and leaves data stale. Change data capture or an outbox table is the correct answer and the most work. Whichever you pick, set a staleness budget ("a price change appears in search within 60 seconds") and measure against it. The ways synchronisation typically breaks are in the integration data sync article, and wiring the budget to an alert belongs with your SLOs and dashboards. Budget too for memory, backups, version upgrades and a reindex window during deployments.

Where hybrid search helps and where it hurts

Semantic search earns its keep when the user's words do not overlap with the catalogue's words: symptom queries, use case queries, questions about policies. Keyword search is unambiguously better on model numbers and exact matches. Hybrid search runs both and fuses the result lists, usually with reciprocal rank fusion. pgvector is an option if you want to stay in Postgres, and both Meilisearch and Typesense support vector search now.

The cost gets underestimated. You take on an embedding pipeline, re-embedding when content changes, pinning a model version, and extra latency per query. Semantic search also has a characteristic failure mode: it confidently promotes a result that looks related and is wrong, offering a 40 watt bulb to someone who asked for 60 watt. Keep one rule in place: when there is an exact identifier match, no semantic result outranks it. Being visible inside AI answer engines is a separate problem with separate mechanics, and we covered that in the AEO and GEO article.

A first step that fits in a week

If you are not logging searches, that is the only thing to do this week. Query text, result count, clicked position. After two weeks of data, produce two lists: the 200 most frequent queries and every query that returned nothing. Reading the second list makes the priority order obvious, and in most teams the top of it looks the same: case and accent normalisation, non-product content missing from the index, and a zero-result page that is a dead end. Those are day-long fixes that require no new engine.

Put the engine migration at the bottom of the list. Without a 100-query regression set and two weeks of measurement you cannot demonstrate that the new engine is better, only that it costs more. If you want help reading your own search log and turning it into a concrete order of work, we can start from the data you already have.