MIS 752 · Lab 5 · LienJ

Speak the Patient's Language: semantic search vs keyword search

A 20-sentence clinical corpus embedded with all-MiniLM-L6-v2, the full cosine similarity matrix, and a scored comparison of semantic search against a keyword baseline on a written gold standard. Runs locally on a free CPU: no API key, no PHI, $0.

The 30-second read

The Model

  • Encoder: sentence-transformers/all-MiniLM-L6-v2
  • Dimensions: 384 numbers per sentence
  • Corpus: 20 sentences, 4 topics, 2 voices
  • Embed time: 713 ms for the whole corpus
  • API cost: $0.00 (runs locally, no key, no network after load)

Does it capture meaning?

  • Mean cosine within a topic: 0.235
  • Mean cosine across topics: 0.129
  • Separation: +0.106

Retrieval vs The Gold Standard (12 queries, top-5)

  • keyword (word overlap): precision 0.38 · recall 0.28 · MRR 0.50
  • semantic (embeddings): precision 0.58 · recall 0.51 · MRR 0.92

Overall: semantic beats keyword on MRR by +0.42

The Finding - MRR split by whether the patient guessed our words

  • patient guessed our words: keyword 0.86 · semantic 1.00
  • patient's own words: keyword 0.00 · semantic 0.80

Example Query: "My blood pressure spikes when I eat candy"

  • #1. [dia1 | diabetes | patient] cosine 0.505
    My sugar goes up after every meal even though I take the pills.
  • #2. [car5 | cardiac | patient] cosine 0.405
    The home blood pressure cuff keeps giving me a high number.
  • #3. [car4 | cardiac | patient] cosine 0.344
    I get out of breath and my heart pounds for no reason at all.
  • #4. [dia5 | diabetes | patient] cosine 0.335
    My feet tingle and sometimes go completely numb.
  • #5. [car1 | cardiac | patient] cosine 0.275
    My chest feels heavy and tight when I walk up the stairs.

Do embeddings capture meaning?

The 20×20 cosine similarity matrix, rows and columns sorted by topic and boxed into the four topic blocks. Each label shows the sentence id and its voice (pat = patient, cli = clinical). The diagonal is blanked and the colour scale spans the off-diagonal range: every sentence scores 1.00 against itself, and leaving that in flattens everything else into one shade. A dark cell spanning both voices is the whole point of the week: same meaning, no shared words. A pale row inside a box is a sentence this model cannot place.

Meaning beats spelling only where spelling runs out

Scored against the written-down gold standard, not by eyeball. MRR asks how high the first right answer was, and it is the metric a patient on a phone actually feels: 1.0 means rank one. The two groups differ by whether the patient happened to guess the corpus's own vocabulary — a split computed from the text, not labelled by hand. In the left group keyword search is fine, because there are words to match. In the right group there are none and it collapses. Semantic search never needed the words to match in the first place.

Per-query detail

Where each method wins and loses, query by query. The keyword baseline is deliberately naive word overlap, so this is not the strongest possible keyword system — BM25 would do better, and a reranker better still. It is enough to show that the vocabulary gap, not the algorithm, is what defeats word matching.

query_type query method retrieved p_at_k recall_at_k rr first_rank
patient guessed our words chest hurts when i climb stairs keyword (word overlap) car1,car2 1.0 0.667 1.0 1.0
patient guessed our words chest hurts when i climb stairs semantic (embeddings) car1,car2 1.0 0.667 1.0 1.0
patient guessed our words my refill is not ready at the pharmacy keyword (word overlap) med5 1.0 0.500 1.0 1.0
patient guessed our words my refill is not ready at the pharmacy semantic (embeddings) med5,dia4 0.5 0.500 1.0 1.0
patient guessed our words is it safe to take ibuprofen with warfarin keyword (word overlap) med1,dia1 0.5 0.500 1.0 1.0
patient guessed our words is it safe to take ibuprofen with warfarin semantic (embeddings) med1,med2 1.0 1.000 1.0 1.0
patient guessed our words i feel low and cannot sleep keyword (word overlap) men1,men3 0.5 0.333 1.0 1.0
patient guessed our words i feel low and cannot sleep semantic (embeddings) men1,dia3 0.5 0.333 1.0 1.0
patient's own words it aches to climb to the second floor keyword (word overlap) (nothing) 0.0 0.000 0.0 NaN
patient's own words it aches to climb to the second floor semantic (embeddings) car1,dia5 0.5 0.333 1.0 1.0
patient's own words the reading on my meter jumped since supper keyword (word overlap) (nothing) 0.0 0.000 0.0 NaN
patient's own words the reading on my meter jumped since supper semantic (embeddings) car5,dia5 0.0 0.000 0.0 NaN
patient guessed our words would the anti-inflammatory clash with what thins my blood keyword (word overlap) car5,med1 0.5 0.500 0.5 2.0
patient guessed our words would the anti-inflammatory clash with what thins my blood semantic (embeddings) med1,med2 1.0 1.000 1.0 1.0
patient guessed our words i have been down and wide awake at 3am for weeks keyword (word overlap) med5,men1 0.5 0.333 0.5 2.0
patient guessed our words i have been down and wide awake at 3am for weeks semantic (embeddings) men1,dia3 0.5 0.333 1.0 1.0
patient's own words i cannot pay for the name brand any more keyword (word overlap) (nothing) 0.0 0.000 0.0 NaN
patient's own words i cannot pay for the name brand any more semantic (embeddings) med3,med5 0.5 0.500 1.0 1.0
patient guessed our words my feet feel like pins and needles keyword (word overlap) dia5,men3 0.5 0.500 1.0 1.0
patient guessed our words my feet feel like pins and needles semantic (embeddings) dia5,car1 0.5 0.500 1.0 1.0
patient's own words the drug store says come back next week keyword (word overlap) (nothing) 0.0 0.000 0.0 NaN
patient's own words the drug store says come back next week semantic (embeddings) med5,men5 0.5 0.500 1.0 1.0
patient's own words i get winded carrying the groceries in keyword (word overlap) (nothing) 0.0 0.000 0.0 NaN
patient's own words i get winded carrying the groceries in semantic (embeddings) car4,car1 0.5 0.500 1.0 1.0

How this was measured

Sentences were embedded with sentence-transformers/all-MiniLM-L6-v2 (384 dimensions, ungated, run locally on a free Colab CPU -- no API key and no network traffic after the model loads). Cosine similarity is computed directly as E @ E.T on L2-normalised rows, not via a library convenience method, so every number is reproducible from the saved embeddings.npy. Retrieval is scored against 12 queries carrying 29 written relevance judgements (GOLD in the notebook). The keyword baseline counts shared content words (stopwords dropped, length > 2). Metrics are precision@k, recall@k and mean reciprocal rank at k=5. This is a 20-sentence teaching corpus with a 12-query gold set -- a demonstration of method, not a clinical validation. The corpus is teaching text I wrote: no PHI, no real records, nothing to act on clinically.