Why AI ranking beats keyword alerts for research

Most research alert tools return papers in one of two orders: chronological (newest first) or relevance-by-keyword-match (how many times your keywords appear). Neither is what you actually want. What you want is: "of the 200 papers published this week that match my topic, which 10 would change my work?"

That question needs ranking that understands your context, not just keyword overlap. This post is about how AI ranking works in practice, what it gets right, and where it's still imperfect.

Keyword matching: why it's not enough

A keyword alert returns a paper if the paper's title, abstract, or full text contains your keywords. Every match is equal in the eyes of the search engine — a passing reference and the paper's central thesis both count as "a match".

This leads to three failure modes:

False positives from tangential mentions. Your alert returns a paper about sepsis management because the word "sepsis" appears once in the introduction. The paper is actually about pneumonia.
False negatives from synonym gaps. Your alert missed a key paper because it uses "septic shock" where your alert matches on "sepsis".
No weighting for study quality or evidence level. A case report about one patient ranks identically to a 15,000-patient RCT.

You compensate by reading abstracts, which takes time. Research monitoring becomes "40 minutes on Sunday evening reading abstracts I should never have seen".

What AI ranking actually does

In Relaylit's pipeline, each candidate paper goes through three stages:

1. Retrieval. The brief is translated into per-database queries (MeSH for PubMed, field tags for arXiv, Elasticsearch syntax for Semantic Scholar) that cast a wide net. Retrieval prioritises recall: miss nothing relevant, accept that many irrelevant papers will come back.

2. Scoring. Every candidate is scored by a language model against the full brief. The score takes into account not just the words in the brief but the context: what population you care about, what study designs you prefer, what outcome measures matter, what you've explicitly excluded.

3. Ranking. Scores are normalised 0–100 across the candidate set. The top N are emailed. The bottom of the list is discarded.

The key is stage 2. A language model reading the abstract understands that "septic shock" and "sepsis" are related; that a case report should be weighted differently from a meta-analysis; that a paper briefly mentioning sepsis while studying pneumonia is not actually a sepsis paper.

Concrete example

Take this brief:

"Long COVID cardiac sequelae in adults, mechanistic and clinical studies only. No case reports. Last 90 days."

Three candidate papers come back from PubMed:

Paper A: "Myocardial injury in post-acute COVID-19: a multi-centre cohort study" — n=2,400, follow-up 12 months, discusses troponin patterns.

Paper B: "Case report: acute pericarditis in a patient with prior SARS-CoV-2 infection."

Paper C: "Long COVID in children: neurological and cardiovascular presentations."

Keyword ranking treats all three as matches. AI ranking (given the brief above) gives Paper A a high score (~85/100), Paper B a low score (~15/100 — it's a case report, which the brief excluded), and Paper C a medium-low score (~45/100 — it's about children, and the brief specifies adults).

The email shows Paper A prominently, de-emphasises Paper B and C. You read one paper instead of three.

Where AI ranking is still imperfect

Novel terminology. For very new terms (first month of a new subfield), the model may underweight papers simply because the vocabulary is unfamiliar.
Evidence-level subtleties. The model approximates study quality from abstract cues. It's right most of the time but doesn't replace reading the methods.
Language bias. Models are trained predominantly on English. Non-English briefs work, but niche terminology in other languages may score less reliably than English-language equivalents.

What to check when evaluating an AI-ranking tool

If you're comparing tools, look for:

Transparency in the score. Each paper should come with a score and, ideally, a one-line justification. Opaque scoring is hard to calibrate against.
Brief-level control. You should be able to specify exclusions ("no case reports", "RCTs preferred") and have them honoured.
Deduplication across sources. If you see the same paper twice from different databases, the ranking is happening after retrieval but before de-duplication, which is backwards.
Cadence control. Weekly is usually the right default; daily is overkill for most monitoring workflows.

What Relaylit does

Relaylit runs the pipeline above across six databases (PubMed, Europe PMC, arXiv, Semantic Scholar, Crossref, OpenAlex), deduplicates by DOI, scores each candidate 0–100, and emails the top N in a single digest. You get a score per paper and can tune the brief to adjust the ranking over time.

The free tier handles two active topics and weekly delivery — enough to evaluate whether AI ranking actually helps your workflow.