Niche weighting is IDF for people

A fifty-year-old idea

In 1972 Karen Spärck Jones published a short paper in the Journal of Documentation with an unglamorous title: "A statistical interpretation of term specificity and its application in retrieval". Its argument changed how every search engine since has ranked results.

"Terms should be weighted according to collection frequency, so that matches on less frequent, more specific, terms are of greater value than matches on frequent terms."

- Karen Spärck Jones, 1972

If a search for "the saison brewery" matches a document on the word "the", that tells you nothing. If it matches on "saison", that tells you a lot. The weight Spärck Jones described became known as inverse document frequency, or IDF. Combined with term frequency it became TF-IDF, which Salton and Buckley's 1988 review helped establish as the default way to weight words in retrieval.

Swap "word" for "interest" and "document" for "person", and you have the central idea of Affinity Atlas niche weighting.

From documents to people

The analogy maps almost one to one:

One difference matters a lot. A search query is one-sided: the document does not need to want the user back. A match is mutual, which is why Affinity Atlas only scores sources both people have connected and why both people always see the same number. We cover that two-sided problem in The Cold Start Problem.

Same intuition, different curve

Classic IDF is logarithmic: log(N / n), where N is the number of documents and n the number containing the term. Affinity Atlas uses a bounded linear transform of a 0 to 100 popularity score, as documented in the algorithm deep-dive:

Put real numbers through both and the shapes diverge in an interesting way. In the table, "share" is the fraction of people with the interest, and IDF is also shown rescaled to 0 to 1 by dividing by ln(100) so the two can be compared.

The log curve saves almost all of its weight for the very rare. The linear curve is more generous to the middle: an interest shared by one person in five still counts for a lot. That is a deliberate choice. In dating, "moderately niche" interests (a favourite director, a beer style, a genre of podcast) are often exactly the overlaps that start a good conversation, and a curve that only rewards the ultra-obscure would make most matches look empty.

The linear form has two practical advantages too. It is legible: "popularity 18, so NicheWeight 0.82" fits on a match card in a way ln(1/0.18) does not. And it is bounded: a truly unique item cannot produce an unbounded weight. The demo engine also caps the top end at 0.95 so that a single obscure overlap cannot swamp everything else.

Recommender systems got there in 1998

We are not the first to move IDF from words to people. In their widely cited 1998 comparison of collaborative filtering methods, Breese, Heckerman and Kadie introduced inverse user frequency: weight each item by log(n / nj), where nj is the number of users who voted for it. Their reasoning is the niche-weighting argument in one line:

"Universally liked items are not as useful in capturing similarity as less common items."

- Breese, Heckerman & Kadie, 1998

Honesty requires the next sentence: in their experiments the effect of inverse user frequency varied by dataset and method, and the gains were modest where they appeared. It helped; it was not magic. That matches our expectation. Niche weighting is one ingredient of a compatibility score, alongside engagement depth, sentiment, Q&A and how much each person cares about a category.

Does it hold for humans?

The retrieval argument is statistical: rare matches are more informative. For dating we also need a psychological argument: do people actually feel more drawn to someone who shares a rare interest? There is direct evidence that they do.

Across five studies, Hans Alves (2018) found that "people's rare compared with their common interests are especially potent to elicit interpersonal attraction". Earlier, Van Zalk and colleagues (2007) found that similarity in less mainstream music preferences predicted which new friends Dutch adolescents chose.

The evidence is not one-sided. Baym and Ledbetter (2009) studied Last.fm users and found that shared musical taste was not associated with developing stronger relationships. Shared rare taste seems to help people notice and like each other; it does not guarantee anything afterwards. That is exactly the job we want it to do: start the right conversations, not promise outcomes.

Three lessons we borrow from search

1. Saturate engagement

Raw term frequency was quickly found to over-reward repetition. BM25, the ranking function that grew out of the probabilistic relevance framework, uses what its authors call a "carefully constructed non-linear saturation": the tenth mention of a term counts for less than the second. Affinity Atlas does the same with engagement. Minutes listened and hours played go through log(1 + value) / log(1 + cap), so the person with 3,000 hours in a game does not drown out the person with 300.

2. Smooth small numbers

IDF misbehaves at the edges. Libraries such as scikit-learn smooth it by default, using log((1 + n) / (1 + df)) + 1, so a term in every document is not ignored entirely and no weight divides by zero. We floor NicheWeight at 0.05 for the same reason: a mainstream overlap is weak evidence, not no evidence. When an interest has only been seen a handful of times, its rarity estimate also needs shrinking towards a sensible prior. That is a big enough topic for its own post: The Cold Start Problem.

3. Know what "popularity" actually measures

Spotify's popularity score runs from 0 to 100 and is based mainly on total plays and how recent they are, and it can lag reality by several days. Spotify's documentation now marks the field as deprecated, which is a useful reminder that third-party popularity numbers can disappear. Last.fm exposes listener and play counts instead, which are cumulative rather than recency-weighted. These are different questions ("popular right now" versus "known by many people ever"), and a band can score low on one and high on the other. This is why the data architecture computes rarity against Affinity Atlas's own population where it can, and treats platform scores as a starting point.

Popularity bias: the opposite failure

Recommender systems research has a name for the failure mode niche weighting is designed to avoid: popularity bias. Algorithms trained on engagement tend to over-recommend what is already popular. Abdollahpouri and colleagues (2019) showed that this hurts some users far more than others: people with niche tastes were routinely recommended items much more popular than anything they actually consumed. Earlier work from the same group showed the bias could be reduced with a regulariser while giving up little ranking accuracy.

Chris Anderson's 2004 essay "The Long Tail" argued that "the future of entertainment is in the millions of niche markets at the shallow end of the bitstream". Dating apps that rank people by desirability are popularity bias applied to humans: they send attention to the head of the distribution and leave everyone in the long tail invisible. We made that argument at length in The Elo Score Problem. Niche weighting is the constructive half of the same argument.

Where the analogy breaks

• Rarity is not importance. IDF happily up-weights typos and junk. An interest can be rare because it is meaningful or because it is noise (a mis-tagged genre, a duplicate artist entry). Cross-platform identity stitching exists partly to stop noise masquerading as rarity.
• Some common things matter enormously. Wanting children, values and lifestyle are not "niche" and should not be discounted because many people share them. That is why the Q&A layer uses importance weights chosen by each person, not niche weights.
• People are not static documents. Tastes drift. Popularity drifts too: today's niche band can be next year's festival headliner. Weights have to be recomputed, not frozen.
• Mutuality changes the game. A search engine only has to satisfy the searcher. A match has to work for two people at once, which is why scores are pairwise and symmetric.

None of these break the core idea. They are the same lessons information retrieval learned over fifty years, applied to a domain where getting it wrong costs people more than a bad search result.