When the Jaccard similarity index isn’t the fitting software for the job, and what to do as a substitute
I’ve been considering these days about one in all my go-to information science instruments, one thing we use fairly a bit at Aampe: the Jaccard index. It’s a similarity metric that you simply compute by taking the dimensions of the intersection of two units and dividing it by the dimensions of the union of two units. In essence, it’s a measure of overlap.
For my fellow visible learners:
Many (myself included) have sung the praises of the Jaccard index as a result of it is useful for lots of use circumstances the place that you must determine the similarity between two teams of parts. Whether or not you’ve acquired a comparatively concrete use case like cross-device id decision, or one thing extra summary, like characterize latent consumer curiosity classes based mostly on historic consumer habits — it’s actually useful to have a software that quantifies what number of elements two issues share.
However Jaccard isn’t a silver bullet. Generally it’s extra informative when it’s used together with different metrics than when it’s used alone. Generally it’s downright deceptive.
Let’s take a better have a look at a number of circumstances when it’s not fairly acceptable, and what you would possibly need to do as a substitute (or alongside).
The issue: The larger one set is than the opposite (holding the dimensions of the intersection equal), the extra it depresses the Jaccard index.
In some circumstances, you don’t care if two units are reciprocally comparable. Perhaps you simply need to know if Set A largely intersects with Set B.
Let’s say you’re making an attempt to determine a taxonomy of consumer curiosity based mostly on looking historical past. You will have a log of all of the customers who visited http://www.luxurygoodsemporium.com and a log of all of the customers who visited http://superexpensiveyachts.com (neither of that are reside hyperlinks at press time; fingers crossed nobody creepy buys these domains sooner or later).
Say that out of 1,000 customers who browsed for tremendous costly yachts, 900 of them additionally appeared up some luxurious items — however 50,000 customers visited the posh items web site. Intuitively, you would possibly interpret these two domains as comparable. Practically everybody who patronized the yacht area additionally went to the posh items area. Looks like we may be detecting a latent dimension of “high-end buy habits.”
However as a result of the variety of customers who have been into yachts was a lot smaller than the variety of customers who have been into luxurious items, the Jaccard index would find yourself being very small (0.018) despite the fact that the overwhelming majority of the yacht-shoppers additionally browsed luxurious items!
What to do as a substitute: Use the overlap coefficient.
The overlap coefficient is the dimensions of the intersection of two units divided by the dimensions of the smaller set. Formally:
Let’s visualize why this may be preferable to Jaccard in some circumstances, utilizing essentially the most excessive model of the issue: Set A is a subset of Set B.
When Set B is fairly shut in dimension to Set B, you’ve acquired an honest Jaccard similarity, as a result of the dimensions of the intersection (which is the dimensions of Set A) is near the dimensions of the union. However as you maintain the dimensions of Set A relentless and improve the dimensions of Set B, the dimensions of the union will increase too, and…the Jaccard index plummets.
The overlap coefficient doesn’t. It stays yoked to the dimensions of the smallest set. That implies that whilst the dimensions of Set B will increase, the dimensions of the intersection (which on this case is the entire dimension of Set A) will all the time be divided by the dimensions of Set A.
Let’s return to our consumer curiosity taxonomy instance. The overlap coefficient is capturing what we’re enthusiastic about right here — the consumer base for yacht-buying is linked to the posh items consumer base. Perhaps the website positioning for the yacht web site is not any good, and that’s why it’s not patronized as a lot as the posh items web site. With the overlap coefficient, you don’t have to fret about one thing like that obscuring the connection between these domains.
Professional tip: if all you could have are the sizes of every set and the dimensions of the intersection, you’ll find the dimensions of the union by summing the sizes of every set and subtracting the dimensions of the intersection. Like this:
Additional studying: https://medium.com/rapids-ai/similarity-in-graphs-jaccard-versus-the-overlap-coefficient-610e083b877d
The issue: When set sizes are very small, your Jaccard index is lower-resolution, and generally that overemphasizes relationships between units.
Let’s say you’re employed at a start-up that produces cellular video games, and also you’re growing a recommender system that means new video games to customers based mostly on their earlier enjoying habits. You’ve acquired two new video games out: Mecha-Crusaders of the Cyber Void II: Prisoners of Vengeance, and Freecell.
A spotlight group in all probability wouldn’t peg these two as being very comparable, however your evaluation exhibits a Jaccard similarity of .4. No nice shakes, but it surely occurs to be on the upper finish of the opposite pairwise Jaccards you’re seeing — in spite of everything, Bubble Crush and Bubble Exploder solely have a Jaccard similarity of .39. Does this imply your cyberpunk RPG and Freecell are extra intently associated (so far as your recommender is worried) than Bubble Crush and Bubble Exploder?
Not essentially. Since you took a better have a look at your information, and solely 3 distinctive system IDs have been logged enjoying Mecha-Crusaders, solely 4 have been logged enjoying Freecell, and a couple of of them simply occurred to have performed each. Whereas Bubble Crush and Bubble Exploder have been every visited by a whole bunch of gadgets. As a result of your samples for the 2 new video games are so small, a presumably coincidental overlap makes the Jaccard similarity look a lot greater than the true inhabitants overlap would in all probability be.
What to do as a substitute: Good information hygiene is all the time one thing to bear in mind right here — you possibly can set a heuristic to wait till you’ve collected a sure pattern dimension to contemplate a set in your similarity matrix. Like all estimates of statistical energy, there’s a component of judgment to this, based mostly on the everyday dimension of the units you’re working with, however keep in mind the final statistical finest follow that bigger samples are typically extra consultant of their populations.
However another choice you could have is to log-transform the dimensions of the intersection and the dimensions of the union. This output ought to solely be interpreted when evaluating two modified indices to one another.
Should you do that for the instance above, you get a rating fairly near what you had earlier than for the 2 new video games (0.431). However since you could have so many extra observations within the Bubble style of video games, the log-transformed intersection and log-transformed union are loads nearer collectively — which interprets to a a lot larger rating.
Caveat: The trade-off right here is that you simply lose some decision when the union has lots of parts in it. Including 100 parts to the intersection of a union with 1000’s of parts might imply the distinction between an everyday Jaccard rating of .94 and .99. Utilizing the log rework method would possibly imply that including 100 parts to the intersection solely strikes the needle from a rating of .998 to .999. It is determined by what’s essential to your use case!
The issue: You’re evaluating two teams of parts, however collapsing the weather into units leads to a lack of sign.
This is the reason utilizing a Jaccard index to check two items of textual content isn’t all the time an incredible thought. It may be tempting to have a look at a pair of paperwork and need to get a measure of their similarity based mostly on what tokens are shared between them. However the Jaccard index assumes that the weather within the two teams to be in contrast are distinctive. Which flattens out phrase frequency. And in pure language evaluation, token frequency is usually actually essential.
Think about you’re evaluating a ebook about vegetable gardening, the Bible, and a dissertation concerning the life cycle of the white-tailed deer. All three of those paperwork would possibly embody the token “deer,” however the relative frequency of the “deer” token will fluctuate dramatically between the paperwork. The a lot larger frequency of the phrase “deer” within the dissertation in all probability has a unique semantic influence than the scarce makes use of of the phrase “deer” within the different paperwork. You wouldn’t desire a similarity measure to only neglect about that sign.
What to do as a substitute: Use cosine similarity. It’s not only for NLP anymore! (But in addition it’s for NLP.)
Briefly, cosine similarity is a strategy to measure how comparable two vectors are in multidimensional house (regardless of the magnitude of the vectors). The course a vector goes in multidimensional house is determined by the frequencies of the size which can be used to outline the house, so details about frequency is baked in.
To make it straightforward to visualise, let’s say there are solely two tokens we care about throughout the three paperwork: “deer” and “bread.” Every textual content makes use of these tokens a unique variety of occasions. The frequency of those tokens develop into the size that we plot the three texts in, and the texts are represented as vectors on this two-dimensional aircraft. For example, the vegetable gardening ebook mentions deer 3 occasions and bread 5 occasions, so we plot a line from the origin to (3, 5).
Right here you need to have a look at the angles between the vectors. θ1 represents the similarity between the dissertation and the Bible; θ2, the similarity between the dissertation and the vegetable gardening ebook; and θ3, the similarity between the Bible and the vegetable gardening ebook.
The angles between the dissertation and both of the opposite texts is fairly massive. We take that to imply that the dissertation is semantically distant from the opposite two — at the very least comparatively talking. The angle between the Bible and the gardening ebook is small relative to every of their angles with the dissertation, so we’d take that to imply there’s much less semantic distance between the 2 of them than from the dissertation.
However we’re speaking right here about similarity, not distance. Cosine similarity is a metamorphosis of the angle measurement of the 2 vectors into an index that goes from 0 to 1*, with the identical intuitive sample as Jaccard — 0 would imply two teams don’t have anything in widespread, and nearer you get to 1 the extra comparable the 2 teams are.
* Technically, cosine similarity can go from -1 to 1, however we’re utilizing it with frequencies right here, and there will be no frequencies lower than zero. So we’re restricted to the interval of 0 to 1.
Cosine similarity is famously utilized to textual content evaluation, like we’ve performed above, however it may be generalized to different use circumstances the place frequency is essential. Let’s return to the posh items and yachts use case. Suppose you don’t merely have a log of which distinctive customers went to every web site, you even have the counts of variety of occasions the consumer visited. Perhaps you discover that every of the 900 customers who went to each web sites solely went to the posh items web site a few times, whereas they went to their yacht web site dozens of occasions. If we consider every consumer as a token, and due to this fact as a unique dimension in multidimensional house, a cosine similarity method would possibly push the yacht-heads somewhat additional away from the posh good patrons. (Observe you can run into scalability points right here, relying on the variety of customers you’re contemplating.)
Additional studying: https://medium.com/geekculture/cosine-similarity-and-cosine-distance-48eed889a5c4
I nonetheless love the Jaccard index. It’s easy to compute and customarily fairly intuitive, and I find yourself utilizing it on a regular basis. So why write a complete weblog put up dunking on it?
As a result of nobody information science software may give you an entire image of your information. Every of those completely different measures let you know one thing barely completely different. You may get priceless data out of seeing the place the outputs of those instruments converge and the place they differ, so long as you already know what the instruments are literally telling you.
Philosophically, we’re towards one-size-fits-all approaches at Aampe. After on a regular basis we’ve spent taking a look at what makes customers distinctive, we’ve realized the worth of leaning into complexity. So we predict the broader the array of instruments you should utilize, the higher — so long as you understand how to make use of them.