Skip to main content

An evaluation of multi-probe locality sensitive hashing for computing similarities over web-scale query logs.

Author
Abstract
:

Many modern applications of AI such as web search, mobile browsing, image processing, and natural language processing rely on finding similar items from a large database of complex objects. Due to the very large scale of data involved (e.g., users' queries from commercial search engines), computing such near or nearest neighbors is a non-trivial task, as the computational cost grows significantly with the number of items. To address this challenge, we adopt Locality Sensitive Hashing (a.k.a, LSH) methods and evaluate four variants in a distributed computing environment (specifically, Hadoop). We identify several optimizations which improve performance, suitable for deployment in very large scale settings. The experimental results demonstrate our variants of LSH achieve the robust performance with better recall compared with "vanilla" LSH, even when using the same amount of space.

Year of Publication
:
0
Journal
:
PloS one
Volume
:
13
Issue
:
1
Number of Pages
:
e0191175
Date Published
:
2018
URL
:
http://dx.plos.org/10.1371/journal.pone.0191175
DOI
:
10.1371/journal.pone.0191175
Short Title
:
PLoS One
Download citation