효율적인 벡터 유사도 조인을 위한 HNSW 기반 Adaptive-K 조인 알고리즘

초록

Recent studies have explored representing diverse data as high-dimensional embedding vectors and using them to search for semanticallyrelated data. Beyond vector similarity search, growing attention has been paid to filtered vector search with structural conditions andto vector similarity join and semantic join for finding semantically related record pairs. In this study, we propose Adaptive-K Join, analgorithm that performs semantic joins on table record embeddings. Nested Loop Join (NLJ) guarantees exact results by comparing allrecord pairs, but is computationally expensive. Hierarchical Navigable Small World(HNSW) enables fast ANN search, but its approximategraph-based search and fixed candidate size k may fail to retrieve all matches required for threshold-based similarity join. This limitationis more pronounced in real-world datasets with imbalanced match distributions across queries. Adaptive-K Join starts with a small initialk and expands it only when needed based on the boundary similarity of the current top-k candidates. This reduces unnecessary candidateverification while exploring more matching record pairs. At a threshold of 0.6, the proposed method achieved 42.2 seconds runtime,compared with 8342.4 seconds for NLJ, and 95.9% recall, outperforming fixed-k HNSW at 77.2%. These results show that Adaptive-KJoin achieves higher recall than HNSW while remaining much faster than NLJ. It can also connect semantically similar records withoutexplicit keys, making it applicable to information enrichment across data sources and semantic-based data integration.

키워드