Improving Performance with Clustering in PHP Using SQLite as Vector Store

The article discusses the use of SQLite as a vector store in PHP and the challenges faced when working with embedding vectors. The author explains that due to limitations in RAM, time, and available extensions, efficient vector stores cannot be used. As a solution, the author uses SQLite as a simple vector store by storing the vectors as JSON blobs. However, the performance of nearest-neighbor searches using cosine similarity decreases as the number of rows increases. To improve performance, the author suggests clustering the data using the K-Means Clustering algorithm. The clusters represent semantically similar topics and are stored in the database with their centers. During query time, the closest cluster center is determined, and a cosine distance search is performed within that cluster. The author also mentions the importance of selecting the right number of samples and clusters for optimal results. This approach significantly improves the performance of the system. This article provides valuable insights for developers working with embedding vectors in PHP and facing similar challenges.