Just curious, are there any techniques other than using embeddings, computing cosine similarity, and sorting the results based on that? RRF could be used but again its very simple as well.
Looks cool!
You can input either a search query or a paper URL on arxiv xplorer. You can even combine paper URLs to search for combinations of ideas by putting + or - before the URL, like `+ 2501.12948 + 1712.01815`
Sure! I first used openai embeddings on all the paper titles, abstracts and authors. When a user submits a search query, I embed the query, find the closest matching papers and return those results. Nothing too fancy involved!
Just curious, are there any techniques other than using embeddings, computing cosine similarity, and sorting the results based on that? RRF could be used but again its very simple as well.
embedding search via https://searchthearxiv.com/ takes either a word vector, or an abs or pdf link to an arxiv paper.
https://news.ycombinator.com/item?id=42519487
I just did a spot check, I think searchthearxiv search results are superior.
There’s also the search and browsing on https://sugaku.net, it’s more focused on math but does also have all of the arxiv on it
Looks cool! You can input either a search query or a paper URL on arxiv xplorer. You can even combine paper URLs to search for combinations of ideas by putting + or - before the URL, like `+ 2501.12948 + 1712.01815`
This is really cool, and very relevant to something I'm working on. Would you be willing to do a quick explanation of the build?
Sure! I first used openai embeddings on all the paper titles, abstracts and authors. When a user submits a search query, I embed the query, find the closest matching papers and return those results. Nothing too fancy involved!
I'm also maintaining a dataset of all the embeddings on kaggle if you want to use them yourself: https://www.kaggle.com/datasets/tomtum/openai-arxiv-embeddin...
So did you just combine Title+Abstracts+Authors into a single chunk and embed them or embedded them individually?
Looks great! Could you add eprint.iacr.org (Cryptology ePrint Archive)?
Do they have a public API/dataset?
They have RSS feeds for new/updated papers: https://eprint.iacr.org/rss/
Oh god, there's a medrxiv?? TIL...
Don't forget chemrXiv!
Sadly I couldn't find a public API for chemrxiv, but would be happy to be proven wrong!
[dead]
[flagged]