New top story on Hacker News: Show HN: Research Hacker News, ArXiv & Google with Hierarchical Bayesian Models

Show HN: Research Hacker News, ArXiv & Google with Hierarchical Bayesian Models
5 by kianN | 1 comments on Hacker News.
Hi Hacker News! I’m a Bayesian statistician that has been working on applying hierarchical mixture models (originally developed for genomics) to structure text data, and in the process, used these models to build (what started as a personal) tool for conducting literature reviews and deep research. My literature review process starts with a broad search to find a few key papers/groups, and from there expands along their citation networks. I needed to conduct a few rounds of literature reviews during the course of my research and decided to build a tool to facilitate this process. The tool started as an experimental wrapper over low-level statistical software in C, quickly became a testing/iteration ground for our api, and is now my personal go-to for lit reviews. The tool organizes corpuses of text content, visualizes the high level themes, and enables me to pull up relevant excerpts. Unlike LLMs, this model transparently organizes the data and can train from scratch quickly on small datasets to learn custom hierarchical taxonomies. My favorite part of the tool is the citation network integration: any research paper it pulls up has a button “Citation Network Deep Dive” that pulls every paper that cites or is cited by the original paper, and organizes it for further exploration. I initially built this tool for academic research, but ended up extending it to support Hacker News to mine technical conversation, the top 200 Google results, and earnings transcripts. We have a gallery of ready to explore results on the homepage. If you are kicking off a custom deep dive, it takes about 1-5 minutes for academic search, 3-7 minutes for Hacker News, and 5-10 minutes for Google. To demonstrate the process, I put together a video walkthrough of a short literature review I conducted on AI hallucinations: https://www.youtube.com/watch?v=OUmDPAcK6Ns I host this tool on my company’s website, free for personal use. I’d love to know if the HN community finds it useful (or to hear what breaks)!