Work/Purpose-built AI
Conversing with fifteen years of journal entries
Fifteen years of journal entries were stored as email and text files. They could be read but not searched by meaning. This project indexed them so the archive could be asked questions and would cite the entries it answered from.
- Organisation
- Personal project
- Scale
- 3,500 entries, 15 years
- Focus
- Semantic search, retrieval
Context
The archive held around 3,500 entries written over fifteen years. Finding anything required remembering roughly when it was written. Keyword search did not help, because the same subject was described in different words at different times.
Organisations have the same problem with research archives, meeting notes and customer feedback. The material exists and cannot be used.
Mandate
Make the archive searchable by meaning, and able to answer a question with references back to the entries the answer came from.
Approach
I built a pipeline that pulled the entries out of email into a local SQLite database. Each entry was tagged by subject and converted into a vector embedding.
A retrieval layer sits on top of that. A question asked in plain language returns the entries closest to it in meaning, together with an answer that cites the entries it drew on.
Outcome
The archive can now be questioned rather than only read. Because every answer cites its sources, the underlying entries can be checked instead of taken on trust.
The same structure applies to any body of unstructured text an organisation holds and cannot currently search.