I'm bad at keeping track of people. Not in a rude way — I genuinely like meeting people — but after a coffee with someone interesting, I'd forget half the conversation by the next day. Three months later I'd run into them and have zero context. "Oh right, you were building something in logistics? Or was it fintech?"
It was embarrassing. And it was costing me real opportunities.
So I built myself a personal CRM. Not another spreadsheet. Not HubSpot with a $1000/month bill. Just a system using transcriptions and AI search that actually works the way my brain does.
Here's how it works.
The problem with existing CRMs
Every CRM I tried felt like homework. You're supposed to log notes after every meeting, tag contacts, set reminders. It assumes you have 20 minutes after each coffee to be a diligent record-keeper.
I don't. And even when I tried, my notes were terrible. "Met Jonas at startup event. Interesting guy. Follow up." That's useless six months later.
The real problem: CRMs are built for salespeople with structured pipelines. I'm a founder who meets interesting people at events, on calls, through Twitter. My "pipeline" is chaos.
What I wanted instead
I wanted something invisible. I talk to someone, have a good call, maybe record a quick voice note on the way home — and then I should be able to ask questions about it later.
"What did Marcus from the Munich meetup say about his B2B sales strategy?" Just search and get an answer, not scroll through 200 messy notes.
That's RAG. Retrieval-Augmented Generation. You store your own data somewhere searchable, and when you ask a question, the AI pulls the relevant bits before answering. The AI stops making things up because it's grounded in your actual notes.
The actual setup
I built this over about two weekends. It's not fancy.
Step 1: Capture
I use Whisper to transcribe voice memos and call recordings. There's a shortcut on my phone: after a meeting I walk outside, record a 2-minute brain dump, drop it in a shared folder. Whisper runs on my Mac mini and auto-transcribes everything overnight.
For calls, I use a call recorder app. Not ideal, but practical. The audio lands in the same folder.
Step 2: Store
Everything goes into a SQLite database. Each transcript gets a date, the person's name, tags if I remembered to add them, and the full text. Then I run embeddings using a local model so I can do semantic search.
I also manually add short "cards" for people I want to track: name, company, how we met, one-line note. These link to any transcripts where they appear.
Step 3: Ask questions
I built a small CLI where I can type "What did I discuss with David about his fintech product?" and it does a semantic search, finds the relevant transcripts, feeds them to Claude with a "answer from this context only" prompt.
The responses are good. Because it's pulling from my actual words and observations, it doesn't hallucinate. It just summarizes what I already captured.
What surprised me
I thought the transcription quality would be the hard part. It wasn't. Whisper is shockingly accurate, even in a noisy coffee shop.
The hard part was building the habit of recording in the first place. For the first month I'd get home and forget. Or I'd record once, then skip for two weeks.
What actually helped: I moved the shortcut to my home screen, right where I'd normally tap my messaging apps. Now it's just there. I record about 70% of meaningful meetings. Not perfect, but way better than zero.
What it's changed
The biggest win: I'm way less awkward at follow-ups. When someone emails me two months after we met, I search their name, pull up what we talked about, and reply with actual context. It feels like I have a better memory. I don't. I just have better infrastructure 😎
I've also started noticing patterns I'd miss otherwise. Three different people in the last month mentioned the same pain point about recruiting. Without the transcripts, that's just noise. With them, it's a signal worth investigating.
Is it worth building?
Honestly, it took longer than I expected. If you're not comfortable with a bit of Python and the occasional broken script, there are products that do parts of this: Granola for call notes, Notion AI for search. Nothing felt as tailored to how I actually work, which is why I built it myself.
If you do build something similar, skip the fancy UI. Just get the data flowing first. The value is entirely in the search, not the dashboard.
What does your network management look like? I'm curious if anyone's doing something similar. 🙂