MemoryCaller

First project in the senior care space. Was really interested in social isolation as a determinant of health, and there were interesting news stories of seniors talking to smart home devices. There were a couple companies marketing "robot companions", and I wanted to make a deviceless version that seniors could call normally on their phone which could theoretically lead to more rapid adoption.

Saw a lot of interesting use cases from our first users, including some who asked for daily jokes, vocab of the day, or as a reminder system.

The system

The system used Twilio for SIP and telephony, LiveKit for cloud agent hosting and observability, speech-to-text through Deepgram, text-to-speech through Cartesia, and Claude Haiku 4.5 as the LLM layer in between. I also built up a basic memory system and a backend service the voice agent could call to access and store memories, inject pre-call context, and schedule outbound calls.

I also experimented with voice-to-voice models since they could theoretically provide better emotional responses and ingest more context on tone, voice inflection, pauses, etc. that are embedded in the received audio. In the current setup, the reasoning layer of the agent can only see text transcripts post-transcription, which is subject to inaccuracy and loses the aforementioned context. Unfortunately, voice-to-voice models weren't great at the time, often having higher latency and outputting strange audio artifacts likely from training. As they get better, this could be a great extension of this project!

Challenges & learnings

Model routing

This was in context of a series of verbal consent (agree to ToS) and language selection deterministic steps in onboarding. In order to enforce this, I had to stop the LLM from generating during the gate and set the response stream to None. However since the release condition (token stream exhausted) couldn't be met, often this would lead to a hard-lock for subsequent requests. We eventually just moved these problematic questions to a web onboarding form. Initially I hoped onboarding could be completed strictly over the phone to minimize friction but there was no easy fix apart from forking the livekit SDK.

Interruption handling

Sometimes agent would go circular when interruptions happened, since it didn't have its own reasoning trace. Not a perfect solution for this, but added a truncating character to show interruption point but still place full returned message in context. Also upped tolerance for interruption audio so it would happen less frequently, less "stacked" interruptions which caused many of the most serious issues.

Context management

Current system could run for ~200 turns and 45 mins in prod before crashing, which was pretty surprising on Haiku's 200k context window. However, during these longer sessions tool calls were pretty unreliable, especially for scheduling or saving memories, and the agent frequently repeated itself. Was working on a system to do background transcript compaction / summarization and mid-call replacement, while maintaining a clean source of truth for the downstream extraction pipeline.

KPI

Realized that majority of retained users completed a scheduled follow-up call on their onboarding call. We structured onboarding as a series of pre-determined questions to build a caller's profile and learn interests, family details, schedule, etc. and this makes future calls significantly better. Picking up a call requires significantly lower activation energy than calling in, and doesn't require the platform to be top-of-mind at all times.

Perceived value

My initial thesis of faster deviceless distribution was incorrect. While it could've worked in theory, there was a clear consumer WTP gap due to perceived value mispricing. For the hardware products, consumers were purchasing tech that they could see and own, rather than faceless software. Additionally, hardware tech fits much better within gifting culture, which is a stronger driver for many of these purchases.