Rocketable (YC W25) acquires profitable software products and builds the AI agent systems to operate them. I worked three months on the customer experience and support systems of one of the company's products. The work covered prompt engineering and LLM evaluation, agent development across OpenAI and Anthropic models, AI cost analysis, and customer conversation analysis at scale. Projects are ordered by impact.
Media-recovery classifier and implementation cost report
93.75% recall detecting recoverable recording problems before customers report them; an LLM labels ffmpeg-extracted evidence packets with a prompt optimized across four automated candidate rounds, lifting validation accuracy from 80% to 90%
88.7% lower API cost from an ablation round, cutting the evidence packet to under a quarter of its size with no loss in test accuracy
The implementation report accounts for the full cost of running the system in production (per-recording AI cost of ~$0.03, projected monthly and annual spend, hosting options); the system is now being integrated on the basis of that report
Customer-experience quality classifier for support conversations
95% recall detecting poor customer experiences in completed support conversations, via a prompt-optimized LLM-judge agent built on OpenAI and Anthropic models, released through two versions with continuous monitoring
500+ support conversations read and hand-labeled from the customer's perspective, building the company's framework for what separates a good customer experience from a bad one, distilled into a generalizable evaluation rubric
Live in production as the company's standard measure of customer-experience quality, scoring every customer support conversation, with the weekly quality report published internally and cited in investor updates
Operator friction analysis and weekly reporting agent
Top drivers of friction for the operators directing the company's AI support agent, identified in a written analysis for leadership; drove prompt revisions, the media-recovery classifier, the support intake redesign, and the subscription cancellation investigation; now automated as a weekly agent-published report used as a recurring prioritization input
Customer support intake redesign
Single-tier support intake now live for all customers, collecting only information the system lacks and providing the CS agent an initial triage in place of multiple automated tiers
Redesign informed by analysis of answer engagement, CS-agent usage, and redundancy against internally held data, weighed against customer preference for response speed
A/B test groups designed to validate the flow changes ahead of rollout
Post-release bug monitoring agent
Deployed in Rocketable's internal systems: an agent monitoring support conversations for post-release bugs and crashes, alerting engineers in Slack; designed the clustering and triage logic, working with the founding product manager
~90% detection of known past outages during training
Subscription cancellation investigation
Triaged recently flagged support conversations, sized post-cancellation charge reports as the largest category, and traced the pattern through Stripe, LogRocket, and internal records to inconsistent cancellation states between systems, ruling out erroneous charges
Internal investigations and operations
Ongoing troubleshooting of Intercom workflows and backend behavior, alongside portfolio operations: affiliate payouts, payment-card management, and knowledge-base updates
View full report →
