Beyond Transcripts: Language Nuances and Audio Signals with Carter Huffman of Modulate
Resources:
- How Many CTOs Pod: https://howmanyctospod.com
- Scott Porad: https://www.linkedin.com/in/scottporad/
- Brad Hefta-Gaub: https://www.linkedin.com/in/bradheftagaub/
- Carter Huffman: https://www.linkedin.com/in/carter-huffman-a9aba05b/
- Modulate: https://www.modulate.ai/
#TechPodcast #EngineeringPodcast #DevTalks #PodcastForDevs #HowManyCTOs #Podcast #CTOs #CTOPodcast #ChiefTechnologyOfficer #Technology #Engineering #SoftwareDevelopment #SoftwareEngineering #TechLeadership #EngineeringLeadership #EngineeringCulture #TechDebates #AI #VoiceTech #MachineLearning #MachineLearningModels #GamingIndustry #AIinnovation #Entrepreneurship #AIConversation #VoiceAssistant #LanguageModeration #GPU #LLMs #LargeLanguageModels
Guest
Brad: With these big foundation models like OpenAI's and everything,
you just throw a ton of data and it'll figure it out. For these small,
tiny, efficient models,
you have to label the data really, really well.
Scott: We would talk on the phone and be like,
oh, this is such a, we should make a podcast.
Then we just started recording the phone calls.
Brad: Oh, amazing.
That's fantastic.
I love
Scott: Yeah.
Brad: that. I should do that with my co-founder just six hours back and forth about like culture and all that kind of stuff.
I'm sure everyone would just listen to that verbatim.
Carter: Oh, well, I mean, at least the 30 listeners that we have would come listen to it.
Brad: Okay,
brilliant.
All right. Perfect.
Wonderful.
Scott: All right. Everybody welcome to this week's episode of How Many CTOs Does It Take?
My name is Scott Porad and this is my trusty co-host.
Bruce and
Carter: Brad Hefta-Gaub.
Scott: Brad who do we have with us here today?
Carter: We have a very special guest with us today,
Carter Huffman,
the CTO and co-founder of Modulate AI.
And Carter,
would you like to introduce yourself and tell us what brought you here?
How'd you get here?
Not flights,
but, you know.
Brad: oh yeah right right so i was in this british airways flight for 11 hours typing like a t-rex yeah and that's why i'm on your podcast everyone wants to hear about that i'm sure uh hi everybody i'm carter hoffman i'm the cto and co-founder over at modulate we're a voice ai company we deal with uh cost effective accurate voice ai models at scale there's a ton of fun stuff to talk about there um and before that i uh was doing machine
machine learning and artificial intelligence over at the Jet Propulsion Lab so working on some cool like spacecraft autonomy machine learning so if we want we can talk a little bit about that too uh
Carter: Very cool.
Scott: That does sound very cool.
Like you were working on autonomous flying airplanes,
autonomous
Brad: autonomous science on spacecraft uh which is which is a autonomous
Scott: science on spacecraft.
Brad: science yeah so so what that is is basically like um
here's the here's the one kind of one i love the most um is okay so you've got like a mission to a meteor like an asteroid comet what have you right and these missions are expensive right you know 100 million dollars you spend 10 years building this spacecraft and you send it up and it travels for two years and it flies by a comet you know a light hour away from earth right that flyby takes
takes 15 minutes maybe max right 15 minutes you're there with this comet all that stuff for 15 minutes and as you're doing this flyby maybe you see something interesting like a little jet of gas coming out of the of the comet or a little bright spot or something and if a human's looking at that on earth they're like oh that's really cool let's go take a closer look but it costs so much money and so much time to get there uh no one's ever going back to that comet ever again this is the only time you
time humanity is going to see it and so the stuff i was working on was like can you with your decade-old you know less powerful than a first generation iphone cpu on this spacecraft find interesting stuff while you're doing the mission and then go kind of do a follow-up observation on it even without a human in the loop it was really cool stuff
That
Carter: That is pretty cool.
Scott: You know, a hundred million dollar spacecraft mission in the age of AI data center spend sounds like a bargain.
Brad: You know what, that's right. We should just send chat GPT to every single asteroid in the solar system.
That's amazing.
Because you know what AI needs? AI needs more capex.
That's what AI really needs. They need to spend more money. That's what was missing.
Carter: Isn't that isn't that the strategy behind?
The Elon merging XAI with SpaceX,
like, you know, all the CapEx just looks the same at that point.
Brad: Those games are so beyond me. When I run multiple trillion dollar companies,
I will come back to you and I will say, okay,
now I finally get it. Now I finally know why Twitter has to go to space,
right?
But, you know, I'm going to have to get back to you on that one.
Carter: Exactly.
Exactly.
Scott: So how was your journey?
From JPL to Modulate. How did that come about?
Brad: It was really cool. Yeah, because you don't necessarily think like, okay,
NASA and then go straight to like, like, like software tech or something.
But there's actually this really cool parallel.
So at JPL,
I was telling you about kind of doing this low power machine learning stuff on the spacecraft.
And that's a lot of kind of analyzing signals and variation in signals over time.
And around that time, this is 2014,
2015 or so, people were doing some really cool stuff with AI and machine learning,
like early days pre-LLM.
So,
for example, one of the things I really loved was called GANs, Generative Adversarial Networks. And that had this cool capability where you could take a picture of me with my glasses and you could use the GAN network to take my glasses off and it would make a kind of realistic image. And you could do all sorts of futzing, change my hair.
my hair color this sort of thing and like nowadays these are in all the apps like nobody thinks that's really impressive but that was the first time you could really do that with high quality images and the really cool thing about neural networks like the entire reason why these have you know because because an llm is just a neural network a transform is just a neural network right the entire reason why these algorithms are so cool is they're very very agnostic to the input kinds of data so you can do something cool with the neural net on an image
or on text or on a video.
I was seeing this stuff and I was like well you could do that for audio too uh and audio is like okay that's just a signal with a bunch of variations and stuff like that you know I was a physicist by training like they deal with waves like quantum waves all the time these sensor measurements those are kind of like waves audio is a wave and I'm just like it's all waves I know the math behind this already why don't I work on audio um and so that's when I kind of just like like nights and weekends you know over
For a few years, I started getting into how can I apply these AI neural networks to audio?
It's got to be possible.
I ran into a ton of challenges,
but that's sort of where it came about.
Carter: Cool. Did you already have a passion for audio? Like, you know, like you just said, are you a music nerd or something like that as well? Or is it or is it more just kind of like what types of waveforms can I apply neural networks to?
Brad: It's so much worse than that. I'm a math and data nerd.
what a nerd so i was like oh there's data type but i understand you know what i was i was in um at stanford i was a math camp in stanford when i was in high school uh and i we did a little impromptu acapella group and so for the talent show i was part of a five person part of your world acapella so um
Carter: There you go.
Brad: yeah probably the nerdiest uh audio there is out there but
um but no it was really the signals in the data type it's like hey i understand how to manipulate this because audio is really cool it's like um and voice is really cool like you think when you're doing machine learning on images you generate a face you can see a face in anything right you can you see faces in the clouds all the time you can see a face in anything so it messes up a little bit you're still going to see a face it's actually fine we're very tolerant if you mess up audio a little bit it
we immediately say that sounds off uncanny valley straight away and there's like evolutionary arguments for this of like okay if your voice sounds wrong maybe you're sick or you know maybe you're lying or something like that but audio is such a severe environment
Carter: Interesting.
Brad: and so you have to massage and manipulate the data to make machine learning work on audio pretty well
Carter: Very cool.
Scott: So, I mean, I got a lot of questions I could ask, but I'm going to kind of just go to the top of the pyramid of questions.
How are people using it?
Brad: Fantastic question.
I love that pyramid.
So primarily right now what our technology does is it understands conversations very,
very thoroughly and cost effectively.
So what happens is we're deployed in video games.
call centers and other areas like that so you have two people talking in a game and they're harassing each other right or somebody's harassing somebody else we're picking up on that and saying okay based on what you said but also the tone of voice the emotion the back and forth of the conversation maybe your history have you been you know spatting at each other for the past 10 minutes or is that just coming out of nowhere
where what is this person's intent are they really being toxic to this other person or are they just trash talking you something like that that's kind of where we got started um and that's a really cool challenge because you have to be right if you ban a gamer and they didn't deserve it they're gonna go on twitter they're gonna yell at you they're gonna be super mad they're never gonna play the game again they'll probably be mad on twitter anyway but uh you know they're they're it's kind of a thing you really don't want to get wrong
wrong
Carter: Right.
Brad: and you also have to do it at scale so like a big game call of duty fortnight they could have a hundred million hours a month or a billion hours a month audio and at transcription costs you know you call it ten cents to a dollar per hour nobody's gonna pay a hundred million dollars per month to just you know make their make their games
Carter: Yeah.
Brad: here
Scott: I kind of be honest I have a hard time imagining how this works so an audio stream comes into your model and then what comes back out you know like if I think about chat GPT I put in a stream of words and I get a stream of words back out right but here like I'm putting in a stream of audio and I actually want to get
like some metadata back out about it like
Brad: Yeah.
Scott: what was the tone yeah how does how does that work
Brad: So we actually have this kind of multi-level representation of a conversation that we build, and then we like update it over time as a conversation is going on.
So you get a transcript,
for example, right, that's kind of easy,
you just get a transcript that's sort of this streaming set of text.
What were you saying?
You're also getting kind of an emotion signal over time. So we have like
you have like labels like okay you're angry right now you're angry right now you're angry right
Scott: Interesting.
Brad: now now you calm down now you come down now you come down um those kinds of things accent labels same exact kind of thing so so that's that's what we call the signals right you take in the audio and you're extracting these low level sort of individual signals from the audio of like what's the tone of voice what's the emotion what's the accent what's the text content of what you're saying etc
That's kind of layer one. And then on top of that, we're running a separate set of models.
They're taking in that stream and constantly saying,
okay,
what's my take on what's happening right now? Is somebody harassing somebody else? Yes or no?
Is somebody trying to manipulate somebody else? Yes or no?
Is somebody talking to a child in a way they're not supposed to? Yes or no.
And we've got these kind of models running along and they're constantly evaluating all of these rich signals, combining them together and saying at any given time, are we confident in any of these bad things happening or any of these interesting things happening?
Carter: Yeah, cool.
And so if I'm picturing this correctly,
whether or not you're using a you're using some.
neural net tech,
which is essentially,
I mean, the really simplified way of thinking about all these neural networks are signal in, neural network.
signal out and you train the neural network to give you signal out that matches what you want the signal out to be based on the signal in so your signal out might be aggressive or friendly that's the signal out that you're training on and the signals that are coming in are these multiple layers of these other signals that have gone through neural networks to produce each of their kind of signal layers that's is that a fair way of describing it or overly simplistic
Brad: No, no, no. I think that's a fantastic view.
view of it and basically the magic comes from layering on a bunch of these so if you want to go straight from just a raw audio signal audio audio waveform to is this person spreading violent radical propaganda that's a really hard problem but if you go first to say well what tone of voice are they using
what are they saying where do they you know what are all these different pieces and then you say okay based on those different pieces if we're confident in those how can I put that together into something more complicated you do like three levels of that hierarchy now one you can be a lot more confident in the result because you're built off of building blocks that apply to more situations you've got kind of more relevant data that you can train on and it's explicable as well
Why do we think you're being this kind of violent extremist? Well,
because you were really angry,
you said these words, and when this person said a dog whistle,
you did this thing to it, right?
All of those different pieces can come together in an explicable way.
Carter: Yeah.
Scott: language. Is it language agnostic or is it English only?
How's that work?
Brad: Good question.
So the whole system has support for over 60 languages.
But this ensemble tech that I'm talking about, when you really dig into it,
it turns out that having just one neural network support 60 different languages is so expensive. So kind of the whole trend for AI.
is build a bigger model build a bigger model build a bigger model and the great thing about big models is they're so flexible you can ask chat gpt anything and it'll have an answer it
Scott: Might not be true.
Brad: might not be true and you're spending so much power and energy and water and everything right to get to that answer and almost you know that the chat gpt also has the bible memorized but it's not using the bible to answer your question probably like
we like
Carter: Right.
Brad: you're wasting so much of that neural network just to get you your one answer about you know what what breed of dog is whatever whatever something so you're wasting so much capacity when you're building this big general network we go literally the exact opposite we say let's have a hundred different tiny networks and each of them aren't going to generalize well they're only going to do one thing on one type of data maybe like we only do you
a good job transcribing romance languages in discord quality audio but you call in on an eight kilohertz phone line it's going to be trash but then you have another model that handles the phone line and another model that handles another thing and then you have to pick you orchestrate between those different models and say at any given time which model is the right one to run and that one will handle it so each individual model handles maybe one or a few languages
languages, but the whole
Carter: Yeah.
Brad: system can handle a bunch.
Carter: Yeah.
And I can picture that like the neural network that's transcribing from, you know, like you said, English to over, you know, the low quality audio stream is.
going to be very different than the neural network that is trying to pick up tone, anger or happiness or whatever the tone, the tone neural network. And I would imagine that some of these neural networks probably are a little bit language agnostic, like maybe the tone one is, is maybe it's not culturally agnostic, but it would be language agnostic, right? So I'm assuming that this is really kind of the magic you've built all these different pieces that are good at really,
really good at.
That one part of the problem and then orchestrating them all together to put it together is the magic.
Brad: You've got it 100% right.
Yeah, each different kind of model has different properties. So like the emotion model,
for example, yes, it's our basic emotion models tend to be more language agnostic.
But on the flip side,
you try to apply it to Mandarin or another tonal language where the tones are actually meaningful.
And it
it can't figure it out at all you have to have a separate total language emotion model otherwise you're going to be super super confused kind of like normal people can be confused when somebody else is speaking a different language and you're like wow that german person is always angry all the time right no it's just you know okay do you have your german emotion model to yes or no right
Carter: Du bist mein Liebchen!
Brad: yeah exactly how dare you right you know um so so it and but that actually brings out a fantastic point really because with with these big foundation models like open ais and everything you just throw a ton of data and it'll figure it out spend a ton of money doing it but you just throw in a ton of data for these small tiny efficient models you have to label the data really really well
Oh, and so if you have a bunch of people, you know, from Boston labeling a ton of German or Russian data,
they're going to get it wrong and our model is going to suck.
And
Scott: Yeah.
Brad: so one of the key pieces for us was actually finding,
vetting and coordinating like globally distributed set of folks who can give us the right labels and take on the data so that we can train these systems correctly.
Carter: That's interesting.
Scott: How big is your team?
Brad: About 45 people right now, about,
you know, maybe 17,
18 engineers and then 15 or so commercial folks,
couple ops people,
few executives that do podcasts and stuff.
Most of the people do the real work.
Carter: I was going to ask sort of similar,
I think, as you're talking about like the people doing labeling and things like that.
I mean, I'm assuming those aren't part of that's part of your magic is like finding the labelers, you know, outsourcing the right labelers, finding a way to vet them.
What is the hardest problem you've had to solve in in in doing this?
Like, I mean, it sounds like so many hard problems.
What's the hardest problem you've had to solve?
Brad: There are a couple. I guess you could say one really difficult one has been all the different definitions.
that different customers or team members or anybody has for certain kinds of behaviors because if you're just transcribing something if we were just a transcription company that's relatively unambiguous not perfect but mostly you can agree on what this person said in any given moment but if you're saying well is this hate speech because you said this awful slur well
it might be but it might be reclaimed language because people use slurs in positive ways but
Carter: Yeah.
Brad: you can only do that in certain situations at certain times uh and it becomes very very nuanced and then you know we kind of go and form this really nuanced opinion about like when is the slur okay or not and the customer comes in and says not super nice uh you just can't say that in my game and so we're like okay well that's a completely separate
kind of take on that problem and
Carter: Yeah.
Brad: so we're trying to build an anti-harassment model and everybody has different definitions of harassment
Carter: Yeah.
Yeah.
Yeah. I was one of the teams that I work with is in the cultural institution space and like working with museums and things like that. And they were confronted with this recently and they're like, well, what do we do about the fact that like.
If someone's going to be talking about
black history museum where slavery and lynching is discussed like we can't just like say you can't use the word lynch like like not if you're having a intellectual conversation about a historical fact right so it's an interesting problem
Brad: Absolutely. Or take, you know, video games where you're talking about, oh,
go over, kill people or OK,
well, I'm just going to, you know, I'm just going to commit suicide and then I'll respawn or whatever.
or whatever right like it is very difficult for a neural network to figure out if that's acceptable language or not it's a really really hard problem and so we had so much time and energy and and effort and money put into how do we build out these really nuanced understandings of all these things and then where do you draw the line between your model because you have to classify some of this stuff otherwise you're just handing people a stream of
stream of text and say you figure it out but
Scott: Yeah,
yeah
Brad: on the other side if you're too opinionated then everybody's going to have a different opinion and they're going to have to circumvent your model anyway so what was the point finding that balance of how much control versus how much do you build
a really really tough problem everybody has different opinions on it
Scott: Totally.
Brad: the other really tough problem maybe if you want to talk more just like hard tech was coordinating all these models so most folks is like a one big model you run the data in and then once you get the answer out you're good
But we have this ensemble of like 100 different models,
and they're all running on different resources.
One of the nice things about our ensemble approach is you can run on a 10-year-old GPU because it's a tiny model,
you'll still get acceptable performance,
you don't need the latest and greatest.
So we've got all those little clusters of different models.
And normally you just kind of push the data through and then you get all the data back out,
you combine it all together and you get an answer. And that's great.
But when you're operating at 100 million hours a month or a billion hours a month,
having to wait for every machine to respond to every answer,
now you've got this massive bottleneck and you slow everything down and lock everything up.
So we have to do really kind of relaxed best effort parallelization, which means not all the data gets together at the time that it's needed to make a decision. And then you make a decision.
and if you want to go audit that decision later you're like well did it even have all the data in the first place like why did it make that decision it's supposed to be explicable but what happens when you know some of the models drop out and so making this stuff robust to how can i give still give good guarantees and good answers when i might have 10 of my models fail to return a result in time and i have to just do the best i can
Carter: That's wild.
At that scale,
like that's.
That's when stuff gets really interesting for It sure.
Brad: was it was really fun when Call of Duty launched, I think this was the November 2023 launch,
and we ran them out of A-10 GPUs in U.S. East 1.
And so we're like,
OK,
we've taken 2,000 of your GPUs, and now we don't have any more.
And we're like,
OK,
you know, we had some people on a team going and being like.
being like okay which other region has these GPUs okay US West 2 all right awesome I'm going to submit a request for quota increase and then the rest of us are like okay which models do we really need how do we balance out like okay you know how important is accent right now you know or or like can we
Get 80% of the accent coming through to get 95% of the transcriptions coming through.
We were like balancing that in real time while this game was launching.
Carter: Wild.
You mentioned video games a handful of times,
and I've been sitting here trying to imagine how,
what are ways that this could be used?
And so I'm just curious if you're willing to share, what are the different ways that the model is being used?
Because I bet there's some ways out there that people,
myself.
I couldn't possibly imagine.
Brad: Well, I think it's it's it's really cool because that's kind of a bit of the evolution of our company was we started out in gaming. That was our beachhead market,
which is why I talk about it a lot.
We just have so much history there and I'm a gamer and we were publishing these case studies and stuff like that.
And we got some folks reaching out to us saying,
hey, you can find harassment in these games.
We're a delivery company.
We have drivers going and delivering stuff physically to people's doors.
And if they have questions, or the person receiving the package has a question or something,
they can call the delivery driver.
And most of the time, that's great.
It's just like, hey, FYI, my door code is whatever.
But sometimes they'll call and say, where's my package?
And if you're like, well, I'm on the way, maybe they're in a rush and they get really upset.
And a smaller fraction of the time people get really upset and start making threats and a very small percentage, but tragically not zero percent of the time they follow through on those threats.
And it's really rough because you've got to go bring this thing to their door.
So you're going to meet this person that is calling you on the phone threatening your life.
And.
And so obviously you just don't want to go do that.
But the problem and a lot of these delivery companies and two-sided marketplaces, right,
all these sorts of things.
The problem you get is that the folks doing these deliveries, their services, if they cancel.
they're worried that that counts against them and they might lose their job and they might
Carter: Yeah.
Brad: lose their source of income and it doesn't matter how much the company says you know that's an okay reason to do that like metric traffic isn't perfect people are worried about losing their jobs so they'll go through with these unsafe situation
Carter: Yeah.
Brad: And so this company reached out to us and said,
we want you to monitor the phone calls and find these threats of violence and we'll stop the delivery so that we don't have to force the driver to choose between potentially their job and their own safety.
Carter: Yeah.
Brad: So we were doing that.
We were running our models, running our models, and we were finding all these high stress, emotionally intense situations that didn't involve threats of violence. And we're like, what could be so stressful about receiving a package or receiving an order?
And we were we were looking in and what we found is that we were actually seeing attempts at manipulation and fraud and the person calling the driver and trying to get them to change the order.
or what have you and emotionally pressuring them and so parts of our models were picking up these really stressful situations the rest of our models didn't know how to handle and so we were looking at that and we were like wow we were finding this stuff isn't this weird and the company was like oh crap okay yeah that's really bad we need to catch that too can you catch that for us as well so customers actually led us into finding these things so now we're doing we're doing anti-fraud we're doing like call centers you're calling in and
and and you know you're like somebody's trying to socially manipulate their way into your you're you're getting a password reset and the agent says sure fine yeah that sounds like a tough situation you lost your phone you're traveling yeah here's here's a new password um we find all that kind of stuff um and escalate it so that folks uh can can solve those problems now the newer use cases come around like
uh when ai's voice agents start misbehaving so
Carter: Yeah,
Brad: i don't know if you all have run into you know any fun little ai voice stories or anything like that we have some good ones over here but
Carter: it's pretty interesting. So we use an outbound.
all ai voice agent a tasker to call piloting right now to call either clients or taskers in certain scenarios and it never occurred to me that the ai could get snippy could get potentially get frustrated and behave badly but it probably could
Brad: Oh, we have one customer who's an interviewing company and they have an AI bot that they were trialing to.
to do their first round interviews and they told the bot to check if the candidates were flexible flexibly minded and it started asking them to do yoga poses on a phone call and and they're just like what you know what on earth so so that's actually kind of a new part of our business which is which is interesting is like you know we're listening to these AI bots on the phone calls and they just do stupid stuff that no human would normally do and you don't even know to look for it
for it and we're like okay can we find all this inappropriate stuff and escalate it so you can be like okay great my interview bot is asking people to do yoga poses that's bad let's go fix it
Carter: I feel like you've also, like, landed on essentially a better version of a polygraph, right?
It's essentially lie detecting or could be used for that. If you're not already doing that, I have to assume someone's using it for that.
Brad: I can't confirm or deny that
Carter: Can't confirm or deny that.
Wait a second.
My model says that may have been a dodge.
Brad: Yes, it is. It's like, I think ultimately speaking,
what I find so compelling about what we're working on here is that conversation is so much more than just the textual content of what you're saying.
Carter: Yeah.
Brad: You pick up so much from my tone of voice, from my emotion,
from my inflection,
how it changes over time,
everything.
And when you're trying to do something.
add value do something worthwhile on top of that voice signal taking in the whole range and dynamic of what's being conveyed in that voice signal is incredibly important and when you throw most of that away and just go with the text transcript you can have the smartest system in the entire world consuming that text it will just not have the data
Carter: Yeah.
Brad: to appropriately interact with you and respond with you and so when you're talking to an AI on the phone and you're like oh my god just get me to a human like that's why that it's not because it's not a smart enough algorithm it's because it doesn't have the data to do it correctly we
Carter: We've talked a lot about this in a commercial sense. I've been thinking about is there any sort of like therapeutic uses for this.
Brad: have actually had some folks
Asking us about putting it on.
like remote telehealth calls and monitoring the emotion both from just to like almost like QA purpose of just like hey you know is this person kind of responding appropriately to what this patient's kind of signals are but also from a diagnostic perspective which is really interesting and challenge like
Like anytime you get into AI diagnostics, you get into a very, very challenging and interesting space.
But what can we pick up?
Based on the fact that we can listen to hundreds of thousands of people talking about, you know, things in a therapy context,
what kinds of signals and cues can we pick out that you would only observe at scale?
And this is especially interesting when you start talking about like more generational trends,
right?
Because doctors have tons of training on what kinds of ways can I.
sort of look into somebody's behavior to help diagnose something that might might be going on with them but everybody's shaped by the society they're in and the people around them and so you get kind of the next generation of folks coming up and they respond to things very very differently from
what your textbook said and but by looking at that behavior across a very very large range can you start to pick out some of these signals and do a better job helping out a doctor who might not have a ton of context on that
Carter: Yeah,
You talked about tele,
you know, digital health,
something Brad and I spent a minute working on.
And I was even thinking about like, just what even even about like ambient in
in the doctor's office.
the doctor's.
doctor's office or even just, you know, like I had that thing happen the other day, right?
where i was talking about getting the tires on my car replaced to my friend and then my friend reported to me that the next day she started getting ads for tires on her phone right because her phone
Oh,
her
I don't
phone
like that.
no nobody likes it but it's true right so her phone heard me talking right so like but think about the ambient you whether it's a doctor's office or just the ambient uses for this thing i don't know what they could be but it's interesting
Brad: I think one of the really cool things, and this starts to get into like a little bit of the future stuff,
which is like, you know, why we do tech,
right?
One of the really cool applications in the future,
because we've talked a lot about the emotion and context and that understanding is also the sense of scale, right?
our our our stuff you know we really focus on that lower cost efficiency because again you're not going to pay a hundred million dollars a month to moderate call of duty right so that was kind of our heritage was we've got to be really really low cost at this stuff and so you can think of like okay well today i'm going to transcribe every zoom call
And you can do that.
But tomorrow,
what if I'm going to transcribe just every random conversation that people are happening all around the office or all around, you know, the wherever?
And that is, on the one hand, super frightening from a privacy perspective.
Carter: Mm
Brad: But on the other hand, there's a lot of potential around being able to keep track of and organize information at a scale that's been completely unheard of in
Carter: -hmm.
Brad: the past that's enabled by being able to do.
do like really really cutting edge highly accurate sophisticated AI on data coming from just everywhere all at once
Carter: Yeah, and even the strategies that you're talking about, you've applied it to audio,
but the notion of an ensemble of small models.
That's a way of thinking about data processing in general,
right?
There's other forms of data, whether that be audio data, like you're doing with modulate or video data or, you know, text data or temperature data,
Brad: Sensor data.
Carter: sensor
Brad: Yeah,
from your
Carter: data.
Brad: car,
your phone,
all that kind of stuff.
Absolutely. Yeah.
And these these ensembles of small models are so powerful when you're talking about the embedded world as well, because.
You can't ship chat GPT to your phone or your car or a pin or something like that.
But you can ship a ton of tiny models. And then as long as you can be pretty smart about picking the best one to actually stick into RAM and run,
you can get really general high performance without having to run this massive neural net all locally, you know, in your 2014 Prius seat.
Carter: I'm curious.
So so far, we've only talked about sort of the the input of signal to the signal out being like, you know, inference of like, you know, is this person angry,
whatever,
harassing, et cetera.
Is Modulate also taking this?
Clever approach that you've done to do the opposite of generating audio that is emotionally relevant,
or where are you guys on that?
Brad: I have so much ambition around that.
Our team is so, so excited about it.
Ultimately,
we love everything about voice.
We love everything about voice. We started with these voice skins,
these voice changers, right?
Like really low latency,
really low power.
run on device while i'm having this call make me sound like scarlett johansson or taylor swift or james earl jones that's where i got started oh god we were working on the research by that a decade ago so we are so interested in high quality audio synthesis all these kinds of things at a 45 person startup you have to find some place to start and so we we are
So, so focused on building the best, most cost effective,
fastest conversation understanding models in the entire world.
And we have so much ambition to take exactly that same strategy and move it to manipulation and synthesis as well.
Carter: Yeah.
Brad: It's actually interesting kind of that you're bringing this up because in the understanding space,
you can just like fan the data out.
run it through a bunch of models and then collect everything back it's not immediately obvious how you would do that just to make like a high quality waveform synthesis right
Carter: Yeah.
Brad: like it's just like do you make a ton of different waveforms and then you average them or combine them together that's probably not not not the right answer but what i think is really interesting about the ensemble models that we work on is it's not just a bunch of tiny models running
running in parallel,
it's also hierarchies across scale.
And
Carter: Yeah.
Brad: so one of the biggest problems in audio synthesis,
especially these voice agents, is latency.
You want to
hear something and respond immediately if you want super low latency you need to run on device on the edge if you're running on device on the edge you have to run a less powerful model and so they're either slow and kind of smart that's still not that smart or they're really fast and super super incompetent if you think about starting to break this stuff up you can run a small edge model that could be super incompetent and you can run a small edge model that could be super incompetent
And then give it context,
very rich context that you're extracting from a larger, more capable model that's somewhere in the cloud.
And you can do this multiple different times at different hierarchies. So the small model that itself would be completely incapable of doing this now has all this extra metadata and information from the bigger models elsewhere.
Carter: yeah that makes a ton of sense like you know the model that's basically you know if we just think of it from like
Text to speech where it's like the model that's taking the text and then generating the voice from it,
you know.
As opposed to, I don't know if this is the way you'd approach it,
but if instead you've broken down the text with the intonation, with the emotional resonance that you're looking for and you've turned that into a different stream of tokens,
which is, you know, maybe closer to phonemes or something like that, where it's no longer,
it's not, it's now just solving like the next layer down that can be much more efficient on a faster,
lighter model.
Brad: exactly right yeah you do the decoding at the edge where you have to do it really fast and then you do the conditioning of somewhere else where you can afford a one second delay if you really really need it and honestly that's kind of how I feel like I work sometimes because like I'll start talking and then as I'm saying the next word I'm like wait a minute what exactly where was I going with what's the next word that I'm supposed to say and so I've got that you know a much more powerful it feels like cloud-based model coming in to be like
Like, where was the synthesis model locally going with that?
Carter: Yeah, exactly.
Yeah.
Maybe that's what happens when you walk into a room to do something or say something and you've forgotten it by the time you get in there,
is that the modeling back is just really latent.
Brad: I feel like that must be like a lossy connection or something.
It's like pinging and it's like, oh, yeah,
you know, retry, retry, retry, just couldn't get the connection.
Carter: Or it's like every time I have to do my two-factor authentication to sign in, I look up and it's an hour later and I've been watching, you know, TikTok reels or something like that. The Edge model has an interrupt that gets interrupted on lower level things.
Brad: oh see see i thought i had that problem solved because i have little key fobs on my on my car key chain um and i'm just like okay great that's not a phone i'm just gonna you know plug it in and press go well of course google decides to log me out when i'm at a hotel that doesn't have any self-parking it's only valet so the valet has my little you know a key fob on my car keys and of course i have a backup that's in my desk drawer at home
from three thousand miles away so that was a that was a rough day i'm just like i hope the rest of the team has it because i can't get into git labs so you know good luck
Carter: that
Brad: you
Carter: is a single point of failure that had never occurred to me before the hotel valet
Brad: you think you've got it because you're like okay it almost never asked me to re-authenticate and when it does i've got my key fob on my keychain and when i don't i've got the one in my desk and somehow
oh but of course because you're traveling so it's like that's a suspicious login and you're away from home so you don't have the backup on your desk and you're out of hotel so it's like it's like a perfect coincidence like i did not think of those correlations of systemic failure
Carter: There you go.
Well, Carter, that seems like a great place to wrap it up with the loss of your car keys.
Brad: Spoiler I did get it back and I did get back in to get lab so just for any compliance folks out there we did not lose the
Carter: We didn't.
Brad: CTO's access to our code base not
Carter: Nobody had to circumvent that particular 2FA.
Brad: this time
Carter: Thank you very much for joining us today.
This has opened by us.
my eyes but I will tell you I don't know if you can see it but my gears are turning here I'm just I've been spending this whole hour thinking about how could we use this what can I do with this this is like a new tool I didn't even know I had my new toy I didn't even know I had my toy box now what am I going to do with this thing
Yeah, for sure.
Brad: Well, I hope you
Yep, try your models for your APIs,
you know, if you're really, you know, going to get...
Getting those gears thinking,
your kind of cloud-based model comes back and says,
wait,
I've got an idea.
Yeah,
go sign up, go try it out. We love hearing about what kind of cool stuff people do with our tech.
So super,
super great to be chatting with you both. Thank you so much. I really
Scott: Yeah.
Brad: appreciate it.
Scott: Yeah.
I'm so glad you're able to come on and chat with us about this.
Very, very cool product.
Very cool solution to a hard problem.
Problem.
I love that you're solving at a scale.
I love that you're thinking outside the box of the current sort of like the current hammer that's solving all of the things.
So yeah,
love it.
Well,
there you go.
I guess we'll say to our listeners,
thanks for listening to another episode of the How Many CTOs Does It Take podcast.
Thanks for coming,
Brad: Carter. Thanks, Carter.
Scott: See you next week.
Brad: See you next week,
Brad.
Scott: This episode of How Many CTOs Does It Take Podcast was edited by Safran Epigow. Our theme song was AI Generated by Brad.
Please follow us on all the social media platforms at at How Many CTOs Pod.
You can find us on YouTube,
LinkedIn,
Instagram,
Facebook,
Threads, X,
TikTok and Patreon or at our website at howmanyctospod.com.
And remember to rate, review and subscribe on the podcast platform of your choice.
Thanks for listening.