Transcript: Beyond Transcripts: Language Nuances and Audio Signals with Carter Huffman of Modulate
0:00 Brad With these big foundation models like OpenAI's and everything,
0:03 Brad you just throw a ton of data and it'll figure it out. For these small,
0:06 Brad tiny, efficient models,
0:07 Brad you have to label the data really, really well.
0:27 Scott We would talk on the phone and be like,
0:28 Scott oh, this is such a, we should make a podcast.
0:30 Scott Then we just started recording the phone calls.
0:32 Brad Oh, amazing.
0:33 Brad That's fantastic.
0:34 Brad I love
0:34 Scott Yeah.
0:35 Brad that. I should do that with my co-founder just six hours back and forth about like culture and all that kind of stuff.
0:40 Brad I'm sure everyone would just listen to that verbatim.
0:42 Carter Oh, well, I mean, at least the 30 listeners that we have would come listen to it.
0:46 Brad Okay,
0:46 Brad brilliant.
0:47 Brad All right. Perfect.
0:48 Brad Wonderful.
0:48 Scott All right. Everybody welcome to this week's episode of How Many CTOs Does It Take?
0:54 Scott My name is Scott Porad and this is my trusty co-host.
0:57 Scott Bruce and
0:58 Carter Brad Hefta-Gaub.
1:00 Scott Brad who do we have with us here today?
1:02 Carter We have a very special guest with us today,
1:04 Carter Carter Huffman,
1:06 Carter the CTO and co-founder of Modulate AI.
1:10 Carter And Carter,
1:11 Carter would you like to introduce yourself and tell us what brought you here?
1:15 Carter How'd you get here?
1:16 Carter Not flights,
1:18 Carter but, you know.
1:18 Brad oh yeah right right so i was in this british airways flight for 11 hours typing like a t-rex yeah and that's why i'm on your podcast everyone wants to hear about that i'm sure uh hi everybody i'm carter hoffman i'm the cto and co-founder over at modulate we're a voice ai company we deal with uh cost effective accurate voice ai models at scale there's a ton of fun stuff to talk about there um and before that i uh was doing machine
1:43 Brad machine learning and artificial intelligence over at the Jet Propulsion Lab so working on some cool like spacecraft autonomy machine learning so if we want we can talk a little bit about that too uh
1:53 Carter Very cool.
1:54 Scott That does sound very cool.
1:55 Scott Like you were working on autonomous flying airplanes,
1:58 Scott autonomous
1:59 Brad autonomous science on spacecraft uh which is which is a autonomous
2:04 Scott science on spacecraft.
2:05 Brad science yeah so so what that is is basically like um
2:09 Brad here's the here's the one kind of one i love the most um is okay so you've got like a mission to a meteor like an asteroid comet what have you right and these missions are expensive right you know 100 million dollars you spend 10 years building this spacecraft and you send it up and it travels for two years and it flies by a comet you know a light hour away from earth right that flyby takes
2:35 Brad takes 15 minutes maybe max right 15 minutes you're there with this comet all that stuff for 15 minutes and as you're doing this flyby maybe you see something interesting like a little jet of gas coming out of the of the comet or a little bright spot or something and if a human's looking at that on earth they're like oh that's really cool let's go take a closer look but it costs so much money and so much time to get there uh no one's ever going back to that comet ever again this is the only time you
3:02 Brad time humanity is going to see it and so the stuff i was working on was like can you with your decade-old you know less powerful than a first generation iphone cpu on this spacecraft find interesting stuff while you're doing the mission and then go kind of do a follow-up observation on it even without a human in the loop it was really cool stuff
3:26 Brad That
3:26 Carter That is pretty cool.
3:28 Scott You know, a hundred million dollar spacecraft mission in the age of AI data center spend sounds like a bargain.
3:36 Brad You know what, that's right. We should just send chat GPT to every single asteroid in the solar system.
3:44 Brad That's amazing.
3:45 Brad Because you know what AI needs? AI needs more capex.
3:49 Brad That's what AI really needs. They need to spend more money. That's what was missing.
3:52 Carter Isn't that isn't that the strategy behind?
3:54 Carter The Elon merging XAI with SpaceX,
3:57 Carter like, you know, all the CapEx just looks the same at that point.
4:01 Brad Those games are so beyond me. When I run multiple trillion dollar companies,
4:07 Brad I will come back to you and I will say, okay,
4:09 Brad now I finally get it. Now I finally know why Twitter has to go to space,
4:13 Brad right?
4:14 Brad But, you know, I'm going to have to get back to you on that one.
4:17 Carter Exactly.
4:17 Carter Exactly.
4:18 Scott So how was your journey?
4:20 Scott From JPL to Modulate. How did that come about?
4:24 Brad It was really cool. Yeah, because you don't necessarily think like, okay,
4:26 Brad NASA and then go straight to like, like, like software tech or something.
4:30 Brad But there's actually this really cool parallel.
4:33 Brad So at JPL,
4:35 Brad I was telling you about kind of doing this low power machine learning stuff on the spacecraft.
4:39 Brad And that's a lot of kind of analyzing signals and variation in signals over time.
4:45 Brad And around that time, this is 2014,
4:48 Brad 2015 or so, people were doing some really cool stuff with AI and machine learning,
4:52 Brad like early days pre-LLM.
4:54 Brad So,
4:55 Brad for example, one of the things I really loved was called GANs, Generative Adversarial Networks. And that had this cool capability where you could take a picture of me with my glasses and you could use the GAN network to take my glasses off and it would make a kind of realistic image. And you could do all sorts of futzing, change my hair.
5:11 Brad my hair color this sort of thing and like nowadays these are in all the apps like nobody thinks that's really impressive but that was the first time you could really do that with high quality images and the really cool thing about neural networks like the entire reason why these have you know because because an llm is just a neural network a transform is just a neural network right the entire reason why these algorithms are so cool is they're very very agnostic to the input kinds of data so you can do something cool with the neural net on an image
5:37 Brad or on text or on a video.
5:40 Brad I was seeing this stuff and I was like well you could do that for audio too uh and audio is like okay that's just a signal with a bunch of variations and stuff like that you know I was a physicist by training like they deal with waves like quantum waves all the time these sensor measurements those are kind of like waves audio is a wave and I'm just like it's all waves I know the math behind this already why don't I work on audio um and so that's when I kind of just like like nights and weekends you know over
6:07 Brad For a few years, I started getting into how can I apply these AI neural networks to audio?
6:12 Brad It's got to be possible.
6:14 Brad I ran into a ton of challenges,
6:16 Brad but that's sort of where it came about.
6:17 Carter Cool. Did you already have a passion for audio? Like, you know, like you just said, are you a music nerd or something like that as well? Or is it or is it more just kind of like what types of waveforms can I apply neural networks to?
6:30 Brad It's so much worse than that. I'm a math and data nerd.
6:33 Brad what a nerd so i was like oh there's data type but i understand you know what i was i was in um at stanford i was a math camp in stanford when i was in high school uh and i we did a little impromptu acapella group and so for the talent show i was part of a five person part of your world acapella so um
6:54 Carter There you go.
6:54 Brad yeah probably the nerdiest uh audio there is out there but
6:58 Brad um but no it was really the signals in the data type it's like hey i understand how to manipulate this because audio is really cool it's like um and voice is really cool like you think when you're doing machine learning on images you generate a face you can see a face in anything right you can you see faces in the clouds all the time you can see a face in anything so it messes up a little bit you're still going to see a face it's actually fine we're very tolerant if you mess up audio a little bit it
7:23 Brad we immediately say that sounds off uncanny valley straight away and there's like evolutionary arguments for this of like okay if your voice sounds wrong maybe you're sick or you know maybe you're lying or something like that but audio is such a severe environment
7:37 Carter Interesting.
7:38 Brad and so you have to massage and manipulate the data to make machine learning work on audio pretty well
7:43 Carter Very cool.
7:45 Scott So, I mean, I got a lot of questions I could ask, but I'm going to kind of just go to the top of the pyramid of questions.
7:50 Scott How are people using it?
7:51 Brad Fantastic question.
7:53 Brad I love that pyramid.
7:54 Brad So primarily right now what our technology does is it understands conversations very,
8:03 Brad very thoroughly and cost effectively.
8:05 Brad So what happens is we're deployed in video games.
8:10 Brad call centers and other areas like that so you have two people talking in a game and they're harassing each other right or somebody's harassing somebody else we're picking up on that and saying okay based on what you said but also the tone of voice the emotion the back and forth of the conversation maybe your history have you been you know spatting at each other for the past 10 minutes or is that just coming out of nowhere
8:35 Brad where what is this person's intent are they really being toxic to this other person or are they just trash talking you something like that that's kind of where we got started um and that's a really cool challenge because you have to be right if you ban a gamer and they didn't deserve it they're gonna go on twitter they're gonna yell at you they're gonna be super mad they're never gonna play the game again they'll probably be mad on twitter anyway but uh you know they're they're it's kind of a thing you really don't want to get wrong
9:02 Brad wrong
9:02 Carter Right.
9:02 Brad and you also have to do it at scale so like a big game call of duty fortnight they could have a hundred million hours a month or a billion hours a month audio and at transcription costs you know you call it ten cents to a dollar per hour nobody's gonna pay a hundred million dollars per month to just you know make their make their games
9:23 Carter Yeah.
9:23 Brad here
9:24 Scott I kind of be honest I have a hard time imagining how this works so an audio stream comes into your model and then what comes back out you know like if I think about chat GPT I put in a stream of words and I get a stream of words back out right but here like I'm putting in a stream of audio and I actually want to get
9:43 Scott like some metadata back out about it like
9:45 Brad Yeah.
9:46 Scott what was the tone yeah how does how does that work
9:48 Brad So we actually have this kind of multi-level representation of a conversation that we build, and then we like update it over time as a conversation is going on.
9:59 Brad So you get a transcript,
10:00 Brad for example, right, that's kind of easy,
10:02 Brad you just get a transcript that's sort of this streaming set of text.
10:05 Brad What were you saying?
10:06 Brad You're also getting kind of an emotion signal over time. So we have like
10:09 Brad you have like labels like okay you're angry right now you're angry right now you're angry right
10:12 Scott Interesting.
10:12 Brad now now you calm down now you come down now you come down um those kinds of things accent labels same exact kind of thing so so that's that's what we call the signals right you take in the audio and you're extracting these low level sort of individual signals from the audio of like what's the tone of voice what's the emotion what's the accent what's the text content of what you're saying etc
10:34 Brad That's kind of layer one. And then on top of that, we're running a separate set of models.
10:40 Brad They're taking in that stream and constantly saying,
10:43 Brad okay,
10:44 Brad what's my take on what's happening right now? Is somebody harassing somebody else? Yes or no?
10:48 Brad Is somebody trying to manipulate somebody else? Yes or no?
10:51 Brad Is somebody talking to a child in a way they're not supposed to? Yes or no.
10:55 Brad And we've got these kind of models running along and they're constantly evaluating all of these rich signals, combining them together and saying at any given time, are we confident in any of these bad things happening or any of these interesting things happening?
11:08 Carter Yeah, cool.
11:09 Carter And so if I'm picturing this correctly,
11:11 Carter whether or not you're using a you're using some.
11:14 Carter neural net tech,
11:16 Carter which is essentially,
11:17 Carter I mean, the really simplified way of thinking about all these neural networks are signal in, neural network.
11:23 Carter signal out and you train the neural network to give you signal out that matches what you want the signal out to be based on the signal in so your signal out might be aggressive or friendly that's the signal out that you're training on and the signals that are coming in are these multiple layers of these other signals that have gone through neural networks to produce each of their kind of signal layers that's is that a fair way of describing it or overly simplistic
11:47 Brad No, no, no. I think that's a fantastic view.
11:49 Brad view of it and basically the magic comes from layering on a bunch of these so if you want to go straight from just a raw audio signal audio audio waveform to is this person spreading violent radical propaganda that's a really hard problem but if you go first to say well what tone of voice are they using
12:08 Brad what are they saying where do they you know what are all these different pieces and then you say okay based on those different pieces if we're confident in those how can I put that together into something more complicated you do like three levels of that hierarchy now one you can be a lot more confident in the result because you're built off of building blocks that apply to more situations you've got kind of more relevant data that you can train on and it's explicable as well
12:33 Brad Why do we think you're being this kind of violent extremist? Well,
12:37 Brad because you were really angry,
12:39 Brad you said these words, and when this person said a dog whistle,
12:43 Brad you did this thing to it, right?
12:44 Brad All of those different pieces can come together in an explicable way.
12:48 Carter Yeah.
12:49 Scott language. Is it language agnostic or is it English only?
12:52 Scott How's that work?
12:52 Brad Good question.
12:53 Brad So the whole system has support for over 60 languages.
12:58 Brad But this ensemble tech that I'm talking about, when you really dig into it,
13:03 Brad it turns out that having just one neural network support 60 different languages is so expensive. So kind of the whole trend for AI.
13:14 Brad is build a bigger model build a bigger model build a bigger model and the great thing about big models is they're so flexible you can ask chat gpt anything and it'll have an answer it
13:24 Scott Might not be true.
13:25 Brad might not be true and you're spending so much power and energy and water and everything right to get to that answer and almost you know that the chat gpt also has the bible memorized but it's not using the bible to answer your question probably like
13:39 Brad we like
13:39 Carter Right.
13:39 Brad you're wasting so much of that neural network just to get you your one answer about you know what what breed of dog is whatever whatever something so you're wasting so much capacity when you're building this big general network we go literally the exact opposite we say let's have a hundred different tiny networks and each of them aren't going to generalize well they're only going to do one thing on one type of data maybe like we only do you
14:03 Brad a good job transcribing romance languages in discord quality audio but you call in on an eight kilohertz phone line it's going to be trash but then you have another model that handles the phone line and another model that handles another thing and then you have to pick you orchestrate between those different models and say at any given time which model is the right one to run and that one will handle it so each individual model handles maybe one or a few languages
14:29 Brad languages, but the whole
14:30 Carter Yeah.
14:30 Brad system can handle a bunch.
14:31 Carter Yeah.
14:32 Carter And I can picture that like the neural network that's transcribing from, you know, like you said, English to over, you know, the low quality audio stream is.
14:42 Carter going to be very different than the neural network that is trying to pick up tone, anger or happiness or whatever the tone, the tone neural network. And I would imagine that some of these neural networks probably are a little bit language agnostic, like maybe the tone one is, is maybe it's not culturally agnostic, but it would be language agnostic, right? So I'm assuming that this is really kind of the magic you've built all these different pieces that are good at really,
15:07 Carter really good at.
15:08 Carter That one part of the problem and then orchestrating them all together to put it together is the magic.
15:14 Brad You've got it 100% right.
15:16 Brad Yeah, each different kind of model has different properties. So like the emotion model,
15:20 Brad for example, yes, it's our basic emotion models tend to be more language agnostic.
15:25 Brad But on the flip side,
15:27 Brad you try to apply it to Mandarin or another tonal language where the tones are actually meaningful.
15:32 Brad And it
15:33 Brad it can't figure it out at all you have to have a separate total language emotion model otherwise you're going to be super super confused kind of like normal people can be confused when somebody else is speaking a different language and you're like wow that german person is always angry all the time right no it's just you know okay do you have your german emotion model to yes or no right
15:51 Carter Du bist mein Liebchen!
15:53 Brad yeah exactly how dare you right you know um so so it and but that actually brings out a fantastic point really because with with these big foundation models like open ais and everything you just throw a ton of data and it'll figure it out spend a ton of money doing it but you just throw in a ton of data for these small tiny efficient models you have to label the data really really well
16:16 Brad Oh, and so if you have a bunch of people, you know, from Boston labeling a ton of German or Russian data,
16:23 Brad they're going to get it wrong and our model is going to suck.
16:26 Brad And
16:27 Scott Yeah.
16:27 Brad so one of the key pieces for us was actually finding,
16:30 Brad vetting and coordinating like globally distributed set of folks who can give us the right labels and take on the data so that we can train these systems correctly.
16:41 Carter That's interesting.
16:42 Scott How big is your team?
16:43 Brad About 45 people right now, about,
16:46 Brad you know, maybe 17,
16:48 Brad 18 engineers and then 15 or so commercial folks,
16:54 Brad couple ops people,
16:55 Brad few executives that do podcasts and stuff.
16:59 Brad Most of the people do the real work.
17:01 Carter I was going to ask sort of similar,
17:03 Carter I think, as you're talking about like the people doing labeling and things like that.
17:07 Carter I mean, I'm assuming those aren't part of that's part of your magic is like finding the labelers, you know, outsourcing the right labelers, finding a way to vet them.
17:16 Carter What is the hardest problem you've had to solve in in in doing this?
17:20 Carter Like, I mean, it sounds like so many hard problems.
17:23 Carter What's the hardest problem you've had to solve?
17:24 Brad There are a couple. I guess you could say one really difficult one has been all the different definitions.
17:32 Brad that different customers or team members or anybody has for certain kinds of behaviors because if you're just transcribing something if we were just a transcription company that's relatively unambiguous not perfect but mostly you can agree on what this person said in any given moment but if you're saying well is this hate speech because you said this awful slur well
17:57 Brad it might be but it might be reclaimed language because people use slurs in positive ways but
18:01 Carter Yeah.
18:02 Brad you can only do that in certain situations at certain times uh and it becomes very very nuanced and then you know we kind of go and form this really nuanced opinion about like when is the slur okay or not and the customer comes in and says not super nice uh you just can't say that in my game and so we're like okay well that's a completely separate
18:23 Brad kind of take on that problem and
18:26 Carter Yeah.
18:27 Brad so we're trying to build an anti-harassment model and everybody has different definitions of harassment
18:34 Carter Yeah.
18:35 Carter Yeah.
18:36 Carter Yeah. I was one of the teams that I work with is in the cultural institution space and like working with museums and things like that. And they were confronted with this recently and they're like, well, what do we do about the fact that like.
18:49 Carter If someone's going to be talking about
18:53 Carter black history museum where slavery and lynching is discussed like we can't just like say you can't use the word lynch like like not if you're having a intellectual conversation about a historical fact right so it's an interesting problem
19:09 Brad Absolutely. Or take, you know, video games where you're talking about, oh,
19:13 Brad go over, kill people or OK,
19:15 Brad well, I'm just going to, you know, I'm just going to commit suicide and then I'll respawn or whatever.
19:18 Brad or whatever right like it is very difficult for a neural network to figure out if that's acceptable language or not it's a really really hard problem and so we had so much time and energy and and effort and money put into how do we build out these really nuanced understandings of all these things and then where do you draw the line between your model because you have to classify some of this stuff otherwise you're just handing people a stream of
19:43 Brad stream of text and say you figure it out but
19:45 Scott Yeah,
19:45 Scott yeah
19:45 Brad on the other side if you're too opinionated then everybody's going to have a different opinion and they're going to have to circumvent your model anyway so what was the point finding that balance of how much control versus how much do you build
19:59 Brad a really really tough problem everybody has different opinions on it
20:03 Scott Totally.
20:03 Brad the other really tough problem maybe if you want to talk more just like hard tech was coordinating all these models so most folks is like a one big model you run the data in and then once you get the answer out you're good
20:17 Brad But we have this ensemble of like 100 different models,
20:20 Brad and they're all running on different resources.
20:23 Brad One of the nice things about our ensemble approach is you can run on a 10-year-old GPU because it's a tiny model,
20:29 Brad you'll still get acceptable performance,
20:31 Brad you don't need the latest and greatest.
20:32 Brad So we've got all those little clusters of different models.
20:35 Brad And normally you just kind of push the data through and then you get all the data back out,
20:39 Brad you combine it all together and you get an answer. And that's great.
20:42 Brad But when you're operating at 100 million hours a month or a billion hours a month,
20:46 Brad having to wait for every machine to respond to every answer,
20:51 Brad now you've got this massive bottleneck and you slow everything down and lock everything up.
20:56 Brad So we have to do really kind of relaxed best effort parallelization, which means not all the data gets together at the time that it's needed to make a decision. And then you make a decision.
21:08 Brad and if you want to go audit that decision later you're like well did it even have all the data in the first place like why did it make that decision it's supposed to be explicable but what happens when you know some of the models drop out and so making this stuff robust to how can i give still give good guarantees and good answers when i might have 10 of my models fail to return a result in time and i have to just do the best i can
21:32 Carter That's wild.
21:33 Carter At that scale,
21:34 Carter like that's.
21:35 Carter That's when stuff gets really interesting for It sure.
21:37 Brad was it was really fun when Call of Duty launched, I think this was the November 2023 launch,
21:43 Brad and we ran them out of A-10 GPUs in U.S. East 1.
21:50 Brad And so we're like,
21:52 Brad OK,
21:52 Brad we've taken 2,000 of your GPUs, and now we don't have any more.
21:56 Brad And we're like,
21:58 Brad OK,
21:58 Brad you know, we had some people on a team going and being like.
22:00 Brad being like okay which other region has these GPUs okay US West 2 all right awesome I'm going to submit a request for quota increase and then the rest of us are like okay which models do we really need how do we balance out like okay you know how important is accent right now you know or or like can we
22:20 Brad Get 80% of the accent coming through to get 95% of the transcriptions coming through.
22:24 Brad We were like balancing that in real time while this game was launching.
22:28 Carter Wild.
22:29 Carter You mentioned video games a handful of times,
22:31 Carter and I've been sitting here trying to imagine how,
22:33 Carter what are ways that this could be used?
22:35 Carter And so I'm just curious if you're willing to share, what are the different ways that the model is being used?
22:40 Carter Because I bet there's some ways out there that people,
22:42 Carter myself.
22:44 Carter I couldn't possibly imagine.
22:45 Brad Well, I think it's it's it's really cool because that's kind of a bit of the evolution of our company was we started out in gaming. That was our beachhead market,
22:52 Brad which is why I talk about it a lot.
22:53 Brad We just have so much history there and I'm a gamer and we were publishing these case studies and stuff like that.
23:00 Brad And we got some folks reaching out to us saying,
23:03 Brad hey, you can find harassment in these games.
23:07 Brad We're a delivery company.
23:09 Brad We have drivers going and delivering stuff physically to people's doors.
23:14 Brad And if they have questions, or the person receiving the package has a question or something,
23:22 Brad they can call the delivery driver.
23:25 Brad And most of the time, that's great.
23:26 Brad It's just like, hey, FYI, my door code is whatever.
23:29 Brad But sometimes they'll call and say, where's my package?
23:33 Brad And if you're like, well, I'm on the way, maybe they're in a rush and they get really upset.
23:37 Brad And a smaller fraction of the time people get really upset and start making threats and a very small percentage, but tragically not zero percent of the time they follow through on those threats.
23:47 Brad And it's really rough because you've got to go bring this thing to their door.
23:50 Brad So you're going to meet this person that is calling you on the phone threatening your life.
23:54 Brad And.
23:56 Brad And so obviously you just don't want to go do that.
23:58 Brad But the problem and a lot of these delivery companies and two-sided marketplaces, right,
24:04 Brad all these sorts of things.
24:05 Brad The problem you get is that the folks doing these deliveries, their services, if they cancel.
24:13 Brad they're worried that that counts against them and they might lose their job and they might
24:17 Carter Yeah.
24:17 Brad lose their source of income and it doesn't matter how much the company says you know that's an okay reason to do that like metric traffic isn't perfect people are worried about losing their jobs so they'll go through with these unsafe situation
24:29 Carter Yeah.
24:30 Brad And so this company reached out to us and said,
24:32 Brad we want you to monitor the phone calls and find these threats of violence and we'll stop the delivery so that we don't have to force the driver to choose between potentially their job and their own safety.
24:43 Carter Yeah.
24:44 Brad So we were doing that.
24:47 Brad We were running our models, running our models, and we were finding all these high stress, emotionally intense situations that didn't involve threats of violence. And we're like, what could be so stressful about receiving a package or receiving an order?
25:00 Brad And we were we were looking in and what we found is that we were actually seeing attempts at manipulation and fraud and the person calling the driver and trying to get them to change the order.
25:13 Brad or what have you and emotionally pressuring them and so parts of our models were picking up these really stressful situations the rest of our models didn't know how to handle and so we were looking at that and we were like wow we were finding this stuff isn't this weird and the company was like oh crap okay yeah that's really bad we need to catch that too can you catch that for us as well so customers actually led us into finding these things so now we're doing we're doing anti-fraud we're doing like call centers you're calling in and
25:39 Brad and and you know you're like somebody's trying to socially manipulate their way into your you're you're getting a password reset and the agent says sure fine yeah that sounds like a tough situation you lost your phone you're traveling yeah here's here's a new password um we find all that kind of stuff um and escalate it so that folks uh can can solve those problems now the newer use cases come around like
26:07 Brad uh when ai's voice agents start misbehaving so
26:11 Carter Yeah,
26:11 Brad i don't know if you all have run into you know any fun little ai voice stories or anything like that we have some good ones over here but
26:18 Carter it's pretty interesting. So we use an outbound.
26:22 Carter all ai voice agent a tasker to call piloting right now to call either clients or taskers in certain scenarios and it never occurred to me that the ai could get snippy could get potentially get frustrated and behave badly but it probably could
26:41 Brad Oh, we have one customer who's an interviewing company and they have an AI bot that they were trialing to.
26:49 Brad to do their first round interviews and they told the bot to check if the candidates were flexible flexibly minded and it started asking them to do yoga poses on a phone call and and they're just like what you know what on earth so so that's actually kind of a new part of our business which is which is interesting is like you know we're listening to these AI bots on the phone calls and they just do stupid stuff that no human would normally do and you don't even know to look for it
27:15 Brad for it and we're like okay can we find all this inappropriate stuff and escalate it so you can be like okay great my interview bot is asking people to do yoga poses that's bad let's go fix it
27:24 Carter I feel like you've also, like, landed on essentially a better version of a polygraph, right?
27:29 Carter It's essentially lie detecting or could be used for that. If you're not already doing that, I have to assume someone's using it for that.
27:34 Brad I can't confirm or deny that
27:36 Carter Can't confirm or deny that.
27:37 Carter Wait a second.
27:38 Carter My model says that may have been a dodge.
27:42 Brad Yes, it is. It's like, I think ultimately speaking,
27:45 Brad what I find so compelling about what we're working on here is that conversation is so much more than just the textual content of what you're saying.
27:55 Carter Yeah.
27:55 Brad You pick up so much from my tone of voice, from my emotion,
27:58 Brad from my inflection,
27:59 Brad how it changes over time,
28:01 Brad everything.
28:02 Brad And when you're trying to do something.
28:06 Brad add value do something worthwhile on top of that voice signal taking in the whole range and dynamic of what's being conveyed in that voice signal is incredibly important and when you throw most of that away and just go with the text transcript you can have the smartest system in the entire world consuming that text it will just not have the data
28:29 Carter Yeah.
28:29 Brad to appropriately interact with you and respond with you and so when you're talking to an AI on the phone and you're like oh my god just get me to a human like that's why that it's not because it's not a smart enough algorithm it's because it doesn't have the data to do it correctly we
28:43 Carter We've talked a lot about this in a commercial sense. I've been thinking about is there any sort of like therapeutic uses for this.
28:49 Brad have actually had some folks
28:53 Brad Asking us about putting it on.
28:58 Brad like remote telehealth calls and monitoring the emotion both from just to like almost like QA purpose of just like hey you know is this person kind of responding appropriately to what this patient's kind of signals are but also from a diagnostic perspective which is really interesting and challenge like
29:23 Brad Like anytime you get into AI diagnostics, you get into a very, very challenging and interesting space.
29:28 Brad But what can we pick up?
29:31 Brad Based on the fact that we can listen to hundreds of thousands of people talking about, you know, things in a therapy context,
29:39 Brad what kinds of signals and cues can we pick out that you would only observe at scale?
29:44 Brad And this is especially interesting when you start talking about like more generational trends,
29:50 Brad right?
29:50 Brad Because doctors have tons of training on what kinds of ways can I.
29:56 Brad sort of look into somebody's behavior to help diagnose something that might might be going on with them but everybody's shaped by the society they're in and the people around them and so you get kind of the next generation of folks coming up and they respond to things very very differently from
30:13 Brad what your textbook said and but by looking at that behavior across a very very large range can you start to pick out some of these signals and do a better job helping out a doctor who might not have a ton of context on that
30:25 Carter Yeah,
30:26 Carter You talked about tele,
30:28 Carter you know, digital health,
30:30 Carter something Brad and I spent a minute working on.
30:33 Carter And I was even thinking about like, just what even even about like ambient in
30:38 Carter in the doctor's office.
30:39 Carter the doctor's.
30:40 Carter doctor's office or even just, you know, like I had that thing happen the other day, right?
30:44 Carter where i was talking about getting the tires on my car replaced to my friend and then my friend reported to me that the next day she started getting ads for tires on her phone right because her phone
30:55 Carter Oh,
30:55 Carter her
30:55 Carter I don't
30:56 Carter phone
30:56 Carter like that.
30:56 Carter no nobody likes it but it's true right so her phone heard me talking right so like but think about the ambient you whether it's a doctor's office or just the ambient uses for this thing i don't know what they could be but it's interesting
31:09 Brad I think one of the really cool things, and this starts to get into like a little bit of the future stuff,
31:14 Brad which is like, you know, why we do tech,
31:16 Brad right?
31:17 Brad One of the really cool applications in the future,
31:20 Brad because we've talked a lot about the emotion and context and that understanding is also the sense of scale, right?
31:29 Brad our our our stuff you know we really focus on that lower cost efficiency because again you're not going to pay a hundred million dollars a month to moderate call of duty right so that was kind of our heritage was we've got to be really really low cost at this stuff and so you can think of like okay well today i'm going to transcribe every zoom call
31:50 Brad And you can do that.
31:51 Brad But tomorrow,
31:52 Brad what if I'm going to transcribe just every random conversation that people are happening all around the office or all around, you know, the wherever?
31:58 Brad And that is, on the one hand, super frightening from a privacy perspective.
32:03 Carter Mm
32:03 Brad But on the other hand, there's a lot of potential around being able to keep track of and organize information at a scale that's been completely unheard of in
32:14 Carter -hmm.
32:14 Brad the past that's enabled by being able to do.
32:17 Brad do like really really cutting edge highly accurate sophisticated AI on data coming from just everywhere all at once
32:25 Carter Yeah, and even the strategies that you're talking about, you've applied it to audio,
32:30 Carter but the notion of an ensemble of small models.
32:35 Carter That's a way of thinking about data processing in general,
32:39 Carter right?
32:39 Carter There's other forms of data, whether that be audio data, like you're doing with modulate or video data or, you know, text data or temperature data,
32:48 Brad Sensor data.
32:49 Carter sensor
32:49 Brad Yeah,
32:49 Brad from your
32:50 Carter data.
32:50 Brad car,
32:50 Brad your phone,
32:51 Brad all that kind of stuff.
32:51 Brad Absolutely. Yeah.
32:52 Brad And these these ensembles of small models are so powerful when you're talking about the embedded world as well, because.
33:00 Brad You can't ship chat GPT to your phone or your car or a pin or something like that.
33:05 Brad But you can ship a ton of tiny models. And then as long as you can be pretty smart about picking the best one to actually stick into RAM and run,
33:14 Brad you can get really general high performance without having to run this massive neural net all locally, you know, in your 2014 Prius seat.
33:24 Carter I'm curious.
33:26 Carter So so far, we've only talked about sort of the the input of signal to the signal out being like, you know, inference of like, you know, is this person angry,
33:38 Carter whatever,
33:38 Carter harassing, et cetera.
33:39 Carter Is Modulate also taking this?
33:43 Carter Clever approach that you've done to do the opposite of generating audio that is emotionally relevant,
33:48 Carter or where are you guys on that?
33:51 Brad I have so much ambition around that.
33:54 Brad Our team is so, so excited about it.
33:56 Brad Ultimately,
33:57 Brad we love everything about voice.
33:59 Brad We love everything about voice. We started with these voice skins,
34:02 Brad these voice changers, right?
34:04 Brad Like really low latency,
34:05 Brad really low power.
34:08 Brad run on device while i'm having this call make me sound like scarlett johansson or taylor swift or james earl jones that's where i got started oh god we were working on the research by that a decade ago so we are so interested in high quality audio synthesis all these kinds of things at a 45 person startup you have to find some place to start and so we we are
34:31 Brad So, so focused on building the best, most cost effective,
34:35 Brad fastest conversation understanding models in the entire world.
34:40 Brad And we have so much ambition to take exactly that same strategy and move it to manipulation and synthesis as well.
34:48 Carter Yeah.
34:48 Brad It's actually interesting kind of that you're bringing this up because in the understanding space,
34:54 Brad you can just like fan the data out.
34:56 Brad run it through a bunch of models and then collect everything back it's not immediately obvious how you would do that just to make like a high quality waveform synthesis right
35:06 Carter Yeah.
35:06 Brad like it's just like do you make a ton of different waveforms and then you average them or combine them together that's probably not not not the right answer but what i think is really interesting about the ensemble models that we work on is it's not just a bunch of tiny models running
35:21 Brad running in parallel,
35:21 Brad it's also hierarchies across scale.
35:24 Brad And
35:24 Carter Yeah.
35:24 Brad so one of the biggest problems in audio synthesis,
35:27 Brad especially these voice agents, is latency.
35:30 Brad You want to
35:31 Brad hear something and respond immediately if you want super low latency you need to run on device on the edge if you're running on device on the edge you have to run a less powerful model and so they're either slow and kind of smart that's still not that smart or they're really fast and super super incompetent if you think about starting to break this stuff up you can run a small edge model that could be super incompetent and you can run a small edge model that could be super incompetent
35:58 Brad And then give it context,
36:01 Brad very rich context that you're extracting from a larger, more capable model that's somewhere in the cloud.
36:08 Brad And you can do this multiple different times at different hierarchies. So the small model that itself would be completely incapable of doing this now has all this extra metadata and information from the bigger models elsewhere.
36:20 Carter yeah that makes a ton of sense like you know the model that's basically you know if we just think of it from like
36:26 Carter Text to speech where it's like the model that's taking the text and then generating the voice from it,
36:31 Carter you know.
36:32 Carter As opposed to, I don't know if this is the way you'd approach it,
36:35 Carter but if instead you've broken down the text with the intonation, with the emotional resonance that you're looking for and you've turned that into a different stream of tokens,
36:44 Carter which is, you know, maybe closer to phonemes or something like that, where it's no longer,
36:48 Carter it's not, it's now just solving like the next layer down that can be much more efficient on a faster,
36:56 Carter lighter model.
36:57 Brad exactly right yeah you do the decoding at the edge where you have to do it really fast and then you do the conditioning of somewhere else where you can afford a one second delay if you really really need it and honestly that's kind of how I feel like I work sometimes because like I'll start talking and then as I'm saying the next word I'm like wait a minute what exactly where was I going with what's the next word that I'm supposed to say and so I've got that you know a much more powerful it feels like cloud-based model coming in to be like
37:24 Brad Like, where was the synthesis model locally going with that?
37:27 Carter Yeah, exactly.
37:29 Carter Yeah.
37:30 Carter Maybe that's what happens when you walk into a room to do something or say something and you've forgotten it by the time you get in there,
37:39 Carter is that the modeling back is just really latent.
37:43 Brad I feel like that must be like a lossy connection or something.
37:46 Brad It's like pinging and it's like, oh, yeah,
37:48 Brad you know, retry, retry, retry, just couldn't get the connection.
37:53 Carter Or it's like every time I have to do my two-factor authentication to sign in, I look up and it's an hour later and I've been watching, you know, TikTok reels or something like that. The Edge model has an interrupt that gets interrupted on lower level things.
38:09 Brad oh see see i thought i had that problem solved because i have little key fobs on my on my car key chain um and i'm just like okay great that's not a phone i'm just gonna you know plug it in and press go well of course google decides to log me out when i'm at a hotel that doesn't have any self-parking it's only valet so the valet has my little you know a key fob on my car keys and of course i have a backup that's in my desk drawer at home
38:33 Brad from three thousand miles away so that was a that was a rough day i'm just like i hope the rest of the team has it because i can't get into git labs so you know good luck
38:42 Carter that
38:42 Brad you
38:42 Carter is a single point of failure that had never occurred to me before the hotel valet
38:48 Brad you think you've got it because you're like okay it almost never asked me to re-authenticate and when it does i've got my key fob on my keychain and when i don't i've got the one in my desk and somehow
38:59 Brad oh but of course because you're traveling so it's like that's a suspicious login and you're away from home so you don't have the backup on your desk and you're out of hotel so it's like it's like a perfect coincidence like i did not think of those correlations of systemic failure
39:12 Carter There you go.
39:13 Carter Well, Carter, that seems like a great place to wrap it up with the loss of your car keys.
39:20 Brad Spoiler I did get it back and I did get back in to get lab so just for any compliance folks out there we did not lose the
39:27 Carter We didn't.
39:28 Brad CTO's access to our code base not
39:31 Carter Nobody had to circumvent that particular 2FA.
39:34 Brad this time
39:34 Carter Thank you very much for joining us today.
39:37 Carter This has opened by us.
39:38 Carter my eyes but I will tell you I don't know if you can see it but my gears are turning here I'm just I've been spending this whole hour thinking about how could we use this what can I do with this this is like a new tool I didn't even know I had my new toy I didn't even know I had my toy box now what am I going to do with this thing
39:56 Carter Yeah, for sure.
39:58 Brad Well, I hope you
39:59 Brad Yep, try your models for your APIs,
40:01 Brad you know, if you're really, you know, going to get...
40:03 Brad Getting those gears thinking,
40:04 Brad your kind of cloud-based model comes back and says,
40:06 Brad wait,
40:06 Brad I've got an idea.
40:08 Brad Yeah,
40:08 Brad go sign up, go try it out. We love hearing about what kind of cool stuff people do with our tech.
40:14 Brad So super,
40:16 Brad super great to be chatting with you both. Thank you so much. I really
40:20 Scott Yeah.
40:20 Brad appreciate it.
40:21 Scott Yeah.
40:21 Scott I'm so glad you're able to come on and chat with us about this.
40:24 Scott Very, very cool product.
40:26 Scott Very cool solution to a hard problem.
40:29 Scott Problem.
40:30 Scott I love that you're solving at a scale.
40:31 Scott I love that you're thinking outside the box of the current sort of like the current hammer that's solving all of the things.
40:40 Scott So yeah,
40:41 Scott love it.
40:42 Scott Well,
40:42 Scott there you go.
40:43 Scott I guess we'll say to our listeners,
40:45 Scott thanks for listening to another episode of the How Many CTOs Does It Take podcast.
40:50 Scott Thanks for coming,
40:51 Brad Carter. Thanks, Carter.
40:51 Scott See you next week.
40:52 Brad See you next week,
40:53 Brad Brad.
40:54 Scott This episode of How Many CTOs Does It Take Podcast was edited by Safran Epigow. Our theme song was AI Generated by Brad.
41:02 Scott Please follow us on all the social media platforms at at How Many CTOs Pod.
41:07 Scott You can find us on YouTube,
41:09 Scott LinkedIn,
41:10 Scott Instagram,
41:11 Scott Facebook,
41:11 Scott Threads, X,
41:12 Scott TikTok and Patreon or at our website at howmanyctospod.com.
41:18 Scott And remember to rate, review and subscribe on the podcast platform of your choice.
41:24 Scott Thanks for listening.