Hello and welcome to a audio data set consisting
of one single episode of a non-existent podcast. Or I
may append this to a podcast that I set up
recently regarding my with my thoughts on speech
tech and AI in particular, more AI in generative AI,
I would say. But in any event, the purpose of
this Voice recording is actually to create a lengthy
voice sample for a quick evaluation, a back of the
envelope evaluation, as they might say, for different speech attack
models. And I'm doing this because I thought I had
made a great breakthrough in my journey with speech tech,
and that was succeeding in the elusive task of fine-tuning
Whisper. Whisper is, and I'm going to just talk, I'm
trying to mix up, I'm going to try a few
different styles of speaking. I might whisper something at some
point. As well. And I'll go back to speaking loud
in, in different parts. I'm going to sound really like
a crazy person because I'm also going to try to
speak at different pitches and cadences in order to really
try to put a speech attacks model through its paces,
which is trying to make sense of is this guy
just rambling on incoherently in one long sentence or are
these just actually a series of step, standalone,
step alone, standalone sentences? And how is it gonna handle
step alone? That's not a word. What happens when you
use speech to text and you use a fake word?
And then you're like, wait, that's not actually, that word
doesn't exist. How does AI handle that? And these and
more are all the questions that I'm seeking to answer
in this training data. Now, why was it trying to
fine tune Whisper? And what is Whisper? As I said,
I'm going to try to record this at a couple
of different levels of technicality for folks who are, you
know, in the normal world and not totally stuck down
the rabbit hole of AI, which I have to say
is a really wonderful rabbit hole to be down. It's
a really interesting area and speech and voice tech is
the aspect of it that I find actually the most,
I'm not sure I would say the most interesting because
there's just so much that is fascinating in AI. But
the most that I find the most personally transformative in
terms of the impact that it's had on my daily
work life and productivity and how I sort of work.
And I'm persevering hard with the task of trying
to get a good solution working for Linux, which if
anyone actually does listen to this, not just for the
training data and for the actual content, this is sparked
I had, besides the fine tune not working, well, that
was the failure. Um, I used Claude code because one
thinks these days that there is nothing short of solving,
you know, the, the reason of life or something, that
Claude and agentic AI can't do, which is not really
the case. Uh, it does seem that way sometimes, but
it fails a lot as well. And this is one
of those, instances where last week I put together an
hour of voice training data, basically speaking, just random things
for 3 minutes. And it was actually kind of tedious
because the texts were really weird. Some of them were
it was like it was AI generated. I tried before
to read Sherlock Holmes for an hour and I just
couldn't. I was so bored after 10 minutes that I
was like, okay, no, I'm just going to have to
find something else to read. So I used a created
with AI studio vibe coded a synthetic text generator. Which
actually I thought was probably a better way of doing
it because it would give me more short samples with
more varied content. So I was like, okay, give me
a voice note, like I'm recording an email, give me
a short story to read, give me prose to read.
So I came up with all these different things and
they added a little timer to it so I could
see how close I was to one hour. And I
spent like an hour one afternoon or probably two hours
by the time you you do retakes. And whatever, because
you want to, it gave me a source of truth,
which I'm not sure if that's the scientific way to
approach this topic of gathering, training data, but I thought
made sense. Um, I have a lot of audio data
from recording voice notes, which I've also kind of used,
been experimenting with using for a different purpose, slightly different
annotating task types. It's more a text classification experiment
or, Well, it's more than that actually. I'm working on
a voice app. So it's a prototype, I guess, is
really more accurate. But you can do that and you
can work backwards. You're like, you listen back to a
voice note and you painfully go through one of those
transcribing, you know, where you start and stop and scrub
around it and you fix the errors, but it's really,
really boring to do that. So I thought it would
be less tedious in the long term if I just
recorded the source of truth. So it gave me these
three minute snippets. I recorded them. It saved an MP3
and a TXT in the same folder, and I created
an error with that data. So I was very hopeful,
quietly, a little bit hopeful that I could actually fine
tune Whisper. I want to fine tune Whisper because when
I got into Voicetech last November, my wife was in
the US and I was alone at home. And when
crazy people like me do really wild things like use
voice to tech technology. That was basically when I started
doing it, I didn't feel like a crazy person speaking
to myself. And my expectations weren't that high. I used
speech tech now and again, tried it out. It was
like, it'd be really cool if you could just, like,
speak into your computer. And whatever I tried out that
had Linux support was just. It was not good, basically.
And this blew me away from the first go. I
mean, it wasn't 100% accurate out of the box and
it took work, but it was good enough that there
was a solid foundation and it kind of passed that
pivot point that it's actually worth doing this. You know,
there's a point where it's so like the transcript is
you don't have to get 100% accuracy for it to
be worth your time for speech attacks to be a
worthwhile addition to your productivity, but you do need to
get above, let's say, I don't know, 85%. If it's
60% or 50%, you inevitably say, screw it, I'll just
type it because you end up missing errors in the
transcript and it becomes actually worse. You end up in
a worse position than you started with. That's been my
experience. So I was like, oh, this is actually really,
really good now. How did that happen? And the answer
is ASR whisper being open source and the transformer
architecture. If you want to go back to the to
the underpinnings, which really blows my mind and it's on
my list. To read through that paper. All you need
is attention as attentively as can be done
with my limited brain because it's super, super high level
stuff, super advanced stuff, I mean. But that, I think
of all the things that are fascinating about the sudden
rise in AI and the dramatic capabilities. I find it
fascinating that a few people are like, hang on, you've
got this thing that can speak to you, like a
chatbot, an LLM, and then you've got image generation. Okay,
so firstly, those two things on the surface have nothing
in common. So like, how are they, how did that
just happen all at the same time? And then when
you extend that further, you're like, Suno, right? You can
sing a song and AI will come up with and
instrumental. And then you've got Whisper and you're like, wait
a second, how did all this stuff, like, if it's
all AI, what's like, there has to be some commonality.
Otherwise, these are totally different technologies on the surface of
it. And the Transformer architecture is, as far as I
know, the answer. And I can't even say, can't even
pretend that I really understand what the Transformer architecture means.
In depth, but I have scanned it and as I
said, I want to print it and really kind of
think over it at some point. And I'll probably feel
bad about myself, I think, because weren't those guys in
their 20s? Like, that's crazy. I think I asked ChatGPT
once who wrote that paper and how old were they
when it was published in Arciv? And I was expecting,
like, I don't know, What do you imagine? I personally
imagine kind of like, you know, you have these breakthroughs
during COVID and things like that where like these kind
of really obscure scientists are like in their 50s and
they've just kind of been laboring in labs and wearily
in writing and publishing in kind of obscure academic publications.
And they finally like hit a big or win a
Nobel Prize and then their household names. So that was
kind of what I had in mind. That was the
mental image I'd formed of the birth of Arcsight. Like
I wasn't expecting 20-somethings in San Francisco, though. I thought
that was both very, very funny, very cool, and actually
kind of inspiring. It's nice to think that people who,
you know, just you might put them in the kind
of milieu or bubble or world that you are in
are credibly in through, you know, the series of connections
that are coming up with such literally world changing innovations.
So that was, I thought, anyway. That's that was cool.
Okay, voice training data. How are we doing? We're about
10 minutes and I'm still talking about voice technology. So
Whisper was brilliant and I was so excited that I
was my first instinct was to like guess like, oh
my gosh, I have to get like a really good
microphone for this. So I didn't go on a spending
spree because I said, I'm gonna have to just wait
a month and see if I still use this. And
It just kind of became, it's become really part of
my daily routine. Like if I'm writing an email, I'll
record a voice note. And then I've developed and it's
nice to see that everyone is like developing the same
things in parallel. Like that's my kind of a weird
thing to say, but when I look, I kind of
came, when I started working on this, these prototypes on
GitHub, which is where I just kind of share very
freely and loosely, ideas and first iterations on concepts.
And for want of a better word, I called it
like LLM post-processing or cleanup or basically a system prompt
that after you get back the raw text from Whisper,
you run it through a model and say, okay, this
is crappy text, like add sentence structure and fix it
up. And now when I'm exploring the different tools that
are out there that people have built, I see quite
a number of projects have basically done the same thing,
lest that be misconstrued. I'm not saying for a millisecond
that I inspired them. I'm sure this has been a
thing that's been integrated into tools for a while, but
it's the kind of thing that when you start using
these tools every day, the need for it is almost
instantly apparent because text that doesn't have any punctuation or
Paragraph spacing takes a long time to, you know, it
takes so long to get it into a presentable email
that again, it's, it's, it, it moves speech tech into
that before that inflection point where you're like, no, it's
just not worth it. It's like, it's, it'll just be
quicker to type this. So it's a big, it's a
little touch that actually is a big deal. Uh, so
I was on Whisper and I've been using Whisper and
I kind of, early on found a couple of tools.
I couldn't find what I was looking for on Linux,
which is basically just something that'll run in the background.
It'll give it an API key and it will just
like transcribe with like a little key to start and
stop the dictation. And the issues were I discovered that
like most people involved in creating these projects were very
much focused on local models, running Whisper locally because you
can. And I tried that a bunch of times and
just never got results that were as good as the
cloud. And when I began looking at the cost of
the speech to text APIs and what I was spending,
I just thought there is, it's actually, in my opinion,
just one of the better deals in API spending and
in cloud. Like it's just not that expensive for very,
very good models that are much more, you know, you're
gonna be able to run the full model. The latest
model versus whatever you can run on your average GPU,
unless you want to buy a crazy GPU. It doesn't
really make sense to me. Now, privacy is another concern
that I know is kind of like a very much
a separate thing that people just don't want their voice
data and their voice leaving their local environment, maybe for
regulatory reasons as well. But I'm not in that. I
neither really care about people listening to my grocery list
consisting of reminding myself that I need to buy more
beer, Cheetos, and hummus, which is kind of the three
staples of my diet during periods of poorer nutrition. But
the kind of stuff that I transcribe, it's just not,
it's not a privacy thing I'm that sort of sensitive
about and I don't do anything so sensitive or secure
that requires air gapping. So I looked at the pricing
and especially the kind of older model mini Some of
them are very, very affordable. And I did a back
of the, I did a calculation once with ChatGPT and
I was like, okay, this is the API price for
I can't remember whatever the model was. Let's say I
just go at it like nonstop, which it rarely happens.
Probably, I would say on average, I might dictate 30
to 60 minutes per day if I was probably summing
up the emails, documents, outlines, which
is a lot, but it's still a fairly modest amount.
And I was like, Some days I do go on
like one or two days where I've been usually when
I'm like kind of out of the house and just
have something like I have nothing else to do. Like
if I'm at a hospital, we have a newborn and
you're waiting for like eight hours and hours for an
appointment. And I would probably have listened to podcasts before
becoming a speech fanatic. And I'm like, oh, wait, let
me just get down. Let me just get these ideas
out of my head. And that's when I'll go on
my speech binges. But those are like once every few
months, like not frequently. But I said, okay, let's just
say if I'm gonna price out Cloud SCT, if I
was like dedicated every second of every waking hour to
transcribing for some odd reason, I mean, I'd have to
like eat and use the toilet. Like, you know, there's
only so many hours I'm awake for. So like, let's
just say a maximum of like 40 hour, 45 minutes.
In the hour. Then I said, all right, let's just
say 50. Who knows? You're dictating on the toilet. We
do it. So it could be. You could just do
60. But whatever I did. And every day, like, you're
going flat out seven days a week dictating non-stop I
was like, what's my monthly API bill gonna be at
this price? And it came out to, like, 70 or
80 bucks. And I was like, well, that would be
an extraordinary. Amount of dictation. And I would hope that
there was some compelling reason more worth more than $70
that I embarked upon that project. So given that that's
kind of the max point for me, I said that's
actually very, very affordable. Now you're gonna, if you want
to spec out the costs and you want to do
the post-processing that I really do feel is valuable, that's
gonna cost some more as well, unless you're using Gemini,
which needless to say is a random person sitting in
Jerusalem. I have no affiliation, nor with Google, nor anthropic,
nor Gemini, nor any major tech vendor for that matter.
I like Gemini not so much as a everyday model.
It's kind of underwhelmed in that respect, I would say.
But for multimodal, I think it's got a lot to
offer. And I think that the transcribing functionality whereby it
can process audio with a system prompt and both give
you transcription that's cleaned up that reduces two steps to
one. And that for me is a very, very big
deal. And I feel like even Google has haven't really
sort of thought through how useful the that modality is
and what kind of use cases you can achieve with
it. Because I found in the course of this year,
just an endless list of really kind of system prompt
system prompt stuff that I can say, okay, I've used
it to capture context data for AI, which is literally
I might speak for if I wanted to have a
good bank of context data about who knows my childhood
more realistically, maybe my career goals, something that would just
be like really boring to type out. So I'll just
like sit in my car and record it for 10
minutes. And that 10 minutes you get a lot of
information in. Um, emails, which is short text, just
there is a whole bunch and all these workflows kind
of require a little bit of treatment afterwards and different
treatment. My context pipeline is kind of like just extract
the bare essentials. So you end up with me talking
very loosely about sort of what I've done in my
career, where I've worked, where I might like to work.
And it goes, it condenses that down to very robotic
language that is easy to chunk parse and maybe put
into a vector database. Daniel has worked in technology. Daniel
has been working in, you know, stuff like that. That's
not how you would speak, but I figure it's probably
easier to parse for, after all, robots. So we've almost
got to 20 minutes and this is actually a success
because I wasted 20 minutes of the evening speaking
into a microphone and the levels were shot and it
was clipping and I said, I can't really do an
evaluation. I have to be fair. I have to give
the models a chance to do their thing. What am
I hoping to achieve in this? Okay, my fine tune
was a dud as mentioned. DeepChrom ST, I'm really, really
hopeful that this prototype will work and it's a build
in public open source, so anyone is welcome to use
it if I make anything good. But that was really
exciting for me last night when after hours of trying
my own prototype, seeing someone just made something that works
like that, you know, you're not gonna have to build
a custom conda environment and image. I have AMD GPU,
which makes things much more complicated. I didn't find it.
And I was about to give up and I said,
all right, let me just give Deep Grams Linux thing
a shot. And if this doesn't work, I'm just going
to go back to trying to Vibe code something myself.
And when I ran the script, I was using Claude
code to do the installation process. It ran the script
and oh my gosh, it works just like that. The
tricky thing For all those who want to know all
the nitty gritty details, was that I
don't think it was actually struggling with transcription, but pasting
Wayland makes life very hard. And I think there was
something not running the right time. Anyway, Deepgram, I looked
at how they actually handled that because it worked out
of the box when other stuff didn't. And it was
quite a clever little mechanism. And but more so than
that, the accuracy was brilliant. Now, what am I doing
here? This is going to be a 20 minute audio
sample. And I think I've done one or two
of these before, but I did it with short snappy
voice notes. This is kind of long form. This actually
might be a better approximation for what's useful to me
than voice memos. Like, I need to buy three Bread,
eaters of milk tomorrow and Peter bread, which is probably
how like half my voice notes sound. Like if anyone
were to, I don't know, like find my phone, they'd
be like, this is the most boring person in the
world. Although actually, there are some like kind of journaling
thoughts as well, but it's a lot of content like
that. And the probably for the evaluation, the most useful
thing is slightly obscure tech, GitHub, NeocleNo, hugging
face, Not so obscure that it's not going to have
a chance of knowing it, but hopefully sufficiently well known
that the model should get it. I tried to do
a little bit of speaking really fast and speaking very
slowly. I would say in general, I've spoken, delivered this
at a faster pace than I usually would owing to
strong coffee flowing through my bloodstream. And the thing that
I'm not going to get in this benchmark is background
noise, which in my first take that I had to
get rid of, My wife came in with my son
and for a goodnight kiss. And that actually would have
been super helpful to get in because it was non
diarized or if we had diarization, a female, I could
say, I want the male voice and that wasn't intended
for transcription. And we're not going to get background noise
like people honking their horns, which is something I've done
in my main data set where I am trying to
go back to some of my voice notes. Annotate them
and run a benchmark. But this is going to be
just a pure quick test. And as someone,
I'm working on a voice note idea. That's my sort
of end motivation. Besides thinking it's an ask to the
outstanding technology that's coming to viability. And really, I know
this sounds cheesy, can actually have a very transformative effect.
It's, you know, voice technology has been life changing for
folks living with disabilities. And I think
there's something really nice about the fact that it can
also benefit, you know, folks who are able bodied and
like we can all in different ways make this tech
as useful as possible, regardless of the exact way that
we're using it. And I think there's something very powerful
in that and it can be very cool. I see
huge potential. What excites me about Voicetech? A lot of
things actually. Firstly, the fact that it's cheap and accurate,
as I mentioned at the very start of this. And
it's getting better and better with stuff like accent handling.
I'm not sure my fine-tune will actually ever come to
fruition in the sense that I'll use it day to
day as I imagine. I get like superb flawless words
error rates because I'm just kind of skeptical about Local
speech to text, as I mentioned, and I think the
pace of innovation and improvement in the models, the main
reasons for fine tuning from what I've seen have been
people who are something that really blows my mind about
ASR is the idea that it's inherently a lingual or
multilingual phonetic based. So as folks who use speak
very obscure languages, that there might be a paucity of
training data or almost none at all, and therefore the
accuracy is significantly reduced. Or folks in very critical
environments, I know this is used extensively in medical transcription
and dispatcher work, the call centers who send out ambulances,
et cetera, where accuracy is absolutely paramount. And in the
case of doctors, radiologist, they might be using very specialized
vocab all the time. So those are kind of the
main two things that I'm not sure that really just
for trying to make it better on a few random
tech words with my slightly, I mean, I have an
accent, but like not, you know, an accent that a
few other million people have ish. I'm not sure that
my little fine tune is gonna actually like the bump
in word error reduction, if I ever actually figure out
how to do it and get it up to the
cloud. By the time we've done that, I suspect that
the next generation of ASR will just be so good
that it will kind of be, well, that would have
been cool if it worked out, but I'll just use
this instead. So that's going to be it for today's
episode of voice training data. Single long shot evaluation.
Who am I going to compare? Whisper is always good
as a benchmark, but I'm more interested in seeing Whisper
head to head with two things, really. One is Whisper
variants. So you've got these projects like faster Distill Whisper,
it's a bit confusing, there's a whole bunch of them.
And the emerging ASRs, which are also a thing. My
intention for this is I'm not sure I'm going to
have the time in any point in the foreseeable future
to go back through this whole episode and create a
proper source truth, where I fix everything. Might do
it if I can get one transcriptions that sufficiently close
to perfection. But what I would actually love to do
on Hugging Face, I think would be a great probably
how I might visualize this is having the audio waveform
play and then have the transcript for each model below
it and maybe even a like, you know, to scale
and maybe even a local one as well, like local
whisper versus OpenAI API, et cetera. And, I
can then actually listen back to segments or anyone who
wants to can listen back to segments of this recording
and see where a particular model struggled and others didn't,
as well as the sort of headline finding of which
had the best WER, but that would require the source
of truth. Okay, that's it. I hope this was, I
don't know, maybe useful for other folks interested in STT.
You want to see that I always feel think I've
just said as something I didn't intend to. STT, I
said for those. Listen carefully, including hopefully the models themselves.
This has been myself, Daniel Rosell. For more jumbled repositories
about my roving interests in AI, but particularly agentic, MCP
and Voicetech, you can find me on GitHub, huggingface.com,
which is my personal website, as well as this podcast,
whose name I sadly cannot remember. Until next time, thanks
for listening.