New gadget for me, a Sandisk Sansa Fuze

Newegg was running a sale on these a few days ago and I got a 4 GB Fuze for $39.99 and free shipping:

The geek in me had to have it just for the Ogg Vorbis support even though I had a bad experience with the Sansa View. Long story short, the View has sluggish performance, buggy firmware, and limited format support. (WMA “Standard” codec, and MP3 for audio, and h.264 for video)

MP3 has been an incredibly resilient audio standard since it was introduced in 1991, and has gone on to be the premier audio codec of nearly every online music store, sans Apple iTunes which uses the MPEG Forum’s “official upgrade”  to MP3. This codec is called MPEG-4 AAC.

That having been said, MP3 its showing age and although it was an acceptable format 10 years ago, times change, formats improve, and out with the old. AAC is not the only thing you can replace it with though. MP3 is far from efficient  even with the best encoders, it has severe limitations that just cannot be fixed without breaking decoding compatibility. This is why AAC has been standardized by the MPEG Forum, and it’s probably the second most likely format to be supported in hardware after MP3 and WMA “Standard”. (Tied for devices supported, but there’s no real customer demand for WMA, Microsoft just muscles it onto every player.)

This is where things get sticky, because Microsoft also just so happened to develop an MP3 competitor (if you can call it that) called Windows Media Audio. Most WMA files use the original codec because it’s the only WMA codec that has any kind of hardware support outside of the Zune, which also supports WMA Professional and Lossless with some limitations (2 channel stereo, 44.1 Khz playback). This may limit you if you use a stereo dock. Even though WMA Standard is promoted by Microsoft as “Better than MP3″, this is just patently untrue, and for this reason and the issue of having an undocumented proprietary format and only one encoder (Microsoft’s), even Windows die hards are better off just pretending Windows Media Audio never existed.

The last codec on the list (I saved the best) is Ogg Vorbis. Ogg Vorbis is fully documented, unpatented, public domain, royalty free, and open source, and otherwise all forms of “Do anything you want with it, and nobody even has to pay us”. Even though Ogg Vorbis is clearly superior to even the best of its competition (AAC), and doesn’t cost anything to license, it has had slow uptake due to not having a major corporate backer that can bribe hardware makers, and not having the mind share of MP3. But trust me when I say that out of all the codecs promising “Same quality as MP3 at around half the bitrate”, Vorbis is the only one that really delivers.

I’ve gauged all popular codecs for transparency with the CD source and here’s what I came up with:

(Terminology: VBR = Variable Bitrate / CBR= Constant Bitrate.  Most encoders use VBR quality settings to provide consistent quality throughout the file, and you should use this if at all possible with very rare exception. Modern codecs (AAC and Vorbis) are not designed to operate in CBR mode and you really should never try it. Obviously, the higher the bitrate in each given codec, the better the audio technically sounds, but file size also increases. There’s a point of diminishing returns where you won’t want to go any further and an average person could not tell the CD from the lossy file, this is what “transparency” means. The codec which can achieve transparency with the lowest bitrate is there fore the “best” codec.)

Lame MP3 at VBR setting 0: Average bitrate is 270-300kbps, you hardly save anything vs. 320kbps CBR (and honestly may as well just use that to err on the side of caution. Other MP3 encoders do not achieve transparency for me at any bitrate.

WMA “Standard” 9.2 at VBR q98: At this point the artifacts are gone and it sounds about like the CD, the problem is the files average about 350 kbps.

AAC: VBR at q 0.60: At this point, you get transparency good frequency response and the bitrate is around 225 kbps.

Ogg Vorbis with the AoTuv encoder: VBR quality 5 gives transparency and excellent frequency response, the bitrate will be averaging 160 kbps (!!!) Don’t use the official libvorbis encoder as AoTuv is a highly optimized fork and you will always get better results.

Despite marketing claims, there are no codecs which deliver transparency at 128 kbps. AoTuv’s Vorbis encoder does better than any of the rest.

(BTW, at low streaming bitrates, Vorbis at q3 (~112kbps) or q4 (~128 kbps) will give you the file sizes necessary without the warbling and other distortions of competing codecs.)

I apologize for the long primer, but these are the audio codecs supported by the Sansa Fuze:

It took me a while to re-rip all of my CDs in Ogg Vorbis q5 (I have a lot of CDs!), and I’m still not done, but I estimate Vorbis is not only saving me between 30-50 megs per CD vs MP3 at Lame V0, but also slightly higher quality.

Which brings me to the Sansa Fuze:

After the disastrous experience with the Sansa View, I was not really sure I wanted to buy another Sansa product, I was pretty underwhelmed by that thing. The price was right on the Fuze and it claimed Ogg Vorbis support, so I bought it, and I’m not regretting it so far.

The Sansa View was giving me the following problems with firmware alone:

Sluggish menu browsing.

Rebuilding the database every time you add a few CDs worth of music to it took forever.

Sometimes eats the database for no reason.

Sometimes it doesn’t update the free space remaining when you delete files. (Requires a format to fix this)

Mass Storage Class transfer mode support was only added in a firmware update *after* it was released and anything you added in this mode, the player failed to organize into album playlists automatically, meaning you had to fall back to Microsoft’s Media Transfer Protocol, which is not exceptionally friendly to anything but Windows. (Linux has extremely spotty support for it and the Mac doesn’t know what it is at all).

Finally, it would play AAC audio files, but always in the wrong order due to an incomplete tag reader. The player had to have the AAC audio codec because it’s what most h.264 videos use, but they crippled playback of AAC audio files to get out of paying for playback royalties and patent licenses. Since MSC mode never works right, and you more or less have to use it in MTP mode anyway, that means Windows Media Player was the only way to properly sync the fucking thing, Windows Media Player doesn’t even know what AAC is and even if you have Windows 7 and WMP *does* know what AAC is, the device tells the player it only works with WMA or MP3 so Windows Media Player will “sync” it by transcoding all your AAC files into WMA. (AGGGHHHH!!!!)

No Ogg Vorbis support.

Personally, I think the Sansa View was such a shoddy device that they should apologize by giving anyone that actually bought one at list price a free Fuze.

In contrast to the View, the Sansa Fuze gives you:

Menus don’t lag.

Doesn’t eat the database, free space updates properly.

Proper MSC support which you should select as soon as you power the unit on. MTP is some nasty, nasty shit.

In MSC mode, you can drop your music folders onto the device’s MUSIC folder and next time you power it on, it’ll figure out the playlist by itself provided you tagged the files properly, it also supports folder hierarchies so you can organize your folders like /MUSIC/Metallica/Ride The Lightning, this drove the View firmware even nuttier but it works now.

Album art: Include the thumbnail for the album art in the appropriate folders as a jpg file and the Fuze will display it.

Proper support for all AAC profiles (including High Efficiency AAC if you must). Proper support for Ogg Vorbis. Proper support for FLAC in or out of an Ogg container (lossless compressed files, but these are massive).

Linux and Mac friendly (in MSC mode). Previous Sansa players have been pretty much dependent on Windows.

Smaller, lighter, and longer lasting battery.

The internal storage can be expanded with Micro SD or Micro SDHC (High Capacity) cards, they’ll show up as two removable drives should you plug a card in and mount it in MSC mode, just make a separate “MUSIC” VIDEO” and “PICTURES” folder on the card. The View supported Micro SD/HC cards and I retasked the 16 gig card from my View for use in the Fuze. (Formatted it first).

The FM tuner has about 50% better range.

All and all, this really is a lot of player for my $40. I recommend it to anyone, especially if you use Ogg Vorbis and want a reputable brand of player that supports Ogg Vorbis or FLAC (Not many name brands do and the ones that do tend to be far more expensive). So the Sansa View sucked, every company has a bad product once in a while, but the Fuze is everything the View should have been and more. Vorbis is catching on, more hardware makers are throwing it in every year, and we should reward the companies that are respecting our freedom to choose above the powers that be that are pushing to maintain the status quo. (It’s free for them to do so and it makes for more happy customers)

——-

Side note:

In the legalese of the included instruction booklet, there’s reference to what Microsoft’s Windows Media DRM can do to your player if you have DRM’d WMA or WMV (video) files files on it.

“Content providers are using the digital rights management technology for Windows Media contained in this device (“WM-DRM”) to protect the integrity of their content (“Secure Content”), so that intellectual property, including copyright, in such content is not misappropriated. This device uses WM-DRM software to play Secure Content (“WM-DRM Software”). If the security of the WM-DRM Software on this device has been compromised, owners of Secure Content (“Secure Content Owners”) may request that Microsoft revoke the WM-DRM Software’s rights to acquire new licenses to copy, display, and/or play Secure Content. Revocation does not alter the WM-DRM Software’s ability to play unprotected content. A list of revoked WM-DRM Software is sent to your device whenever you download a license for Secure Content from the Internet or from a PC. Microsoft, may, in conjunction with such license, also download revocation lists onto your device on behalf of Secure Content Owners.”

So if you are a fool who buys DRM’d Windows Media files, and someone cracks the Windows Media DRM software on your portable, Microsoft can take control of your property and force it to not be able to play the songs you bought. Likewise, they can also revoke your license to play all of your subscription WMA files on the device. You agree to let them do this and to trojan in the blacklists whenever you think you’re just downloading songs from Rhapsody or Napster or whatever.

This invasive bullshit is enough reason to avoid Windows Media. I only want the device to play Ogg Vorbis, I would have paid them extra to not include Microsoft stuff on my player. :P

Every player does this though unless you’re using custom firmware like Rockbox. You just can’t have a garden without a few of Microsoft’s stinging nettles popping up, can you?

How do you know when Paul Thurrott is lying?

His lips are moving.

Seriously, if Microsoft was the Republican Party, Paul Thurrott would be Fox News Channel.

Some people are constitutionally incapable of shutting the fuck up no matter how many times and how thoroughly they’re proven wrong, some people incessantly insist that they were taken out of context when they’re called out, and some people are just pathological liars that deliberately construct every sentence to be as misleading as possible. Paul Thurrott, in my opinion, is three for three.

On top of that, he has this odd mix of gung ho patsy and outright stupid that sometimes makes it difficult to tell when he’s lying and when he really has no idea what he’s saying.

This guy has bothered me for years with his total and utter disregard for journalistic quality, instead often opting to resort to hearsay, weasel words, the statistics that most favor his opinion, and anything that makes Apple and Linux look tiny or irrelevant.

Just a few of the lies and contradictions he’s been unable to escape (in order of how they amuse me):

(Windows Me in Thurrottland) “It is, quite possibly, the most under-hyped version of Windows ever created…It’s easy to ridicule Microsoft for milking the Windows 9x cash cow yet again. But the reality is that this release is exceptional.”

(2007 in Thurrottland) Tries bashing Firefox 2 and gets called out by Asa Dotzler

(2008 in reality) Gives up and writes a positive review of Firefox 3 but insists his abortive attack from the year before was “misconstrued”, goes on to list a few trivial gimmicky features IE 8 has that Firefox doesn’t. (Thurrott, you don’t want to go into feature comparisons between IE and Firefox, it can’t end well for IE)

(2004 in Thurrotland) Like zOMG!!!!1111 MSN Music is gonna be HUGE!

(2006 in reality) MSN Music is shutting down and your licensed files are toast if anything happens to Windows. (Such as upgrading XP to Vista or a slightly less devastating catastrophe like a hard disk crash.)

(2002 in Thurrottland) Like zOMG!!!!1111 Windows Media Audio is the second coming!

(2007 in reality) Finally forced to eat crow and admit that WMA is dead, and everybody uses MP3.

BONUS:

“AAC doesn’t play nice with products made by Microsoft and its partners. AAC isn’t compatible with Windows Media Player or Media Center” -Thurrott, October 2007

“Put simply, I am a fan of the Zune…The online marketplace is good, but not as good as iTunes Store, though that matters less with music because MP3/AAC is universally compatible.” -Thurrott, July 2009 (Note that Zune supported AAC in 2007 when he made the first post)

And just for an added face palm:

“..there are audiophiles and technology trolls out there who might recommend [lossless formats]…Don’t be confused by the term “lossless,” however: These formats are still compressed…This is a foolhardy idea, unless you will never use a portable media device or enjoy the thought of storing and managing two copies of your music collection, one in lossless and one in another format that’s been transcoded from the lossless masters.” – Thurrott, October 2007

I don’t suppose he bothered to mention that you can decompress lossless files back into WAV or onto another CD and the CRC checksums will even match the original disc! Or perhaps that both Windows Media Player and iTunes can transcode from your lossless library on the fly and put the resulting lossy files on your device? This isn’t new folks, they’ve both been able to do this since at least 2003-2004.

(2009) Admits the Zune is going nowhere but tries to play it off by comparing it to a Macintosh computer’s supposedly miniscule market share.

In a survey conducted last fall, IDC’s Kevorkian said only 4.8% of those with a portable media player reported having a Zune, while 61% had some sort of iPod.

So, in late 2008, the Zune actually had 50 percent more usage share in the MP3 player market than the Mac did in the worldwide PC market. (Hey, math can be fun.)

Of course it is, especially when it’s wrong/fake/conjured up with the rest of his delusions.

While we’re comparing apples and bowling balls, Mac rounded out the fourth quarter of 2008 with 8.87% of the desktop computer market (nearly double the market share of the Zune in the MP3 player market), and the Mac has gone on to 9.81% as of May of 2009. *source*

Mac and Linux have driven Windows down to an 87.75% market share, which is still a commanding lead, but in 2004 they had 96.34% *source*. While Windows isn’t dying off as fast as Microsoft’s other products (Read: Dropping like a brick), it *is* shrinking, and it should worry any investor when a company cannot at least break even year-over-year.

Mr. Thurrott, since IDG has Kevorkian on hand, can they possibly put the Zune under? It can be so quick and painless… No reason to punish the people that unwittingly put Microsoft in their 401(k) for another 2 years.

This of course brings me to today’s Paul Thurrott crap.

Thurrott slanders anything that competes with Microsoft, but seems to go out of his way to bash Apple. Now Opera is on his vendetta list since they won freedom of choice for European consumers to decide what browser they want with Windows 7.

The fact that he is bashing the browser with the most strict adherence to World Wide Web Consortium markup standards should not be overlooked, because Thurrott has a history of bashing industry standards like AAC while promoting Microsoft’s dead end (WMA).

His tirade is, essentially   “Well, uhhhm, Internet Explorer has 66% of the browser market, so it should call all the shots”. (It had 92% at the end of 2004 *source* )

Now, Mr. Thurrott has been gay for Internet Explorer for a long long time, and has stood faithfully by it despite several hundred security flaws, the fact that its rendering engine is prehistoric and buggy, and that you can’t extend it with anything but toolbars (oh do we know about IE toolbars…). (Although I have my doubts about whether or not even Thurrott could stomach any version of IE for longer than it takes to glaze over some Microsoft PR notes and grab a few screen shots)

To be dramatic, Paul Thurrott kind of reminds me of that episode of South Park “AhhhH!!! My baby is killing again! Don’t worry, mommy will protect you! I have such a good boy, such a nice boy…”, but there’s only so many bodies you can hide in the backyard and IE is a fuck up that people witness first hand from day one, so there’s really no point in even trying to defend it.

Apparently Thurrott has his panties in a twist this time because he can’t stand that the European Union, unlike the United States, actually has and enforces consumer protection laws. (And it will be interesting to see how the eradication of IE bundling in the Euro Zone affects the spread of spyware over there…)

Thurrott argues, plainly, that users should not be presented with a choice of what browser to use, that IE should remain welded onto Windows, inseparable and popping up even when you thought you hid the fucking thing, and that naive users should continue using what’s there and getting their system deluged with porn dialers, trojans, search page hijackings, and every kind of web annoyance and active content abuse that Adblock Plus for Firefox (or an ad blocking file loaded into Opera’s content blocker) can silence once and for all.

Paul Thurrott remains as pro-Microsoft and anti-user as ever. One could only assume that his yellow journalism is the kind that only Microsoft Monopoly Money could afford.

SoundConverter on Linux mutilates the MP3 files it creates

So I’ve been playing around with SoundConvertor on Ubuntu which is a front end for LAME, and a lame frontend for LAME as it turns out. :P

MP3 has some well known problems as you may be aware, which is why newer codecs like AAC and Vorbis and Windows Media Audio were created. One of these shortcomings is the so-called Scale Factor Band 21 problem. Put simply, it affects MP3 encoders in Variable Bitrate mode and causes a bit of file bloat when encoding audio above 16 Khz.

Various solutions have been proposed to this in the past, including eliminating everything in sfb21 completely by using what is called a “lowpass filter”, the encoder can simply throw everything away above 16 Khz and not even bother to encode it. This is the default for Real’s Helix MP3 encoder.

That solution figures that most people “probably” won’t hear those high frequencies on most sound systems, most of the time. The problem is that eliminating everything in this band can cause subtle distortions, so most encoders also have a mode that lets them selectively encode audio that is relevent to what is currently going on at that point in the song, if it won’t bloat the file too much. This helps with things like drum solos and such. Fruenhofer’s official encoder does this by default, Helix can do it with the -HF2 switch, and LAME can do this with the -Y switch.

The argument over what to throw away and what to keep is still subjective and going this route can still (though rarely) cause distortion in the resulting MP3 file. (Although I would say that LAME -Y handles these situations better than most).

The only way to ensure that the resulting MP3 file is of high quality is to encode at least *some* of sfb21. LAME’s psychoacoustic model (npsytune formerly known as gspycho) will strive to keep enough of sfb21 to keep the MP3 file from sounding distorted, and from V0 (Extreme quality) to V2 (Standard quality), Lame will adapt the lowpass to mitigate file bloat but at the same time, not carve too deeply into the frequency range you’re liable to hear. (Beyond V3 there’s really not much of sfb21 left and it has the -Y switch built-in) LAME’s developers have worked long and hard to fine tune these quality settings so that at V2 (Standard) you’d be hard pressed to tell the CD from the MP3 file.

Enter SoundConverter. I had SoundConverter suggested to me by an Ubuntu user when I asked if there was an easy way to batch transcode my lossless FLAC files to MP3, I’ve never much bothered with MP3 until I got an MP3 player (My old iPod accepted AAC files, this Sandisk Sansa does not), but I noticed that when SoundConverter on Linux converted them, they didn’t sound rich and full, they sounded flat and distorted. Nothing at all like when you use LAME directly or LAME frontends on Windows such as Winamp or Foobar2000.

A little investigating answered my question nicely:

This is the first 25 seconds of Seether’s “Driven Under” from Disclaimer II in a spectrogram as encoded by SoundConverter which claims to be using LAME 3.98.2’s “Extreme/Insane”  V0 quality (as seen in Foobar200’s spectrogram):

SoundConverter's "LAME 3.98.2 V0" Click for full resolution.

Lame 3.98.2 V0 Click for full resolution

What’s immediately obvious is that Gstreamer’s LAME plugin / SoundConverter are using a hard cutoff at 16 Khz whereas the real LAME V0 (again, EXTREME quality) faithfully represents the track up very close to 20 Khz.

What has Gstreamer/ SoundConverter saved us by mutilating our MP3 file? Well, its “V0″ file is 8.02 megs and LAME 3.98.2’s is 8.97 megs, SoundConverter’s hard cutoff file mutilation has saved us a whole smegging 0.95 MB, roughly 11% of the file size.

If you’re really worried about saving space, the real LAME and V2 would probably still sound better at around the same size, or less than, SoundConverter’s Extreme setting. Some pinheads that are developing Gstreamer and SoundConverter think they know better than the user (or the people that made the fucking encoder for that matter) what settings to use. Not only that, SoundConverter arbitrarily picks five VBR quality modes you can select from where LAME itself has TEN.

This “Users are idiots and confused by options” disease is clearly spreading even further than I thought.

It looks like you’re going to have to drop down to using shell scripts for batch encoding of MP3 unless you want the unholy union of Gstreamer and SoundConverter screwing up everything you encode into MP3. I wouldn’t trust anything that’s using Gstreamer’s LAME plug-in as a backend as it seems they all do the 16 Khz lowpass.

Edit: Seems like SoundKonverter (the KDE counterpart to SoundConverter) addresses LAME directly and isn’t as dumbed down as SoundConverter. Defining your own profile and choosing VBR quality 75 seems to correspond to LAME V2 (Standard quality) which is all I really wanted to begin with. If you don’t feel like dicking around with shell scripts, hauling in 200 megs of KDE to get SoundKonverter seems to be your other option.

I think this is the first time I’ve ever needed the KDE version of anything.