Sunday, December 30, 2012

Music in the Cloud: Scanning and Matching

I've had the chance now to work with the three big scanning and matching service, iTunes Match, Amazon Cloud Player, and Google Play.  I wanted to share my experience with these services and show the strength and weaknesses of all three.

First I'll go over how each service works, and then compare how each of the services compare to each other in each segment of the process.

1. iTunes Match.  
The first cloud based service to offer the scan and match feature, it is also the most mature.  Rooted in Apple's software, iTunes, its focus is your iTunes library.  The service is $24.99 a year and as long as your library is smaller than 25,000 songs, starting is as simple as clicking the Match button at the top of main window when in your music library.  This simplicity is both it's strength and it's weakness.  If the song is in your library, then it will be added to your cloud music library, but if you have some songs outside of iTunes (such as your secret passion of Aqua), there isn't any way of getting the file into your cloud library.
The software will gather information about your library, send it back to Apple's servers, retrieve the parsed data, and go through each song and match each song to one in your country's iTunes Store.  Your purchase history through the store doesn't count against your 25K limit and will appear automatically in your cloud library.  Once all of the music has been compared, any non-matching entries will be uploaded to your account.  All metadata is synced the way you have it, so it will not correct any incorrect tags, but keep your data as organized as you keep it yourself.  Custom genres, lyrics, BPM, comments, grouping, and album artwork you've manually tagged are maintained on the copy in the cloud. If you haven't tagged artwork, and iTunes finds a match in it's own collection, it will automatically add it to the track.
Once the initial scan & match has completed, you are ready to go.  New songs added to iTunes will be scanned after a few moments of being added to iTunes.  Up to 10 devices can access your cloud library, any mix of iOS devices running iOS 5+ or iTunes 10.5.1+.  On another iTunes computer, to access your cloud music, go to Store, then Turn on iTunes Match.  Sign in and it will scan any music in that library as well.  What is currently already in the cloud will appear in your main music library as long as Show Music in the Cloud under view is active.
All matched tracks are guaranteed 256 kbps in AAC format.  I say that, but in reality it seems that the files are encoded more in the style of 256 ABR.  Running the tracks through VLC to watch the stream bitrate reveals that some songs if needed will have additional bits.  I've seen  it spike up to 280 at times.  Another piece of evidence is that a song converted to CBR 256 is generally a tad smaller, sometimes by a few megabytes.  
To replace a lower bitrate song, simple delete the track out of iTunes (making sure to NOT delete out of iCloud & to move file to the trash can), right click and select download.  It will download into your iTunes library and you now have the equivalent track that you would have bought from the iTunes Store.

2. Amazon Cloud Player
Amazon was the second music cloud hosting service to introduce the scan and match feature. 
Before it was a truly unlimited number of songs (with a limit of 100 mb per song) now has a limit, but is absurdly high at 250,000 songs.  I have a very large library, but even I won't be able to hit that limit.  I could triple my collection and still be able to fit it all in the cloud.  At this point it's obvious that any of the limits imposed by cloud player services are not physical.  Allowing such a enormous amount of music (that doesn't have to be replicated locally) illustrates the material worries such as storage or bandwidth are such of little cost that they themselves aren't imposing limits on these service, but it's the agreements with record companies.  I'm sure up front money is also a concern, along with royalty rates, but the fact that $25 a year can get my virtually unlimited cloud storage for music is a bit mind blowing.  There is also a free version, but it's limited to 250 songs. Just like iTunes Match, songs you have purchased through the Amazon MP3 Store will automatically be included in your cloud player and won't count against your sound count limit.
The scan and match feature was advertised as a major upgrade to the cloud player, but it dramatically changed how I was forced into using the service. Previously I was able to just select folders and drag them to the amazon cloud drive app in my menu bar extras area.  This would upload the songs as is to your Amazon Cloud Drive.  Any file that was a supported music format would automatically put in your Cloud Player and wouldn't count against your storage limit.
Now it's a radically different ball game.  Their system is centralized on the Cloud Player website.  The whole service is very much a web service.  The apps you use to upload and download can only be used in conjunction w/ the the website.
Getting music into your cloud start from the Cloud Player website.  Click the button "Import your Music."
Let me take this moment to explain one big caveat.  Google Chrome with its built in flash play is problematic.  It WILL NOT import music, and while you can get it to download music, it will count as a second desktop even if you have already authorized the machine with a different browser.  Playing your library from Chrome work just fine, but I would recommend using a different browser to upload or download music from your cloud library.
Once your desktop has been authorized, a helper app will be downloaded and needs to be installed, the "Amazon Music Importer."  (Amazon is located in Washington State just like Microsoft and I think they have the same naming committee.  Very boring and descriptive naming of all their products.)  Once it installed, the app will launch and you will need to authorize your desktop.  You are allowed 10 devices, but it's not as straight cut was it sounds.  You only have to authorize a desktop by either uploading or downloading music.  If all you want to do is listen to your music, I doesn't seem to count against your device limit.
After authorization, you are given two buttons, Start Scan or browse manually.  
To be honest, I've never been able to get the Start Scan button to work right, but it could be due to the complex nature of my iTunes setup.  I don't store my library in the default location and it's freaking gigantic (~80K songs).  Maybe with a more normal setup would work with their software, but I don't have any experience with it.  Browse manually works exactly like expected.  Select the folder you want it to scan and let it scan through that folder and all subfolders.  Once it has scanned and processed the files it found, the software will pause and await your input.  It will ask if you want to import all songs, or select ones it found eligible to add.  Then the software will run through all the tracks it found and try to match them.  After all songs has been run through, the matched ones are already in your cloud library, while the unmatched ones are uploaded to it.  
Downloading tracks is relatively simple process.  Anywhere in your Cloud Player next to a song on the left is a check box.  You can select up to 500 tracks at once.  Select the tracks you want to download and then click the download button at the top right.  
If this is your first time, another helper app will need to be installed first.  This time it's the "Amazon MP3 Downloader."  Install the app (which has an extraordinarily old & outdated splash screen), and unfortunately the browser you initiated the download with will need to be restarted.
Once the downloader software is installed, the browser will download a small directory file w/ an extension of .amz.  Depending on the browser you use you can have the file automatically open w/ the associated program or just save it.  Either way, double clicking the file will open up the Amazon MP3 Downloader.  Here you can specify where it downloads to, such as iTunes library or a folder you choose.  Then it will download each track you selected one at a time.  You can specify whether you want the newly downloaded songs to be automatically put in iTunes or not.
There isn't an easy way to figure out what tracks are matched and what ones are uploaded.  You will have to look at the metadata of each track to see whether it's an Amazon copy or not.  In the comments of a matched track there will be a song ID number.  The quality of a matched track is advertised as 256 kbps, but there is more variance than is stated.  Many tracks (I'd say ratio is about 50/50) come in as VBR tracks set at 80%.  iTunes will report the bitrate as a VBR track and list the average bitrate of the entire track.

3. Google Play
The newest kid on the block (at least here in the USA) is Google Play.  Recently introduced here in America, but had been previously available in Europe for several months, Google Play scan and match service is also the least vocal about this feature.  
Google's offering allows up to 20,000 tracks, but it's completely free.  There isn't an option to pay more to get more.  Like the other two, songs bought through this service appear in your cloud library, BUT, despite my research on their web site, I couldn't find a good answer either way whether these tracks count against your song count limit.  From the way it's talked about, it seems they do, but I'm not 100% sure of that.
Once your Google Play account is setup, adding songs to it is done via the Music Manager app.  While the app does install to the Application folder on a Mac, it really all lives in the system preferences.  Sorry windows users, I don't work on a windows machine so I can't inform you of how the software works on a PC. 
Opening Music Manager for the first time greats you with a choice, download your music or upload.  It's a bit misleading, but it's not a one time choice, but a choice you have to make first run.  Going to the app later on you can choose to do either.  
To start the process, you are given a chance to choose iTunes / Windows Media Library, Music folder, or another folder of your choosing.  (Disclaimer, I have only used the choose folder option as I was just testing out the scan & match feature.)  You can't have both an iTunes library and a folder selected as a source at the same time.  Songs that are matched with Google's library are indicated as being "uploaded" but it really is just syncing the artwork you had embedded.  This is what I meant by being less vocal about the scan and match feature.  Everywhere Google Play keeps talking about uploading, and you have to really dig around to figure out about the matching part.
Like Amazon, finding out which track is matched is a manual process of downloading the file or watching the upload process.   There are two options to download your music, either via the web site or with the music manager.  The web site limits you to downloading a track twice, ever.  The music manager allows you to either download your entire uploaded/match library or your entire purchased library, but either way it's an all or nothing deal.
Google claims that songs matched are matched up to 320 kbps, and that is technically true.  Don't expect to start getting droves and droves of songs back in maximum quality, as I saw a wild variation in quality of files in my testing of the service.  I will go into more detail of that later.

Now that I've described each service's workings, both uploading & downloading, let's compare them in different aspects of the process.

All three sift through the music you designate, but iTunes Match by far is the fastest.  Apple's software maturity in the segment really shows.  On my home Mac, it took Google a solid 5 minutes to prepare one test folder of 9 songs before it uploaded.  Amazon took several hours (I think it was around 3) to sort through my music library folder just to start figuring out what I had in the cloud already and what needed to be uploaded.  When I matched about 14,000 songs on iTunes, it took about 20 minutes.  And that was when the service first started, now it's been improved several times.  To scan a folder of just a few songs with amazon, it still take several minutes before it's ready to attempt to match the songs.

iTunes has an entire column dedicated to telling you what the status of each track.  Matched, uploaded, duplicate, or error are the various categories each track can be listed as, giving you detailed information about your catalog.  This kind of information is missing in the other two, and it's partly due to the fact they are both web services with helper apps.  Now it's feasible that some kind of feedback could be provided on the website, but neither does so.  The only reliable way to know what the results are the matching process with Google or Amazon are to redownload the tracks to your desktop.  
One issue with Amazon's matching is that it doesn't always accurately represent all metadata of tracks.  For example, uploading a box set of Elvis songs, Amazon matched part of the album, but not all.  When viewing the album on their website, the songs had different artwork and did not all have the correct disc or date.  Amazon replaced the matched tracks ID3 data in these fields with the data from the album it matched the track to.  Since this was compilation of songs released from different albums, dates were all over the place as the album those songs originally appeared on were released in different years.  This mismatching data make the songs appear all jumbled when viewing the album on the website, and when the tracks play their matched artwork is displayed as opposed to the embedded art. 
Amazon's matching is solid but does not get as many tracks as iTunes Match.  It will almost always find at least one track in an album it can ID, but rarely gets a whole album.  The percentage goes up the bigger name the artist is and how popular they are right now.
Google's flaw is that it's very hit and miss with its matching.  With the albums I tested more songs were not matched than were.  Very big name artists such as Metallica and Anthrax with both older and newer material were not able to be matched.  Dethklok's Dethalbum III along with Crowbar's Sever the Wicked Hand, both very recent albums, were plainly uploaded. I had the most success w/ non-metal tracks, with most of the Elvis songs being picked up, Aerosmith's Toys in the Attic, and the new Mumford and Son's Babel.
iTunes is by no means perfect.  It still misses single songs out of entire albums for no reason.  When playing tracks on an iOS device, at times there isn't any album artwork.  Occasionally lyrics don't sync across devices.  But these hiccups I feel are minor to how solid the service is overall.  
Apple has had the longest to work on it's algorithms and has the largest user base to vet it's software.  I'm glad to see competition and hope to see the other players strengthen their products in the future.

Google here shows off its web-savyness.  The service was able to upload tracks faster than any of the other two.  It needs to, with the weak sauce that is its matching feature.  It would saturate my upload as much as it could and would work on uploading two tracks at a time, so there wasn't any downtime in switching between songs being uploaded.  
iTunes is a close second.  It still is daily quick with uploading music files, but works one at a time.  Sometimes it will say it's uploading a track, but is really just uploading the custom artwork.
Amazon's Music Importer is a sluggish brute.  It also works one at a time, but doesn't even come close to uploading as fast as your connection will allow.  It also seems to want to take a back seat to other programs trying to upload at the same time.  Amazon could stand to focus on improving its upload speeds. 
Each browser has it's own quirks with the Importer.  Chrome won't upload songs, Firefox asks several times to grant permission to the Importer.  Opera loads the main page blank and you need to click on a different section and then click back to see the songs section.  So far no issues with Safari and I have no idea how it works on Internet Explorer.

Amazon, since it doesn't create a mobile OS itself, tries to be the most platform agnostic.  The cloud player will work in just about any modern browser.  Amazon's website is decent for playback, but there is lots of loading when scrolling through lists.  They also have both an iPhone app and an Android app.  Their web player even works fully on an iPad's Safari as well.  
One issue I'm having personally is that the iPhone app is terrible. It won't run on any of my iOS devices, my two iPhone 4 or my 1st gen iPad.  The app crashes when trying to start.  I believe this is due to a combination of the size of my collection, an added feature of displaying more unicode characters and the general unstableness of the program.  Several calls have been made to their support line, w/ an ongoing support ticket with them, but no fix works yet.  So take that as a warning if you plan on using their iOS app, it can fail completely on you.
iTunes Match can play back on iOS devices, iTunes on both Mac & PC, and Apple TVs of 2nd or 3rd gen.  On iOS 6, songs will stream when clicked to be played, while on iOS 5 will start to download to the device before playback starts.  On iTunes, playback will stream the song, or a user can manually download it.  The Apple TV only allows streaming.
Google stresses it's Android app and also provides a web player for any desktop OS.  iOS users supposedly have some third party offerings, but I haven't tried any of these out.  The Google Play app is compatible with Android 2.2 or newer, so the majority of devices will be able to use the feature.

Downloading Songs:
iTunes makes it very easy to download any of your tracks to your mobile device or desktop.  Right clicking in iTunes gives the option to download any track to the computer, and if the option is checked (it is be default) it will download 3 at a time, saturating all available.  iOS allows to get whole albums or artists, but in 6 the single song has been removed for some bizarre reason.  You can control whether the mobile device uses cellular data or not to download the track.  
Amazon is a little more difficult.  You can manually select tracks, or select a whole Artist or Album using a check box atop the list you are looking at.  Playlists are more difficult, you can only select songs individually, which you can only select up to 500 songs at once.  As stated earlier, a file w/ a list of songs to download is the downloaded by the browser itself, while the helper app actually does the downloading.  This app, just like the Importer, is one song at a time and SLOW.  I have ample bandwidth available to download with, but it seems to plod along at around 4-5 Mbps.  Both my work and home connection are much faster than that. 
Downloading on the mobile app is little easier, just click download on the song, artist, or album you want and it seemed to download just as quick as iOS did w/ iTunes Match.
Google isn't much better at download speed than Amazon, and severely restricts how you can access your own music offline.  The 2 song limit on the website is ridiculous, and the all or nothing of your entire library is not any better with the Music Manager, especially if you really start using the service.  Again, I have no experience with the mobile app so I can't provide any usage details on it.

Song Quality:
The most consistent and reliable is again Apple's iTunes Match.  The songs are always at least if not more than 256 kbps.  Amazon's files are almost as good, and still very reliably good quality.  Google, I have a bone to pick with you here.
With my tests, I got all manner of quality of files, and even within an album there were different bit rates.  The worst quality files were of older albums, and they came back as 160 kbps but w/ a sample rate of only 22.05 kHz, and the format was listed as Mpeg 2 Layer III as opposed the traditional Mpeg 1 Layer III.  That's half the rate of CDs and a subpar bitrate!  On other albums I would get a mix of 256 and 320.  I don't know why Google's offerings are so variable, but it left me disappointed.  I don't want to risk having a high quality song I've personally ripped to be downgraded by my cloud service.

Each service has its high points and pitfalls.  Even though I have my personal favorite, I know each person has their own preferences and circumstances.  

iTunes Match:
Saving Grace: Matching strength and song quality.  You will have more songs matched and get your music in the cloud faster with iTunes.  You will also have the most consistently high quality songs to choose from when listening to matched tracks.
Weak Sauce: Song limit.  25,000 songs is too low for paying $25 a year.  I have over 3 times that in my collection and I have to create a special separate iTunes library to get the service to work.

Amazon Cloud Player:
Saving Grace: Song Limit.  Having such a large limit allows people such as I to have access to all songs anywhere.  I don't have to choose & I like that.
Weak Sauce: Import & Export of Songs.  Way too slow and kind of a headache.  Other service make this much easier.  Look to Google on how to improve your web service.

Google Play:
Saving Grace: Price.  20,000 songs for free is a great enticement.  For people on a budget or just can't justify paying money to host their own music, Google offers a great service.
Weak Sauce: Matching/Song Quality. Way too many songs aren't matched, and the unreliable nature of the songs that are matched makes it wary to any one concerned with the quality of their music files.  

I hope that this epic post has helped inform you of the music cloud services out there.


  1. This is a very thorough evaluation of these three services. Thanks for taking the time to review these.

  2. Very good stuff! It's hard to keep these things straight so I admire that you do.