The Mystery of iCloud Photo Library Optimization

To say I was thrilled with the introduction of the iCloud Photo Library is an understatement.    I have been waiting for this type of single source library for years.  Of course I immediately turned it on for all  my devices and went all in (not completely in Photos for OS X, because my library was all referenced and it is a slow process moving them over).  I was also excited about the cool Optimize Storage option in iOS because in 2012, when I bought my iPhone 5, 16GB seemed like plenty of space.  

However, as I soon found out the Optimize Storage option would cause me more stress than I anticipated.  You see Apple decided that the optimization would be completely automatic with no opportunity for manual intervention for the user.  There is clearly some sort of algorithm that determines what full size versions of photos to keep on the device.  So far one of the rules revolves around photos taken within the last 30 days and there is also one based on photos that were downloaded locally to the device in the recent past.  It seems that no matter how low the available storage level goes the full size of these photos will remain on the device.  

I learned this because the other day I tried to take a video of my son playing baseball and got the dreaded “Video couldn’t be saved because of insufficient space” message. My heart sunk because I thought barring some insane offline photo taking spree this error would never appear on my devices again.  My expectation was that in the worst case scenario if space was running low (as in I didn’t have enough space to take one picture) that the “Optimize” setting would delete ANY photos that had already been uploaded to the cloud.  I figured maybe there were tiers:
if (availableStorage > 500MB) then don’t do any optimization
if (available Storage < 500MB && availableStorage > 250 MB) then get rid of any full size copies of photos from more than 30 days ago that have been uploaded to the cloud
if (availableStorage < 250MB) then delete any full size photos that have been uploaded to the cloud

No such luck though, it seems to insist on keeping some amount of full size photos on the device even if they exist in the cloud.  I was bit by this recently because about 2 weeks ago I went on a trip and recorded a bunch of video, which went up to the cloud nearly the same day.  However, now the 1GB of videos from last month sits on my device and I have no way of deleting it without also deleting the copy in the cloud too.

I tried all sorts of trickery like trying to delete the photos using the Image Capture app on my MacBook.  I turns out the delete functionality isn’t available if iCloud Photo Library is turned on for the device.  So I tried turning off the iCloud Photo Library on the phone and while the delete button appeared in Image Capture it didn’t really do anything.

In the end I found the best way to get me back some space was to turn off the iCloud Photo Library on my iPhone, which freed up some space because it removed all the iCloud Photo Library thumbnail photos, but all the full size photos were still on the device.  At this point I was able to use the Photos app on my iPhone to remove photos and videos from my device, but not from the iCloud Photo Library. I also had to delete them from the “Recently Deleted” album too. Once I got rid of some of the videos that had been taking up so much space I turned the iCloud Photo Library back on and it started downloading the thumbnails again, but I now have a 1.5 more GB of free space on my iPhone. 

Photos for OS X Learning Curve

It has been a while since I have posted, but I figured it would be helpful to document my experience with the new Photos for OS X.

My Library Situation
Prior to the upgrade I had a few different Aperture libraries (stored in a sub directory of the Pictures directory). My primary library has a combination of internal library and externally referenced photos, some of the external are on an external drive.  A second library has all externally referenced photos, while a third library has all its photos stored in the library.  The primary library is the one that has had the photo stream turned on and syncs all of my photos from my iOS devices.

Since November I have had the iCloud Photo Library turned on for my iPad and iPhone, which has worked pretty well.  According the my iCloud.com Photos page there are 1389 photos and videos stored in my iCloud Photo Library at the current moment.

Photos for OS X Upgrade
I was very excited to come home and update my Mac to OS X 10.10.3 and to try out the new Photos app.  The upgrade took about a half an hour to download and install, but I was up and running.  I had heard a lot of talk and read a lot of reviews on the application, but looking back a lot of these seemed to focus on the standard single library that you got with your Mac use case.

I opened the Photos app and was expecting to be prompted to choose a library for importing, but instead it just asked if I wanted to create an iCloud Photo Library.  I click next and before I knew it the 1389 photos and videos stored in my iCloud Photo Library magically appeared before me. Nice!!!

But wait, what about my existing libraries?  I quickly searched online and started doing research, but I also did a little playing around with a test library containing both internal and externally referenced images.  Here are some questions I was wondering about and here are the answers.

Why wasn’t I prompted to select a library for migration? 
Since my libraries were not in the Pictures directory, but instead in a subdirectory of the Pictures directory, it just went ahead and created a new library.  Apparently if you have just one library (iPhoto or Aperture) in the folder it will automatically try to use it.

What is the deal with the ‘System Photo Library’?
You can have several Photos for OS X libraries on your system, just as you could have many different libraries for iPhoto and Aperture.  However, only one of these can be the ‘System Photo Library’.  The big difference between the ‘System Photo Library’ and the other libraries is that this is the only one that will sync up with your iCloud Photo Library.  Opening a non ‘System Photo Library’ will show you the photos in that library, but it won’t show the photos you have in the cloud and it also won’t sync your photos to the iCloud photo library.  However, you can change a library to be the ‘System Photo Library’ after the fact so don’t fret if it isn’t the one you wanted (note that you want to switch this as infrequently as possible because it does trigger a download of at preview size copies of all the photos in your iCloud Photo library.

How do I convert Aperture and iPhoto libraries over to Photos and what does that mean exactly?
If you right click on an Aperture library (MyLibrary.apLibrary) in the Finder and choose Photos from the Open With menu it will ask you if you want to start the conversion process.  Once it is complete you should see all of your photos in the Photos app.  Back in the Finder you will see that there is now a MyLibrary.photoslibrary and MyLibrary.migratedaplibrary where the original library used to be.  Going forward these are now two separate entities that can be used with their respective apps, but any changes made in one won’t appear in the other.  The first time you open a .migratedaplibrary you will be warned that any changes made to the photos will not be made in the Photos app, but if you click OK the extension will go back to .aplibrary.

What happens to referenced files when migrating to Photos for OS X?
Any files that were referenced in the Aperture library will continue to be referenced in Photos.  However, the tools for managing referenced files are very limited in Photos and essentially let you either see the referenced file in the Finder or Consolidate referenced files aka bring them into the library.  The other caveat with referenced files is that they will not be uploaded to your iCloud Photo Library.  The good news is that as soon as you “Consolidate” photos (which can be done on one or many photos at a time) they will be uploaded to the iCloud Photo Library, assuming that the library you are working on is the ‘System Photo Library’.

Can I delete the old Aperture Library after migrating?
First of all any files that were stored in the Aperture library will be stored in the new Photos library. I have read in various places that the masters in one is a link to the other so in reality there is only one copy, but I am not sure what is going on there.  In my testing I took an Aperture library with photos stored in the library, migrated it to Photos and then deleted the Aperture library without event.

My Plan
After some research and testing I think I have a plan that should work for me.  I am going to convert my primary library from Aperture to Photos.  Initially most of the library will not end up in my iCloud Photo Library because most of the files are referenced files.  This will create a single library for all of my photos going forward, with any new photos from my iOS devices or imported into my Mac automatically going into the iCloud Photo Library.  Over time I will slowly run the “Consolidate photos” operation on my referenced photos so they will be added to the library and consequently uploaded to my iCloud Photo Library.  Since I have the option turned on to Optimize Storage, space won’t become an issue and eventually all of my photos will be in the cloud.  As for my other libraries, I will probably just convert the smaller ones to Photos and the larger one leave as an Aperture Library right now because it has a ton of referenced images, which I might want the flexibility to easily move in the future which is much easier given the tools in Aperture.