Restore hidden files from Time Machine

About backups

A working backup is like a stock of 10 billion masks waiting for a pandemic. You only use it when things go bad and since you never know when things go bad your backup needs to work flawlessly all the time.

Time Machine makes it extremely easy to restore anything that's on the backup disk to your machine. If you have not yet installed an external drive for your Time Machine backups, do it NOW. There is nothing more important than having a reliable backup of all your important files.

Time Machine has an optional setting (System Preferences > Time Machine > [Options...] button) where you decide what not to backup. Anything that is not there is basically copied to the Time Machine disk.

Hidden files are also hidden in Time Machine

This is not a Time Machine tutorial, so I'll stop here. But, what matters is that when you click on the "Enter Time Machine" menu item, macos shows a nice time based display of the contents of the current window, but it only shows what is visible. Not hidden files that are used in Unix applications as preference files and that are routinely created at the root of your home folder.

Yesterday I messed with one such file and since it was hidden, I could not restore it from the Time Machine display.

The easy way, for moderately experienced people

Time Machine can be accessed from the command line with the tmutil command.

$ man tmutil

TMUTIL(8)                 BSD System Manager's Manual                TMUTIL(8)

NAME
     tmutil -- Time Machine utility

SYNOPSIS
     tmutil verb [options]

DESCRIPTION
     tmutil provides methods of controlling and interacting with Time Machine,
     as well as examining and manipulating Time Machine backups.
     Common abilities include restoring data from backups, editing exclusions,
     and comparing backups.

The "verb" we're looking for is restore and its syntax is:

restore [-v] src ... dst
        Restore the item src, which is inside a snapshot, to the location dst.
        The dst argument mimics the destination path semantics of the cp tool.
        You may provide multiple source paths to restore. The last path
        argument must be a destination.

The src is where the restore date/time can be chosen since the backups have their own folders based on date/time.

So what I needed was:

$ tmutil restore /Time/Machine/path/to/.hidden.file ~/.hidden.file.test

I restored to a .test file just to make sure that really was the file I wanted.

The longer road, a half command-line, half GUI approach

Another solution is to make hidden files visible in Finder, start Time Machine, restore from the graphical interface and re-hide the files (they're hidden to avoid messing with them by accident).

There is no visible Finder setting that allows for that, so to get the job done one still has to use the command line to modify some non-GUI Finder setting from Terminal. This has been documented for years but I keep forgetting about it.

In Terminal, enter the following command:

$ defaults write com.apple.finder AppleShowAllFiles TRUE

What does all that mean ?

In Terminal, enter the following comment:

$ man defaults

To display the "defaults" command "man"ual. You should see:

DEFAULTS(1)               BSD General Commands Manual              DEFAULTS(1)

NAME
    defaults -- access the Mac OS X user defaults system

SYNOPSIS
    defaults [-currentHost | -host hostname] read [domain [key]]
    defaults [-currentHost | -host hostname] read-type domain key
    defaults [-currentHost | -host hostname] write domain { 'plist' | key 'value' }
    defaults [-currentHost | -host hostname] rename domain old_key new_key
    defaults [-currentHost | -host hostname] delete [domain [key]]
    defaults [-currentHost | -host hostname] { domains | find word | help }

(etc.)

Do you see the "defaults [-currentHost | -host hostname] write domain { 'plist' | key 'value' }" line ?

What that means is that the command syntax is:

defaults + optionally either "-currentHost" or "-host hostname" + write + the "domain" to which belongs the preference + a compulsory settings value.

In our case, the "domain" is "com.apple.finder", the "key" is "AppleShowAllFiles" and the "value" is "TRUE".

which means "Please, write down somewhere that Finder is now required to show all the files."

Next step

When you're done with your un-hiding hidden files, you can enter Time Machine, restore the file you want and then re-hide the hidden files with:

$ defaults write com.apple.finder AppleShowAllFiles FALSE

Et voilà !

Now, Finder doesn't know about that change of settings until you actually relaunch it. You can do that from Finder itself, or, since you're already in Terminal, you can do it from there by using the following command:

$ killall Finder

where "man killall" gives you:

KILLALL(1)                BSD General Commands Manual               KILLALL(1)

NAME
    killall -- kill processes by name
...

$ killall Finder

will actually kill and restart Finder.

To do the un-hidding and killalling in one fell swoop just tell Terminal that the two commands should be followed in order:

$ defaults write com.apple.finder AppleShowAllFiles TRUE; killall Finder

And the same for hidding them again when you're done with Time Machine:

$ defaults write com.apple.finder AppleShowAllFiles FALSE; killall Finder

Knowing about the command line is a must

Knowing about the command line and Terminal in a must. You can't pretend that system does not exist and trying to make sense of all that will drastically increase the amount of stuff you can do on your machine. There are plenty of tutorials on the web. One day I'll eventually write one for "us" translators, with commands that are relevant to our work...

Adventures in Kernel Panic Land

Jimmy McGill chose "Saul Goodman" as his pseudo for a new venture to sell his now useless TV commercial slots, because "it's all good, man!"
That's exactly what your Mac tells you every day. Everything works fine, and everything will work out fine tomorrow, until it doesn't, but then, it's a Mac, so you're all good.

You never know that you need a backup until you do. That's the rule. I learned it about 10 years ago, the hard way. Massive hard disk failure, no backup at all, no extra software to check/fix/save the disk. Nothing, but a big deadline 2 days away. At the time I was running an Itanium Powerbook. I had bought it second hand a while before and it was a real workhorse. I also had a greenish clamshell iBook that my wife used to work with, before her company lent her a Windows machine.

When the machine's hard disk died on me, I called the project manager. He roughly told me that I still had 2 days and that the schedule would not move an inch. I don't remember what I did to fix the mess, but I did manage to scrap most of the data from the disk, then I must have moved all that to the iBook, resumed work, delivered, celebrated, and decided to create regular backups.

In the 21st century, external drives are cheap. They are cheaper than whatever hourly rate you apply to the time spent on fixing a non-backed-up dead system. They come in all sorts of flavour, shape, colour and capacity. And if your main machine is a laptop and it's a pain to physically connect it to your backup disk every time you put it back on your desk, you even have Wifi backups.

We tried half-backed solutions for a while. I bought terabytes of external drives, ugly as crap, all Firewiry and USB-Cy, but when you want a transparent solution that just works, all this just doesn't scale. Full disclosure: I have no financial interests in Apple. But damn, Time Capsule rocks. So yes, we got ourselves a Time Capsule, just 2 TB. We had 2 iMacs and 2 notebooks at the time, we thought we'd be short with the capacity but the 3TB Capsule seemed too expensive for just one extra terabyte. But even with the "low" capacity, it's been with us since. It does everything over Wifi, no need to connect the machines, just turn them on and Time Machine does its thing, and in no time you have your incremental backup ready for when you need it. You do need to turn the Time Machine on and select Time Capsule as your backup target, but it's a one time setting, and then you can forget about it, pretty much.

When things work fine, the only time you need a thorough backup is when you upgrade your system. I don't know how many more OS upgrades this "early 2011 Macbook Pro" will be allowed (our 24" iMac could not go further than Yosemite) but I was not going to miss that High Sierra upgrade after seeing the APFS talks on internet. The rest of the icing, I was not very interested in, although I love the way Safari does a lot to keep advertisers from wasting our CPU (and brain) cycles.

What had been bothering me for a while was that the machine was getting slower and slower everywhere and after having an issue with how slow BBEdit was, the Barebones support team confirmed that it was most probably a hard disk issue. With APFS coming I was not going to replace my disk with a rotating thing again. I was done with that. I was going to go the SSD way. All flashy, all speedy, all slick electronics.

My son had done it a few months back. We bought him an MBP when he entered university, one of those new but still coming with the antiquated DVD player. In his department, he was the only freshman to use a Mac. When he came back for Christmas last year, I paid for the replacement Samsung SSD and he spent a few hours doing the thing: backup your data and follow the instructions on Youtube. Easy enough. Then he reboot the machine and everything was faster. And it was before High Sierra and APFS.

I was going to go incremental on this. First install High Sierra, then get an SSD, and last reinstall everything. But before all, the backup. Time Machine being automatic, you really don't have to do much usually. But when you want to do it "now", well, there is a menu item for that. It's called "Back Up Now". To make sure you don't forget to save a few last-minute settings, just quit all the applications you run, leaving only Finder on (although if you know how to quit it, you can too), then let Time machine prepare the backup, do the backup and clean up the backup.

Then, you can start the upgrade process itself. Go to apple.com, click on "Upgrade" and everything goes super smoothly. At one point you are presented with a dialog that asks whether you really agree with all the terms, you just click "Agree" unless you're a hard-core free software advocate, but in this case you're not running a Mac and you're not reading this page, and then files get downloaded and installed and you can go watch your favourite show in the meanwhile, or read a book.

That's what I had planned anyway. Either finish season 3 of "Better call Saul" or read "Consider Phlebas", but it did not happen that way. What happened is that after a long while and not much happening, the Mac complained that it could not be upgraded. Just like that, two thirds into the install process, when the machine had overwritten all those important parts of the disk and you know that if you don't find a solution, you're done.

Rebooting won't do it. The old OS has been blown to pieces by the newer files partially written over it and the machine is programmatically speaking a giant spaghetti mess that the CPU has no idea how to deal with. When that happens, you have a Kernel Panic, and when the kernel panics it is of the utmost importance that you don't panic. Whatever you imagine has happened. All the pics of the newborn, all the thousands of whatever currency you spent on music and movies. Everything is gone. Except that, it's on your backup, and the likeliness that your computer and your backup die on you at the exact same time is so low that it's even lower than the likeliness of you being under the atomic fire of a North Korean missile. Although, granted, the day this happens, you'll have things more important to worry about than a broken backup.

So, you see the Kernel Panic thing that actually looks like a personal reboot of The Matrix on your screen and you feel all warm inside because you know it's a Mac. The idea here is that you need a handy way to get information from the Net. I don't have a phone, but I have that extra 24" iMac that's now mostly used as a Minecraft machine by the 2 little ones, but it also runs Safari (mostly used by the same little ones to check new "mods" for said Minecraft) and so I have a powerful entry point into the internets of information.

What you need to know here (and Safari is your friend if you don't) is that your Mac comes with secret programs that you can launch when you need them because they are located somewhere that doesn't get to be erased by a gigantic mess up of overwritten OS files. One such program is "Recovery Mode". You can launch it by hitting Command+R right after you restart your machine. If your system is down, like mine was, and you don't have a boot disk or USB key, then you want to enter Internet Recovery Mode, which requires an internet connection to download to your Mac all the things necessary to fix your issues (you may want to physically connect to your router/Time Capsule with anything that is faster than Wifi).

"Recovery Mode" be it local or internet based can do four things. The first one is get a Time Machine backup and restore the full machine from there. The second one is reinstall the OS from scratch. The third one is launch Safari to give you access to Apple Support for help. And the fourth one is give you access to Disk Utility so that you can diagnose/fix disk problems.

I figured that since I did not have a separate boot system my best bet was a full recovery from Time Machine. But I tried Disk Utility before that, checked the disk and since it told me everything was normal I just went on and decided to test an OS install. Depending on the key you call Recovery with, the version of the OS that's proposed to you is different. Either you have the OS your machine came installed with when you bought it, or you get the latest version that was installed, or you get the latest compatible version.

I tried High Sierra. And it did not work. I got a number of "cannot access the recovery server" or similar messages and gave up. I rebooted, again in Recovery Mode, tried again with the originally bundled version (Lion ?) and install started. And never stopped. It was late in the night when a dialog announced that the install would really not work.

This kind of moments are when you desperately need to cling to your faith in Macs (slightly backed by actually very few horror stories of messed up machines that refused to recover), and try one last thing before you go to bed. Reboot/Recovery Mode/Reinstall, it was staring to become a routine. But for some reason, this time it worked, and in the morning I had a (not so High) Sierra machine up and running, or rather ready to accept my backup data. A few hours and 250 GB later, I was running my machine, with a fresh Sierra install and all my data in place. I was basically at the exact same point I was the day before, but stronger in my faith that Macs just work and when they don't they still do.

The SSD arrived a few days later. That kind of manipulation involves a slightly higher level of wizardry. You need a few tools and an iFix video. Tools mean screws, and you really don't want to lose the screws. So here we go. Old hard disk removed, screws put in separate boxes to not mix them all, new SSD thing installed, screws back in place, power supply back on, reboot and Internet Recovery Mode launch.

Normally, an all brand new disk is not formatted to accept data copying. It is just a blank slate that the OS must put in shape for it to be of any use. But the OS still recognizes that there is a disk willing to be formatted attached to the machine. Normally.

In my case, the SSD thing was nowhere to be seen. The law of probability was plain about that: the thing itself is not at issue. Maybe an internal connection ? Would I have to open the Mac again and check everything ? I'm lazy and I prefer the Aspirin of Computer Fixing: reboot. A reboot/recovery mode later, I had the thing under my eyes. Go figure. Disk Utility was offering me to format the disk to APFS, which would also erase the data I had spent one full day reinstalling. Go for it, I thought, I'd just spend one more day going through the recovery hoops to get my machine running with the new OS.

Here again, things did not happen as planned. It took a small number of attempts to get High Sierra to be installed on the disk, and then I had to reinstall my data again from the Time Capsule backup.

I was lucky. We had a 3 day weekend here and I had decided to remove the curtains in both our room and the kids' and replace everything with wood or cloth window shades. Plus I had a lot of TV shows to watch, and all the books I brought back from France last summer just waiting to be read. Every once in a while I would check the Mac's screen, just to make sure, and this morning, eventually, I was able to contemplate the new setup, running smoothly, with a few new visible features, but mostly the new boot and launch speed (and damn, that SSD boot is faster than anything I've seen in a long while).

I know that putting a new third party disk into a box that's starting to show its age, adding the new version of the OS when it's just fresh out of master copy and using a new file system that was still in beta not so long ago is the exact opposite of hedging one's bets. And after reading a few cautionary articles after the fact I'm realizing that I may have been a little bold, but whatever happens, I have a backup that's regularly and automatically updated and I can just start from scratch any time I need... So I guess, it's all good, man!

"Open With" too many applications...

All this started a few weeks ago, and it is most probably related to Leopard's Time Machine. It may be related to other things too but then I have no idea what they are. The result is that you end up with duplicates of all your applications popping up here and there, first in the SpotLight result window, then in Mail, when you right-click on an attachment, in Finder too...

Sometimes you launch the backup instead of the original, and you realize when the backup disks starts to scramble like crazy...

Eventually, I found a solution. A little drastic, and maybe it was not all necessary, but it also contributed to rationalize my backup procedures.

Before:

1) Time Machine copied _all_ my disk to an external backup disk
2) I had Sync!Sync!Sync! do an extra daily backup of my whole home hierarchy (including all the "user space" applications)
3) I had Spotlight index everything

What I wanted was Spotlight indexing everything _but_ the applications.

The problem was that the Privacy setting of SpotLight did not allow for the selection of a subfolder in the Time Machine backup (like Applications, for ex).

Now:

1) Time Machine copies everything but the system Application folder, the user Application folder, the whole Developer folder and the Download folder.
2) Sync!Sync!Sync! does an extra daily backup of my whole home hierarchy, including the user space Application folder
3) Sync!Sync!Sync! does also a daily backup of the system's Application folder
4) I have Spotlight index everything (including the TimeMachine backup, of course) but the Sync!Sync!Sync! backups where all the Application duplicates are now.

I have also reindexed Spotlight using the following command in Terminal:

 $ sudo mdutil -Es / /Volumes/backup_disk_name/

And I have rebuilt the LaunchServices database, the one that contains all the "knowledge" about which application is supposed to launch which file, with the following command, also in Terminal:

 $ /System/Library/Frameworks/CoreServices.framework/\
Versions/A/Frameworks/LaunchServices.framework/\
Versions/A/Support/lsregister\
 -kill -r -domain local -domain system -domain user

SInce the whole "recipe" is based on saving applications outside Time Machine, I also had to erase all the data Time Machine had accumulated for a month... Setting Time Machine to not save /Applications from any day does not remove /Applications from the previous days backup sets... There may be smarter ways to do that, but I thought that with my already existing extra daily backup I was on the safe side.

Now, I have much saner "Open With" drop down lists when I right-click a file with my mouse...