19 October 2017

Applescript everywhere

Artificial languages are no different from natural languages. Either you're fluent and you will have fewer problems doing what you want to do, or you're not, and you're going to spend a lot of time on trivial issues that don't seem to make much sense.

I was in the middle of writing a very optimistic article on how automating your tasks could greatly enhance your workflow and your life in general, and how automation on Mac generally meant "Applescript" when I stumbled on some of those issues when trying to work with Terminal.

I will finish the article because computers are automation systems and using them without trying to automate more than what's provided by bundled programs means letting programs govern the way we work. But that episode painfully reminded me how non-trivial it was for me to work in Applescript, even though I am not exactly a beginner anymore, which does not mean that I know much, just that I know where to look for information when I don't know, and that I can more or less make sense of what I find.

With Applescript, I never read the "manual". I started with online examples, I copy pasted a lot, modified a few scripts, checked the errors, modified again until I got where I wanted, and checked the official reference only rarely (only 333 pages for the PDF I have, but still). And I'm saying to myself, now that I can sort of see how things fall in place, that I should definitely spare some time to go through either the reference or one of the excellent books that are out there.

If Applescript is a language, you're most likely going to use one of its "dialects" too, since any application that supports Applescript actually has a list of words that are specific to it and if the words look the same between applications, they are, like we say in French about some English words "false friends". They look the same but don't always mean the same. Applescript is not a language that you will use to make heavy-duty computations (although you could). Applescript was made to work with "Apple Events", which are used to connect applications together. As you'll eventually read in that other article, one of my script creates a Finder group of folders based on a job request I received in Mail that will end up as a Calendar item.

Applications that don't support Applescript can also be talked to: think about the primitive sign language that we use when we don't know a language. But that can be pretty powerful because just as you can show a person how to click the button that will move a robot, you can also tell Applescript to click on any button or menu item of pretty much any application that runs on a Mac. In fact, some parts of Applescript supporting applications are only available through their user interface, for example, asking a Terminal window to enter full-screen mode requires an Applescript that says "click on such menu item of such menu bar" or even better "hit F while pressing Command and Control" to simulate the shortcut. That's the part that bit me last weekend when I was trying to create a tabbed full-screen window in Terminal opened on the folder that was selected in Finder.

And Applescript can also call the services that are hidden under your Mac's hood: the command line applications that you can usually only access through Terminal, and reciprocally, you can call Applescript from the Terminal. I do that when I select an OmegaT project folder in Finder and use Applescript to tell the command line to use that folder as a parameter to launch OmegaT.

All this is very powerful but can be pretty daunting for amateurs. In my next article, I'll try to show you how I got where I'm at today, which is, in all honesty, not very far ahead of any person who's reading about Applescript today for the first time. I'll publish the Applescript code I'm using, with extensive comments, so that you don't have to just blindingly take for granted what I wrote but get to understand why I wrote it that way. And hopefully that will help you being more productive while working on your Mac.

Update (a few minutes after posting):
I clicked on the "Applescript" tag that's in the column on the right side of the articles, which brought me back to November 2007 when I wrote "I have never found a practical way to use AppleScript in my workflows..." Well, things have changed and now I'm pretty sure I could not live without Applescript on my machine...

10 October 2017

Adventures in Kernel Panic Land

Jimmy McGill chose "Saul Goodman" as his pseudo for a new venture to sell his now useless TV commercial slots, because "it's all good, man!"
That's exactly what your Mac tells you every day. Everything works fine, and everything will work out fine tomorrow, until it doesn't, but then, it's a Mac, so you're all good.

You never know that you need a backup until you do. That's the rule. I learned it about 10 years ago, the hard way. Massive hard disk failure, no backup at all, no extra software to check/fix/save the disk. Nothing, but a big deadline 2 days away. At the time I was running an Itanium Powerbook. I had bought it second hand a while before and it was a real workhorse. I also had a greenish clamshell iBook that my wife used to work with, before her company lent her a Windows machine.

When the machine's hard disk died on me, I called the project manager. He roughly told me that I still had 2 days and that the schedule would not move an inch. I don't remember what I did to fix the mess, but I did manage to scrap most of the data from the disk, then I must have moved all that to the iBook, resumed work, delivered, celebrated, and decided to create regular backups.

In the 21st century, external drives are cheap. They are cheaper than whatever hourly rate you apply to the time spent on fixing a non-backed-up dead system. They come in all sorts of flavour, shape, colour and capacity. And if your main machine is a laptop and it's a pain to physically connect it to your backup disk every time you put it back on your desk, you even have Wifi backups.

We tried half-backed solutions for a while. I bought terabytes of external drives, ugly as crap, all Firewiry and USB-Cy, but when you want a transparent solution that just works, all this just doesn't scale. Full disclosure: I have no financial interests in Apple. But damn, Time Capsule rocks. So yes, we got ourselves a Time Capsule, just 2 TB. We had 2 iMacs and 2 notebooks at the time, we thought we'd be short with the capacity but the 3TB Capsule seemed too expensive for just one extra terabyte. But even with the "low" capacity, it's been with us since. It does everything over Wifi, no need to connect the machines, just turn them on and Time Machine does its thing, and in no time you have your incremental backup ready for when you need it. You do need to turn the Time Machine on and select Time Capsule as your backup target, but it's a one time setting, and then you can forget about it, pretty much.

When things work fine, the only time you need a thorough backup is when you upgrade your system. I don't know how many more OS upgrades this "early 2011 Macbook Pro" will be allowed (our 24" iMac could not go further than Yosemite) but I was not going to miss that High Sierra upgrade after seeing the APFS talks on internet. The rest of the icing, I was not very interested in, although I love the way Safari does a lot to keep advertisers from wasting our CPU (and brain) cycles.

What had been bothering me for a while was that the machine was getting slower and slower everywhere and after having an issue with how slow BBEdit was, the Barebones support team confirmed that it was most probably a hard disk issue. With APFS coming I was not going to replace my disk with a rotating thing again. I was done with that. I was going to go the SSD way. All flashy, all speedy, all slick electronics.

My son had done it a few months back. We bought him an MBP when he entered university, one of those new but still coming with the antiquated DVD player. In his department, he was the only freshman to use a Mac. When he came back for Christmas last year, I paid for the replacement Samsung SSD and he spent a few hours doing the thing: backup your data and follow the instructions on Youtube. Easy enough. Then he reboot the machine and everything was faster. And it was before High Sierra and APFS.

I was going to go incremental on this. First install High Sierra, then get an SSD, and last reinstall everything. But before all, the backup. Time Machine being automatic, you really don't have to do much usually. But when you want to do it "now", well, there is a menu item for that. It's called "Back Up Now". To make sure you don't forget to save a few last-minute settings, just quit all the applications you run, leaving only Finder on (although if you know how to quit it, you can too), then let Time machine prepare the backup, do the backup and clean up the backup.

Then, you can start the upgrade process itself. Go to apple.com, click on "Upgrade" and everything goes super smoothly. At one point you are presented with a dialog that asks whether you really agree with all the terms, you just click "Agree" unless you're a hard-core free software advocate, but in this case you're not running a Mac and you're not reading this page, and then files get downloaded and installed and you can go watch your favourite show in the meanwhile, or read a book.

That's what I had planned anyway. Either finish season 3 of "Better call Saul" or read "Consider Phlebas", but it did not happen that way. What happened is that after a long while and not much happening, the Mac complained that it could not be upgraded. Just like that, two thirds into the install process, when the machine had overwritten all those important parts of the disk and you know that if you don't find a solution, you're done.

Rebooting won't do it. The old OS has been blown to pieces by the newer files partially written over it and the machine is programmatically speaking a giant spaghetti mess that the CPU has no idea how to deal with. When that happens, you have a Kernel Panic, and when the kernel panics it is of the utmost importance that you don't panic. Whatever you imagine has happened. All the pics of the newborn, all the thousands of whatever currency you spent on music and movies. Everything is gone. Except that, it's on your backup, and the likeliness that your computer and your backup die on you at the exact same time is so low that it's even lower than the likeliness of you being under the atomic fire of a North Korean missile. Although, granted, the day this happens, you'll have things more important to worry about than a broken backup.

So, you see the Kernel Panic thing that actually looks like a personal reboot of The Matrix on your screen and you feel all warm inside because you know it's a Mac. The idea here is that you need a handy way to get information from the Net. I don't have a phone, but I have that extra 24" iMac that's now mostly used as a Minecraft machine by the 2 little ones, but it also runs Safari (mostly used by the same little ones to check new "mods" for said Minecraft) and so I have a powerful entry point into the internets of information.

What you need to know here (and Safari is your friend if you don't) is that your Mac comes with secret programs that you can launch when you need them because they are located somewhere that doesn't get to be erased by a gigantic mess up of overwritten OS files. One such program is "Recovery Mode". You can launch it by hitting Command+R right after you restart your machine. If your system is down, like mine was, and you don't have a boot disk or USB key, then you want to enter Internet Recovery Mode, which requires an internet connection to download to your Mac all the things necessary to fix your issues (you may want to physically connect to your router/Time Capsule with anything that is faster than Wifi).

"Recovery Mode" be it local or internet based can do four things. The first one is get a Time Machine backup and restore the full machine from there. The second one is reinstall the OS from scratch. The third one is launch Safari to give you access to Apple Support for help. And the fourth one is give you access to Disk Utility so that you can diagnose/fix disk problems.

I figured that since I did not have a separate boot system my best bet was a full recovery from Time Machine. But I tried Disk Utility before that, checked the disk and since it told me everything was normal I just went on and decided to test an OS install. Depending on the key you call Recovery with, the version of the OS that's proposed to you is different. Either you have the OS your machine came installed with when you bought it, or you get the latest version that was installed, or you get the latest compatible version.

I tried High Sierra. And it did not work. I got a number of "cannot access the recovery server" or similar messages and gave up. I rebooted, again in Recovery Mode, tried again with the originally bundled version (Lion ?) and install started. And never stopped. It was late in the night when a dialog announced that the install would really not work.

This kind of moments are when you desperately need to cling to your faith in Macs (slightly backed by actually very few horror stories of messed up machines that refused to recover), and try one last thing before you go to bed. Reboot/Recovery Mode/Reinstall, it was staring to become a routine. But for some reason, this time it worked, and in the morning I had a (not so High) Sierra machine up and running, or rather ready to accept my backup data. A few hours and 250 GB later, I was running my machine, with a fresh Sierra install and all my data in place. I was basically at the exact same point I was the day before, but stronger in my faith that Macs just work and when they don't they still do.

The SSD arrived a few days later. That kind of manipulation involves a slightly higher level of wizardry. You need a few tools and an iFix video. Tools mean screws, and you really don't want to lose the screws. So here we go. Old hard disk removed, screws put in separate boxes to not mix them all, new SSD thing installed, screws back in place, power supply back on, reboot and Internet Recovery Mode launch.

Normally, an all brand new disk is not formatted to accept data copying. It is just a blank slate that the OS must put in shape for it to be of any use. But the OS still recognizes that there is a disk willing to be formatted attached to the machine. Normally.

In my case, the SSD thing was nowhere to be seen. The law of probability was plain about that: the thing itself is not at issue. Maybe an internal connection ? Would I have to open the Mac again and check everything ? I'm lazy and I prefer the Aspirin of Computer Fixing: reboot. A reboot/recovery mode later, I had the thing under my eyes. Go figure. Disk Utility was offering me to format the disk to APFS, which would also erase the data I had spent one full day reinstalling. Go for it, I thought, I'd just spend one more day going through the recovery hoops to get my machine running with the new OS.

Here again, things did not happen as planned. It took a small number of attempts to get High Sierra to be installed on the disk, and then I had to reinstall my data again from the Time Capsule backup.

I was lucky. We had a 3 day weekend here and I had decided to remove the curtains in both our room and the kids' and replace everything with wood or cloth window shades. Plus I had a lot of TV shows to watch, and all the books I brought back from France last summer just waiting to be read. Every once in a while I would check the Mac's screen, just to make sure, and this morning, eventually, I was able to contemplate the new setup, running smoothly, with a few new visible features, but mostly the new boot and launch speed (and damn, that SSD boot is faster than anything I've seen in a long while).

I know that putting a new third party disk into a box that's starting to show its age, adding the new version of the OS when it's just fresh out of master copy and using a new file system that was still in beta not so long ago is the exact opposite of hedging one's bets. And after reading a few cautionary articles after the fact I'm realizing that I may have been a little bold, but whatever happens, I have a backup that's regularly and automatically updated and I can just start from scratch any time I need... So I guess, it's all good, man!

23 September 2017

Writing, because Emacs

Blogging is writing and publishing. I've not been writing here much for lack of a relatively frictionless process. And also because, well, translators on Mac don't seem to have so many issues anymore with the platform. Or maybe they don't really exist. I'm not sure.

Some people say they've moved to Emacs because of org-mode. Emacs? org-mode? The names are probably unknown to most translators. But they're what brought me back to writing, and eventually to publishing, at least here and other places.

Emacs is a venerable text editor. It is one of the first "officially" free programs published ever, and it is the beginning of the Free software movement, in a way. Emacs is in fact more than a text editor. And a lot of people are confused by that because Emacs is before anything a Lisp virtual machine that was made to run on any platform.

Lisp machines used to be hot in the 80s, when research on AI was at its peak. Lisp machines were computers that understood Lisp down to the CPU... And Lisp was the language of choice for AI back then. Lisp itself is the second-oldest programming language still in use today. Its implementation started in autumn 1958 explicitly as a language to research AI at the MIT. A few years later, copyright issues and headhunting resulted in fewer and fewer people able to freely use and develop Lisp, and that's when Emacs was conceived, at first as a way to put Lisp on any possible machine without limiting them to Lisp Machines, to fight against the people who were attempting to close access to that knowledge.

Emacs now looks like a text editor because a text editor is a required tool to write Lisp code. But besides for Lisp code writing, which it does very well, Emacs is able to write any kind of things. And programs in Lisp written for Emacs have been extending Emacs functions to areas that were never envisioned by the original creators.

org-mode is one such program. It does not look like a program, of course. It looks like you're doing things in Emacs and it's easy to organize them because org-mode is here to help you.

I wrote above that a lot of people have moved to Emacs because of org-mode. In Emacs talk, a mode is a group of specialized functions. In my Emacs, I have installed a chess mode, that allows me to play chess in Emacs, a "deft" mode that emulates in Emacs the behavior of Notational Velocity, the famous note taking app for OSX, I also have apples-mode that transforms Emacs into an Applescript editor or writeroom-mode, that works like all the "distraction-less editors" around, except that it turns Emacs into that distraction-less editor. Some people say that Emacs is an operating system and that you don't have to leave it to do your computing. You can browse the web, do your mail, write text, read PDF files, etc.

org-mode was created over the outliner-mode that Emacs provides. Outliners exist everywhere. Mac has many nice outlining applications where you can put notes in a given hierarchy, sort them, search their contents, use them as reminders or as todo lists, etc. org-mode original creators started to add functions to the Emacs outliner, and then everything grew so big that a separate mode was born.

I am just a beginner in Emacs. Although I started using it about 20 years ago, I never got to actually use it because most of my "writing" work then involved regular expressions and the tool of choice on Mac at the time was BBEdit Lite. I still don't do regular expressions in Emacs. When I need to do relatively complex searches, I still switch to BBEdit (Textwrangler is going to be discontinued and BareBones is planning to default BBEdit to a lite version, except for the paying users who'll be able to unlock the full thing).

I am just a beginner but I have traces of Emacs edited files on my machine dating back at least 10 years. I remember meeting a then not yet world-famous Emacs contributor/blogger in Tokyo, almost 13 years ago and showing her an Emacs book in Japanese I had just bought. And I remember reading about and using Emacs at the end of the 90s when I was attempting to install and run Linux on that IBM/Lenovo machine I had just bought when I came to Japan. It worked eventually, but Unicode was not widely spread then and I had to constantly switch between a French environment and a Japanese environment to do my work on Linux, and that was not convenient. So I moved (back) to Mac, which I had started using a few years before, where Emacs then was not as attractive for the casual writer as it is today.

Why Emacs? For me it was not org-mode, which did not exist then. It was just the intuition that Lisp was an amazing language and that I ought to learn it. 12 years ago there was another peak Lisp with the publishing of "Practical Common Lisp". There were Emacs distributions that were tweaked to include the latest Lisp editing environment, there were blogs that discussed the marvels of writing Lisp in Emacs. There were beginning attempts at curating the various libraries that were spread all over the place. There were discussions about creating the best Lisp ever... It was like a Lisp rebirth, and the Emacs mode that everybody talked about then was slime, the best Lisp editing environment.

I never got to go beyond the cover of the Lisp books I kept buying though. And it is only many years later, that org-mode and the articles I kept reading about it finally got me started, along with the nagging feeling that I should be better organized and that I should start writing, not just resume blogging, but write my own stuff. And for that I needed no bells and whistles, no fancy WYSIWYG application, just white characters on a black screen with wide margins and nothing else.

I'm writing this article from org-mode. The article is one "node" in a big "blog.org" file where all the nodes are listed in chronological order, with the date and time I started them in the header title. I start writing an article by calling a "capture template". This one has the name of this blog. I have 9 different templates. One for a big TODO file, one for a personal FAQ where I add questions and answers about all the stuff that I want to know about Emacs, org-mode, git, and various other things. Then I have one for taking various notes, I have a daily journal for things that won't be published, then 2 templates for 2 different blogs (including this one), 2 others for 2 novels I have in mind and a last one for an introduction to Emacs Lisp that I've wanted to write for a long time now.

It took me a while to realize that the feature that would hook me to Emacs was this "capture" thing. org-mode is a lot about headers, subheaders, TODO states, lists, deadlines and calendars and timers. It is a huge planner machine that goes in so many directions that it is not possible to fully use it all. The terms used in the manual are sometimes obscure and the manual itself goes into a lot of details not relevant to the beginning user. So you really have to wander on the net to find the tutorial that works for you, tweak some settings and start. I have 51 documents on my machine that I tagged with "org". I've browsed them all at least once. I keep them for easy reference to see what people do with org-mode, but the documents I consult the most now are the Org Manual and the Emacs manual.

When you start tweaking, it never ends, but it's OK. Your documents are being written in a live Lisp environment. Everything you do, from typing a single letter to reorganizing your paragraphs or checking your calendar is done by calling functions that you can modify right away by just editing and evaluating them. Assigning shortcuts is also a Lisp function away. But you don't have to worry about learning Lisp before you start using Emacs. It just comes, almost naturally, little by little. And it is not difficult. It took me just a few days of reading and researching before proposing code that fixed a bug in an OSX service Emacs.app provided, and that code was eventually accepted for release in Emacs. I was totally new to Emacs Lisp and even though it was only a few lines of trivial code the whole thing was not trivial, but it was not hard to find the issue, understand it, and fix it, because I could do that right there, inside that live Lisp environment that Emacs is.

I wish there was a powerful translation mode in Emacs. There is a po-mode that does interesting things, but it is nowhere near what professional translators use. There was an attempt a long time ago at linking the current target segment of an OmegaT project to Emacs, but the device involved a lot of intermediate technologies (a bridge from Java to Python and then one from python to Emacs if I remember well). Doing CAT based translation work in Emacs is not possible now, as far as I can tell. But that's fine. I just need to know that I have found a pretty good writing environment and that I can just "capture" my thoughts, put them on virtual paper and move on.