emacs regex with emacs lisp

June 28 update: using \#1 instead of (string-to-number \1)

A reader on reddit mentionned that the manual also had the "\#d" construct to replace the often used (string-to-number \d) function.

That regex-replace improvement was mentionned in Stevve Yegge's emacs 22 introduction, back in 2006.

Last but not least, I noticed that the whole post was originally written with (string-as-number ...) when the correct function name is (string-to-number ...)

---

Not strictly related to translation but here is what's happening...

I've resumed studies last year, trying to finish an MA in Japan Studies I started 25 years ago.

For the first year, I only have to write a 30ish pages dissertation on my subject (representation of women in kendo magazines in Japan) and I decided to go the emacs + org-mode way, with the easy export to ODF function that's packaged with the thing.

So I decided to write each chapter in a different org file, and send them one by one to my director. But then, for the final delivery I needed to put all that in one big file and was faced with the fact that all my footnotes would need to be re-indexed manually because each file had notes starting at 1...

I usually use BBedit for any serious regex work. Mostly because the interface is clearer than emacs, and the regexp feels more modern (\d vs [:digit:])

But one thing you can't do in BBEdit is to send commands to the replace string. For ex, in my case, add 21 to the matching number, which seems pretty trivial, when you think of it, but doing that will involve other technologies, like using perl or some other command line thing.

In emacs, however, everything can be interpreted as an expression, hence you can insert code wherever you want and get the result from that code right in the document.

The org-mode footnotes all look like [fn:12], where "12" is the note number that I need to replace with an incremented number. Since there are no instances of fn:\d+ without the brackets that are not footnotes, I figured I could just be searching for that string:

fn:\([:digit:]+\)

Notice that in emacs, "(" and ")" need to be escaped, also I could have used the [0-9] class.

In BBedit I'd just need:

fn:(\d+)

And now I need to replace that with the expression that will add 21 to the number.

In BBEdit, I'd be stuck here. I just can't add anything to a match. In emacs, I can replace the match with that:

fn:\,(+ 21 (string-to-number \1))

The emacs lisp expression is "(+ 21 (string-to-number \1))", which means "convert the \1 match that is a string into its numerical value and add 21 to it".

But, wasn't \1 supposed to match [0-9]+, which is a number? Well, yes, but really it's just digits, hence strings, that have no numerical value whatsoever, so first, we need the expression to convert them to a numerical value before adding 21 to them.

Now, the trick is to have the expression be handled as an operation and not as an arbitrary string, and that's where the "\," prefix comes into play.

"\," tells the replace engine that the string that follows must be interpreted as an emacs lisp expression and not as a mere string. With it, the regexp replaces properly adds 21 to my note numbers, and I get two dozen footnotes updated in one fell swoop...

I love BBEdit and its people, but emacs is really a gift that keeps on giving.

---

Here are some handy references:

the emacs regexp-replace function

Regexp Replacement

the emacs regexp syntax

Syntax of Regular Expressions

the emacs-lisp string-to-number function

Conversion of Characters and Strings

And here is a really super short introduction to lisp syntax

There is no real need to have a very deep understanding of emacs lisp to use this regexp-replace function. Just remember that a lisp expression generally looks like this:

(operator operands)

Where the operator is a generally a function, like + or string-to-number above, and the operands can be any expression that is accepted by the operator. So, here:

(+ 21 (string-to-number \1))

means:

add 21 to the result of the expression (string-to-number \1)

with (string-to-number \1) meaning:

convert the string matched by \1 to its numeric value

Obviously, if \1 is not a string, the conversion will fail and the addition won't work. And without that conversion, if we had just added \1 as a string, the addition that expects numbers as operands would have failed.

I just realized that this is my first emacs lisp related post ever ! I'd like to thank that person I met in Tokyo about 15 years ago who showed me the way. It's an egg that definitely took some time to hatch...

Easily launch scripts from Spotlight

I've been advertising AppleScript a lot here. Automating a task is something, but easy access to that automation is quite important too.

Since I try to stick to Apple solutions and free software* I prefer using Spotlight instead of all the smart launchers that we have for macOS.

What I do is that I call my script names that are easy to call first in Spotlight. A few screenshots speak louder than words so here we are:

The series that start with ">" usually is scripts that I use to open something.

See how just typing ">" suggests ">BB". That ">BB" is what I use to open the files selected in Finder with BBEdit.

The one below is ">Tedit", short for "TextEdit" and opens selected files in TextEdit, etc.

If I type "c" after ">", I get the following list of choices. ">command" is to launch an arbitrary command in Terminal, ">Capture" is to use org capture in Emacs (see here and here for more information) and ">cd" just opens a Terminal tab on the front Finder window, or selected Finder folder.

When I use "<" to initiate the search, I get a different list. That series is for scripts that usually act by themselves. "<text file" creates a text file in the front Finder window and proposes to open it in BBEdit for editing, "<facturation" is an invoicing script for the job selected in Finder, "<job" is a job managing script that creates a job hierarchy in Finder based on a mail, along with an event in calendar, and then "<xls2tmx" is a TMX converter for multicolumn Excel reference data (I'll publish it when it's more polished, but creating XML data with AppleScript is documented here).

I have a few more scripts (a dozen) that I routinely call with Spotlight, which I find totally sufficient for my needs.

As you know, hitting Command+Enter when you have a selection in Spotlight is a way to reveal that selection in the Finder if it is available. So when I want to edit a script, I start by calling it in Spotlight, I hit Command+Enter when it is selected and then I call ">SEditor" (Script Editor) or ">debugger" (Script Debugger) on the selection, to open it with the appropriate application...


* Ok, I do have BBEdit, Microsoft Office and Illustrator... And maybe a few others...

Open a file in your editor of choice

[Update]
Chris Stone has a nice follow-up on the BBEdit user forum.

---

You know how it is. You double-click on a file thinking it will open in the application that you're working with at the moment and you forget that the file type was associated with a different application...

Back to Finder, right-click on the file, select "Open with..." and you're good. But mousing around macOS can be tedious at times so here is a simple script that I was pretty much given by an ASUL co-lister and that I barely had to adapt to my workflow to make it more general.

property targetApplication : "BBEdit"

tell application "Finder" to set mySelectionList to selection as alias list

if length of mySelectionList = 0 then error "No files were selected in the Finder!"

tell application targetApplication

repeat with myFile in mySelectionList

open myFile

end repeat

activate

end tell

You notice right away that the only reference to the opening application is in the first line. You can change the application name to anything you want and have multiple copies of the script, one for each application, with an appropriate name so that you can open any file with any supporting application you want.


Here is the exact same code for TextEdit:

property targetApplication : "TextEdit"

tell application "Finder" to set mySelectionList to selection as alias list

if length of mySelectionList = 0 then error "No files were selected in the Finder!"

tell application targetApplication

repeat with myFile in mySelectionList

open myFile

end repeat

activate

end tell

The same for Script Editor:

property targetApplication : "Script Editor"

tell application "Finder" to set mySelectionList to selection as alias list

if length of mySelectionList = 0 then error "No files were selected in the Finder!"

tell application targetApplication

repeat with myFile in mySelectionList

open myFile

end repeat

activate

end tell

The same for Script Debugger:

property targetApplication : "Script Debugger"

tell application "Finder" to set mySelectionList to selection as alias list

if length of mySelectionList = 0 then error "No files were selected in the Finder!"

tell application targetApplication

repeat with myFile in mySelectionList

open myFile

end repeat

activate

end tell

The same for Word:

property targetApplication : "Microsoft Word"

tell application "Finder" to set mySelectionList to selection as alias list

if length of mySelectionList = 0 then error "No files were selected in the Finder!"

tell application targetApplication

repeat with myFile in mySelectionList

open myFile

end repeat

activate

end tell

You get the drift.

Some applications don't work that way, so here is the code for Emacs.app:

use AppleScript version "2.4" -- Yosemite (10.10) or later

use scripting additions

tell application "Finder" to set fileSelection to selection as alias list

try

tell application "System Events" to tell process "Emacs" to set frontmost to true

on error

tell application "/Users/suzume/Documents/Code/emacs/nextstep/Emacs.app"

activate

delay 1

end tell

end try

# return

repeat with selectedFile in fileSelection

set the clipboard to (POSIX path of selectedFile)

tell application "/Users/suzume/Documents/Code/emacs/nextstep/Emacs.app"

tell application "System Events"

delay 0.5

keystroke "x" using {control down}

keystroke "f" using {control down}

keystroke "a" using {control down}

keystroke "v" using {command down}

keystroke "k" using {control down}

key code 36 # Escape

delay 0.1

end tell

end tell

end repeat

The code makes use of UI scripting because Emacs does not support Applescript, but it works just as well.

What you do now, is create one script for each application you want to open your files with, save the script with a name that Spotlight will easily call first (something like ">BB" for opening in BBedit) and you're done!

Thank you Chris Stone on ASUL (and all the others) for your help !

Writing, because Emacs

Blogging is writing and publishing. I've not been writing here much for lack of a relatively frictionless process. And also because, well, translators on Mac don't seem to have so many issues anymore with the platform. Or maybe they don't really exist. I'm not sure.

Some people say they've moved to Emacs because of org-mode. Emacs? org-mode? The names are probably unknown to most translators. But they're what brought me back to writing, and eventually to publishing, at least here and other places.

Emacs is a venerable text editor. It is one of the first "officially" free programs published ever, and it is the beginning of the Free software movement, in a way. Emacs is in fact more than a text editor. And a lot of people are confused by that because Emacs is before anything a Lisp virtual machine that was made to run on any platform.

Lisp machines used to be hot in the 80s, when research on AI was at its peak. Lisp machines were computers that understood Lisp down to the CPU... And Lisp was the language of choice for AI back then. Lisp itself is the second-oldest programming language still in use today. Its implementation started in autumn 1958 explicitly as a language to research AI at the MIT. A few years later, copyright issues and headhunting resulted in fewer and fewer people able to freely use and develop Lisp, and that's when Emacs was conceived, at first as a way to put Lisp on any possible machine without limiting them to Lisp Machines, to fight against the people who were attempting to close access to that knowledge.

Emacs now looks like a text editor because a text editor is a required tool to write Lisp code. But besides for Lisp code writing, which it does very well, Emacs is able to write any kind of things. And programs in Lisp written for Emacs have been extending Emacs functions to areas that were never envisioned by the original creators.

org-mode is one such program. It does not look like a program, of course. It looks like you're doing things in Emacs and it's easy to organize them because org-mode is here to help you.

I wrote above that a lot of people have moved to Emacs because of org-mode. In Emacs talk, a mode is a group of specialized functions. In my Emacs, I have installed a chess mode, that allows me to play chess in Emacs, a "deft" mode that emulates in Emacs the behavior of Notational Velocity, the famous note taking app for OSX, I also have apples-mode that transforms Emacs into an Applescript editor or writeroom-mode, that works like all the "distraction-less editors" around, except that it turns Emacs into that distraction-less editor. Some people say that Emacs is an operating system and that you don't have to leave it to do your computing. You can browse the web, do your mail, write text, read PDF files, etc.

org-mode was created over the outliner-mode that Emacs provides. Outliners exist everywhere. Mac has many nice outlining applications where you can put notes in a given hierarchy, sort them, search their contents, use them as reminders or as todo lists, etc. org-mode original creators started to add functions to the Emacs outliner, and then everything grew so big that a separate mode was born.

I am just a beginner in Emacs. Although I started using it about 20 years ago, I never got to actually use it because most of my "writing" work then involved regular expressions and the tool of choice on Mac at the time was BBEdit Lite. I still don't do regular expressions in Emacs. When I need to do relatively complex searches, I still switch to BBEdit (Textwrangler is going to be discontinued and BareBones is planning to default BBEdit to a lite version, except for the paying users who'll be able to unlock the full thing).

I am just a beginner but I have traces of Emacs edited files on my machine dating back at least 10 years. I remember meeting a then not yet world-famous Emacs contributor/blogger in Tokyo, almost 13 years ago and showing her an Emacs book in Japanese I had just bought. And I remember reading about and using Emacs at the end of the 90s when I was attempting to install and run Linux on that IBM/Lenovo machine I had just bought when I came to Japan. It worked eventually, but Unicode was not widely spread then and I had to constantly switch between a French environment and a Japanese environment to do my work on Linux, and that was not convenient. So I moved (back) to Mac, which I had started using a few years before, where Emacs then was not as attractive for the casual writer as it is today.

Why Emacs? For me it was not org-mode, which did not exist then. It was just the intuition that Lisp was an amazing language and that I ought to learn it. 12 years ago there was another peak Lisp with the publishing of "Practical Common Lisp". There were Emacs distributions that were tweaked to include the latest Lisp editing environment, there were blogs that discussed the marvels of writing Lisp in Emacs. There were beginning attempts at curating the various libraries that were spread all over the place. There were discussions about creating the best Lisp ever... It was like a Lisp rebirth, and the Emacs mode that everybody talked about then was slime, the best Lisp editing environment.

I never got to go beyond the cover of the Lisp books I kept buying though. And it is only many years later, that org-mode and the articles I kept reading about it finally got me started, along with the nagging feeling that I should be better organized and that I should start writing, not just resume blogging, but write my own stuff. And for that I needed no bells and whistles, no fancy WYSIWYG application, just white characters on a black screen with wide margins and nothing else.

I'm writing this article from org-mode. The article is one "node" in a big "blog.org" file where all the nodes are listed in chronological order, with the date and time I started them in the header title. I start writing an article by calling a "capture template". This one has the name of this blog. I have 9 different templates. One for a big TODO file, one for a personal FAQ where I add questions and answers about all the stuff that I want to know about Emacs, org-mode, git, and various other things. Then I have one for taking various notes, I have a daily journal for things that won't be published, then 2 templates for 2 different blogs (including this one), 2 others for 2 novels I have in mind and a last one for an introduction to Emacs Lisp that I've wanted to write for a long time now.

It took me a while to realize that the feature that would hook me to Emacs was this "capture" thing. org-mode is a lot about headers, subheaders, TODO states, lists, deadlines and calendars and timers. It is a huge planner machine that goes in so many directions that it is not possible to fully use it all. The terms used in the manual are sometimes obscure and the manual itself goes into a lot of details not relevant to the beginning user. So you really have to wander on the net to find the tutorial that works for you, tweak some settings and start. I have 51 documents on my machine that I tagged with "org". I've browsed them all at least once. I keep them for easy reference to see what people do with org-mode, but the documents I consult the most now are the Org Manual and the Emacs manual.

When you start tweaking, it never ends, but it's OK. Your documents are being written in a live Lisp environment. Everything you do, from typing a single letter to reorganizing your paragraphs or checking your calendar is done by calling functions that you can modify right away by just editing and evaluating them. Assigning shortcuts is also a Lisp function away. But you don't have to worry about learning Lisp before you start using Emacs. It just comes, almost naturally, little by little. And it is not difficult. It took me just a few days of reading and researching before proposing code that fixed a bug in an OSX service Emacs.app provided, and that code was eventually accepted for release in Emacs. I was totally new to Emacs Lisp and even though it was only a few lines of trivial code the whole thing was not trivial, but it was not hard to find the issue, understand it, and fix it, because I could do that right there, inside that live Lisp environment that Emacs is.

I wish there was a powerful translation mode in Emacs. There is a po-mode that does interesting things, but it is nowhere near what professional translators use. There was an attempt a long time ago at linking the current target segment of an OmegaT project to Emacs, but the device involved a lot of intermediate technologies (a bridge from Java to Python and then one from python to Emacs if I remember well). Doing CAT based translation work in Emacs is not possible now, as far as I can tell. But that's fine. I just need to know that I have found a pretty good writing environment and that I can just "capture" my thoughts, put them on virtual paper and move on.

Introduction to regular expressions

[2018 update]

Textwrangler has now been replaced by BBEdit that offers a free trial and then turns into a BBEdit "free mode". BBEdit "free mode" is a better TextWrangler and also happens to be a 64 bits application compatible with High Sierra.
https://www.barebones.com/products/bbedit/comparison.html

I mention Aquamacs at the end of the article. Regular expressions for Aquamacs but also for Emacs are documented in the Emacs manual available here:
https://www.gnu.org/software/emacs/manual/html_node/emacs/Regexps.html

Check this article too:
https://www.johndcook.com/blog/2018/01/27/emacs-features-that-use-regular-expressions/

In OmegaT, you lose a lot if you're not familiar with regular expressions. OmegaT uses the Java flavor of regular expressions:
https://omegat.sourceforge.io/manual-standard/en/appendix.regexp.html

OmegaT allows you to create "tags" that can be protected and checked during the translation by using regular expressions. In a recent project I had created a 872 characters long regular expression that described 71 different tags.

As of February 2018, OmegaT also allows for replacements with capture groups as described below:
https://sourceforge.net/p/omegat/feature-requests/953/

And you can also use regular expressions to search for empty segments, as I document in this article:
https://mac4translators.blogspot.com/2018/03/searching-for-empty-translations-in.html

---

The technology that had the most impact on my workflow is definitely "regular expressions".

I discovered them at the end of the 90' when I was working on the conversion of a database output to a set of about 6000 static HTML pages. At the time, the editor of choice on the Mac was BBEdit from Barebones Software, but its free and "lite" version "BBEdit Lite" was also immensely popular. BBedit Lite has now been replaced by Textwrangler and just like its predecessor, Textwrangler can be used without paying a user license fee*.

What are regular expressions?

Regular expressions are a "search" function on steroids. Regular expressions were created to find patterns in strings. They can find simple patterns like the word "pattern" in this text, or more complex patterns like "a string that starts with 'pa', followed by a letter that's repeated twice, followed by any three characters that are neither 'space' nor '@' or '^' and followed by a space".

This document uses its first two paragraphs (the paragraphs in italics, above) as a test ground. Paste that paragraph in your favorite regular expressions supporting text editor (I use Textwrangler for all the descriptions so you might want to use it too) then call the search window, check the "grep" box at its bottom and search for:

re[^ ]*

You should see colors appearing while you type the search terms.

What that expression means is:
r followed by e followed by a group of characters that are not a space, or by nothing.

Hit Next and see what you get, then hit Cmd+G and see what you get. If you start from the top of the paragraph, you should have 8 "matches".

Normal characters

Most characters represent themselves in regular expressions (regex), like a "normal" search.

→ r means r and e means e, " " means a space. In the same sequence. No magic here.

Special effects

Some characters have special effects:

→ [ starts a group of characters
→ ] ends that group
→ ^ means "not"
→ * means "zero or more of what just came"

So, our above simple regular expression re[^ ]* means:

"look for any string that has a r followed by a e followed by zero or more characters that are not a space."

Now, what if you need to find characters like ^, [, ] or *?

Cancelling special effects

When you want to find characters that have a special effect without "triggering" that special effect, you put a "\" in front of them:

→ \* means the character *
→ \[ means the character [


And since the character "\" has the special effect of removing the special effect of a character that has a special effect... then:

→ \\ means the character \

etc.

By the way, the character . has the special effect of matching "any one character" so if you're looking for a period, then you really want to look for the \. string...

Examples:

The regular expression ". " (. followed by space) will match any one character followed by a space. There are 78 strings that match this pattern in the paragraph.

The regular expression "\. " (\ followed by . followed by space) will match any period followed by a space. There are only 2 strings that match this pattern in the paragraph.

The regular expressions "re*." (re followed by * followed by .  will match any string that is composed of a r followed by zero or more e, followed by any one character. There are 22 matches in the paragraph. Verify that you understand them all.

The regular expression ".e\*\." (. followed by e followed by \ followed by * followed by \ followed by .) will match the 4 characters string ee*. that you find at the end of the paragraph.

Triggering special effects

Some characters work the other way round: by themselves they do not have a special effect but if you stick the \ character before them, then their special effect is triggered.

→ t means t but \t means tabulation
→ r means r but \r means line break (specifically "carriage return")
→ s means s but \s means all sorts of white space, which includes spaces, tabulations, line breaks, etc.

If the character does not have a special effect then using \ has no effect.

→ i means i and \i too means i

Such sequences (\ followed by a character) are usually called "escape sequences".

Remembering matches

If you want to "memorize" a match, for later use in the expression or in the "Replace:" field, then you put the corresponding expressions between parenthesis:

→ (re)[^ ]+ will produce the same matches as above, but will memorize the re part and not the rest.

→ re([^ ]+) will produce the same matches as above, but will not memorize the re part and instead will memorize the rest.

→ (re)([^ ]+) will produce the same matches as above and will memorize the 2 parts separately.

Using memorized matches

Now that the matches are remembered, you can use them. Use \1 to refer to the first memorized string, \2 to refer to the second memorized string, etc...

→ (e)\1\*\. will match the "ee*." string that you find at the end of the text.

→ search for (re)([^ ]+) and put \2\1 in the Replace: field:

(re) is the first group
([^ ]+) is the second group

\2\1 will thus put the second group before the first group.

The term "regular" matches the pattern: (re) matches re and ([^ ]+) matches gular. The replaced string will thus be "gularre".

→ search for (re)([^ ]+) and put \1\1_\[\2\] in the Replace: field:


(re) is the first group
([^ ]+) is the second group

\1\1_\[\2\] will put 2 instances of the first group, then an underbar, then [, then the second group, then ].

In the case of "regular", we'd have the following replacement string:
rere_[gular]

That's only the beginning...

What you need to check now is the special effects of some characters. If you've used Textwrangler it is all in the user manual, page 133, Chapter 8 (Searching with Grep)**, or you can call the Help with Cmd+? and you'll find a relevant link right away.

Textwrangler's regex is pretty standard so once you're used to it there, you can use it in other editors too. If what works in Textwrangler does not work there, check the idiosyncrasies of the editor you use.

Now, take a real world document and try to transform it by using a few regular expressions. A typical use case for a translator would be to convert a TMX file into a 2 column tab separated data set, or the opposite: to convert a 2 column tab separated data sets into a TMX file. If you manage to do that you've created your first alignment based TMX converter!

---

* I try to use or discuss free software when possible because I think that is the way to go. People who want to use a free text editor on the Mac can use Aquamacs. It comes with all the goodness of emacs (including the same regular expressions) and looks and feels a lot like a "normal" Mac text editor.
** [2018 Update] In the BBEdit manual, Chapter 8 is on page 165.

Regular expressions and text editing

Regular expressions

When you edit glossaries or translation memories a few regular expressions always come in handy.

Regular expressions are pattern matching expressions. You create a pattern in the search field of a text editor and the text editor will look for anything that matches the pattern. Similarly, once you have found the pattern, you can replace it with a second pattern. That way, you end up with super powerful search-replace routines that can save you hours of stress on thorny texts...

Here are two articles that you can use to get up to speed on the topic:

• Regular Expressions from Princeton University
• Regular Expressions Unfettered from Apple Developer Connection

As you can see, regular expression creation is not always an easy task and a helper application can sometimes save you a lot of time.

There are a few free regexp testers for OSX but they basically all work from outside your text editor.

The one that seems to be the easiest to use is reggy, a nice piece of software licensed under the GNU General Public License v2.

On his blog, Bill Clementson talks about regex-tool.el. As the name indicates, regex-tool.el is an elisp library for Emacs

Besides for text editors, regular expressions are supported by all kinds of software. Including, of course, the major Office applications on the market. Look at their user manuals to find more information about regexp handling there.

Text editing

A number of very good text editors are available for the Mac. TextEdit, OSX's default text editing application, is fine but lacks even basic regular expressions support. It does plenty of other things though, and can be considered as a simple word processor with most of what is needed in that field (read and write Word files etc).

Others major text editors on Mac include:
• Smultron (free as in "speech"),
• TextWrangler (free as in "beer"),
• SubEthaEdit (not free, except for the old version),
• BBEdit (not free at all) and
• TextMate (not free either and with very bad multibyte characters support).

There are also the "ancestors" that are VI (VIM) and Emacs. Both are available from the Terminal application but require some time to get used to. Still, they are definitely some of the most powerful application OSX hosts...

Emacs is not exactly the text editor that I'd recommend to my wife. But Aquamacs, a "Mac" version in terms of interface is much friendlier and can almost right away be used as a replacement for TextEdit as far as, well, text editing is concerned. Being an adaptation of Emacs, it is just as free and also distributed under the GNU General Public License...

Emacs is written in Elisp. So anything you write in Elisp within Emacs can de-facto extend the functionality of Emacs. In other words, Emacs is just a huge macro editing environment...

Whichever text editor you decide to use, don't forget to read the user manual and especially the "searches" and "regular expressions" chapters.