emacs regex with emacs lisp

June 28 update: using \#1 instead of (string-to-number \1)

A reader on reddit mentionned that the manual also had the "\#d" construct to replace the often used (string-to-number \d) function.

That regex-replace improvement was mentionned in Stevve Yegge's emacs 22 introduction, back in 2006.

Last but not least, I noticed that the whole post was originally written with (string-as-number ...) when the correct function name is (string-to-number ...)

---

Not strictly related to translation but here is what's happening...

I've resumed studies last year, trying to finish an MA in Japan Studies I started 25 years ago.

For the first year, I only have to write a 30ish pages dissertation on my subject (representation of women in kendo magazines in Japan) and I decided to go the emacs + org-mode way, with the easy export to ODF function that's packaged with the thing.

So I decided to write each chapter in a different org file, and send them one by one to my director. But then, for the final delivery I needed to put all that in one big file and was faced with the fact that all my footnotes would need to be re-indexed manually because each file had notes starting at 1...

I usually use BBedit for any serious regex work. Mostly because the interface is clearer than emacs, and the regexp feels more modern (\d vs [:digit:])

But one thing you can't do in BBEdit is to send commands to the replace string. For ex, in my case, add 21 to the matching number, which seems pretty trivial, when you think of it, but doing that will involve other technologies, like using perl or some other command line thing.

In emacs, however, everything can be interpreted as an expression, hence you can insert code wherever you want and get the result from that code right in the document.

The org-mode footnotes all look like [fn:12], where "12" is the note number that I need to replace with an incremented number. Since there are no instances of fn:\d+ without the brackets that are not footnotes, I figured I could just be searching for that string:

fn:\([:digit:]+\)

Notice that in emacs, "(" and ")" need to be escaped, also I could have used the [0-9] class.

In BBedit I'd just need:

fn:(\d+)

And now I need to replace that with the expression that will add 21 to the number.

In BBEdit, I'd be stuck here. I just can't add anything to a match. In emacs, I can replace the match with that:

fn:\,(+ 21 (string-to-number \1))

The emacs lisp expression is "(+ 21 (string-to-number \1))", which means "convert the \1 match that is a string into its numerical value and add 21 to it".

But, wasn't \1 supposed to match [0-9]+, which is a number? Well, yes, but really it's just digits, hence strings, that have no numerical value whatsoever, so first, we need the expression to convert them to a numerical value before adding 21 to them.

Now, the trick is to have the expression be handled as an operation and not as an arbitrary string, and that's where the "\," prefix comes into play.

"\," tells the replace engine that the string that follows must be interpreted as an emacs lisp expression and not as a mere string. With it, the regexp replaces properly adds 21 to my note numbers, and I get two dozen footnotes updated in one fell swoop...

I love BBEdit and its people, but emacs is really a gift that keeps on giving.

---

Here are some handy references:

the emacs regexp-replace function

Regexp Replacement

the emacs regexp syntax

Syntax of Regular Expressions

the emacs-lisp string-to-number function

Conversion of Characters and Strings

And here is a really super short introduction to lisp syntax

There is no real need to have a very deep understanding of emacs lisp to use this regexp-replace function. Just remember that a lisp expression generally looks like this:

(operator operands)

Where the operator is a generally a function, like + or string-to-number above, and the operands can be any expression that is accepted by the operator. So, here:

(+ 21 (string-to-number \1))

means:

add 21 to the result of the expression (string-to-number \1)

with (string-to-number \1) meaning:

convert the string matched by \1 to its numeric value

Obviously, if \1 is not a string, the conversion will fail and the addition won't work. And without that conversion, if we had just added \1 as a string, the addition that expects numbers as operands would have failed.

I just realized that this is my first emacs lisp related post ever ! I'd like to thank that person I met in Tokyo about 15 years ago who showed me the way. It's an egg that definitely took some time to hatch...

Searching for empty translations in OmegaT

OmegaT searches are very powerful. One feature OmegaT does not have (yet?) is the ability to register common searches for later use.
There are at least 2 RFEs for that on the OmegaT development site, one registered in 2006 (by me) and one in 2014.

In the meanwhile, there are at least 2 searches that you want to remember:

1. searches for non translated segments
2. searches for segments that have been set to <EMPTY> (they are translated but the translation being empty, the source won't appear in the target document).

Searches for non translated segments

The logic is straightforward:

You want to search for any one character in source that is untranslated.

In a search, "any one character" can be expressed by the regular expression "." (period). So you put a period in the search field and you make sure you have selected "Regular expressions" below it. That "any one character" is to be found in "source", so you uncheck anything else but "source" in the line below that. Then you specify "Untranslated", since that's exactly what your looking for, and you can eventually select "all matching segments" to make sure non unique segments are all displayed.

The search scope will be "Project" and only "Memory".

Searches for <EMPTY> segments

In OmegaT, when a segment is left untranslated, the source text appears in the target document. An easy way to get around this is to translate the segment with a space. The problem is that sometimes you really want to not use anything in target. For this, OmegaT allows you to "Set [an] empty translation" in the Edit menu. Once you do that, the segment will be empty but translated (it appears in the same color as the other translated segments when you select "Mark translated segments" in the View menu), and OmegaT will display it with the <EMPTY> string when you've left it.

To search for such segments, the logic is a bit different:

You want to search for an empty string in target that is translated.

In a search, "an empty string" can be expressed by the regular expression "^$" (caret, followed by dollar). The ^ stands for the beginning of the line and the $ for its end, with nothing in between the string is empty. So you put a caret followed by a dollar sign in the search field and you make sure you have selected "Regular expressions" below it. That "empty string" is to be found in "target", so you uncheck anything else but "target" in the line below that. Then you specify "Translated", since that's exactly what your looking for, and you can eventually select "all matching segments" to make sure non unique segments are all displayed.

The search scope will be "Project" and only "Memory".

Introduction to regular expressions

[2018 update]

Textwrangler has now been replaced by BBEdit that offers a free trial and then turns into a BBEdit "free mode". BBEdit "free mode" is a better TextWrangler and also happens to be a 64 bits application compatible with High Sierra.
https://www.barebones.com/products/bbedit/comparison.html

I mention Aquamacs at the end of the article. Regular expressions for Aquamacs but also for Emacs are documented in the Emacs manual available here:
https://www.gnu.org/software/emacs/manual/html_node/emacs/Regexps.html

Check this article too:
https://www.johndcook.com/blog/2018/01/27/emacs-features-that-use-regular-expressions/

In OmegaT, you lose a lot if you're not familiar with regular expressions. OmegaT uses the Java flavor of regular expressions:
https://omegat.sourceforge.io/manual-standard/en/appendix.regexp.html

OmegaT allows you to create "tags" that can be protected and checked during the translation by using regular expressions. In a recent project I had created a 872 characters long regular expression that described 71 different tags.

As of February 2018, OmegaT also allows for replacements with capture groups as described below:
https://sourceforge.net/p/omegat/feature-requests/953/

And you can also use regular expressions to search for empty segments, as I document in this article:
https://mac4translators.blogspot.com/2018/03/searching-for-empty-translations-in.html

---

The technology that had the most impact on my workflow is definitely "regular expressions".

I discovered them at the end of the 90' when I was working on the conversion of a database output to a set of about 6000 static HTML pages. At the time, the editor of choice on the Mac was BBEdit from Barebones Software, but its free and "lite" version "BBEdit Lite" was also immensely popular. BBedit Lite has now been replaced by Textwrangler and just like its predecessor, Textwrangler can be used without paying a user license fee*.

What are regular expressions?

Regular expressions are a "search" function on steroids. Regular expressions were created to find patterns in strings. They can find simple patterns like the word "pattern" in this text, or more complex patterns like "a string that starts with 'pa', followed by a letter that's repeated twice, followed by any three characters that are neither 'space' nor '@' or '^' and followed by a space".

This document uses its first two paragraphs (the paragraphs in italics, above) as a test ground. Paste that paragraph in your favorite regular expressions supporting text editor (I use Textwrangler for all the descriptions so you might want to use it too) then call the search window, check the "grep" box at its bottom and search for:

re[^ ]*

You should see colors appearing while you type the search terms.

What that expression means is:
r followed by e followed by a group of characters that are not a space, or by nothing.

Hit Next and see what you get, then hit Cmd+G and see what you get. If you start from the top of the paragraph, you should have 8 "matches".

Normal characters

Most characters represent themselves in regular expressions (regex), like a "normal" search.

→ r means r and e means e, " " means a space. In the same sequence. No magic here.

Special effects

Some characters have special effects:

→ [ starts a group of characters
→ ] ends that group
→ ^ means "not"
→ * means "zero or more of what just came"

So, our above simple regular expression re[^ ]* means:

"look for any string that has a r followed by a e followed by zero or more characters that are not a space."

Now, what if you need to find characters like ^, [, ] or *?

Cancelling special effects

When you want to find characters that have a special effect without "triggering" that special effect, you put a "\" in front of them:

→ \* means the character *
→ \[ means the character [


And since the character "\" has the special effect of removing the special effect of a character that has a special effect... then:

→ \\ means the character \

etc.

By the way, the character . has the special effect of matching "any one character" so if you're looking for a period, then you really want to look for the \. string...

Examples:

The regular expression ". " (. followed by space) will match any one character followed by a space. There are 78 strings that match this pattern in the paragraph.

The regular expression "\. " (\ followed by . followed by space) will match any period followed by a space. There are only 2 strings that match this pattern in the paragraph.

The regular expressions "re*." (re followed by * followed by .  will match any string that is composed of a r followed by zero or more e, followed by any one character. There are 22 matches in the paragraph. Verify that you understand them all.

The regular expression ".e\*\." (. followed by e followed by \ followed by * followed by \ followed by .) will match the 4 characters string ee*. that you find at the end of the paragraph.

Triggering special effects

Some characters work the other way round: by themselves they do not have a special effect but if you stick the \ character before them, then their special effect is triggered.

→ t means t but \t means tabulation
→ r means r but \r means line break (specifically "carriage return")
→ s means s but \s means all sorts of white space, which includes spaces, tabulations, line breaks, etc.

If the character does not have a special effect then using \ has no effect.

→ i means i and \i too means i

Such sequences (\ followed by a character) are usually called "escape sequences".

Remembering matches

If you want to "memorize" a match, for later use in the expression or in the "Replace:" field, then you put the corresponding expressions between parenthesis:

→ (re)[^ ]+ will produce the same matches as above, but will memorize the re part and not the rest.

→ re([^ ]+) will produce the same matches as above, but will not memorize the re part and instead will memorize the rest.

→ (re)([^ ]+) will produce the same matches as above and will memorize the 2 parts separately.

Using memorized matches

Now that the matches are remembered, you can use them. Use \1 to refer to the first memorized string, \2 to refer to the second memorized string, etc...

→ (e)\1\*\. will match the "ee*." string that you find at the end of the text.

→ search for (re)([^ ]+) and put \2\1 in the Replace: field:

(re) is the first group
([^ ]+) is the second group

\2\1 will thus put the second group before the first group.

The term "regular" matches the pattern: (re) matches re and ([^ ]+) matches gular. The replaced string will thus be "gularre".

→ search for (re)([^ ]+) and put \1\1_\[\2\] in the Replace: field:


(re) is the first group
([^ ]+) is the second group

\1\1_\[\2\] will put 2 instances of the first group, then an underbar, then [, then the second group, then ].

In the case of "regular", we'd have the following replacement string:
rere_[gular]

That's only the beginning...

What you need to check now is the special effects of some characters. If you've used Textwrangler it is all in the user manual, page 133, Chapter 8 (Searching with Grep)**, or you can call the Help with Cmd+? and you'll find a relevant link right away.

Textwrangler's regex is pretty standard so once you're used to it there, you can use it in other editors too. If what works in Textwrangler does not work there, check the idiosyncrasies of the editor you use.

Now, take a real world document and try to transform it by using a few regular expressions. A typical use case for a translator would be to convert a TMX file into a 2 column tab separated data set, or the opposite: to convert a 2 column tab separated data sets into a TMX file. If you manage to do that you've created your first alignment based TMX converter!

---

* I try to use or discuss free software when possible because I think that is the way to go. People who want to use a free text editor on the Mac can use Aquamacs. It comes with all the goodness of emacs (including the same regular expressions) and looks and feels a lot like a "normal" Mac text editor.
** [2018 Update] In the BBEdit manual, Chapter 8 is on page 165.

Regular expressions and text editing

Regular expressions

When you edit glossaries or translation memories a few regular expressions always come in handy.

Regular expressions are pattern matching expressions. You create a pattern in the search field of a text editor and the text editor will look for anything that matches the pattern. Similarly, once you have found the pattern, you can replace it with a second pattern. That way, you end up with super powerful search-replace routines that can save you hours of stress on thorny texts...

Here are two articles that you can use to get up to speed on the topic:

• Regular Expressions from Princeton University
• Regular Expressions Unfettered from Apple Developer Connection

As you can see, regular expression creation is not always an easy task and a helper application can sometimes save you a lot of time.

There are a few free regexp testers for OSX but they basically all work from outside your text editor.

The one that seems to be the easiest to use is reggy, a nice piece of software licensed under the GNU General Public License v2.

On his blog, Bill Clementson talks about regex-tool.el. As the name indicates, regex-tool.el is an elisp library for Emacs

Besides for text editors, regular expressions are supported by all kinds of software. Including, of course, the major Office applications on the market. Look at their user manuals to find more information about regexp handling there.

Text editing

A number of very good text editors are available for the Mac. TextEdit, OSX's default text editing application, is fine but lacks even basic regular expressions support. It does plenty of other things though, and can be considered as a simple word processor with most of what is needed in that field (read and write Word files etc).

Others major text editors on Mac include:
• Smultron (free as in "speech"),
• TextWrangler (free as in "beer"),
• SubEthaEdit (not free, except for the old version),
• BBEdit (not free at all) and
• TextMate (not free either and with very bad multibyte characters support).

There are also the "ancestors" that are VI (VIM) and Emacs. Both are available from the Terminal application but require some time to get used to. Still, they are definitely some of the most powerful application OSX hosts...

Emacs is not exactly the text editor that I'd recommend to my wife. But Aquamacs, a "Mac" version in terms of interface is much friendlier and can almost right away be used as a replacement for TextEdit as far as, well, text editing is concerned. Being an adaptation of Emacs, it is just as free and also distributed under the GNU General Public License...

Emacs is written in Elisp. So anything you write in Elisp within Emacs can de-facto extend the functionality of Emacs. In other words, Emacs is just a huge macro editing environment...

Whichever text editor you decide to use, don't forget to read the user manual and especially the "searches" and "regular expressions" chapters.