This is monotone.info, produced by makeinfo version 5.2 from
monotone.texi.

INFO-DIR-SECTION Programming
START-INFO-DIR-ENTRY
* monotone: (monotone).         Monotone version control system
END-INFO-DIR-ENTRY


File: monotone.info,  Node: VCS,  Prev: Automation,  Up: Command Reference

4.12 VCS
========

'mtn cvs_import PATHNAME'

     This command imports all the file versions in each RCS file found
     in the tree of files starting at PATHNAME, then reconstructs the
     tree-wide history of logical changes by comparing RCS time stamps
     and change log entries.  For each logical tree-wide change,
     monotone synthesizes a manifest and revision, and commits them
     (along with all associated file deltas) to your database.  It also
     copies all change log entries, author identifiers, and date stamps
     to manifest certificates.

     In normal use, PATHNAME will be a CVS module, though it is possible
     to point it at a directory within a module as well.  Whatever
     directory you point it at will become the root of monotone's
     version of the tree.

'mtn git_export [--authors-file=AUTHORS-FILE] [--branches-file=BRANCHES-FILE]'
'[--import-marks=MARKS-FILE] [--export-marks=MARKS-FILE]'
'[--log-revids] [--log-certs] [--use-one-changelog]'
'[--refs=TYPE]'

     This command exports all revisions from the current monotone
     database to 'stdout' in a format that can be piped directly to git
     fast-import.

     The '--authors-file' option may be used to map monotone author and
     committer names to different values in a manner similar to that
     documented in git-svn(1).  Mappings are specified in the file as:
     loginname = Joe User <user@example.com>

     The list of authors that might need to be mapped can be extracted
     from a monotone database with the following sql query:

     $ mtn db execute 'select distinct value from revision_certs where name = "author"'

     The list of committers that might need to be mapped can be
     extracted from a monotone database with the following sql query:

     $ mtn db execute 'select distinct public_keys.name
                       from public_keys
                       left join revision_certs on revision_certs.keypair_id = public_keys.id
                       where revision_certs.name = "author"'

     The '--branches-file' option may be used to map monotone branch
     names to different values.  This may be required as monotone allows
     branch names that are not valid according to git.  Branch mappings
     are specified in the BRANCHES-FILE as:

     monotone-branch-name = git-branch-name

     Revisions with no author cert will use "Unknown <unknown>" for both
     the author and the committer.  These can be mapped to other values
     using the AUTHORS-FILE option.

     The list of branches that might need to be mapped can be extracted
     from a monotone database with using the 'ls branches' command:

     $ mtn ls branches --ignore-suspend-certs

     The '--import-marks' and '--export-marks' options are similar to
     those documented in git-fast-export(1) and git-fast-import(1).
     These may be used for incremental exports and may also be useful
     for repository verification.  The MARKS-FILE is read on startup if
     '--import-marks' is specified and all marked revs are excluded from
     the export.  The MARKS-FILE is written on completion if
     '--export-marks' is specified and will contain marks for all revs
     that were exported in addition to any marks that were read on
     startup.  It is safe to use the same file for both '--import-marks'
     and '--export-marks' but different files may also be used.

     The original monotone revision ids may be included in the exported
     git commit messages using the '--log-revids' option.  These will
     appear as:

     Monotone-Parent: ...
     Monotone-Revision: ...

     in the git commit messages and may be useful for repository
     verification.  Merge revisions with two parents will include two
     'Monotone-Parent' lines.

     The original monotone values for 'author', 'date', 'branch' and
     'tag' certificates may be included in the exported git commit
     messages using the '--log-certs' option.  These will appear as:

     Monotone-Author: ...
     Monotone-Date: ...
     Monotone-Branch: ...
     Monotone-Tag: ...

     in the git commit messages and may be useful for repository
     verification or maintaining information that is otherwise not
     maintained by git.  In particular monotone may have several values
     for each of these certs and git only represents a single 'author'
     and 'date' value.

     By default, the values of all changelog and comment certs on a
     revision are concatenated to form the git commit message for that
     revision.  The '--use-one-changelog' option will prevent this
     behaviour and use the value from the first changelog cert
     encountered.  _WARNING:_ this will explicitly lose information in
     the exported data.  Changelog certs have no implicit ordering, one
     will be selected arbitrarily to be used as the git commit message
     and all others will be excluded from the exported data.
     Additionally, all comment certs will be excluded from the exported
     data.

     References to the original monotone branches and tags are exported
     as 'refs/tags/<tag-name>' and 'refs/heads/<branch-name>'.  In
     addition, references to the original monotone revision ids, the
     root revisions and the leaf revisions may be exported using
     '--refs=revs', '--refs=roots' and '--refs=leaves' respectively.
     These references are exported as 'refs/mtn/revs/<id>',
     'refs/mtn/roots/<id>' and 'refs/mtn/leaves/<id>' and may be useful
     for repository verification.  These additional references should
     probably not be maintained after the exported repository has been
     verified as they cause considerable clutter in tools like gitk.


File: monotone.info,  Node: Formats,  Next: Lua Reference,  Prev: Command Reference,  Up: Top

5 Formats
*********

Monotone uses three formats to output or to accept input of data (mostly
structured), 'basic_io', 'stdio', and 'packet'.  'basic_io' is often
used to represent data or for configuration, while 'stdio' is used for
communication with 'automate stdio'.

   The 'stdio' format is not described here.  For information on it, see
'*note mtn automate stdio::'.

   The 'packet' format is not described here; see *note Packet I/O::.
For examples, see '*note mtn automate packet_for_fdelta::', '*note mtn
automate packet_for_fdata::'.

5.1 'basic_io' Format
=====================

'basic_io' is a very simple format, yet expressive, and can be described
with the following ABNF (see RFC 2234) specification:

     basic_io = *( item )

         item = *sp symbol *sp *( string / hexid ) *sp
                ; a symbol followed by zero or more strings and/or hexes.

       symbol = 1*( lcalpha / "_" )
                ; One or more letters or underscores.
       string = DQUOTE *( charnqnb / "\" "\" / "\" DQUOTE ) DQUOTE
                ; A series of any character, enclosed with double quotes
                ; To include a double quote or a backslash in the string,
                ; escape it with a backslash.
        hexid = "[" 40HEXDIG "]"
                ; 40 hexadecimal characters enclosed in brackets.
                ; Note: the amount might change as we move away from sha1.

           sp = ( WSP CR LF )
     charnqnb = %x01-21 / %x23-5B / %x5D-FF
                ; All characters except NUL (%x00) , " (%x22) and \ (%x5C).
      lcalpha = %x61-7A
                ; Lower case a-z.

   In essence, it's a stream of items, where each item is a 'symbol'
followed by zero or more 'string's and 'hex'es.

   There is an idea of stanzas, which is a group of items that logically
belongs together.  However, stanzas aren't defined in the 'basic_io'
syntax, and are instead defined in terms of order and sequence for what
symbols may be part of the stream, how they may appear, and how many and
what arguments should follow them.  This is defined by each command or
program using this format.

   For example, revisions have associated data; the structure is header
stanzas followed by a list of changes vs.  each parent revision, with
each change represented by a separate stanza; reordering the stanzas
could cause a particular change to be interpreted against the wrong
parent.

   Another example is the 'read-permissions' file that's read by the
default definition of the *note get_netsync_read_permitted:: Lua hook.

   Monotone will format its output in 'basic_io' as follows:
   * Each 'symbol' begins a new line.
   * Tokens on the same line are separated by a single space.
   * Lines are grouped into logical 'stanza's, separated by a single
     empty line.  Each stanza then describes a particular item, with the
     different lines ('symbol's followed by 'string' and 'hex' values)
     describing different aspects of that item.
   * 'symbol's in a stanza are right-aligned with each other; each
     'symbol' other than the longest is preceeded by spaces.  The
     longest 'symbol' in a stanza is not preceeded by spaces, even if
     there are longer 'symbol's in other stanzas.
   * Lines within stanzas have a consistent order, although some may be
     optional in some commands.


File: monotone.info,  Node: Lua Reference,  Next: Special Topics,  Prev: Formats,  Up: Top

6 Lua Reference
***************

Monotone makes use of the Lua (http://www.lua.org) programming language
to customize and extend its behaviour.  By writing "functions" which are
loaded and evaluated at runtime, you can help monotone to make a
particular decision, set a suitable default or preference or perform a
certain action.

   Lua functions are defined in "rcfiles" which will be read every time
monotone runs.  'rcfiles' consist of:

   * '$HOME/.monotone/monotonerc' on Unix or
     '%APPDATA%\monotone\monotonerc' on Windows

   * '_MTN/monotonerc' in the current workspace

   * Files given in the '--rcfile=FILE' option on the command line.

   * All files in a directory given by '--rcfile=DIRECTORY' on the
     command line.

   The files are loaded in the order '.monotone/monotonerc',
'_MTN/monotonerc', '--rcfile' in the command line order.  Definitions
loaded later shadow (override) earlier definitions.

   There are two uses for Lua functions; hooks and user-defined
commands.  This section documents hooks; see '*note register_command::'
for user-defined commands.  The source distribution contains some
example user commands in the 'contrib/command' directory.

   Hooks are Lua functions that are called from monotone code in many
places.  Monotone provides default definitions for some hooks; see *note
Default hooks:: for their complete source.  For other hooks, if no
definition is provided, a default return value is used.  When writing
new hooks, it may be helpful to reuse some code from the default ones.
Since Lua is a lexically scoped language with closures, this can be
achieved with the following code:

     do
         local old_hook = default_hook
         function default_hook(arg)
             if not old_hook(arg) then
                 -- do other stuff
             end
         end
     end

   Now the default hook is trapped in a variable local to this block,
and can only be seen by the new hook.  Since in Lua functions default to
the global scope, the new hook is seen from inside monotone.

   Monotone also provides a number of helper functions to hook writers
exposing functionality not available with standard Lua.

* Menu:

* Hooks::                       All hooks called by monotone.
* Additional Lua Functions::    Extra functionality available to hook writers.
* Implementation Differences::  Functional differences from standard Lua


File: monotone.info,  Node: Hooks,  Next: Additional Lua Functions,  Prev: Lua Reference,  Up: Lua Reference

6.1 Hooks
=========

This section documents the existing hook functions and their default
definitions.

* Menu:

* Common Data Types::
* Event Notifications and Triggers::
* User Defaults::
* Netsync Permission Hooks::
* Netsync Transport Hooks::
* Trust Evaluation Hooks::
* External Diff Tools::
* External Merge Tools::
* Selector Expansion::
* Attribute Handling::
* GIT Export Hooks::
* Validation Hooks::


File: monotone.info,  Node: Common Data Types,  Next: Event Notifications and Triggers,  Prev: Hooks,  Up: Hooks

6.1.1 Common Data Types
-----------------------

Some hooks take arguments which are more complex than a simple string or
number (they are a Lua "table").  Where multiple hooks take the same
kind of argument, we generally try to make them take that kind of
argument in the same format, described here.

'key_identity'

          {
             id = "key hash/fingerprint, in hex",
             given_name = "name given when creating the key",
             name = "local alias of the key"
          }

     When a key is created, it is given a meaningful name.  Because this
     is chosen by the key creator and may be unreliable, trust hooks
     need to be able to look at the key hash as well.  This may be
     inconvenient, so a local alias is also provided.  This alias is
     obtained from a Lua hook, which can provide reliable easy-to-use
     names based on the key hash (or however else you want).


File: monotone.info,  Node: Event Notifications and Triggers,  Next: User Defaults,  Prev: Common Data Types,  Up: Hooks

6.1.2 Event Notifications and Triggers
--------------------------------------

There are a number of hooks that are called when noteworthy events
occur, such as commits or new revisions arriving over the network.
These hooks can be used to feed the events into external notification
systems, such as generating email.

   By default, these hooks are undefined, so no special external actions
are taken.

   Hooks are presented not in alphabetic order; they are grouped by
function, and then by the order they are typically called by monotone.

'note_commit (NEW_ID, REVISION, CERTS)'

     Called by monotone after the revsion NEW_ID is committed.  REVISION
     is the text of the revision, what would be given by 'mtn automate
     get_revision NEW_ID'.  CERTS is a Lua table containing the set of
     certificate names and values committed along with this version.

     There is no default definition for this hook.

     Note that since the CERTS table does not contain cryptographic or
     trust information, and only contains one entry per cert name, it is
     an incomplete source of information about the committed version.
     This hook is only intended as an aid for integrating monotone with
     informal commit-notification systems such as mailing lists or news
     services.  It should not perform any security-critical operations.

'note_netsync_start (SESSION_ID, MY_ROLE, SYNC_TYPE, REMOTE_HOST, REMOTE_KEY, INCLUDES, EXCLUDES)'

     Called by monotone just after a netsync session is started, before
     any of the other netsync notification hooks are called.

     The arguments are:

     SESSION_ID
          Identifies the current netsync session in case several are
          happening at the same time (only possible on a server).

     MY_ROLE

          One of "client" or "server".

     SYNC_TYPE

          One of "sync", "push", or "pull".

     REMOTE_HOST

          The network address of the remote host.  At the client, this
          will be the name it was told to connect to; at the server,
          this will use the numerical IP address the connection was
          received from.

     REMOTE_KEY

          The identity of the key being used by the other end of the
          connection.  The fields may be empty at the server if the key
          used by the client is not present at the server.

     INCLUDES
     EXCLUDES

          The include and exclude patterns used by the client.

'note_netsync_revision_received (NEW_ID, REVISION, CERTS, SESSION_ID)'
'note_netsync_revision_sent (REV_ID, REVISION, CERTS, SESSION_ID)'

     Called after the revision NEW_ID is received or sent through
     netsync.

     There are no default definitions for these hooks.

     Arguments:

     NEW_ID
          The revision id.

     REVISION
          The text of the revision; what would be given by 'mtn automate
          get_revision NEW_ID'.

     CERTS
          A Lua table containing one subtable for each cert attached to
          the revision NEW_ID.  These subtables have fields named "key",
          "name", and "value", containing the identity of the signing
          key for the cert, the name of the cert, and the value of the
          cert.

     SESSION_ID
          Identifies the current netsync session.

'note_netsync_cert_received (REV_ID, KEY_IDENTITY, NAME, VALUE, SESSION_ID)'
'note_netsync_cert_sent (REV_ID, KEY_IDENTITY, NAME, VALUE, SESSION_ID)'

     Called by monotone after a cert is received (or sent) through
     netsync, if the revision that the cert is attached to was not also
     received (or sent) in the same netsync operation.

     There is no default definition for this hook.

     Arguments:

     REV_ID
          The revision id that the cert is attached to.

     KEY_IDENTITY
          The key that the cert is signed with; see *NOTE
          KEY_IDENTITY::.

     NAME
          The name of the cert.

     VALUE
          The cert value.

     SESSION_ID
          Identifies the netsync session.

'note_netsync_pubkey_received (KEY_IDENTITY, SESSION_ID)'
'note_netsync_pubkey_sent (KEY_IDENTITY, SESSION_ID)'

     Called by monotone after a pubkey is received or sent through
     netsync.

     There is no default definition for this hook.

     Arguments:

     KEY_IDENTITY
          The identity of the key received; see *NOTE KEY_IDENTITY::.

     SESSION_ID
          Identifies the current netsync session.

'note_netsync_end (SESSION_ID, STATUS, BYTES_IN, BYTES_OUT, CERTS_IN, CERTS_OUT, REVS_IN, REVS_OUT, KEYS_IN, KEYS_OUT)'

     Called by monotone after a netsync session ends.  This hook would
     usually be used for post-netsync purposes, like collecting all the
     data from all other netsync notification hooks, make some nice
     output from them and finally send the result somewhere.  It could
     also be used to prepare parallel databases with all the data to be
     displayed through something like viewmtn.

     Arguments:

     SESSION_ID
          Identifies the current netsync session.

     STATUS
          A three digit integer that tells whether there was an error,
          and if so what kind of error it was:

          *200*
               No error, connection successful.

          *211*
               The connection was interrupted after some data may have
               been transferred.

          *212*
               The connection was interrupted before any data could be
               transferred.

          *412*
               The request is not permitted.

          *422*
               The client tried to use a key that the server doesn't
               know about.

          *432*
               The client and server have different epochs for a branch.

          *512*
               Protocol error (source/sink confusion).

          *521*
               Protocol error (packet received at a time when it doesn't
               make sense).

          *532*
               The client did not identify itself correctly.  (Possible
               replay attack?)

          In general, 2xx means there was no error, 4xx means there was
          a permissions error, and 5xx means there was a protocol error.
          xx1 means some data may have been transferred, xx2 means no
          data was transferred, and xx0 means all data was transferred.

     BYTES_IN
     BYTES_OUT
          The number of bytes received/sent during the session.

     CERTS_IN
     CERTS_OUT
          The number of certs received/sent during the session.

     REVS_IN
     REVS_OUT
          The number of revisions received/sent during the session.

     KEYS_IN
     KEYS_OUT
          The number of keys received/sent during the session.

'note_mtn_startup (...)'

     Called by monotone when it is first started, after all command
     completion and option processing, before the command starts
     executing.

     There is no default definition of this hook.

     One use of this hook is to monitor usage of monotone, for user
     interface testing.

     The arguments to the hook are the command line arguments to
     monotone, without the initial 'mtn' command.  They can be accessed
     through the lua ARG variable as in this example:

          function note_mtn_startup(...)
              print("Beginning note_mtn_startup")
              for i = 1,arg.n do
                  print(arg[i])
              end
              print("Ending note_mtn_startup")
          end


File: monotone.info,  Node: User Defaults,  Next: Netsync Permission Hooks,  Prev: Event Notifications and Triggers,  Up: Hooks

6.1.3 User Defaults
-------------------

These are hooks that can be used to provide smart, context-sensitive
default values for a number of parameters the user might otherwise be
prompted for.

'get_branch_key (BRANCHNAME)'

     Called whenever monotone needs a key to sign a certificate, and
     '*note --key::' was not given.

     Returns a string which is the name or hash of a private key used to
     sign certificates in a particular branch BRANCHNAME, or nil for no
     key.

     See '*note --key::' for a description of how monotone determines
     what key to use.

     There is no default definition for this hook; it returns nil.

'get_netsync_client_key(SERVER, INCLUDE, EXCLUDE)'

     Called by the client when a netsync connection is being
     established, and '*note --key::' was not given.  Returns a string
     which is the name or hash of the key to use to authenticate the
     client side of a netsync connection.

     Note that netsync commands do not need a signing key; they only
     transmit already signed information.

     See '*note --key::' for a discussion of how monotone determines
     what key to use.

     Arguments, when called:

     SERVER
          The scheme, user, host, port, and path fields from the URI
          provided on the command line.  See *note netsync uri::.

     INCLUDE
          The include pattern in the URI provided on the command line.

     EXCLUDE
          The exclude pattern in the URI provided on the command line.

'get_netsync_server_key(ADDRESSES)'

     Called by the server when a new netsync server instance is created,
     and '*note --key::' was not given.  Returns a string which is the
     name or hash of the key to use to authenticate the server side of a
     netsync connection.

     Note that netsync commands do not need a signing key; they only
     transmit already signed information.

     See '*note --key::' for a discussion of how monotone determines
     what key to use.

     Arguments, when called:

     ADDRESSES
          A table of addresses given to monotone via the '--bind' option
          that denote the addresses and / or ports monotone is listening
          on for connections.  If the address in one of the table
          entries is omitted, the port must be given with a leading
          colon; in this case monotone listens on all interfaces on this
          port.

'get_default_command_options(COMMAND)'
     Called after a command is completed, before the workspace options
     are loaded, and before the command line options are processed.

     Returns a table of options.  The options must be valid for the
     given command or global options.  The returned option values
     overwrite the standard default options values; workspace and
     command line options override the new defaults as usual.

     The argument is a table containing the command keywords, indexed by
     the integer position of the keyword in the command.  For example,
     'mtn list branches' passes a table '(1 => "list", 2 =>
     "branches")'.

     The default definition of this hook returns an empty table.

     Simple example which enables recursive directory scanning for 'mtn
     add' by default:
          function get_default_command_options(command)
              local default_options = {}
              if (command[1] == "add") then
                  table.insert(default_options, "--recursive")
              end
              return default_options
          end

'get_passphrase (KEY_IDENTITY)'
     Called whenever monotone needs to use a private key.

     Returns a string which must match the passphrase used to encrypt
     the private *NOTE KEY_IDENTITY:: in your key store.

     This hook has no default definition.

     If this hook is not defined or returns false, and ssh keys are not
     enabled (see '*note mtn ssh_agent_export::'), monotone will prompt
     you for a passphrase each time it needs to use a private key.

'get_local_key_name (KEY_IDENTITY)'

     Called whenever monotone retrieves a key from the keystore or
     database, or completes a user-provided key, to provide the local
     alias (the NAME field in *NOTE KEY_IDENTITY::).

     Returns the local alias for the given KEY_IDENTITY.  The ID and
     GIVEN_NAME fields of KEY_IDENTITY will be populated, and the NAME
     field will not be.  The return value indicates what the NAME field
     should contain.

     The default implementation of this hook returns GIVEN_NAME.

'get_author (BRANCHNAME, KEY_IDENTITY)'

     Called by any command that needs an author name for an author cert
     for a commit, if '*note --author::' is not given.

     Returns a string which is the author name.  If it returns nil, the
     key local name (as provided by '*note get_local_key_name::') is
     used for the author name.

     BRANCHNAME is the branch for the commit, *NOTE KEY_IDENTITY:: is
     the key.

     There is no default definition for this hook; it returns nil.

     Example definitions:
          function get_author(branchname, key_identity)
                  -- Key pair identity ignored.
                  local user = os.getenv("USER")
                  local host = os.getenv("HOSTNAME")
                  if ((user == nil) or (host == nil)) then return nil end
                  return string.format("%s@%s", user, host)
          end
          function get_author(branchname, key_identity)
                  -- Branch name ignored.
                  if (key_identity.given_name == "joe@example.com") then
                          return "Joe Random <joe@example.com>"
                  end
                  return key_identity
          end

'get_default_database_locations ()'
     Called whenever monotone converts a database name to an absolute
     path.

     Returns a table of paths where monotone should look for *note
     Managed Databases::.

     The default implementation returns a table with a single entry,
     '$HOME/.monotone/databases' on Unix and
     '%APPDATA%\monotone\databases' on Windows.

'get_default_database_alias ()'
     Called whenever monotone converts a database name to an absolute
     path.

     Returns the alias of the managed database which should be used as
     default.  See *note Managed Databases::.

     The default implementation returns ':default.mtn'.

'get_default_database_glob ()'
     Returns a pattern to let monotone distinguish a valid from an
     invalid database file name.  This pattern is used in two places:

        * When monotone scans for database files in its default database
          locations, it will skip all files which do not match the
          pattern return from this hook.
        * When a partial database alias is validated and the alias' name
          does not match the pattern returned from this hook, '.mtn' is
          appended to the alias' name.

     The default implementation returns '*.{mtn,db}'.

'edit_comment (USER_LOG_MESSAGE)'
     Called by '*note mtn comment::', '*note mtn commit::', '*note mtn
     import::' when a comment is not provided on the command line.

     Returns a commit comment for the command.  USER_LOG_MESSAGE depends
     on the calling command; see the command definitions.

     This hook is intended to interface with some sort of editor, so
     that you can interactively document each change you make.

     The default definition of 'edit_comment' invokes the user's editor
     (specified by the environment variables 'VISUAL' and 'EDITOR', or
     'editor, vi', or 'notepad' on Windows).  See *note Default hooks::.

'persist_phrase_ok ()'
     Called whenever a passphrase is provided to decrypt a private key.

     Returns 'true' if you want monotone to remember the passphrase of a
     private key for the duration of a single command, or 'false' if you
     want monotone to prompt you for a passphrase for each certificate
     it generates.  Since monotone often generates several certificates
     in quick succession, unless you are very concerned about security
     you probably want this hook to return 'true'.

     The default definition of this hook returns 'true'.

'use_inodeprints ()'
     Called when creating a new workspace.

     Returns 'true' if you want monotone to automatically enable *note
     Inodeprints:: support in the workspace.

     The default definition of this hook returns 'false'.

'ignore_file (FILENAME)'
     Called by many monotone commands when considering whether to
     process a file in a workspace.

     Returns 'true' if FILENAME should be ignored by the command,
     'false' otherwise.

     This is most important when performing recursive actions on
     directories; if 'ignore_file' returns 'true' for a directory, all
     files under that directory will be ignored.

     In some commands, '--no-ignore' will cause this hook to not be
     called.

     The default definition of this hook recognises a number of common
     file types and extensions for temporary and generated file types
     that users typically don't want to track.  In addition, if the file
     '.mtn-ignore' exists in the root workspace directory, this hook
     will read a list of regular expressions from the file, one per
     line, and ignore all files matching one of these expressions.  See
     *note Default hooks::.

'ignore_branch (BRANCHNAME)'

     Called by '*note mtn automate branches::', '*note mtn automate
     tags::', '*note mtn list branches::'.

     Returns 'true' if BRANCHNAME should be ignored by the command,
     otherwise returns 'false'.

     This hook has no default definition; it acts as if it returns
     false.

'get_date_format_spec (WANTED)'
     Called by commands that display dates, when neither '--date-format'
     nor '--no-format-dates' is given.

     Returns a 'strftime' format specification.

     If an empty string is returned, monotone uses the date format
     "yyyy-mm-ddThh:mm:ss".

     The default definition returns '%x' for long and short date
     formats, '%X' for long and short time formats and '%x %X' for long
     and short date time formats, which is equivalent to '22/05/09',
     '09:06:14' and '22/05/09 09:06:14' in an English locale.

     WANTED can be one of 'date_long', 'date_short', 'time_long',
     'time_short', 'date_time_long', 'date_time_short'.

'get_man_page_formatter_command ()'
     Called by '*note mtn manpage::', if '--formatted' is specified.

     Returns a command string that is passed to the operating system
     function 'popen' to format man pages.  The input to the pipe is
     'nroff' markup.

     Note that on the native Windows build of monotone, 'popen' runs the
     'cmd.exe' shell to execute the command string.  On Unix and Windows
     Cygwin, 'popen' runs the 'sh' shell.

     The default hook returns a string that runs 'nroff' and pipes that
     output into the 'less' pager, with appropriate options.  On Windows
     native, the default hook assumes that Cygwin or equivalent is
     installed, and includes 'sh' in the command string.


File: monotone.info,  Node: Netsync Permission Hooks,  Next: Netsync Transport Hooks,  Prev: User Defaults,  Up: Hooks

6.1.4 Netsync Permission Hooks
------------------------------

These hooks are used when running a netsync server, via 'mtn serve'.
They are evaluated by the server for each new connection, based on the
certificate used for authentication by the client.  Note that a
long-running server will need to be restarted in order to reload the
hook definitions if the 'montonerc' file is changed.

'get_netsync_read_permitted (BRANCH, KEY_IDENTITY)'

     Returns 'true' if a peer authenticated as key KEY_IDENTITY should
     be allowed to read from your database certs, revisions, manifests,
     and files associated with BRANCH; otherwise 'false'.  The default
     definition of this hook reads a file 'read-permissions' and all
     files in the subdirectory 'read-permissions.d' in the configuration
     directory.  All the files are treated as if they were one file,
     with 'read-permissions' first followed by the files in
     'read-permissions.d' in alphanumerical order.  The files look like
          pattern "net.example.project.{private,security}*"
          allow "joe@example.net"
          allow "f3f62f86eba204a3ce15174083a53349201993d6"

          comment "everyone can read these branches"
          pattern "net.example.{public,project}*"
          allow "*"
     This example allows everyone access to branches
     'net.example.project' and 'net.example.public' and their
     sub-branches, except for the branches in
     'net.example.project.security' and 'net.example.project.private',
     which are only readable by Joe and Jim.

     The file is divided into stanzas of one 'pattern' line followed by
     any number of 'allow' and 'deny' lines, and possibly a 'continue'
     line.  Anything from the unquoted word 'comment' until the next
     unquoted word is ignored.  A stanza is processed if the argument to
     'pattern' is a glob that matches BRANCH.  Any keys with a hash or
     local_name that matches an 'allow' line are given access, and any
     keys which match a 'deny' line are denied access.  If there is a
     'continue "true"' line, then if the key is not granted or denied
     access in this stanza the next matching stanza will be processed.
     If there is not a 'continue "true"' line, then any key which has
     not been given access will be denied access even if it doesn't
     match any 'deny' lines.  Thus, deny lines are redundant unless
     there is also a 'continue "true"' line.

     If a client connects anonymously, this hook will be called with a
     KEY_IDENTITY of 'nil'.  An empty string value must be used in
     'read-permissions' to represent clients connecting anonymously.
     e.g.  'allow ""'.

'get_netsync_write_permitted (KEY_IDENTITY)'

     Returns 'true' if a peer authenticated as key KEY_IDENTITY should
     be allowed to write into your database certs, revisions, manifests,
     and files; otherwise 'false'.  The default definition of this hook
     reads a file 'write-permissions' and all files in the subdirectory
     'write-permissions.d' in the configuration directory which contains
     a list of key hashes or local key names, one per line, which are
     allowed write access.  The special value '*' means to allow access
     to anyone whose public key we already have.

     If a client connects anonymously, it will be unconditionally denied
     write access; this hook will _not_ be called with a KEY_IDENTITY of
     'nil'.

     Note also that, unlike the equivalent read permission hook, the
     write permission hook does not take a BRANCH name as an argument.
     There is presently no way to selectively grant write access to
     different branches via netsync, for a number of reasons.
     Contributions in the database from different authors can be
     selectively trusted using the *note Trust Evaluation Hooks::
     instead.

'get_remote_automate_permitted(KEY_IDENTITY, COMMAND, OPTIONS)'

     Returns true if a peer authenticated as key KEY_IDENTITY should be
     allowed to execute the given COMMAND with the given OPTIONS, as if
     through 'automate stdio'.


File: monotone.info,  Node: Netsync Transport Hooks,  Next: Trust Evaluation Hooks,  Prev: Netsync Permission Hooks,  Up: Hooks

6.1.5 Netsync Transport Hooks
-----------------------------

When a monotone client initiates a netsync connection, these hooks are
called to attempt to parse the host argument provided on the command
line.  If the hooks fail or return nil, monotone will interpret the host
argument as a network name (possibly with a port number) and open a TCP
socket.

'get_netsync_connect_command (URI, ARGS)'

     Returns a table describing a command to run to connect to the
     specified host.  The URI argument is a table containing between 0
     and 7 components:

        * 'uri["scheme"]', such as '"ssh"' or '"file"'
        * 'uri["user"]', the name of a remote user
        * 'uri["host"]', the name or address of a remote host
        * 'uri["port"]', a network port number
        * 'uri["path"]', a filesystem path
        * 'uri["query"]', for additional parameters
        * 'uri["fragment"]', to describe a sub-location within the
          remote resource

     The ARGS argument is a table containing between 0 and 3 components:

        * 'args["include"]', the branch pattern to include
        * 'args["exclude"]', the branch pattern to exclude
        * 'args["debug"]', whether to run the connection in debug mode

     The default definition of this hook follows:

          function get_netsync_connect_command(uri, args)

                  local argv = nil

                  if uri["scheme"] == "ssh"
                          and uri["host"]
                          and uri["path"] then

                          argv = { "ssh" }
                          if uri["user"] then
                                  table.insert(argv, "-l")
                                  table.insert(argv, uri["user"])
                          end
                          if uri["port"] then
                                  table.insert(argv, "-p")
                                  table.insert(argv, uri["port"])
                          end

                          table.insert(argv, uri["host"])
                  end

                  if uri["scheme"] == "file" and uri["path"] then
                          argv = { }
                  end

                  if argv then

                          table.insert(argv, get_mtn_command(uri["host"]))

                          if args["debug"] then
                                  table.insert(argv, "--debug")
                          else
                                  table.insert(argv, "--quiet")
                          end

                          table.insert(argv, "--db")
                          table.insert(argv, uri["path"])
                          table.insert(argv, "serve")
                          table.insert(argv, "--stdio")
                          table.insert(argv, "--no-transport-auth")

                          if args["include"] then
                                  table.insert(argv, args["include"])
                          end

                          if args["exclude"] then
                                  table.insert(argv, "--exclude")
                                  table.insert(argv, args["exclude"])
                          end
                  end
                  return argv
          end

'use_transport_auth (URI)'

     Returns a boolean indicating whether monotone should use transport
     authentication mechanisms when communicating with URI.  If this
     hook fails, the return value is assumed to be 'true'.  The form of
     the URI argument is a table, identical to the table provided as an
     argument to 'get_netsync_connect_command'.

     Note that the return value of this hook must "match" the semantics
     of the command returned by 'get_netsync_connect_command'.  In
     particular, if this hook returns 'false', the 'serve' command line
     arguments passed to the remote end of the connection should include
     the '--no-transport-auth' option.  A mismatch between this hook's
     return value and the command line returned by
     'get_netsync_connect_command' will cause a communication failure,
     as the local and remote monotone processes will have mismatched
     authentication assumptions.

          function use_transport_auth(uri)
                  if uri["scheme"] == "ssh"
                  or uri["scheme"] == "file" then
                          return false
                  else
                          return true
                  end
          end

'get_mtn_command(HOST)'

     Returns a string containing the monotone command to be executed on
     HOST when communicating over 'ssh'.  The HOST argument is a string
     containing the name of the host to which 'ssh' is connecting, from
     the server URI. This is useful when there are multiple monotone
     binaries on the remote host, or the monotone binary is not in the
     default path.

          function get_mtn_command(host)
              return "mtn"
          end


File: monotone.info,  Node: Trust Evaluation Hooks,  Next: External Diff Tools,  Prev: Netsync Transport Hooks,  Up: Hooks

6.1.6 Trust Evaluation Hooks
----------------------------

Monotone makes heavy use of certs to provide descriptive information
about revisions.  In many projects, not all developers should have the
same privileges, or be trusted for the same purposes (indeed, some
signers might be automated robots, with very specific purposes).

   These hooks allow the user to configure which signers will be trusted
to make which kinds of assertions using certs.  Monotone uses these
certs when selecting available revisions for commands such as 'update'.

   Each user, or even each workspace, can have their own implementation
of these hooks, and thus a different filtered view of valid revisions,
according to their own preferences and purposes.

   See *note Quality Assurance::.

'get_revision_cert_trust (SIGNERS, ID, NAME, VAL)'

     Returns 'true' if you _trust_ the assertion NAME=VALUE on a given
     revision ID, given a valid signature from all the keys in SIGNERS;
     'false' otherwise.  SIGNERS is a table containing a '*note
     key_identity::' for all signatures on this cert, the other three
     parameters are strings.

     The default definition of this hook returns 'true'.

     The default definition corresponds to a form of trust where every
     key which is defined in your database is trusted.  This is a _weak_
     trust setting.  A possible example of a stronger trust function
     (along with a utility function for computing the intersection of
     tables) is the following:

          function intersection(a,b)
             local s={}
             local t={}
             for k,v in pairs(a) do s[v.name] = 1 end
             for k,v in pairs(b) do if s[v] ~= nil then table.insert(t,v) end end
             return t
          end

          function get_revision_cert_trust(signers, id, name, val)
             local trusted_signers = { "bob@happyplace.example.com",
                                       "friend@trustedplace.example.com",
                                       "myself@home.example.com" }
             local t = intersection(signers, trusted_signers)

             if t == nil then return false end

             if    (name ~= "branch" and table.getn(t) >= 1)
                or (name == "branch" and table.getn(t) >= 2)
             then
                return true
             else
                return false
             end
          end

     In this example, any revision certificate is trusted if it is
     signed by at least one of three "trusted" keys, unless it is an
     'branch' certificate, in which case it must be signed by _two_ or
     more trusted keys.  This is one way of requiring that the revision
     has been approved by an extra "reviewer" who used the 'approve'
     command.

'accept_testresult_change (OLD_RESULTS, NEW_RESULTS)'
     Called by '*note mtn update::'.

     This hook is used by the update algorithm to determine whether a
     change in test results between update source and update target is
     acceptable.  The hook is called with two tables, each of which maps
     a signing key hash (as 20 raw bytes) - representing a particular
     testsuite - to a boolean value indicating whether or not the test
     run was successful (calculated from the 'testresult' cert).  The
     function should return 'true' if you consider an update from the
     version carrying the OLD_RESULTS to the version carrying the
     NEW_RESULTS to be acceptable.

     The default definition of this hook returns 'true' if
     '_MTN/wanted-testresults' does not exist.  Otherwise, the file
     should contain a list of signing key hex-encoded hashes in
     lowercase (40 characters).  The hook returns 'false' if a listed
     signing key hash is present in both OLD_RESULTS and NEW_RESULTS,
     with the value 'true' in OLD_RESULTS and 'false' in NEW_RESULTS;
     otherwise it returns 'true'.


File: monotone.info,  Node: External Diff Tools,  Next: External Merge Tools,  Prev: Trust Evaluation Hooks,  Up: Hooks

6.1.7 External Diff Tools
-------------------------

Differences between files can be shown in a number of ways, varying
according to user preference and file type.  These hooks allow
customization of the way file differences are shown in 'diff' and 'log'
commands.

'get_encloser_pattern (FILE_PATH)'

     Called by '*note mtn diff::', '*note mtn automate content_diff::',
     '*note mtn log::', '*note mtn automate log::', for each file in the
     diff output, if '--no-show-encloser' is not given.

     FILE_PATH is the pathname of the file that is being diffed.

     The hook should return a string constant containing a regular
     expression; this regular expression will be used to find lines in
     the file that name the "top-level" constructs enclosing each "hunk"
     of changes.

     If a null string is returned, no enclosers are shown.

     See *note Regexps::, for the regular expression syntax used by
     monotone.

     The default hook treats looks at the file extension, as follows:

     '.texi'
          Returns a regular expression matching nodes, subsections,
          headings.

     '.tex'
     '.ltx'
     '.latex'
          Returns a regular expression matching part, chapter,
          [sub]sections, paragraphs.

     '.txt'
          Empty string.

          Otherwise returns '^[[:alnum:]$_]', which is correct for many
          programming languages.

'external_diff (FILE_PATH, OLD_DATA, NEW_DATA, IS_BINARY, DIFF_ARGS, OLD_REV, NEW_REV)'

     Called by '*note mtn diff::', '*note mtn automate content_diff::',
     '*note mtn log::', '*note mtn automate log::', for each file in the
     diff output, if '--external' is given.

     The hook should run a program that displays the differences between
     two versions of a file.  The return value of the hook is not used.

     FILE_PATH is the pathname of the file that is being diffed.

     OLD_DATA and NEW_DATA are the contents of the old and the new file
     (OLD_DATA is nil if the file is new).  If the content is binary,
     IS_BINARY will be true, otherwise false.  OLD_REV and NEW_REV are
     the revision IDs of the old and new data.

     If an extra arguments are given via '--diff-args', the string will
     be passed in as DIFF_ARGS.  Otherwise DIFF_ARGS will be nil.

     The default implementation of this hook calls the program 'diff',
     and if '--diff-args' were not passed, takes default arguments from
     the Lua variable 'external_diff_default_args'.  You can override
     this variable in your configuration file, without overriding the
     whole hook.


File: monotone.info,  Node: External Merge Tools,  Next: Selector Expansion,  Prev: External Diff Tools,  Up: Hooks

6.1.8 External Merge Tools
--------------------------

These hooks allow the user to use their favorite tools when resolving
*note File Content Conflict::s.

   The *note Default hooks:: include helper functions used by the hooks
below to invoke a number of external merge tools known to monotone, and
you can override or extend these hooks if you have a preferred tool, or
if you have a tool specific to certain file types.

'merge3 (ANCESTOR_PATH, LEFT_PATH, RIGHT_PATH, MERGED_PATH, ANCESTOR_TEXT, LEFT_TEXT, RIGHT_TEXT)'

     Called by '*note mtn conflicts resolve_first::' when 'interactive'
     is given; by '*note mtn merge::', '*note mtn explicit_merge::' when
     '--resolve-conflicts' is not specified and the internal merger
     fails for a file content conflict; and by '*note mtn update::' or
     any command that accepts '--update' for workspace file content
     conflicts.

     The actual ancestor, left, and right contents of the file are
     passed in the ANCESTOR_TEXT, LEFT_TEXT, and RIGHT_TEXT strings.  In
     addition, the hook is given the names that this file had in the
     ancestor (ANCESTOR_PATH), left (LEFT_PATH), and right (RIGHT_PATH)
     trees, and the name it will end up having in the merged tree
     (MERGED_PATH).  These paths are useful for merge tools that can
     display the names of files in their GUI, since the actual path
     names are likely more meaningful than the temporary file names the
     merge tool will actually be working on.

     On success, 'merge3' returns a string, which should be the new file
     contents, the result of merging the given texts.  If nil is
     returned, the merge command fails; this is how the user can abort a
     merge.

     The default definition of this hook writes the texts to temporary
     files, then delegates the actual merge to the result of *note
     get_preferred_merge3_command::.  The default definition of *note
     get_preferred_merge3_command:: checks to see if the 'MTN_MERGE'
     environment variable, or the Lua variable 'merger' are set to the
     name of a merge tool that it recognizes, and if not, then simply
     searches for several popular tools that might be installed on the
     local system.

     The default hook then invokes the merge tool, waits for it to
     return, and checks to see if MERGED_PATH was written.  If not, it
     returns nil, and the merge command fails.

     For details, see the code in *note Default hooks::.

     By redefining this hook the user can choose a preferred merge tool,
     or select different merge tools for different file types.  For
     example, gimp for .png files, OpenOffice.org for .doc, and so on.

'get_preferred_merge3_command(TBL)'

     Returns a table 'command, mkey' telling '*note merge3::' what
     external merge tool to run.

     TBL wraps up the various arguments for each merge command and is
     always provided by *note merge3::.  If there is a particular editor
     that you would like to use to perform merge3 operations, override
     this hook to specify it.


File: monotone.info,  Node: Selector Expansion,  Next: Attribute Handling,  Prev: External Merge Tools,  Up: Hooks

6.1.9 Selector Expansion
------------------------

Monotone's selectors are a powerful mechanism used to refer to revisions
with symbolic names or groupings.  Thanks to the hooks described in this
section, it is possible to use various forms of shorthand in selection
strings; these hooks are designed to recognise shorthand patterns and
expand them to their full form.

   For more detail on the use of selectors, see *note Selectors::.

'expand_selector (STR)'

     Called by any command that can take a selector argument, for each
     element of a combined selector string (between '/' separators).

     The input STR is the command line argument.  The hook should return
     a string that is a valid monotone selector.

     Expansion generally means providing a type prefix for the selector,
     such as 'a:' for authors or 'd:' for dates.

     The default definition of this hook attempts to recognize certs,
     email addresses, branch names, hex ids, and dates, and adds the
     appropriate prefix.  See *note Default hooks::.

'expand_date (STR)'
     Called when processing a date selector ('d:').

     The input STR is the command line selector, after processing by
     'expand_selector'.  The hook should return a date that can be used
     by a standard SQL select statement.

     The default hook recognizes special words such as 'yesterday' or '6
     months ago' and converts them into well formed date expressions.
     See *note Default hooks::.


File: monotone.info,  Node: Attribute Handling,  Next: GIT Export Hooks,  Prev: Selector Expansion,  Up: Hooks

6.1.10 Attribute Handling
-------------------------

Some files in a project are special; they may require different handling
(such as binary or structured files that should always be manually
merged - see *note File Content Conflict::), or they may represent
executable scripts or programs.

   Monotone allows each file (or directory) in a repository to carry
arbitrary *note File Attributes::.  Persistent attributes are stored in
each revision's manifest.  The hooks in this section allow files to be
automatically recognised as having certain attributes at the time
they're added, and for custom triggers to be invoked on each file
according to its attributes when the workspace is changed.

'attr_functions [ATTRIBUTE] (FILENAME, VALUE)'

     This is not a hook function, but a _table_ of hook functions,
     indexed by ATTRIBUTE.  Each entry in the table 'attr_functions' is
     a function taking a file name FILENAME and an attribute value
     VALUE.  The function should "apply" the attribute to the file in
     the file system, possibly in a platform-specific way.

     These hooks are called by any command that modifies workspace
     files, including '*note mtn revert::', '*note mtn undrop::', '*note
     mtn update::', '*note mtn merge_into_workspace::', '*note mtn
     pluck::', '*note mtn clone::' and '*note mtn checkout::'.

     When called to set an attribute, VALUE is a string representing the
     value of the attribute.

     When called to clear an attribute, VALUE is 'nil'.

     This facility can be used to extend monotone's understanding of
     files with platform-specific attributes, such as permission bits,
     access control lists, or special file types.

     By default, there is only one entry in this table, for the
     'mtn:execute' attribute.  It calls platform-specific functions to
     mark files as executable or not.  See *note Default hooks::.

'attr_init_functions [ATTRIBUTE] (FILENAME)'

     This is not a hook function, but a _table_ of hook functions.  Each
     entry in the table 'attr_init_functions', at table entry ATTRIBUTE,
     is a function taking a file (or directory) name FILENAME.  Each
     function returns 'true' if the attribute should be set on FILENAME.
     This table of hook functions is called once for each file during an
     "add".

     By default, there are only two entries in this table, for the
     'mtn:execute' and 'mtn:manual_merge' attributes.  Their definition
     is:

          attr_init_functions["mtn:execute"] =
             function(filename)
                if (is_executable(filename)) then
                  return "true"
                else
                  return nil
                end
             end
          attr_init_functions["mtn:manual_merge"] =
             function(filename)
                if (binary_file(filename)) then
                  return "true" -- binary files must be merged manually
                else
                  return nil
                end
             end

     The 'binary_file' function is also defined as a Lua hook.  See
     *note Default hooks::.


File: monotone.info,  Node: GIT Export Hooks,  Next: Validation Hooks,  Prev: Attribute Handling,  Up: Hooks

6.1.11 GIT Export Hooks
-----------------------

Exporting monotone revisions in git-fast-import(1) format often requires
translation of monotone author cert values and associated signing keys
into corresponding git author and committer values.  Translation of
author and committer values and validation of the results is controlled
by these hooks.  See '*note mtn git_export::'.

'unmapped_git_author(AUTHOR)'

     This hook is called for any git author or committer value that does
     not come from the current _author map_ file, specified by
     '--authors-file'.  If no _author map_ file is specified this hook
     will be called for _every_ unique git author and committer value.
     It may return the value unchanged or modify it in some way in an
     effort to ensure that it is valid.  The default implementation
     attempts several common pattern replacements to produce valid
     authors from monotone authors.

'validate_git_author(AUTHOR)'

     This hook is called before the git author or committer value is
     written to the export output stream.  The AUTHOR value is either
     the mapped value from the current _author map_ file or the value
     produced by the 'unmapped_git_author' hook.  This hook may return
     true if the author is valid or false if it is not.  The export will
     be aborted if this hook returns false for any value.

     The default definition enforces the 'Name <email>' pattern.


File: monotone.info,  Node: Validation Hooks,  Prev: GIT Export Hooks,  Up: Hooks

6.1.12 Validation Hooks
-----------------------

If there is a policy decision to make, Monotone defines certain hooks to
allow a client to validate or reject certain behaviors.

'validate_changes (REVISION_TEXT, BRANCHNAME)'

     This hook is called by '*note mtn commit::' just after the
     to-be-committed changes and the branch have been logically verified
     and before the user enters his/her commit message.  REVISION_TEXT
     is the full text of the changes for this revision, which can be
     parsed with the '*note parse_basic_io::' function.  The second
     parameter, BRANCHNAME, is the branch for this commit - unless it
     gets changed in '*note edit_comment::' later on.

     If the hook finds the changes satisfactory, it should return 'true,
     ""'.  If it finds fault, then it should return 'false, reason'
     where REASON is a string containing the reason the changes were
     rejected.  By default, this hook is not defined and acts as if it
     returned 'true, ""'.

'validate_commit_message (MESSAGE, REVISION_TEXT, BRANCHNAME)'

     This hook is called by '*note mtn commit::' after '*note
     edit_comment::'.  MESSAGE is the result of 'edit_comment' and
     REVISION_TEXT is the full text of the changes for this revision,
     which can be parsed with the '*note parse_basic_io::' function.
     BRANCHNAME is parsed from the result of 'edit_comment'.

     If the hook finds the commit message satisfactory, it should return
     'true, ""'.  If it finds fault, then it should return 'false,
     reason' where REASON is a string containing the reason the message
     was rejected.  By default, this hook is not defined and acts as if
     it returned 'true, ""'.


File: monotone.info,  Node: Additional Lua Functions,  Next: Implementation Differences,  Prev: Hooks,  Up: Lua Reference

6.2 Additional Lua Functions
============================

This section documents the additional Lua functions made available to
hook and user command writers.

'alias_command(ORIGINAL, ALIAS)'

     This function adds a new alias for a monotone command.  ORIGINAL
     must be an existing montone command, ALIAS is the new alias.

'change_workspace(DIRECTORY)'

     Change to workspace DIRECTORY, which may be absolute or relative to
     the directory that was current when monotone was started.

     Throws an error if DIRECTORY does not exist or is not a workspace
     (does not contain '_MTN').

     After calling 'change_workspace', the next operation should read
     the workspace options.  '*note mtn_automate::' does this; other Lua
     functions do not.

'existonpath(COMMAND)'

     Return 0 if COMMAND (a string) exists on '$PATH' and is executable,
     -1 otherwise.

     For example, 'existonpath("xxdiff")' returns 0 if the program
     xxdiff is available.

     On Windows, this function automatically appends ".exe" to the
     program name if it is not present.  In the previous example,
     'existonpath' would search for "xxdiff.exe".

'get_confdir()'

     Returns the path to the configuration directory; see '*note
     --confdir::'.

'get_ostype()'

     Returns the operating system flavor as a string.

'guess_binary_file_contents(FILESPEC)'

     Returns true if the file contains one or more of the following
     bytes:
          0x00 thru 0x06
          0x0E thru 0x1f
          0x7f

'guess_terminal_width()'

     Returns the size of the terminal window as number or a sane default
     (72) if the information cannot be retrieved.

'include(SCRIPTFILE)'

     This function tries to load and execute the Lua script contained in
     SCRIPTFILE.  It returns true for success and false if there is an
     error.

     This is the same as Lua 'loadfile'.

'includedir(SCRIPTPATH)'

     This function loads and executes in alphabetical order all the
     scripts contained in the directory SCRIPTPATH.

     If one of the scripts has an error, the function doesn't process
     the remaining scripts.

     There is no useful return value.

'includedirpattern(SCRIPTPATH, PATTERN)'

     This function loads and executes in alphabetical order all the
     scripts contained into the directory SCRIPTPATH that match PATTERN
     (a glob pattern).

     If one of the scripts has an error, the functions doesn't process
     the remaining scripts.

     There is no useful return value.

'is_executable(FILE)'

     This function returns true if FILE is executable, false otherwise.
     On Windows this function always returns false.

'kill(PID [, SIGNAL])'

     This function calls the kill() C library function on POSIX systems
     and TerminateProcess on Win32 (in that case PID is the process
     handle).  If the optional SIGNAL parameter is missing, SIGTERM will
     be used.

     Returns 0 on success, -1 on error.

'make_executable(FILE)'

     This function marks FILE as executable.  On Windows this has no
     effect.

'globish.match(GLOB, STRING)'

     Returns 'true' if GLOB matches STR, return 'false' otherwise.

'mkstemp(TEMPLATE)'

     Returns a unique name for a writeable temporary file.  Note that it
     does not create the file.

     TEMPLATE should be a string that consists of contiguous, legal file
     and path name characters followed by six 'X's.

     'mkstemp' replaces the 'X's by an alpha-numeric sequence that is
     unique in the directory given in TEMPLATE.

     Subsequent calls to 'mkstemp' within the same process each yield
     different file names.

     Unlike other implementations, monotone mkstemp allows TEMPLATE to
     contain a complete path, not only a filename, allowing users to
     create temporary files outside the current directory.

     *Important notice:*
     To create a temporary file, you should use the 'temp_file'
     function, unless you need to run monotone with the
     '--no-builtin-rcfiles' option.  'temp_file()' builds on 'mkstemp()'
     and creates a file in the standard TMP/TEMP directories.  For the
     definition of 'temp_file()', see *note Default hooks::.

'mtn_automate(COMMAND ARGS... )'

     'mtn_automate' calls the monotone 'automate' COMMAND, with ARGS...
     (may include options).  The result of the call is a pair consisting
     of a boolean ('true' on success), and a string containing the
     'stdout' output from 'automate COMMAND'.

     This function is not for use in ordinary Lua hooks, but rather for
     Lua based commands as defined by the Lua function *note
     register_command::.

     Note that keyboard interaction is disabled, just as if
     '--non-interactive' is specified.  Actions which require operations
     on password-encrypted private keys will therefor fail unless the
     '*note get_passphrase::' hook is set up locally.

'parse_basic_io(DATA)'

     Parse the string DATA, which should be in 'basic_io' format.
     Returns nil if it can't parse the string; otherwise returns a
     table, with each entry in the table corresponding to one 'basic_io'
     line, consisting of a 'name' element and a 'values' element.

     For example, given this as input:

          thingy "foo" "bar"
          thingy "baz"
          spork
          frob "oops"

     The output table will be:
          {
             1 = { name = "thingy", values = { 1 = "foo", 2 = "bar" } },
             2 = { name = "thingy", values = { 1 = "baz" } },
             3 = { name = "spork", values = { } },
             4 = { name = "frob", values = { 1 = "oops" } }
          }

'regex.search(REGEXP, STRING)'

     Returns 'true' if a match for REGEXP is found in STR, 'false'
     otherwise.  *Note Regexps::, for the syntax of REGEXP.

'register_command(NAME, PARAMS, ABSTRACT, DESCRIPTION, FUNCTION)'

     Add a command named NAME to the user command group in monotone.
     When the user issues the registered command, monotone will call the
     lua FUNCTION name supplied.  That function would then normally use
     '*note mtn_automate::' calls to service the request.  PARAMS is a
     string with the list of parameters for the command, ABSTRACT gives
     a brief description, DESCRIPTION a longer description.  PARAMS,
     ABSTRACT, DESCRIPTION are output by 'mtn help'.

'server_request_sync(WHAT, ADDRESS, INCLUDE, EXCLUDE)'

     Initiate a netsync connection to the server at ADDRESS, with the
     given INCLUDE and EXCLUDE branch patterns.  WHAT is one of 'sync',
     'push', or 'pull', indicating netsync operation.

     When called by a monotone instance which is not running the 'serve'
     command, this function has no effect.

'server_set_listening(BOOLEAN)'

     If the argument is FALSE, make the server not listen for incoming
     connections, and exit when all existing connections have closed.

     If the argument is TRUE, cancel an earlier call with FALSE given.

'sleep(SECONDS)'

     Makes the calling process sleep for the specified number of
     seconds.

'spawn(EXECUTABLE [, ARGS ...])'

     Starts the named executable with the given arguments.  Returns the
     process PID on POSIX systems, the process handle on Win32 or -1 if
     there was an error.

     Calls fork/execvp on POSIX, CreateProcess on Win32.

     *Important notice:*
     To spawn a process and wait for its completion, use the 'execute()'
     function, unless you need to run monotone with the
     '--no-builtin-rcfiles' option.  'execute()' builds on 'spawn()' and
     'wait()' in a standardized way.

'spawn_pipe(EXECUTABLE [, ARGS ...])'

     Like spawn(), but returns three values, where the first two are the
     subprocess' standard input and standard output, and the last is the
     process PID on POSIX systems, the process handle on Win32 or -1 if
     there was an error.

'spawn_redirected(INFILE, OUTFILE, ERRFILE, EXECUTABLE [, ARGS ...])'

     Like spawn(), but with standard input, standard output and standard
     error redirected to the given files.

'wait(PID)'

     Wait until the process with given PID (process handle on Win32)
     exits.  Returns two values: a result value and the exit code of the
     waited-for process.

     The exit code is meaningful only if the result value is 0.


File: monotone.info,  Node: Implementation Differences,  Prev: Additional Lua Functions,  Up: Lua Reference

6.3 Implementation Differences
==============================

This section documents disabled and overwritten functions from the
standard Lua libraries.

'print, io.write'

     Both functions are overwritten to print to standard error (instead
     of standard output), formatted as monotone progress message with
     the prefix "lua:" in front of every line.

     The rationale behind this is to catch most of the informational
     messages from hooks which use these functions and present them via
     monotone's user interface.  This allows them to be logged just as
     other progress messages, and prevents unexpected out-of-band output
     in the *note Automation:: interface.

     If you explicitly need to write to either stdout or stderr, please
     use 'io.stdout:write' and 'io.stderr.write'.

'os.execute, io.popen'

     Both functions have been disabled for security reasons.  You should
     use their replacements 'spawn' and 'spawn_pipe'.

     The original versions do not apply any kind of shell character
     escaping, which might lead to serious problems f.e.  when a merger
     is automatically invoked by such a function.


File: monotone.info,  Node: Special Topics,  Next: Default hooks,  Prev: Lua Reference,  Up: Top

7 Special Topics
****************

This chapter describes some "special" issues which are not directly
related to monotone's _use_, but which are occasionally of interest to
people researching monotone or trying to learn the specifics of how it
works.  Most users can ignore these sections.

* Menu:

* Internationalization::        Using monotone in non-English locales.
* Hash Integrity::              Notes on probability and failure.
* Rebuilding ancestry::         In case of corruption.
* Regexps::                     The syntax of regular expressions
                                  used by Monotone.


File: monotone.info,  Node: Internationalization,  Next: Hash Integrity,  Prev: Special Topics,  Up: Special Topics

7.1 Internationalization
========================

Monotone initially dealt with only ASCII characters, in file path names,
certificate names, key names, and packets.  Some conservative extensions
are provided to permit internationalized use.  These extensions can be
summarized as follows:

   * Monotone uses GNU gettext to provide localized progress and error
     messages.  Translations may or may not exist for your locale, but
     the infrastructure is present to add them.

   * All command-line arguments are mapped from your local character set
     to UTF-8 before processing.  This means that monotone can _only_
     handle key names, file names and certificate names which map
     cleanly into UTF-8.

   * Monotone's control files are stored in UTF-8.  This includes:
     revisions and manifests, both inside the database and when written
     to the '_MTN/' directory of the workspace; the '_MTN/options' and
     '_MTN/revision' files.  Converting these files to any other
     character set will cause monotone to break; do not do so.

   * File path names in the workspace are converted to the locale's
     character set (determined via the LANG or CHARSET environment
     variables) before monotone interacts with the file system.  If you
     are accustomed to being able to use file names in your locale's
     character set, this should "just work" with monotone.

   * Key and cert names, and similar "name-like" entities are subject to
     some cleaning and normalization, and conversion into network-safe
     subsets of ASCII (typically ACE). Generally, you should be able to
     use "sensible" strings in your locale's character set as names, but
     they may appear mangled or escaped in certain contexts such as
     network transmission.

   * Monotone's transmission and storage forms are otherwise unchanged.
     Packets and database contents are 7-bit clean ASCII.

   The remainder of this section is a precise specification of
monotone's internationalization behavior.

General Terms
=============

Character set conversion
     The process of mapping a string of bytes representing wide
     characters from one encoding to another.  Per-file character set
     conversions are specified by a Lua hook 'get_charset_conv' which
     takes a filename and returns a table of two strings: the first
     represents the "internal" (database) charset, the second represents
     the "external" (file system) charset.

LDH
     Letters, digits, and hyphen: the set of ASCII bytes '0x2D',
     '0x30..0x39', '0x41..0x5A', and '0x61..0x7A'.

stringprep
     RFC 3454, a general framework for mapping, normalizing, prohibiting
     and bidirectionality checking for international names prior to use
     in public network protocols.

nameprep
     RFC 3491, a specific profile of stringprep, used for preparing
     international domain names (IDNs)

punycode
     RFC 3492, a "bootstring" encoding of Unicode into ASCII.

IDNA
     RFC 3490, international domain names for applications, a
     combination of the above technologies (nameprep, punycoding,
     limiting to LDH characters) to form a specific "ASCII compatible
     encoding" (ACE) of Unicode, signified by the presence of an
     "unlikely" ACE prefix string "xn-".  IDNA is intended to make it
     possible to use Unicode relatively "safely" over legacy ASCII-based
     applications.  the general picture of an IDNA string is this:

                {ACE-prefix}{LDH-sanitized(punycode(nameprep(UTF-8-string)))}

     It is important to understand that IDNA encoding does _not_
     preserve the input string: it both prohibits a wide variety of
     possible strings and normalizes non-equal strings to supposedly
     "equivalent" forms.

     By default, monotone does _not_ decode IDNA when printing to the
     console (IDNA names are ASCII, which is a subset of UTF-8, so this
     normal form conversion can still apply, albeit oddly).  this
     behavior is to protect users against security problems associated
     with malicious use of "similar-looking" characters.

Filenames
=========

   * Filenames are subject to normal form conversion.

   * Filenames are subject to an additional normal form stage which
     adjusts for platform name semantics, for example changing the
     Windows '0x5C' '\' path separator to '0x2F' '/'.  This extra
     processing is performed by boost::filesystem.

   * Monotone does not properly handle case insensitivity on Windows.

   * A filename (in normal form) is constrained to be a nonempty
     sequence of path components, separated by byte '0x2F' (ASCII / ),
     and without a leading or trailing '0x2F'.

   * A path component is a nonempty sequence of any UTF-8 character
     codes except the path separator byte '0x2F' and any ASCII "control
     codes" ('0x00..0x1F' and '0x7F').

   * The path components "."  and ".."  are prohibited.

   * Manifests and revisions are constructed from the normal form
     (UTF-8).  The LC_COLLATE locale category is _not_ used to sort
     manifest or revision entries.

File contents
=============

   * Files are subject to character set conversion and line ending
     conversion.

   * File SHA1 values are calculated from the internal form of the
     conversions.  If the external form of a file differs from the
     internal form, running a 3rd party program such as 'sha1sum' will
     produce different results than those entries shown in a
     corresponding manifest.

UI messages
===========

UI messages are displayed via calls to 'gettext()'.

Host names
==========

Host names are read on the command-line and subject to normal form
conversion.  Host names are then split at '0x2E' (ASCII '.'), each
component is subject to IDNA encoding, and the components are rejoined.

   After processing, host names are stored internally as ASCII. The
invariant is that a host name inside monotone contains only sequences of
LDH separated by '0x2E'.

Cert names
==========

Read on the command line and subject to normal form conversion and IDNA
encoding as a single component.  The invariant is that a cert name
inside monotone is a single LDH ASCII string.

Cert values
===========

Cert values may be either text or binary, depending on the return value
of the hook 'cert_is_binary'.  If binary, the cert value is never
printed to the screen (the literal string "<binary>" is displayed,
instead), and is never subjected to line ending or character conversion.
If text, the cert value is subject to normal form conversion, as well as
having all UTF-8 codes corresponding to ASCII control codes ('0x0..0x1F'
and '0x7F') prohibited in the normal form, except '0x0A' (ASCII LF).

Var domains
===========

Read on the command line and subject to normal form conversion and IDNA
encoding as a single component.  The invariant is that a var domain
inside monotone is a single LDH ASCII string.

Var names and values
====================

Var names and values are assumed to be text, and subject to normal form
conversion.

Key names
=========

Read on the command line and subject to normal form conversion and IDNA
encoding as an email address (split and joined at '.'  and '@'
characters).  The invariant is that a key name inside monotone contains
only LDH, '0x2E' (ASCII '.')  and '0x40' (ASCII '@') characters.

Packets
=======

Packets are 7-bit ASCII. The characters permitted in packets are the
union of these character sets:

   * The 65 characters of base64 encoding (64 coding + "=" pad).
   * The 16 characters of hex encoding.
   * LDH, '@' and '.'  characters, as required for key and cert names.
   * '[' and ']', the packet delimiters.
   * ASCII codes 0x0D (CR), 0x0A (LF), 0x09 (HT), and 0x20 (SP).


File: monotone.info,  Node: Hash Integrity,  Next: Rebuilding ancestry,  Prev: Internationalization,  Up: Special Topics

7.2 Hash Integrity
==================

Some proponents of a competing, proprietary version control system have
suggested, in a usenix paper
(http://www.usenix.org/events/hotos03/tech/full_papers/henson/henson_html/),
that the use of a cryptographic hash function such as SHA1 as an
identifier for a version is unacceptably unsafe.  This section addresses
the argument presented in that paper and describes monotone's additional
precautions.

   To summarize our position:
   * the analysis in the paper is wrong,
   * even if it were right, monotone is sufficiently safe.

The analysis is wrong
=====================

The paper displays a fundamental lack of understanding about what a
_cryptographic_ hash function is, and how it differs from a normal hash
function.  Furthermore it confuses accidental collision with attack
scenarios, and mixes up its analysis of the risk involved in each.  We
will try to untangle these issues here.

   A cryptographic hash function such as SHA1 is more than just a
uniform spread of inputs to an output range.  Rather, it must be
designed to withstand attempts at:

   * reversal: deriving an input value from the output
   * collision: finding two different inputs which hash to the same
     output

   Collision is the problem the paper is concerned with.  Formally, an
n-bit cryptographic hash should cost 2^n work units to collide against a
given value, and sqrt(2^n) tries to find a random pair of colliding
values.  This latter probability is sometimes called the hash's
"birthday paradox probability".

Accidental collision
--------------------

One way of measuring these bounds is by measuring how single-bit changes
in the input affect bits in the hash output.  The SHA1 hash has a strong
_avalanche property_, which means that flipping _any one bit_ in the
input will cause on average half the 160 bits in the output code to
change.  The fanciful VAL1 hash presented in the paper does not have
such a property -- flipping its first bit when all the rest are zero
causes _no change_ to any of the 160 output bits -- and is completely
unsuited for use as a _cryptographic hash_, regardless of the general
shape of its probability distribution.

   The paper also suggests that birthday paradox probability cannot be
used to measure the chance of accidental SHA1 collision on "real
inputs", because birthday paradox probability assumes a uniformly random
sample and "real inputs" are not uniformly random.  The paper is wrong:
the inputs to SHA1 are not what is being measured (and in any case can
be arbitrarily long); the collision probability being measured is of
_output space_.  On output space, the hash is designed to produce
uniformly random spread, even given nearly identical inputs.  In other
words, it is _a primary design criterion_ of such a hash that a birthday
paradox probability is a valid approximation of its collision
probability.

   The paper's characterization of risk when hashing "non-random inputs"
is similarly deceptive.  It presents a fanciful case of a program which
is _storing_ every possible 2kb block in a file system addressed by SHA1
(the program is trying to find a SHA1 collision).  While this scenario
_will_ very likely encounter a collision _somewhere_ in the course of
storing all such blocks, the paper neglects to mention that we only
expect it to collide after storing about 2^{80} of the 2^{16384}
possible such blocks (not to mention the requirements for compute time
to search, or disk space to store 2^{80} 2kb blocks).

   Noting that monotone can only store 2^{41} bytes in a database, and
thus probably some lower number (say 2^{32} or so) active rows, we
consider such birthday paradox probability well out of practical sight.
Perhaps it will be a serious concern when multi-yottabyte hard disks are
common.

Collision attacks
-----------------

Setting aside accidental collisions, then, the paper's underlying theme
of vulnerability rests on the assertion that someone will break SHA1.
Breaking a cryptographic hash usually means finding a way to collide it
trivially.  While we note that SHA1 has in fact resisted attempts at
breaking for 8 years already, we cannot say that it will last forever.
Someone might break it.  We can say, however, that finding a way to
trivially collide it only changes the resistance to _active attack_,
rather than the behavior of the hash on benign inputs.

   Therefore the vulnerability is not that the hash might suddenly cease
to address benign blocks well, but merely that additional security
precautions might become a requirement to ensure that blocks are benign,
rather than malicious.  The paper fails to make this distinction,
suggesting that a hash becomes "unusable" when it is broken.  This is
plainly not true, as a number of systems continue to get useful low
collision hashing behavior -- just not good security behavior -- out of
"broken" cryptographic hashes such as MD4.

Monotone is probably safe anyways
=================================

Perhaps our arguments above are unconvincing, or perhaps you are the
sort of person who thinks that practice never lines up with theory.
Fair enough.  Below we present _practical_ procedures you can follow to
compensate for the supposed threats presented in the paper.

Collision attacks
-----------------

A successful collision attack on SHA1, as mentioned, does not disrupt
the _probability_ features of SHA1 on benign blocks.  So if, at any
time, you believe SHA1 is "broken", it does _not_ mean that you cannot
use it for your work with monotone.  It means, rather, that you cannot
base your _trust_ on SHA1 values anymore.  You must trust who you
communicate with.

   The way around this is reasonably simple: if you do not trust SHA1 to
prevent malicious blocks from slipping into your communications, you can
always augment it by enclosing your communications in more security,
such as tunnels or additional signatures on your email posts.  If you
choose to do this, you will still have the benefit of self-identifying
blocks, you will simply cease to trust such blocks unless they come with
additional authentication information.

   If in the future SHA1 (or, indeed, RSA) becomes accepted as broken we
will naturally upgrade monotone to a newer hash or public key scheme,
and provide migration commands to recalculate existing databases based
on the new algorithm.

   Similarly, if you do not trust our vigilance in keeping up to date
with cryptography literature, you can modify monotone to use any
stronger hash you like, at the cost of isolating your own communications
to a group using the modified version.  Monotone is free software, and
runs atop 'botan', so it is both legal and relatively simple to change
it to use some other algorithm.


File: monotone.info,  Node: Rebuilding ancestry,  Next: Regexps,  Prev: Hash Integrity,  Up: Special Topics

7.3 Rebuilding ancestry
=======================

As described in *note Historical records::, monotone revisions contain
the SHA1 hashes of their predecessors, which in turn contain the SHA1
hashes of _their_ predecessors, and so on until the beginning of
history.  This means that it is _mathematically impossible_ to modify
the history of a revision, without some way to defeat SHA1.  This is
generally a good thing; having immutable history is the point of a
version control system, after all, and it turns out to be very important
to building a _distributed_ version control system like monotone.

   It does have one unfortunate consequence, though.  It means that in
the rare occasion where one _needs_ to change a historical revision, it
will change the SHA1 of that revision, which will change the text of its
children, which will change their SHA1s, and so on; basically the entire
history graph will diverge from that point (invalidating all certs in
the process).

   In practice there are two situations where this might be necessary:
   * bugs: monotone has occasionally allowed nonsense, uninterpretable
     changesets to be generated and stored in the database, and this was
     not detected until further work had been based off of them.
   * advances in crypto: if or when SHA1 is broken, we will need to
     migrate to a different secure hash.

   Obviously, we hope neither of these things will happen, and we've
taken lots of precautions against the first recurring; but it is better
to be prepared.

   If either of these events occur, we will provide migration commands
and explain how to use them for the situation in question; this much is
necessarily somewhat unpredictable.  In the past we've used the (now
defunct) 'db rebuild' command, and more recently the 'db rosterify'
command, for such changes as monotone developed.  These commands were
used to recreate revisions with new formats.  Because the revision id's
changed, all the existing certs that you trust also must be reissued,
signed with your key.(1)

   While such commands can reconstruct the ancestry graph in _your_
database, there are practical problems which arise when working in a
distributed work group.  For example, suppose our group consists of the
fictional developers Jim and Beth, and they need to rebuild their
ancestry graph.  Jim performs a rebuild, and sends Beth an email telling
her that he has done so, but the email gets caught by Beth's spam
filter, she doesn't see it, and she blithely syncs her database with
Jim's.  This creates a problem: Jim and Beth have combined the
pre-rebuild and post-rebuild databases.  Their databases now contain two
complete, parallel (but possibly overlapping) copies of their project's
ancestry.  The "bad" old revisions that they were trying to get rid of
are still there, mixed up with the "good" new revisions.

   To prevent such messy situations, monotone keeps a table of branch
"epochs" in each database.  An epoch is just a large bit string
associated with a branch.  Initially each branch's epoch is zero.  Most
monotone commands ignore epochs; they are relevant in only two
circumstances:

   * When monotone rebuilds ancestry, it generates a new _random_ epoch
     for each branch in the database.

   * When monotone runs netsync between databases, it checks to make
     sure that all branches involved in the synchronization have the
     same epochs.  If any epochs differ, the netsync is aborted with no
     changes made to either database.  If either side is seeing a branch
     for the first time, it adopts the epoch of the other side.

   Thus, when a user rebuilds their ancestry graph, they select a new
epoch and thus effectively disassociate with the group of colleagues
they had previously been communicating with.  Other members of that
group can then decide whether to follow the rebuild user into a new
group -- by pulling the newly rebuilt ancestry -- or to remain behind in
the old group.

   In our example, if Jim and Beth have epochs, Jim's rebuild creates a
new epoch for their branch, in his database.  This causes monotone to
reject netsync operations between Jim and Beth; it doesn't matter if
Beth loses Jim's email.  When she tries to synchronize with him, she
receives an error message indicating that the epoch does not match.  She
must then discuss the matter with Jim and settle on a new course of
action -- probably pulling Jim's database into a fresh database on
Beth's end - before future synchronizations will succeed.

Best practices
==============

The previous section described the theory and rationale behind rebuilds
and epochs.  Here we discuss the practical consequences of that
discussion.

   If you decide you must rebuild your ancestry graph -- generally by
announcement of a bug from the monotone developers -- the first thing to
do is get everyone to sync their changes with the central server; if
people have unshared changes when the database is rebuilt, they will
have trouble sharing them afterwards.

   Next, the project should pick a designated person to take down the
netsync server, rebuild their database, and put the server back up with
the rebuilt ancestry in it.  Everybody else should then pull this
history into a fresh database, check out again from this database, and
continue working as normal.

   In complicated situations, where people have private branches, or
ancestries cross organizational boundaries, matters are more complex.
The basic approach is to do a local rebuild, then after carefully
examining the new revision IDs to convince yourself that the rebuilt
graph is the same as the upstream subgraph, use the special 'db epoch'
commands to force your local epochs to match the upstream ones.  (You
may also want to do some fiddling with certs, to avoid getting duplicate
copies of all of them; if this situation ever arises in real life we'll
figure out how exactly that should work.)  Be very careful when doing
this; you're explicitly telling monotone to let you shoot yourself in
the foot, and it will let you.

   Fortunately, this process should be extremely rare; with luck, it
will never happen at all.  But this way we're prepared.

   ---------- Footnotes ----------

   (1) Regardless of who originally signed the certs, after the rebuild
they will be signed by you.  This means you should be somewhat careful
when rebuilding, but it is unavoidable -- if you could sign with other
people's keys, that would be a rather serious security problem!


File: monotone.info,  Node: Regexps,  Prev: Rebuilding ancestry,  Up: Special Topics

7.4 Regular Expression Syntax
=============================

Monotone expects user-provided regular expressions in '.mtn-ignore'
files and as the result of the 'get_encloser_pattern' Lua hook (for the
'diff' command).  User-written Lua hooks may also use the function
'regex.search' as they see fit.  All these regular expressions should be
written with the same syntax, which is that expected by the
Perl-Compatible Regular Expression library (PCRE).

* Menu:

* Regexp Summary::  Quick reference to regular expression syntax.
* Regexp Details::  Details of regular expression syntax and semantics.


File: monotone.info,  Node: Regexp Summary,  Next: Regexp Details,  Up: Regexps

7.4.1 Regexp Syntax Summary
---------------------------

This is a quick-reference summary of the regular expression syntax used
in Monotone.

Quoting
.......

'\X'
     where X is non-alphanumeric is a literal X
'\Q...\E'
     treat enclosed characters as literal

Characters
..........

'\a'
     alarm, that is, the BEL character (hex 07)
'\cX'
     "control-X", where X is any character
'\e'
     escape (hex 1B)
'\f'
     formfeed (hex 0C)
'\n'
     newline (hex 0A)
'\r'
     carriage return (hex 0D)
'\t'
     tab (hex 09)
'\DDD'
     character with octal code DDD, or backreference
'\xHH'
     character with hex code HH
'\x{HHH...}'
     character with hex code HHH...

Character Types
...............

'.'
     any character except newline; in dotall mode, any character
     whatsoever
'\C'
     one byte, even in UTF-8 mode (best avoided)
'\d'
     a decimal digit
'\D'
     a character that is not a decimal digit
'\h'
     a horizontal whitespace character
'\H'
     a character that is not a horizontal whitespace character
'\p{XX}'
     a character with the XX property
'\P{XX}'
     a character without the XX property
'\R'
     a newline sequence
'\s'
     a whitespace character
'\S'
     a character that is not a whitespace character
'\v'
     a vertical whitespace character
'\V'
     a character that is not a vertical whitespace character
'\w'
     a "word" character
'\W'
     a "non-word" character
'\X'
     an extended Unicode sequence

'\d', '\D', '\s', '\S', '\w', and '\W' recognize only ASCII characters.

General category property codes for '\p' and '\P'
.................................................

'C'
     Other
'Cc'
     Control
'Cf'
     Format
'Cn'
     Unassigned
'Co'
     Private use
'Cs'
     Surrogate

'L'
     Letter
'Ll'
     Lower case letter
'Lm'
     Modifier letter
'Lo'
     Other letter
'Lt'
     Title case letter
'Lu'
     Upper case letter
'L&'
     Ll, Lu, or Lt

'M'
     Mark
'Mc'
     Spacing mark
'Me'
     Enclosing mark
'Mn'
     Non-spacing mark

'N'
     Number
'Nd'
     Decimal number
'Nl'
     Letter number
'No'
     Other number

'P'
     Punctuation
'Pc'
     Connector punctuation
'Pd'
     Dash punctuation
'Pe'
     Close punctuation
'Pf'
     Final punctuation
'Pi'
     Initial punctuation
'Po'
     Other punctuation
'Ps'
     Open punctuation

'S'
     Symbol
'Sc'
     Currency symbol
'Sk'
     Modifier symbol
'Sm'
     Mathematical symbol
'So'
     Other symbol

'Z'
     Separator
'Zl'
     Line separator
'Zp'
     Paragraph separator
'Zs'
     Space separator

Script names for '\p' and '\P'
..............................

Arabic, Armenian, Balinese, Bengali, Bopomofo, Braille, Buginese, Buhid,
Canadian_Aboriginal, Cherokee, Common, Coptic, Cuneiform, Cypriot,
Cyrillic, Deseret, Devanagari, Ethiopic, Georgian, Glagolitic, Gothic,
Greek, Gujarati, Gurmukhi, Han, Hangul, Hanunoo, Hebrew, Hiragana,
Inherited, Kannada, Katakana, Kharoshthi, Khmer, Lao, Latin, Limbu,
Linear_B, Malayalam, Mongolian, Myanmar, New_Tai_Lue, Nko, Ogham,
Old_Italic, Old_Persian, Oriya, Osmanya, Phags_Pa, Phoenician, Runic,
Shavian, Sinhala, Syloti_Nagri, Syriac, Tagalog, Tagbanwa, Tai_Le,
Tamil, Telugu, Thaana, Thai, Tibetan, Tifinagh, Ugaritic, Yi.

Character Classes
.................

'[...]'
     positive character class
'[^...]'
     negative character class
'[X-Y]'
     range (can be used for hex characters)
'[[:XXX:]]'
     positive POSIX named set
'[[:^XXX:]]'
     negative POSIX named set

'alnum'
     alphanumeric
'alpha'
     alphabetic
'ascii'
     0-127
'blank'
     space or tab
'cntrl'
     control character
'digit'
     decimal digit
'graph'
     printing, excluding space
'lower'
     lower case letter
'print'
     printing, including space
'punct'
     printing, excluding alphanumeric
'space'
     whitespace
'upper'
     upper case letter
'word'
     same as '\w'
'xdigit'
     hexadecimal digit

   In PCRE, POSIX character set names recognize only ASCII characters.
You can use '\Q...\E' inside a character class.

Quantifiers
...........

'?'
     0 or 1, greedy
'?+'
     0 or 1, possessive
'??'
     0 or 1, lazy
'*'
     0 or more, greedy
'*+'
     0 or more, possessive
'*?'
     0 or more, lazy
'+'
     1 or more, greedy
'++'
     1 or more, possessive
'+?'
     1 or more, lazy
'{N}'
     exactly N
'{N,M}'
     at least N, no more than M, greedy
'{N,M}+'
     at least N, no more than M, possessive
'{N,M}?'
     at least N, no more than M, lazy
'{N,}'
     N or more, greedy
'{N,}+'
     N or more, possessive
'{N,}?'
     N or more, lazy

Anchors and Simple Assertions
.............................

'\b'
     word boundary
'\B'
     not a word boundary
'^'
     start of subject also after internal newline in multiline mode
'\A'
     start of subject
'$'
     end of subject also before newline at end of subject also before
     internal newline in multiline mode
'\Z'
     end of subject also before newline at end of subject
'\z'
     end of subject
'\G'
     first matching position in subject

Match Point Reset
.................

'\K'
     reset start of match

Alternation
...........

'EXPR|EXPR|EXPR...'

Capturing
.........

'(...)'
     capturing group
'(?<NAME>...)'
     named capturing group (like Perl)
'(?'NAME'...)'
     named capturing group (like Perl)
'(?P<NAME>...)'
     named capturing group (like Python)
'(?:...)'
     non-capturing group
'(?|...)'
     non-capturing group; reset group numbers for capturing groups in
     each alternative

Atomic Groups
.............

'(?>...)'
     atomic, non-capturing group

Comment
.......

'(?#....)'
     comment (not nestable)

Option Setting
..............

'(?i)'
     caseless
'(?J)'
     allow duplicate names
'(?m)'
     multiline
'(?s)'
     single line (dotall)
'(?U)'
     default ungreedy (lazy)
'(?x)'
     extended (ignore white space)
'(?-...)'
     unset option(s)

Lookahead and Lookbehind Assertions
...................................

'(?=...)'
     positive look ahead
'(?!...)'
     negative look ahead
'(?<=...)'
     positive look behind
'(?<!...)'
     negative look behind

   Each top-level branch of a look behind must be of a fixed length.

Backreferences
..............

'\N'
     reference by number (can be ambiguous)
'\gN'
     reference by number
'\g{N}'
     reference by number
'\g{-N}'
     relative reference by number
'\k<NAME>'
     reference by name (like Perl)
'\k'NAME''
     reference by name (like Perl)
'\g{NAME}'
     reference by name (like Perl)
'\k{NAME}'
     reference by name (like .NET)
'(?P=NAME)'
     reference by name (like Python)

Subroutine References (possibly recursive)
..........................................

'(?R)'
     recurse whole pattern
'(?N)'
     call subpattern by absolute number
'(?+N)'
     call subpattern by relative number
'(?-N)'
     call subpattern by relative number
'(?&NAME)'
     call subpattern by name (like Perl)
'(?P>NAME)'
     call subpattern by name (like Python)

Conditional Patterns
....................

'(?(CONDITION)YES-PATTERN)'
'(?(CONDITION)YES-PATTERN|NO-PATTERN)'

'(?(N)...'
     absolute reference condition
'(?(+N)...'
     relative reference condition
'(?(-N)...'
     relative reference condition
'(?(<NAME>)...'
     named reference condition (like Perl)
'(?('NAME')...'
     named reference condition (like Perl)
'(?(NAME)...'
     named reference condition (PCRE only)
'(?(R)...'
     overall recursion condition
'(?(RN)...'
     specific group recursion condition
'(?(R&NAME)...'
     specific recursion condition
'(?(DEFINE)...'
     define subpattern for reference
'(?(assert)...'
     assertion condition

Backtracking Control
....................

The following act immediately they are reached:

'(*ACCEPT)'
     force successful match
'(*FAIL)'
     force backtrack; synonym '(*F)'

   The following act only when a subsequent match failure causes a
backtrack to reach them.  They all force a match failure, but they
differ in what happens afterwards.  Those that advance the
start-of-match point do so only if the pattern is not anchored.

'(*COMMIT)'
     overall failure, no advance of starting point
'(*PRUNE)'
     advance to next starting character
'(*SKIP)'
     advance start to current matching position
'(*THEN)'
     local failure, backtrack to next alternation

Newline Conventions
...................

These are recognized only at the very start of the pattern or after a
'(*BSR_...)' option.

'(*CR)'
'(*LF)'
'(*CRLF)'
'(*ANYCRLF)'
'(*ANY)'

What '\R' Matches
.................

These are recognized only at the very start of the pattern or after a
'(*...)' option that sets the newline convention.

'(*BSR_ANYCRLF)'
'(*BSR_UNICODE)'


File: monotone.info,  Node: Regexp Details,  Prev: Regexp Summary,  Up: Regexps

7.4.2 Regexp Details
--------------------

The syntax and semantics of PCRE regular expressions, as used in
Monotone, are described in detail below.  Regular expressions in general
are covered in a number of books, some of which have copious examples.
Jeffrey Friedl's "Mastering Regular Expressions," published by O'Reilly,
covers regular expressions in great detail.  This description is
intended as reference material.

Characters and Metacharacters
.............................

A regular expression is a pattern that is matched against a subject
string from left to right.  Most characters stand for themselves in a
pattern, and match the corresponding characters in the subject.  As a
trivial example, the pattern

         The quick brown fox

matches a portion of a subject string that is identical to itself.  When
caseless matching is specified, letters are matched independently of
case.

   The power of regular expressions comes from the ability to include
alternatives and repetitions in the pattern.  These are encoded in the
pattern by the use of "metacharacters", which do not stand for
themselves but instead are interpreted in some special way.

   There are two different sets of metacharacters: those that are
recognized anywhere in the pattern except within square brackets, and
those that are recognized within square brackets.  Outside square
brackets, the metacharacters are as follows:

'\'
     general escape character with several uses
'^'
     assert start of string (or line, in multiline mode)
'$'
     assert end of string (or line, in multiline mode)
'.'
     match any character except newline (by default)
'['
     start character class definition
'|'
     start of alternative branch
'('
     start subpattern
')'
     end subpattern
'?'
     extends the meaning of '(' also 0 or 1 quantifier also quantifier
     minimizer
'*'
     0 or more quantifier
'+'
     1 or more quantifier also "possessive quantifier"
'{'
     start min/max quantifier

Part of a pattern that is in square brackets is called a "character
class".  In a character class the only metacharacters are:

'\'
     general escape character
'^'
     negate the class, but only if the first character
'-'
     indicates character range
'['
     POSIX character class (only if followed by POSIX syntax)
']'
     terminates the character class

The following sections describe the use of each of the metacharacters.

Backslash
.........

The backslash character has several uses.  Firstly, if it is followed by
a non-alphanumeric character, it takes away any special meaning that
character may have.  This use of backslash as an escape character
applies both inside and outside character classes.

   For example, if you want to match a '*' character, you write '\*' in
the pattern.  This escaping action applies whether or not the following
character would otherwise be interpreted as a metacharacter, so it is
always safe to precede a non-alphanumeric with backslash to specify that
it stands for itself.  In particular, if you want to match a backslash,
you write '\\'.

   If a pattern is compiled with the '(?x)' option, whitespace in the
pattern (other than in a character class) and characters between a '#'
outside a character class and the next newline are ignored.  An escaping
backslash can be used to include a whitespace or '#' character as part
of the pattern.

   If you want to remove the special meaning from a sequence of
characters, you can do so by putting them between '\Q' and '\E'.  The
'\Q...\E' sequence is recognized both inside and outside character
classes.

Non-printing Characters
.......................

A second use of backslash provides a way of encoding non-printing
characters in patterns in a visible manner.  There is no restriction on
the appearance of non-printing characters, apart from the binary zero
that terminates a pattern, but when a pattern is being prepared by text
editing, it is usually easier to use one of the following escape
sequences than the binary character it represents:

'\a'
     alarm, that is, the BEL character (hex 07)
'\cX'
     "control-X", where X is any character
'\e'
     escape (hex 1B)
'\f'
     formfeed (hex 0C)
'\n'
     linefeed (hex 0A)
'\r'
     carriage return (hex 0D)
'\t'
     tab (hex 09)
'\DDD'
     character with octal code DDD, or backreference
'\xHH'
     character with hex code HH
'\x{HHH...}'
     character with hex code HHH...

   The precise effect of '\cX' is as follows: if X is a lower case
letter, it is converted to upper case.  Then bit 6 of the character (hex
40) is inverted.  Thus '\cz' becomes hex 1A (the <SUB> control
character, in ASCII), but '\c{' becomes hex 3B (';'), and '\c;' becomes
hex 7B ('{').

   After '\x', from zero to two hexadecimal digits are read (letters can
be in upper or lower case).  Any number of hexadecimal digits may appear
between '\x{' and '}', but the value of the character code must be less
than 256 in non-UTF-8 mode, and less than 2**31 in UTF-8 mode.  That is,
the maximum value in hexadecimal is 7FFFFFFF. Note that this is bigger
than the largest Unicode code point, which is 10FFFF.

   If characters other than hexadecimal digits appear between '\x{' and
'}', or if there is no terminating '}', this form of escape is not
recognized.  Instead, the initial '\x' will be interpreted as a basic
hexadecimal escape, with no following digits, giving a character whose
value is zero.

   Characters whose value is less than 256 can be defined by either of
the two syntaxes for '\x'.  There is no difference in the way they are
handled.  For example, '\xdc' is exactly the same as '\x{dc}'.

   After '\0' up to two further octal digits are read.  If there are
fewer than two digits, just those that are present are used.  Thus the
sequence '\0\x\07' specifies two binary zeros followed by a <BEL>
character (octal 007).  Make sure you supply two digits after the
initial zero if the pattern character that follows is itself an octal
digit.

   The handling of a backslash followed by a digit other than 0 is
complicated.  Outside a character class, PCRE reads it and any following
digits as a decimal number.  If the number is less than 10, or if there
have been at least that many previous capturing left parentheses in the
expression, the entire sequence is taken as a "back reference".  A
description of how this works is given later, following the discussion
of parenthesized subpatterns.

   Inside a character class, or if the decimal number is greater than 9
and there have not been that many capturing subpatterns, PCRE re-reads
up to three octal digits following the backslash, and uses them to
generate a data character.  Any subsequent digits stand for themselves.
In non-UTF-8 mode, the value of a character specified in octal must be
less than '\400'.  In UTF-8 mode, values up to '\777' are permitted.
For example:

'\040'
     is another way of writing a space
'\40'
     is the same, provided there are fewer than 40 previous capturing
     subpatterns
'\7'
     is always a back reference
'\11'
     might be a back reference, or another way of writing a tab
'\011'
     is always a tab
'\0113'
     is a tab followed by the character '3'
'\113'
     might be a back reference, otherwise the character with octal code
     113
'\377'
     might be a back reference, otherwise the byte consisting entirely
     of 1 bits
'\81'
     is either a back reference, or a binary zero followed by the two
     characters '8' and '1'

   Note that octal values of 100 or greater must not be introduced by a
leading zero, because no more than three octal digits are ever read.

   All the sequences that define a single character value can be used
both inside and outside character classes.  In addition, inside a
character class, the sequence '\b' is interpreted as the <BS> character
(hex 08), and the sequences '\R' and '\X' are interpreted as the
characters 'R' and 'X', respectively.  Outside a character class, these
sequences have different meanings (see below).

Absolute and Relative Back References
.....................................

The sequence '\g' followed by an unsigned or a negative number,
optionally enclosed in braces, is an absolute or relative back
reference.  A named back reference can be coded as '\g{name}'.  Back
references are discussed later, following the discussion of
parenthesized subpatterns.

Generic character types
.......................

Another use of backslash is for specifying generic character types.  The
following are always recognized:

'\d'
     any decimal digit
'\D'
     any character that is not a decimal digit
'\h'
     any horizontal whitespace character
'\H'
     any character that is not a horizontal whitespace character
'\s'
     any whitespace character
'\S'
     any character that is not a whitespace character
'\v'
     any vertical whitespace character
'\V'
     any character that is not a vertical whitespace character
'\w'
     any "word" character
'\W'
     any "non-word" character

   Each pair of escape sequences partitions the complete set of
characters into two disjoint sets.  Any given character matches one, and
only one, of each pair.

   These character type sequences can appear both inside and outside
character classes.  They each match one character of the appropriate
type.  If the current matching point is at the end of the subject
string, all of them fail, since there is no character to match.

   For compatibility with Perl, '\s' does not match the <VT> character
(code 11).  This makes it different from the the POSIX "space" class.
The '\s' characters are <TAB> (9), <LF> (10), <FF> (12), <CR> (13), and
<SPACE> (32).

   In UTF-8 mode, characters with values greater than 128 never match
'\d', '\s', or '\w', and always match '\D', '\S', and '\W'.  These
sequences retain their original meanings from before UTF-8 support was
available, mainly for efficiency reasons.

   The sequences '\h', '\H', '\v', and '\V' are Perl 5.10 features.  In
contrast to the other sequences, these do match certain high-valued
codepoints in UTF-8 mode.  The horizontal space characters are:

'U+0009'
     Horizontal tab
'U+0020'
     Space
'U+00A0'
     Non-break space
'U+1680'
     Ogham space mark
'U+180E'
     Mongolian vowel separator
'U+2000'
     En quad
'U+2001'
     Em quad
'U+2002'
     En space
'U+2003'
     Em space
'U+2004'
     Three-per-em space
'U+2005'
     Four-per-em space
'U+2006'
     Six-per-em space
'U+2007'
     Figure space
'U+2008'
     Punctuation space
'U+2009'
     Thin space
'U+200A'
     Hair space
'U+202F'
     Narrow no-break space
'U+205F'
     Medium mathematical space
'U+3000'
     Ideographic space

The vertical space characters are:

'U+000A'
     Linefeed
'U+000B'
     Vertical tab
'U+000C'
     Formfeed
'U+000D'
     Carriage return
'U+0085'
     Next line
'U+2028'
     Line separator
'U+2029'
     Paragraph separator

A "word" character is an underscore or any character less than 256 that
is a letter or digit.  The definition of letters and digits is that used
for the "C" locale.

Newline Conventions
...................

PCRE supports five different conventions for indicating line breaks in
strings: a single CR (carriage return) character, a single LF (linefeed)
character, the two-character sequence CRLF, any of the three preceding,
or any Unicode newline sequence.  The default is to match any Unicode
newline sequence.  It is possible to override the default newline
convention by starting a pattern string with one of the following five
sequences:

'(*CR)'
     carriage return
'(*LF)'
     linefeed
'(*CRLF)'
     carriage return, followed by linefeed
'(*ANYCRLF)'
     any of the three above
'(*ANY)'
     all Unicode newline sequences

   For example, the pattern

         (*CR)a.b

changes the convention to CR. That pattern matches 'a\nb' because LF is
no longer a newline.  Note that these special settings, which are not
Perl-compatible, are recognized only at the very start of a pattern, and
that they must be in upper case.  If more than one of them is present,
the last one is used.

   The newline convention does not affect what the '\R' escape sequence
matches.  By default, this is any Unicode newline sequence, for Perl
compatibility.  However, this can be changed; see the description of
'\R' below.  A change of '\R' setting can be combined with a change of
newline convention.

Newline Sequences
.................

Outside a character class, by default, the escape sequence '\R' matches
any Unicode newline sequence.  This is a Perl 5.10 feature.  In
non-UTF-8 mode '\R' is equivalent to the following:

         (?>\r\n|\n|\x0b|\f|\r|\x85)

   This is an example of an "atomic group", details of which are given
below.  This particular group matches either the two-character sequence
<CR> followed by <LF>, or one of the single characters <LF> (linefeed,
'U+000A'), <VT> (vertical tab, 'U+000B'), <FF> (formfeed, 'U+000C'),
<CR> (carriage return, 'U+000D'), or <NEL> (next line, 'U+0085').  The
two-character sequence is treated as a single unit that cannot be split.
In UTF-8 mode, two additional characters whose codepoints are greater
than 255 are added: <LS> (line separator, 'U+2028') and <PS> (paragraph
separator, 'U+2029').

   It is possible to change the meaning of '\R' by starting a pattern
string with one of the following sequences:

'(*BSR_ANYCRLF)'
     <CR>, <LF>, or <CR><LF> only
'(*BSR_UNICODE)'
     any Unicode newline sequence (the default)

   Note that these special settings, which are not Perl-compatible, are
recognized only at the very start of a pattern, and that they must be in
upper case.  If more than one of them is present, the last one is used.
They can be combined with a change of newline convention, for example, a
pattern can start with:

         (*ANY)(*BSR_ANYCRLF)

Inside a character class, '\R' matches the letter 'R'.

Unicode Character Properties
............................

Three additional escape sequences match characters with specific Unicode
properties.  When not in UTF-8 mode, these sequences are of course
limited to testing characters whose codepoints are less than 256, but
they do work in this mode.  The extra escape sequences are:

'\p{XX}'
     a character with the XX property
'\P{XX}'
     a character without the XX property
'\X'
     an extended Unicode sequence

   The property names represented by XX above are limited to the Unicode
script names, the general category properties, and 'Any', which matches
any character (including newline).  Other properties such as
'InMusicalSymbols' are not currently supported by PCRE. Note that
'\P{Any}' does not match any characters, so always causes a match
failure.

   Sets of Unicode characters are defined as belonging to certain
scripts.  A character from one of these sets can be matched using a
script name.  For example:

         \p{Greek}
         \P{Han}

Those that are not part of an identified script are lumped together as
"Common."  The current list of scripts is:

   Arabic, Armenian, Balinese, Bengali, Bopomofo, Braille, Buginese,
Buhid, Canadian_Aboriginal, Cherokee, Common, Coptic, Cuneiform,
Cypriot, Cyrillic, Deseret, Devanagari, Ethiopic, Georgian, Glagolitic,
Gothic, Greek, Gujarati, Gurmukhi, Han, Hangul, Hanunoo, Hebrew,
Hiragana, Inherited, Kannada, Katakana, Kharoshthi, Khmer, Lao, Latin,
Limbu, Linear_B, Malayalam, Mongolian, Myanmar, New_Tai_Lue, Nko, Ogham,
Old_Italic, Old_Persian, Oriya, Osmanya, Phags_Pa, Phoenician, Runic,
Shavian, Sinhala, Syloti_Nagri, Syriac, Tagalog, Tagbanwa, Tai_Le,
Tamil, Telugu, Thaana, Thai, Tibetan, Tifinagh, Ugaritic, Yi.

   Each character has exactly one general category property, specified
by a two-letter abbreviation.  For compatibility with Perl, negation can
be specified by including a circumflex between the opening brace and the
property name.  For example, '\p{^Lu}' is the same as '\P{Lu}'.

   If only one letter is specified with '\p' or '\P', it includes all
the general category properties that start with that letter.  In this
case, in the absence of negation, the curly brackets in the escape
sequence are optional; these two examples have the same effect:

         \p{L}
         \pL

The following general category property codes are supported:

'C'
     Other
'Cc'
     Control
'Cf'
     Format
'Cn'
     Unassigned
'Co'
     Private use
'Cs'
     Surrogate

'L'
     Letter
'Ll'
     Lower case letter
'Lm'
     Modifier letter
'Lo'
     Other letter
'Lt'
     Title case letter
'Lu'
     Upper case letter

'M'
     Mark
'Mc'
     Spacing mark
'Me'
     Enclosing mark
'Mn'
     Non-spacing mark

'N'
     Number
'Nd'
     Decimal number
'Nl'
     Letter number
'No'
     Other number

'P'
     Punctuation
'Pc'
     Connector punctuation
'Pd'
     Dash punctuation
'Pe'
     Close punctuation
'Pf'
     Final punctuation
'Pi'
     Initial punctuation
'Po'
     Other punctuation
'Ps'
     Open punctuation

'S'
     Symbol
'Sc'
     Currency symbol
'Sk'
     Modifier symbol
'Sm'
     Mathematical symbol
'So'
     Other symbol

'Z'
     Separator
'Zl'
     Line separator
'Zp'
     Paragraph separator
'Zs'
     Space separator

   The special property 'L&' is also supported: it matches a character
that has the 'Lu', 'Ll', or 'Lt' property, in other words, a letter that
is not classified as a modifier or "other."

   The 'Cs' (Surrogate) property applies only to characters in the range
'U+D800' to 'U+DFFF'.  Such characters are not valid in UTF-8 strings
(see RFC 3629) and so cannot be tested by PCRE.

   The long synonyms for these properties that Perl supports (such as
'\p{Letter}') are not supported by PCRE, nor is it permitted to prefix
any of these properties with 'Is'.

   No character that is in the Unicode table has the 'Cn' (unassigned)
property.  Instead, this property is assumed for any code point that is
not in the Unicode table.

   Specifying caseless matching does not affect these escape sequences.
For example, '\p{Lu}' always matches only upper case letters.

   The '\X' escape matches any number of Unicode characters that form an
extended Unicode sequence.  '\X' is equivalent to

         (?>\PM\pM*)

   That is, it matches a character without the "mark" property, followed
by zero or more characters with the "mark" property, and treats the
sequence as an atomic group (see below).  Characters with the "mark"
property are typically accents that affect the preceding character.
None of them have codepoints less than 256, so in non-UTF-8 mode '\X'
matches any one character.

   Matching characters by Unicode property is not fast, because PCRE has
to search a structure that contains data for over fifteen thousand
characters.  That is why the traditional escape sequences such as '\d'
and '\w' do not use Unicode properties in PCRE.

Resetting the Match Start
.........................

The escape sequence '\K', which is a Perl 5.10 feature, causes any
previously matched characters not to be included in the final matched
sequence.  For example, the pattern:

         foo\Kbar

matches 'foobar', but reports that it has matched 'bar'.  This feature
is similar to a lookbehind assertion (described below).  However, in
this case, the part of the subject before the real match does not have
to be of fixed length, as lookbehind assertions do.  The use of '\K'
does not interfere with the setting of captured substrings.  For
example, when the pattern

         (foo)\Kbar

matches 'foobar', the first substring is still set to 'foo'.

Simple assertions
.................

The final use of backslash is for certain simple assertions.  An
assertion specifies a condition that has to be met at a particular point
in a match, without consuming any characters from the subject string.
The use of subpatterns for more complicated assertions is described
below.  The backslashed assertions are:

'\b'
     matches at a word boundary
'\B'
     matches when not at a word boundary
'\A'
     matches at the start of the subject
'\Z'
     matches at the end of the subject also matches before a newline at
     the end of the subject
'\z'
     matches only at the end of the subject
'\G'
     matches at the first matching position in the subject

These assertions may not appear in character classes (but note that '\b'
has a different meaning, namely the backspace character, inside a
character class).

   A word boundary is a position in the subject string where the current
character and the previous character do not both match '\w' or '\W'
(i.e. one matches '\w' and the other matches '\W'), or the start or end
of the string if the first or last character matches '\w', respectively.

   The '\A', '\Z', and '\z' assertions differ from the traditional
circumflex and dollar (described in the next section) in that they only
ever match at the very start and end of the subject string, whatever
options are set.  Thus, they are independent of multiline mode.  The
difference between '\Z' and '\z' is that '\Z' matches before a newline
at the end of the string as well as at the very end, whereas '\z'
matches only at the end.

   The '\G' assertion is true only when the current matching position is
at the start point of the match.  As used in Monotone, '\G' is always
equal to '\A'.

Circumflex and Dollar
.....................

Outside a character class, in the default matching mode, the circumflex
character, '^', is an assertion that is true only if the current
matching point is at the start of the subject string.  Inside a
character class, circumflex has an entirely different meaning (see
below).

   Circumflex need not be the first character of the pattern if a number
of alternatives are involved, but it should be the first thing in each
alternative in which it appears if the pattern is ever to match that
branch.  If all possible alternatives start with a circumflex, that is,
if the pattern is constrained to match only at the start of the subject,
it is said to be an "anchored" pattern.  (There are also other
constructs that can cause a pattern to be anchored.)

   A dollar character, '$', is an assertion that is true only if the
current matching point is at the end of the subject string, or
immediately before a newline at the end of the string (by default).
Dollar need not be the last character of the pattern if a number of
alternatives are involved, but it should be the last item in any branch
in which it appears.  Dollar has no special meaning in a character
class.

   The meanings of the circumflex and dollar characters are changed if
the '(?m)' option is set.  When this is the case, a circumflex matches
immediately after internal newlines as well as at the start of the
subject string.  It does not match after a newline that ends the string.
A dollar matches before any newlines in the string, as well as at the
very end, when '(?m)' is set.  When newline is specified as the
two-character sequence <CR><LF>, isolated <CR> and <LF> characters do
not indicate newlines.

   For example, the pattern '^abc$' matches the subject string
'def\nabc' (where '\n' represents a newline) in multiline mode, but not
otherwise.  Consequently, patterns that are anchored in single line mode
because all branches start with ^ are not anchored in multiline mode.

   Note that the sequences '\A', '\Z', and '\z' can be used to match the
start and end of the subject in both modes, and if all branches of a
pattern start with '\A' it is always anchored.

Full Stop (Period, Dot)
.......................

Outside a character class, a dot in the pattern matches any one
character in the subject string except (by default) a character that
signifies the end of a line.  In UTF-8 mode, the matched character may
be more than one byte long.

   When a line ending is defined as a single character, dot never
matches that character; when the two-character sequence <CR><LF> is
used, dot does not match <CR> if it is immediately followed by <LF>, but
otherwise it matches all characters (including isolated <CR>s and
<LF>s).  When any Unicode line endings are being recognized, dot does
not match <CR> or <LF> or any of the other line ending characters.

   The behaviour of dot with regard to newlines can be changed.  If the
'(?s)' option is set, a dot matches any one character, without
exception.  If the two-character sequence <CR><LF> is present in the
subject string, it takes two dots to match it.

   The handling of dot is entirely independent of the handling of
circumflex and dollar, the only relationship being that they both
involve newlines.  Dot has no special meaning in a character class.

Matching a Single Byte
......................

Outside a character class, the escape sequence '\C' matches any one
byte, both in and out of UTF-8 mode.  Unlike a dot, it always matches
any line-ending characters.  The feature is provided in Perl in order to
match individual bytes in UTF-8 mode.  Because it breaks up UTF-8
characters into individual bytes, what remains in the string may be a
malformed UTF-8 string.  For this reason, the '\C' escape sequence is
best avoided.

   PCRE does not allow '\C' to appear in lookbehind assertions
(described below), because in UTF-8 mode this would make it impossible
to calculate the length of the lookbehind.

Square Brackets and Character Classes
.....................................

An opening square bracket introduces a character class, terminated by a
closing square bracket.  A closing square bracket on its own is not
special.  If a closing square bracket is required as a member of the
class, it should be the first data character in the class (after an
initial circumflex, if present) or escaped with a backslash.

   A character class matches a single character in the subject.  In
UTF-8 mode, the character may occupy more than one byte.  A matched
character must be in the set of characters defined by the class, unless
the first character in the class definition is a circumflex, in which
case the subject character must not be in the set defined by the class.
If a circumflex is actually required as a member of the class, ensure it
is not the first character, or escape it with a backslash.

   For example, the character class '[aeiou]' matches any lower case
vowel, while '[^aeiou]' matches any character that is not a lower case
vowel.  Note that a circumflex is just a convenient notation for
specifying the characters that are in the class by enumerating those
that are not.  A class that starts with a circumflex is not an
assertion: it still consumes a character from the subject string, and
therefore it fails if the current pointer is at the end of the string.

   In UTF-8 mode, characters with values greater than 255 can be
included in a class as a literal string of bytes, or by using the '\x{'
escaping mechanism.

   When caseless matching is set, any letters in a class represent both
their upper case and lower case versions, so for example, a caseless
'[aeiou]' matches 'A' as well as 'a', and a caseless [^aeiou] does not
match 'A', whereas a caseful version would.  In UTF-8 mode, PCRE always
understands the concept of case for characters whose values are less
than 128, so caseless matching is always possible.  For characters with
higher values, the concept of case is supported if PCRE is compiled with
Unicode property support, but not otherwise.  If you want to use
caseless matching for characters 128 and above, you must ensure that
PCRE is compiled with Unicode property support as well as with UTF-8
support.

   Characters that might indicate line breaks are never treated in any
special way when matching character classes, whatever line-ending
sequence is in use, and whatever setting of the '(?s)' and '(?m)'
options is used.  A class such as '[^a]' always matches one of these
characters.

   The minus (hyphen) character can be used to specify a range of
characters in a character class.  For example, '[d-m]' matches any
letter between 'd' and 'm', inclusive.  If a minus character is required
in a class, it must be escaped with a backslash or appear in a position
where it cannot be interpreted as indicating a range, typically as the
first or last character in the class.

   It is not possible to have the literal character ']' as the end
character of a range.  A pattern such as '[W-]46]' is interpreted as a
class of two characters ('W' and '-') followed by a literal string
'46]', so it would match 'W46]' or '-46]'.  However, if the ']' is
escaped with a backslash it is interpreted as the end of range, so
'[W-\]46]' is interpreted as a class containing a range followed by two
other characters.  The octal or hexadecimal representation of ']' can
also be used to end a range.

   Ranges operate in the collating sequence of character values.  They
can also be used for characters specified numerically, for example
'[\000-\037]'.  In UTF-8 mode, ranges can include characters whose
values are greater than 255, for example '[\x{100}-\x{2ff}]'.

   If a range that includes letters is used when caseless matching is
set, it matches the letters in either case.  For example, '[W-c]' is
equivalent to '[][\\^_`wxyzabc]', matched caselessly.

   The character types '\d', '\D', '\p', '\P', '\s', '\S', '\w', and
'\W' may also appear in a character class, and add the characters that
they match to the class.  For example, '[\dABCDEF]' matches any
hexadecimal digit.  A circumflex can conveniently be used with the upper
case character types to specify a more restricted set of characters than
the matching lower case type.  For example, the class '[^\W_]' matches
any letter or digit, but not underscore.

   The only metacharacters that are recognized in character classes are
backslash, hyphen (only where it can be interpreted as specifying a
range), circumflex (only at the start), opening square bracket (only
when it can be interpreted as introducing a POSIX class name--see the
next section), and the terminating closing square bracket.  However,
escaping other non-alphanumeric characters does no harm.

POSIX Character Classes
.......................

Perl supports the POSIX notation for character classes.  This uses names
enclosed by '[:' and ':]' within the enclosing square brackets.  PCRE
also supports this notation.  For example,

         [01[:alpha:]%]

matches '0', '1', any alphabetic character, or '%'.  The supported class
names are

'alnum'
     letters and digits
'alpha'
     letters
'ascii'
     character codes 0 - 127
'blank'
     space or tab only
'cntrl'
     control characters
'digit'
     decimal digits (same as '\d')
'graph'
     printing characters, excluding space
'lower'
     lower case letters
'print'
     printing characters, including space
'punct'
     printing characters, excluding letters and digits
'space'
     white space (not quite the same as '\s')
'upper'
     upper case letters
'word'
     "word" characters (same as '\w')
'xdigit'
     hexadecimal digits

   The "space" characters are <HT> (9), <LF> (10), <VT> (11), <FF> (12),
<CR> (13), and space (32).  Notice that this list includes the <VT>
character (code 11).  This makes "space" different to '\s', which does
not include <VT> (for Perl compatibility).

   The name "word" is a Perl extension, and "blank" is a GNU extension
from Perl 5.8.  Another Perl extension is negation, which is indicated
by a '^' character after the colon.  For example,

         [12[:^digit:]]

matches '1', '2', or any non-digit.  PCRE (and Perl) also recognize the
POSIX syntax '[.CH.]' and '[=CH=]' where CH is a "collating element,"
but these are not supported, and an error is given if they are
encountered.

   In UTF-8 mode, characters with values greater than 128 do not match
any of the POSIX character classes.

Vertical Bar
............

Vertical bar characters are used to separate alternative patterns.  For
example, the pattern

         gilbert|sullivan

matches either 'gilbert' or 'sullivan'.  Any number of alternatives may
appear, and an empty alternative is permitted (matching the empty
string).  The matching process tries each alternative in turn, from left
to right, and the first one that succeeds is used.  If the alternatives
are within a subpattern (defined below), "succeeds" means matching the
rest of the main pattern as well as the alternative in the subpattern.

Internal Option Setting
.......................

The behavior of the matching engine can be adjusted from within the
pattern by a sequence of option letters enclosed between '(?' and ')'.
The option letters are

'i'
     Caseless: characters in one case match the corresponding characters
     in other cases as well.
'm'
     Multiline: '^' and '$' match at newlines as well as at beginning
     and end of string.
's'
     Dotall: dot matches any character, including newline characters.
'x'
     Extended syntax: unescaped white space is ignored and embedded
     comments are possible.
'J'
     Dupnames: names for capturing subpattern need not be unique.
'U'
     Ungreedy: quantifiers match as few times as possible by default.
'X'
     Extra: for forward compatibility, give an error if any escape
     sequence with no defined meaning appears.

   For example, '(?im)' sets caseless, multiline matching.  It is also
possible to unset these options by preceding the letters with a hyphen,
and a combined setting and unsetting such as '(?im-sx)' is also
permitted.  (This would set the caseless and multiline options while
unsetting the dotall and extended-syntax options.)  If a letter appears
both before and after the hyphen, the option is unset.  The lowercase
option letters are Perl-compatible; the uppercase ones are PCRE only.

   When an option change occurs at top level (that is, not inside
subpattern parentheses), the change applies to the remainder of the
pattern that follows.  An option change within a subpattern (see below
for a description of subpatterns) affects only that part of the current
pattern that follows it, so

         (a(?i)b)c

matches 'abc' and 'aBc' and no other strings.  By this means, options
can be made to have different settings in different parts of the
pattern.  Any changes made in one alternative do carry on into
subsequent branches within the same subpattern.  For example,

         (a(?i)b|c)

matches 'ab', 'aB', 'c', and 'C', even though when matching 'C' the
first branch is abandoned before the option setting.  This is because
the effects of option settings happen when the pattern is parsed.  There
would be some very weird behaviour otherwise.

   _Note:_ Unlike these options, the similar, PCRE-specific option
sequences that start with '(*' may appear only at the very beginning of
the pattern.  Details of these sequences are given in the section
entitled "Newline sequences," above.

Subpatterns
...........

Subpatterns are delimited by parentheses (round brackets), which can be
nested.  Turning part of a pattern into a subpattern does two things:

  1. It localizes a set of alternatives.  For example, the pattern

              cat(aract|erpillar|)

     matches one of the words 'cat', 'cataract', or 'caterpillar'.
     Without the parentheses, it would match 'cataract', 'erpillar' or
     an empty string.

  2. It sets up the subpattern as a capturing subpattern.  As used in
     Monotone this only means that during matching, the portion of the
     subject string that matched the subpattern is available for back
     references.  Captured subpatterns are, for instance, not available
     to callers of 'regex.search'.  Opening parentheses are counted from
     left to right (starting from 1) to obtain numbers for the capturing
     subpatterns.

     For example, if the string 'the red king' is matched against the
     pattern

              the ((red|white) (king|queen))

     the captured substrings are 'red king', 'red', and 'king', and are
     numbered 1, 2, and 3, respectively.

   The fact that plain parentheses fulfil two functions is not always
helpful.  There are often times when a grouping subpattern is required
without a capturing requirement.  If an opening parenthesis is followed
by a question mark and a colon, the subpattern does not do any
capturing, and is not counted when computing the number of any
subsequent capturing subpatterns.  For example, if the string 'the white
queen' is matched against the pattern

         the ((?:red|white) (king|queen))

the captured substrings are 'white queen' and 'queen', and are numbered
1 and 2.  The maximum number of capturing subpatterns is 65535.

   As a convenient shorthand, if any option settings are required at the
start of a non-capturing subpattern, the option letters may appear
between the '?' and the ':'.  Thus the two patterns

         (?i:saturday|sunday)
         (?:(?i)saturday|sunday)

match exactly the same set of strings.  Because alternative branches are
tried from left to right, and options are not reset until the end of the
subpattern is reached, an option setting in one branch does affect
subsequent branches, so the above patterns match 'SUNDAY' as well as
'Saturday'.

Duplicate Subpattern Numbers
............................

Perl 5.10 introduced a feature whereby each alternative in a subpattern
uses the same numbers for its capturing parentheses.  Such a subpattern
starts with '(?|' and is itself a non-capturing subpattern.  For
example, consider this pattern:

         (?|(Sat)ur|(Sun))day

Because the two alternatives are inside a '(?|' group, both sets of
capturing parentheses are numbered one.  Thus, when the pattern matches,
you can look at captured substring number one, whichever alternative
matched.  This construct is useful when you want to capture part, but
not all, of one of a number of alternatives.  Inside a '(?|' group,
parentheses are numbered as usual, but the number is reset at the start
of each branch.  The numbers of any capturing buffers that follow the
subpattern start after the highest number used in any branch.  The
following example is taken from the Perl documentation.  The numbers
underneath show in which buffer the captured content will be stored.

  # before  ---------------branch-reset----------- after
  / ( a )  (?| x ( y ) z | (p (q) r) | (t) u (v) ) ( z ) /x
  # 1            2         2  3        2     3     4

A backreference or a recursive call to a numbered subpattern always
refers to the first one in the pattern with the given number.

   An alternative approach to using this "branch reset" feature is to
use duplicate named subpatterns, as described in the next section.

Named Subpatterns
.................

Identifying capturing parentheses by number is simple, but it can be
very hard to keep track of the numbers in complicated regular
expressions.  Furthermore, if an expression is modified, the numbers may
change.  To help with this difficulty, PCRE supports the naming of
subpatterns.  This feature was not added to Perl until release 5.10.
Python had the feature earlier, and PCRE introduced it at release 4.0,
using the Python syntax.  PCRE now supports both the Perl and the Python
syntax.

   In PCRE, a subpattern can be named in one of three ways:
'(?<NAME>...)' or '(?'NAME'...)' as in Perl, or '(?P<NAME>...)' as in
Python.  References to capturing parentheses from other parts of the
pattern, such as backreferences, recursion, and conditions, can be made
by name as well as by number.

   Names consist of up to 32 alphanumeric characters and underscores.
Named capturing parentheses are still allocated numbers as well as
names, exactly as if the names were not present.

   By default, a name must be unique within a pattern, but it is
possible to relax this constraint by setting the '(?J)' option.  This
can be useful for patterns where only one instance of the named
parentheses can match.  Suppose you want to match the name of a weekday,
either as a 3-letter abbreviation or as the full name, and in both cases
you want to extract the abbreviation.  This pattern (ignoring the line
breaks) does the job:

         (?Jx)
         (?<DN>Mon|Fri|Sun)(?:day)?|
         (?<DN>Tue)(?:sday)?|
         (?<DN>Wed)(?:nesday)?|
         (?<DN>Thu)(?:rsday)?|
         (?<DN>Sat)(?:urday)?

There are five capturing substrings, but only one is ever set after a
match.  (An alternative way of solving this problem is to use a "branch
reset" subpattern, as described in the previous section.)

Repetition
..........

Repetition is specified by "quantifiers", which can follow any of the
following items:

   * a literal data character
   * the dot metacharacter
   * the '\C' escape sequence
   * the '\X' escape sequence (in UTF-8 mode with Unicode properties)
   * the '\R' escape sequence
   * an escape such as '\d' that matches a single character
   * a character class
   * a back reference (see next section)
   * a parenthesized subpattern (unless it is an assertion)

   The general repetition quantifier specifies a minimum and maximum
number of permitted matches, by giving the two numbers in curly brackets
(braces), separated by a comma.  The numbers must be less than 65536,
and the first must be less than or equal to the second.  For example:

         z{2,4}

matches 'zz', 'zzz', or 'zzzz'.  A closing brace on its own is not a
special character.  If the second number is omitted, but the comma is
present, there is no upper limit; if the second number and the comma are
both omitted, the quantifier specifies an exact number of required
matches.  Thus

         [aeiou]{3,}

matches at least 3 successive vowels, but may match many more, while

         \d{8}

matches exactly 8 digits.  An opening curly bracket that appears in a
position where a quantifier is not allowed, or one that does not match
the syntax of a quantifier, is taken as a literal character.  For
example, '{,6}' is not a quantifier, but a literal string of four
characters.

   In UTF-8 mode, quantifiers apply to UTF-8 characters rather than to
individual bytes.  Thus, for example, '\x{100}{2}' matches two UTF-8
characters, each of which is represented by a two-byte sequence.
Similarly, '\X{3}' matches three Unicode extended sequences, each of
which may be several bytes long (and they may be of different lengths).

   The quantifier '{0}' is permitted, causing the expression to behave
as if the previous item and the quantifier were not present.

   For convenience, the three most common quantifiers have
single-character abbreviations:

'*'
     is equivalent to {0,}
'+'
     is equivalent to {1,}
'?'
     is equivalent to {0,1}

   It is possible to construct infinite loops by following a subpattern
that can match no characters with a quantifier that has no upper limit,
for example:

         (a?)*

Earlier versions of Perl and PCRE used to give an error at compile time
for such patterns.  However, because there are cases where this can be
useful, such patterns are now accepted, but if any repetition of the
subpattern does in fact match no characters, the loop is forcibly
broken.

   By default, the quantifiers are "greedy", that is, they match as much
as possible (up to the maximum number of permitted times), without
causing the rest of the pattern to fail.  The classic example of where
this gives problems is in trying to match comments in C programs.  These
appear between '/*' and '*/', and within the comment, individual '*' and
'/' characters may appear.  An attempt to match C comments by applying
the pattern

         /\*.*\*/

to the string

         /* first comment */  not comment  /* second comment */

fails, because it matches the entire string owing to the greediness of
the '.*' item.

   However, if a quantifier is followed by a question mark, it ceases to
be greedy, and instead matches the minimum number of times possible, so
the pattern

         /\*.*?\*/

does the right thing with the C comments.  The meaning of the various
quantifiers is not otherwise changed, just the preferred number of
matches.  Do not confuse this use of question mark with its use as a
quantifier in its own right.  Because it has two uses, it can sometimes
appear doubled, as in

         \d??\d

which matches one digit by preference, but can match two if that is the
only way the rest of the pattern matches.

   If the '(?U)' option is set (an option that is not available in
Perl), the quantifiers are not greedy by default, but individual ones
can be made greedy by following them with a question mark.  In other
words, it inverts the default behaviour.

   When a parenthesized subpattern is quantified with a minimum repeat
count that is greater than 1 or with a limited maximum, more memory is
required for the compiled pattern, in proportion to the size of the
minimum or maximum.

   If a pattern starts with '.*' or '.{0,}' and the '(?s)' option is
set, thus allowing the dot to match newlines, the pattern is implicitly
anchored, because whatever follows will be tried against every character
position in the subject string, so there is no point in retrying the
overall match at any position after the first.  PCRE normally treats
such a pattern as though it were preceded by '\A'.

   In cases where it is known that the subject string contains no
newlines, it is worth setting '(?s)' in order to obtain this
optimization, or alternatively using '^' or '\A' to indicate anchoring
explicitly.

   However, there is one situation where the optimization cannot be
used.  When .* is inside capturing parentheses that are the subject of a
backreference elsewhere in the pattern, a match at the start may fail
where a later one succeeds.  Consider, for example:

         (.*)abc\1

If the subject is 'xyz123abc123' the match point is the fourth
character.  For this reason, such a pattern is not implicitly anchored.

   When a capturing subpattern is repeated, the value captured is the
substring that matched the final iteration.  For example, after

         (tweedle[dume]{3}\s*)+

has matched 'tweedledum tweedledee' the value of the captured substring
is 'tweedledee'.  However, if there are nested capturing subpatterns,
the corresponding captured values may have been set in previous
iterations.  For example, after

         (a|(b))+

matches 'aba' the value of the second captured substring is 'b'.

Atomic Grouping and Possessive Quantifiers
..........................................

With both maximizing ("greedy") and minimizing ("ungreedy" or "lazy")
repetition, failure of what follows normally causes the repeated item to
be re-evaluated to see if a different number of repeats allows the rest
of the pattern to match.  Sometimes it is useful to prevent this, either
to change the nature of the match, or to cause it fail earlier than it
otherwise might, when the author of the pattern knows there is no point
in carrying on.

   Consider, for example, the pattern '\d+foo' when applied to the
subject line

         123456bar

   After matching all 6 digits and then failing to match 'foo', the
normal action of the matcher is to try again with only 5 digits matching
the '\d+' item, and then with 4, and so on, before ultimately failing.
"Atomic grouping" (a term taken from Jeffrey Friedl's book) provides the
means for specifying that once a subpattern has matched, it is not to be
re-evaluated in this way.

   If we use atomic grouping for the previous example, the matcher gives
up immediately on failing to match 'foo' the first time.  The notation
is a kind of special parenthesis, starting with '(?>' as in this
example:

         (?>\d+)foo

This kind of parenthesis "locks up" the part of the pattern it contains
once it has matched, and a failure further into the pattern is prevented
from backtracking into it.  Backtracking past it to previous items,
however, works as normal.  Atomic grouping subpatterns are not capturing
subpatterns.

   An alternative description is that a subpattern of this type matches
the string of characters that an identical standalone pattern would
match, if anchored at the current point in the subject string.

   Simple cases such as the above example can be thought of as a
maximizing repeat that must swallow everything it can.  So, while both
'\d+' and '\d+?' are prepared to adjust the number of digits they match
in order to make the rest of the pattern match, '(?>\d+)' can only match
an entire sequence of digits.

   Atomic groups in general can of course contain arbitrarily
complicated subpatterns, and can be nested.  However, when the
subpattern for an atomic group is just a single repeated item, as in the
example above, a simpler notation, called a "possessive quantifier", can
be used.  This consists of an additional '+' character following a
quantifier.  Using this notation, the previous example can be rewritten
as

         \d++foo

Note that a possessive quantifier can be used with an entire group, for
example:

         (abc|xyz){2,3}+

Possessive quantifiers are always greedy; the setting of the '(?U)'
option is ignored.  They are a convenient notation for the simpler forms
of atomic group.  However, there is no difference in the meaning of a
possessive quantifier and the equivalent atomic group, though there may
be a performance difference; possessive quantifiers should be slightly
faster.

   The possessive quantifier syntax is an extension to the Perl 5.8
syntax.  Jeffrey Friedl originated the idea (and the name) in the first
edition of his book.  Mike McCloskey liked it, so implemented it when he
built Sun's Java package, and PCRE copied it from there.  It ultimately
found its way into Perl at release 5.10.

   PCRE has an optimization that automatically "possessifies" certain
simple pattern constructs.  For example, the sequence 'A+B' is treated
as 'A++B' because there is no point in backtracking into a sequence of
'A's when 'B' must follow.

   When a pattern contains an unlimited repeat inside a subpattern that
can itself be repeated an unlimited number of times, the use of an
atomic group is the only way to avoid some failing matches taking a very
long time indeed.  The pattern

         (\D+|<\d+>)*[!?]

matches an unlimited number of substrings that either consist of
non-digits, or digits enclosed in '<>', followed by either '!' or '?'.
When it matches, it runs quickly.  However, if it is applied to

         aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

it takes a long time before reporting failure.  This is because the
string can be divided between the internal '\D+' repeat and the external
'*' repeat in a large number of ways, and all have to be tried.  (The
example uses '[!?]' rather than a single character at the end, because
both PCRE and Perl have an optimization that allows for fast failure
when a single character is used.  They remember the last single
character that is required for a match, and fail early if it is not
present in the string.)  If the pattern is changed so that it uses an
atomic group, like this:

         ((?>\D+)|<\d+>)*[!?]

sequences of non-digits cannot be broken, and failure happens quickly.

Back References
...............

Outside a character class, a backslash followed by a digit greater than
0 (and possibly further digits) is a back reference to a capturing
subpattern earlier (that is, to its left) in the pattern, provided there
have been that many previous capturing left parentheses.

   However, if the decimal number following the backslash is less than
10, it is always taken as a back reference, and causes an error only if
there are not that many capturing left parentheses in the entire
pattern.  In other words, the parentheses that are referenced need not
be to the left of the reference for numbers less than 10.  A "forward
back reference" of this type can make sense when a repetition is
involved and the subpattern to the right has participated in an earlier
iteration.

   It is not possible to have a numerical "forward back reference" to a
subpattern whose number is 10 or more using this syntax because a
sequence such as '\50' is interpreted as a character defined in octal.
See the subsection entitled "Non-printing characters" above for further
details of the handling of digits following a backslash.  There is no
such problem when named parentheses are used.  A back reference to any
subpattern is possible using named parentheses (see below).

   Another way of avoiding the ambiguity inherent in the use of digits
following a backslash is to use the '\g' escape sequence, which is a
feature introduced in Perl 5.10.  This escape must be followed by an
unsigned number or a negative number, optionally enclosed in braces.
These examples are all identical:

         (ring), \1
         (ring), \g1
         (ring), \g{1}

   An unsigned number specifies an absolute reference without the
ambiguity that is present in the older syntax.  It is also useful when
literal digits follow the reference.  A negative number is a relative
reference.  Consider this example:

         (abc(def)ghi)\g{-1}

The sequence '\g{-1}' is a reference to the most recently started
capturing subpattern before '\g', that is, is it equivalent to '\2'.
Similarly, '\g{-2}' would be equivalent to '\1'.  The use of relative
references can be helpful in long patterns, and also in patterns that
are created by joining together fragments that contain references within
themselves.

   A back reference matches whatever actually matched the capturing
subpattern in the current subject string, rather than anything matching
the subpattern itself (see "Subpatterns as subroutines" below for a way
of doing that).  So the pattern

         (sens|respons)e and \1ibility

matches 'sense and sensibility' and 'response and responsibility', but
not 'sense and responsibility'.  If caseful matching is in force at the
time of the back reference, the case of letters is relevant.  For
example,

         ((?i)rah)\s+\1

matches 'rah rah' and 'RAH RAH', but not 'RAH rah', even though the
original capturing subpattern is matched caselessly.

   There are several different ways of writing back references to named
subpatterns.  The .NET syntax '\k{name}' and the Perl syntax '\k<name>'
or '\k'name'' are supported, as is the Python syntax (?P=name).  Perl
5.10's unified back reference syntax, in which '\g' can be used for both
numeric and named references, is also supported.  We could rewrite the
above example in any of the following ways:

         (?<p1>(?i)rah)\s+\k<p1>
         (?'p1'(?i)rah)\s+\k{p1}
         (?P<p1>(?i)rah)\s+(?P=p1)
         (?<p1>(?i)rah)\s+\g{p1}

A subpattern that is referenced by name may appear in the pattern before
or after the reference.

   There may be more than one back reference to the same subpattern.  If
a subpattern has not actually been used in a particular match, any back
references to it always fail.  For example, the pattern

         (a|(bc))\2

always fails if it starts to match 'a' rather than 'bc'.  Because there
may be many capturing parentheses in a pattern, all digits following the
backslash are taken as part of a potential back reference number.  If
the pattern continues with a digit character, some delimiter must be
used to terminate the back reference.  If the '(?x)' option is set, this
can be whitespace.  Otherwise an empty comment (see "Comments" below)
can be used.

   A back reference that occurs inside the parentheses to which it
refers fails when the subpattern is first used, so, for example, '(a\1)'
never matches.  However, such references can be useful inside repeated
subpatterns.  For example, the pattern

         (a|b\1)+

matches any number of 'a's and also 'aba', 'ababbaa' etc.  At each
iteration of the subpattern, the back reference matches the character
string corresponding to the previous iteration.  In order for this to
work, the pattern must be such that the first iteration does not need to
match the back reference.  This can be done using alternation, as in the
example above, or by a quantifier with a minimum of zero.

Assertions
..........

An assertion is a test on the characters following or preceding the
current matching point that does not actually consume any characters.
The simple assertions coded as '\b', '\B', '\A', '\G', '\Z', '\z', '^'
and '$' are described above.

   More complicated assertions are coded as subpatterns.  There are two
kinds: those that look ahead of the current position in the subject
string, and those that look behind it.  An assertion subpattern is
matched in the normal way, except that it does not cause the current
matching position to be changed.

   Assertion subpatterns are not capturing subpatterns, and may not be
repeated, because it makes no sense to assert the same thing several
times.  If any kind of assertion contains capturing subpatterns within
it, these are counted for the purposes of numbering the capturing
subpatterns in the whole pattern.  However, substring capturing is
carried out only for positive assertions, because it does not make sense
for negative assertions.

Lookahead Assertions
....................

Lookahead assertions start with '(?=' for positive assertions and '(?!'
for negative assertions.  For example,

         \w+(?=;)

matches a word followed by a semicolon, but does not include the
semicolon in the match, and

         foo(?!bar)

matches any occurrence of 'foo' that is not followed by 'bar'.  Note
that the apparently similar pattern

         (?!foo)bar

does not find an occurrence of 'bar' that is preceded by something other
than 'foo'; it finds any occurrence of 'bar' whatsoever, because the
assertion '(?!foo)' is always true when the next three characters are
'bar'.  A lookbehind assertion is needed to achieve the other effect.

   If you want to force a matching failure at some point in a pattern,
the most convenient way to do it is with '(?!)' because an empty string
always matches, so an assertion that requires there not to be an empty
string must always fail.

Lookbehind Assertions
.....................

Lookbehind assertions start with '(?<=' for positive assertions and
'(?<!' for negative assertions.  For example,

         (?<!foo)bar

matches an occurrence of 'bar' that is not preceded by 'foo'.  The
contents of a lookbehind assertion are restricted such that all the
strings it matches must have a fixed length.  However, if there are
several top-level alternatives, they do not all have to have the same
fixed length.  Thus

         (?<=bullock|donkey)

is permitted, but

         (?<!dogs?|cats?)

causes an error at compile time.  Branches that match different length
strings are permitted only at the top level of a lookbehind assertion.
This is an extension compared with Perl (at least for 5.8), which
requires all branches to match the same length of string.  An assertion
such as

         (?<=ab(c|de))

is not permitted, because its single top-level branch can match two
different lengths, but it is acceptable if rewritten to use two
top-level branches:

         (?<=abc|abde)

   In some cases, the Perl 5.10 escape sequence '\K' (see above) can be
used instead of a lookbehind assertion; this is not restricted to a
fixed-length.

   The implementation of lookbehind assertions is, for each alternative,
to temporarily move the current position back by the fixed length and
then try to match.  If there are insufficient characters before the
current position, the assertion fails.

   PCRE does not allow the '\C' escape (which matches a single byte in
UTF-8 mode) to appear in lookbehind assertions, because it makes it
impossible to calculate the length of the lookbehind.  The '\X' and '\R'
escapes, which can match different numbers of bytes, are also not
permitted.

   Possessive quantifiers can be used in conjunction with lookbehind
assertions to specify efficient matching at the end of the subject
string.  Consider a simple pattern such as

         abcd$

when applied to a long string that does not match.  Because matching
proceeds from left to right, PCRE will look for each 'a' in the subject
and then see if what follows matches the rest of the pattern.  If the
pattern is specified as

         ^.*abcd$

the initial '.*' matches the entire string at first, but when this fails
(because there is no following 'a'), it backtracks to match all but the
last character, then all but the last two characters, and so on.  Once
again the search for 'a' covers the entire string, from right to left,
so we are no better off.  However, if the pattern is written as

         ^.*+(?<=abcd)

there can be no backtracking for the '.*+' item; it can match only the
entire string.  The subsequent lookbehind assertion does a single test
on the last four characters.  If it fails, the match fails immediately.
For long strings, this approach makes a significant difference to the
processing time.

Using multiple assertions
.........................

Several assertions (of any sort) may occur in succession.  For example,

         (?<=\d{3})(?<!999)foo

matches 'foo' preceded by three digits that are not '999'.  Notice that
each of the assertions is applied independently at the same point in the
subject string.  First there is a check that the previous three
characters are all digits, and then there is a check that the same three
characters are not '999'.  This pattern does _not_ match 'foo' preceded
by six characters, the first of which are digits and the last three of
which are not '999'.  For example, it doesn't match '123abcfoo'.  A
pattern to do that is

         (?<=\d{3}...)(?<!999)foo

This time the first assertion looks at the preceding six characters,
checking that the first three are digits, and then the second assertion
checks that the preceding three characters are not '999'.

   Assertions can be nested in any combination.  For example,

         (?<=(?<!foo)bar)baz

matches an occurrence of 'baz' that is preceded by 'bar' which in turn
is not preceded by 'foo', while

         (?<=\d{3}(?!999)...)foo

is another pattern that matches 'foo' preceded by three digits and any
three characters that are not '999'.

Conditional Subpatterns
.......................

It is possible to cause the matching process to obey a subpattern
conditionally or to choose between two alternative subpatterns,
depending on the result of an assertion, or whether a previous capturing
subpattern matched or not.  The two possible forms of conditional
subpattern are

   * (?(CONDITION)YES-PATTERN)
   * (?(CONDITION)YES-PATTERN|NO-PATTERN)

   If the CONDITION is satisfied, the YES-PATTERN is used; otherwise the
NO-PATTERN (if present) is used.  If there are more than two
alternatives in the subpattern, a compile-time error occurs.

   There are four kinds of condition: references to subpatterns,
references to recursion, a pseudo-condition called 'DEFINE', and
assertions.

Checking for a used subpattern by number
........................................

If the text between the parentheses consists of a sequence of digits,
the condition is true if the capturing subpattern of that number has
previously matched.  An alternative notation is to precede the digits
with a plus or minus sign.  In this case, the subpattern number is
relative rather than absolute.  The most recently opened parentheses can
be referenced by '(?(-1)', the next most recent by '(?(-2)', and so on.
In looping constructs it can also make sense to refer to subsequent
groups with constructs such as '(?(+2)'.

   Consider the following pattern, which contains non-significant white
space to make it more readable and to divide it into three parts for
ease of discussion (assume a preceding '(?x)'):

         ( \( )?    [^()]+    (?(1) \) )

   The first part matches an optional opening parenthesis, and if that
character is present, sets it as the first captured substring.  The
second part matches one or more characters that are not parentheses.
The third part is a conditional subpattern that tests whether the first
set of parentheses matched or not.  If they did, that is, if subject
started with an opening parenthesis, the condition is true, and so the
yes-pattern is executed and a closing parenthesis is required.
Otherwise, since no-pattern is not present, the subpattern matches
nothing.  In other words, this pattern matches a sequence of
non-parentheses, optionally enclosed in parentheses.

   If you were embedding this pattern in a larger one, you could use a
relative reference:

         ...other stuff... ( \( )?    [^()]+    (?(-1) \) ) ...

   This makes the fragment independent of the parentheses in the larger
pattern.

Checking for a used subpattern by name
......................................

Perl uses the syntax '(?(<name>)...)' or '(?('name')...)' to test for a
used subpattern by name.  For compatibility with earlier versions of
PCRE, which had this facility before Perl, the syntax '(?(name)...)' is
also recognized.  However, there is a possible ambiguity with this
syntax, because subpattern names may consist entirely of digits.  PCRE
looks first for a named subpattern; if it cannot find one and the name
consists entirely of digits, PCRE looks for a subpattern of that number,
which must be greater than zero.  Using subpattern names that consist
entirely of digits is not recommended.

   Rewriting the above example to use a named subpattern gives this:

         (?<OPEN> \( )?    [^()]+    (?(<OPEN>) \) )

Checking for pattern recursion
..............................

If the condition is the string '(R)', and there is no subpattern with
the name 'R', the condition is true if a recursive call to the whole
pattern or any subpattern has been made.  If digits or a name preceded
by ampersand follow the letter 'R', for example:

         (?(R3)...) or (?(R&name)...)

the condition is true if the most recent recursion is into the
subpattern whose number or name is given.  This condition does not check
the entire recursion stack.

   At "top level," all these recursion test conditions are false.
Recursive patterns are described below.

Defining subpatterns for use by reference only
..............................................

If the condition is the string '(DEFINE)', and there is no subpattern
with the name 'DEFINE', the condition is always false.  In this case,
there may be only one alternative in the subpattern.  It is always
skipped if control reaches this point in the pattern; the idea of DEFINE
is that it can be used to define "subroutines" that can be referenced
from elsewhere.  (The use of subroutines is described below.)  For
example, a pattern to match an IPv4 address could be written like this
(ignore whitespace and line breaks):

         (?(DEFINE) (?<byte> 2[0-4]\d | 25[0-5] | 1\d\d | [1-9]?\d) )
         \b (?&byte) (\.(?&byte)){3} \b

   The first part of the pattern is a DEFINE group inside which a
another group named "byte" is defined.  This matches an individual
component of an IPv4 address (a number less than 256).  When matching
takes place, this part of the pattern is skipped because DEFINE acts
like a false condition.

   The rest of the pattern uses references to the named group to match
the four dot-separated components of an IPv4 address, insisting on a
word boundary at each end.

Assertion conditions
....................

If the condition is not in any of the above formats, it must be an
assertion.  This may be a positive or negative lookahead or lookbehind
assertion.  Consider this pattern, again containing non-significant
white space, and with the two alternatives on the second line:

         (?(?=[^a-z]*[a-z])
         \d{2}-[a-z]{3}-\d{2}  |  \d{2}-\d{2}-\d{2} )

   The condition is a positive lookahead assertion that matches an
optional sequence of non-letters followed by a letter.  In other words,
it tests for the presence of at least one letter in the subject.  If a
letter is found, the subject is matched against the first alternative;
otherwise it is matched against the second.  This pattern matches
strings in one of the two forms 'DD-AAA-DD' or 'DD-DD-DD', where AAA are
letters and DD are digits.

Comments
........

The sequence '(?#' marks the start of a comment that continues up to the
next closing parenthesis.  Nested parentheses are not permitted.  The
characters that make up a comment play no part in the pattern matching
at all.

   If the '(?x)' option is set, an unescaped '#' character outside a
character class introduces a comment that continues to immediately after
the next newline in the pattern.

Recursive Patterns
..................

Consider the problem of matching a string in parentheses, allowing for
unlimited nested parentheses.  Without the use of recursion, the best
that can be done is to use a pattern that matches up to some fixed depth
of nesting.  It is not possible to handle an arbitrary nesting depth.

   PCRE supports special syntax for recursion of the entire pattern, and
also for individual subpattern recursion.  After its introduction in
PCRE and Python, this kind of recursion was introduced into Perl at
release 5.10.

   A special item that consists of '(?' followed by a number greater
than zero and a closing parenthesis is a recursive call of the
subpattern of the given number, provided that it occurs inside that
subpattern.  (If not, it is a subroutine call, which is described in the
next section.)  The special item '(?R)' or '(?0)' is a recursive call of
the entire regular expression.

   In PCRE (like Python, but unlike Perl), a recursive subpattern call
is always treated as an atomic group.  That is, once it has matched some
of the subject string, it is never re-entered, even if it contains
untried alternatives and there is a subsequent matching failure.

   This PCRE pattern solves the nested parentheses problem (whitespace
is insignificant):

         \( ( (?>[^()]+) | (?R) )* \)

   First it matches an opening parenthesis.  Then it matches any number
of substrings which can either be a sequence of non-parentheses, or a
recursive match of the pattern itself (that is, a correctly
parenthesized substring).  Finally there is a closing parenthesis.

   If this were part of a larger pattern, you would not want to recurse
the entire pattern, so instead you could use this:

         ( \( ( (?>[^()]+) | (?1) )* \) )

We have put the pattern into parentheses, and caused the recursion to
refer to them instead of the whole pattern.

   In a larger pattern, keeping track of parenthesis numbers can be
tricky.  This is made easier by the use of relative references.  (A Perl
5.10 feature.)  Instead of '(?1)' in the pattern above you can write
'(?-2)' to refer to the second most recently opened parentheses
preceding the recursion.  In other words, a negative number counts
capturing parentheses leftwards from the point at which it is
encountered.

   It is also possible to refer to subsequently opened parentheses, by
writing references such as '(?+2)'.  However, these cannot be recursive
because the reference is not inside the parentheses that are referenced.
They are always subroutine calls, as described in the next section.

   An alternative approach is to use named parentheses instead.  The
Perl syntax for this is '(?&name)'; PCRE's earlier syntax '(?P>name)' is
also supported.  We could rewrite the above example as follows:

         (?<pn> \( ( (?>[^()]+) | (?&pn) )* \) )

If there is more than one subpattern with the same name, the earliest
one is used.

   This particular example pattern that we have been looking at contains
nested unlimited repeats, and so the use of atomic grouping for matching
strings of non-parentheses is important when applying the pattern to
strings that do not match.  For example, when this pattern is applied to

         (aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa()

it fails quickly.  However, if atomic grouping is not used, the match
runs for a very long time indeed because there are so many different
ways the '+' and '*' repeats can carve up the subject, and all have to
be tested before failure can be reported.

   At the end of a match, the values set for any capturing subpatterns
are those from the outermost level of the recursion at which the
subpattern value is set.  If the pattern above is matched against

         (ab(cd)ef)

the value for the capturing parentheses is 'ef', which is the last value
taken on at the top level.  If additional parentheses are added, giving

         \( ( ( (?>[^()]+) | (?R) )* ) \)
            ^                        ^

the string they capture is 'ab(cd)ef', the contents of the top level
parentheses.

   Do not confuse the '(?R)' item with the condition '(?(R)', which
tests for recursion.  Consider this pattern, which matches text in angle
brackets, allowing for arbitrary nesting.  Only digits are allowed in
nested brackets (that is, when recursing), whereas any characters are
permitted at the outer level.

         < (?: (?(R) \d++  | [^<>]*+) | (?R)) * >

In this pattern, '(?(R)' is the start of a conditional subpattern, with
two different alternatives for the recursive and non-recursive cases.
The '(?R)' item is the actual recursive call.

Subpatterns as Subroutines
..........................

If the syntax for a recursive subpattern reference (either by number or
by name) is used outside the parentheses to which it refers, it operates
like a subroutine in a programming language.  The called subpattern may
be defined before or after the reference.  A numbered reference can be
absolute or relative, as in these examples:

         (...(absolute)...)...(?2)...
         (...(relative)...)...(?-1)...
         (...(?+1)...(relative)...

   An earlier example pointed out that the pattern

         (sens|respons)e and \1ibility

matches 'sense and sensibility' and 'response and responsibility', but
not 'sense and responsibility'.  If instead the pattern

         (sens|respons)e and (?1)ibility

is used, it does match 'sense and responsibility' as well as the other
two strings.  Another example is given in the discussion of DEFINE
above.

   Like recursive subpatterns, a subroutine call is always treated as an
atomic group.  That is, once it has matched some of the subject string,
it is never re-entered, even if it contains untried alternatives and
there is a subsequent matching failure.

   When a subpattern is used as a subroutine, processing options such as
case-independence are fixed when the subpattern is defined.  They cannot
be changed for different calls.  For example, consider this pattern:

         (abc)(?i:(?-1))

   It matches 'abcabc'.  It does not match 'abcABC' because the change
of processing option does not affect the called subpattern.

Backtracking Control
....................

Perl 5.10 introduced a number of special "backtracking control verbs",
which are described in the Perl documentation as "experimental and
subject to change or removal in a future version of Perl."  It goes on
to say: "Their usage in production code should be noted to avoid
problems during upgrades."  The same remarks apply to the PCRE features
described in this section.

   The new verbs make use of what was previously invalid syntax: an
opening parenthesis followed by an asterisk.  In Perl, they are
generally of the form '(*VERB:ARG)' but PCRE does not support the use of
arguments, so its general form is just '(*VERB)'.  Any number of these
verbs may occur in a pattern.  There are two kinds:

Verbs that act immediately
..........................

The following verbs act as soon as they are encountered:

'(*ACCEPT)'

     This verb causes the match to end successfully, skipping the
     remainder of the pattern.  When inside a recursion, only the
     innermost pattern is ended immediately.  PCRE differs from Perl in
     what happens if the '(*ACCEPT)' is inside capturing parentheses.
     In Perl, the data so far is captured: in PCRE no data is captured.
     For example:

              A(A|B(*ACCEPT)|C)D

     This matches 'AB', 'AAD', or 'ACD', but when it matches 'AB', no
     data is captured.

'(*FAIL) or (*F)'

     This verb causes the match to fail, forcing backtracking to occur.
     It is equivalent to '(?!)' but easier to read.  It is not clear
     whether there is any use for this without the ability to execute
     code in the middle of the pattern (which Perl has but PCRE in
     Monotone does not).

Verbs that act after backtracking
.................................

The following verbs do nothing when they are encountered.  Matching
continues with what follows, but if there is no subsequent match, a
failure is forced.  The verbs differ in exactly what kind of failure
occurs.

'(*COMMIT)'

     This verb causes the whole match to fail outright if the rest of
     the pattern does not match.  Even if the pattern is unanchored, no
     further attempts to find a match by advancing the start point take
     place.  Once '(*COMMIT)' has been passed, the regular expression
     engine is committed to finding a match at the current starting
     point, or not at all.  For example:

              a+(*COMMIT)b

     This matches 'xxaab' but not 'aacaab'.  It can be thought of as a
     kind of dynamic anchor, or "I've started, so I must finish."

'(*PRUNE)'

     This verb causes the match to fail at the current position if the
     rest of the pattern does not match.  If the pattern is unanchored,
     the normal "bump-along" advance to the next starting character then
     happens.  Backtracking can occur as usual to the left of
     '(*PRUNE)', or when matching to the right of '(*PRUNE)', but if
     there is no match to the right, backtracking cannot cross
     '(*PRUNE)'.  In simple cases, the use of '(*PRUNE)' is just an
     alternative to an atomic group or possessive quantifier, but there
     are some uses of '(*PRUNE)' that cannot be expressed in any other
     way.

'(*SKIP)'

     This verb is like '(*PRUNE)', except that if the pattern is
     unanchored, the "bumpalong" advance is not to the next character,
     but to the position in the subject where '(*SKIP)' was encountered.
     '(*SKIP)' signifies that whatever text was matched leading up to it
     cannot be part of a successful match.  Consider:

              a+(*SKIP)b

     If the subject is 'aaaac...', after the first match attempt fails
     (starting at the first character in the string), the starting point
     skips on to start the next attempt at 'c'.  Note that a possessive
     quantifer does not have the same effect in this example; although
     it would suppress backtracking during the first match attempt, the
     second attempt would start at the second character instead of
     skipping on to 'c'.

'(*THEN)'

     This verb causes a skip to the next alternation if the rest of the
     pattern does not match.  That is, it cancels pending backtracking,
     but only within the current alternation.  Its name comes from the
     observation that it can be used for a pattern-based if-then-else
     block:

              ( COND1 (*THEN) FOO
              | COND2 (*THEN) BAR
              | COND3 (*THEN) BAZ ) ...

     If the 'COND1' pattern matches, 'FOO' is tried (and possibly
     further items after the end of the group if 'FOO' succeeds); on
     failure the matcher skips to the second alternative and tries
     'COND2', without backtracking into COND1.  If '(*THEN)' is used
     outside of any alternation, it acts exactly like '(*PRUNE)'.


File: monotone.info,  Node: Default hooks,  Next: General Index,  Prev: Special Topics,  Up: Top

Appendix A Default hooks
************************

This section contains the entire source code of the standard hook file,
that is built in to the monotone executable, and read before any user
hooks files (unless '--no-builtin-rcfiles' is passed).  It contains the
default values for all hooks.  See *note rcfiles::.

-- Copyright (C) 2003 Graydon Hoare <graydon@pobox.com>
--
-- This program is made available under the GNU GPL version 2.0 or
-- greater. See the accompanying file COPYING for details.
--
-- This program is distributed WITHOUT ANY WARRANTY; without even the
-- implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
-- PURPOSE.

-- this is the standard set of lua hooks for monotone;
-- user-provided files can override it or add to it.

-- Since Lua 5.2, unpack and loadstrings are deprecated and are either moved
-- to table.unpack() or replaced by load(). If lua was compiled without
-- LUA_COMPAT_UNPACK and/or LUA_COMPAT_LOADSTRING, these two are not
-- available and we add a similar compatibility layer, ourselves.
unpack = unpack or table.unpack
loadstring = loadstring or load

function temp_file(namehint, filemodehint)
   local tdir
   tdir = os.getenv("TMPDIR")
   if tdir == nil then tdir = os.getenv("TMP") end
   if tdir == nil then tdir = os.getenv("TEMP") end
   if tdir == nil then tdir = "/tmp" end
   local filename
   if namehint == nil then
      filename = string.format("%s/mtn.XXXXXX", tdir)
   else
      filename = string.format("%s/mtn.%s.XXXXXX", tdir, namehint)
   end
   local filemode
   if filemodehint == nil then
      filemode = "r+"
   else
      filemode = filemodehint
   end
   local name = mkstemp(filename)
   local file = io.open(name, filemode)
   return file, name
end

function execute(path, ...)
   local pid
   local ret = -1
   pid = spawn(path, ...)
   if (pid ~= -1) then ret, pid = wait(pid) end
   return ret
end

function execute_redirected(stdin, stdout, stderr, path, ...)
   local pid
   local ret = -1
   io.flush();
   pid = spawn_redirected(stdin, stdout, stderr, path, ...)
   if (pid ~= -1) then ret, pid = wait(pid) end
   return ret
end

-- Wrapper around execute to let user confirm in the case where a subprocess
-- returns immediately
-- This is needed to work around some brokenness with some merge tools
-- (e.g. on OS X)
function execute_confirm(path, ...)
   ret = execute(path, ...)

   if (ret ~= 0)
   then
      print(gettext("Press enter"))
   else
      print(gettext("Press enter when the subprocess has completed"))
   end
   io.read()
   return ret
end

-- attributes are persistent metadata about files (such as execute
-- bit, ACLs, various special flags) which we want to have set and
-- re-set any time the files are modified. the attributes themselves
-- are stored in the roster associated with the revision. each (f,k,v)
-- attribute triple turns into a call to attr_functions[k](f,v) in lua.

if (attr_init_functions == nil) then
   attr_init_functions = {}
end

attr_init_functions["mtn:execute"] =
   function(filename)
      if (is_executable(filename)) then
        return "true"
      else
        return nil
      end
   end

attr_init_functions["mtn:manual_merge"] =
   function(filename)
      if (binary_file(filename)) then
        return "true" -- binary files must be merged manually
      else
        return nil
      end
   end

if (attr_functions == nil) then
   attr_functions = {}
end

attr_functions["mtn:execute"] =
   function(filename, value)
      if (value == "true") then
         set_executable(filename)
      else
         clear_executable(filename)
      end
   end

function dir_matches(name, dir)
   -- helper for ignore_file, matching files within dir, or dir itself.
   -- eg for dir of 'CVS', matches CVS/, CVS/*, */CVS/ and */CVS/*
   if (string.find(name, "^" .. dir .. "/")) then return true end
   if (string.find(name, "^" .. dir .. "$")) then return true end
   if (string.find(name, "/" .. dir .. "/")) then return true end
   if (string.find(name, "/" .. dir .. "$")) then return true end
   return false
end

function portable_readline(f)
    line = f:read()
    if line ~= nil then
        line = string.gsub(line, "\r$","") -- strip possible \r left from windows editing
    end
    return line
end

function ignore_file(name)
   -- project specific
   if (ignored_files == nil) then
      ignored_files = {}
      local ignfile = io.open(".mtn-ignore", "r")
      if (ignfile ~= nil) then
         local line = portable_readline(ignfile)
         while (line ~= nil) do
            if line ~= "" then
                table.insert(ignored_files, line)
            end
            line = portable_readline(ignfile)
         end
         io.close(ignfile)
      end
   end

   local warn_reported_file = false
   for i, line in pairs(ignored_files)
   do
      if (line ~= nil) then
         local pcallstatus, result = pcall(function()
        return regex.search(line, name)
     end)
         if pcallstatus == true then
            -- no error from the regex.search call
            if result == true then return true end
         else
            -- regex.search had a problem, warn the user their
            -- .mtn-ignore file syntax is wrong
        if not warn_reported_file then
           io.stderr:write("mtn: warning: while matching file '"
                       .. name .. "':\n")
           warn_reported_file = true
        end
        local prefix = ".mtn-ignore:" .. i .. ": warning: "
            io.stderr:write(prefix
                            .. string.gsub(result, "\n", "\n" .. prefix)
                               .. "\n\t- skipping this regex for "
                               .. "all remaining files.\n")
            ignored_files[i] = nil
         end
      end
   end

   local file_pats = {
      -- c/c++
      "%.a$", "%.so$", "%.o$", "%.la$", "%.lo$", "^core$",
      "/core$", "/core%.%d+$",
      -- java
      "%.class$",
      -- python
      "%.pyc$", "%.pyo$",
      -- gettext
      "%.g?mo$",
      -- intltool
      "%.intltool%-merge%-cache$",
      -- TeX
      "%.aux$",
      -- backup files
      "%.bak$", "%.orig$", "%.rej$", "%~$",
      -- vim creates .foo.swp files
      "%.[^/]*%.swp$",
      -- emacs creates #foo# files
      "%#[^/]*%#$",
      -- other VCSes (where metadata is stored in named files):
      "%.scc$",
      -- desktop/directory configuration metadata
      "^%.DS_Store$", "/%.DS_Store$", "^desktop%.ini$", "/desktop%.ini$"
   }

   local dir_pats = {
      -- autotools detritus:
      "autom4te%.cache", "%.deps", "%.libs",
      -- Cons/SCons detritus:
      "%.consign", "%.sconsign",
      -- other VCSes (where metadata is stored in named dirs):
      "CVS", "%.svn", "SCCS", "_darcs", "%.cdv", "%.git", "%.bzr", "%.hg"
   }

   for _, pat in ipairs(file_pats) do
      if string.find(name, pat) then return true end
   end
   for _, pat in ipairs(dir_pats) do
      if dir_matches(name, pat) then return true end
   end

   return false;
end

-- return true means "binary", false means "text",
-- nil means "unknown, try to guess"
function binary_file(name)
   -- some known binaries, return true
   local bin_pats = {
      "%.gif$", "%.jpe?g$", "%.png$", "%.bz2$", "%.gz$", "%.zip$",
      "%.class$", "%.jar$", "%.war$", "%.ear$"
   }

   -- some known text, return false
   local txt_pats = {
      "%.cc?$", "%.cxx$", "%.hh?$", "%.hxx$", "%.cpp$", "%.hpp$",
      "%.lua$", "%.texi$", "%.sql$", "%.java$"
   }

   local lowname=string.lower(name)
   for _, pat in ipairs(bin_pats) do
      if string.find(lowname, pat) then return true end
   end
   for _, pat in ipairs(txt_pats) do
      if string.find(lowname, pat) then return false end
   end

   -- unknown - read file and use the guess-binary
   -- monotone built-in function
   return guess_binary_file_contents(name)
end

-- given a file name, return a regular expression which will match
-- lines that name top-level constructs in that file, or "", to disable
-- matching.
function get_encloser_pattern(name)
   -- texinfo has special sectioning commands
   if (string.find(name, "%.texi$")) then
      -- sectioning commands in texinfo: @node, @chapter, @top,
      -- @((sub)?sub)?section, @unnumbered(((sub)?sub)?sec)?,
      -- @appendix(((sub)?sub)?sec)?, @(|major|chap|sub(sub)?)heading
      return ("^@("
              .. "node|chapter|top"
              .. "|((sub)?sub)?section"
              .. "|(unnumbered|appendix)(((sub)?sub)?sec)?"
              .. "|(major|chap|sub(sub)?)?heading"
              .. ")")
   end
   -- LaTeX has special sectioning commands.  This rule is applied to ordinary
   -- .tex files too, since there's no reliable way to distinguish those from
   -- latex files anyway, and there's no good pattern we could use for
   -- arbitrary plain TeX anyway.
   if (string.find(name, "%.tex$")
       or string.find(name, "%.ltx$")
       or string.find(name, "%.latex$")) then
      return ("\\\\("
              .. "part|chapter|paragraph|subparagraph"
              .. "|((sub)?sub)?section"
              .. ")")
   end
   -- There's no good way to find section headings in raw text, and trying
   -- just gives distracting output, so don't even try.
   if (string.find(name, "%.txt$")
       or string.upper(name) == "README") then
      return ""
   end
   -- This default is correct surprisingly often -- in pretty much any text
   -- written with code-like indentation.
   return "^[[:alnum:]$_]"
end

function edit_comment(user_log_message)
   local exe = nil

   -- top priority is VISUAL, then EDITOR, then a series of hardcoded
   -- defaults, if available.

   local visual = os.getenv("VISUAL")
   local editor = os.getenv("EDITOR")
   if (visual ~= nil) then exe = visual
   elseif (editor ~= nil) then exe = editor
   elseif (program_exists_in_path("editor")) then exe = "editor"
   elseif (program_exists_in_path("vi")) then exe = "vi"
   elseif (string.sub(get_ostype(), 1, 6) ~= "CYGWIN" and
       program_exists_in_path("notepad.exe")) then exe = "notepad"
   else
      io.write(gettext("Could not find editor to enter commit message\n"
               .. "Try setting the environment variable EDITOR\n"))
      return nil
   end

   local tmp, tname = temp_file()
   if (tmp == nil) then return nil end
   tmp:write(user_log_message)
   if user_log_message == "" or string.sub(user_log_message, -1) ~= "\n" then
      tmp:write("\n")
   end
   io.close(tmp)

   -- By historical convention, VISUAL and EDITOR can contain arguments
   -- (and, in fact, arbitrarily complicated shell constructs).  Since Lua
   -- has no word-splitting functionality, we invoke the shell to deal with
   -- anything more complicated than a single word with no metacharacters.
   -- This, unfortunately, means we have to quote the file argument.

   if (not string.find(exe, "[^%w_.+-]")) then
      -- safe to call spawn directly
      if (execute(exe, tname) ~= 0) then
         io.write(string.format(gettext("Error running editor '%s' "..
                                        "to enter log message\n"),
                                exe))
         os.remove(tname)
         return nil
      end
   else
      -- must use shell
      local shell = os.getenv("SHELL")
      if (shell == nil) then shell = "sh" end
      if (not program_exists_in_path(shell)) then
         io.write(string.format(gettext("Editor command '%s' needs a shell, "..
                                        "but '%s' is not to be found"),
                                exe, shell))
         os.remove(tname)
         return nil
      end

      -- Single-quoted strings in both Bourne shell and csh can contain
      -- anything but a single quote.
      local safe_tname = " '" .. string.gsub(tname, "'", "'\\''") .. "'"

      if (execute(shell, "-c", editor .. safe_tname) ~= 0) then
         io.write(string.format(gettext("Error running editor '%s' "..
                                        "to enter log message\n"),
                                exe))
         os.remove(tname)
         return nil
      end
   end

   tmp = io.open(tname, "r")
   if (tmp == nil) then os.remove(tname); return nil end
   local res = tmp:read("*a")
   io.close(tmp)
   os.remove(tname)
   return res
end


function get_local_key_name(key_identity)
   return key_identity.given_name
end


function persist_phrase_ok()
   return true
end


function use_inodeprints()
   return false
end

function get_date_format_spec(wanted)
   -- Return the strftime(3) specification to be used to print dates
   -- in human-readable format after conversion to the local timezone.
   -- The default uses the preferred date and time representation for
   -- the current locale, e.g. the output looks like this: "09/08/2009
   -- 06:49:26 PM" for en_US and "date_time_long", or "08.09.2009"
   -- for de_DE and "date_short"
   --
   -- A sampling of other possible formats you might want:
   --   default for your locale: "%c" (may include a confusing timezone label)
   --   12 hour format: "%d %b %Y, %I:%M:%S %p"
   --   like ctime(3):  "%a %b %d %H:%M:%S %Y"
   --   email style:    "%a, %d %b %Y %H:%M:%S"
   --   ISO 8601:       "%Y-%m-%d %H:%M:%S" or "%Y-%m-%dT%H:%M:%S"
   --
   --   ISO 8601, no timezone conversion: ""
   --.
   if (wanted == "date_long" or wanted == "date_short") then
       return "%x"
   end
   if (wanted == "time_long" or wanted == "time_short") then
       return "%X"
   end
   return "%x %X"
end

-- trust evaluation hooks

function intersection(a,b)
   local s={}
   local t={}
   for k,v in pairs(a) do s[v.name] = 1 end
   for k,v in pairs(b) do if s[v] ~= nil then table.insert(t,v) end end
   return t
end

function get_revision_cert_trust(signers, id, name, val)
   return true
end

-- This is only used by migration from old manifest-style ancestry
function get_manifest_cert_trust(signers, id, name, val)
   return true
end

-- http://snippets.luacode.org/?p=snippets/String_to_Hex_String_68
function hex_dump(str,spacer)
   return (string.gsub(str,"(.)",
      function (c)
         return string.format("%02x%s",string.byte(c), spacer or "")
      end)
   )
end

function accept_testresult_change_hex(old_results, new_results)
   local reqfile = io.open("_MTN/wanted-testresults", "r")
   if (reqfile == nil) then return true end
   local line = reqfile:read()
   local required = {}
   while (line ~= nil)
   do
      required[line] = true
      line = reqfile:read()
   end
   io.close(reqfile)
   for test, res in pairs(required)
   do
      if old_results[test] == true and new_results[test] ~= true
      then
         return false
      end
   end
   return true
end

function accept_testresult_change(old_results, new_results)
   -- Hex encode each of the key hashes to match those in 'wanted-testresults'
   local old_results_hex = {}
   for k, v in pairs(old_results) do
	old_results_hex[hex_dump(k)] = v
   end

   local new_results_hex = {}
   for k, v in pairs(new_results) do
      new_results_hex[hex_dump(k)] = v
   end

   return accept_testresult_change_hex(old_results_hex, new_results_hex)
end

-- merger support

-- Fields in the mergers structure:
-- cmd       : a function that performs the merge operation using the chosen
--             program, best try.
-- available : a function that checks that the needed program is installed and
--             in $PATH
-- wanted    : a function that checks if the user doesn't want to use this
--             method, and returns false if so.  This should normally return
--             true, but in some cases, especially when the merger is really
--             an editor, the user might have a preference in EDITOR and we
--             need to respect that.
--             NOTE: wanted is only used when the user has NOT defined the
--             `merger' variable or the MTN_MERGE environment variable.
mergers = {}

-- This merger is designed to fail if there are any conflicts without trying to resolve them
mergers.fail = {
   cmd = function (tbl) return false end,
   available = function () return true end,
   wanted = function () return true end
}

mergers.meld = {
   cmd = function (tbl)
      io.write(string.format(
        "\nWARNING: 'meld' was chosen to perform an external 3-way merge.\n"..
        "You must merge all changes to the *CENTER* file.\n\n"
      ))
      local path = "meld"
      local ret = execute(path, tbl.lfile, tbl.afile, tbl.rfile)
      if (ret ~= 0) then
         io.write(string.format(gettext("Error running merger '%s'\n"), path))
         return false
      end
      return tbl.afile
   end ,
   available = function () return program_exists_in_path("meld") end,
   wanted = function () return true end
}

mergers.diffuse = {
   cmd = function (tbl)
      io.write(string.format(
        "\nWARNING: 'diffuse' was chosen to perform an external 3-way merge.\n"..
        "You must merge all changes to the *CENTER* file.\n\n"
      ))
      local path = "diffuse"
      local ret = execute(path, tbl.lfile, tbl.afile, tbl.rfile)
      if (ret ~= 0) then
         io.write(string.format(gettext("Error running merger '%s'\n"), path))
         return false
      end
      return tbl.afile
   end ,
   available = function () return program_exists_in_path("diffuse") end,
   wanted = function () return true end
}

mergers.tortoise = {
   cmd = function (tbl)
      local path = "tortoisemerge"
      local ret = execute(path,
                          string.format("/base:%s", tbl.afile),
                          string.format("/theirs:%s", tbl.lfile),
                          string.format("/mine:%s", tbl.rfile),
                          string.format("/merged:%s", tbl.outfile))
      if (ret ~= 0) then
         io.write(string.format(gettext("Error running merger '%s'\n"), path))
         return false
      end
      return tbl.outfile
   end ,
   available = function() return program_exists_in_path ("tortoisemerge") end,
   wanted = function () return true end
}

mergers.vim = {
   cmd = function (tbl)
      function execute_diff3(mine, yours, out)
     local diff3_args = {
        "diff3",
        "--merge",
        "--easy-only",
     }
     table.insert(diff3_args, string.gsub(mine, "\\", "/") .. "")
     table.insert(diff3_args, string.gsub(tbl.afile, "\\", "/") .. "")
     table.insert(diff3_args, string.gsub(yours, "\\", "/") .. "")

     return execute_redirected("", string.gsub(out, "\\", "/"), "", unpack(diff3_args))
      end

      io.write (string.format("\nWARNING: 'vim' was chosen to perform "..
                  "an external 3-way merge.\n"..
                  "You must merge all changes to the "..
                  "*LEFT* file.\n"))

      local vim
      if os.getenv ("DISPLAY") ~= nil and program_exists_in_path ("gvim") then
     vim = "gvim"
      else
     vim = "vim"
      end

      local lfile_merged = tbl.lfile .. ".merged"
      local rfile_merged = tbl.rfile .. ".merged"

      -- first merge lfile using diff3
      local ret = execute_diff3(tbl.lfile, tbl.rfile, lfile_merged)
      if ret == 2 then
         io.write(string.format(gettext("Error running diff3 for merger '%s'\n"), vim))
         os.remove(lfile_merged)
     return false
      end

      -- now merge rfile using diff3
      ret = execute_diff3(tbl.rfile, tbl.lfile, rfile_merged)
      if ret == 2 then
         io.write(string.format(gettext("Error running diff3 for merger '%s'\n"), vim))
         os.remove(lfile_merged)
         os.remove(rfile_merged)
     return false
      end

      os.rename(lfile_merged, tbl.lfile)
      os.rename(rfile_merged, tbl.rfile)

      local ret = execute(vim, "-f", "-d", "-c", string.format("silent file %s", tbl.outfile),
                          tbl.lfile, tbl.rfile)
      if (ret ~= 0) then
         io.write(string.format(gettext("Error running merger '%s'\n"), vim))
         return false
      end
      return tbl.outfile
   end ,
   available =
      function ()
     return program_exists_in_path("diff3") and
            (program_exists_in_path("vim") or
        program_exists_in_path("gvim"))
      end ,
   wanted =
      function ()
     local editor = os.getenv("EDITOR")
     if editor and
        not (string.find(editor, "vim") or
         string.find(editor, "gvim")) then
        return false
     end
     return true
      end
}

mergers.rcsmerge = {
   cmd = function (tbl)
      -- XXX: This is tough - should we check if conflict markers stay or not?
      -- If so, we should certainly give the user some way to still force
      -- the merge to proceed since they can appear in the files (and I saw
      -- that). --pasky
      local merge = os.getenv("MTN_RCSMERGE")
      if execute(merge, tbl.lfile, tbl.afile, tbl.rfile) == 0 then
         copy_text_file(tbl.lfile, tbl.outfile);
         return tbl.outfile
      end
      local ret = execute("vim", "-f", "-c", string.format("file %s", tbl.outfile
),
                          tbl.lfile)
      if (ret ~= 0) then
         io.write(string.format(gettext("Error running merger '%s'\n"), "vim"))
         return false
      end
      return tbl.outfile
   end,
   available =
      function ()
     local merge = os.getenv("MTN_RCSMERGE")
     return merge and
        program_exists_in_path(merge) and program_exists_in_path("vim")
      end ,
   wanted = function () return os.getenv("MTN_RCSMERGE") ~= nil end
}

--  GNU diffutils based merging
mergers.diffutils = {
    --  merge procedure execution
    cmd = function (tbl)
        --  parse options
        local option = {}
        option.partial = false
        option.diff3opts = ""
        option.sdiffopts = ""
        local options = os.getenv("MTN_MERGE_DIFFUTILS")
        if options ~= nil then
            for spec in string.gmatch(options, "%s*(%w[^,]*)%s*,?") do
                local name, value = string.match(spec, "^(%w+)=([^,]*)")
                if name == nil then
                    name = spec
                    value = true
                end
                if type(option[name]) == "nil" then
                    io.write("mtn: " .. string.format(gettext("invalid \"diffutils\" merger option \"%s\""), name) .. "\n")
                    return false
                end
                option[name] = value
            end
        end

        --  determine the diff3(1) command
        local diff3 = {
            "diff3",
            "--merge",
            "--label", string.format("%s [left]",     tbl.left_path ),
            "--label", string.format("%s [ancestor]", tbl.anc_path  ),
            "--label", string.format("%s [right]",    tbl.right_path),
        }
        if option.diff3opts ~= "" then
            for opt in string.gmatch(option.diff3opts, "%s*([^%s]+)%s*") do
                table.insert(diff3, opt)
            end
        end
        table.insert(diff3, string.gsub(tbl.lfile, "\\", "/") .. "")
        table.insert(diff3, string.gsub(tbl.afile, "\\", "/") .. "")
        table.insert(diff3, string.gsub(tbl.rfile, "\\", "/") .. "")

        --  dispatch according to major operation mode
        if option.partial then
            --  partial batch/non-modal 3-way merge "resolution":
            --  simply merge content with help of conflict markers
            io.write("mtn: " .. gettext("3-way merge via GNU diffutils, resolving conflicts via conflict markers") .. "\n")
            local ret = execute_redirected("", string.gsub(tbl.outfile, "\\", "/"), "", unpack(diff3))
            if ret == 2 then
                io.write("mtn: " .. gettext("error running GNU diffutils 3-way difference/merge tool \"diff3\"") .. "\n")
                return false
            end
            return tbl.outfile
        else
            --  real interactive/modal 3/2-way merge resolution:
            --  display 3-way merge conflict and perform 2-way merge resolution
            io.write("mtn: " .. gettext("3-way merge via GNU diffutils, resolving conflicts via interactive prompt") .. "\n")

            --  display 3-way merge conflict (batch)
            io.write("\n")
            io.write("mtn: " .. gettext("---- CONFLICT SUMMARY ------------------------------------------------") .. "\n")
            local ret = execute(unpack(diff3))
            if ret == 2 then
                io.write("mtn: " .. gettext("error running GNU diffutils 3-way difference/merge tool \"diff3\"") .. "\n")
                return false
            end

            --  perform 2-way merge resolution (interactive)
            io.write("\n")
            io.write("mtn: " .. gettext("---- CONFLICT RESOLUTION ---------------------------------------------") .. "\n")
            local sdiff = {
                "sdiff",
                "--diff-program=diff",
                "--suppress-common-lines",
                "--minimal",
                "--output=" .. string.gsub(tbl.outfile, "\\", "/")
            }
            if option.sdiffopts ~= "" then
                for opt in string.gmatch(option.sdiffopts, "%s*([^%s]+)%s*") do
                    table.insert(sdiff, opt)
                end
            end
            table.insert(sdiff, string.gsub(tbl.lfile, "\\", "/") .. "")
            table.insert(sdiff, string.gsub(tbl.rfile, "\\", "/") .. "")
            local ret = execute(unpack(sdiff))
            if ret == 2 then
                io.write("mtn: " .. gettext("error running GNU diffutils 2-way merging tool \"sdiff\"") .. "\n")
                return false
            end
            return tbl.outfile
        end
    end,

    --  merge procedure availability check
    available = function ()
        --  make sure the GNU diffutils tools are available
        return program_exists_in_path("diff3") and
               program_exists_in_path("sdiff") and
               program_exists_in_path("diff");
    end,

    --  merge procedure request check
    wanted = function ()
        --  assume it is requested (if it is available at all)
        return true
    end
}

mergers.emacs = {
   cmd = function (tbl)
      local emacs
      if program_exists_in_path("xemacs") then
         emacs = "xemacs"
      else
         emacs = "emacs"
      end
      local elisp = "(ediff-merge-files-with-ancestor \"%s\" \"%s\" \"%s\" nil \"%s\")"
      -- Converting backslashes is necessary on Win32 MinGW; emacs
      -- lisp string syntax says '\' is an escape.
      local ret = execute(emacs, "--eval",
                          string.format(elisp,
                          string.gsub (tbl.lfile, "\\", "/"),
                          string.gsub (tbl.rfile, "\\", "/"),
                          string.gsub (tbl.afile, "\\", "/"),
                          string.gsub (tbl.outfile, "\\", "/")))
      if (ret ~= 0) then
         io.write(string.format(gettext("Error running merger '%s'\n"), emacs))
         return false
      end
      return tbl.outfile
   end,
   available =
      function ()
     return program_exists_in_path("xemacs") or
        program_exists_in_path("emacs")
      end ,
   wanted =
      function ()
     local editor = os.getenv("EDITOR")
     if editor and
        not (string.find(editor, "emacs") or
         string.find(editor, "gnu")) then
        return false
     end
     return true
      end
}

mergers.xxdiff = {
   cmd = function (tbl)
      local path = "xxdiff"
      local ret = execute(path,
                        "--title1", tbl.left_path,
                        "--title2", tbl.right_path,
                        "--title3", tbl.merged_path,
                        tbl.lfile, tbl.afile, tbl.rfile,
                        "--merge",
                        "--merged-filename", tbl.outfile,
                        "--exit-with-merge-status")
      if (ret ~= 0) then
         io.write(string.format(gettext("Error running merger '%s'\n"), path))
         return false
      end
      return tbl.outfile
   end,
   available = function () return program_exists_in_path("xxdiff") end,
   wanted = function () return true end
}

mergers.kdiff3 = {
   cmd = function (tbl)
      local path = "kdiff3"
      local ret = execute(path,
                          "--L1", tbl.anc_path,
                          "--L2", tbl.left_path,
                          "--L3", tbl.right_path,
                          tbl.afile, tbl.lfile, tbl.rfile,
                          "--merge",
                          "--o", tbl.outfile)
      if (ret ~= 0) then
         io.write(string.format(gettext("Error running merger '%s'\n"), path))
         return false
      end
      return tbl.outfile
   end,
   available = function () return program_exists_in_path("kdiff3") end,
   wanted = function () return true end
}

mergers.opendiff = {
   cmd = function (tbl)
      local path = "opendiff"
      -- As opendiff immediately returns, let user confirm manually
      local ret = execute_confirm(path,
                                  tbl.lfile,tbl.rfile,
                                  "-ancestor",tbl.afile,
                                  "-merge",tbl.outfile)
      if (ret ~= 0) then
         io.write(string.format(gettext("Error running merger '%s'\n"), path))
         return false
      end
      return tbl.outfile
   end,
   available = function () return program_exists_in_path("opendiff") end,
   wanted = function () return true end
}

function write_to_temporary_file(data, namehint, filemodehint)
   tmp, filename = temp_file(namehint, filemodehint)
   if (tmp == nil) then
      return nil
   end;
   tmp:write(data)
   io.close(tmp)
   return filename
end

function copy_text_file(srcname, destname)
   src = io.open(srcname, "r")
   if (src == nil) then return nil end
   dest = io.open(destname, "w")
   if (dest == nil) then return nil end

   while true do
      local line = src:read()
      if line == nil then break end
      dest:write(line, "\n")
   end

   io.close(dest)
   io.close(src)
end

function read_contents_of_file(filename, mode)
   tmp = io.open(filename, mode)
   if (tmp == nil) then
      return nil
   end
   local data = tmp:read("*a")
   io.close(tmp)
   return data
end

function program_exists_in_path(program)
   return existsonpath(program) == 0
end

function get_preferred_merge3_command (tbl)
   local default_order = {"diffuse", "kdiff3", "xxdiff", "opendiff",
                          "tortoise", "emacs", "vim", "meld", "diffutils"}
   local function existmerger(name)
      local m = mergers[name]
      if type(m) == "table" and m.available(tbl) then
         return m.cmd
      end
      return nil
   end
   local function trymerger(name)
      local m = mergers[name]
      if type(m) == "table" and m.available(tbl) and m.wanted(tbl) then
         return m.cmd
      end
      return nil
   end
   -- Check if there's a merger given by the user.
   local mkey = os.getenv("MTN_MERGE")
   if not mkey then mkey = merger end
   if not mkey and os.getenv("MTN_RCSMERGE") then mkey = "rcsmerge" end
   -- If there was a user-given merger, see if it exists.  If it does, return
   -- the cmd function.  If not, return nil.
   local c
   if mkey then c = existmerger(mkey) end
   if c then return c,mkey end
   if mkey then return nil,mkey end
   -- If there wasn't any user-given merger, take the first that's available
   -- and wanted.
   for _,mkey in ipairs(default_order) do
      c = trymerger(mkey) ; if c then return c,mkey end
   end
end

function merge3 (anc_path, left_path, right_path, merged_path, ancestor, left, right)
   local ret = nil
   local tbl = {}

   tbl.anc_path = anc_path
   tbl.left_path = left_path
   tbl.right_path = right_path

   tbl.merged_path = merged_path
   tbl.afile = nil
   tbl.lfile = nil
   tbl.rfile = nil
   tbl.outfile = nil
   tbl.meld_exists = false
   tbl.lfile = write_to_temporary_file (left, "left", "r+b")
   tbl.afile = write_to_temporary_file (ancestor, "ancestor", "r+b")
   tbl.rfile = write_to_temporary_file (right, "right", "r+b")
   tbl.outfile = write_to_temporary_file ("", "merged", "r+b")

   if tbl.lfile ~= nil and tbl.rfile ~= nil and tbl.afile ~= nil and tbl.outfile ~= nil
   then
      local cmd,mkey = get_preferred_merge3_command (tbl)
      if cmd ~=nil
      then
         io.write ("mtn: " .. string.format(gettext("executing external 3-way merge via \"%s\" merger\n"), mkey))
         ret = cmd (tbl)
         if not ret then
            ret = nil
         else
            ret = read_contents_of_file (ret, "rb")
            if string.len (ret) == 0
            then
               ret = nil
            end
         end
      else
     if mkey then
        io.write (string.format("The possible commands for the "..mkey.." merger aren't available.\n"..
                "You may want to check that $MTN_MERGE or the lua variable `merger' is set\n"..
                "to something available.  If you want to use vim or emacs, you can also\n"..
        "set $EDITOR to something appropriate.\n"))
     else
        io.write (string.format("No external 3-way merge command found.\n"..
                "You may want to check that $EDITOR is set to an editor that supports 3-way\n"..
                "merge, set this explicitly in your get_preferred_merge3_command hook,\n"..
                "or add a 3-way merge program to your path.\n"))
     end
      end
   end

   os.remove (tbl.lfile)
   os.remove (tbl.rfile)
   os.remove (tbl.afile)
   os.remove (tbl.outfile)

   return ret
end

-- expansion of values used in selector completion

function expand_selector(str)

   -- something which looks like a generic cert pattern
   if string.find(str, "^[^=]*=.*$")
   then
      return ("c:" .. str)
   end

   -- something which looks like an email address
   if string.find(str, "[%w%-_]+@[%w%-_]+")
   then
      return ("a:" .. str)
   end

   -- something which looks like a branch name
   if string.find(str, "[%w%-]+%.[%w%-]+")
   then
      return ("b:" .. str)
   end

   -- a sequence of nothing but hex digits
   if string.find(str, "^%x+$")
   then
      return ("i:" .. str)
   end

   -- tries to expand as a date
   local dtstr = expand_date(str)
   if  dtstr ~= nil
   then
      return ("d:" .. dtstr)
   end

   return nil
end

-- expansion of a date expression
function expand_date(str)
   -- simple date patterns
   if string.find(str, "^19%d%d%-%d%d")
      or string.find(str, "^20%d%d%-%d%d")
   then
      return (str)
   end

   -- "now"
   if str == "now"
   then
      local t = os.time(os.date('!*t'))
      return os.date("!%Y-%m-%dT%H:%M:%S", t)
   end

   -- today don't uses the time         # for xgettext's sake, an extra quote
   if str == "today"
   then
      local t = os.time(os.date('!*t'))
      return os.date("!%Y-%m-%d", t)
   end

   -- "yesterday", the source of all hangovers
   if str == "yesterday"
   then
      local t = os.time(os.date('!*t'))
      return os.date("!%Y-%m-%d", t - 86400)
   end

   -- "CVS style" relative dates such as "3 weeks ago"
   local trans = {
      minute = 60;
      hour = 3600;
      day = 86400;
      week = 604800;
      month = 2678400;
      year = 31536000
   }
   local pos, len, n, type = string.find(str, "(%d+) ([minutehordaywk]+)s? ago")
   if trans[type] ~= nil
   then
      local t = os.time(os.date('!*t'))
      if trans[type] <= 3600
      then
        return os.date("!%Y-%m-%dT%H:%M:%S", t - (n * trans[type]))
      else
        return os.date("!%Y-%m-%d", t - (n * trans[type]))
      end
   end

   return nil
end


external_diff_default_args = "-u"

-- default external diff, works for gnu diff
function external_diff(file_path, data_old, data_new, is_binary, diff_args, rev_old, rev_new)
   local old_file = write_to_temporary_file(data_old, nil, "r+b");
   local new_file = write_to_temporary_file(data_new, nil, "r+b");

   if diff_args == nil then diff_args = external_diff_default_args end
   execute("diff", diff_args, "--label", file_path .. "\told", old_file, "--label", file_path .. "\tnew", new_file);

   os.remove (old_file);
   os.remove (new_file);
end

-- netsync permissions hooks (and helper)

function globish_match(glob, str)
      local pcallstatus, result = pcall(function() if (globish.match(glob, str)) then return true else return false end end)
      if pcallstatus == true then
          -- no error
          return result
      else
          -- globish.match had a problem
          return nil
      end
end

function _get_netsync_read_permitted(branch, ident, permfilename, state)
   if not exists(permfilename) or isdir(permfilename) then
      return false
   end
   local permfile = io.open(permfilename, "r")
   if (permfile == nil) then return false end
   local dat = permfile:read("*a")
   io.close(permfile)
   local res = parse_basic_io(dat)
   if res == nil then
      io.stderr:write("file "..permfilename.." cannot be parsed\n")
      return false,"continue"
   end
   state["matches"] = state["matches"] or false
   state["cont"] = state["cont"] or false
   for i, item in pairs(res)
   do
      -- legal names: pattern, allow, deny, continue
      if item.name == "pattern" then
         if state["matches"] and not state["cont"] then return false end
         state["matches"] = false
         state["cont"] = false
         for j, val in pairs(item.values) do
            if globish_match(val, branch) then state["matches"] = true end
         end
      elseif item.name == "allow" then if state["matches"] then
         for j, val in pairs(item.values) do
            if val == "*" then return true end
            if val == "" and ident == nil then return true end
            if ident ~= nil and val == ident.id then return true end
            if ident ~= nil and globish_match(val, ident.name) then return true end
         end
      end elseif item.name == "deny" then if state["matches"] then
         for j, val in pairs(item.values) do
            if val == "*" then return false end
            if val == "" and ident == nil then return false end
            if ident ~= nil and val == ident.id then return false end
            if ident ~= nil and globish_match(val, ident.name) then return false end
         end
      end elseif item.name == "continue" then if state["matches"] then
         state["cont"] = true
         for j, val in pairs(item.values) do
            if val == "false" or val == "no" then
              state["cont"] = false
            end
         end
      end elseif item.name ~= "comment" then
         io.stderr:write("unknown symbol in read-permissions: " .. item.name .. "\n")
         return false
      end
   end
   return false
end

function get_netsync_read_permitted(branch, ident)
   local permfilename = get_confdir() .. "/read-permissions"
   local permdirname = permfilename .. ".d"
   local state = {}
   if _get_netsync_read_permitted(branch, ident, permfilename, state) then
      return true
   end
   if isdir(permdirname) then
      local files = read_directory(permdirname)
      table.sort(files)
      for _,f in ipairs(files) do
        pf = permdirname.."/"..f
        if _get_netsync_read_permitted(branch, ident, pf, state) then
          return true
        end
      end
   end
   return false
end

function _get_netsync_write_permitted(ident, permfilename)
   if not exists(permfilename) or isdir(permfilename) then return false end
   local permfile = io.open(permfilename, "r")
   if (permfile == nil) then
      return false
   end
   local matches = false
   local line = permfile:read()
   while (not matches and line ~= nil) do
      local _, _, ln = string.find(line, "%s*([^%s]*)%s*")
      if ln == "*" then matches = true end
      if ln == ident.id then matches = true end
      if globish_match(ln, ident.name) then matches = true end
      line = permfile:read()
   end
   io.close(permfile)
   return matches
end

function get_netsync_write_permitted(ident)
   local permfilename = get_confdir() .. "/write-permissions"
   local permdirname = permfilename .. ".d"
   if _get_netsync_write_permitted(ident, permfilename) then return true end
   if isdir(permdirname) then
      local files = read_directory(permdirname)
      table.sort(files)
      for _,f in ipairs(files) do
        pf = permdirname.."/"..f
        if _get_netsync_write_permitted(ident, pf) then return true end
      end
   end
   return false
end

-- This is a simple function which assumes you're going to be spawning
-- a copy of mtn, so reuses a common bit at the end for converting
-- local args into remote args. You might need to massage the logic a
-- bit if this doesn't fit your assumptions.

function get_netsync_connect_command(uri, args)

        local argv = nil

        if uri["scheme"] == "ssh"
                and uri["host"]
                and uri["path"] then

                argv = { "ssh" }
                if uri["user"] then
                        table.insert(argv, "-l")
                        table.insert(argv, uri["user"])
                end
                if uri["port"] then
                        table.insert(argv, "-p")
                        table.insert(argv, uri["port"])
                end

                -- ssh://host/~/dir/file.mtn or
                -- ssh://host/~user/dir/file.mtn should be home-relative
                if string.find(uri["path"], "^/~") then
                        uri["path"] = string.sub(uri["path"], 2)
                end

                table.insert(argv, uri["host"])
        end

        if uri["scheme"] == "file" and uri["path"] then
                argv = { }
        end

        if uri["scheme"] == "ssh+ux"
                and uri["host"]
                and uri["path"] then

                argv = { "ssh" }
                if uri["user"] then
                        table.insert(argv, "-l")
                        table.insert(argv, uri["user"])
                end
                if uri["port"] then
                        table.insert(argv, "-p")
                        table.insert(argv, uri["port"])
                end

                -- ssh://host/~/dir/file.mtn or
                -- ssh://host/~user/dir/file.mtn should be home-relative
                if string.find(uri["path"], "^/~") then
                        uri["path"] = string.sub(uri["path"], 2)
                end

                table.insert(argv, uri["host"])
                table.insert(argv, get_remote_unix_socket_command(uri["host"]))
                table.insert(argv, "-")
                table.insert(argv, "UNIX-CONNECT:" .. uri["path"])
        else
            if argv then
                    -- start remote monotone process

                    table.insert(argv, get_mtn_command(uri["host"]))

                    if args["debug"] then
                            table.insert(argv, "--verbose")
                    else
                            table.insert(argv, "--quiet")
                    end

                    table.insert(argv, "--db")
                    table.insert(argv, uri["path"])
                    table.insert(argv, "serve")
                    table.insert(argv, "--stdio")
                    table.insert(argv, "--no-transport-auth")

            -- else scheme does not require starting a new remote
            -- process (ie mtn:)
            end
        end
        return argv
end

function use_transport_auth(uri)
        if uri["scheme"] == "ssh"
        or uri["scheme"] == "ssh+ux"
        or uri["scheme"] == "file" then
                return false
        else
                return true
        end
end

function get_mtn_command(host)
        return "mtn"
end

function get_remote_unix_socket_command(host)
    return "socat"
end

function get_default_command_options(command)
    local default_args = {}
    return default_args
end

function get_default_database_alias()
    return ":default.mtn"
end

function get_default_database_locations()
    local paths = {}
    table.insert(paths, get_confdir() .. "/databases")
    return paths
end

function get_default_database_glob()
    return "*.{mtn,db}"
end

hook_wrapper_dump                = {}
hook_wrapper_dump.depth          = 0
hook_wrapper_dump._string        = function(s) return string.format("%q", s) end
hook_wrapper_dump._number        = function(n) return tostring(n) end
hook_wrapper_dump._boolean       = function(b) if (b) then return "true" end return "false" end
hook_wrapper_dump._userdata      = function(u) return "nil --[[userdata]]" end
-- if we really need to return / serialize functions we could do it
-- like cbreak@irc.freenode.net did here: http://lua-users.org/wiki/TablePersistence
hook_wrapper_dump._function      = function(f) return "nil --[[function]]" end
hook_wrapper_dump._nil           = function(n) return "nil" end
hook_wrapper_dump._thread        = function(t) return "nil --[[thread]]" end
hook_wrapper_dump._lightuserdata = function(l) return "nil --[[lightuserdata]]" end

hook_wrapper_dump._table = function(t)
    local buf = ''
    if (hook_wrapper_dump.depth > 0) then
        buf = buf .. '{\n'
    end
    hook_wrapper_dump.depth = hook_wrapper_dump.depth + 1;
    for k,v in pairs(t) do
        buf = buf..string.format('%s[%s] = %s;\n',
              string.rep("\t", hook_wrapper_dump.depth - 1),
              hook_wrapper_dump["_" .. type(k)](k),
              hook_wrapper_dump["_" .. type(v)](v))
    end
    hook_wrapper_dump.depth = hook_wrapper_dump.depth - 1;
    if (hook_wrapper_dump.depth > 0) then
        buf = buf .. string.rep("\t", hook_wrapper_dump.depth - 1) .. '}'
    end
    return buf
end

function hook_wrapper(func_name, ...)
    -- we have to ensure that nil arguments are restored properly for the
    -- function call, see http://lua-users.org/wiki/StoringNilsInTables
    local args = { n=select('#', ...), ... }
    for i=1,args.n do
        local val = assert(loadstring("return " .. args[i]),
                         "argument "..args[i].." could not be evaluated")()
        assert(val ~= nil or args[i] == "nil",
               "argument "..args[i].." was evaluated to nil")
        args[i] = val
    end
    local res = { _G[func_name](unpack(args, 1, args.n)) }
    return hook_wrapper_dump._table(res)
end

do
   -- Hook functions are tables containing any of the following 6 items
   -- with associated functions:
   --
   --   startup			Corresponds to note_mtn_startup()
   --   start			Corresponds to note_netsync_start()
   --   revision_received	Corresponds to note_netsync_revision_received()
   --   revision_sent		Corresponds to note_netsync_revision_sent()
   --   cert_received		Corresponds to note_netsync_cert_received()
   --   cert_sent		Corresponds to note_netsync_cert_sent()
   --   pubkey_received		Corresponds to note_netsync_pubkey_received()
   --   pubkey_sent		Corresponds to note_netsync_pubkey_sent()
   --   end			Corresponds to note_netsync_end()
   --
   -- Those functions take exactly the same arguments as the corresponding
   -- global functions, but return a different kind of value, a tuple
   -- composed of a return code and a value to be returned back to monotone.
   -- The codes are strings:
   -- "continue" and "stop"
   -- When the code "continue" is returned and there's another notifier, the
   -- second value is ignored and the next notifier is called.  Otherwise,
   -- the second value is returned immediately.
   local hook_functions = {}
   local supported_items = {
      "startup",
      "start", "revision_received", "revision_sent", "cert_received", "cert_sent",
      "pubkey_received", "pubkey_sent", "end"
   }

   function _hook_functions_helper(f,...)
      local s = "continue"
      local v = nil
      for _,n in pairs(hook_functions) do
         if n[f] then
            s,v = n[f](...)
         end
         if s ~= "continue" then
            break
         end
      end
      return v
   end
   function note_mtn_startup(...)
      return _hook_functions_helper("startup",...)
   end
   function note_netsync_start(...)
      return _hook_functions_helper("start",...)
   end
   function note_netsync_revision_received(...)
      return _hook_functions_helper("revision_received",...)
   end
   function note_netsync_revision_sent(...)
      return _hook_functions_helper("revision_sent",...)
   end
   function note_netsync_cert_received(...)
      return _hook_functions_helper("cert_received",...)
   end
   function note_netsync_cert_sent(...)
      return _hook_functions_helper("cert_sent",...)
   end
   function note_netsync_pubkey_received(...)
      return _hook_functions_helper("pubkey_received",...)
   end
   function note_netsync_pubkey_sent(...)
      return _hook_functions_helper("pubkey_sent",...)
   end
   function note_netsync_end(...)
      return _hook_functions_helper("end",...)
   end

   function add_hook_functions(functions, precedence)
      if type(functions) ~= "table" or type(precedence) ~= "number" then
         return false, "Invalid type"
      end
      if hook_functions[precedence] then
         return false, "Precedence already taken"
      end

      local unknown_items = ""
      local warning = nil
      local is_member =
         function (s,t)
            for k,v in pairs(t) do if s == v then return true end end
            return false
         end

      for n,f in pairs(functions) do
         if type(n) == "string" then
            if not is_member(n, supported_items) then
               if unknown_items ~= "" then
                  unknown_items = unknown_items .. ","
               end
               unknown_items = unknown_items .. n
            end
            if type(f) ~= "function" then
               return false, "Value for functions item "..n.." isn't a function"
            end
         else
            warning = "Non-string item keys found in functions table"
         end
      end

      if warning == nil and unknown_items ~= "" then
         warning = "Unknown item(s) " .. unknown_items .. " in functions table"
      end

      hook_functions[precedence] = functions
      return true, warning
   end
   function push_hook_functions(functions)
      local n = #hook_functions + 1
      return add_hook_functions(functions, n)
   end

   -- Kept for backward compatibility
   function add_netsync_notifier(notifier, precedence)
      return add_hook_functions(notifier, precedence)
   end
   function push_netsync_notifier(notifier)
      return push_hook_functions(notifier)
   end
end

-- to ensure only mapped authors are allowed through
-- return "" from unmapped_git_author
-- and validate_git_author will fail

function unmapped_git_author(author)
   -- replace "foo@bar" with "foo <foo@bar>"
   name = author:match("^([^<>]+)@[^<>]+$")
   if name then
      return name .. " <" .. author .. ">"
   end

   -- replace "<foo@bar>" with "foo <foo@bar>"
   name = author:match("^<([^<>]+)@[^<>]+>$")
   if name then
      return name .. " " .. author
   end

   -- replace "foo" with "foo <foo>"
   name = author:match("^[^<>@]+$")
   if name then
      return name .. " <" .. name .. ">"
   end

   return author -- unchanged
end

function validate_git_author(author)
   -- ensure author matches the "Name <email>" format git expects
   if author:match("^[^<]+ <[^>]*>$") then
      return true
   end

   return false
end

function get_man_page_formatter_command()
   local term_width = guess_terminal_width() - 2
   -- The string returned is run in a process created with 'popen'
   -- (see cmd.cc manpage).
   --
   -- On Unix (and POSIX compliant systems), 'popen' runs 'sh' with
   -- the inherited path.
   --
   -- On MinGW, 'popen' runs 'cmd.exe' with the inherited path. MinGW
   -- does not (currently) provide nroff or equivalent. So we assume
   -- sh, nroff, locale and less are also installed, from Cygwin or
   -- some other toolset.
   --
   -- GROFF_ENCODING is an environment variable that, when set, tells
   -- groff (called by nroff where applicable) to use preconv to convert
   -- the input from the given encoding to something groff understands.
   -- For example, groff doesn NOT understand raw UTF-8 as input, but
   -- it does understand unicode, which preconv will happily provide.
   -- This doesn't help people that don't use groff, unfortunately.
   -- Patches are welcome!
   if string.sub(get_ostype(), 1, 7) == "Windows" then
      return string.format("sh -c 'GROFF_ENCODING=`locale charmap` nroff -man -rLL=%dn' | less -R", term_width)
   else
      return string.format("GROFF_ENCODING=`locale charmap` nroff -man -rLL=%dn | less -R", term_width)
   end
end


