This is guile.info, produced by makeinfo version 6.7 from guile.texi.

This manual documents Guile version 2.2.7.

   Copyright (C) 1996, 1997, 2000, 2001, 2002, 2003, 2004, 2005, 2009,
2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Free Software
Foundation, Inc.

   Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with no
Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  A
copy of the license is included in the section entitled “GNU Free
Documentation License.”
INFO-DIR-SECTION The Algorithmic Language Scheme
START-INFO-DIR-ENTRY
* Guile Reference: (guile).     The Guile reference manual.
END-INFO-DIR-ENTRY


File: guile.info,  Node: Buffering,  Next: Random Access,  Prev: Simple Output,  Up: Input and Output

6.14.6 Buffering
----------------

Every port has associated input and output buffers.  You can think of
ports as being backed by some mutable store, and that store might be far
away.  For example, ports backed by file descriptors have to go all the
way to the kernel to read and write their data.  To avoid this
round-trip cost, Guile usually reads in data from the mutable store in
chunks, and then services small requests like ‘get-char’ out of that
intermediate buffer.  Similarly, small writes like ‘write-char’ first go
to a buffer, and are sent to the store when the buffer is full (or when
port is flushed).  Buffered ports speed up your program by reducing the
number of round-trips to the mutable store, and they do so in a way that
is mostly transparent to the user.

   There are two major ways, however, in which buffering affects program
semantics.  Building correct, performant programs requires understanding
these situations.

   The first case is in random-access read/write ports (*note Random
Access::).  These ports, usually backed by a file, logically operate
over the same mutable store when both reading and writing.  So, if you
read a character, causing the buffer to fill, then write a character,
the bytes you filled in your read buffer are now invalid.  Every time
you switch between reading and writing, Guile has to flush any pending
buffer.  If this happens frequently, the cost can be high.  In that case
you should reduce the amount that you buffer, in both directions.
Similarly, Guile has to flush buffers before seeking.  None of these
considerations apply to sockets, which don’t logically read from and
write to the same mutable store, and are not seekable.  Note also that
sockets are unbuffered by default.  *Note Network Sockets and
Communication::.

   The second case is the more pernicious one.  If you write data to a
buffered port, it probably doesn’t go out to the mutable store directly.
(This “probably” introduces some indeterminism in your program: what
goes to the store, and when, depends on how full the buffer is.  It is
something that the user needs to explicitly be aware of.)  The data is
written to the store later – when the buffer fills up due to another
write, or when ‘force-output’ is called, or when ‘close-port’ is called,
or when the program exits, or even when the garbage collector runs.  The
salient point is, _the errors are signalled then too_.  Buffered writes
defer error detection (and defer the side effects to the mutable store),
perhaps indefinitely if the port type does not need to be closed at GC.

   One common heuristic that works well for textual ports is to flush
output when a newline (‘\n’) is written.  This “line buffering” mode is
on by default for TTY ports.  Most other ports are “block buffered”,
meaning that once the output buffer reaches the block size, which
depends on the port and its configuration, the output is flushed as a
block, without regard to what is in the block.  Likewise reads are read
in at the block size, though if there are fewer bytes available to read,
the buffer may not be entirely filled.

   Note that binary reads or writes that are larger than the buffer size
go directly to the mutable store without passing through the buffers.
If your access pattern involves many big reads or writes, buffering
might not matter so much to you.

   To control the buffering behavior of a port, use ‘setvbuf’.

 -- Scheme Procedure: setvbuf port mode [size]
 -- C Function: scm_setvbuf (port, mode, size)
     Set the buffering mode for PORT.  MODE can be one of the following
     symbols:

     ‘none’
          non-buffered
     ‘line’
          line buffered
     ‘block’
          block buffered, using a newly allocated buffer of SIZE bytes.
          If SIZE is omitted, a default size will be used.

   Another way to set the buffering, for file ports, is to open the file
with ‘0’ or ‘l’ as part of the mode string, for unbuffered or
line-buffered ports, respectively.  *Note File Ports::, for more.

   Any buffered output data will be written out when the port is closed.
To make sure to flush it at specific points in your program, use
‘force-otput’.

 -- Scheme Procedure: force-output [port]
 -- C Function: scm_force_output (port)
     Flush the specified output port, or the current output port if PORT
     is omitted.  The current output buffer contents, if any, are passed
     to the underlying port implementation.

     The return value is unspecified.

 -- Scheme Procedure: flush-all-ports
 -- C Function: scm_flush_all_ports ()
     Equivalent to calling ‘force-output’ on all open output ports.  The
     return value is unspecified.

   Similarly, sometimes you might want to switch from using Guile’s
ports to working directly on file descriptors.  In that case, for input
ports use ‘drain-input’ to get any buffered input from that port.

 -- Scheme Procedure: drain-input port
 -- C Function: scm_drain_input (port)
     This procedure clears a port’s input buffers, similar to the way
     that force-output clears the output buffer.  The contents of the
     buffers are returned as a single string, e.g.,

          (define p (open-input-file ...))
          (drain-input p) => empty string, nothing buffered yet.
          (unread-char (read-char p) p)
          (drain-input p) => initial chars from p, up to the buffer size.

   All of these considerations are very similar to those of streams in
the C library, although Guile’s ports are not built on top of C streams.
Still, it is useful to read what other systems do.  *Note
(libc)Streams::, for more discussion on C streams.


File: guile.info,  Node: Random Access,  Next: Line/Delimited,  Prev: Buffering,  Up: Input and Output

6.14.7 Random Access
--------------------

 -- Scheme Procedure: seek fd_port offset whence
 -- C Function: scm_seek (fd_port, offset, whence)
     Sets the current position of FD_PORT to the integer OFFSET.  For a
     file port, OFFSET is expressed as a number of bytes; for other
     types of ports, such as string ports, OFFSET is an abstract
     representation of the position within the port’s data, not
     necessarily expressed as a number of bytes.  OFFSET is interpreted
     according to the value of WHENCE.

     One of the following variables should be supplied for WHENCE:
      -- Variable: SEEK_SET
          Seek from the beginning of the file.
      -- Variable: SEEK_CUR
          Seek from the current position.
      -- Variable: SEEK_END
          Seek from the end of the file.
     If FD_PORT is a file descriptor, the underlying system call is
     ‘lseek’.  PORT may be a string port.

     The value returned is the new position in FD_PORT.  This means that
     the current position of a port can be obtained using:
          (seek port 0 SEEK_CUR)

 -- Scheme Procedure: ftell fd_port
 -- C Function: scm_ftell (fd_port)
     Return an integer representing the current position of FD_PORT,
     measured from the beginning.  Equivalent to:

          (seek port 0 SEEK_CUR)

 -- Scheme Procedure: truncate-file file [length]
 -- C Function: scm_truncate_file (file, length)
     Truncate FILE to LENGTH bytes.  FILE can be a filename string, a
     port object, or an integer file descriptor.  The return value is
     unspecified.

     For a port or file descriptor LENGTH can be omitted, in which case
     the file is truncated at the current position (per ‘ftell’ above).

     On most systems a file can be extended by giving a length greater
     than the current size, but this is not mandatory in the POSIX
     standard.


File: guile.info,  Node: Line/Delimited,  Next: Default Ports,  Prev: Random Access,  Up: Input and Output

6.14.8 Line Oriented and Delimited Text
---------------------------------------

The delimited-I/O module can be accessed with:

     (use-modules (ice-9 rdelim))

   It can be used to read or write lines of text, or read text delimited
by a specified set of characters.

 -- Scheme Procedure: read-line [port] [handle-delim]
     Return a line of text from PORT if specified, otherwise from the
     value returned by ‘(current-input-port)’.  Under Unix, a line of
     text is terminated by the first end-of-line character or by
     end-of-file.

     If HANDLE-DELIM is specified, it should be one of the following
     symbols:
     ‘trim’
          Discard the terminating delimiter.  This is the default, but
          it will be impossible to tell whether the read terminated with
          a delimiter or end-of-file.
     ‘concat’
          Append the terminating delimiter (if any) to the returned
          string.
     ‘peek’
          Push the terminating delimiter (if any) back on to the port.
     ‘split’
          Return a pair containing the string read from the port and the
          terminating delimiter or end-of-file object.

 -- Scheme Procedure: read-line! buf [port]
     Read a line of text into the supplied string BUF and return the
     number of characters added to BUF.  If BUF is filled, then ‘#f’ is
     returned.  Read from PORT if specified, otherwise from the value
     returned by ‘(current-input-port)’.

 -- Scheme Procedure: read-delimited delims [port] [handle-delim]
     Read text until one of the characters in the string DELIMS is found
     or end-of-file is reached.  Read from PORT if supplied, otherwise
     from the value returned by ‘(current-input-port)’.  HANDLE-DELIM
     takes the same values as described for ‘read-line’.

 -- Scheme Procedure: read-delimited! delims buf [port] [handle-delim]
          [start] [end]
     Read text into the supplied string BUF.

     If a delimiter was found, return the number of characters written,
     except if HANDLE-DELIM is ‘split’, in which case the return value
     is a pair, as noted above.

     As a special case, if PORT was already at end-of-stream, the EOF
     object is returned.  Also, if no characters were written because
     the buffer was full, ‘#f’ is returned.

     It’s something of a wacky interface, to be honest.

 -- Scheme Procedure: %read-delimited! delims str gobble [port [start
          [end]]]
 -- C Function: scm_read_delimited_x (delims, str, gobble, port, start,
          end)
     Read characters from PORT into STR until one of the characters in
     the DELIMS string is encountered.  If GOBBLE is true, discard the
     delimiter character; otherwise, leave it in the input stream for
     the next read.  If PORT is not specified, use the value of
     ‘(current-input-port)’.  If START or END are specified, store data
     only into the substring of STR bounded by START and END (which
     default to the beginning and end of the string, respectively).

     Return a pair consisting of the delimiter that terminated the
     string and the number of characters read.  If reading stopped at
     the end of file, the delimiter returned is the EOF-OBJECT; if the
     string was filled without encountering a delimiter, this value is
     ‘#f’.

 -- Scheme Procedure: %read-line [port]
 -- C Function: scm_read_line (port)
     Read a newline-terminated line from PORT, allocating storage as
     necessary.  The newline terminator (if any) is removed from the
     string, and a pair consisting of the line and its delimiter is
     returned.  The delimiter may be either a newline or the EOF-OBJECT;
     if ‘%read-line’ is called at the end of file, it returns the pair
     ‘(#<eof> . #<eof>)’.


File: guile.info,  Node: Default Ports,  Next: Port Types,  Prev: Line/Delimited,  Up: Input and Output

6.14.9 Default Ports for Input, Output and Errors
-------------------------------------------------

 -- Scheme Procedure: current-input-port
 -- C Function: scm_current_input_port ()
     Return the current input port.  This is the default port used by
     many input procedures.

     Initially this is the “standard input” in Unix and C terminology.
     When the standard input is a tty the port is unbuffered, otherwise
     it’s fully buffered.

     Unbuffered input is good if an application runs an interactive
     subprocess, since any type-ahead input won’t go into Guile’s buffer
     and be unavailable to the subprocess.

     Note that Guile buffering is completely separate from the tty “line
     discipline”.  In the usual cooked mode on a tty Guile only sees a
     line of input once the user presses <Return>.

 -- Scheme Procedure: current-output-port
 -- C Function: scm_current_output_port ()
     Return the current output port.  This is the default port used by
     many output procedures.

     Initially this is the “standard output” in Unix and C terminology.
     When the standard output is a tty this port is unbuffered,
     otherwise it’s fully buffered.

     Unbuffered output to a tty is good for ensuring progress output or
     a prompt is seen.  But an application which always prints whole
     lines could change to line buffered, or an application with a lot
     of output could go fully buffered and perhaps make explicit
     ‘force-output’ calls (*note Buffering::) at selected points.

 -- Scheme Procedure: current-error-port
 -- C Function: scm_current_error_port ()
     Return the port to which errors and warnings should be sent.

     Initially this is the “standard error” in Unix and C terminology.
     When the standard error is a tty this port is unbuffered, otherwise
     it’s fully buffered.

 -- Scheme Procedure: set-current-input-port port
 -- Scheme Procedure: set-current-output-port port
 -- Scheme Procedure: set-current-error-port port
 -- C Function: scm_set_current_input_port (port)
 -- C Function: scm_set_current_output_port (port)
 -- C Function: scm_set_current_error_port (port)
     Change the ports returned by ‘current-input-port’,
     ‘current-output-port’ and ‘current-error-port’, respectively, so
     that they use the supplied PORT for input or output.

 -- Scheme Procedure: with-input-from-port port thunk
 -- Scheme Procedure: with-output-to-port port thunk
 -- Scheme Procedure: with-error-to-port port thunk
     Call THUNK in a dynamic environment in which ‘current-input-port’,
     ‘current-output-port’ or ‘current-error-port’ is rebound to the
     given PORT.

 -- C Function: void scm_dynwind_current_input_port (SCM port)
 -- C Function: void scm_dynwind_current_output_port (SCM port)
 -- C Function: void scm_dynwind_current_error_port (SCM port)
     These functions must be used inside a pair of calls to
     ‘scm_dynwind_begin’ and ‘scm_dynwind_end’ (*note Dynamic Wind::).
     During the dynwind context, the indicated port is set to PORT.

     More precisely, the current port is swapped with a ‘backup’ value
     whenever the dynwind context is entered or left.  The backup value
     is initialized with the PORT argument.


File: guile.info,  Node: Port Types,  Next: Venerable Port Interfaces,  Prev: Default Ports,  Up: Input and Output

6.14.10 Types of Port
---------------------

* Menu:

* File Ports:: Ports on an operating system file.
* Bytevector Ports:: Ports on a bytevector.
* String Ports:: Ports on a Scheme string.
* Custom Ports:: Ports whose implementation you control.
* Soft Ports:: An older version of custom ports.
* Void Ports:: Ports on nothing at all.


File: guile.info,  Node: File Ports,  Next: Bytevector Ports,  Up: Port Types

6.14.10.1 File Ports
....................

The following procedures are used to open file ports.  See also *note
open: Ports and File Descriptors, for an interface to the Unix ‘open’
system call.

   All file access uses the “LFS” large file support functions when
available, so files bigger than 2 Gbytes (2^31 bytes) can be read and
written on a 32-bit system.

   Most systems have limits on how many files can be open, so it’s
strongly recommended that file ports be closed explicitly when no longer
required (*note Ports::).

 -- Scheme Procedure: open-file filename mode [#:guess-encoding=#f]
          [#:encoding=#f]
 -- C Function: scm_open_file_with_encoding (filename, mode,
          guess_encoding, encoding)
 -- C Function: scm_open_file (filename, mode)
     Open the file whose name is FILENAME, and return a port
     representing that file.  The attributes of the port are determined
     by the MODE string.  The way in which this is interpreted is
     similar to C stdio.  The first character must be one of the
     following:

     ‘r’
          Open an existing file for input.
     ‘w’
          Open a file for output, creating it if it doesn’t already
          exist or removing its contents if it does.
     ‘a’
          Open a file for output, creating it if it doesn’t already
          exist.  All writes to the port will go to the end of the file.
          The "append mode" can be turned off while the port is in use
          *note fcntl: Ports and File Descriptors.

     The following additional characters can be appended:

     ‘+’
          Open the port for both input and output.  E.g., ‘r+’: open an
          existing file for both input and output.
     ‘0’
          Create an "unbuffered" port.  In this case input and output
          operations are passed directly to the underlying port
          implementation without additional buffering.  This is likely
          to slow down I/O operations.  The buffering mode can be
          changed while a port is in use (*note Buffering::).
     ‘l’
          Add line-buffering to the port.  The port output buffer will
          be automatically flushed whenever a newline character is
          written.
     ‘b’
          Use binary mode, ensuring that each byte in the file will be
          read as one Scheme character.

          To provide this property, the file will be opened with the
          8-bit character encoding "ISO-8859-1", ignoring the default
          port encoding.  *Note Ports::, for more information on port
          encodings.

          Note that while it is possible to read and write binary data
          as characters or strings, it is usually better to treat bytes
          as octets, and byte sequences as bytevectors.  *Note Binary
          I/O::, for more.

          This option had another historical meaning, for DOS
          compatibility: in the default (textual) mode, DOS reads a
          CR-LF sequence as one LF byte.  The ‘b’ flag prevents this
          from happening, adding ‘O_BINARY’ to the underlying ‘open’
          call.  Still, the flag is generally useful because of its port
          encoding ramifications.

     Unless binary mode is requested, the character encoding of the new
     port is determined as follows: First, if GUESS-ENCODING is true,
     the ‘file-encoding’ procedure is used to guess the encoding of the
     file (*note Character Encoding of Source Files::).  If
     GUESS-ENCODING is false or if ‘file-encoding’ fails, ENCODING is
     used unless it is also false.  As a last resort, the default port
     encoding is used.  *Note Ports::, for more information on port
     encodings.  It is an error to pass a non-false GUESS-ENCODING or
     ENCODING if binary mode is requested.

     If a file cannot be opened with the access requested, ‘open-file’
     throws an exception.

 -- Scheme Procedure: open-input-file filename [#:guess-encoding=#f]
          [#:encoding=#f] [#:binary=#f]

     Open FILENAME for input.  If BINARY is true, open the port in
     binary mode, otherwise use text mode.  ENCODING and GUESS-ENCODING
     determine the character encoding as described above for
     ‘open-file’.  Equivalent to
          (open-file FILENAME
                     (if BINARY "rb" "r")
                     #:guess-encoding GUESS-ENCODING
                     #:encoding ENCODING)

 -- Scheme Procedure: open-output-file filename [#:encoding=#f]
          [#:binary=#f]

     Open FILENAME for output.  If BINARY is true, open the port in
     binary mode, otherwise use text mode.  ENCODING specifies the
     character encoding as described above for ‘open-file’.  Equivalent
     to
          (open-file FILENAME
                     (if BINARY "wb" "w")
                     #:encoding ENCODING)

 -- Scheme Procedure: call-with-input-file filename proc
          [#:guess-encoding=#f] [#:encoding=#f] [#:binary=#f]
 -- Scheme Procedure: call-with-output-file filename proc
          [#:encoding=#f] [#:binary=#f]
     Open FILENAME for input or output, and call ‘(PROC port)’ with the
     resulting port.  Return the value returned by PROC.  FILENAME is
     opened as per ‘open-input-file’ or ‘open-output-file’ respectively,
     and an error is signaled if it cannot be opened.

     When PROC returns, the port is closed.  If PROC does not return
     (e.g. if it throws an error), then the port might not be closed
     automatically, though it will be garbage collected in the usual way
     if not otherwise referenced.

 -- Scheme Procedure: with-input-from-file filename thunk
          [#:guess-encoding=#f] [#:encoding=#f] [#:binary=#f]
 -- Scheme Procedure: with-output-to-file filename thunk [#:encoding=#f]
          [#:binary=#f]
 -- Scheme Procedure: with-error-to-file filename thunk [#:encoding=#f]
          [#:binary=#f]
     Open FILENAME and call ‘(THUNK)’ with the new port setup as
     respectively the ‘current-input-port’, ‘current-output-port’, or
     ‘current-error-port’.  Return the value returned by THUNK.
     FILENAME is opened as per ‘open-input-file’ or ‘open-output-file’
     respectively, and an error is signaled if it cannot be opened.

     When THUNK returns, the port is closed and the previous setting of
     the respective current port is restored.

     The current port setting is managed with ‘dynamic-wind’, so the
     previous value is restored no matter how THUNK exits (eg. an
     exception), and if THUNK is re-entered (via a captured
     continuation) then it’s set again to the FILENAME port.

     The port is closed when THUNK returns normally, but not when exited
     via an exception or new continuation.  This ensures it’s still
     ready for use if THUNK is re-entered by a captured continuation.
     Of course the port is always garbage collected and closed in the
     usual way when no longer referenced anywhere.

 -- Scheme Procedure: port-mode port
 -- C Function: scm_port_mode (port)
     Return the port modes associated with the open port PORT.  These
     will not necessarily be identical to the modes used when the port
     was opened, since modes such as "append" which are used only during
     port creation are not retained.

 -- Scheme Procedure: port-filename port
 -- C Function: scm_port_filename (port)
     Return the filename associated with PORT, or ‘#f’ if no filename is
     associated with the port.

     PORT must be open; ‘port-filename’ cannot be used once the port is
     closed.

 -- Scheme Procedure: set-port-filename! port filename
 -- C Function: scm_set_port_filename_x (port, filename)
     Change the filename associated with PORT, using the current input
     port if none is specified.  Note that this does not change the
     port’s source of data, but only the value that is returned by
     ‘port-filename’ and reported in diagnostic output.

 -- Scheme Procedure: file-port? obj
 -- C Function: scm_file_port_p (obj)
     Determine whether OBJ is a port that is related to a file.


File: guile.info,  Node: Bytevector Ports,  Next: String Ports,  Prev: File Ports,  Up: Port Types

6.14.10.2 Bytevector Ports
..........................

 -- Scheme Procedure: open-bytevector-input-port bv [transcoder]
 -- C Function: scm_open_bytevector_input_port (bv, transcoder)
     Return an input port whose contents are drawn from bytevector BV
     (*note Bytevectors::).

     The TRANSCODER argument is currently not supported.

 -- Scheme Procedure: open-bytevector-output-port [transcoder]
 -- C Function: scm_open_bytevector_output_port (transcoder)
     Return two values: a binary output port and a procedure.  The
     latter should be called with zero arguments to obtain a bytevector
     containing the data accumulated by the port, as illustrated below.

          (call-with-values
            (lambda ()
              (open-bytevector-output-port))
            (lambda (port get-bytevector)
              (display "hello" port)
              (get-bytevector)))

          ⇒ #vu8(104 101 108 108 111)

     The TRANSCODER argument is currently not supported.


File: guile.info,  Node: String Ports,  Next: Custom Ports,  Prev: Bytevector Ports,  Up: Port Types

6.14.10.3 String Ports
......................

 -- Scheme Procedure: call-with-output-string proc
 -- C Function: scm_call_with_output_string (proc)
     Calls the one-argument procedure PROC with a newly created output
     port.  When the function returns, the string composed of the
     characters written into the port is returned.  PROC should not
     close the port.

 -- Scheme Procedure: call-with-input-string string proc
 -- C Function: scm_call_with_input_string (string, proc)
     Calls the one-argument procedure PROC with a newly created input
     port from which STRING’s contents may be read.  The value yielded
     by the PROC is returned.

 -- Scheme Procedure: with-output-to-string thunk
     Calls the zero-argument procedure THUNK with the current output
     port set temporarily to a new string port.  It returns a string
     composed of the characters written to the current output.

 -- Scheme Procedure: with-input-from-string string thunk
     Calls the zero-argument procedure THUNK with the current input port
     set temporarily to a string port opened on the specified STRING.
     The value yielded by THUNK is returned.

 -- Scheme Procedure: open-input-string str
 -- C Function: scm_open_input_string (str)
     Take a string and return an input port that delivers characters
     from the string.  The port can be closed by ‘close-input-port’,
     though its storage will be reclaimed by the garbage collector if it
     becomes inaccessible.

 -- Scheme Procedure: open-output-string
 -- C Function: scm_open_output_string ()
     Return an output port that will accumulate characters for retrieval
     by ‘get-output-string’.  The port can be closed by the procedure
     ‘close-output-port’, though its storage will be reclaimed by the
     garbage collector if it becomes inaccessible.

 -- Scheme Procedure: get-output-string port
 -- C Function: scm_get_output_string (port)
     Given an output port created by ‘open-output-string’, return a
     string consisting of the characters that have been output to the
     port so far.

     ‘get-output-string’ must be used before closing PORT, once closed
     the string cannot be obtained.

   With string ports, the port-encoding is treated differently than
other types of ports.  When string ports are created, they do not
inherit a character encoding from the current locale.  They are given a
default locale that allows them to handle all valid string characters.
Typically one should not modify a string port’s character encoding away
from its default.  *Note Encoding::.


File: guile.info,  Node: Custom Ports,  Next: Soft Ports,  Prev: String Ports,  Up: Port Types

6.14.10.4 Custom Ports
......................

Custom ports allow the user to provide input and handle output via
user-supplied procedures.  Guile currently only provides custom binary
ports, not textual ports; for custom textual ports, *Note Soft Ports::.
We should add the R6RS custom textual port interfaces though.
Contributions are appreciated.

 -- Scheme Procedure: make-custom-binary-input-port id read!
          get-position set-position! close
     Return a new custom binary input port(1) named ID (a string) whose
     input is drained by invoking READ! and passing it a bytevector, an
     index where bytes should be written, and the number of bytes to
     read.  The ‘read!’ procedure must return an integer indicating the
     number of bytes read, or ‘0’ to indicate the end-of-file.

     Optionally, if GET-POSITION is not ‘#f’, it must be a thunk that
     will be called when ‘port-position’ is invoked on the custom binary
     port and should return an integer indicating the position within
     the underlying data stream; if GET-POSITION was not supplied, the
     returned port does not support ‘port-position’.

     Likewise, if SET-POSITION! is not ‘#f’, it should be a one-argument
     procedure.  When ‘set-port-position!’ is invoked on the custom
     binary input port, SET-POSITION! is passed an integer indicating
     the position of the next byte is to read.

     Finally, if CLOSE is not ‘#f’, it must be a thunk.  It is invoked
     when the custom binary input port is closed.

     The returned port is fully buffered by default, but its buffering
     mode can be changed using ‘setvbuf’ (*note Buffering::).

     Using a custom binary input port, the ‘open-bytevector-input-port’
     procedure (*note Bytevector Ports::) could be implemented as
     follows:

          (define (open-bytevector-input-port source)
            (define position 0)
            (define length (bytevector-length source))

            (define (read! bv start count)
              (let ((count (min count (- length position))))
                (bytevector-copy! source position
                                  bv start count)
                (set! position (+ position count))
                count))

            (define (get-position) position)

            (define (set-position! new-position)
              (set! position new-position))

            (make-custom-binary-input-port "the port" read!
                                            get-position set-position!
                                            #f))

          (read (open-bytevector-input-port (string->utf8 "hello")))
          ⇒ hello

 -- Scheme Procedure: make-custom-binary-output-port id write!
          get-position set-position! close
     Return a new custom binary output port named ID (a string) whose
     output is sunk by invoking WRITE! and passing it a bytevector, an
     index where bytes should be read from this bytevector, and the
     number of bytes to be “written”.  The ‘write!’ procedure must
     return an integer indicating the number of bytes actually written;
     when it is passed ‘0’ as the number of bytes to write, it should
     behave as though an end-of-file was sent to the byte sink.

     The other arguments are as for ‘make-custom-binary-input-port’.

 -- Scheme Procedure: make-custom-binary-input/output-port id read!
          write! get-position set-position! close
     Return a new custom binary input/output port named ID (a string).
     The various arguments are the same as for The other arguments are
     as for ‘make-custom-binary-input-port’ and
     ‘make-custom-binary-output-port’.  If buffering is enabled on the
     port, as is the case by default, input will be buffered in both
     directions; *Note Buffering::.  If the SET-POSITION! function is
     provided and not ‘#f’, then the port will also be marked as
     random-access, causing the buffer to be flushed between reads and
     writes.

   ---------- Footnotes ----------

   (1) This is similar in spirit to Guile’s “soft ports” (*note Soft
Ports::).


File: guile.info,  Node: Soft Ports,  Next: Void Ports,  Prev: Custom Ports,  Up: Port Types

6.14.10.5 Soft Ports
....................

A “soft port” is a port based on a vector of procedures capable of
accepting or delivering characters.  It allows emulation of I/O ports.

 -- Scheme Procedure: make-soft-port pv modes
     Return a port capable of receiving or delivering characters as
     specified by the MODES string (*note open-file: File Ports.).  PV
     must be a vector of length 5 or 6.  Its components are as follows:

       0. procedure accepting one character for output
       1. procedure accepting a string for output
       2. thunk for flushing output
       3. thunk for getting one character
       4. thunk for closing port (not by garbage collection)
       5. (if present and not ‘#f’) thunk for computing the number of
          characters that can be read from the port without blocking.

     For an output-only port only elements 0, 1, 2, and 4 need be
     procedures.  For an input-only port only elements 3 and 4 need be
     procedures.  Thunks 2 and 4 can instead be ‘#f’ if there is no
     useful operation for them to perform.

     If thunk 3 returns ‘#f’ or an ‘eof-object’ (*note eof-object?:
     (r5rs)Input.) it indicates that the port has reached end-of-file.
     For example:

          (define stdout (current-output-port))
          (define p (make-soft-port
                     (vector
                      (lambda (c) (write c stdout))
                      (lambda (s) (display s stdout))
                      (lambda () (display "." stdout))
                      (lambda () (char-upcase (read-char)))
                      (lambda () (display "@" stdout)))
                     "rw"))

          (write p p) ⇒ #<input-output: soft 8081e20>


File: guile.info,  Node: Void Ports,  Prev: Soft Ports,  Up: Port Types

6.14.10.6 Void Ports
....................

This kind of port causes any data to be discarded when written to, and
always returns the end-of-file object when read from.

 -- Scheme Procedure: %make-void-port mode
 -- C Function: scm_sys_make_void_port (mode)
     Create and return a new void port.  A void port acts like
     ‘/dev/null’.  The MODE argument specifies the input/output modes
     for this port: see the documentation for ‘open-file’ in *note File
     Ports::.


File: guile.info,  Node: Venerable Port Interfaces,  Next: Using Ports from C,  Prev: Port Types,  Up: Input and Output

6.14.11 Venerable Port Interfaces
---------------------------------

Over the 25 years or so that Guile has been around, its port system has
evolved, adding many useful features.  At the same time there have been
four major Scheme standards released in those 25 years, which also
evolve the common Scheme understanding of what a port interface should
be.  Alas, it would be too much to ask for all of these evolutionary
branches to be consistent.  Some of Guile’s original interfaces don’t
mesh with the later Scheme standards, and yet Guile can’t just drop old
interfaces.  Sadly as well, the R6RS and R7RS standards both part from a
base of R5RS, but end up in different and somewhat incompatible designs.

   Guile’s approach is to pick a set of port primitives that make sense
together.  We document that set of primitives, design our internal
interfaces around them, and recommend them to users.  As the R6RS I/O
system is the most capable standard that Scheme has yet produced in this
domain, we mostly recommend that; ‘(ice-9 binary-ports)’ and ‘(ice-9
textual-ports)’ are wholly modelled on ‘(rnrs io ports)’.  Guile does
not wholly copy R6RS, however; *Note R6RS Incompatibilities::.

   At the same time, we have many venerable port interfaces, lore handed
down to us from our hacker ancestors.  Most of these interfaces even
predate the expectation that Scheme should have modules, so they are
present in the default environment.  In Guile we support them as well
and we have no plans to remove them, but again we don’t recommend them
for new users.

 -- Scheme Procedure: char-ready? [port]
     Return ‘#t’ if a character is ready on input PORT and return ‘#f’
     otherwise.  If ‘char-ready?’ returns ‘#t’ then the next ‘read-char’
     operation on PORT is guaranteed not to hang.  If PORT is a file
     port at end of file then ‘char-ready?’ returns ‘#t’.

     ‘char-ready?’ exists to make it possible for a program to accept
     characters from interactive ports without getting stuck waiting for
     input.  Any input editors associated with such ports must make sure
     that characters whose existence has been asserted by ‘char-ready?’
     cannot be rubbed out.  If ‘char-ready?’ were to return ‘#f’ at end
     of file, a port at end of file would be indistinguishable from an
     interactive port that has no ready characters.

     Note that ‘char-ready?’ only works reliably for terminals and
     sockets with one-byte encodings.  Under the hood it will return
     ‘#t’ if the port has any input buffered, or if the file descriptor
     that backs the port polls as readable, indicating that Guile can
     fetch more bytes from the kernel.  However being able to fetch one
     byte doesn’t mean that a full character is available; *Note
     Encoding::.  Also, on many systems it’s possible for a file
     descriptor to poll as readable, but then block when it comes time
     to read bytes.  Note also that on Linux kernels, all file ports
     backed by files always poll as readable.  For non-file ports, this
     procedure always returns ‘#t’, except for soft ports, which have a
     ‘char-ready?’ handler.  *Note Soft Ports::.

     In short, this is a legacy procedure whose semantics are hard to
     provide.  However it is a useful check to see if any input is
     buffered.  *Note Non-Blocking I/O::.

 -- Scheme Procedure: read-char [port]
     The same as ‘get-char’, except that PORT defaults to the current
     input port.  *Note Textual I/O::.

 -- Scheme Procedure: peek-char [port]
     The same as ‘lookahead-char’, except that PORT defaults to the
     current input port.  *Note Textual I/O::.

 -- Scheme Procedure: unread-char cobj [port]
     The same as ‘unget-char’, except that PORT defaults to the current
     input port, and the arguments are swapped.  *Note Textual I/O::.

 -- Scheme Procedure: unread-string str port
 -- C Function: scm_unread_string (str, port)
     The same as ‘unget-string’, except that PORT defaults to the
     current input port, and the arguments are swapped.  *Note Textual
     I/O::.

 -- Scheme Procedure: newline [port]
     Send a newline to PORT.  If PORT is omitted, send to the current
     output port.  Equivalent to ‘(put-char port #\newline)’.

 -- Scheme Procedure: write-char chr [port]
     The same as ‘put-char’, except that PORT defaults to the current
     input port, and the arguments are swapped.  *Note Textual I/O::.


File: guile.info,  Node: Using Ports from C,  Next: I/O Extensions,  Prev: Venerable Port Interfaces,  Up: Input and Output

6.14.12 Using Ports from C
--------------------------

Guile’s C interfaces provides some niceties for sending and receiving
bytes and characters in a way that works better with C.

 -- C Function: size_t scm_c_read (SCM port, void *buffer, size_t size)
     Read up to SIZE bytes from PORT and store them in BUFFER.  The
     return value is the number of bytes actually read, which can be
     less than SIZE if end-of-file has been reached.

     Note that as this is a binary input procedure, this function does
     not update ‘port-line’ and ‘port-column’ (*note Textual I/O::).

 -- C Function: void scm_c_write (SCM port, const void *buffer, size_t
          size)
     Write SIZE bytes at BUFFER to PORT.

     Note that as this is a binary output procedure, this function does
     not update ‘port-line’ and ‘port-column’ (*note Textual I/O::).

 -- C Function: size_t scm_c_read_bytes (SCM port, SCM bv, size_t start,
          size_t count)
 -- C Function: void scm_c_write_bytes (SCM port, SCM bv, size_t start,
          size_t count)
     Like ‘scm_c_read’ and ‘scm_c_write’, but reading into or writing
     from the bytevector BV.  COUNT indicates the byte index at which to
     start in the bytevector, and the read or write will continue for
     COUNT bytes.

 -- C Function: void scm_unget_bytes (const unsigned char *buf, size_t
          len, SCM port)
 -- C Function: void scm_unget_byte (int c, SCM port)
 -- C Function: void scm_ungetc (scm_t_wchar c, SCM port)
     Like ‘unget-bytevector’, ‘unget-byte’, and ‘unget-char’,
     respectively.  *Note Textual I/O::.

 -- C Function: void scm_c_put_latin1_chars (SCM port, const scm_t_uint8
          *buf, size_t len)
 -- C Function: void scm_c_put_utf32_chars (SCM port, const scm_t_uint32
          *buf, size_t len);
     Write a string to PORT.  In the first case, the ‘scm_t_uint8*’
     buffer is a string in the latin-1 encoding.  In the second, the
     ‘scm_t_uint32*’ buffer is a string in the UTF-32 encoding.  These
     routines will update the port’s line and column.


File: guile.info,  Node: I/O Extensions,  Next: Non-Blocking I/O,  Prev: Using Ports from C,  Up: Input and Output

6.14.13 Implementing New Port Types in C
----------------------------------------

This section describes how to implement a new port type in C. Although
ports support many operations, as a data structure they present an
opaque interface to the user.  To the port implementor, you have two
pieces of information to work with: the port type, and the port’s
“stream”.  The port type is an opaque pointer allocated when defining
your port type.  It is your key into the port API, and it helps you
identify which ports are actually yours.  The “stream” is a pointer you
control, and which you set when you create a port.  Get a stream from a
port using the ‘SCM_STREAM’ macro.  Note that your port methods are only
ever called with ports of your type.

   A port type is created by calling ‘scm_make_port_type’.  Once you
have your port type, you can create ports with ‘scm_c_make_port’, or
‘scm_c_make_port_with_encoding’.

 -- Function: scm_t_port_type* scm_make_port_type (char *name, size_t
          (*read) (SCM port, SCM dst, size_t start, size_t count),
          size_t (*write) (SCM port, SCM src, size_t start, size_t
          count))
     Define a new port type.  The NAME, READ and WRITE parameters are
     initial values for those port type fields, as described below.  The
     other fields are initialized with default values and can be changed
     later.

 -- Function: SCM scm_c_make_port_with_encoding (scm_t_port_type *type,
          unsigned long mode_bits, SCM encoding, SCM
          conversion_strategy, scm_t_bits stream)
 -- Function: SCM scm_c_make_port (scm_t_port_type *type, unsigned long
          mode_bits, scm_t_bits stream)
     Make a port with the given TYPE.  The STREAM indicates the private
     data associated with the port, which your port implementation may
     later retrieve with ‘SCM_STREAM’.  The mode bits should include one
     or more of the flags ‘SCM_RDNG’ or ‘SCM_WRTNG’, indicating that the
     port is an input and/or an output port, respectively.  The mode
     bits may also include ‘SCM_BUF0’ or ‘SCM_BUFLINE’, indicating that
     the port should be unbuffered or line-buffered, respectively.  The
     default is that the port will be block-buffered.  *Note
     Buffering::.

     As you would imagine, ENCODING and CONVERSION_STRATEGY specify the
     port’s initial textual encoding and conversion strategy.  Both are
     symbols.  ‘scm_c_make_port’ is the same as
     ‘scm_c_make_port_with_encoding’, except it uses the default port
     encoding and conversion strategy.

   The port type has a number of associate procedures and properties
which collectively implement the port’s behavior.  Creating a new port
type mostly involves writing these procedures.

‘name’
     A pointer to a NUL terminated string: the name of the port type.
     This property is initialized via the first argument to
     ‘scm_make_port_type’.

‘read’
     A port’s ‘read’ implementation fills read buffers.  It should copy
     bytes to the supplied bytevector ‘dst’, starting at offset ‘start’
     and continuing for ‘count’ bytes, returning the number of bytes
     read.

‘write’
     A port’s ‘write’ implementation flushes write buffers to the
     mutable store.  A port’s ‘read’ implementation fills read buffers.
     It should write out bytes from the supplied bytevector ‘src’,
     starting at offset ‘start’ and continuing for ‘count’ bytes, and
     return the number of bytes that were written.

‘read_wait_fd’
‘write_wait_fd’
     If a port’s ‘read’ or ‘write’ function returns ‘(size_t) -1’, that
     indicates that reading or writing would block.  In that case to
     preserve the illusion of a blocking read or write operation,
     Guile’s C port run-time will ‘poll’ on the file descriptor returned
     by either the port’s ‘read_wait_fd’ or ‘write_wait_fd’ function.
     Set using

      -- Function: void scm_set_port_read_wait_fd (scm_t_port_type
               *type, int (*wait_fd) (SCM port))
      -- Function: void scm_set_port_write_wait_fd (scm_t_port_type
               *type, int (*wait_fd) (SCM port))

     Only a port type which implements the ‘read_wait_fd’ or
     ‘write_wait_fd’ port methods can usefully return ‘(size_t) -1’ from
     a read or write function.  *Note Non-Blocking I/O::, for more on
     non-blocking I/O in Guile.

‘print’
     Called when ‘write’ is called on the port, to print a port
     description.  For example, for a file port it may produce something
     like: ‘#<input: /etc/passwd 3>’.  Set using

      -- Function: void scm_set_port_print (scm_t_port_type *type, int
               (*print) (SCM port, SCM dest_port, scm_print_state
               *pstate))
          The first argument PORT is the port being printed, the second
          argument DEST_PORT is where its description should go.

‘close’
     Called when the port is closed.  It should free any resources used
     by the port.  Set using

      -- Function: void scm_set_port_close (scm_t_port_type *type, void
               (*close) (SCM port))

     By default, ports that are garbage collected just go away without
     closing.  If your port type needs to release some external resource
     like a file descriptor, or needs to make sure that its internal
     buffers are flushed even if the port is collected while it was
     open, then mark the port type as needing a close on GC.

      -- Function: void scm_set_port_needs_close_on_gc (scm_t_port_type
               *type, int needs_close_p)

‘seek’
     Set the current position of the port.  Guile will flush read and/or
     write buffers before seeking, as appropriate.

      -- Function: void scm_set_port_seek (scm_t_port_type *type,
               scm_t_off (*seek) (SCM port, scm_t_off offset, int
               whence))

‘truncate’
     Truncate the port data to be specified length.  Guile will flush
     buffers before hand, as appropriate.  Set using

      -- Function: void scm_set_port_truncate (scm_t_port_type *type,
               void (*truncate) (SCM port, scm_t_off length))

‘random_access_p’
     Determine whether this port is a random-access port.

     Seeking on a random-access port with buffered input, or switching
     to writing after reading, will cause the buffered input to be
     discarded and Guile will seek the port back the buffered number of
     bytes.  Likewise seeking on a random-access port with buffered
     output, or switching to reading after writing, will flush pending
     bytes with a call to the ‘write’ procedure.  *Note Buffering::.

     Indicate to Guile that your port needs this behavior by returning a
     nonzero value from your ‘random_access_p’ function.  The default
     implementation of this function returns nonzero if the port type
     supplies a seek implementation.

      -- Function: void scm_set_port_random_access_p (scm_t_port_type
               *type, int (*random_access_p) (SCM port));

‘get_natural_buffer_sizes’
     Guile will internally attach buffers to ports.  An input port
     always has a read buffer and an output port always has a write
     buffer.  *Note Buffering::.  A port buffer consists of a
     bytevector, along with some cursors into that bytevector denoting
     where to get and put data.

     Port implementations generally don’t have to be concerned with
     buffering: a port type’s ‘read’ or ‘write’ function will receive
     the buffer’s bytevector as an argument, along with an offset and a
     length into that bytevector, and should then either fill or empty
     that bytevector.  However in some cases, port implementations may
     be able to provide an appropriate default buffer size to Guile.

      -- Function: void scm_set_port_get_natural_buffer_sizes
               (scm_t_port_type *type, void (*get_natural_buffer_sizes)
               (SCM, size_t *read_buf_size, size_t *write_buf_size))
          Fill in READ_BUF_SIZE and WRITE_BUF_SIZE with an appropriate
          buffer size for this port, if one is known.

     File ports implement a ‘get_natural_buffer_sizes’ to let the
     operating system inform Guile about the appropriate buffer sizes
     for the particular file opened by the port.

   Note that calls to all of these methods can proceed in parallel and
concurrently and from any thread up until the point that the port is
closed.  The call to ‘close’ will happen when no other method is
running, and no method will be called after the ‘close’ method is
called.  If your port implementation needs mutual exclusion to prevent
concurrency, it is responsible for locking appropriately.


File: guile.info,  Node: Non-Blocking I/O,  Next: BOM Handling,  Prev: I/O Extensions,  Up: Input and Output

6.14.14 Non-Blocking I/O
------------------------

Most ports in Guile are “blocking”: when you try to read a character
from a port, Guile will block on the read until a character is ready, or
end-of-stream is detected.  Likewise whenever Guile goes to write
(possibly buffered) data to an output port, Guile will block until all
the data is written.

   Interacting with ports in blocking mode is very convenient: you can
write straightforward, sequential algorithms whose code flow reflects
the flow of data.  However, blocking I/O has two main limitations.

   The first is that it’s easy to get into a situation where code is
waiting on data.  Time spent waiting on data when code could be doing
something else is wasteful and prevents your program from reaching its
peak throughput.  If you implement a web server that sequentially
handles requests from clients, it’s very easy for the server to end up
waiting on a client to finish its HTTP request, or waiting on it to
consume the response.  The end result is that you are able to serve
fewer requests per second than you’d like to serve.

   The second limitation is related: a blocking parser over
user-controlled input is a denial-of-service vulnerability.  Indeed the
so-called “slow loris” attack of the early 2010s was just that: an
attack on common web servers that drip-fed HTTP requests, one character
at a time.  All it took was a handful of slow loris connections to
occupy an entire web server.

   In Guile we would like to preserve the ability to write
straightforward blocking networking processes of all kinds, but under
the hood to allow those processes to suspend their requests if they
would block.

   To do this, the first piece is to allow Guile ports to declare
themselves as being nonblocking.  This is currently supported only for
file ports, which also includes sockets, terminals, or any other port
that is backed by a file descriptor.  To do that, we use an arcane UNIX
incantation:

     (let ((flags (fcntl socket F_GETFL)))
       (fcntl socket F_SETFL (logior O_NONBLOCK flags)))

   Now the file descriptor is open in non-blocking mode.  If Guile tries
to read or write from this file and the read or write returns a result
indicating that more data can only be had by doing a blocking read or
write, Guile will block by polling on the socket’s ‘read-wait-fd’ or
‘write-wait-fd’, to preserve the illusion of a blocking read or write.
*Note I/O Extensions:: for more on those internal interfaces.

   So far we have just reproduced the status quo: the file descriptor is
non-blocking, but the operations on the port do block.  To go farther,
it would be nice if we could suspend the “thread” using delimited
continuations, and only resume the thread once the file descriptor is
readable or writable.  (*Note Prompts::).

   But here we run into a difficulty.  The ports code is implemented in
C, which means that although we can suspend the computation to some
outer prompt, we can’t resume it because Guile can’t resume delimited
continuations that capture the C stack.

   To overcome this difficulty we have created a compatible but entirely
parallel implementation of port operations.  To use this implementation,
do the following:

     (use-modules (ice-9 suspendable-ports))
     (install-suspendable-ports!)

   This will replace the core I/O primitives like ‘get-char’ and
‘put-bytevector’ with new versions that are exactly the same as the ones
in the standard library, but with two differences.  One is that when a
read or a write would block, the suspendable port operations call out
the value of the ‘current-read-waiter’ or ‘current-write-waiter’
parameter, as appropriate.  *Note Parameters::.  The default read and
write waiters do the same thing that the C read and write waiters do,
which is to poll.  User code can parameterize the waiters, though,
enabling the computation to suspend and allow the program to process
other I/O operations.  Because the new suspendable ports implementation
is written in Scheme, that suspended computation can resume again later
when it is able to make progress.  Success!

   The other main difference is that because the new ports
implementation is written in Scheme, it is slower than C, currently by a
factor of 3 or 4, though it depends on many factors.  For this reason we
have to keep the C implementations as the default ones.  One day when
Guile’s compiler is better, we can close this gap and have only one port
operation implementation again.

   Note that Guile does not currently include an implementation of the
facility to suspend the current thread and schedule other threads in the
meantime.  Before adding such a thing, we want to make sure that we’re
providing the right primitives that can be used to build schedulers and
other user-space concurrency patterns, and that the patterns that we
settle on are the right patterns.  In the meantime, have a look at 8sync
(<https://gnu.org/software/8sync>) for a prototype of an asynchronous
I/O and concurrency facility.

 -- Scheme Procedure: install-suspendable-ports!
     Replace the core ports implementation with suspendable ports, as
     described above.  This will mutate the values of the bindings like
     ‘get-char’, ‘put-u8’, and so on in place.

 -- Scheme Procedure: uninstall-suspendable-ports!
     Restore the original core ports implementation, un-doing the effect
     of ‘install-suspendable-ports!’.

 -- Scheme Parameter: current-read-waiter
 -- Scheme Parameter: current-write-waiter
     Parameters whose values are procedures of one argument, called when
     a suspendable port operation would block on a port while reading or
     writing, respectively.  The default values of these parameters do a
     blocking ‘poll’ on the port’s file descriptor.  The procedures are
     passed the port in question as their one argument.


File: guile.info,  Node: BOM Handling,  Prev: Non-Blocking I/O,  Up: Input and Output

6.14.15 Handling of Unicode Byte Order Marks
--------------------------------------------

This section documents the finer points of Guile’s handling of Unicode
byte order marks (BOMs).  A byte order mark (U+FEFF) is typically found
at the start of a UTF-16 or UTF-32 stream, to allow readers to reliably
determine the byte order.  Occasionally, a BOM is found at the start of
a UTF-8 stream, but this is much less common and not generally
recommended.

   Guile attempts to handle BOMs automatically, and in accordance with
the recommendations of the Unicode Standard, when the port encoding is
set to ‘UTF-8’, ‘UTF-16’, or ‘UTF-32’.  In brief, Guile automatically
writes a BOM at the start of a UTF-16 or UTF-32 stream, and
automatically consumes one from the start of a UTF-8, UTF-16, or UTF-32
stream.

   As specified in the Unicode Standard, a BOM is only handled specially
at the start of a stream, and only if the port encoding is set to
‘UTF-8’, ‘UTF-16’ or ‘UTF-32’.  If the port encoding is set to
‘UTF-16BE’, ‘UTF-16LE’, ‘UTF-32BE’, or ‘UTF-32LE’, then BOMs are _not_
handled specially, and none of the special handling described in this
section applies.

   • To ensure that Guile will properly detect the byte order of a
     UTF-16 or UTF-32 stream, you must perform a textual read before any
     writes, seeks, or binary I/O. Guile will not attempt to read a BOM
     unless a read is explicitly requested at the start of the stream.

   • If a textual write is performed before the first read, then an
     arbitrary byte order will be chosen.  Currently, big endian is the
     default on all platforms, but that may change in the future.  If
     you wish to explicitly control the byte order of an output stream,
     set the port encoding to ‘UTF-16BE’, ‘UTF-16LE’, ‘UTF-32BE’, or
     ‘UTF-32LE’, and explicitly write a BOM (‘#\xFEFF’) if desired.

   • If ‘set-port-encoding!’ is called in the middle of a stream, Guile
     treats this as a new logical “start of stream” for purposes of BOM
     handling, and will forget about any BOMs that had previously been
     seen.  Therefore, it may choose a different byte order than had
     been used previously.  This is intended to support multiple logical
     text streams embedded within a larger binary stream.

   • Binary I/O operations are not guaranteed to update Guile’s notion
     of whether the port is at the “start of the stream”, nor are they
     guaranteed to produce or consume BOMs.

   • For ports that support seeking (e.g.  normal files), the input and
     output streams are considered linked: if the user reads first, then
     a BOM will be consumed (if appropriate), but later writes will
     _not_ produce a BOM. Similarly, if the user writes first, then
     later reads will _not_ consume a BOM.

   • For ports that are not random access (e.g.  pipes, sockets, and
     terminals), the input and output streams are considered
     _independent_ for purposes of BOM handling: the first read will
     consume a BOM (if appropriate), and the first write will _also_
     produce a BOM (if appropriate).  However, the input and output
     streams will always use the same byte order.

   • Seeks to the beginning of a file will set the “start of stream”
     flags.  Therefore, a subsequent textual read or write will consume
     or produce a BOM. However, unlike ‘set-port-encoding!’, if a byte
     order had already been chosen for the port, it will remain in
     effect after a seek, and cannot be changed by the presence of a
     BOM. Seeks anywhere other than the beginning of a file clear the
     “start of stream” flags.


File: guile.info,  Node: Regular Expressions,  Next: LALR(1) Parsing,  Prev: Input and Output,  Up: API Reference

6.15 Regular Expressions
========================

A “regular expression” (or “regexp”) is a pattern that describes a whole
class of strings.  A full description of regular expressions and their
syntax is beyond the scope of this manual.

   If your system does not include a POSIX regular expression library,
and you have not linked Guile with a third-party regexp library such as
Rx, these functions will not be available.  You can tell whether your
Guile installation includes regular expression support by checking
whether ‘(provided? 'regex)’ returns true.

   The following regexp and string matching features are provided by the
‘(ice-9 regex)’ module.  Before using the described functions, you
should load this module by executing ‘(use-modules (ice-9 regex))’.

* Menu:

* Regexp Functions::            Functions that create and match regexps.
* Match Structures::            Finding what was matched by a regexp.
* Backslash Escapes::           Removing the special meaning of regexp
                                meta-characters.


File: guile.info,  Node: Regexp Functions,  Next: Match Structures,  Up: Regular Expressions

6.15.1 Regexp Functions
-----------------------

By default, Guile supports POSIX extended regular expressions.  That
means that the characters ‘(’, ‘)’, ‘+’ and ‘?’ are special, and must be
escaped if you wish to match the literal characters and there is no
support for “non-greedy” variants of ‘*’, ‘+’ or ‘?’.

   This regular expression interface was modeled after that implemented
by SCSH, the Scheme Shell.  It is intended to be upwardly compatible
with SCSH regular expressions.

   Zero bytes (‘#\nul’) cannot be used in regex patterns or input
strings, since the underlying C functions treat that as the end of
string.  If there’s a zero byte an error is thrown.

   Internally, patterns and input strings are converted to the current
locale’s encoding, and then passed to the C library’s regular expression
routines (*note (libc)Regular Expressions::).  The returned match
structures always point to characters in the strings, not to individual
bytes, even in the case of multi-byte encodings.

 -- Scheme Procedure: string-match pattern str [start]
     Compile the string PATTERN into a regular expression and compare it
     with STR.  The optional numeric argument START specifies the
     position of STR at which to begin matching.

     ‘string-match’ returns a “match structure” which describes what, if
     anything, was matched by the regular expression.  *Note Match
     Structures::.  If STR does not match PATTERN at all, ‘string-match’
     returns ‘#f’.

   Two examples of a match follow.  In the first example, the pattern
matches the four digits in the match string.  In the second, the pattern
matches nothing.

     (string-match "[0-9][0-9][0-9][0-9]" "blah2002")
     ⇒ #("blah2002" (4 . 8))

     (string-match "[A-Za-z]" "123456")
     ⇒ #f

   Each time ‘string-match’ is called, it must compile its PATTERN
argument into a regular expression structure.  This operation is
expensive, which makes ‘string-match’ inefficient if the same regular
expression is used several times (for example, in a loop).  For better
performance, you can compile a regular expression in advance and then
match strings against the compiled regexp.

 -- Scheme Procedure: make-regexp pat flag...
 -- C Function: scm_make_regexp (pat, flaglst)
     Compile the regular expression described by PAT, and return the
     compiled regexp structure.  If PAT does not describe a legal
     regular expression, ‘make-regexp’ throws a
     ‘regular-expression-syntax’ error.

     The FLAG arguments change the behavior of the compiled regular
     expression.  The following values may be supplied:

      -- Variable: regexp/icase
          Consider uppercase and lowercase letters to be the same when
          matching.

      -- Variable: regexp/newline
          If a newline appears in the target string, then permit the ‘^’
          and ‘$’ operators to match immediately after or immediately
          before the newline, respectively.  Also, the ‘.’ and ‘[^...]’
          operators will never match a newline character.  The intent of
          this flag is to treat the target string as a buffer containing
          many lines of text, and the regular expression as a pattern
          that may match a single one of those lines.

      -- Variable: regexp/basic
          Compile a basic (“obsolete”) regexp instead of the extended
          (“modern”) regexps that are the default.  Basic regexps do not
          consider ‘|’, ‘+’ or ‘?’ to be special characters, and require
          the ‘{...}’ and ‘(...)’ metacharacters to be backslash-escaped
          (*note Backslash Escapes::).  There are several other
          differences between basic and extended regular expressions,
          but these are the most significant.

      -- Variable: regexp/extended
          Compile an extended regular expression rather than a basic
          regexp.  This is the default behavior; this flag will not
          usually be needed.  If a call to ‘make-regexp’ includes both
          ‘regexp/basic’ and ‘regexp/extended’ flags, the one which
          comes last will override the earlier one.

 -- Scheme Procedure: regexp-exec rx str [start [flags]]
 -- C Function: scm_regexp_exec (rx, str, start, flags)
     Match the compiled regular expression RX against ‘str’.  If the
     optional integer START argument is provided, begin matching from
     that position in the string.  Return a match structure describing
     the results of the match, or ‘#f’ if no match could be found.

     The FLAGS argument changes the matching behavior.  The following
     flag values may be supplied, use ‘logior’ (*note Bitwise
     Operations::) to combine them,

      -- Variable: regexp/notbol
          Consider that the START offset into STR is not the beginning
          of a line and should not match operator ‘^’.

          If RX was created with the ‘regexp/newline’ option above, ‘^’
          will still match after a newline in STR.

      -- Variable: regexp/noteol
          Consider that the end of STR is not the end of a line and
          should not match operator ‘$’.

          If RX was created with the ‘regexp/newline’ option above, ‘$’
          will still match before a newline in STR.

     ;; Regexp to match uppercase letters
     (define r (make-regexp "[A-Z]*"))

     ;; Regexp to match letters, ignoring case
     (define ri (make-regexp "[A-Z]*" regexp/icase))

     ;; Search for bob using regexp r
     (match:substring (regexp-exec r "bob"))
     ⇒ ""                  ; no match

     ;; Search for bob using regexp ri
     (match:substring (regexp-exec ri "Bob"))
     ⇒ "Bob"               ; matched case insensitive

 -- Scheme Procedure: regexp? obj
 -- C Function: scm_regexp_p (obj)
     Return ‘#t’ if OBJ is a compiled regular expression, or ‘#f’
     otherwise.


 -- Scheme Procedure: list-matches regexp str [flags]
     Return a list of match structures which are the non-overlapping
     matches of REGEXP in STR.  REGEXP can be either a pattern string or
     a compiled regexp.  The FLAGS argument is as per ‘regexp-exec’
     above.

          (map match:substring (list-matches "[a-z]+" "abc 42 def 78"))
          ⇒ ("abc" "def")

 -- Scheme Procedure: fold-matches regexp str init proc [flags]
     Apply PROC to the non-overlapping matches of REGEXP in STR, to
     build a result.  REGEXP can be either a pattern string or a
     compiled regexp.  The FLAGS argument is as per ‘regexp-exec’ above.

     PROC is called as ‘(PROC match prev)’ where MATCH is a match
     structure and PREV is the previous return from PROC.  For the first
     call PREV is the given INIT parameter.  ‘fold-matches’ returns the
     final value from PROC.

     For example to count matches,

          (fold-matches "[a-z][0-9]" "abc x1 def y2" 0
                        (lambda (match count)
                          (1+ count)))
          ⇒ 2


   Regular expressions are commonly used to find patterns in one string
and replace them with the contents of another string.  The following
functions are convenient ways to do this.

 -- Scheme Procedure: regexp-substitute port match item ...
     Write to PORT selected parts of the match structure MATCH.  Or if
     PORT is ‘#f’ then form a string from those parts and return that.

     Each ITEM specifies a part to be written, and may be one of the
     following,

        • A string.  String arguments are written out verbatim.

        • An integer.  The submatch with that number is written
          (‘match:substring’).  Zero is the entire match.

        • The symbol ‘pre’.  The portion of the matched string preceding
          the regexp match is written (‘match:prefix’).

        • The symbol ‘post’.  The portion of the matched string
          following the regexp match is written (‘match:suffix’).

     For example, changing a match and retaining the text before and
     after,

          (regexp-substitute #f (string-match "[0-9]+" "number 25 is good")
                             'pre "37" 'post)
          ⇒ "number 37 is good"

     Or matching a YYYYMMDD format date such as ‘20020828’ and
     re-ordering and hyphenating the fields.

          (define date-regex
             "([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])")
          (define s "Date 20020429 12am.")
          (regexp-substitute #f (string-match date-regex s)
                             'pre 2 "-" 3 "-" 1 'post " (" 0 ")")
          ⇒ "Date 04-29-2002 12am. (20020429)"

 -- Scheme Procedure: regexp-substitute/global port regexp target
          item...
     Write to PORT selected parts of matches of REGEXP in TARGET.  If
     PORT is ‘#f’ then form a string from those parts and return that.
     REGEXP can be a string or a compiled regex.

     This is similar to ‘regexp-substitute’, but allows global
     substitutions on TARGET.  Each ITEM behaves as per
     ‘regexp-substitute’, with the following differences,

        • A function.  Called as ‘(ITEM match)’ with the match structure
          for the REGEXP match, it should return a string to be written
          to PORT.

        • The symbol ‘post’.  This doesn’t output anything, but instead
          causes ‘regexp-substitute/global’ to recurse on the unmatched
          portion of TARGET.

          This _must_ be supplied to perform a global search and replace
          on TARGET; without it ‘regexp-substitute/global’ returns after
          a single match and output.

     For example, to collapse runs of tabs and spaces to a single hyphen
     each,

          (regexp-substitute/global #f "[ \t]+"  "this   is   the text"
                                    'pre "-" 'post)
          ⇒ "this-is-the-text"

     Or using a function to reverse the letters in each word,

          (regexp-substitute/global #f "[a-z]+"  "to do and not-do"
            'pre (lambda (m) (string-reverse (match:substring m))) 'post)
          ⇒ "ot od dna ton-od"

     Without the ‘post’ symbol, just one regexp match is made.  For
     example the following is the date example from ‘regexp-substitute’
     above, without the need for the separate ‘string-match’ call.

          (define date-regex
             "([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])")
          (define s "Date 20020429 12am.")
          (regexp-substitute/global #f date-regex s
                                    'pre 2 "-" 3 "-" 1 'post " (" 0 ")")

          ⇒ "Date 04-29-2002 12am. (20020429)"


File: guile.info,  Node: Match Structures,  Next: Backslash Escapes,  Prev: Regexp Functions,  Up: Regular Expressions

6.15.2 Match Structures
-----------------------

A “match structure” is the object returned by ‘string-match’ and
‘regexp-exec’.  It describes which portion of a string, if any, matched
the given regular expression.  Match structures include: a reference to
the string that was checked for matches; the starting and ending
positions of the regexp match; and, if the regexp included any
parenthesized subexpressions, the starting and ending positions of each
submatch.

   In each of the regexp match functions described below, the ‘match’
argument must be a match structure returned by a previous call to
‘string-match’ or ‘regexp-exec’.  Most of these functions return some
information about the original target string that was matched against a
regular expression; we will call that string TARGET for easy reference.

 -- Scheme Procedure: regexp-match? obj
     Return ‘#t’ if OBJ is a match structure returned by a previous call
     to ‘regexp-exec’, or ‘#f’ otherwise.

 -- Scheme Procedure: match:substring match [n]
     Return the portion of TARGET matched by subexpression number N.
     Submatch 0 (the default) represents the entire regexp match.  If
     the regular expression as a whole matched, but the subexpression
     number N did not match, return ‘#f’.

     (define s (string-match "[0-9][0-9][0-9][0-9]" "blah2002foo"))
     (match:substring s)
     ⇒ "2002"

     ;; match starting at offset 6 in the string
     (match:substring
       (string-match "[0-9][0-9][0-9][0-9]" "blah987654" 6))
     ⇒ "7654"

 -- Scheme Procedure: match:start match [n]
     Return the starting position of submatch number N.

   In the following example, the result is 4, since the match starts at
character index 4:

     (define s (string-match "[0-9][0-9][0-9][0-9]" "blah2002foo"))
     (match:start s)
     ⇒ 4

 -- Scheme Procedure: match:end match [n]
     Return the ending position of submatch number N.

   In the following example, the result is 8, since the match runs
between characters 4 and 8 (i.e. the “2002”).

     (define s (string-match "[0-9][0-9][0-9][0-9]" "blah2002foo"))
     (match:end s)
     ⇒ 8

 -- Scheme Procedure: match:prefix match
     Return the unmatched portion of TARGET preceding the regexp match.

          (define s (string-match "[0-9][0-9][0-9][0-9]" "blah2002foo"))
          (match:prefix s)
          ⇒ "blah"

 -- Scheme Procedure: match:suffix match
     Return the unmatched portion of TARGET following the regexp match.

     (define s (string-match "[0-9][0-9][0-9][0-9]" "blah2002foo"))
     (match:suffix s)
     ⇒ "foo"

 -- Scheme Procedure: match:count match
     Return the number of parenthesized subexpressions from MATCH.  Note
     that the entire regular expression match itself counts as a
     subexpression, and failed submatches are included in the count.

 -- Scheme Procedure: match:string match
     Return the original TARGET string.

     (define s (string-match "[0-9][0-9][0-9][0-9]" "blah2002foo"))
     (match:string s)
     ⇒ "blah2002foo"


File: guile.info,  Node: Backslash Escapes,  Prev: Match Structures,  Up: Regular Expressions

6.15.3 Backslash Escapes
------------------------

Sometimes you will want a regexp to match characters like ‘*’ or ‘$’
exactly.  For example, to check whether a particular string represents a
menu entry from an Info node, it would be useful to match it against a
regexp like ‘^* [^:]*::’.  However, this won’t work; because the
asterisk is a metacharacter, it won’t match the ‘*’ at the beginning of
the string.  In this case, we want to make the first asterisk un-magic.

   You can do this by preceding the metacharacter with a backslash
character ‘\’.  (This is also called “quoting” the metacharacter, and is
known as a “backslash escape”.)  When Guile sees a backslash in a
regular expression, it considers the following glyph to be an ordinary
character, no matter what special meaning it would ordinarily have.
Therefore, we can make the above example work by changing the regexp to
‘^\* [^:]*::’.  The ‘\*’ sequence tells the regular expression engine to
match only a single asterisk in the target string.

   Since the backslash is itself a metacharacter, you may force a regexp
to match a backslash in the target string by preceding the backslash
with itself.  For example, to find variable references in a TeX program,
you might want to find occurrences of the string ‘\let\’ followed by any
number of alphabetic characters.  The regular expression
‘\\let\\[A-Za-z]*’ would do this: the double backslashes in the regexp
each match a single backslash in the target string.

 -- Scheme Procedure: regexp-quote str
     Quote each special character found in STR with a backslash, and
     return the resulting string.

   *Very important:* Using backslash escapes in Guile source code (as in
Emacs Lisp or C) can be tricky, because the backslash character has
special meaning for the Guile reader.  For example, if Guile encounters
the character sequence ‘\n’ in the middle of a string while processing
Scheme code, it replaces those characters with a newline character.
Similarly, the character sequence ‘\t’ is replaced by a horizontal tab.
Several of these “escape sequences” are processed by the Guile reader
before your code is executed.  Unrecognized escape sequences are
ignored: if the characters ‘\*’ appear in a string, they will be
translated to the single character ‘*’.

   This translation is obviously undesirable for regular expressions,
since we want to be able to include backslashes in a string in order to
escape regexp metacharacters.  Therefore, to make sure that a backslash
is preserved in a string in your Guile program, you must use _two_
consecutive backslashes:

     (define Info-menu-entry-pattern (make-regexp "^\\* [^:]*"))

   The string in this example is preprocessed by the Guile reader before
any code is executed.  The resulting argument to ‘make-regexp’ is the
string ‘^\* [^:]*’, which is what we really want.

   This also means that in order to write a regular expression that
matches a single backslash character, the regular expression string in
the source code must include _four_ backslashes.  Each consecutive pair
of backslashes gets translated by the Guile reader to a single
backslash, and the resulting double-backslash is interpreted by the
regexp engine as matching a single backslash character.  Hence:

     (define tex-variable-pattern (make-regexp "\\\\let\\\\=[A-Za-z]*"))

   The reason for the unwieldiness of this syntax is historical.  Both
regular expression pattern matchers and Unix string processing systems
have traditionally used backslashes with the special meanings described
above.  The POSIX regular expression specification and ANSI C standard
both require these semantics.  Attempting to abandon either convention
would cause other kinds of compatibility problems, possibly more severe
ones.  Therefore, without extending the Scheme reader to support strings
with different quoting conventions (an ungainly and confusing extension
when implemented in other languages), we must adhere to this cumbersome
escape syntax.


File: guile.info,  Node: LALR(1) Parsing,  Next: PEG Parsing,  Prev: Regular Expressions,  Up: API Reference

6.16 LALR(1) Parsing
====================

The ‘(system base lalr)’ module provides the ‘lalr-scm’ LALR(1) parser
generator by Dominique Boucher (https://github.com/schemeway/lalr-scm/).
‘lalr-scm’ uses the same algorithm as GNU Bison (*note Introduction to
Bison: (bison)Introduction.).  Parsers are defined using the
‘lalr-parser’ macro.

 -- Scheme Syntax: lalr-parser [OPTIONS] TOKENS RULES...
     Generate an LALR(1) syntax analyzer.  TOKENS is a list of symbols
     representing the terminal symbols of the grammar.  RULES are the
     grammar production rules.

     Each rule has the form ‘(NON-TERMINAL (RHS ...) : ACTION ...)’,
     where NON-TERMINAL is the name of the rule, RHS are the right-hand
     sides, i.e., the production rule, and ACTION is a semantic action
     associated with the rule.

     The generated parser is a two-argument procedure that takes a
     “tokenizer” and a “syntax error procedure”.  The tokenizer should
     be a thunk that returns lexical tokens as produced by
     ‘make-lexical-token’.  The syntax error procedure may be called
     with at least an error message (a string), and optionally the
     lexical token that caused the error.

   Please refer to the ‘lalr-scm’ documentation for details.


File: guile.info,  Node: PEG Parsing,  Next: Read/Load/Eval/Compile,  Prev: LALR(1) Parsing,  Up: API Reference

6.17 PEG Parsing
================

Parsing Expression Grammars (PEGs) are a way of specifying formal
languages for text processing.  They can be used either for matching
(like regular expressions) or for building recursive descent parsers
(like lex/yacc).  Guile uses a superset of PEG syntax that allows more
control over what information is preserved during parsing.

   Wikipedia has a clear and concise introduction to PEGs if you want to
familiarize yourself with the syntax:
<http://en.wikipedia.org/wiki/Parsing_expression_grammar>.

   The ‘(ice-9 peg)’ module works by compiling PEGs down to lambda
expressions.  These can either be stored in variables at compile-time by
the define macros (‘define-peg-pattern’ and
‘define-peg-string-patterns’) or calculated explicitly at runtime with
the compile functions (‘compile-peg-pattern’ and ‘peg-string-compile’).

   They can then be used for either parsing (‘match-pattern’) or
searching (‘search-for-pattern’).  For convenience, ‘search-for-pattern’
also takes pattern literals in case you want to inline a simple search
(people often use regular expressions this way).

   The rest of this documentation consists of a syntax reference, an API
reference, and a tutorial.

* Menu:

* PEG Syntax Reference::
* PEG API Reference::
* PEG Tutorial::
* PEG Internals::


File: guile.info,  Node: PEG Syntax Reference,  Next: PEG API Reference,  Up: PEG Parsing

6.17.1 PEG Syntax Reference
---------------------------

Normal PEG Syntax:
..................

 -- PEG Pattern: sequence a b
     Parses A.  If this succeeds, continues to parse B from the end of
     the text parsed as A.  Succeeds if both A and B succeed.

     ‘"a b"’

     ‘(and a b)’

 -- PEG Pattern: ordered choice a b
     Parses A.  If this fails, backtracks and parses B.  Succeeds if
     either A or B succeeds.

     ‘"a/b"’

     ‘(or a b)’

 -- PEG Pattern: zero or more a
     Parses A as many times in a row as it can, starting each A at the
     end of the text parsed by the previous A.  Always succeeds.

     ‘"a*"’

     ‘(* a)’

 -- PEG Pattern: one or more a
     Parses A as many times in a row as it can, starting each A at the
     end of the text parsed by the previous A.  Succeeds if at least one
     A was parsed.

     ‘"a+"’

     ‘(+ a)’

 -- PEG Pattern: optional a
     Tries to parse A.  Succeeds if A succeeds.

     ‘"a?"’

     ‘(? a)’

 -- PEG Pattern: followed by a
     Makes sure it is possible to parse A, but does not actually parse
     it.  Succeeds if A would succeed.

     ‘"&a"’

     ‘(followed-by a)’

 -- PEG Pattern: not followed by a
     Makes sure it is impossible to parse A, but does not actually parse
     it.  Succeeds if A would fail.

     ‘"!a"’

     ‘(not-followed-by a)’

 -- PEG Pattern: string literal ``abc''
     Parses the string "ABC".  Succeeds if that parsing succeeds.

     ‘"'abc'"’

     ‘"abc"’

 -- PEG Pattern: any character
     Parses any single character.  Succeeds unless there is no more text
     to be parsed.

     ‘"."’

     ‘peg-any’

 -- PEG Pattern: character class a b
     Alternative syntax for “Ordered Choice A B” if A and B are
     characters.

     ‘"[ab]"’

     ‘(or "a" "b")’

 -- PEG Pattern: range of characters a z
     Parses any character falling between A and Z.

     ‘"[a-z]"’

     ‘(range #\a #\z)’

   Example:

     "(a !b / c &d*) 'e'+"

   Would be:

     (and
      (or
       (and a (not-followed-by b))
       (and c (followed-by (* d))))
      (+ "e"))

Extended Syntax
...............

There is some extra syntax for S-expressions.

 -- PEG Pattern: ignore a
     Ignore the text matching A

 -- PEG Pattern: capture a
     Capture the text matching A.

 -- PEG Pattern: peg a
     Embed the PEG pattern A using string syntax.

   Example:

     "!a / 'b'"

   Is equivalent to

     (or (peg "!a") "b")

   and

     (or (not-followed-by a) "b")


File: guile.info,  Node: PEG API Reference,  Next: PEG Tutorial,  Prev: PEG Syntax Reference,  Up: PEG Parsing

6.17.2 PEG API Reference
------------------------

Define Macros
.............

The most straightforward way to define a PEG is by using one of the
define macros (both of these macroexpand into ‘define’ expressions).
These macros bind parsing functions to variables.  These parsing
functions may be invoked by ‘match-pattern’ or ‘search-for-pattern’,
which return a PEG match record.  Raw data can be retrieved from this
record with the PEG match deconstructor functions.  More complicated
(and perhaps enlightening) examples can be found in the tutorial.

 -- Scheme Macro: define-peg-string-patterns peg-string
     Defines all the nonterminals in the PEG PEG-STRING.  More
     precisely, ‘define-peg-string-patterns’ takes a superset of PEGs.
     A normal PEG has a ‘<-’ between the nonterminal and the pattern.
     ‘define-peg-string-patterns’ uses this symbol to determine what
     information it should propagate up the parse tree.  The normal ‘<-’
     propagates the matched text up the parse tree, ‘<--’ propagates the
     matched text up the parse tree tagged with the name of the
     nonterminal, and ‘<’ discards that matched text and propagates
     nothing up the parse tree.  Also, nonterminals may consist of any
     alphanumeric character or a “-” character (in normal PEGs
     nonterminals can only be alphabetic).

     For example, if we:
          (define-peg-string-patterns
            "as <- 'a'+
          bs <- 'b'+
          as-or-bs <- as/bs")
          (define-peg-string-patterns
            "as-tag <-- 'a'+
          bs-tag <-- 'b'+
          as-or-bs-tag <-- as-tag/bs-tag")
     Then:
          (match-pattern as-or-bs "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: aa>
          (match-pattern as-or-bs-tag "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: (as-or-bs-tag (as-tag aa))>

     Note that in doing this, we have bound 6 variables at the toplevel
     (AS, BS, AS-OR-BS, AS-TAG, BS-TAG, and AS-OR-BS-TAG).

 -- Scheme Macro: define-peg-pattern name capture-type peg-sexp
     Defines a single nonterminal NAME.  CAPTURE-TYPE determines how
     much information is passed up the parse tree.  PEG-SEXP is a PEG in
     S-expression form.

     Possible values for capture-type:

     ‘all’
          passes the matched text up the parse tree tagged with the name
          of the nonterminal.
     ‘body’
          passes the matched text up the parse tree.
     ‘none’
          passes nothing up the parse tree.

     For Example, if we:
          (define-peg-pattern as body (+ "a"))
          (define-peg-pattern bs body (+ "b"))
          (define-peg-pattern as-or-bs body (or as bs))
          (define-peg-pattern as-tag all (+ "a"))
          (define-peg-pattern bs-tag all (+ "b"))
          (define-peg-pattern as-or-bs-tag all (or as-tag bs-tag))
     Then:
          (match-pattern as-or-bs "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: aa>
          (match-pattern as-or-bs-tag "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: (as-or-bs-tag (as-tag aa))>

     Note that in doing this, we have bound 6 variables at the toplevel
     (AS, BS, AS-OR-BS, AS-TAG, BS-TAG, and AS-OR-BS-TAG).

Compile Functions
.................

It is sometimes useful to be able to compile anonymous PEG patterns at
runtime.  These functions let you do that using either syntax.

 -- Scheme Procedure: peg-string-compile peg-string capture-type
     Compiles the PEG pattern in PEG-STRING propagating according to
     CAPTURE-TYPE (capture-type can be any of the values from
     ‘define-peg-pattern’).

 -- Scheme Procedure: compile-peg-pattern peg-sexp capture-type
     Compiles the PEG pattern in PEG-SEXP propagating according to
     CAPTURE-TYPE (capture-type can be any of the values from
     ‘define-peg-pattern’).

   The functions return syntax objects, which can be useful if you want
to use them in macros.  If all you want is to define a new nonterminal,
you can do the following:

     (define exp '(+ "a"))
     (define as (compile (compile-peg-pattern exp 'body)))

   You can use this nonterminal with all of the regular PEG functions:

     (match-pattern as "aaaaa") ⇒
     #<peg start: 0 end: 5 string: bbbbb tree: bbbbb>

Parsing & Matching Functions
............................

For our purposes, “parsing” means parsing a string into a tree starting
from the first character, while “matching” means searching through the
string for a substring.  In practice, the only difference between the
two functions is that ‘match-pattern’ gives up if it can’t find a valid
substring starting at index 0 and ‘search-for-pattern’ keeps looking.
They are both equally capable of “parsing” and “matching” given those
constraints.

 -- Scheme Procedure: match-pattern nonterm string
     Parses STRING using the PEG stored in NONTERM.  If no match was
     found, ‘match-pattern’ returns false.  If a match was found, a PEG
     match record is returned.

     The ‘capture-type’ argument to ‘define-peg-pattern’ allows you to
     choose what information to hold on to while parsing.  The options
     are:

     ‘all’
          tag the matched text with the nonterminal
     ‘body’
          just the matched text
     ‘none’
          nothing

          (define-peg-pattern as all (+ "a"))
          (match-pattern as "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: (as aa)>

          (define-peg-pattern as body (+ "a"))
          (match-pattern as "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: aa>

          (define-peg-pattern as none (+ "a"))
          (match-pattern as "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: ()>

          (define-peg-pattern bs body (+ "b"))
          (match-pattern bs "aabbcc") ⇒
          #f

 -- Scheme Macro: search-for-pattern nonterm-or-peg string
     Searches through STRING looking for a matching subexpression.
     NONTERM-OR-PEG can either be a nonterminal or a literal PEG
     pattern.  When a literal PEG pattern is provided,
     ‘search-for-pattern’ works very similarly to the regular expression
     searches many hackers are used to.  If no match was found,
     ‘search-for-pattern’ returns false.  If a match was found, a PEG
     match record is returned.

          (define-peg-pattern as body (+ "a"))
          (search-for-pattern as "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: aa>
          (search-for-pattern (+ "a") "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: aa>
          (search-for-pattern "'a'+" "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: aa>

          (define-peg-pattern as all (+ "a"))
          (search-for-pattern as "aabbcc") ⇒
          #<peg start: 0 end: 2 string: aabbcc tree: (as aa)>

          (define-peg-pattern bs body (+ "b"))
          (search-for-pattern bs "aabbcc") ⇒
          #<peg start: 2 end: 4 string: aabbcc tree: bb>
          (search-for-pattern (+ "b") "aabbcc") ⇒
          #<peg start: 2 end: 4 string: aabbcc tree: bb>
          (search-for-pattern "'b'+" "aabbcc") ⇒
          #<peg start: 2 end: 4 string: aabbcc tree: bb>

          (define-peg-pattern zs body (+ "z"))
          (search-for-pattern zs "aabbcc") ⇒
          #f
          (search-for-pattern (+ "z") "aabbcc") ⇒
          #f
          (search-for-pattern "'z'+" "aabbcc") ⇒
          #f

PEG Match Records
.................

The ‘match-pattern’ and ‘search-for-pattern’ functions both return PEG
match records.  Actual information can be extracted from these with the
following functions.

 -- Scheme Procedure: peg:string match-record
     Returns the original string that was parsed in the creation of
     ‘match-record’.

 -- Scheme Procedure: peg:start match-record
     Returns the index of the first parsed character in the original
     string (from ‘peg:string’).  If this is the same as ‘peg:end’,
     nothing was parsed.

 -- Scheme Procedure: peg:end match-record
     Returns one more than the index of the last parsed character in the
     original string (from ‘peg:string’).  If this is the same as
     ‘peg:start’, nothing was parsed.

 -- Scheme Procedure: peg:substring match-record
     Returns the substring parsed by ‘match-record’.  This is equivalent
     to ‘(substring (peg:string match-record) (peg:start match-record)
     (peg:end match-record))’.

 -- Scheme Procedure: peg:tree match-record
     Returns the tree parsed by ‘match-record’.

 -- Scheme Procedure: peg-record? match-record
     Returns true if ‘match-record’ is a PEG match record, or false
     otherwise.

   Example:
     (define-peg-pattern bs all (peg "'b'+"))

     (search-for-pattern bs "aabbcc") ⇒
     #<peg start: 2 end: 4 string: aabbcc tree: (bs bb)>

     (let ((pm (search-for-pattern bs "aabbcc")))
        `((string ,(peg:string pm))
          (start ,(peg:start pm))
          (end ,(peg:end pm))
          (substring ,(peg:substring pm))
          (tree ,(peg:tree pm))
          (record? ,(peg-record? pm)))) ⇒
     ((string "aabbcc")
      (start 2)
      (end 4)
      (substring "bb")
      (tree (bs "bb"))
      (record? #t))

Miscellaneous
.............

 -- Scheme Procedure: context-flatten tst lst
     Takes a predicate TST and a list LST.  Flattens LST until all
     elements are either atoms or satisfy TST.  If LST itself satisfies
     TST, ‘(list lst)’ is returned (this is a flat list whose only
     element satisfies TST).

          (context-flatten (lambda (x) (and (number? (car x)) (= (car x) 1))) '(2 2 (1 1 (2 2)) (2 2 (1 1)))) ⇒
          (2 2 (1 1 (2 2)) 2 2 (1 1))
          (context-flatten (lambda (x) (and (number? (car x)) (= (car x) 1))) '(1 1 (1 1 (2 2)) (2 2 (1 1)))) ⇒
          ((1 1 (1 1 (2 2)) (2 2 (1 1))))

     If you’re wondering why this is here, take a look at the tutorial.

 -- Scheme Procedure: keyword-flatten terms lst
     A less general form of ‘context-flatten’.  Takes a list of terminal
     atoms ‘terms’ and flattens LST until all elements are either atoms,
     or lists which have an atom from ‘terms’ as their first element.
          (keyword-flatten '(a b) '(c a b (a c) (b c) (c (b a) (c a)))) ⇒
          (c a b (a c) (b c) c (b a) c a)

     If you’re wondering why this is here, take a look at the tutorial.


File: guile.info,  Node: PEG Tutorial,  Next: PEG Internals,  Prev: PEG API Reference,  Up: PEG Parsing

6.17.3 PEG Tutorial
-------------------

Parsing /etc/passwd
...................

This example will show how to parse /etc/passwd using PEGs.

   First we define an example /etc/passwd file:

     (define *etc-passwd*
       "root:x:0:0:root:/root:/bin/bash
     daemon:x:1:1:daemon:/usr/sbin:/bin/sh
     bin:x:2:2:bin:/bin:/bin/sh
     sys:x:3:3:sys:/dev:/bin/sh
     nobody:x:65534:65534:nobody:/nonexistent:/bin/sh
     messagebus:x:103:107::/var/run/dbus:/bin/false
     ")

   As a first pass at this, we might want to have all the entries in
/etc/passwd in a list.

   Doing this with string-based PEG syntax would look like this:
     (define-peg-string-patterns
       "passwd <- entry* !.
     entry <-- (! NL .)* NL*
     NL < '\n'")

   A ‘passwd’ file is 0 or more entries (‘entry*’) until the end of the
file (‘!.’ (‘.’ is any character, so ‘!.’ means “not anything”)).  We
want to capture the data in the nonterminal ‘passwd’, but not tag it
with the name, so we use ‘<-’.

   An entry is a series of 0 or more characters that aren’t newlines
(‘(! NL .)*’) followed by 0 or more newlines (‘NL*’).  We want to tag
all the entries with ‘entry’, so we use ‘<--’.

   A newline is just a literal newline (‘'\n'’).  We don’t want a bunch
of newlines cluttering up the output, so we use ‘<’ to throw away the
captured data.

   Here is the same PEG defined using S-expressions:
     (define-peg-pattern passwd body (and (* entry) (not-followed-by peg-any)))
     (define-peg-pattern entry all (and (* (and (not-followed-by NL) peg-any))
     			       (* NL)))
     (define-peg-pattern NL none "\n")

   Obviously this is much more verbose.  On the other hand, it’s more
explicit, and thus easier to build automatically.  However, there are
some tricks that make S-expressions easier to use in some cases.  One is
the ‘ignore’ keyword; the string syntax has no way to say “throw away
this text” except breaking it out into a separate nonterminal.  For
instance, to throw away the newlines we had to define ‘NL’.  In the
S-expression syntax, we could have simply written ‘(ignore "\n")’.
Also, for the cases where string syntax is really much cleaner, the
‘peg’ keyword can be used to embed string syntax in S-expression syntax.
For instance, we could have written:

     (define-peg-pattern passwd body (peg "entry* !."))

   However we define it, parsing ‘*etc-passwd*’ with the ‘passwd’
nonterminal yields the same results:

     (peg:tree (match-pattern passwd *etc-passwd*)) ⇒
     ((entry "root:x:0:0:root:/root:/bin/bash")
      (entry "daemon:x:1:1:daemon:/usr/sbin:/bin/sh")
      (entry "bin:x:2:2:bin:/bin:/bin/sh")
      (entry "sys:x:3:3:sys:/dev:/bin/sh")
      (entry "nobody:x:65534:65534:nobody:/nonexistent:/bin/sh")
      (entry "messagebus:x:103:107::/var/run/dbus:/bin/false"))

   However, here is something to be wary of:

     (peg:tree (match-pattern passwd "one entry")) ⇒
     (entry "one entry")

   By default, the parse trees generated by PEGs are compressed as much
as possible without losing information.  It may not look like this is
what you want at first, but uncompressed parse trees are an enormous
headache (there’s no easy way to predict how deep particular lists will
nest, there are empty lists littered everywhere, etc.  etc.).  One
side-effect of this, however, is that sometimes the compressor is too
aggressive.  No information is discarded when ‘((entry "one entry"))’ is
compressed to ‘(entry "one entry")’, but in this particular case it
probably isn’t what we want.

   There are two functions for easily dealing with this:
‘keyword-flatten’ and ‘context-flatten’.  The ‘keyword-flatten’ function
takes a list of keywords and a list to flatten, then tries to coerce the
list such that the first element of all sublists is one of the keywords.
The ‘context-flatten’ function is similar, but instead of a list of
keywords it takes a predicate that should indicate whether a given
sublist is good enough (refer to the API reference for more details).

   What we want here is ‘keyword-flatten’.
     (keyword-flatten '(entry) (peg:tree (match-pattern passwd *etc-passwd*))) ⇒
     ((entry "root:x:0:0:root:/root:/bin/bash")
      (entry "daemon:x:1:1:daemon:/usr/sbin:/bin/sh")
      (entry "bin:x:2:2:bin:/bin:/bin/sh")
      (entry "sys:x:3:3:sys:/dev:/bin/sh")
      (entry "nobody:x:65534:65534:nobody:/nonexistent:/bin/sh")
      (entry "messagebus:x:103:107::/var/run/dbus:/bin/false"))
     (keyword-flatten '(entry) (peg:tree (match-pattern passwd "one entry"))) ⇒
     ((entry "one entry"))

   Of course, this is a somewhat contrived example.  In practice we
would probably just tag the ‘passwd’ nonterminal to remove the ambiguity
(using either the ‘all’ keyword for S-expressions or the ‘<--’ symbol
for strings)..

     (define-peg-pattern tag-passwd all (peg "entry* !."))
     (peg:tree (match-pattern tag-passwd *etc-passwd*)) ⇒
     (tag-passwd
       (entry "root:x:0:0:root:/root:/bin/bash")
       (entry "daemon:x:1:1:daemon:/usr/sbin:/bin/sh")
       (entry "bin:x:2:2:bin:/bin:/bin/sh")
       (entry "sys:x:3:3:sys:/dev:/bin/sh")
       (entry "nobody:x:65534:65534:nobody:/nonexistent:/bin/sh")
       (entry "messagebus:x:103:107::/var/run/dbus:/bin/false"))
     (peg:tree (match-pattern tag-passwd "one entry"))
     (tag-passwd
       (entry "one entry"))

   If you’re ever uncertain about the potential results of parsing
something, remember the two absolute rules:
  1. No parsing information will ever be discarded.
  2. There will never be any lists with fewer than 2 elements.

   For the purposes of (1), "parsing information" means things tagged
with the ‘any’ keyword or the ‘<--’ symbol.  Plain strings will be
concatenated.

   Let’s extend this example a bit more and actually pull some useful
information out of the passwd file:

     (define-peg-string-patterns
       "passwd <-- entry* !.
     entry <-- login C pass C uid C gid C nameORcomment C homedir C shell NL*
     login <-- text
     pass <-- text
     uid <-- [0-9]*
     gid <-- [0-9]*
     nameORcomment <-- text
     homedir <-- path
     shell <-- path
     path <-- (SLASH pathELEMENT)*
     pathELEMENT <-- (!NL !C  !'/' .)*
     text <- (!NL !C  .)*
     C < ':'
     NL < '\n'
     SLASH < '/'")

   This produces rather pretty parse trees:
     (passwd
       (entry (login "root")
              (pass "x")
              (uid "0")
              (gid "0")
              (nameORcomment "root")
              (homedir (path (pathELEMENT "root")))
              (shell (path (pathELEMENT "bin") (pathELEMENT "bash"))))
       (entry (login "daemon")
              (pass "x")
              (uid "1")
              (gid "1")
              (nameORcomment "daemon")
              (homedir
                (path (pathELEMENT "usr") (pathELEMENT "sbin")))
              (shell (path (pathELEMENT "bin") (pathELEMENT "sh"))))
       (entry (login "bin")
              (pass "x")
              (uid "2")
              (gid "2")
              (nameORcomment "bin")
              (homedir (path (pathELEMENT "bin")))
              (shell (path (pathELEMENT "bin") (pathELEMENT "sh"))))
       (entry (login "sys")
              (pass "x")
              (uid "3")
              (gid "3")
              (nameORcomment "sys")
              (homedir (path (pathELEMENT "dev")))
              (shell (path (pathELEMENT "bin") (pathELEMENT "sh"))))
       (entry (login "nobody")
              (pass "x")
              (uid "65534")
              (gid "65534")
              (nameORcomment "nobody")
              (homedir (path (pathELEMENT "nonexistent")))
              (shell (path (pathELEMENT "bin") (pathELEMENT "sh"))))
       (entry (login "messagebus")
              (pass "x")
              (uid "103")
              (gid "107")
              nameORcomment
              (homedir
                (path (pathELEMENT "var")
                      (pathELEMENT "run")
                      (pathELEMENT "dbus")))
              (shell (path (pathELEMENT "bin") (pathELEMENT "false")))))

   Notice that when there’s no entry in a field (e.g.  ‘nameORcomment’
for messagebus) the symbol is inserted.  This is the “don’t throw away
any information” rule—we succesfully matched a ‘nameORcomment’ of 0
characters (since we used ‘*’ when defining it).  This is usually what
you want, because it allows you to e.g.  use ‘list-ref’ to pull out
elements (since they all have known offsets).

   If you’d prefer not to have symbols for empty matches, you can
replace the ‘*’ with a ‘+’ and add a ‘?’ after the ‘nameORcomment’ in
‘entry’.  Then it will try to parse 1 or more characters, fail
(inserting nothing into the parse tree), but continue because it didn’t
have to match the nameORcomment to continue.

Embedding Arithmetic Expressions
................................

We can parse simple mathematical expressions with the following PEG:

     (define-peg-string-patterns
       "expr <- sum
     sum <-- (product ('+' / '-') sum) / product
     product <-- (value ('*' / '/') product) / value
     value <-- number / '(' expr ')'
     number <-- [0-9]+")

   Then:
     (peg:tree (match-pattern expr "1+1/2*3+(1+1)/2")) ⇒
     (sum (product (value (number "1")))
          "+"
          (sum (product
                 (value (number "1"))
                 "/"
                 (product
                   (value (number "2"))
                   "*"
                   (product (value (number "3")))))
               "+"
               (sum (product
                      (value "("
                             (sum (product (value (number "1")))
                                  "+"
                                  (sum (product (value (number "1")))))
                             ")")
                      "/"
                      (product (value (number "2")))))))

   There is very little wasted effort in this PEG. The ‘number’
nonterminal has to be tagged because otherwise the numbers might run
together with the arithmetic expressions during the string concatenation
stage of parse-tree compression (the parser will see “1” followed by “/”
and decide to call it “1/”).  When in doubt, tag.

   It is very easy to turn these parse trees into lisp expressions:

     (define (parse-sum sum left . rest)
       (if (null? rest)
           (apply parse-product left)
           (list (string->symbol (car rest))
     	    (apply parse-product left)
     	    (apply parse-sum (cadr rest)))))

     (define (parse-product product left . rest)
       (if (null? rest)
           (apply parse-value left)
           (list (string->symbol (car rest))
     	    (apply parse-value left)
     	    (apply parse-product (cadr rest)))))

     (define (parse-value value first . rest)
       (if (null? rest)
           (string->number (cadr first))
           (apply parse-sum (car rest))))

     (define parse-expr parse-sum)

   (Notice all these functions look very similar; for a more complicated
PEG, it would be worth abstracting.)

   Then:
     (apply parse-expr (peg:tree (match-pattern expr "1+1/2*3+(1+1)/2"))) ⇒
     (+ 1 (+ (/ 1 (* 2 3)) (/ (+ 1 1) 2)))

   But wait!  The associativity is wrong!  Where it says ‘(/ 1 (* 2
3))’, it should say ‘(* (/ 1 2) 3)’.

   It’s tempting to try replacing e.g.  ‘"sum <-- (product ('+' / '-')
sum) / product"’ with ‘"sum <-- (sum ('+' / '-') product) / product"’,
but this is a Bad Idea.  PEGs don’t support left recursion.  To see why,
imagine what the parser will do here.  When it tries to parse ‘sum’, it
first has to try and parse ‘sum’.  But to do that, it first has to try
and parse ‘sum’.  This will continue until the stack gets blown off.

   So how does one parse left-associative binary operators with PEGs?
Honestly, this is one of their major shortcomings.  There’s no
general-purpose way of doing this, but here the repetition operators are
a good choice:

     (use-modules (srfi srfi-1))

     (define-peg-string-patterns
       "expr <- sum
     sum <-- (product ('+' / '-'))* product
     product <-- (value ('*' / '/'))* value
     value <-- number / '(' expr ')'
     number <-- [0-9]+")

     ;; take a deep breath...
     (define (make-left-parser next-func)
       (lambda (sum first . rest) ;; general form, comments below assume
         ;; that we're dealing with a sum expression
         (if (null? rest) ;; form (sum (product ...))
           (apply next-func first)
           (if (string? (cadr first));; form (sum ((product ...) "+") (product ...))
     	  (list (string->symbol (cadr first))
     		(apply next-func (car first))
     		(apply next-func (car rest)))
               ;; form (sum (((product ...) "+") ((product ...) "+")) (product ...))
     	  (car
     	   (reduce ;; walk through the list and build a left-associative tree
     	    (lambda (l r)
     	      (list (list (cadr r) (car r) (apply next-func (car l)))
     		    (string->symbol (cadr l))))
     	    'ignore
     	    (append ;; make a list of all the products
                  ;; the first one should be pre-parsed
     	     (list (list (apply next-func (caar first))
     			 (string->symbol (cadar first))))
     	     (cdr first)
                  ;; the last one has to be added in
     	     (list (append rest '("done"))))))))))

     (define (parse-value value first . rest)
       (if (null? rest)
           (string->number (cadr first))
           (apply parse-sum (car rest))))
     (define parse-product (make-left-parser parse-value))
     (define parse-sum (make-left-parser parse-product))
     (define parse-expr parse-sum)

   Then:
     (apply parse-expr (peg:tree (match-pattern expr "1+1/2*3+(1+1)/2"))) ⇒
     (+ (+ 1 (* (/ 1 2) 3)) (/ (+ 1 1) 2))

   As you can see, this is much uglier (it could be made prettier by
using ‘context-flatten’, but the way it’s written above makes it clear
how we deal with the three ways the zero-or-more ‘*’ expression can
parse).  Fortunately, most of the time we can get away with only using
right-associativity.

Simplified Functions
....................

For a more tantalizing example, consider the following grammar that
parses (highly) simplified C functions:

     (define-peg-string-patterns
       "cfunc <-- cSP ctype cSP cname cSP cargs cLB cSP cbody cRB
     ctype <-- cidentifier
     cname <-- cidentifier
     cargs <-- cLP (! (cSP cRP) carg cSP (cCOMMA / cRP) cSP)* cSP
     carg <-- cSP ctype cSP cname
     cbody <-- cstatement *
     cidentifier <- [a-zA-z][a-zA-Z0-9_]*
     cstatement <-- (!';'.)*cSC cSP
     cSC < ';'
     cCOMMA < ','
     cLP < '('
     cRP < ')'
     cLB < '{'
     cRB < '}'
     cSP < [ \t\n]*")

   Then:
     (match-pattern cfunc "int square(int a) { return a*a;}") ⇒
     (32
      (cfunc (ctype "int")
             (cname "square")
             (cargs (carg (ctype "int") (cname "a")))
             (cbody (cstatement "return a*a"))))

   And:
     (match-pattern cfunc "int mod(int a, int b) { int c = a/b;return a-b*c; }") ⇒
     (52
      (cfunc (ctype "int")
             (cname "mod")
             (cargs (carg (ctype "int") (cname "a"))
                    (carg (ctype "int") (cname "b")))
             (cbody (cstatement "int c = a/b")
                    (cstatement "return a- b*c"))))

   By wrapping all the ‘carg’ nonterminals in a ‘cargs’ nonterminal, we
were able to remove any ambiguity in the parsing structure and avoid
having to call ‘context-flatten’ on the output of ‘match-pattern’.  We
used the same trick with the ‘cstatement’ nonterminals, wrapping them in
a ‘cbody’ nonterminal.

   The whitespace nonterminal ‘cSP’ used here is a (very) useful
instantiation of a common pattern for matching syntactically irrelevant
information.  Since it’s tagged with ‘<’ and ends with ‘*’ it won’t
clutter up the parse trees (all the empty lists will be discarded during
the compression step) and it will never cause parsing to fail.


File: guile.info,  Node: PEG Internals,  Prev: PEG Tutorial,  Up: PEG Parsing

6.17.4 PEG Internals
--------------------

A PEG parser takes a string as input and attempts to parse it as a given
nonterminal.  The key idea of the PEG implementation is that every
nonterminal is just a function that takes a string as an argument and
attempts to parse that string as its nonterminal.  The functions always
start from the beginning, but a parse is considered successful if there
is material left over at the end.

   This makes it easy to model different PEG parsing operations.  For
instance, consider the PEG grammar ‘"ab"’, which could also be written
‘(and "a" "b")’.  It matches the string “ab”.  Here’s how that might be
implemented in the PEG style:

     (define (match-and-a-b str)
       (match-a str)
       (match-b str))

   As you can see, the use of functions provides an easy way to model
sequencing.  In a similar way, one could model ‘(or a b)’ with something
like the following:

     (define (match-or-a-b str)
       (or (match-a str) (match-b str)))

   Here the semantics of a PEG ‘or’ expression map naturally onto
Scheme’s ‘or’ operator.  This function will attempt to run ‘(match-a
str)’, and return its result if it succeeds.  Otherwise it will run
‘(match-b str)’.

   Of course, the code above wouldn’t quite work.  We need some way for
the parsing functions to communicate.  The actual interface used is
below.

Parsing Function Interface
..........................

A parsing function takes three arguments - a string, the length of that
string, and the position in that string it should start parsing at.  In
effect, the parsing functions pass around substrings in pieces - the
first argument is a buffer of characters, and the second two give a
range within that buffer that the parsing function should look at.

   Parsing functions return either #f, if they failed to match their
nonterminal, or a list whose first element must be an integer
representing the final position in the string they matched and whose cdr
can be any other data the function wishes to return, or ’() if it
doesn’t have any more data.

   The one caveat is that if the extra data it returns is a list, any
adjacent strings in that list will be appended by ‘match-pattern’.  For
instance, if a parsing function returns ‘(13 ("a" "b" "c"))’,
‘match-pattern’ will take ‘(13 ("abc"))’ as its value.

   For example, here is a function to match “ab” using the actual
interface.

     (define (match-a-b str len pos)
        (and (<= (+ pos 2) len)
             (string= str "ab" pos (+ pos 2))
             (list (+ pos 2) '()))) ; we return no extra information

   The above function can be used to match a string by running
‘(match-pattern match-a-b "ab")’.

Code Generators and Extensible Syntax
.....................................

PEG expressions, such as those in a ‘define-peg-pattern’ form, are
interpreted internally in two steps.

   First, any string PEG is expanded into an s-expression PEG by the
code in the ‘(ice-9 peg string-peg)’ module.

   Then, the s-expression PEG that results is compiled into a parsing
function by the ‘(ice-9 peg codegen)’ module.  In particular, the
function ‘compile-peg-pattern’ is called on the s-expression.  It then
decides what to do based on the form it is passed.

   The PEG syntax can be expanded by providing ‘compile-peg-pattern’
more options for what to do with its forms.  The extended syntax will be
associated with a symbol, for instance ‘my-parsing-form’, and will be
called on all PEG expressions of the form
     (my-parsing-form ...)

   The parsing function should take two arguments.  The first will be a
syntax object containing a list with all of the arguments to the form
(but not the form’s name), and the second will be the ‘capture-type’
argument that is passed to ‘define-peg-pattern’.

   New functions can be registered by calling ‘(add-peg-compiler! symbol
function)’, where ‘symbol’ is the symbol that will indicate a form of
this type and ‘function’ is the code generating function described
above.  The function ‘add-peg-compiler!’ is exported from the ‘(ice-9
peg codegen)’ module.


File: guile.info,  Node: Read/Load/Eval/Compile,  Next: Memory Management,  Prev: PEG Parsing,  Up: API Reference

6.18 Reading and Evaluating Scheme Code
=======================================

This chapter describes Guile functions that are concerned with reading,
loading, evaluating, and compiling Scheme code at run time.

* Menu:

* Scheme Syntax::               Standard and extended Scheme syntax.
* Scheme Read::                 Reading Scheme code.
* Scheme Write::                Writing Scheme values to a port.
* Fly Evaluation::              Procedures for on the fly evaluation.
* Compilation::                 How to compile Scheme files and procedures.
* Loading::                     Loading Scheme code from file.
* Load Paths::                  Where Guile looks for code.
* Character Encoding of Source Files:: Loading non-ASCII Scheme code from file.
* Delayed Evaluation::          Postponing evaluation until it is needed.
* Local Evaluation::            Evaluation in a local lexical environment.
* Local Inclusion::             Compile-time inclusion of one file in another.
* Sandboxed Evaluation::        Evaluation with limited capabilities.
* REPL Servers::                Serving a REPL over a socket.
* Cooperative REPL Servers::    REPL server for single-threaded applications.


File: guile.info,  Node: Scheme Syntax,  Next: Scheme Read,  Up: Read/Load/Eval/Compile

6.18.1 Scheme Syntax: Standard and Guile Extensions
---------------------------------------------------

* Menu:

* Expression Syntax::
* Comments::
* Block Comments::
* Case Sensitivity::
* Keyword Syntax::
* Reader Extensions::


File: guile.info,  Node: Expression Syntax,  Next: Comments,  Up: Scheme Syntax

6.18.1.1 Expression Syntax
..........................

An expression to be evaluated takes one of the following forms.

SYMBOL
     A symbol is evaluated by dereferencing.  A binding of that symbol
     is sought and the value there used.  For example,

          (define x 123)
          x ⇒ 123

(PROC ARGS...)
     A parenthesised expression is a function call.  PROC and each
     argument are evaluated, then the function (which PROC evaluated to)
     is called with those arguments.

     The order in which PROC and the arguments are evaluated is
     unspecified, so be careful when using expressions with side
     effects.

          (max 1 2 3) ⇒ 3

          (define (get-some-proc)  min)
          ((get-some-proc) 1 2 3) ⇒ 1

     The same sort of parenthesised form is used for a macro invocation,
     but in that case the arguments are not evaluated.  See the
     descriptions of macros for more on this (*note Macros::, and *note
     Syntax Rules::).

CONSTANT
     Number, string, character and boolean constants evaluate “to
     themselves”, so can appear as literals.

          123     ⇒ 123
          99.9    ⇒ 99.9
          "hello" ⇒ "hello"
          #\z     ⇒ #\z
          #t      ⇒ #t

     Note that an application must not attempt to modify literal
     strings, since they may be in read-only memory.

(quote DATA)
’DATA
     Quoting is used to obtain a literal symbol (instead of a variable
     reference), a literal list (instead of a function call), or a
     literal vector.  ’ is simply a shorthand for a ‘quote’ form.  For
     example,

          'x                   ⇒ x
          '(1 2 3)             ⇒ (1 2 3)
          '#(1 (2 3) 4)        ⇒ #(1 (2 3) 4)
          (quote x)            ⇒ x
          (quote (1 2 3))      ⇒ (1 2 3)
          (quote #(1 (2 3) 4)) ⇒ #(1 (2 3) 4)

     Note that an application must not attempt to modify literal lists
     or vectors obtained from a ‘quote’ form, since they may be in
     read-only memory.

(quasiquote DATA)
‘DATA
     Backquote quasi-quotation is like ‘quote’, but selected
     sub-expressions are evaluated.  This is a convenient way to
     construct a list or vector structure most of which is constant, but
     at certain points should have expressions substituted.

     The same effect can always be had with suitable ‘list’, ‘cons’ or
     ‘vector’ calls, but quasi-quoting is often easier.

     (unquote EXPR)
     ,EXPR
          Within the quasiquote DATA, ‘unquote’ or ‘,’ indicates an
          expression to be evaluated and inserted.  The comma syntax ‘,’
          is simply a shorthand for an ‘unquote’ form.  For example,

               `(1 2 (* 9 9) 3 4)       ⇒ (1 2 (* 9 9) 3 4)
               `(1 2 ,(* 9 9) 3 4)      ⇒ (1 2 81 3 4)
               `(1 (unquote (+ 1 1)) 3) ⇒ (1 2 3)
               `#(1 ,(/ 12 2))          ⇒ #(1 6)

     (unquote-splicing EXPR)
     ,@EXPR
          Within the quasiquote DATA, ‘unquote-splicing’ or ‘,@’
          indicates an expression to be evaluated and the elements of
          the returned list inserted.  EXPR must evaluate to a list.
          The “comma-at” syntax ‘,@’ is simply a shorthand for an
          ‘unquote-splicing’ form.

               (define x '(2 3))
               `(1 ,x 4)                           ⇒ (1 (2 3) 4)
               `(1 ,@x 4)                         ⇒ (1 2 3 4)
               `(1 (unquote-splicing (map 1+ x)))  ⇒ (1 3 4)
               `#(9 ,@x 9)                        ⇒ #(9 2 3 9)

          Notice ‘,@’ differs from plain ‘,’ in the way one level of
          nesting is stripped.  For ‘,@’ the elements of a returned list
          are inserted, whereas with ‘,’ it would be the list itself
          inserted.


File: guile.info,  Node: Comments,  Next: Block Comments,  Prev: Expression Syntax,  Up: Scheme Syntax

6.18.1.2 Comments
.................

Comments in Scheme source files are written by starting them with a
semicolon character (‘;’).  The comment then reaches up to the end of
the line.  Comments can begin at any column, and the may be inserted on
the same line as Scheme code.

     ; Comment
     ;; Comment too
     (define x 1)        ; Comment after expression
     (let ((y 1))
       ;; Display something.
       (display y)
     ;;; Comment at left margin.
       (display (+ y 1)))

   It is common to use a single semicolon for comments following
expressions on a line, to use two semicolons for comments which are
indented like code, and three semicolons for comments which start at
column 0, even if they are inside an indented code block.  This
convention is used when indenting code in Emacs’ Scheme mode.


File: guile.info,  Node: Block Comments,  Next: Case Sensitivity,  Prev: Comments,  Up: Scheme Syntax

6.18.1.3 Block Comments
.......................

In addition to the standard line comments defined by R5RS, Guile has
another comment type for multiline comments, called “block comments”.
This type of comment begins with the character sequence ‘#!’ and ends
with the characters ‘!#’.

   These comments are compatible with the block comments in the Scheme
Shell ‘scsh’ (*note The Scheme shell (scsh)::).  The characters ‘#!’
were chosen because they are the magic characters used in shell scripts
for indicating that the name of the program for executing the script
follows on the same line.

   Thus a Guile script often starts like this.

     #! /usr/local/bin/guile -s
     !#

   More details on Guile scripting can be found in the scripting section
(*note Guile Scripting::).

   Similarly, Guile (starting from version 2.0) supports nested block
comments as specified by R6RS and SRFI-30
(http://srfi.schemers.org/srfi-30/srfi-30.html):

     (+ 1 #| this is a #| nested |# block comment |# 2)
     ⇒ 3

   For backward compatibility, this syntax can be overridden with
‘read-hash-extend’ (*note ‘read-hash-extend’: Reader Extensions.).

   There is one special case where the contents of a comment can
actually affect the interpretation of code.  When a character encoding
declaration, such as ‘coding: utf-8’ appears in one of the first few
lines of a source file, it indicates to Guile’s default reader that this
source code file is not ASCII. For details see *note Character Encoding
of Source Files::.


File: guile.info,  Node: Case Sensitivity,  Next: Keyword Syntax,  Prev: Block Comments,  Up: Scheme Syntax

6.18.1.4 Case Sensitivity
.........................

Scheme as defined in R5RS is not case sensitive when reading symbols.
Guile, on the contrary is case sensitive by default, so the identifiers

     guile-whuzzy
     Guile-Whuzzy

   are the same in R5RS Scheme, but are different in Guile.

   It is possible to turn off case sensitivity in Guile by setting the
reader option ‘case-insensitive’.  For more information on reader
options, *Note Scheme Read::.

     (read-enable 'case-insensitive)

   It is also possible to disable (or enable) case sensitivity within a
single file by placing the reader directives ‘#!fold-case’ (or
‘#!no-fold-case’) within the file itself.


File: guile.info,  Node: Keyword Syntax,  Next: Reader Extensions,  Prev: Case Sensitivity,  Up: Scheme Syntax

6.18.1.5 Keyword Syntax
.......................


File: guile.info,  Node: Reader Extensions,  Prev: Keyword Syntax,  Up: Scheme Syntax

6.18.1.6 Reader Extensions
..........................

 -- Scheme Procedure: read-hash-extend chr proc
 -- C Function: scm_read_hash_extend (chr, proc)
     Install the procedure PROC for reading expressions starting with
     the character sequence ‘#’ and CHR.  PROC will be called with two
     arguments: the character CHR and the port to read further data
     from.  The object returned will be the return value of ‘read’.
     Passing ‘#f’ for PROC will remove a previous setting.


File: guile.info,  Node: Scheme Read,  Next: Scheme Write,  Prev: Scheme Syntax,  Up: Read/Load/Eval/Compile

6.18.2 Reading Scheme Code
--------------------------

 -- Scheme Procedure: read [port]
 -- C Function: scm_read (port)
     Read an s-expression from the input port PORT, or from the current
     input port if PORT is not specified.  Any whitespace before the
     next token is discarded.

   The behaviour of Guile’s Scheme reader can be modified by
manipulating its read options.

 -- Scheme Procedure: read-options [setting]
     Display the current settings of the global read options.  If
     SETTING is omitted, only a short form of the current read options
     is printed.  Otherwise if SETTING is the symbol ‘help’, a complete
     options description is displayed.

   The set of available options, and their default values, may be had by
invoking ‘read-options’ at the prompt.

     scheme@(guile-user)> (read-options)
     (square-brackets keywords #f positions)
     scheme@(guile-user)> (read-options 'help)
     copy              no    Copy source code expressions.
     positions         yes   Record positions of source code expressions.
     case-insensitive  no    Convert symbols to lower case.
     keywords          #f    Style of keyword recognition: #f, 'prefix or 'postfix.
     r6rs-hex-escapes  no    Use R6RS variable-length character and string hex escapes.
     square-brackets   yes   Treat `[' and `]' as parentheses, for R6RS compatibility.
     hungry-eol-escapes no   In strings, consume leading whitespace after an
                             escaped end-of-line.
     curly-infix       no    Support SRFI-105 curly infix expressions.
     r7rs-symbols      no    Support R7RS |...| symbol notation.

   Note that Guile also includes a preliminary mechanism for setting
read options on a per-port basis.  For instance, the ‘case-insensitive’
read option is set (or unset) on the port when the reader encounters the
‘#!fold-case’ or ‘#!no-fold-case’ reader directives.  Similarly, the
‘#!curly-infix’ reader directive sets the ‘curly-infix’ read option on
the port, and ‘#!curly-infix-and-bracket-lists’ sets ‘curly-infix’ and
unsets ‘square-brackets’ on the port (*note SRFI-105::).  There is
currently no other way to access or set the per-port read options.

   The boolean options may be toggled with ‘read-enable’ and
‘read-disable’.  The non-boolean ‘keywords’ option must be set using
‘read-set!’.

 -- Scheme Procedure: read-enable option-name
 -- Scheme Procedure: read-disable option-name
 -- Scheme Syntax: read-set! option-name value
     Modify the read options.  ‘read-enable’ should be used with boolean
     options and switches them on, ‘read-disable’ switches them off.

     ‘read-set!’ can be used to set an option to a specific value.  Due
     to historical oddities, it is a macro that expects an unquoted
     option name.

   For example, to make ‘read’ fold all symbols to their lower case
(perhaps for compatibility with older Scheme code), you can enter:

     (read-enable 'case-insensitive)

   For more information on the effect of the ‘r6rs-hex-escapes’ and
‘hungry-eol-escapes’ options, see (*note String Syntax::).

   For more information on the ‘r7rs-symbols’ option, see (*note Symbol
Read Syntax::).


File: guile.info,  Node: Scheme Write,  Next: Fly Evaluation,  Prev: Scheme Read,  Up: Read/Load/Eval/Compile

6.18.3 Writing Scheme Values
----------------------------

Any scheme value may be written to a port.  Not all values may be read
back in (*note Scheme Read::), however.

 -- Scheme Procedure: write obj [port]
     Send a representation of OBJ to PORT or to the current output port
     if not given.

     The output is designed to be machine readable, and can be read back
     with ‘read’ (*note Scheme Read::).  Strings are printed in double
     quotes, with escapes if necessary, and characters are printed in
     ‘#\’ notation.

 -- Scheme Procedure: display obj [port]
     Send a representation of OBJ to PORT or to the current output port
     if not given.

     The output is designed for human readability, it differs from
     ‘write’ in that strings are printed without double quotes and
     escapes, and characters are printed as per ‘write-char’, not in
     ‘#\’ form.

   As was the case with the Scheme reader, there are a few options that
affect the behavior of the Scheme printer.

 -- Scheme Procedure: print-options [setting]
     Display the current settings of the read options.  If SETTING is
     omitted, only a short form of the current read options is printed.
     Otherwise if SETTING is the symbol ‘help’, a complete options
     description is displayed.

   The set of available options, and their default values, may be had by
invoking ‘print-options’ at the prompt.

     scheme@(guile-user)> (print-options)
     (quote-keywordish-symbols reader highlight-suffix "}" highlight-prefix "{")
     scheme@(guile-user)> (print-options 'help)
     highlight-prefix          {       The string to print before highlighted values.
     highlight-suffix          }       The string to print after highlighted values.
     quote-keywordish-symbols  reader  How to print symbols that have a colon
                                       as their first or last character. The
                                       value '#f' does not quote the colons;
                                       '#t' quotes them; 'reader' quotes them
                                       when the reader option 'keywords' is
                                       not '#f'.
     escape-newlines           yes     Render newlines as \n when printing
                                       using `write'.
     r7rs-symbols              no      Escape symbols using R7RS |...| symbol
                                       notation.

   These options may be modified with the print-set!  syntax.

 -- Scheme Syntax: print-set! option-name value
     Modify the print options.  Due to historical oddities, ‘print-set!’
     is a macro that expects an unquoted option name.


File: guile.info,  Node: Fly Evaluation,  Next: Compilation,  Prev: Scheme Write,  Up: Read/Load/Eval/Compile

6.18.4 Procedures for On the Fly Evaluation
-------------------------------------------

Scheme has the lovely property that its expressions may be represented
as data.  The ‘eval’ procedure takes a Scheme datum and evaluates it as
code.

 -- Scheme Procedure: eval exp module_or_state
 -- C Function: scm_eval (exp, module_or_state)
     Evaluate EXP, a list representing a Scheme expression, in the
     top-level environment specified by MODULE_OR_STATE.  While EXP is
     evaluated (using ‘primitive-eval’), MODULE_OR_STATE is made the
     current module.  The current module is reset to its previous value
     when ‘eval’ returns.  XXX - dynamic states.  Example: (eval ’(+ 1
     2) (interaction-environment))

 -- Scheme Procedure: interaction-environment
 -- C Function: scm_interaction_environment ()
     Return a specifier for the environment that contains
     implementation–defined bindings, typically a superset of those
     listed in the report.  The intent is that this procedure will
     return the environment in which the implementation would evaluate
     expressions dynamically typed by the user.

   *Note Environments::, for other environments.

   One does not always receive code as Scheme data, of course, and this
is especially the case for Guile’s other language implementations (*note
Other Languages::).  For the case in which all you have is a string, we
have ‘eval-string’.  There is a legacy version of this procedure in the
default environment, but you really want the one from ‘(ice-9
eval-string)’, so load it up:

     (use-modules (ice-9 eval-string))

 -- Scheme Procedure: eval-string string [#:module=#f] [#:file=#f]
          [#:line=#f] [#:column=#f] [#:lang=(current-language)]
          [#:compile?=#f]
     Parse STRING according to the current language, normally Scheme.
     Evaluate or compile the expressions it contains, in order,
     returning the last expression.

     If the MODULE keyword argument is set, save a module excursion
     (*note Module System Reflection::) and set the current module to
     MODULE before evaluation.

     The FILE, LINE, and COLUMN keyword arguments can be used to
     indicate that the source string begins at a particular source
     location.

     Finally, LANG is a language, defaulting to the current language,
     and the expression is compiled if COMPILE? is true or there is no
     evaluator for the given language.

 -- C Function: scm_eval_string (string)
 -- C Function: scm_eval_string_in_module (string, module)
     These C bindings call ‘eval-string’ from ‘(ice-9 eval-string)’,
     evaluating within MODULE or the current module.

 -- C Function: SCM scm_c_eval_string (const char *string)
     ‘scm_eval_string’, but taking a C string in locale encoding instead
     of an ‘SCM’.

 -- Scheme Procedure: apply proc arg ... arglst
 -- C Function: scm_apply_0 (proc, arglst)
 -- C Function: scm_apply_1 (proc, arg1, arglst)
 -- C Function: scm_apply_2 (proc, arg1, arg2, arglst)
 -- C Function: scm_apply_3 (proc, arg1, arg2, arg3, arglst)
 -- C Function: scm_apply (proc, arg, rest)
     Call PROC with arguments ARG ... and the elements of the ARGLST
     list.

     ‘scm_apply’ takes parameters corresponding to a Scheme level
     ‘(lambda (proc arg1 . rest) ...)’.  So ARG1 and all but the last
     element of the REST list make up ARG ..., and the last element of
     REST is the ARGLST list.  Or if REST is the empty list ‘SCM_EOL’
     then there’s no ARG ..., and (ARG1) is the ARGLST.

     ARGLST is not modified, but the REST list passed to ‘scm_apply’ is
     modified.

 -- C Function: scm_call_0 (proc)
 -- C Function: scm_call_1 (proc, arg1)
 -- C Function: scm_call_2 (proc, arg1, arg2)
 -- C Function: scm_call_3 (proc, arg1, arg2, arg3)
 -- C Function: scm_call_4 (proc, arg1, arg2, arg3, arg4)
 -- C Function: scm_call_5 (proc, arg1, arg2, arg3, arg4, arg5)
 -- C Function: scm_call_6 (proc, arg1, arg2, arg3, arg4, arg5, arg6)
 -- C Function: scm_call_7 (proc, arg1, arg2, arg3, arg4, arg5, arg6,
          arg7)
 -- C Function: scm_call_8 (proc, arg1, arg2, arg3, arg4, arg5, arg6,
          arg7, arg8)
 -- C Function: scm_call_9 (proc, arg1, arg2, arg3, arg4, arg5, arg6,
          arg7, arg8, arg9)
     Call PROC with the given arguments.

 -- C Function: scm_call (proc, ...)
     Call PROC with any number of arguments.  The argument list must be
     terminated by ‘SCM_UNDEFINED’.  For example:

          scm_call (scm_c_public_ref ("guile", "+"),
                    scm_from_int (1),
                    scm_from_int (2),
                    SCM_UNDEFINED);

 -- C Function: scm_call_n (proc, argv, nargs)
     Call PROC with the array of arguments ARGV, as a ‘SCM*’.  The
     length of the arguments should be passed in NARGS, as a ‘size_t’.

 -- Scheme Procedure: primitive-eval exp
 -- C Function: scm_primitive_eval (exp)
     Evaluate EXP in the top-level environment specified by the current
     module.


File: guile.info,  Node: Compilation,  Next: Loading,  Prev: Fly Evaluation,  Up: Read/Load/Eval/Compile

6.18.5 Compiling Scheme Code
----------------------------

The ‘eval’ procedure directly interprets the S-expression representation
of Scheme.  An alternate strategy for evaluation is to determine ahead
of time what computations will be necessary to evaluate the expression,
and then use that recipe to produce the desired results.  This is known
as “compilation”.

   While it is possible to compile simple Scheme expressions such as ‘(+
2 2)’ or even ‘"Hello world!"’, compilation is most interesting in the
context of procedures.  Compiling a lambda expression produces a
compiled procedure, which is just like a normal procedure except
typically much faster, because it can bypass the generic interpreter.

   Functions from system modules in a Guile installation are normally
compiled already, so they load and run quickly.

   Note that well-written Scheme programs will not typically call the
procedures in this section, for the same reason that it is often bad
taste to use ‘eval’.  By default, Guile automatically compiles any files
it encounters that have not been compiled yet (*note ‘--auto-compile’:
Invoking Guile.).  The compiler can also be invoked explicitly from the
shell as ‘guild compile foo.scm’.

   (Why are calls to ‘eval’ and ‘compile’ usually in bad taste?  Because
they are limited, in that they can only really make sense for top-level
expressions.  Also, most needs for “compile-time” computation are
fulfilled by macros and closures.  Of course one good counterexample is
the REPL itself, or any code that reads expressions from a port.)

   Automatic compilation generally works transparently, without any need
for user intervention.  However Guile does not yet do proper dependency
tracking, so that if file ‘A.scm’ uses macros from ‘B.scm’, and B.SCM
changes, ‘A.scm’ would not be automatically recompiled.  To forcibly
invalidate the auto-compilation cache, pass the ‘--fresh-auto-compile’
option to Guile, or set the ‘GUILE_AUTO_COMPILE’ environment variable to
‘fresh’ (instead of to ‘0’ or ‘1’).

   For more information on the compiler itself, see *note Compiling to
the Virtual Machine::.  For information on the virtual machine, see
*note A Virtual Machine for Guile::.

   The command-line interface to Guile’s compiler is the ‘guild compile’
command:

 -- Command: guild compile [‘option’...] FILE...
     Compile FILE, a source file, and store bytecode in the compilation
     cache or in the file specified by the ‘-o’ option.  The following
     options are available:

     ‘-L DIR’
     ‘--load-path=DIR’
          Add DIR to the front of the module load path.

     ‘-o OFILE’
     ‘--output=OFILE’
          Write output bytecode to OFILE.  By convention, bytecode file
          names end in ‘.go’.  When ‘-o’ is omitted, the output file
          name is as for ‘compile-file’ (see below).

     ‘-x EXTENSION’
          Recognize EXTENSION as a valid source file name extension.

          For example, to compile R6RS code, you might want to pass ‘-x
          .sls’ so that files ending in ‘.sls’ can be found.

     ‘-W WARNING’
     ‘--warn=WARNING’
          Emit warnings of type WARNING; use ‘--warn=help’ for a list of
          available warnings and their description.  Currently
          recognized warnings include ‘unused-variable’,
          ‘unused-toplevel’, ‘shadowed-toplevel’, ‘unbound-variable’,
          ‘macro-use-before-definition’, ‘arity-mismatch’, ‘format’,
          ‘duplicate-case-datum’, and ‘bad-case-datum’.

     ‘-f LANG’
     ‘--from=LANG’
          Use LANG as the source language of FILE.  If this option is
          omitted, ‘scheme’ is assumed.

     ‘-t LANG’
     ‘--to=LANG’
          Use LANG as the target language of FILE.  If this option is
          omitted, ‘rtl’ is assumed.

     ‘-T TARGET’
     ‘--target=TARGET’
          Produce code for TARGET instead of %HOST-TYPE (*note
          %host-type: Build Config.).  Target must be a valid GNU
          triplet, such as ‘armv5tel-unknown-linux-gnueabi’ (*note
          (autoconf)Specifying Target Triplets::).

     Each FILE is assumed to be UTF-8-encoded, unless it contains a
     coding declaration as recognized by ‘file-encoding’ (*note
     Character Encoding of Source Files::).

   The compiler can also be invoked directly by Scheme code using the
procedures below:

 -- Scheme Procedure: compile exp [#:env=#f] [#:from=(current-language)]
          [#:to=value] [#:opts=()]
     Compile the expression EXP in the environment ENV.  If EXP is a
     procedure, the result will be a compiled procedure; otherwise
     ‘compile’ is mostly equivalent to ‘eval’.

     For a discussion of languages and compiler options, *Note Compiling
     to the Virtual Machine::.

 -- Scheme Procedure: compile-file file [#:output-file=#f]
          [#:from=(current-language)] [#:to='rtl]
          [#:env=(default-environment from)] [#:opts='()]
          [#:canonicalization='relative]
     Compile the file named FILE.

     Output will be written to a OUTPUT-FILE.  If you do not supply an
     output file name, output is written to a file in the cache
     directory, as computed by ‘(compiled-file-name FILE)’.

     FROM and TO specify the source and target languages.  *Note
     Compiling to the Virtual Machine::, for more information on these
     options, and on ENV and OPTS.

     As with ‘guild compile’, FILE is assumed to be UTF-8-encoded unless
     it contains a coding declaration.

 -- Scheme Procedure: compiled-file-name file
     Compute a cached location for a compiled version of a Scheme file
     named FILE.

     This file will usually be below the ‘$HOME/.cache/guile/ccache’
     directory, depending on the value of the ‘XDG_CACHE_HOME’
     environment variable.  The intention is that ‘compiled-file-name’
     provides a fallback location for caching auto-compiled files.  If
     you want to place a compile file in the ‘%load-compiled-path’, you
     should pass the OUTPUT-FILE option to ‘compile-file’, explicitly.

 -- Scheme Variable: %auto-compilation-options
     This variable contains the options passed to the ‘compile-file’
     procedure when auto-compiling source files.  By default, it enables
     useful compilation warnings.  It can be customized from ‘~/.guile’.


File: guile.info,  Node: Loading,  Next: Load Paths,  Prev: Compilation,  Up: Read/Load/Eval/Compile

6.18.6 Loading Scheme Code from File
------------------------------------

 -- Scheme Procedure: load filename [reader]
     Load FILENAME and evaluate its contents in the top-level
     environment.

     READER if provided should be either ‘#f’, or a procedure with the
     signature ‘(lambda (port) ...)’ which reads the next expression
     from PORT.  If READER is ‘#f’ or absent, Guile’s built-in ‘read’
     procedure is used (*note Scheme Read::).

     The READER argument takes effect by setting the value of the
     ‘current-reader’ fluid (see below) before loading the file, and
     restoring its previous value when loading is complete.  The Scheme
     code inside FILENAME can itself change the current reader procedure
     on the fly by setting ‘current-reader’ fluid.

     If the variable ‘%load-hook’ is defined, it should be bound to a
     procedure that will be called before any code is loaded.  See
     documentation for ‘%load-hook’ later in this section.

 -- Scheme Procedure: load-compiled filename
     Load the compiled file named FILENAME.

     Compiling a source file (*note Read/Load/Eval/Compile::) and then
     calling ‘load-compiled’ on the resulting file is equivalent to
     calling ‘load’ on the source file.

 -- Scheme Procedure: primitive-load filename
 -- C Function: scm_primitive_load (filename)
     Load the file named FILENAME and evaluate its contents in the
     top-level environment.  FILENAME must either be a full pathname or
     be a pathname relative to the current directory.  If the variable
     ‘%load-hook’ is defined, it should be bound to a procedure that
     will be called before any code is loaded.  See the documentation
     for ‘%load-hook’ later in this section.

 -- C Function: SCM scm_c_primitive_load (const char *filename)
     ‘scm_primitive_load’, but taking a C string instead of an ‘SCM’.

 -- Variable: current-reader
     ‘current-reader’ holds the read procedure that is currently being
     used by the above loading procedures to read expressions (from the
     file that they are loading).  ‘current-reader’ is a fluid, so it
     has an independent value in each dynamic root and should be read
     and set using ‘fluid-ref’ and ‘fluid-set!’ (*note Fluids and
     Dynamic States::).

     Changing ‘current-reader’ is typically useful to introduce local
     syntactic changes, such that code following the ‘fluid-set!’ call
     is read using the newly installed reader.  The ‘current-reader’
     change should take place at evaluation time when the code is
     evaluated, or at compilation time when the code is compiled:

          (eval-when (compile eval)
            (fluid-set! current-reader my-own-reader))

     The ‘eval-when’ form above ensures that the ‘current-reader’ change
     occurs at the right time.

 -- Variable: %load-hook
     A procedure to be called ‘(%load-hook FILENAME)’ whenever a file is
     loaded, or ‘#f’ for no such call.  ‘%load-hook’ is used by all of
     the loading functions (‘load’ and ‘primitive-load’, and
     ‘load-from-path’ and ‘primitive-load-path’ documented in the next
     section).

     For example an application can set this to show what’s loaded,

          (set! %load-hook (lambda (filename)
                             (format #t "Loading ~a ...\n" filename)))
          (load-from-path "foo.scm")
          ⊣ Loading /usr/local/share/guile/site/foo.scm ...

 -- Scheme Procedure: current-load-port
 -- C Function: scm_current_load_port ()
     Return the current-load-port.  The load port is used internally by
     ‘primitive-load’.


File: guile.info,  Node: Load Paths,  Next: Character Encoding of Source Files,  Prev: Loading,  Up: Read/Load/Eval/Compile

6.18.7 Load Paths
-----------------

The procedure in the previous section look for Scheme code in the file
system at specific location.  Guile also has some procedures to search
the load path for code.

 -- Variable: %load-path
     List of directories which should be searched for Scheme modules and
     libraries.  When Guile starts up, ‘%load-path’ is initialized to
     the default load path ‘(list (%library-dir) (%site-dir)
     (%global-site-dir) (%package-data-dir))’.  The ‘GUILE_LOAD_PATH’
     environment variable can be used to prepend or append additional
     directories (*note Environment Variables::).

     *Note Build Config::, for more on ‘%site-dir’ and related
     procedures.

 -- Scheme Procedure: load-from-path filename
     Similar to ‘load’, but searches for FILENAME in the load paths.
     Preferentially loads a compiled version of the file, if it is
     available and up-to-date.

   A user can extend the load path by calling ‘add-to-load-path’.

 -- Scheme Syntax: add-to-load-path dir
     Add DIR to the load path.

   For example, a script might include this form to add the directory
that it is in to the load path:

     (add-to-load-path (dirname (current-filename)))

   It’s better to use ‘add-to-load-path’ than to modify ‘%load-path’
directly, because ‘add-to-load-path’ takes care of modifying the path
both at compile-time and at run-time.

 -- Scheme Procedure: primitive-load-path filename
          [exception-on-not-found]
 -- C Function: scm_primitive_load_path (filename)
     Search ‘%load-path’ for the file named FILENAME and load it into
     the top-level environment.  If FILENAME is a relative pathname and
     is not found in the list of search paths, an error is signalled.
     Preferentially loads a compiled version of the file, if it is
     available and up-to-date.

     If FILENAME is a relative pathname and is not found in the list of
     search paths, one of three things may happen, depending on the
     optional second argument, EXCEPTION-ON-NOT-FOUND.  If it is ‘#f’,
     ‘#f’ will be returned.  If it is a procedure, it will be called
     with no arguments.  (This allows a distinction to be made between
     exceptions raised by loading a file, and exceptions related to the
     loader itself.)  Otherwise an error is signalled.

     For compatibility with Guile 1.8 and earlier, the C function takes
     only one argument, which can be either a string (the file name) or
     an argument list.

 -- Scheme Procedure: %search-load-path filename
 -- C Function: scm_sys_search_load_path (filename)
     Search ‘%load-path’ for the file named FILENAME, which must be
     readable by the current user.  If FILENAME is found in the list of
     paths to search or is an absolute pathname, return its full
     pathname.  Otherwise, return ‘#f’.  Filenames may have any of the
     optional extensions in the ‘%load-extensions’ list;
     ‘%search-load-path’ will try each extension automatically.

 -- Variable: %load-extensions
     A list of default file extensions for files containing Scheme code.
     ‘%search-load-path’ tries each of these extensions when looking for
     a file to load.  By default, ‘%load-extensions’ is bound to the
     list ‘("" ".scm")’.

   As mentioned above, when Guile searches the ‘%load-path’ for a source
file, it will also search the ‘%load-compiled-path’ for a corresponding
compiled file.  If the compiled file is as new or newer than the source
file, it will be loaded instead of the source file, using
‘load-compiled’.

 -- Variable: %load-compiled-path
     Like ‘%load-path’, but for compiled files.  By default, this path
     has two entries: one for compiled files from Guile itself, and one
     for site packages.  The ‘GUILE_LOAD_COMPILED_PATH’ environment
     variable can be used to prepend or append additional directories
     (*note Environment Variables::).

   When ‘primitive-load-path’ searches the ‘%load-compiled-path’ for a
corresponding compiled file for a relative path it does so by appending
‘.go’ to the relative path.  For example, searching for ‘ice-9/popen’
could find ‘/usr/lib/guile/2.2/ccache/ice-9/popen.go’, and use it
instead of ‘/usr/share/guile/2.2/ice-9/popen.scm’.

   If ‘primitive-load-path’ does not find a corresponding ‘.go’ file in
the ‘%load-compiled-path’, or the ‘.go’ file is out of date, it will
search for a corresponding auto-compiled file in the fallback path,
possibly creating one if one does not exist.

   *Note Installing Site Packages::, for more on how to correctly
install site packages.  *Note Modules and the File System::, for more on
the relationship between load paths and modules.  *Note Compilation::,
for more on the fallback path and auto-compilation.

   Finally, there are a couple of helper procedures for general path
manipulation.

 -- Scheme Procedure: parse-path path [tail]
 -- C Function: scm_parse_path (path, tail)
     Parse PATH, which is expected to be a colon-separated string, into
     a list and return the resulting list with TAIL appended.  If PATH
     is ‘#f’, TAIL is returned.

 -- Scheme Procedure: parse-path-with-ellipsis path base
 -- C Function: scm_parse_path_with_ellipsis (path, base)
     Parse PATH, which is expected to be a colon-separated string, into
     a list and return the resulting list with BASE (a list) spliced in
     place of the ‘...’ path component, if present, or else BASE is
     added to the end.  If PATH is ‘#f’, BASE is returned.

 -- Scheme Procedure: search-path path filename [extensions
          [require-exts?]]
 -- C Function: scm_search_path (path, filename, rest)
     Search PATH for a directory containing a file named FILENAME.  The
     file must be readable, and not a directory.  If we find one, return
     its full filename; otherwise, return ‘#f’.  If FILENAME is
     absolute, return it unchanged.  If given, EXTENSIONS is a list of
     strings; for each directory in PATH, we search for FILENAME
     concatenated with each EXTENSION.  If REQUIRE-EXTS? is true,
     require that the returned file name have one of the given
     extensions; if REQUIRE-EXTS? is not given, it defaults to ‘#f’.

     For compatibility with Guile 1.8 and earlier, the C function takes
     only three arguments.


File: guile.info,  Node: Character Encoding of Source Files,  Next: Delayed Evaluation,  Prev: Load Paths,  Up: Read/Load/Eval/Compile

6.18.8 Character Encoding of Source Files
-----------------------------------------

Scheme source code files are usually encoded in ASCII or UTF-8, but the
built-in reader can interpret other character encodings as well.  When
Guile loads Scheme source code, it uses the ‘file-encoding’ procedure
(described below) to try to guess the encoding of the file.  In the
absence of any hints, UTF-8 is assumed.  One way to provide a hint about
the encoding of a source file is to place a coding declaration in the
top 500 characters of the file.

   A coding declaration has the form ‘coding: XXXXXX’, where ‘XXXXXX’ is
the name of a character encoding in which the source code file has been
encoded.  The coding declaration must appear in a scheme comment.  It
can either be a semicolon-initiated comment, or the first block ‘#!’
comment in the file.

   The name of the character encoding in the coding declaration is
typically lower case and containing only letters, numbers, and hyphens,
as recognized by ‘set-port-encoding!’ (*note ‘set-port-encoding!’:
Ports.).  Common examples of character encoding names are ‘utf-8’ and
‘iso-8859-1’, as defined by IANA
(http://www.iana.org/assignments/character-sets).  Thus, the coding
declaration is mostly compatible with Emacs.

   However, there are some differences in encoding names recognized by
Emacs and encoding names defined by IANA, the latter being essentially a
subset of the former.  For instance, ‘latin-1’ is a valid encoding name
for Emacs, but it’s not according to the IANA standard, which Guile
follows; instead, you should use ‘iso-8859-1’, which is both understood
by Emacs and dubbed by IANA (IANA writes it uppercase but Emacs wants it
lowercase and Guile is case insensitive.)

   For source code, only a subset of all possible character encodings
can be interpreted by the built-in source code reader.  Only those
character encodings in which ASCII text appears unmodified can be used.
This includes ‘UTF-8’ and ‘ISO-8859-1’ through ‘ISO-8859-15’.  The
multi-byte character encodings ‘UTF-16’ and ‘UTF-32’ may not be used
because they are not compatible with ASCII.

   There might be a scenario in which one would want to read non-ASCII
code from a port, such as with the function ‘read’, instead of with
‘load’.  If the port’s character encoding is the same as the encoding of
the code to be read by the port, not other special handling is
necessary.  The port will automatically do the character encoding
conversion.  The functions ‘setlocale’ or by ‘set-port-encoding!’ are
used to set port encodings (*note Ports::).

   If a port is used to read code of unknown character encoding, it can
accomplish this in three steps.  First, the character encoding of the
port should be set to ISO-8859-1 using ‘set-port-encoding!’.  Then, the
procedure ‘file-encoding’, described below, is used to scan for a coding
declaration when reading from the port.  As a side effect, it rewinds
the port after its scan is complete.  After that, the port’s character
encoding should be set to the encoding returned by ‘file-encoding’, if
any, again by using ‘set-port-encoding!’.  Then the code can be read as
normal.

   Alternatively, one can use the ‘#:guess-encoding’ keyword argument of
‘open-file’ and related procedures.  *Note File Ports::.

 -- Scheme Procedure: file-encoding port
 -- C Function: scm_file_encoding (port)
     Attempt to scan the first few hundred bytes from the PORT for hints
     about its character encoding.  Return a string containing the
     encoding name or ‘#f’ if the encoding cannot be determined.  The
     port is rewound.

     Currently, the only supported method is to look for an Emacs-like
     character coding declaration (*note how Emacs recognizes file
     encoding: (emacs)Recognize Coding.).  The coding declaration is of
     the form ‘coding: XXXXX’ and must appear in a Scheme comment.
     Additional heuristics may be added in the future.


File: guile.info,  Node: Delayed Evaluation,  Next: Local Evaluation,  Prev: Character Encoding of Source Files,  Up: Read/Load/Eval/Compile

6.18.9 Delayed Evaluation
-------------------------

Promises are a convenient way to defer a calculation until its result is
actually needed, and to run such a calculation only once.  Also *note
SRFI-45::.

 -- syntax: delay expr
     Return a promise object which holds the given EXPR expression,
     ready to be evaluated by a later ‘force’.

 -- Scheme Procedure: promise? obj
 -- C Function: scm_promise_p (obj)
     Return true if OBJ is a promise.

 -- Scheme Procedure: force p
 -- C Function: scm_force (p)
     Return the value obtained from evaluating the EXPR in the given
     promise P.  If P has previously been forced then its EXPR is not
     evaluated again, instead the value obtained at that time is simply
     returned.

     During a ‘force’, an EXPR can call ‘force’ again on its own
     promise, resulting in a recursive evaluation of that EXPR.  The
     first evaluation to return gives the value for the promise.  Higher
     evaluations run to completion in the normal way, but their results
     are ignored, ‘force’ always returns the first value.


File: guile.info,  Node: Local Evaluation,  Next: Local Inclusion,  Prev: Delayed Evaluation,  Up: Read/Load/Eval/Compile

6.18.10 Local Evaluation
------------------------

Guile includes a facility to capture a lexical environment, and later
evaluate a new expression within that environment.  This code is
implemented in a module.

     (use-modules (ice-9 local-eval))

 -- syntax: the-environment
     Captures and returns a lexical environment for use with
     ‘local-eval’ or ‘local-compile’.

 -- Scheme Procedure: local-eval exp env
 -- C Function: scm_local_eval (exp, env)
 -- Scheme Procedure: local-compile exp env [opts=()]
     Evaluate or compile the expression EXP in the lexical environment
     ENV.

   Here is a simple example, illustrating that it is the variable that
gets captured, not just its value at one point in time.

     (define e (let ((x 100)) (the-environment)))
     (define fetch-x (local-eval '(lambda () x) e))
     (fetch-x)
     ⇒ 100
     (local-eval '(set! x 42) e)
     (fetch-x)
     ⇒ 42

   While EXP is evaluated within the lexical environment of
‘(the-environment)’, it has the dynamic environment of the call to
‘local-eval’.

   ‘local-eval’ and ‘local-compile’ can only evaluate expressions, not
definitions.

     (local-eval '(define foo 42)
                 (let ((x 100)) (the-environment)))
     ⇒ syntax error: definition in expression context

   Note that the current implementation of ‘(the-environment)’ only
captures “normal” lexical bindings, and pattern variables bound by
‘syntax-case’.  It does not currently capture local syntax transformers
bound by ‘let-syntax’, ‘letrec-syntax’ or non-top-level ‘define-syntax’
forms.  Any attempt to reference such captured syntactic keywords via
‘local-eval’ or ‘local-compile’ produces an error.


File: guile.info,  Node: Local Inclusion,  Next: Sandboxed Evaluation,  Prev: Local Evaluation,  Up: Read/Load/Eval/Compile

6.18.11 Local Inclusion
-----------------------

This section has discussed various means of linking Scheme code
together: fundamentally, loading up files at run-time using ‘load’ and
‘load-compiled’.  Guile provides another option to compose parts of
programs together at expansion-time instead of at run-time.

 -- Scheme Syntax: include file-name
     Open FILE-NAME, at expansion-time, and read the Scheme forms that
     it contains, splicing them into the location of the ‘include’,
     within a ‘begin’.

     If FILE-NAME is a relative path, it is searched for relative to the
     path that contains the file that the ‘include’ form appears in.

   If you are a C programmer, if ‘load’ in Scheme is like ‘dlopen’ in C,
consider ‘include’ to be like the C preprocessor’s ‘#include’.  When you
use ‘include’, it is as if the contents of the included file were typed
in instead of the ‘include’ form.

   Because the code is included at compile-time, it is available to the
macroexpander.  Syntax definitions in the included file are available to
later code in the form in which the ‘include’ appears, without the need
for ‘eval-when’.  (*Note Eval When::.)

   For the same reason, compiling a form that uses ‘include’ results in
one compilation unit, composed of multiple files.  Loading the compiled
file is one ‘stat’ operation for the compilation unit, instead of ‘2*N’
in the case of ‘load’ (once for each loaded source file, and once each
corresponding compiled file, in the best case).

   Unlike ‘load’, ‘include’ also works within nested lexical contexts.
It so happens that the optimizer works best within a lexical context,
because all of the uses of bindings in a lexical context are visible, so
composing files by including them within a ‘(let () ...)’ can sometimes
lead to important speed improvements.

   On the other hand, ‘include’ does have all the disadvantages of early
binding: once the code with the ‘include’ is compiled, no change to the
included file is reflected in the future behavior of the including form.

   Also, the particular form of ‘include’, which requires an absolute
path, or a path relative to the current directory at compile-time, is
not very amenable to compiling the source in one place, but then
installing the source to another place.  For this reason, Guile provides
another form, ‘include-from-path’, which looks for the source file to
include within a load path.

 -- Scheme Syntax: include-from-path file-name
     Like ‘include’, but instead of expecting ‘file-name’ to be an
     absolute file name, it is expected to be a relative path to search
     in the ‘%load-path’.

   ‘include-from-path’ is more useful when you want to install all of
the source files for a package (as you should!).  It makes it possible
to evaluate an installed file from source, instead of relying on the
‘.go’ file being up to date.


File: guile.info,  Node: Sandboxed Evaluation,  Next: REPL Servers,  Prev: Local Inclusion,  Up: Read/Load/Eval/Compile

6.18.12 Sandboxed Evaluation
----------------------------

Sometimes you would like to evaluate code that comes from an untrusted
party.  The safest way to do this is to buy a new computer, evaluate the
code on that computer, then throw the machine away.  However if you are
unwilling to take this simple approach, Guile does include a limited
“sandbox” facility that can allow untrusted code to be evaluated with
some confidence.

   To use the sandboxed evaluator, load its module:

     (use-modules (ice-9 sandbox))

   Guile’s sandboxing facility starts with the ability to restrict the
time and space used by a piece of code.

 -- Scheme Procedure: call-with-time-limit limit thunk limit-reached
     Call THUNK, but cancel it if LIMIT seconds of wall-clock time have
     elapsed.  If the computation is cancelled, call LIMIT-REACHED in
     tail position.  THUNK must not disable interrupts or prevent an
     abort via a ‘dynamic-wind’ unwind handler.

 -- Scheme Procedure: call-with-allocation-limit limit thunk
          limit-reached
     Call THUNK, but cancel it if LIMIT bytes have been allocated.  If
     the computation is cancelled, call LIMIT-REACHED in tail position.
     THUNK must not disable interrupts or prevent an abort via a
     ‘dynamic-wind’ unwind handler.

     This limit applies to both stack and heap allocation.  The
     computation will not be aborted before LIMIT bytes have been
     allocated, but for the heap allocation limit, the check may be
     postponed until the next garbage collection.

     Note that as a current shortcoming, the heap size limit applies to
     all threads; concurrent allocation by other unrelated threads
     counts towards the allocation limit.

 -- Scheme Procedure: call-with-time-and-allocation-limits time-limit
          allocation-limit thunk
     Invoke THUNK in a dynamic extent in which its execution is limited
     to TIME-LIMIT seconds of wall-clock time, and its allocation to
     ALLOCATION-LIMIT bytes.  THUNK must not disable interrupts or
     prevent an abort via a ‘dynamic-wind’ unwind handler.

     If successful, return all values produced by invoking THUNK.  Any
     uncaught exception thrown by the thunk will propagate out.  If the
     time or allocation limit is exceeded, an exception will be thrown
     to the ‘limit-exceeded’ key.

   The time limit and stack limit are both very precise, but the heap
limit only gets checked asynchronously, after a garbage collection.  In
particular, if the heap is already very large, the number of allocated
bytes between garbage collections will be large, and therefore the
precision of the check is reduced.

   Additionally, due to the mechanism used by the allocation limit (the
‘after-gc-hook’), large single allocations like ‘(make-vector #e1e7)’
are only detected after the allocation completes, even if the allocation
itself causes garbage collection.  It’s possible therefore for user code
to not only exceed the allocation limit set, but also to exhaust all
available memory, causing out-of-memory conditions at any allocation
site.  Failure to allocate memory in Guile itself should be safe and
cause an exception to be thrown, but most systems are not designed to
handle ‘malloc’ failures.  An allocation failure may therefore exercise
unexpected code paths in your system, so it is a weakness of the sandbox
(and therefore an interesting point of attack).

   The main sandbox interface is ‘eval-in-sandbox’.

 -- Scheme Procedure: eval-in-sandbox exp [#:time-limit 0.1]
          [#:allocation-limit #e10e6] [#:bindings all-pure-bindings]
          [#:module (make-sandbox-module bindings)] [#:sever-module? #t]
     Evaluate the Scheme expression EXP within an isolated "sandbox".
     Limit its execution to TIME-LIMIT seconds of wall-clock time, and
     limit its allocation to ALLOCATION-LIMIT bytes.

     The evaluation will occur in MODULE, which defaults to the result
     of calling ‘make-sandbox-module’ on BINDINGS, which itself defaults
     to ‘all-pure-bindings’.  This is the core of the sandbox: creating
     a scope for the expression that is “safe”.

     A safe sandbox module has two characteristics.  Firstly, it will
     not allow the expression being evaluated to avoid being cancelled
     due to time or allocation limits.  This ensures that the expression
     terminates in a timely fashion.

     Secondly, a safe sandbox module will prevent the evaluation from
     receiving information from previous evaluations, or from affecting
     future evaluations.  All combinations of binding sets exported by
     ‘(ice-9 sandbox)’ form safe sandbox modules.

     The BINDINGS should be given as a list of import sets.  One import
     set is a list whose car names an interface, like ‘(ice-9 q)’, and
     whose cdr is a list of imports.  An import is either a bare symbol
     or a pair of ‘(OUT . IN)’, where OUT and IN are both symbols and
     denote the name under which a binding is exported from the module,
     and the name under which to make the binding available,
     respectively.  Note that BINDINGS is only used as an input to the
     default initializer for the MODULE argument; if you pass
     ‘#:module’, BINDINGS is unused.  If SEVER-MODULE? is true (the
     default), the module will be unlinked from the global module tree
     after the evaluation returns, to allow MOD to be garbage-collected.

     If successful, return all values produced by EXP.  Any uncaught
     exception thrown by the expression will propagate out.  If the time
     or allocation limit is exceeded, an exception will be thrown to the
     ‘limit-exceeded’ key.

   Constructing a safe sandbox module is tricky in general.  Guile
defines an easy way to construct safe modules from predefined sets of
bindings.  Before getting to that interface, here are some general notes
on safety.

  1. The time and allocation limits rely on the ability to interrupt and
     cancel a computation.  For this reason, no binding included in a
     sandbox module should be able to indefinitely postpone interrupt
     handling, nor should a binding be able to prevent an abort.  In
     practice this second consideration means that ‘dynamic-wind’ should
     not be included in any binding set.
  2. The time and allocation limits apply only to the ‘eval-in-sandbox’
     call.  If the call returns a procedure which is later called, no
     limit is “automatically” in place.  Users of ‘eval-in-sandbox’ have
     to be very careful to reimpose limits when calling procedures that
     escape from sandboxes.
  3. Similarly, the dynamic environment of the ‘eval-in-sandbox’ call is
     not necessarily in place when any procedure that escapes from the
     sandbox is later called.

     This detail prevents us from exposing ‘primitive-eval’ to the
     sandbox, for two reasons.  The first is that it’s possible for
     legacy code to forge references to any binding, if the
     ‘allow-legacy-syntax-objects?’ parameter is true.  The default for
     this parameter is true; *note Syntax Transformer Helpers:: for the
     details.  The parameter is bound to ‘#f’ for the duration of the
     ‘eval-in-sandbox’ call itself, but that will not be in place during
     calls to escaped procedures.

     The second reason we don’t expose ‘primitive-eval’ is that
     ‘primitive-eval’ implicitly works in the current module, which for
     an escaped procedure will probably be different than the module
     that is current for the ‘eval-in-sandbox’ call itself.

     The common denominator here is that if an interface exposed to the
     sandbox relies on dynamic environments, it is easy to mistakenly
     grant the sandboxed procedure additional capabilities in the form
     of bindings that it should not have access to.  For this reason,
     the default sets of predefined bindings do not depend on any
     dynamically scoped value.
  4. Mutation may allow a sandboxed evaluation to break some invariant
     in users of data supplied to it.  A lot of code culturally doesn’t
     expect mutation, but if you hand mutable data to a sandboxed
     evaluation and you also grant mutating capabilities to that
     evaluation, then the sandboxed code may indeed mutate that data.
     The default set of bindings to the sandbox do not include any
     mutating primitives.

     Relatedly, ‘set!’ may allow a sandbox to mutate a primitive,
     invalidating many system-wide invariants.  Guile is currently quite
     permissive when it comes to imported bindings and mutability.
     Although ‘set!’ to a module-local or lexically bound variable would
     be fine, we don’t currently have an easy way to disallow ‘set!’ to
     an imported binding, so currently no binding set includes ‘set!’.
  5. Mutation may allow a sandboxed evaluation to keep state, or make a
     communication mechanism with other code.  On the one hand this
     sounds cool, but on the other hand maybe this is part of your
     threat model.  Again, the default set of bindings doesn’t include
     mutating primitives, preventing sandboxed evaluations from keeping
     state.
  6. The sandbox should probably not be able to open a network
     connection, or write to a file, or open a file from disk.  The
     default binding set includes no interaction with the operating
     system.

   If you, dear reader, find the above discussion interesting, you will
enjoy Jonathan Rees’ dissertation, “A Security Kernel Based on the
Lambda Calculus”.

 -- Scheme Variable: all-pure-bindings
     All “pure” bindings that together form a safe subset of those
     bindings available by default to Guile user code.

 -- Scheme Variable: all-pure-and-impure-bindings
     Like ‘all-pure-bindings’, but additionally including mutating
     primitives like ‘vector-set!’.  This set is still safe in the sense
     mentioned above, with the caveats about mutation.

   The components of these composite sets are as follows:
 -- Scheme Variable: alist-bindings
 -- Scheme Variable: array-bindings
 -- Scheme Variable: bit-bindings
 -- Scheme Variable: bitvector-bindings
 -- Scheme Variable: char-bindings
 -- Scheme Variable: char-set-bindings
 -- Scheme Variable: clock-bindings
 -- Scheme Variable: core-bindings
 -- Scheme Variable: error-bindings
 -- Scheme Variable: fluid-bindings
 -- Scheme Variable: hash-bindings
 -- Scheme Variable: iteration-bindings
 -- Scheme Variable: keyword-bindings
 -- Scheme Variable: list-bindings
 -- Scheme Variable: macro-bindings
 -- Scheme Variable: nil-bindings
 -- Scheme Variable: number-bindings
 -- Scheme Variable: pair-bindings
 -- Scheme Variable: predicate-bindings
 -- Scheme Variable: procedure-bindings
 -- Scheme Variable: promise-bindings
 -- Scheme Variable: prompt-bindings
 -- Scheme Variable: regexp-bindings
 -- Scheme Variable: sort-bindings
 -- Scheme Variable: srfi-4-bindings
 -- Scheme Variable: string-bindings
 -- Scheme Variable: symbol-bindings
 -- Scheme Variable: unspecified-bindings
 -- Scheme Variable: variable-bindings
 -- Scheme Variable: vector-bindings
 -- Scheme Variable: version-bindings
     The components of ‘all-pure-bindings’.

 -- Scheme Variable: mutating-alist-bindings
 -- Scheme Variable: mutating-array-bindings
 -- Scheme Variable: mutating-bitvector-bindings
 -- Scheme Variable: mutating-fluid-bindings
 -- Scheme Variable: mutating-hash-bindings
 -- Scheme Variable: mutating-list-bindings
 -- Scheme Variable: mutating-pair-bindings
 -- Scheme Variable: mutating-sort-bindings
 -- Scheme Variable: mutating-srfi-4-bindings
 -- Scheme Variable: mutating-string-bindings
 -- Scheme Variable: mutating-variable-bindings
 -- Scheme Variable: mutating-vector-bindings
     The additional components of ‘all-pure-and-impure-bindings’.

   Finally, what do you do with a binding set?  What is a binding set
anyway?  ‘make-sandbox-module’ is here for you.

 -- Scheme Procedure: make-sandbox-module bindings
     Return a fresh module that only contains BINDINGS.

     The BINDINGS should be given as a list of import sets.  One import
     set is a list whose car names an interface, like ‘(ice-9 q)’, and
     whose cdr is a list of imports.  An import is either a bare symbol
     or a pair of ‘(OUT . IN)’, where OUT and IN are both symbols and
     denote the name under which a binding is exported from the module,
     and the name under which to make the binding available,
     respectively.

   So you see that binding sets are just lists, and
‘all-pure-and-impure-bindings’ is really just the result of appending
all of the component binding sets.


File: guile.info,  Node: REPL Servers,  Next: Cooperative REPL Servers,  Prev: Sandboxed Evaluation,  Up: Read/Load/Eval/Compile

6.18.13 REPL Servers
--------------------

The procedures in this section are provided by
     (use-modules (system repl server))

   When an application is written in Guile, it is often convenient to
allow the user to be able to interact with it by evaluating Scheme
expressions in a REPL.

   The procedures of this module allow you to spawn a “REPL server”,
which permits interaction over a local or TCP connection.  Guile itself
uses them internally to implement the ‘--listen’ switch, *note
Command-line Options::.

 -- Scheme Procedure: make-tcp-server-socket [#:host=#f] [#:addr]
          [#:port=37146]
     Return a stream socket bound to a given address ADDR and port
     number PORT.  If the HOST is given, and ADDR is not, then the HOST
     string is converted to an address.  If neither is given, we use the
     loopback address.

 -- Scheme Procedure: make-unix-domain-server-socket
          [#:path="/tmp/guile-socket"]
     Return a UNIX domain socket, bound to a given PATH.

 -- Scheme Procedure: run-server [server-socket]
 -- Scheme Procedure: spawn-server [server-socket]
     Create and run a REPL, making it available over the given
     SERVER-SOCKET.  If SERVER-SOCKET is not provided, it defaults to
     the socket created by calling ‘make-tcp-server-socket’ with no
     arguments.

     ‘run-server’ runs the server in the current thread, whereas
     ‘spawn-server’ runs the server in a new thread.

 -- Scheme Procedure: stop-server-and-clients!
     Closes the connection on all running server sockets.

     Please note that in the current implementation, the REPL threads
     are cancelled without unwinding their stacks.  If any of them are
     holding mutexes or are within a critical section, the results are
     unspecified.


File: guile.info,  Node: Cooperative REPL Servers,  Prev: REPL Servers,  Up: Read/Load/Eval/Compile

6.18.14 Cooperative REPL Servers
--------------------------------

The procedures in this section are provided by
     (use-modules (system repl coop-server))

   Whereas ordinary REPL servers run in their own threads (*note REPL
Servers::), sometimes it is more convenient to provide REPLs that run at
specified times within an existing thread, for example in programs
utilizing an event loop or in single-threaded programs.  This allows for
safe access and mutation of a program’s data structures from the REPL,
without concern for thread synchronization.

   Although the REPLs are run in the thread that calls
‘spawn-coop-repl-server’ and ‘poll-coop-repl-server’, dedicated threads
are spawned so that the calling thread is not blocked.  The spawned
threads read input for the REPLs and to listen for new connections.

   Cooperative REPL servers must be polled periodically to evaluate any
pending expressions by calling ‘poll-coop-repl-server’ with the object
returned from ‘spawn-coop-repl-server’.  The thread that calls
‘poll-coop-repl-server’ will be blocked for as long as the expression
takes to be evaluated or if the debugger is entered.

 -- Scheme Procedure: spawn-coop-repl-server [server-socket]
     Create and return a new cooperative REPL server object, and spawn a
     new thread to listen for connections on SERVER-SOCKET.  Proper
     functioning of the REPL server requires that
     ‘poll-coop-repl-server’ be called periodically on the returned
     server object.

 -- Scheme Procedure: poll-coop-repl-server coop-server
     Poll the cooperative REPL server COOP-SERVER and apply a pending
     operation if there is one, such as evaluating an expression typed
     at the REPL prompt.  This procedure must be called from the same
     thread that called ‘spawn-coop-repl-server’.


File: guile.info,  Node: Memory Management,  Next: Modules,  Prev: Read/Load/Eval/Compile,  Up: API Reference

6.19 Memory Management and Garbage Collection
=============================================

Guile uses a _garbage collector_ to manage most of its objects.  While
the garbage collector is designed to be mostly invisible, you sometimes
need to interact with it explicitly.

   See *note Garbage Collection:: for a general discussion of how
garbage collection relates to using Guile from C.

* Menu:

* Garbage Collection Functions::
* Memory Blocks::
* Weak References::
* Guardians::


File: guile.info,  Node: Garbage Collection Functions,  Next: Memory Blocks,  Up: Memory Management

6.19.1 Function related to Garbage Collection
---------------------------------------------

 -- Scheme Procedure: gc
 -- C Function: scm_gc ()
     Finds all of the “live” ‘SCM’ objects and reclaims for further use
     those that are no longer accessible.  You normally don’t need to
     call this function explicitly.  Its functionality is invoked
     automatically as needed.

 -- C Function: SCM scm_gc_protect_object (SCM OBJ)
     Protects OBJ from being freed by the garbage collector, when it
     otherwise might be.  When you are done with the object, call
     ‘scm_gc_unprotect_object’ on the object.  Calls to
     ‘scm_gc_protect_object’/‘scm_gc_unprotect_object’ can be nested,
     and the object remains protected until it has been unprotected as
     many times as it was protected.  It is an error to unprotect an
     object more times than it has been protected.  Returns the SCM
     object it was passed.

     Note that storing OBJ in a C global variable has the same
     effect(1).

 -- C Function: SCM scm_gc_unprotect_object (SCM OBJ)

     Unprotects an object from the garbage collector which was protected
     by ‘scm_gc_unprotect_object’.  Returns the SCM object it was
     passed.

 -- C Function: SCM scm_permanent_object (SCM OBJ)

     Similar to ‘scm_gc_protect_object’ in that it causes the collector
     to always mark the object, except that it should not be nested
     (only call ‘scm_permanent_object’ on an object once), and it has no
     corresponding unpermanent function.  Once an object is declared
     permanent, it will never be freed.  Returns the SCM object it was
     passed.

 -- C Macro: void scm_remember_upto_here_1 (SCM obj)
 -- C Macro: void scm_remember_upto_here_2 (SCM obj1, SCM obj2)
     Create a reference to the given object or objects, so they’re
     certain to be present on the stack or in a register and hence will
     not be freed by the garbage collector before this point.

     Note that these functions can only be applied to ordinary C local
     variables (ie. “automatics”).  Objects held in global or static
     variables or some malloced block or the like cannot be protected
     with this mechanism.

 -- Scheme Procedure: gc-stats
 -- C Function: scm_gc_stats ()
     Return an association list of statistics about Guile’s current use
     of storage.

 -- Scheme Procedure: gc-live-object-stats
 -- C Function: scm_gc_live_object_stats ()
     Return an alist of statistics of the current live objects.

 -- Function: void scm_gc_mark (SCM X)
     Mark the object X, and recurse on any objects X refers to.  If X’s
     mark bit is already set, return immediately.  This function must
     only be called during the mark-phase of garbage collection,
     typically from a smob _mark_ function.

   ---------- Footnotes ----------

   (1) In Guile up to version 1.8, C global variables were not visited
by the garbage collector in the mark phase; hence,
‘scm_gc_protect_object’ was the only way in C to prevent a Scheme object
from being freed.


File: guile.info,  Node: Memory Blocks,  Next: Weak References,  Prev: Garbage Collection Functions,  Up: Memory Management

6.19.2 Memory Blocks
--------------------

In C programs, dynamic management of memory blocks is normally done with
the functions malloc, realloc, and free.  Guile has additional functions
for dynamic memory allocation that are integrated into the garbage
collector and the error reporting system.

   Memory blocks that are associated with Scheme objects (for example a
foreign object) should be allocated with ‘scm_gc_malloc’ or
‘scm_gc_malloc_pointerless’.  These two functions will either return a
valid pointer or signal an error.  Memory blocks allocated this way may
be released explicitly; however, this is not strictly needed, and we
recommend _not_ calling ‘scm_gc_free’.  All memory allocated with
‘scm_gc_malloc’ or ‘scm_gc_malloc_pointerless’ is automatically
reclaimed when the garbage collector no longer sees any live reference
to it(1).

   When garbage collection occurs, Guile will visit the words in memory
allocated with ‘scm_gc_malloc’, looking for live pointers.  This means
that if ‘scm_gc_malloc’-allocated memory contains a pointer to some
other part of the memory, the garbage collector notices it and prevents
it from being reclaimed(2).  Conversely, memory allocated with
‘scm_gc_malloc_pointerless’ is assumed to be “pointer-less” and is not
scanned for pointers.

   For memory that is not associated with a Scheme object, you can use
‘scm_malloc’ instead of ‘malloc’.  Like ‘scm_gc_malloc’, it will either
return a valid pointer or signal an error.  However, it will not assume
that the new memory block can be freed by a garbage collection.  The
memory must be explicitly freed with ‘free’.

   There is also ‘scm_gc_realloc’ and ‘scm_realloc’, to be used in place
of ‘realloc’ when appropriate, and ‘scm_gc_calloc’ and ‘scm_calloc’, to
be used in place of ‘calloc’ when appropriate.

   The function ‘scm_dynwind_free’ can be useful when memory should be
freed with libc’s ‘free’ when leaving a dynwind context, *Note Dynamic
Wind::.

 -- C Function: void * scm_malloc (size_t SIZE)
 -- C Function: void * scm_calloc (size_t SIZE)
     Allocate SIZE bytes of memory and return a pointer to it.  When
     SIZE is 0, return ‘NULL’.  When not enough memory is available,
     signal an error.  This function runs the GC to free up some memory
     when it deems it appropriate.

     The memory is allocated by the libc ‘malloc’ function and can be
     freed with ‘free’.  There is no ‘scm_free’ function to go with
     ‘scm_malloc’ to make it easier to pass memory back and forth
     between different modules.

     The function ‘scm_calloc’ is similar to ‘scm_malloc’, but
     initializes the block of memory to zero as well.

     These functions will (indirectly) call
     ‘scm_gc_register_allocation’.

 -- C Function: void * scm_realloc (void *MEM, size_t NEW_SIZE)
     Change the size of the memory block at MEM to NEW_SIZE and return
     its new location.  When NEW_SIZE is 0, this is the same as calling
     ‘free’ on MEM and ‘NULL’ is returned.  When MEM is ‘NULL’, this
     function behaves like ‘scm_malloc’ and allocates a new block of
     size NEW_SIZE.

     When not enough memory is available, signal an error.  This
     function runs the GC to free up some memory when it deems it
     appropriate.

     This function will call ‘scm_gc_register_allocation’.

 -- C Function: void * scm_gc_malloc (size_t SIZE, const char *WHAT)
 -- C Function: void * scm_gc_malloc_pointerless (size_t SIZE, const
          char *WHAT)
 -- C Function: void * scm_gc_realloc (void *MEM, size_t OLD_SIZE,
          size_t NEW_SIZE, const char *WHAT);
 -- C Function: void * scm_gc_calloc (size_t SIZE, const char *WHAT)
     Allocate SIZE bytes of automatically-managed memory.  The memory is
     automatically freed when no longer referenced from any live memory
     block.

     When garbage collection occurs, Guile will visit the words in
     memory allocated with ‘scm_gc_malloc’ or ‘scm_gc_calloc’, looking
     for pointers to other memory allocations that are managed by the
     GC. In contrast, memory allocated by ‘scm_gc_malloc_pointerless’ is
     not scanned for pointers.

     The ‘scm_gc_realloc’ call preserves the “pointerlessness” of the
     memory area pointed to by MEM.  Note that you need to pass the old
     size of a reallocated memory block as well.  See below for a
     motivation.

 -- C Function: void scm_gc_free (void *MEM, size_t SIZE, const char
          *WHAT)
     Explicitly free the memory block pointed to by MEM, which was
     previously allocated by one of the above ‘scm_gc’ functions.  This
     function is almost always unnecessary, except for codebases that
     still need to compile on Guile 1.8.

     Note that you need to explicitly pass the SIZE parameter.  This is
     done since it should normally be easy to provide this parameter
     (for memory that is associated with GC controlled objects) and help
     keep the memory management overhead very low.  However, in Guile
     2.x, SIZE is always ignored.

 -- C Function: void scm_gc_register_allocation (size_t SIZE)
     Informs the garbage collector that SIZE bytes have been allocated,
     which the collector would otherwise not have known about.

     In general, Scheme will decide to collect garbage only after some
     amount of memory has been allocated.  Calling this function will
     make the Scheme garbage collector know about more allocation, and
     thus run more often (as appropriate).

     It is especially important to call this function when large
     unmanaged allocations, like images, may be freed by small Scheme
     allocations, like foreign objects.

 -- C Function: void scm_dynwind_free (void *mem)
     Equivalent to ‘scm_dynwind_unwind_handler (free, MEM,
     SCM_F_WIND_EXPLICITLY)’.  That is, the memory block at MEM will be
     freed (using ‘free’ from the C library) when the current dynwind is
     left.

 -- Scheme Procedure: malloc-stats
     Return an alist ((WHAT .  N) ...)  describing number of malloced
     objects.  WHAT is the second argument to ‘scm_gc_malloc’, N is the
     number of objects of that type currently allocated.

     This function is only available if the ‘GUILE_DEBUG_MALLOC’
     preprocessor macro was defined when Guile was compiled.

   ---------- Footnotes ----------

   (1) In Guile up to version 1.8, memory allocated with ‘scm_gc_malloc’
_had_ to be freed with ‘scm_gc_free’.

   (2) In Guile up to 1.8, memory allocated with ‘scm_gc_malloc’ was
_not_ visited by the collector in the mark phase.  Consequently, the GC
had to be told explicitly about pointers to live objects contained in
the memory block, e.g., via SMOB mark functions (*note
‘scm_set_smob_mark’: Smobs.)


File: guile.info,  Node: Weak References,  Next: Guardians,  Prev: Memory Blocks,  Up: Memory Management

6.19.3 Weak References
----------------------

[FIXME: This chapter is based on Mikael Djurfeldt’s answer to a question
by Michael Livshin.  Any mistakes are not theirs, of course.  ]

   Weak references let you attach bookkeeping information to data so
that the additional information automatically disappears when the
original data is no longer in use and gets garbage collected.  In a weak
key hash, the hash entry for that key disappears as soon as the key is
no longer referenced from anywhere else.  For weak value hashes, the
same happens as soon as the value is no longer in use.  Entries in a
doubly weak hash disappear when either the key or the value are not used
anywhere else anymore.

   Object properties offer the same kind of functionality as weak key
hashes in many situations.  (*note Object Properties::)

   Here’s an example (a little bit strained perhaps, but one of the
examples is actually used in Guile):

   Assume that you’re implementing a debugging system where you want to
associate information about filename and position of source code
expressions with the expressions themselves.

   Hashtables can be used for that, but if you use ordinary hash tables
it will be impossible for the scheme interpreter to "forget" old source
when, for example, a file is reloaded.

   To implement the mapping from source code expressions to positional
information it is necessary to use weak-key tables since we don’t want
the expressions to be remembered just because they are in our table.

   To implement a mapping from source file line numbers to source code
expressions you would use a weak-value table.

   To implement a mapping from source code expressions to the procedures
they constitute a doubly-weak table has to be used.

* Menu:

* Weak hash tables::
* Weak vectors::


File: guile.info,  Node: Weak hash tables,  Next: Weak vectors,  Up: Weak References

6.19.3.1 Weak hash tables
.........................

 -- Scheme Procedure: make-weak-key-hash-table [size]
 -- Scheme Procedure: make-weak-value-hash-table [size]
 -- Scheme Procedure: make-doubly-weak-hash-table [size]
 -- C Function: scm_make_weak_key_hash_table (size)
 -- C Function: scm_make_weak_value_hash_table (size)
 -- C Function: scm_make_doubly_weak_hash_table (size)
     Return a weak hash table with SIZE buckets.  As with any hash
     table, choosing a good size for the table requires some caution.

     You can modify weak hash tables in exactly the same way you would
     modify regular hash tables, with the exception of the routines that
     act on handles.  Weak tables have a different implementation behind
     the scenes that doesn’t have handles.  *note Hash Tables::, for
     more on ‘hashq-ref’ et al.

   Note that in a weak-key hash table, the reference to the value is
strong.  This means that if the value references the key, even
indirectly, the key will never be collected, which can lead to a memory
leak.  The reverse is true for weak value tables.

 -- Scheme Procedure: weak-key-hash-table? obj
 -- Scheme Procedure: weak-value-hash-table? obj
 -- Scheme Procedure: doubly-weak-hash-table? obj
 -- C Function: scm_weak_key_hash_table_p (obj)
 -- C Function: scm_weak_value_hash_table_p (obj)
 -- C Function: scm_doubly_weak_hash_table_p (obj)
     Return ‘#t’ if OBJ is the specified weak hash table.  Note that a
     doubly weak hash table is neither a weak key nor a weak value hash
     table.


File: guile.info,  Node: Weak vectors,  Prev: Weak hash tables,  Up: Weak References

6.19.3.2 Weak vectors
.....................

 -- Scheme Procedure: make-weak-vector size [fill]
 -- C Function: scm_make_weak_vector (size, fill)
     Return a weak vector with SIZE elements.  If the optional argument
     FILL is given, all entries in the vector will be set to FILL.  The
     default value for FILL is the empty list.

 -- Scheme Procedure: weak-vector elem ...
 -- Scheme Procedure: list->weak-vector l
 -- C Function: scm_weak_vector (l)
     Construct a weak vector from a list: ‘weak-vector’ uses the list of
     its arguments while ‘list->weak-vector’ uses its only argument L (a
     list) to construct a weak vector the same way ‘list->vector’ would.

 -- Scheme Procedure: weak-vector? obj
 -- C Function: scm_weak_vector_p (obj)
     Return ‘#t’ if OBJ is a weak vector.

 -- Scheme Procedure: weak-vector-ref wvect k
 -- C Function: scm_weak_vector_ref (wvect, k)
     Return the Kth element of the weak vector WVECT, or ‘#f’ if that
     element has been collected.

 -- Scheme Procedure: weak-vector-set! wvect k elt
 -- C Function: scm_weak_vector_set_x (wvect, k, elt)
     Set the Kth element of the weak vector WVECT to ELT.


File: guile.info,  Node: Guardians,  Prev: Weak References,  Up: Memory Management

6.19.4 Guardians
----------------

Guardians provide a way to be notified about objects that would
otherwise be collected as garbage.  Guarding them prevents the objects
from being collected and cleanup actions can be performed on them, for
example.

   See R. Kent Dybvig, Carl Bruggeman, and David Eby (1993) "Guardians
in a Generation-Based Garbage Collector".  ACM SIGPLAN Conference on
Programming Language Design and Implementation, June 1993.

 -- Scheme Procedure: make-guardian
 -- C Function: scm_make_guardian ()
     Create a new guardian.  A guardian protects a set of objects from
     garbage collection, allowing a program to apply cleanup or other
     actions.

     ‘make-guardian’ returns a procedure representing the guardian.
     Calling the guardian procedure with an argument adds the argument
     to the guardian’s set of protected objects.  Calling the guardian
     procedure without an argument returns one of the protected objects
     which are ready for garbage collection, or ‘#f’ if no such object
     is available.  Objects which are returned in this way are removed
     from the guardian.

     You can put a single object into a guardian more than once and you
     can put a single object into more than one guardian.  The object
     will then be returned multiple times by the guardian procedures.

     An object is eligible to be returned from a guardian when it is no
     longer referenced from outside any guardian.

     There is no guarantee about the order in which objects are returned
     from a guardian.  If you want to impose an order on finalization
     actions, for example, you can do that by keeping objects alive in
     some global data structure until they are no longer needed for
     finalizing other objects.

     Being an element in a weak vector, a key in a hash table with weak
     keys, or a value in a hash table with weak values does not prevent
     an object from being returned by a guardian.  But as long as an
     object can be returned from a guardian it will not be removed from
     such a weak vector or hash table.  In other words, a weak link does
     not prevent an object from being considered collectable, but being
     inside a guardian prevents a weak link from being broken.

     A key in a weak key hash table can be thought of as having a strong
     reference to its associated value as long as the key is accessible.
     Consequently, when the key is only accessible from within a
     guardian, the reference from the key to the value is also
     considered to be coming from within a guardian.  Thus, if there is
     no other reference to the value, it is eligible to be returned from
     a guardian.


File: guile.info,  Node: Modules,  Next: Foreign Function Interface,  Prev: Memory Management,  Up: API Reference

6.20 Modules
============

When programs become large, naming conflicts can occur when a function
or global variable defined in one file has the same name as a function
or global variable in another file.  Even just a _similarity_ between
function names can cause hard-to-find bugs, since a programmer might
type the wrong function name.

   The approach used to tackle this problem is called _information
encapsulation_, which consists of packaging functional units into a
given name space that is clearly separated from other name spaces.

   The language features that allow this are usually called _the module
system_ because programs are broken up into modules that are compiled
separately (or loaded separately in an interpreter).

   Older languages, like C, have limited support for name space
manipulation and protection.  In C a variable or function is public by
default, and can be made local to a module with the ‘static’ keyword.
But you cannot reference public variables and functions from another
module with different names.

   More advanced module systems have become a common feature in recently
designed languages: ML, Python, Perl, and Modula 3 all allow the
_renaming_ of objects from a foreign module, so they will not clutter
the global name space.

   In addition, Guile offers variables as first-class objects.  They can
be used for interacting with the module system.

* Menu:

* General Information about Modules::  Guile module basics.
* Using Guile Modules::         How to use existing modules.
* Creating Guile Modules::      How to package your code into modules.
* Modules and the File System:: Installing modules in the file system.
* R6RS Version References::     Using version numbers with modules.
* R6RS Libraries::              The library and import forms.
* Variables::                   First-class variables.
* Module System Reflection::    First-class modules.
* Accessing Modules from C::    How to work with modules with C code.
* provide and require::         The SLIB feature mechanism.
* Environments::                R5RS top-level environments.


File: guile.info,  Node: General Information about Modules,  Next: Using Guile Modules,  Up: Modules

6.20.1 General Information about Modules
----------------------------------------

A Guile module can be thought of as a collection of named procedures,
variables and macros.  More precisely, it is a set of “bindings” of
symbols (names) to Scheme objects.

   Within a module, all bindings are visible.  Certain bindings can be
declared “public”, in which case they are added to the module’s
so-called “export list”; this set of public bindings is called the
module’s “public interface” (*note Creating Guile Modules::).

   A client module “uses” a providing module’s bindings by either
accessing the providing module’s public interface, or by building a
custom interface (and then accessing that).  In a custom interface, the
client module can “select” which bindings to access and can also
algorithmically “rename” bindings.  In contrast, when using the
providing module’s public interface, the entire export list is available
without renaming (*note Using Guile Modules::).

   All Guile modules have a unique “module name”, for example ‘(ice-9
popen)’ or ‘(srfi srfi-11)’.  Module names are lists of one or more
symbols.

   When Guile goes to use an interface from a module, for example
‘(ice-9 popen)’, Guile first looks to see if it has loaded ‘(ice-9
popen)’ for any reason.  If the module has not been loaded yet, Guile
searches a “load path” for a file that might define it, and loads that
file.

   The following subsections go into more detail on using, creating,
installing, and otherwise manipulating modules and the module system.


File: guile.info,  Node: Using Guile Modules,  Next: Creating Guile Modules,  Prev: General Information about Modules,  Up: Modules

6.20.2 Using Guile Modules
--------------------------

To use a Guile module is to access either its public interface or a
custom interface (*note General Information about Modules::).  Both
types of access are handled by the syntactic form ‘use-modules’, which
accepts one or more interface specifications and, upon evaluation,
arranges for those interfaces to be available to the current module.
This process may include locating and loading code for a given module if
that code has not yet been loaded, following ‘%load-path’ (*note Modules
and the File System::).

   An “interface specification” has one of two forms.  The first
variation is simply to name the module, in which case its public
interface is the one accessed.  For example:

     (use-modules (ice-9 popen))

   Here, the interface specification is ‘(ice-9 popen)’, and the result
is that the current module now has access to ‘open-pipe’, ‘close-pipe’,
‘open-input-pipe’, and so on (*note Pipes::).

   Note in the previous example that if the current module had already
defined ‘open-pipe’, that definition would be overwritten by the
definition in ‘(ice-9 popen)’.  For this reason (and others), there is a
second variation of interface specification that not only names a module
to be accessed, but also selects bindings from it and renames them to
suit the current module’s needs.  For example:

     (use-modules ((ice-9 popen)
                   #:select ((open-pipe . pipe-open) close-pipe)
                   #:renamer (symbol-prefix-proc 'unixy:)))

or more simply:

     (use-modules ((ice-9 popen)
                   #:select ((open-pipe . pipe-open) close-pipe)
                   #:prefix unixy:))

   Here, the interface specification is more complex than before, and
the result is that a custom interface with only two bindings is created
and subsequently accessed by the current module.  The mapping of old to
new names is as follows:

     (ice-9 popen) sees:             current module sees:
     open-pipe                       unixy:pipe-open
     close-pipe                      unixy:close-pipe

   This example also shows how to use the convenience procedure
‘symbol-prefix-proc’.

   You can also directly refer to bindings in a module by using the ‘@’
syntax.  For example, instead of using the ‘use-modules’ statement from
above and writing ‘unixy:pipe-open’ to refer to the ‘pipe-open’ from the
‘(ice-9 popen)’, you could also write ‘(@ (ice-9 popen) open-pipe)’.
Thus an alternative to the complete ‘use-modules’ statement would be

     (define unixy:pipe-open (@ (ice-9 popen) open-pipe))
     (define unixy:close-pipe (@ (ice-9 popen) close-pipe))

   There is also ‘@@’, which can be used like ‘@’, but does not check
whether the variable that is being accessed is actually exported.  Thus,
‘@@’ can be thought of as the impolite version of ‘@’ and should only be
used as a last resort or for debugging, for example.

   Note that just as with a ‘use-modules’ statement, any module that has
not yet been loaded will be loaded when referenced by a ‘@’ or ‘@@’
form.

   You can also use the ‘@’ and ‘@@’ syntaxes as the target of a ‘set!’
when the binding refers to a variable.

 -- Scheme Procedure: symbol-prefix-proc prefix-sym
     Return a procedure that prefixes its arg (a symbol) with
     PREFIX-SYM.

 -- syntax: use-modules spec ...
     Resolve each interface specification SPEC into an interface and
     arrange for these to be accessible by the current module.  The
     return value is unspecified.

     SPEC can be a list of symbols, in which case it names a module
     whose public interface is found and used.

     SPEC can also be of the form:

           (MODULE-NAME [#:select SELECTION]
                        [#:prefix PREFIX]
                        [#:renamer RENAMER])

     in which case a custom interface is newly created and used.
     MODULE-NAME is a list of symbols, as above; SELECTION is a list of
     selection-specs; PREFIX is a symbol that is prepended to imported
     names; and RENAMER is a procedure that takes a symbol and returns
     its new name.  A selection-spec is either a symbol or a pair of
     symbols ‘(ORIG . SEEN)’, where ORIG is the name in the used module
     and SEEN is the name in the using module.  Note that SEEN is also
     modified by PREFIX and RENAMER.

     The ‘#:select’, ‘#:prefix’, and ‘#:renamer’ clauses are optional.
     If all are omitted, the returned interface has no bindings.  If the
     ‘#:select’ clause is omitted, PREFIX and RENAMER operate on the
     used module’s public interface.

     In addition to the above, SPEC can also include a ‘#:version’
     clause, of the form:

           #:version VERSION-SPEC

     where VERSION-SPEC is an R6RS-compatible version reference.  An
     error will be signaled in the case in which a module with the same
     name has already been loaded, if that module specifies a version
     and that version is not compatible with VERSION-SPEC.  *Note R6RS
     Version References::, for more on version references.

     If the module name is not resolvable, ‘use-modules’ will signal an
     error.

 -- syntax: @ module-name binding-name
     Refer to the binding named BINDING-NAME in module MODULE-NAME.  The
     binding must have been exported by the module.

 -- syntax: @@ module-name binding-name
     Refer to the binding named BINDING-NAME in module MODULE-NAME.  The
     binding must not have been exported by the module.  This syntax is
     only intended for debugging purposes or as a last resort.


File: guile.info,  Node: Creating Guile Modules,  Next: Modules and the File System,  Prev: Using Guile Modules,  Up: Modules

6.20.3 Creating Guile Modules
-----------------------------

When you want to create your own modules, you have to take the following
steps:

   • Create a Scheme source file and add all variables and procedures
     you wish to export, or which are required by the exported
     procedures.

   • Add a ‘define-module’ form at the beginning.

   • Export all bindings which should be in the public interface, either
     by using ‘define-public’ or ‘export’ (both documented below).

 -- syntax: define-module module-name option ...
     MODULE-NAME is a list of one or more symbols.

          (define-module (ice-9 popen))

     ‘define-module’ makes this module available to Guile programs under
     the given MODULE-NAME.

     OPTION ... are keyword/value pairs which specify more about the
     defined module.  The recognized options and their meaning are shown
     in the following table.

     ‘#:use-module INTERFACE-SPECIFICATION’
          Equivalent to a ‘(use-modules INTERFACE-SPECIFICATION)’ (*note
          Using Guile Modules::).

     ‘#:autoload MODULE SYMBOL-LIST’
          Load MODULE when any of SYMBOL-LIST are accessed.  For
          example,

               (define-module (my mod)
                 #:autoload (srfi srfi-1) (partition delete-duplicates))
               ...
               (if something
                   (set! foo (delete-duplicates ...)))

          When a module is autoloaded, all its bindings become
          available.  SYMBOL-LIST is just those that will first trigger
          the load.

          An autoload is a good way to put off loading a big module
          until it’s really needed, for instance for faster startup or
          if it will only be needed in certain circumstances.

          ‘@’ can do a similar thing (*note Using Guile Modules::), but
          in that case an ‘@’ form must be written every time a binding
          from the module is used.

     ‘#:export LIST’
          Export all identifiers in LIST which must be a list of symbols
          or pairs of symbols.  This is equivalent to ‘(export LIST)’ in
          the module body.

     ‘#:re-export LIST’
          Re-export all identifiers in LIST which must be a list of
          symbols or pairs of symbols.  The symbols in LIST must be
          imported by the current module from other modules.  This is
          equivalent to ‘re-export’ below.

     ‘#:replace LIST’
          Export all identifiers in LIST (a list of symbols or pairs of
          symbols) and mark them as “replacing bindings”.  In the module
          user’s name space, this will have the effect of replacing any
          binding with the same name that is not also “replacing”.
          Normally a replacement results in an “override” warning
          message, ‘#:replace’ avoids that.

          In general, a module that exports a binding for which the
          ‘(guile)’ module already has a definition should use
          ‘#:replace’ instead of ‘#:export’.  ‘#:replace’, in a sense,
          lets Guile know that the module _purposefully_ replaces a core
          binding.  It is important to note, however, that this binding
          replacement is confined to the name space of the module user.
          In other words, the value of the core binding in question
          remains unchanged for other modules.

          Note that although it is often a good idea for the replaced
          binding to remain compatible with a binding in ‘(guile)’, to
          avoid surprising the user, sometimes the bindings will be
          incompatible.  For example, SRFI-19 exports its own version of
          ‘current-time’ (*note SRFI-19 Time::) which is not compatible
          with the core ‘current-time’ function (*note Time::).  Guile
          assumes that a user importing a module knows what she is
          doing, and uses ‘#:replace’ for this binding rather than
          ‘#:export’.

          A ‘#:replace’ clause is equivalent to ‘(export! LIST)’ in the
          module body.

          The ‘#:duplicates’ (see below) provides fine-grain control
          about duplicate binding handling on the module-user side.

     ‘#:version LIST’
          Specify a version for the module in the form of LIST, a list
          of zero or more exact, nonnegative integers.  The
          corresponding ‘#:version’ option in the ‘use-modules’ form
          allows callers to restrict the value of this option in various
          ways.

     ‘#:duplicates LIST’
          Tell Guile to handle duplicate bindings for the bindings
          imported by the current module according to the policy defined
          by LIST, a list of symbols.  LIST must contain symbols
          representing a duplicate binding handling policy chosen among
          the following:

          ‘check’
               Raises an error when a binding is imported from more than
               one place.
          ‘warn’
               Issue a warning when a binding is imported from more than
               one place and leave the responsibility of actually
               handling the duplication to the next duplicate binding
               handler.
          ‘replace’
               When a new binding is imported that has the same name as
               a previously imported binding, then do the following:

                 1. If the old binding was said to be “replacing” (via
                    the ‘#:replace’ option above) and the new binding is
                    not replacing, the keep the old binding.
                 2. If the old binding was not said to be replacing and
                    the new binding is replacing, then replace the old
                    binding with the new one.
                 3. If neither the old nor the new binding is replacing,
                    then keep the old one.

          ‘warn-override-core’
               Issue a warning when a core binding is being overwritten
               and actually override the core binding with the new one.
          ‘first’
               In case of duplicate bindings, the firstly imported
               binding is always the one which is kept.
          ‘last’
               In case of duplicate bindings, the lastly imported
               binding is always the one which is kept.
          ‘noop’
               In case of duplicate bindings, leave the responsibility
               to the next duplicate handler.

          If LIST contains more than one symbol, then the duplicate
          binding handlers which appear first will be used first when
          resolving a duplicate binding situation.  As mentioned above,
          some resolution policies may explicitly leave the
          responsibility of handling the duplication to the next handler
          in LIST.

          If GOOPS has been loaded before the ‘#:duplicates’ clause is
          processed, there are additional strategies available for
          dealing with generic functions.  *Note Merging Generics::, for
          more information.

          The default duplicate binding resolution policy is given by
          the ‘default-duplicate-binding-handler’ procedure, and is

               (replace warn-override-core warn last)

     ‘#:pure’
          Create a “pure” module, that is a module which does not
          contain any of the standard procedure bindings except for the
          syntax forms.  This is useful if you want to create “safe”
          modules, that is modules which do not know anything about
          dangerous procedures.

 -- syntax: export variable ...
     Add all VARIABLEs (which must be symbols or pairs of symbols) to
     the list of exported bindings of the current module.  If VARIABLE
     is a pair, its ‘car’ gives the name of the variable as seen by the
     current module and its ‘cdr’ specifies a name for the binding in
     the current module’s public interface.

 -- syntax: define-public ...
     Equivalent to ‘(begin (define foo ...) (export foo))’.

 -- syntax: re-export variable ...
     Add all VARIABLEs (which must be symbols or pairs of symbols) to
     the list of re-exported bindings of the current module.  Pairs of
     symbols are handled as in ‘export’.  Re-exported bindings must be
     imported by the current module from some other module.

 -- syntax: export! variable ...
     Like ‘export’, but marking the exported variables as replacing.
     Using a module with replacing bindings will cause any existing
     bindings to be replaced without issuing any warnings.  See the
     discussion of ‘#:replace’ above.


File: guile.info,  Node: Modules and the File System,  Next: R6RS Version References,  Prev: Creating Guile Modules,  Up: Modules

6.20.4 Modules and the File System
----------------------------------

Typical programs only use a small subset of modules installed on a Guile
system.  In order to keep startup time down, Guile only loads modules
when a program uses them, on demand.

   When a program evaluates ‘(use-modules (ice-9 popen))’, and the
module is not loaded, Guile searches for a conventionally-named file
from in the “load path”.

   In this case, loading ‘(ice-9 popen)’ will eventually cause Guile to
run ‘(primitive-load-path "ice-9/popen")’.  ‘primitive-load-path’ will
search for a file ‘ice-9/popen’ in the ‘%load-path’ (*note Load
Paths::).  For each directory in ‘%load-path’, Guile will try to find
the file name, concatenated with the extensions from ‘%load-extensions’.
By default, this will cause Guile to ‘stat’ ‘ice-9/popen.scm’, and then
‘ice-9/popen’.  *Note Load Paths::, for more on ‘primitive-load-path’.

   If a corresponding compiled ‘.go’ file is found in the
‘%load-compiled-path’ or in the fallback path, and is as fresh as the
source file, it will be loaded instead of the source file.  If no
compiled file is found, Guile may try to compile the source file and
cache away the resulting ‘.go’ file.  *Note Compilation::, for more on
compilation.

   Once Guile finds a suitable source or compiled file is found, the
file will be loaded.  If, after loading the file, the module under
consideration is still not defined, Guile will signal an error.

   For more information on where and how to install Scheme modules,
*Note Installing Site Packages::.


File: guile.info,  Node: R6RS Version References,  Next: R6RS Libraries,  Prev: Modules and the File System,  Up: Modules

6.20.5 R6RS Version References
------------------------------

Guile’s module system includes support for locating modules based on a
declared version specifier of the same form as the one described in R6RS
(*note R6RS Library Form: (r6rs)Library form.).  By using the
‘#:version’ keyword in a ‘define-module’ form, a module may specify a
version as a list of zero or more exact, nonnegative integers.

   This version can then be used to locate the module during the module
search process.  Client modules and callers of the ‘use-modules’
function may specify constraints on the versions of target modules by
providing a “version reference”, which has one of the following forms:

      (SUB-VERSION-REFERENCE ...)
      (and VERSION-REFERENCE ...)
      (or VERSION-REFERENCE ...)
      (not VERSION-REFERENCE)

   in which SUB-VERSION-REFERENCE is in turn one of:

      (SUB-VERSION)
      (>= SUB-VERSION)
      (<= SUB-VERSION)
      (and SUB-VERSION-REFERENCE ...)
      (or SUB-VERSION-REFERENCE ...)
      (not SUB-VERSION-REFERENCE)

   in which SUB-VERSION is an exact, nonnegative integer as above.  A
version reference matches a declared module version if each element of
the version reference matches a corresponding element of the module
version, according to the following rules:

   • The ‘and’ sub-form matches a version or version element if every
     element in the tail of the sub-form matches the specified version
     or version element.

   • The ‘or’ sub-form matches a version or version element if any
     element in the tail of the sub-form matches the specified version
     or version element.

   • The ‘not’ sub-form matches a version or version element if the tail
     of the sub-form does not match the version or version element.

   • The ‘>=’ sub-form matches a version element if the element is
     greater than or equal to the SUB-VERSION in the tail of the
     sub-form.

   • The ‘<=’ sub-form matches a version element if the version is less
     than or equal to the SUB-VERSION in the tail of the sub-form.

   • A SUB-VERSION matches a version element if one is EQV? to the
     other.

   For example, a module declared as:

      (define-module (mylib mymodule) #:version (1 2 0))

   would be successfully loaded by any of the following ‘use-modules’
expressions:

      (use-modules ((mylib mymodule) #:version (1 2 (>= 0))))
      (use-modules ((mylib mymodule) #:version (or (1 2 0) (1 2 1))))
      (use-modules ((mylib mymodule) #:version ((and (>= 1) (not 2)) 2 0)))


File: guile.info,  Node: R6RS Libraries,  Next: Variables,  Prev: R6RS Version References,  Up: Modules

6.20.6 R6RS Libraries
---------------------

In addition to the API described in the previous sections, you also have
the option to create modules using the portable ‘library’ form described
in R6RS (*note R6RS Library Form: (r6rs)Library form.), and to import
libraries created in this format by other programmers.  Guile’s R6RS
library implementation takes advantage of the flexibility built into the
module system by expanding the R6RS library form into a corresponding
Guile ‘define-module’ form that specifies equivalent import and export
requirements and includes the same body expressions.  The library
expression:

       (library (mylib (1 2))
         (export mybinding)
         (import (otherlib (3))))

   is equivalent to the module definition:

       (define-module (mylib)
         #:version (1 2)
         #:use-module ((otherlib) #:version (3))
         #:export (mybinding))

   Central to the mechanics of R6RS libraries is the concept of import
and export “levels”, which control the visibility of bindings at various
phases of a library’s lifecycle — macros necessary to expand forms in
the library’s body need to be available at expand time; variables used
in the body of a procedure exported by the library must be available at
runtime.  R6RS specifies the optional ‘for’ sub-form of an _import set_
specification (see below) as a mechanism by which a library author can
indicate that a particular library import should take place at a
particular phase with respect to the lifecycle of the importing library.

   Guile’s library implementation uses a technique called “implicit
phasing” (first described by Abdulaziz Ghuloum and R. Kent Dybvig),
which allows the expander and compiler to automatically determine the
necessary visibility of a binding imported from another library.  As
such, the ‘for’ sub-form described below is ignored by Guile (but may be
required by Schemes in which phasing is explicit).

 -- Scheme Syntax: library name (export export-spec ...) (import
          import-spec ...) body ...
     Defines a new library with the specified name, exports, and
     imports, and evaluates the specified body expressions in this
     library’s environment.

     The library NAME is a non-empty list of identifiers, optionally
     ending with a version specification of the form described above
     (*note Creating Guile Modules::).

     Each EXPORT-SPEC is the name of a variable defined or imported by
     the library, or must take the form ‘(rename (internal-name
     external-name) ...)’, where the identifier INTERNAL-NAME names a
     variable defined or imported by the library and EXTERNAL-NAME is
     the name by which the variable is seen by importing libraries.

     Each IMPORT-SPEC must be either an “import set” (see below) or must
     be of the form ‘(for import-set import-level ...)’, where each
     IMPORT-LEVEL is one of:

            run
            expand
            (meta LEVEL)

     where LEVEL is an integer.  Note that since Guile does not require
     explicit phase specification, any IMPORT-SETs found inside of ‘for’
     sub-forms will be “unwrapped” during expansion and processed as if
     they had been specified directly.

     Import sets in turn take one of the following forms:

            LIBRARY-REFERENCE
            (library LIBRARY-REFERENCE)
            (only IMPORT-SET IDENTIFIER ...)
            (except IMPORT-SET IDENTIFIER ...)
            (prefix IMPORT-SET IDENTIFIER)
            (rename IMPORT-SET (INTERNAL-IDENTIFIER EXTERNAL-IDENTIFIER) ...)

     where LIBRARY-REFERENCE is a non-empty list of identifiers ending
     with an optional version reference (*note R6RS Version
     References::), and the other sub-forms have the following
     semantics, defined recursively on nested IMPORT-SETs:

        • The ‘library’ sub-form is used to specify libraries for import
          whose names begin with the identifier “library.”

        • The ‘only’ sub-form imports only the specified IDENTIFIERs
          from the given IMPORT-SET.

        • The ‘except’ sub-form imports all of the bindings exported by
          IMPORT-SET except for those that appear in the specified list
          of IDENTIFIERs.

        • The ‘prefix’ sub-form imports all of the bindings exported by
          IMPORT-SET, first prefixing them with the specified
          IDENTIFIER.

        • The ‘rename’ sub-form imports all of the identifiers exported
          by IMPORT-SET.  The binding for each INTERNAL-IDENTIFIER among
          these identifiers is made visible to the importing library as
          the corresponding EXTERNAL-IDENTIFIER; all other bindings are
          imported using the names provided by IMPORT-SET.

     Note that because Guile translates R6RS libraries into module
     definitions, an import specification may be used to declare a
     dependency on a native Guile module — although doing so may make
     your libraries less portable to other Schemes.

 -- Scheme Syntax: import import-spec ...
     Import into the current environment the libraries specified by the
     given import specifications, where each IMPORT-SPEC takes the same
     form as in the ‘library’ form described above.


File: guile.info,  Node: Variables,  Next: Module System Reflection,  Prev: R6RS Libraries,  Up: Modules

6.20.7 Variables
----------------

Each module has its own hash table, sometimes known as an “obarray”,
that maps the names defined in that module to their corresponding
variable objects.

   A variable is a box-like object that can hold any Scheme value.  It
is said to be “undefined” if its box holds a special Scheme value that
denotes undefined-ness (which is different from all other Scheme values,
including for example ‘#f’); otherwise the variable is “defined”.

   On its own, a variable object is anonymous.  A variable is said to be
“bound” when it is associated with a name in some way, usually a symbol
in a module obarray.  When this happens, the name is said to be bound to
the variable, in that module.

   (That’s the theory, anyway.  In practice, defined-ness and bound-ness
sometimes get confused, because Lisp and Scheme implementations have
often conflated — or deliberately drawn no distinction between — a name
that is unbound and a name that is bound to a variable whose value is
undefined.  We will try to be clear about the difference and explain any
confusion where it is unavoidable.)

   Variables do not have a read syntax.  Most commonly they are created
and bound implicitly by ‘define’ expressions: a top-level ‘define’
expression of the form

     (define NAME VALUE)

creates a variable with initial value VALUE and binds it to the name
NAME in the current module.  But they can also be created dynamically by
calling one of the constructor procedures ‘make-variable’ and
‘make-undefined-variable’.

 -- Scheme Procedure: make-undefined-variable
 -- C Function: scm_make_undefined_variable ()
     Return a variable that is initially unbound.

 -- Scheme Procedure: make-variable init
 -- C Function: scm_make_variable (init)
     Return a variable initialized to value INIT.

 -- Scheme Procedure: variable-bound? var
 -- C Function: scm_variable_bound_p (var)
     Return ‘#t’ if VAR is bound to a value, or ‘#f’ otherwise.  Throws
     an error if VAR is not a variable object.

 -- Scheme Procedure: variable-ref var
 -- C Function: scm_variable_ref (var)
     Dereference VAR and return its value.  VAR must be a variable
     object; see ‘make-variable’ and ‘make-undefined-variable’.

 -- Scheme Procedure: variable-set! var val
 -- C Function: scm_variable_set_x (var, val)
     Set the value of the variable VAR to VAL.  VAR must be a variable
     object, VAL can be any value.  Return an unspecified value.

 -- Scheme Procedure: variable-unset! var
 -- C Function: scm_variable_unset_x (var)
     Unset the value of the variable VAR, leaving VAR unbound.

 -- Scheme Procedure: variable? obj
 -- C Function: scm_variable_p (obj)
     Return ‘#t’ if OBJ is a variable object, else return ‘#f’.


File: guile.info,  Node: Module System Reflection,  Next: Accessing Modules from C,  Prev: Variables,  Up: Modules

6.20.8 Module System Reflection
-------------------------------

The previous sections have described a declarative view of the module
system.  You can also work with it programmatically by accessing and
modifying various parts of the Scheme objects that Guile uses to
implement the module system.

   At any time, there is a “current module”.  This module is the one
where a top-level ‘define’ and similar syntax will add new bindings.
You can find other module objects with ‘resolve-module’, for example.

   These module objects can be used as the second argument to ‘eval’.

 -- Scheme Procedure: current-module
 -- C Function: scm_current_module ()
     Return the current module object.

 -- Scheme Procedure: set-current-module module
 -- C Function: scm_set_current_module (module)
     Set the current module to MODULE and return the previous current
     module.

 -- Scheme Procedure: save-module-excursion thunk
     Call THUNK within a ‘dynamic-wind’ such that the module that is
     current at invocation time is restored when THUNK’s dynamic extent
     is left (*note Dynamic Wind::).

     More precisely, if THUNK escapes non-locally, the current module
     (at the time of escape) is saved, and the original current module
     (at the time THUNK’s dynamic extent was last entered) is restored.
     If THUNK’s dynamic extent is re-entered, then the current module is
     saved, and the previously saved inner module is set current again.

 -- Scheme Procedure: resolve-module name [autoload=#t] [version=#f]
          [#:ensure=#t]
 -- C Function: scm_resolve_module (name)
     Find the module named NAME and return it.  When it has not already
     been defined and AUTOLOAD is true, try to auto-load it.  When it
     can’t be found that way either, create an empty module if ENSURE is
     true, otherwise return ‘#f’.  If VERSION is true, ensure that the
     resulting module is compatible with the given version reference
     (*note R6RS Version References::).  The name is a list of symbols.

 -- Scheme Procedure: resolve-interface name [#:select=#f] [#:hide='()]
          [#:prefix=#f] [#:renamer=#f] [#:version=#f]
     Find the module named NAME as with ‘resolve-module’ and return its
     interface.  The interface of a module is also a module object, but
     it contains only the exported bindings.

 -- Scheme Procedure: module-uses module
     Return a list of the interfaces used by MODULE.

 -- Scheme Procedure: module-use! module interface
     Add INTERFACE to the front of the use-list of MODULE.  Both
     arguments should be module objects, and INTERFACE should very
     likely be a module returned by ‘resolve-interface’.

 -- Scheme Procedure: reload-module module
     Revisit the source file that corresponds to MODULE.  Raises an
     error if no source file is associated with the given module.

   As mentioned in the previous section, modules contain a mapping
between identifiers (as symbols) and storage locations (as variables).
Guile defines a number of procedures to allow access to this mapping.
If you are programming in C, *note Accessing Modules from C::.

 -- Scheme Procedure: module-variable module name
     Return the variable bound to NAME (a symbol) in MODULE, or ‘#f’ if
     NAME is unbound.

 -- Scheme Procedure: module-add! module name var
     Define a new binding between NAME (a symbol) and VAR (a variable)
     in MODULE.

 -- Scheme Procedure: module-ref module name
     Look up the value bound to NAME in MODULE.  Like ‘module-variable’,
     but also does a ‘variable-ref’ on the resulting variable, raising
     an error if NAME is unbound.

 -- Scheme Procedure: module-define! module name value
     Locally bind NAME to VALUE in MODULE.  If NAME was already locally
     bound in MODULE, i.e., defined locally and not by an imported
     module, the value stored in the existing variable will be updated.
     Otherwise, a new variable will be added to the module, via
     ‘module-add!’.

 -- Scheme Procedure: module-set! module name value
     Update the binding of NAME in MODULE to VALUE, raising an error if
     NAME is not already bound in MODULE.

   There are many other reflective procedures available in the default
environment.  If you find yourself using one of them, please contact the
Guile developers so that we can commit to stability for that interface.


File: guile.info,  Node: Accessing Modules from C,  Next: provide and require,  Prev: Module System Reflection,  Up: Modules

6.20.9 Accessing Modules from C
-------------------------------

The last sections have described how modules are used in Scheme code,
which is the recommended way of creating and accessing modules.  You can
also work with modules from C, but it is more cumbersome.

   The following procedures are available.

 -- C Function: SCM scm_c_call_with_current_module (SCM MODULE, SCM
          (*FUNC)(void *), void *DATA)
     Call FUNC and make MODULE the current module during the call.  The
     argument DATA is passed to FUNC.  The return value of
     ‘scm_c_call_with_current_module’ is the return value of FUNC.

 -- C Function: SCM scm_public_variable (SCM MODULE_NAME, SCM NAME)
 -- C Function: SCM scm_c_public_variable (const char *MODULE_NAME,
          const char *NAME)
     Find a the variable bound to the symbol NAME in the public
     interface of the module named MODULE_NAME.

     MODULE_NAME should be a list of symbols, when represented as a
     Scheme object, or a space-separated string, in the ‘const char *’
     case.  See ‘scm_c_define_module’ below, for more examples.

     Signals an error if no module was found with the given name.  If
     NAME is not bound in the module, just returns ‘#f’.

 -- C Function: SCM scm_private_variable (SCM MODULE_NAME, SCM NAME)
 -- C Function: SCM scm_c_private_variable (const char *MODULE_NAME,
          const char *NAME)
     Like ‘scm_public_variable’, but looks in the internals of the
     module named MODULE_NAME instead of the public interface.
     Logically, these procedures should only be called on modules you
     write.

 -- C Function: SCM scm_public_lookup (SCM MODULE_NAME, SCM NAME)
 -- C Function: SCM scm_c_public_lookup (const char *MODULE_NAME, const
          char *NAME)
 -- C Function: SCM scm_private_lookup (SCM MODULE_NAME, SCM NAME)
 -- C Function: SCM scm_c_private_lookup (const char *MODULE_NAME, const
          char *NAME)
     Like ‘scm_public_variable’ or ‘scm_private_variable’, but if the
     NAME is not bound in the module, signals an error.  Returns a
     variable, always.

          static SCM eval_string_var;

          /* NOTE: It is important that the call to 'my_init'
             happens-before all calls to 'my_eval_string'. */
          void my_init (void)
          {
            eval_string_var = scm_c_public_lookup ("ice-9 eval-string",
                                                   "eval-string");
          }

          SCM my_eval_string (SCM str)
          {
            return scm_call_1 (scm_variable_ref (eval_string_var), str);
          }

 -- C Function: SCM scm_public_ref (SCM MODULE_NAME, SCM NAME)
 -- C Function: SCM scm_c_public_ref (const char *MODULE_NAME, const
          char *NAME)
 -- C Function: SCM scm_private_ref (SCM MODULE_NAME, SCM NAME)
 -- C Function: SCM scm_c_private_ref (const char *MODULE_NAME, const
          char *NAME)
     Like ‘scm_public_lookup’ or ‘scm_private_lookup’, but additionally
     dereferences the variable.  If the variable object is unbound,
     signals an error.  Returns the value bound to NAME in MODULE_NAME.

   In addition, there are a number of other lookup-related procedures.
We suggest that you use the ‘scm_public_’ and ‘scm_private_’ family of
procedures instead, if possible.

 -- C Function: SCM scm_c_lookup (const char *NAME)
     Return the variable bound to the symbol indicated by NAME in the
     current module.  If there is no such binding or the symbol is not
     bound to a variable, signal an error.

 -- C Function: SCM scm_lookup (SCM NAME)
     Like ‘scm_c_lookup’, but the symbol is specified directly.

 -- C Function: SCM scm_c_module_lookup (SCM MODULE, const char *NAME)
 -- C Function: SCM scm_module_lookup (SCM MODULE, SCM NAME)
     Like ‘scm_c_lookup’ and ‘scm_lookup’, but the specified module is
     used instead of the current one.

 -- C Function: SCM scm_module_variable (SCM MODULE, SCM NAME)
     Like ‘scm_module_lookup’, but if the binding does not exist, just
     returns ‘#f’ instead of raising an error.

   To define a value, use ‘scm_define’:

 -- C Function: SCM scm_c_define (const char *NAME, SCM VAL)
     Bind the symbol indicated by NAME to a variable in the current
     module and set that variable to VAL.  When NAME is already bound to
     a variable, use that.  Else create a new variable.

 -- C Function: SCM scm_define (SCM NAME, SCM VAL)
     Like ‘scm_c_define’, but the symbol is specified directly.

 -- C Function: SCM scm_c_module_define (SCM MODULE, const char *NAME,
          SCM VAL)
 -- C Function: SCM scm_module_define (SCM MODULE, SCM NAME, SCM VAL)
     Like ‘scm_c_define’ and ‘scm_define’, but the specified module is
     used instead of the current one.

   In some rare cases, you may need to access the variable that
‘scm_module_define’ would have accessed, without changing the binding of
the existing variable, if one is present.  In that case, use
‘scm_module_ensure_local_variable’:

 -- C Function: SCM scm_module_ensure_local_variable (SCM MODULE, SCM
          SYM)
     Like ‘scm_module_define’, but if the SYM is already locally bound
     in that module, the variable’s existing binding is not reset.
     Returns a variable.

 -- C Function: SCM scm_module_reverse_lookup (SCM MODULE, SCM VARIABLE)
     Find the symbol that is bound to VARIABLE in MODULE.  When no such
     binding is found, return ‘#f’.

 -- C Function: SCM scm_c_define_module (const char *NAME, void
          (*INIT)(void *), void *DATA)
     Define a new module named NAME and make it current while INIT is
     called, passing it DATA.  Return the module.

     The parameter NAME is a string with the symbols that make up the
     module name, separated by spaces.  For example, ‘"foo bar"’ names
     the module ‘(foo bar)’.

     When there already exists a module named NAME, it is used
     unchanged, otherwise, an empty module is created.

 -- C Function: SCM scm_c_resolve_module (const char *NAME)
     Find the module name NAME and return it.  When it has not already
     been defined, try to auto-load it.  When it can’t be found that way
     either, create an empty module.  The name is interpreted as for
     ‘scm_c_define_module’.

 -- C Function: SCM scm_c_use_module (const char *NAME)
     Add the module named NAME to the uses list of the current module,
     as with ‘(use-modules NAME)’.  The name is interpreted as for
     ‘scm_c_define_module’.

 -- C Function: void scm_c_export (const char *NAME, ...)
     Add the bindings designated by NAME, ...  to the public interface
     of the current module.  The list of names is terminated by ‘NULL’.


File: guile.info,  Node: provide and require,  Next: Environments,  Prev: Accessing Modules from C,  Up: Modules

6.20.10 provide and require
---------------------------

Aubrey Jaffer, mostly to support his portable Scheme library SLIB,
implemented a provide/require mechanism for many Scheme implementations.
Library files in SLIB _provide_ a feature, and when user programs
_require_ that feature, the library file is loaded in.

   For example, the file ‘random.scm’ in the SLIB package contains the
line

     (provide 'random)

   so to use its procedures, a user would type

     (require 'random)

   and they would magically become available, _but still have the same
names!_  So this method is nice, but not as good as a full-featured
module system.

   When SLIB is used with Guile, provide and require can be used to
access its facilities.


File: guile.info,  Node: Environments,  Prev: provide and require,  Up: Modules

6.20.11 Environments
--------------------

Scheme, as defined in R5RS, does _not_ have a full module system.
However it does define the concept of a top-level “environment”.  Such
an environment maps identifiers (symbols) to Scheme objects such as
procedures and lists: *note About Closure::.  In other words, it
implements a set of “bindings”.

   Environments in R5RS can be passed as the second argument to ‘eval’
(*note Fly Evaluation::).  Three procedures are defined to return
environments: ‘scheme-report-environment’, ‘null-environment’ and
‘interaction-environment’ (*note Fly Evaluation::).

   In addition, in Guile any module can be used as an R5RS environment,
i.e., passed as the second argument to ‘eval’.

   Note: the following two procedures are available only when the
‘(ice-9 r5rs)’ module is loaded:

     (use-modules (ice-9 r5rs))

 -- Scheme Procedure: scheme-report-environment version
 -- Scheme Procedure: null-environment version
     VERSION must be the exact integer ‘5’, corresponding to revision 5
     of the Scheme report (the Revised^5 Report on Scheme).
     ‘scheme-report-environment’ returns a specifier for an environment
     that is empty except for all bindings defined in the report that
     are either required or both optional and supported by the
     implementation.  ‘null-environment’ returns a specifier for an
     environment that is empty except for the (syntactic) bindings for
     all syntactic keywords defined in the report that are either
     required or both optional and supported by the implementation.

     Currently Guile does not support values of VERSION for other
     revisions of the report.

     The effect of assigning (through the use of ‘eval’) a variable
     bound in a ‘scheme-report-environment’ (for example ‘car’) is
     unspecified.  Currently the environments specified by
     ‘scheme-report-environment’ are not immutable in Guile.


File: guile.info,  Node: Foreign Function Interface,  Next: Scheduling,  Prev: Modules,  Up: API Reference

6.21 Foreign Function Interface
===============================

The more one hacks in Scheme, the more one realizes that there are
actually two computational worlds: one which is warm and alive, that
land of parentheses, and one cold and dead, the land of C and its ilk.

   But yet we as programmers live in both worlds, and Guile itself is
half implemented in C. So it is that Guile’s living half pays respect to
its dead counterpart, via a spectrum of interfaces to C ranging from
dynamic loading of Scheme primitives to dynamic binding of stock C
library procedures.

* Menu:

* Foreign Libraries::           Dynamically linking to libraries.
* Foreign Functions::           Simple calls to C procedures.
* C Extensions::                Extending Guile in C with loadable modules.
* Modules and Extensions::      Loading C extensions into modules.
* Foreign Pointers::            Accessing global variables.
* Dynamic FFI::                 Calling arbitrary C functions.


File: guile.info,  Node: Foreign Libraries,  Next: Foreign Functions,  Up: Foreign Function Interface

6.21.1 Foreign Libraries
------------------------

Most modern Unices have something called “shared libraries”.  This
ordinarily means that they have the capability to share the executable
image of a library between several running programs to save memory and
disk space.  But generally, shared libraries give a lot of additional
flexibility compared to the traditional static libraries.  In fact,
calling them ‘dynamic’ libraries is as correct as calling them ‘shared’.

   Shared libraries really give you a lot of flexibility in addition to
the memory and disk space savings.  When you link a program against a
shared library, that library is not closely incorporated into the final
executable.  Instead, the executable of your program only contains
enough information to find the needed shared libraries when the program
is actually run.  Only then, when the program is starting, is the final
step of the linking process performed.  This means that you need not
recompile all programs when you install a new, only slightly modified
version of a shared library.  The programs will pick up the changes
automatically the next time they are run.

   Now, when all the necessary machinery is there to perform part of the
linking at run-time, why not take the next step and allow the programmer
to explicitly take advantage of it from within their program?  Of
course, many operating systems that support shared libraries do just
that, and chances are that Guile will allow you to access this feature
from within your Scheme programs.  As you might have guessed already,
this feature is called “dynamic linking”.(1)

   We titled this section “foreign libraries” because although the name
“foreign” doesn’t leak into the API, the world of C really is foreign to
Scheme – and that estrangement extends to components of foreign
libraries as well, as we see in future sections.

 -- Scheme Procedure: dynamic-link [library]
 -- C Function: scm_dynamic_link (library)
     Find the shared library denoted by LIBRARY (a string) and link it
     into the running Guile application.  When everything works out,
     return a Scheme object suitable for representing the linked object
     file.  Otherwise an error is thrown.  How object files are searched
     is system dependent.

     Normally, LIBRARY is just the name of some shared library file that
     will be searched for in the places where shared libraries usually
     reside, such as in ‘/usr/lib’ and ‘/usr/local/lib’.

     LIBRARY should not contain an extension such as ‘.so’.  The correct
     file name extension for the host operating system is provided
     automatically, according to libltdl’s rules (*note lt_dlopenext:
     (libtool)Libltdl interface.).

     When LIBRARY is omitted, a “global symbol handle” is returned.
     This handle provides access to the symbols available to the program
     at run-time, including those exported by the program itself and the
     shared libraries already loaded.

     Note that on hosts that use dynamic-link libraries (DLLs), the
     global symbol handle may not be able to provide access to symbols
     from recursively-loaded DLLs.  Only exported symbols from those
     DLLs directly loaded by the program may be available.

 -- Scheme Procedure: dynamic-object? obj
 -- C Function: scm_dynamic_object_p (obj)
     Return ‘#t’ if OBJ is a dynamic library handle, or ‘#f’ otherwise.

 -- Scheme Procedure: dynamic-unlink dobj
 -- C Function: scm_dynamic_unlink (dobj)
     Unlink the indicated object file from the application.  The
     argument DOBJ must have been obtained by a call to ‘dynamic-link’.
     After ‘dynamic-unlink’ has been called on DOBJ, its content is no
     longer accessible.

     (define libgl-obj (dynamic-link "libGL"))
     libgl-obj
     ⇒ #<dynamic-object "libGL">
     (dynamic-unlink libGL-obj)
     libGL-obj
     ⇒ #<dynamic-object "libGL" (unlinked)>

   As you can see, after calling ‘dynamic-unlink’ on a dynamically
linked library, it is marked as ‘(unlinked)’ and you are no longer able
to use it with ‘dynamic-call’, etc.  Whether the library is really
removed from you program is system-dependent and will generally not
happen when some other parts of your program still use it.

   When dynamic linking is disabled or not supported on your system, the
above functions throw errors, but they are still available.

   ---------- Footnotes ----------

   (1) Some people also refer to the final linking stage at program
startup as ‘dynamic linking’, so if you want to make yourself perfectly
clear, it is probably best to use the more technical term “dlopening”,
as suggested by Gordon Matzigkeit in his libtool documentation.


File: guile.info,  Node: Foreign Functions,  Next: C Extensions,  Prev: Foreign Libraries,  Up: Foreign Function Interface

6.21.2 Foreign Functions
------------------------

The most natural thing to do with a dynamic library is to grovel around
in it for a function pointer: a “foreign function”.  ‘dynamic-func’
exists for that purpose.

 -- Scheme Procedure: dynamic-func name dobj
 -- C Function: scm_dynamic_func (name, dobj)
     Return a “handle” for the func NAME in the shared object referred
     to by DOBJ.  The handle can be passed to ‘dynamic-call’ to actually
     call the function.

     Regardless whether your C compiler prepends an underscore ‘_’ to
     the global names in a program, you should *not* include this
     underscore in NAME since it will be added automatically when
     necessary.

   Guile has static support for calling functions with no arguments,
‘dynamic-call’.

 -- Scheme Procedure: dynamic-call func dobj
 -- C Function: scm_dynamic_call (func, dobj)
     Call the C function indicated by FUNC and DOBJ.  The function is
     passed no arguments and its return value is ignored.  When FUNCTION
     is something returned by ‘dynamic-func’, call that function and
     ignore DOBJ.  When FUNC is a string , look it up in DYNOBJ; this is
     equivalent to
          (dynamic-call (dynamic-func FUNC DOBJ) #f)

   ‘dynamic-call’ is not very powerful.  It is mostly intended to be
used for calling specially written initialization functions that will
then add new primitives to Guile.  For example, we do not expect that
you will dynamically link ‘libX11’ with ‘dynamic-link’ and then
construct a beautiful graphical user interface just by using
‘dynamic-call’.  Instead, the usual way would be to write a special
Guile-to-X11 glue library that has intimate knowledge about both Guile
and X11 and does whatever is necessary to make them inter-operate
smoothly.  This glue library could then be dynamically linked into a
vanilla Guile interpreter and activated by calling its initialization
function.  That function would add all the new types and primitives to
the Guile interpreter that it has to offer.

   (There is actually another, better option: simply to create a
‘libX11’ wrapper in Scheme via the dynamic FFI. *Note Dynamic FFI::, for
more information.)

   Given some set of C extensions to Guile, the next logical step is to
integrate these glue libraries into the module system of Guile so that
you can load new primitives into a running system just as you can load
new Scheme code.

 -- Scheme Procedure: load-extension lib init
 -- C Function: scm_load_extension (lib, init)
     Load and initialize the extension designated by LIB and INIT. When
     there is no pre-registered function for LIB/INIT, this is
     equivalent to

          (dynamic-call INIT (dynamic-link LIB))

     When there is a pre-registered function, that function is called
     instead.

     Normally, there is no pre-registered function.  This option exists
     only for situations where dynamic linking is unavailable or
     unwanted.  In that case, you would statically link your program
     with the desired library, and register its init function right
     after Guile has been initialized.

     As for ‘dynamic-link’, LIB should not contain any suffix such as
     ‘.so’ (*note dynamic-link: Foreign Libraries.).  It should also not
     contain any directory components.  Libraries that implement Guile
     Extensions should be put into the normal locations for shared
     libraries.  We recommend to use the naming convention
     ‘libguile-bla-blum’ for a extension related to a module ‘(bla
     blum)’.

     The normal way for a extension to be used is to write a small
     Scheme file that defines a module, and to load the extension into
     this module.  When the module is auto-loaded, the extension is
     loaded as well.  For example,

          (define-module (bla blum))

          (load-extension "libguile-bla-blum" "bla_init_blum")


File: guile.info,  Node: C Extensions,  Next: Modules and Extensions,  Prev: Foreign Functions,  Up: Foreign Function Interface

6.21.3 C Extensions
-------------------

The most interesting application of dynamically linked libraries is
probably to use them for providing _compiled code modules_ to Scheme
programs.  As much fun as programming in Scheme is, every now and then
comes the need to write some low-level C stuff to make Scheme even more
fun.

   Not only can you put these new primitives into their own module (see
the previous section), you can even put them into a shared library that
is only then linked to your running Guile image when it is actually
needed.

   An example will hopefully make everything clear.  Suppose we want to
make the Bessel functions of the C library available to Scheme in the
module ‘(math bessel)’.  First we need to write the appropriate glue
code to convert the arguments and return values of the functions from
Scheme to C and back.  Additionally, we need a function that will add
them to the set of Guile primitives.  Because this is just an example,
we will only implement this for the ‘j0’ function.

     #include <math.h>
     #include <libguile.h>

     SCM
     j0_wrapper (SCM x)
     {
       return scm_from_double (j0 (scm_to_double (x, "j0")));
     }

     void
     init_math_bessel ()
     {
       scm_c_define_gsubr ("j0", 1, 0, 0, j0_wrapper);
     }

   We can already try to bring this into action by manually calling the
low level functions for performing dynamic linking.  The C source file
needs to be compiled into a shared library.  Here is how to do it on
GNU/Linux, please refer to the ‘libtool’ documentation for how to create
dynamically linkable libraries portably.

     gcc -shared -o libbessel.so -fPIC bessel.c

   Now fire up Guile:

     (define bessel-lib (dynamic-link "./libbessel.so"))
     (dynamic-call "init_math_bessel" bessel-lib)
     (j0 2)
     ⇒ 0.223890779141236

   The filename ‘./libbessel.so’ should be pointing to the shared
library produced with the ‘gcc’ command above, of course.  The second
line of the Guile interaction will call the ‘init_math_bessel’ function
which in turn will register the C function ‘j0_wrapper’ with the Guile
interpreter under the name ‘j0’.  This function becomes immediately
available and we can call it from Scheme.

   Fun, isn’t it?  But we are only half way there.  This is what
‘apropos’ has to say about ‘j0’:

     (apropos "j0")
     ⊣ (guile-user): j0     #<primitive-procedure j0>

   As you can see, ‘j0’ is contained in the root module, where all the
other Guile primitives like ‘display’, etc live.  In general, a
primitive is put into whatever module is the “current module” at the
time ‘scm_c_define_gsubr’ is called.

   A compiled module should have a specially named “module init
function”.  Guile knows about this special name and will call that
function automatically after having linked in the shared library.  For
our example, we replace ‘init_math_bessel’ with the following code in
‘bessel.c’:

     void
     init_math_bessel (void *unused)
     {
       scm_c_define_gsubr ("j0", 1, 0, 0, j0_wrapper);
       scm_c_export ("j0", NULL);
     }

     void
     scm_init_math_bessel_module ()
     {
       scm_c_define_module ("math bessel", init_math_bessel, NULL);
     }

   The general pattern for the name of a module init function is:
‘scm_init_’, followed by the name of the module where the individual
hierarchical components are concatenated with underscores, followed by
‘_module’.

   After ‘libbessel.so’ has been rebuilt, we need to place the shared
library into the right place.

   Once the module has been correctly installed, it should be possible
to use it like this:

     guile> (load-extension "./libbessel.so" "scm_init_math_bessel_module")
     guile> (use-modules (math bessel))
     guile> (j0 2)
     0.223890779141236
     guile> (apropos "j0")
     ⊣ (math bessel): j0      #<primitive-procedure j0>

   That’s it!


File: guile.info,  Node: Modules and Extensions,  Next: Foreign Pointers,  Prev: C Extensions,  Up: Foreign Function Interface

6.21.4 Modules and Extensions
-----------------------------

The new primitives that you add to Guile with ‘scm_c_define_gsubr’
(*note Primitive Procedures::) or with any of the other mechanisms are
placed into the module that is current when the ‘scm_c_define_gsubr’ is
executed.  Extensions loaded from the REPL, for example, will be placed
into the ‘(guile-user)’ module, if the REPL module was not changed.

   To define C primitives within a specific module, the simplest way is:

     (define-module (foo bar))
     (load-extension "foobar-c-code" "foo_bar_init")

   When loaded with ‘(use-modules (foo bar))’, the ‘load-extension’ call
looks for the ‘foobar-c-code.so’ (etc) object file in Guile’s
‘extensiondir’, which is usually a subdirectory of the ‘libdir’.  For
example, if your libdir is ‘/usr/lib’, the ‘extensiondir’ for the Guile
2.2.X series will be ‘/usr/lib/guile/2.2/’.

   The extension path includes the major and minor version of Guile (the
“effective version”), because Guile guarantees compatibility within a
given effective version.  This allows you to install different versions
of the same extension for different versions of Guile.

   If the extension is not found in the ‘extensiondir’, Guile will also
search the standard system locations, such as ‘/usr/lib’ or
‘/usr/local/lib’.  It is preferable, however, to keep your extension out
of the system library path, to prevent unintended interference with
other dynamically-linked C libraries.

   If someone installs your module to a non-standard location then the
object file won’t be found.  You can address this by inserting the
install location in the ‘foo/bar.scm’ file.  This is convenient for the
user and also guarantees the intended object is read, even if stray
older or newer versions are in the loader’s path.

   The usual way to specify an install location is with a ‘prefix’ at
the configure stage, for instance ‘./configure prefix=/opt’ results in
library files as say ‘/opt/lib/foobar-c-code.so’.  When using Autoconf
(*note Introduction: (autoconf)Top.), the library location is in a
‘libdir’ variable.  Its value is intended to be expanded by ‘make’, and
can by substituted into a source file like ‘foo.scm.in’

     (define-module (foo bar))
     (load-extension "XXextensiondirXX/foobar-c-code" "foo_bar_init")

with the following in a ‘Makefile’, using ‘sed’ (*note Introduction:
(sed)Top.),

     foo.scm: foo.scm.in
             sed 's|XXextensiondirXX|$(libdir)/guile/2.2|' <foo.scm.in >foo.scm

   The actual pattern ‘XXextensiondirXX’ is arbitrary, it’s only
something which doesn’t otherwise occur.  If several modules need the
value, it can be easier to create one ‘foo/config.scm’ with a define of
the ‘extensiondir’ location, and use that as required.

     (define-module (foo config))
     (define-public foo-config-extensiondir "XXextensiondirXX"")

   Such a file might have other locations too, for instance a data
directory for auxiliary files, or ‘localedir’ if the module has its own
‘gettext’ message catalogue (*note Internationalization::).

   It will be noted all of the above requires that the Scheme code to be
found in ‘%load-path’ (*note Load Paths::).  Presently it’s left up to
the system administrator or each user to augment that path when
installing Guile modules in non-default locations.  But having reached
the Scheme code, that code should take care of hitting any of its own
private files etc.


File: guile.info,  Node: Foreign Pointers,  Next: Dynamic FFI,  Prev: Modules and Extensions,  Up: Foreign Function Interface

6.21.5 Foreign Pointers
-----------------------

The previous sections have shown how Guile can be extended at runtime by
loading compiled C extensions.  This approach is all well and good, but
wouldn’t it be nice if we didn’t have to write any C at all?  This
section takes up the problem of accessing C values from Scheme, and the
next discusses C functions.

* Menu:

* Foreign Types::                  Expressing C types in Scheme.
* Foreign Variables::              Pointers to C symbols.
* Void Pointers and Byte Access::  Pointers into the ether.
* Foreign Structs::                Packing and unpacking structs.


File: guile.info,  Node: Foreign Types,  Next: Foreign Variables,  Up: Foreign Pointers

6.21.5.1 Foreign Types
......................

The first impedance mismatch that one sees between C and Scheme is that
in C, the storage locations (variables) are typed, but in Scheme types
are associated with values, not variables.  *Note Values and
Variables::.

   So when describing a C function or a C structure so that it can be
accessed from Scheme, the data types of the parameters or fields must be
passed explicitly.

   These “C type values” may be constructed using the constants and
procedures from the ‘(system foreign)’ module, which may be loaded like
this:

     (use-modules (system foreign))

   ‘(system foreign)’ exports a number of values expressing the basic C
types:

 -- Scheme Variable: int8
 -- Scheme Variable: uint8
 -- Scheme Variable: uint16
 -- Scheme Variable: int16
 -- Scheme Variable: uint32
 -- Scheme Variable: int32
 -- Scheme Variable: uint64
 -- Scheme Variable: int64
 -- Scheme Variable: float
 -- Scheme Variable: double
     These values represent the C numeric types of the specified sizes
     and signednesses.

   In addition there are some convenience bindings for indicating types
of platform-dependent size:

 -- Scheme Variable: int
 -- Scheme Variable: unsigned-int
 -- Scheme Variable: long
 -- Scheme Variable: unsigned-long
 -- Scheme Variable: short
 -- Scheme Variable: unsigned-short
 -- Scheme Variable: size_t
 -- Scheme Variable: ssize_t
 -- Scheme Variable: ptrdiff_t
 -- Scheme Variable: intptr_t
 -- Scheme Variable: uintptr_t
     Values exported by the ‘(system foreign)’ module, representing C
     numeric types.  For example, ‘long’ may be ‘equal?’ to ‘int64’ on a
     64-bit platform.

 -- Scheme Variable: void
     The ‘void’ type.  It can be used as the first argument to
     ‘pointer->procedure’ to wrap a C function that returns nothing.

   In addition, the symbol ‘*’ is used by convention to denote pointer
types.  Procedures detailed in the following sections, such as
‘pointer->procedure’, accept it as a type descriptor.


File: guile.info,  Node: Foreign Variables,  Next: Void Pointers and Byte Access,  Prev: Foreign Types,  Up: Foreign Pointers

6.21.5.2 Foreign Variables
..........................

Pointers to variables in the current address space may be looked up
dynamically using ‘dynamic-pointer’.

 -- Scheme Procedure: dynamic-pointer name dobj
 -- C Function: scm_dynamic_pointer (name, dobj)
     Return a “wrapped pointer” for the symbol NAME in the shared object
     referred to by DOBJ.  The returned pointer points to a C object.

     Regardless whether your C compiler prepends an underscore ‘_’ to
     the global names in a program, you should *not* include this
     underscore in NAME since it will be added automatically when
     necessary.

   For example, currently Guile has a variable, ‘scm_numptob’, as part
of its API. It is declared as a C ‘long’.  So, to create a handle
pointing to that foreign value, we do:

     (use-modules (system foreign))
     (define numptob (dynamic-pointer "scm_numptob" (dynamic-link)))
     numptob
     ⇒ #<pointer 0x7fb35b1b4688>

   (The next section discusses ways to dereference pointers.)

   A value returned by ‘dynamic-pointer’ is a Scheme wrapper for a C
pointer.

 -- Scheme Procedure: pointer-address pointer
 -- C Function: scm_pointer_address (pointer)
     Return the numerical value of POINTER.

          (pointer-address numptob)
          ⇒ 139984413364296 ; YMMV

 -- Scheme Procedure: make-pointer address [finalizer]
     Return a foreign pointer object pointing to ADDRESS.  If FINALIZER
     is passed, it should be a pointer to a one-argument C function that
     will be called when the pointer object becomes unreachable.

 -- Scheme Procedure: pointer? obj
     Return ‘#t’ if OBJ is a pointer object, ‘#f’ otherwise.

 -- Scheme Variable: %null-pointer
     A foreign pointer whose value is 0.

 -- Scheme Procedure: null-pointer? pointer
     Return ‘#t’ if POINTER is the null pointer, ‘#f’ otherwise.

   For the purpose of passing SCM values directly to foreign functions,
and allowing them to return SCM values, Guile also supports some unsafe
casting operators.

 -- Scheme Procedure: scm->pointer scm
     Return a foreign pointer object with the ‘object-address’ of SCM.

 -- Scheme Procedure: pointer->scm pointer
     Unsafely cast POINTER to a Scheme object.  Cross your fingers!

   Sometimes you want to give C extensions access to the dynamic FFI. At
that point, the names get confusing, because “pointer” can refer to a
‘SCM’ object that wraps a pointer, or to a ‘void*’ value.  We will try
to use “pointer object” to refer to Scheme objects, and “pointer value”
to refer to ‘void *’ values.

 -- C Function: SCM scm_from_pointer (void *ptr, void (*finalizer)
          (void*))
     Create a pointer object from a pointer value.

     If FINALIZER is non-null, Guile arranges to call it on the pointer
     value at some point after the pointer object becomes collectable.

 -- C Function: void* scm_to_pointer (SCM obj)
     Unpack the pointer value from a pointer object.


File: guile.info,  Node: Void Pointers and Byte Access,  Next: Foreign Structs,  Prev: Foreign Variables,  Up: Foreign Pointers

6.21.5.3 Void Pointers and Byte Access
......................................

Wrapped pointers are untyped, so they are essentially equivalent to C
‘void’ pointers.  As in C, the memory region pointed to by a pointer can
be accessed at the byte level.  This is achieved using _bytevectors_
(*note Bytevectors::).  The ‘(rnrs bytevectors)’ module contains
procedures that can be used to convert byte sequences to Scheme objects
such as strings, floating point numbers, or integers.

 -- Scheme Procedure: pointer->bytevector pointer len [offset
          [uvec_type]]
 -- C Function: scm_pointer_to_bytevector (pointer, len, offset,
          uvec_type)
     Return a bytevector aliasing the LEN bytes pointed to by POINTER.

     The user may specify an alternate default interpretation for the
     memory by passing the UVEC_TYPE argument, to indicate that the
     memory is an array of elements of that type.  UVEC_TYPE should be
     something that ‘array-type’ would return, like ‘f32’ or ‘s16’.

     When OFFSET is passed, it specifies the offset in bytes relative to
     POINTER of the memory region aliased by the returned bytevector.

     Mutating the returned bytevector mutates the memory pointed to by
     POINTER, so buckle your seatbelts.

 -- Scheme Procedure: bytevector->pointer bv [offset]
 -- C Function: scm_bytevector_to_pointer (bv, offset)
     Return a pointer pointer aliasing the memory pointed to by BV or
     OFFSET bytes after BV when OFFSET is passed.

   In addition to these primitives, convenience procedures are
available:

 -- Scheme Procedure: dereference-pointer pointer
     Assuming POINTER points to a memory region that holds a pointer,
     return this pointer.

 -- Scheme Procedure: string->pointer string [encoding]
     Return a foreign pointer to a nul-terminated copy of STRING in the
     given ENCODING, defaulting to the current locale encoding.  The C
     string is freed when the returned foreign pointer becomes
     unreachable.

     This is the Scheme equivalent of ‘scm_to_stringn’.

 -- Scheme Procedure: pointer->string pointer [length] [encoding]
     Return the string representing the C string pointed to by POINTER.
     If LENGTH is omitted or ‘-1’, the string is assumed to be
     nul-terminated.  Otherwise LENGTH is the number of bytes in memory
     pointed to by POINTER.  The C string is assumed to be in the given
     ENCODING, defaulting to the current locale encoding.

     This is the Scheme equivalent of ‘scm_from_stringn’.

   Most object-oriented C libraries use pointers to specific data
structures to identify objects.  It is useful in such cases to reify the
different pointer types as disjoint Scheme types.  The
‘define-wrapped-pointer-type’ macro simplifies this.

 -- Scheme Syntax: define-wrapped-pointer-type type-name pred wrap
          unwrap print
     Define helper procedures to wrap pointer objects into Scheme
     objects with a disjoint type.  Specifically, this macro defines:

        • PRED, a predicate for the new Scheme type;
        • WRAP, a procedure that takes a pointer object and returns an
          object that satisfies PRED;
        • UNWRAP, which does the reverse.

     WRAP preserves pointer identity, for two pointer objects P1 and P2
     that are ‘equal?’, ‘(eq? (WRAP P1) (WRAP P2)) ⇒ #t’.

     Finally, PRINT should name a user-defined procedure to print such
     objects.  The procedure is passed the wrapped object and a port to
     write to.

     For example, assume we are wrapping a C library that defines a
     type, ‘bottle_t’, and functions that can be passed ‘bottle_t *’
     pointers to manipulate them.  We could write:

          (define-wrapped-pointer-type bottle
            bottle?
            wrap-bottle unwrap-bottle
            (lambda (b p)
              (format p "#<bottle of ~a ~x>"
                      (bottle-contents b)
                      (pointer-address (unwrap-bottle b)))))

          (define grab-bottle
            ;; Wrapper for `bottle_t *grab (void)'.
            (let ((grab (pointer->procedure '*
                                            (dynamic-func "grab_bottle" libbottle)
                                            '())))
              (lambda ()
                "Return a new bottle."
                (wrap-bottle (grab)))))

          (define bottle-contents
            ;; Wrapper for `const char *bottle_contents (bottle_t *)'.
            (let ((contents (pointer->procedure '*
                                                (dynamic-func "bottle_contents"
                                                               libbottle)
                                                '(*))))
              (lambda (b)
                "Return the contents of B."
                (pointer->string (contents (unwrap-bottle b))))))

          (write (grab-bottle))
          ⇒ #<bottle of Château Haut-Brion 803d36>

     In this example, ‘grab-bottle’ is guaranteed to return a genuine
     ‘bottle’ object satisfying ‘bottle?’.  Likewise, ‘bottle-contents’
     errors out when its argument is not a genuine ‘bottle’ object.

   Going back to the ‘scm_numptob’ example above, here is how we can
read its value as a C ‘long’ integer:

     (use-modules (rnrs bytevectors))

     (bytevector-uint-ref (pointer->bytevector numptob (sizeof long))
                          0 (native-endianness)
                          (sizeof long))
     ⇒ 8

   If we wanted to corrupt Guile’s internal state, we could set
‘scm_numptob’ to another value; but we shouldn’t, because that variable
is not meant to be set.  Indeed this point applies more widely: the C
API is a dangerous place to be.  Not only might setting a value crash
your program, simply accessing the data pointed to by a dangling pointer
or similar can prove equally disastrous.


File: guile.info,  Node: Foreign Structs,  Prev: Void Pointers and Byte Access,  Up: Foreign Pointers

6.21.5.4 Foreign Structs
........................

Finally, one last note on foreign values before moving on to actually
calling foreign functions.  Sometimes you need to deal with C structs,
which requires interpreting each element of the struct according to the
its type, offset, and alignment.  Guile has some primitives to support
this.

 -- Scheme Procedure: sizeof type
 -- C Function: scm_sizeof (type)
     Return the size of TYPE, in bytes.

     TYPE should be a valid C type, like ‘int’.  Alternately TYPE may be
     the symbol ‘*’, in which case the size of a pointer is returned.
     TYPE may also be a list of types, in which case the size of a
     ‘struct’ with ABI-conventional packing is returned.

 -- Scheme Procedure: alignof type
 -- C Function: scm_alignof (type)
     Return the alignment of TYPE, in bytes.

     TYPE should be a valid C type, like ‘int’.  Alternately TYPE may be
     the symbol ‘*’, in which case the alignment of a pointer is
     returned.  TYPE may also be a list of types, in which case the
     alignment of a ‘struct’ with ABI-conventional packing is returned.

   Guile also provides some convenience methods to pack and unpack
foreign pointers wrapping C structs.

 -- Scheme Procedure: make-c-struct types vals
     Create a foreign pointer to a C struct containing VALS with types
     ‘types’.

     VALS and ‘types’ should be lists of the same length.

 -- Scheme Procedure: parse-c-struct foreign types
     Parse a foreign pointer to a C struct, returning a list of values.

     ‘types’ should be a list of C types.

   For example, to create and parse the equivalent of a ‘struct {
int64_t a; uint8_t b; }’:

     (parse-c-struct (make-c-struct (list int64 uint8)
                                    (list 300 43))
                     (list int64 uint8))
     ⇒ (300 43)

   As yet, Guile only has convenience routines to support
conventionally-packed structs.  But given the ‘bytevector->pointer’ and
‘pointer->bytevector’ routines, one can create and parse tightly packed
structs and unions by hand.  See the code for ‘(system foreign)’ for
details.


File: guile.info,  Node: Dynamic FFI,  Prev: Foreign Pointers,  Up: Foreign Function Interface

6.21.6 Dynamic FFI
------------------

Of course, the land of C is not all nouns and no verbs: there are
functions too, and Guile allows you to call them.

 -- Scheme Procedure: pointer->procedure return_type func_ptr arg_types
          [#:return-errno?=#f]
 -- C Function: scm_pointer_to_procedure (return_type, func_ptr,
          arg_types)
 -- C Function: scm_pointer_to_procedure_with_errno (return_type,
          func_ptr, arg_types)

     Make a foreign function.

     Given the foreign void pointer FUNC_PTR, its argument and return
     types ARG_TYPES and RETURN_TYPE, return a procedure that will pass
     arguments to the foreign function and return appropriate values.

     ARG_TYPES should be a list of foreign types.  ‘return_type’ should
     be a foreign type.  *Note Foreign Types::, for more information on
     foreign types.

     If RETURN-ERRNO? is true, or when calling
     ‘scm_pointer_to_procedure_with_errno’, the returned procedure will
     return two values, with ‘errno’ as the second value.

   Here is a better definition of ‘(math bessel)’:

     (define-module (math bessel)
       #:use-module (system foreign)
       #:export (j0))

     (define libm (dynamic-link "libm"))

     (define j0
       (pointer->procedure double
                           (dynamic-func "j0" libm)
                           (list double)))

   That’s it!  No C at all.

   Numeric arguments and return values from foreign functions are
represented as Scheme values.  For example, ‘j0’ in the above example
takes a Scheme number as its argument, and returns a Scheme number.

   Pointers may be passed to and returned from foreign functions as
well.  In that case the type of the argument or return value should be
the symbol ‘*’, indicating a pointer.  For example, the following code
makes ‘memcpy’ available to Scheme:

     (define memcpy
       (let ((this (dynamic-link)))
         (pointer->procedure '*
                             (dynamic-func "memcpy" this)
                             (list '* '* size_t))))

   To invoke ‘memcpy’, one must pass it foreign pointers:

     (use-modules (rnrs bytevectors))

     (define src-bits
       (u8-list->bytevector '(0 1 2 3 4 5 6 7)))
     (define src
       (bytevector->pointer src-bits))
     (define dest
       (bytevector->pointer (make-bytevector 16 0)))

     (memcpy dest src (bytevector-length src-bits))

     (bytevector->u8-list (pointer->bytevector dest 16))
     ⇒ (0 1 2 3 4 5 6 7 0 0 0 0 0 0 0 0)

   One may also pass structs as values, passing structs as foreign
pointers.  *Note Foreign Structs::, for more information on how to
express struct types and struct values.

   “Out” arguments are passed as foreign pointers.  The memory pointed
to by the foreign pointer is mutated in place.

     ;; struct timeval {
     ;;      time_t      tv_sec;     /* seconds */
     ;;      suseconds_t tv_usec;    /* microseconds */
     ;; };
     ;; assuming fields are of type "long"

     (define gettimeofday
       (let ((f (pointer->procedure
                 int
                 (dynamic-func "gettimeofday" (dynamic-link))
                 (list '* '*)))
             (tv-type (list long long)))
         (lambda ()
           (let* ((timeval (make-c-struct tv-type (list 0 0)))
                  (ret (f timeval %null-pointer)))
             (if (zero? ret)
                 (apply values (parse-c-struct timeval tv-type))
                 (error "gettimeofday returned an error" ret))))))

     (gettimeofday)
     ⇒ 1270587589
     ⇒ 499553

   As you can see, this interface to foreign functions is at a very low,
somewhat dangerous level(1).

   The FFI can also work in the opposite direction: making Scheme
procedures callable from C. This makes it possible to use Scheme
procedures as “callbacks” expected by C function.

 -- Scheme Procedure: procedure->pointer return-type proc arg-types
 -- C Function: scm_procedure_to_pointer (return_type, proc, arg_types)
     Return a pointer to a C function of type RETURN-TYPE taking
     arguments of types ARG-TYPES (a list) and behaving as a proxy to
     procedure PROC.  Thus PROC’s arity, supported argument types, and
     return type should match RETURN-TYPE and ARG-TYPES.

   As an example, here’s how the C library’s ‘qsort’ array sorting
function can be made accessible to Scheme (*note ‘qsort’: (libc)Array
Sort Function.):

     (define qsort!
       (let ((qsort (pointer->procedure void
                                        (dynamic-func "qsort"
                                                      (dynamic-link))
                                        (list '* size_t size_t '*))))
         (lambda (bv compare)
           ;; Sort bytevector BV in-place according to comparison
           ;; procedure COMPARE.
           (let ((ptr (procedure->pointer int
                                          (lambda (x y)
                                            ;; X and Y are pointers so,
                                            ;; for convenience, dereference
                                            ;; them before calling COMPARE.
                                            (compare (dereference-uint8* x)
                                                     (dereference-uint8* y)))
                                          (list '* '*))))
             (qsort (bytevector->pointer bv)
                    (bytevector-length bv) 1 ;; we're sorting bytes
                    ptr)))))

     (define (dereference-uint8* ptr)
       ;; Helper function: dereference the byte pointed to by PTR.
       (let ((b (pointer->bytevector ptr 1)))
         (bytevector-u8-ref b 0)))

     (define bv
       ;; An unsorted array of bytes.
       (u8-list->bytevector '(7 1 127 3 5 4 77 2 9 0)))

     ;; Sort BV.
     (qsort! bv (lambda (x y) (- x y)))

     ;; Let's see what the sorted array looks like:
     (bytevector->u8-list bv)
     ⇒ (0 1 2 3 4 5 7 9 77 127)

   And voilà!

   Note that ‘procedure->pointer’ is not supported (and not defined) on
a few exotic architectures.  Thus, user code may need to check
‘(defined? 'procedure->pointer)’.  Nevertheless, it is available on many
architectures, including (as of libffi 3.0.9) x86, ia64, SPARC, PowerPC,
ARM, and MIPS, to name a few.

   ---------- Footnotes ----------

   (1) A contribution to Guile in the form of a high-level FFI would be
most welcome.


File: guile.info,  Node: Scheduling,  Next: Options and Config,  Prev: Foreign Function Interface,  Up: API Reference

6.22 Threads, Mutexes, Asyncs and Dynamic Roots
===============================================

* Menu:

* Threads::                     Multiple threads of execution.
* Thread Local Variables::      Some fluids are thread-local.
* Asyncs::                      Asynchronous interrupts.
* Atomics::                     Atomic references.
* Mutexes and Condition Variables:: Synchronization primitives.
* Blocking::                    How to block properly in guile mode.
* Futures::                     Fine-grain parallelism.
* Parallel Forms::              Parallel execution of forms.


File: guile.info,  Node: Threads,  Next: Thread Local Variables,  Up: Scheduling

6.22.1 Threads
--------------

Guile supports POSIX threads, unless it was configured with
‘--without-threads’ or the host lacks POSIX thread support.  When thread
support is available, the ‘threads’ feature is provided (*note
‘provided?’: Feature Manipulation.).

   The procedures below manipulate Guile threads, which are wrappers
around the system’s POSIX threads.  For application-level parallelism,
using higher-level constructs, such as futures, is recommended (*note
Futures::).

   To use these facilities, load the ‘(ice-9 threads)’ module.

     (use-modules (ice-9 threads))

 -- Scheme Procedure: all-threads
 -- C Function: scm_all_threads ()
     Return a list of all threads.

 -- Scheme Procedure: current-thread
 -- C Function: scm_current_thread ()
     Return the thread that called this function.

 -- Scheme Procedure: call-with-new-thread thunk [handler]
     Call ‘thunk’ in a new thread and with a new dynamic state,
     returning the new thread.  The procedure THUNK is called via
     ‘with-continuation-barrier’.

     When HANDLER is specified, then THUNK is called from within a
     ‘catch’ with tag ‘#t’ that has HANDLER as its handler.  This catch
     is established inside the continuation barrier.

     Once THUNK or HANDLER returns, the return value is made the _exit
     value_ of the thread and the thread is terminated.

 -- C Function: SCM scm_spawn_thread (scm_t_catch_body body, void
          *body_data, scm_t_catch_handler handler, void *handler_data)
     Call BODY in a new thread, passing it BODY_DATA, returning the new
     thread.  The function BODY is called via
     ‘scm_c_with_continuation_barrier’.

     When HANDLER is non-‘NULL’, BODY is called via ‘scm_internal_catch’
     with tag ‘SCM_BOOL_T’ that has HANDLER and HANDLER_DATA as the
     handler and its data.  This catch is established inside the
     continuation barrier.

     Once BODY or HANDLER returns, the return value is made the _exit
     value_ of the thread and the thread is terminated.

 -- Scheme Procedure: thread? obj
 -- C Function: scm_thread_p (obj)
     Return ‘#t’ ff OBJ is a thread; otherwise, return ‘#f’.

 -- Scheme Procedure: join-thread thread [timeout [timeoutval]]
 -- C Function: scm_join_thread (thread)
 -- C Function: scm_join_thread_timed (thread, timeout, timeoutval)
     Wait for THREAD to terminate and return its exit value.  Only
     threads that were created with ‘call-with-new-thread’ or
     ‘scm_spawn_thread’ can be joinable; attempting to join a foreign
     thread will raise an error.

     When TIMEOUT is given, it specifies a point in time where the
     waiting should be aborted.  It can be either an integer as returned
     by ‘current-time’ or a pair as returned by ‘gettimeofday’.  When
     the waiting is aborted, TIMEOUTVAL is returned (if it is specified;
     ‘#f’ is returned otherwise).

 -- Scheme Procedure: thread-exited? thread
 -- C Function: scm_thread_exited_p (thread)
     Return ‘#t’ if THREAD has exited, or ‘#f’ otherwise.

 -- Scheme Procedure: yield
 -- C Function: scm_yield (thread)
     If one or more threads are waiting to execute, calling yield forces
     an immediate context switch to one of them.  Otherwise, yield has
     no effect.

 -- Scheme Procedure: cancel-thread thread . values
 -- C Function: scm_cancel_thread (thread)
     Asynchronously interrupt THREAD and ask it to terminate.
     ‘dynamic-wind’ post thunks will run, but throw handlers will not.
     If THREAD has already terminated or been signaled to terminate,
     this function is a no-op.  Calling ‘join-thread’ on the thread will
     return the given VALUES, if the cancel succeeded.

     Under the hood, thread cancellation uses ‘system-async-mark’ and
     ‘abort-to-prompt’.  *Note Asyncs:: for more on asynchronous
     interrupts.

 -- macro: make-thread proc arg ...
     Apply PROC to ARG ... in a new thread formed by
     ‘call-with-new-thread’ using a default error handler that display
     the error to the current error port.  The ARG ... expressions are
     evaluated in the new thread.

 -- macro: begin-thread expr1 expr2 ...
     Evaluate forms EXPR1 EXPR2 ... in a new thread formed by
     ‘call-with-new-thread’ using a default error handler that display
     the error to the current error port.

   One often wants to limit the number of threads running to be
proportional to the number of available processors.  These interfaces
are therefore exported by (ice-9 threads) as well.

 -- Scheme Procedure: total-processor-count
 -- C Function: scm_total_processor_count ()
     Return the total number of processors of the machine, which is
     guaranteed to be at least 1.  A “processor” here is a thread
     execution unit, which can be either:

        • an execution core in a (possibly multi-core) chip, in a
          (possibly multi- chip) module, in a single computer, or
        • a thread execution unit inside a core in the case of
          “hyper-threaded” CPUs.

     Which of the two definitions is used, is unspecified.

 -- Scheme Procedure: current-processor-count
 -- C Function: scm_current_processor_count ()
     Like ‘total-processor-count’, but return the number of processors
     available to the current process.  See ‘setaffinity’ and
     ‘getaffinity’ for more information.


File: guile.info,  Node: Thread Local Variables,  Next: Asyncs,  Prev: Threads,  Up: Scheduling

6.22.2 Thread-Local Variables
-----------------------------

Sometimes you want to establish a variable binding that is only valid
for a given thread: a “thread-local variable”.

   You would think that fluids or parameters would be Guile’s answer for
thread-local variables, since establishing a new fluid binding doesn’t
affect bindings in other threads.  *Note Fluids and Dynamic States::, or
*Note Parameters::.  However, new threads inherit the fluid bindings
that were in place in their creator threads.  In this way, a binding
established using a fluid (or a parameter) in a thread can escape to
other threads, which might not be what you want.  Or, it might escape
via explicit reification via ‘current-dynamic-state’.

   Of course, this dynamic scoping might be exactly what you want;
that’s why fluids and parameters work this way, and is what you want for
for many common parameters such as the current input and output ports,
the current locale conversion parameters, and the like.  Perhaps this is
the case for most parameters, even.  If your use case for thread-local
bindings comes from a desire to isolate a binding from its setting in
unrelated threads, then fluids and parameters apply nicely.

   On the other hand, if your use case is to prevent concurrent access
to a value from multiple threads, then using vanilla fluids or
parameters is not appropriate.  For this purpose, Guile has
“thread-local fluids”.  A fluid created with ‘make-thread-local-fluid’
won’t be captured by ‘current-dynamic-state’ and won’t be propagated to
new threads.

 -- Scheme Procedure: make-thread-local-fluid [dflt]
 -- C Function: scm_make_thread_local_fluid (dflt)
     Return a newly created fluid, whose initial value is DFLT, or ‘#f’
     if DFLT is not given.  Unlike fluids made with ‘make-fluid’, thread
     local fluids are not captured by ‘make-dynamic-state’.  Similarly,
     a newly spawned child thread does not inherit thread-local fluid
     values from the parent thread.

 -- Scheme Procedure: fluid-thread-local? fluid
 -- C Function: scm_fluid_thread_local_p (fluid)
     Return ‘#t’ if the fluid FLUID is is thread-local, or ‘#f’
     otherwise.

   For example:

     (define %thread-local (make-thread-local-fluid))

     (with-fluids ((%thread-local (compute-data)))
       ... (fluid-ref %thread-local) ...)

   You can also make a thread-local parameter out of a thread-local
fluid using the normal ‘fluid->parameter’:

     (define param (fluid->parameter (make-thread-local-fluid)))

     (parameterize ((param (compute-data)))
       ... (param) ...)


File: guile.info,  Node: Asyncs,  Next: Atomics,  Prev: Thread Local Variables,  Up: Scheduling

6.22.3 Asynchronous Interrupts
------------------------------

Every Guile thread can be interrupted.  Threads running Guile code will
periodically check if there are pending interrupts and run them if
necessary.  To interrupt a thread, call ‘system-async-mark’ on that
thread.

 -- Scheme Procedure: system-async-mark proc [thread]
 -- C Function: scm_system_async_mark (proc)
 -- C Function: scm_system_async_mark_for_thread (proc, thread)
     Enqueue PROC (a procedure with zero arguments) for future execution
     in THREAD.  When PROC has already been enqueued for THREAD but has
     not been executed yet, this call has no effect.  When THREAD is
     omitted, the thread that called ‘system-async-mark’ is used.

   Note that ‘scm_system_async_mark_for_thread’ is not
“async-signal-safe” and so cannot be called from a C signal handler.
(Indeed in general, ‘libguile’ functions are not safe to call from C
signal handlers.)

   Though an interrupt procedure can have any side effect permitted to
Guile code, asynchronous interrupts are generally used either for
profiling or for prematurely cancelling a computation.  The former case
is mostly transparent to the program being run, by design, but the
latter case can introduce bugs.  Like finalizers (*note Foreign Object
Memory Management::), asynchronous interrupts introduce concurrency in a
program.  An asyncronous interrupt can run in the middle of some
mutex-protected operation, for example, and potentially corrupt the
program’s state.

   If some bit of Guile code needs to temporarily inhibit interrupts, it
can use ‘call-with-blocked-asyncs’.  This function works by temporarily
increasing the _async blocking level_ of the current thread while a
given procedure is running.  The blocking level starts out at zero, and
whenever a safe point is reached, a blocking level greater than zero
will prevent the execution of queued asyncs.

   Analogously, the procedure ‘call-with-unblocked-asyncs’ will
temporarily decrease the blocking level of the current thread.  You can
use it when you want to disable asyncs by default and only allow them
temporarily.

   In addition to the C versions of ‘call-with-blocked-asyncs’ and
‘call-with-unblocked-asyncs’, C code can use ‘scm_dynwind_block_asyncs’
and ‘scm_dynwind_unblock_asyncs’ inside a “dynamic context” (*note
Dynamic Wind::) to block or unblock asyncs temporarily.

 -- Scheme Procedure: call-with-blocked-asyncs proc
 -- C Function: scm_call_with_blocked_asyncs (proc)
     Call PROC and block the execution of asyncs by one level for the
     current thread while it is running.  Return the value returned by
     PROC.  For the first two variants, call PROC with no arguments; for
     the third, call it with DATA.

 -- C Function: void * scm_c_call_with_blocked_asyncs (void * (*proc)
          (void *data), void *data)
     The same but with a C function PROC instead of a Scheme thunk.

 -- Scheme Procedure: call-with-unblocked-asyncs proc
 -- C Function: scm_call_with_unblocked_asyncs (proc)
     Call PROC and unblock the execution of asyncs by one level for the
     current thread while it is running.  Return the value returned by
     PROC.  For the first two variants, call PROC with no arguments; for
     the third, call it with DATA.

 -- C Function: void * scm_c_call_with_unblocked_asyncs (void *(*proc)
          (void *data), void *data)
     The same but with a C function PROC instead of a Scheme thunk.

 -- C Function: void scm_dynwind_block_asyncs ()
     During the current dynwind context, increase the blocking of asyncs
     by one level.  This function must be used inside a pair of calls to
     ‘scm_dynwind_begin’ and ‘scm_dynwind_end’ (*note Dynamic Wind::).

 -- C Function: void scm_dynwind_unblock_asyncs ()
     During the current dynwind context, decrease the blocking of asyncs
     by one level.  This function must be used inside a pair of calls to
     ‘scm_dynwind_begin’ and ‘scm_dynwind_end’ (*note Dynamic Wind::).

   Sometimes you want to interrupt a thread that might be waiting for
something to happen, for example on a file descriptor or a condition
variable.  In that case you can inform Guile of how to interrupt that
wait using the following procedures:

 -- C Function: int scm_c_prepare_to_wait_on_fd (int fd)
     Inform Guile that the current thread is about to sleep, and that if
     an asynchronous interrupt is signalled on this thread, Guile should
     wake up the thread by writing a zero byte to FD.  Returns zero if
     the prepare succeeded, or nonzero if the thread already has a
     pending async and that it should avoid waiting.

 -- C Function: int scm_c_prepare_to_wait_on_cond (scm_i_pthread_mutex_t
          *mutex, scm_i_pthread_cond_t *cond)
     Inform Guile that the current thread is about to sleep, and that if
     an asynchronous interrupt is signalled on this thread, Guile should
     wake up the thread by acquiring MUTEX and signalling COND.  The
     caller must already hold MUTEX and only drop it as part of the
     ‘pthread_cond_wait’ call.  Returns zero if the prepare succeeded,
     or nonzero if the thread already has a pending async and that it
     should avoid waiting.

 -- C Function: void scm_c_wait_finished (void)
     Inform Guile that the current thread has finished waiting, and that
     asynchronous interrupts no longer need any special wakeup action;
     the current thread will periodically poll its internal queue
     instead.

   Guile’s own interface to ‘sleep’, ‘wait-condition-variable’,
‘select’, and so on all call the above routines as appropriate.

   Finally, note that threads can also be interrupted via POSIX signals.
*Note Signals::.  As an implementation detail, signal handlers will
effectively call ‘system-async-mark’ in a signal-safe way, eventually
running the signal handler using the same async mechanism.  In this way
you can temporarily inhibit signal handlers from running using the above
interfaces.

