% This LaTeX document was generated using the LaTeX backend of PlDoc,
% The SWI-Prolog documentation system



\subsection{library(http/http_open): HTTP client library}

\label{sec:httpopen}

\begin{tags}
\mtag{See also}- \predref{load_html}{3} and \predref{xpath}{3} can be used to parse and navigate HTML
documents. \\- \predref{http_get}{3} and \predref{http_post}{4} provide an alternative interface that
convert the reply depending on the \verb$Content-Type$ header.
\end{tags}

This library defines \predref{http_open}{3}, which opens a URL as a Prolog stream.
The functionality of the library can be extended by loading two
additional modules that act as plugins:

\begin{description}
    \termitem{library}{http/http_ssl_plugin}
Loading this library causes \predref{http_open}{3} to handle HTTPS connections.
Relevant options for SSL certificate handling are handed to
\predref{ssl_context}{3}. This plugin is loaded automatically if the scheme
\const{https} is requested using a default SSL context. See the plugin for
additional information regarding security.
    \termitem{library}{http/http_cookie}
Loading this library adds tracking cookies to \predref{http_open}{3}. Returned
cookies are collected in the Prolog database and supplied for
subsequent requests.
\end{description}

Here is a simple example to fetch a web-page:

\begin{code}
?- http_open('http://www.google.com/search?q=prolog', In, []),
   copy_stream_data(In, user_output),
   close(In).
<!doctype html><head><title>prolog - Google Search</title><script>
...
\end{code}

The example below fetches the modification time of a web-page. Note that
Modified is '' (the empty atom) if the web-server does not provide a
time-stamp for the resource. See also \predref{parse_time}{2}.

\begin{code}
modified(URL, Stamp) :-
        http_open(URL, In,
                  [ method(head),
                    header(last_modified, Modified)
                  ]),
        close(In),
        Modified \== '',
        parse_time(Modified, Stamp).
\end{code}

Then next example uses Google search. It exploits \verb$library(uri)$ to manage
URIs, \verb$library(sgml)$ to load an HTML document and \verb$library(xpath)$ to
navigate the parsed HTML. Note that you may need to adjust the XPath
queries if the data returned by Google changes.

\begin{code}
:- use_module(library(http/http_open)).
:- use_module(library(xpath)).
:- use_module(library(sgml)).
:- use_module(library(uri)).

google(For, Title, HREF) :-
        uri_encoded(query_value, For, Encoded),
        atom_concat('http://www.google.com/search?q=', Encoded, URL),
        http_open(URL, In, []),
        call_cleanup(
            load_html(In, DOM, []),
            close(In)),
        xpath(DOM, //h3(@class=r), Result),
        xpath(Result, //a(@href=HREF0, text), Title),
        uri_components(HREF0, Components),
        uri_data(search, Components, Query),
        uri_query_components(Query, Parts),
        memberchk(q=HREF, Parts).
\end{code}

An example query is below:

\begin{code}
?- google(prolog, Title, HREF).
Title = 'SWI-Prolog',
HREF = 'http://www.swi-prolog.org/' ;
Title = 'Prolog - Wikipedia',
HREF = 'https://nl.wikipedia.org/wiki/Prolog' ;
Title = 'Prolog - Wikipedia, the free encyclopedia',
HREF = 'https://en.wikipedia.org/wiki/Prolog' ;
Title = 'Pro-Log is logistiek dienstverlener m.b.t. vervoer over water.',
HREF = 'http://www.pro-log.nl/' ;
Title = 'Learn Prolog Now!',
HREF = 'http://www.learnprolognow.org/' ;
Title = 'Free Online Version - Learn Prolog
...
\end{code}

\vspace{0.7cm}

\begin{description}
    \predicate[det]{http_open}{3}{+URL, -Stream, +Options}
Open the data at the HTTP server as a Prolog stream. \arg{URL} is
either an atom specifying a \arg{URL} or a list representing a
broken-down \arg{URL} as specified below. After this predicate
succeeds the data can be read from \arg{Stream}. After completion this
stream must be closed using the built-in Prolog predicate
\predref{close}{1}. \arg{Options} provides additional options:

\begin{description}
    \termitem{authenticate}{+Boolean}
If \const{false} (default \const{true}), do \textit{not} try to automatically
authenticate the client if a 401 (Unauthorized) status code
is received.
    \termitem{authorization}{+Term}
Send authorization. See also \predref{http_set_authorization}{2}. Supported
schemes:

\begin{description}
    \termitem{basic}{+User, +Password}
HTTP Basic authentication.
    \termitem{bearer}{+Token}
HTTP Bearer authentication.
    \termitem{digest}{+User, +Password}
HTTP Digest authentication. This option is only provided
if the plugin \verb$library(http/http_digest)$ is also loaded.
\end{description}

    \termitem{connection}{+Connection}
Specify the \verb$Connection$ header. Default is \const{close}. The
alternative is \verb$Keep-alive$. This maintains a pool of
available connections as determined by \predref{keep_connection}{1}.
The \verb$library(http/websockets)$ uses \verb$Keep-alive, Upgrade$.
Keep-alive connections can be closed explicitly using
\predref{http_close_keep_alive}{1}. Keep-alive connections may
significantly improve repetitive requests on the same server,
especially if the IP route is long, HTTPS is used or the
connection uses a proxy.
    \termitem{final_url}{-FinalURL}
Unify \arg{FinalURL} with the final destination. This differs from
the original \arg{URL} if the returned head of the original
indicates an HTTP redirect (codes 301, 302 or 303). Without a
redirect, \arg{FinalURL} is the same as \arg{URL} if \arg{URL} is an atom, or a
\arg{URL} constructed from the parts.
    \termitem{header}{Name, -AtomValue}
If provided, \arg{AtomValue} is unified with the value of the
indicated field in the reply header. \arg{Name} is matched
case-insensitive and the underscore (_) matches the hyphen
(-). Multiple of these options may be provided to extract
multiple header fields. If the header is not available
\arg{AtomValue} is unified to the empty atom ('').
    \termitem{headers}{-List}
If provided, \arg{List} is unified with a list of Name(Value) pairs
corresponding to fields in the reply header. Name and Value
follow the same conventions used by the \verb$header(Name,Value)$
option.
    \termitem{method}{+Method}
One of \const{get} (default), \const{head}, \const{delete}, \const{post}, \const{put} or
\const{patch}.
The \const{head} message can be
used in combination with the \verb$header(Name, Value)$ option to
access information on the resource without actually fetching
the resource itself. The returned stream must be closed
immediately.

If \verb$post(Data)$ is provided, the default is \const{post}.
    \termitem{size}{-Size}
\arg{Size} is unified with the integer value of \verb$Content-Length$
in the reply header.
    \termitem{version}{-Version}
\arg{Version} is a \textit{pair} \verb$Major-Minor$, where \arg{Major} and \arg{Minor}
are integers representing the HTTP version in the reply header.
    \termitem{range}{+Range}
Ask for partial content. \arg{Range} is a term \textit{Unit(From,To)},
where \arg{From} is an integer and \arg{To} is either an integer or
the atom \const{end}. HTTP 1.1 only supports Unit = \const{bytes}. E.g.,
to ask for bytes 1000-1999, use the option
\verb$range(bytes(1000,1999))$
    \termitem{redirect}{+Boolean}
If \const{false} (default \const{true}), do \textit{not} automatically redirect
if a 3XX code is received. Must be combined with
\verb$status_code(Code)$ and one of the header options to read the
redirect reply. In particular, without \verb$status_code(Code)$ a
redirect is mapped to an exception.
    \termitem{status_code}{-Code}
If this option is present and \arg{Code} unifies with the HTTP
status code, do \textbf{not} translate errors (4xx, 5xx) into an
exception. Instead, \predref{http_open}{3} behaves as if 2xx (success) is
returned, providing the application to read the error document
from the returned stream.
    \termitem{output}{-Out}
Unify the output stream with \arg{Out} and do not close it. This can
be used to upgrade a connection.
    \termitem{timeout}{+Timeout}
If provided, set a timeout on the stream using \predref{set_stream}{2}.
With this option if no new data arrives within \arg{Timeout} seconds
the stream raises an exception. Default is to wait forever
(\const{infinite}).
    \termitem{post}{+Data}
Issue a \verb$POST$ request on the HTTP server. \arg{Data} is
handed to \predref{http_post_data}{3}.
    \termitem{proxy}{+Host:Port}
Use an HTTP proxy to connect to the outside world. See also
\qpredref{socket}{proxy_for_url}{3}. This option overrules the proxy
specification defined by \qpredref{socket}{proxy_for_url}{3}.
    \termitem{proxy}{+Host, +Port}
Synonym for \verb$proxy(+Host:Port)$. Deprecated.
    \termitem{proxy_authorization}{+Authorization}
Send authorization to the proxy. Otherwise the same as the
\const{authorization} option.
    \termitem{bypass_proxy}{+Boolean}
If \const{true}, bypass proxy hooks. Default is \const{false}.
    \termitem{request_header}{Name=Value}
Additional name-value parts are added in the order of
appearance to the HTTP request header. No interpretation is
done.
    \termitem{max_redirect}{+Max}
Sets the maximum length of a redirection chain. This is needed
for some IRIs that redirect indefinitely to other IRIs without
looping (e.g., redirecting to IRIs with a random element in them).
\arg{Max} must be either a non-negative integer or the atom \const{infinite}.
The default value is \verb$10$.
    \termitem{user_agent}{+Agent}
Defines the value of the \verb$User-Agent$ field of the HTTP
header. Default is \verb$SWI-Prolog$.
\end{description}

The hook \qpredref{http}{open_options}{2} can be used to provide default
options based on the broken-down \arg{URL}. The option
\verb$status_code(-Code)$ is particularly useful to query \textbf{REST}
interfaces that commonly return status codes other than \verb$200$
that need to be be processed by the client code.

\begin{arguments}
\arg{URL} & is either an atom or string (url) or a list of \textit{parts}.

When provided, this list may contain the fields
\const{scheme}, \const{user}, \const{password}, \const{host}, \const{port}, \const{path}
and either \verb$query_string$ (whose argument is an atom)
or \const{search} (whose argument is a list of
\verb$Name(Value)$ or \verb$Name=Value$ compound terms).
Only \const{host} is mandatory. The example below opens the
\arg{URL} \verb$http://www.example.com/my/path?q=Hello%20World&lang=en$.
Note that values must \textbf{not} be quoted because the
library inserts the required quotes.

\begin{code}
http_open([ host('www.example.com'),
            path('/my/path'),
            search([ q='Hello world',
                     lang=en
                   ])
          ])
\end{code}

 \\
\end{arguments}

\begin{tags}
    \tag{throws}
\verb$error(existence_error(url, Id),Context)$ is raised if the
HTTP result code is not in the range 200..299. Context has the
shape \verb$context(Message, status(Code, TextCode))$, where \arg{Code} is the
numeric HTTP code and \arg{TextCode} is the textual description thereof
provided by the server. \arg{Message} may provide additional details or
may be unbound.
    \tag{See also}
\predref{ssl_context}{3} for SSL related options if
\verb$library(http/http_ssl_plugin)$ is loaded.
\end{tags}

\qpredicate[semidet,multifile]{http}{disable_encoding_filter}{1}{+ContentType}Do not use the \verb$Content-encoding$ as \verb$Transfer-encoding$
encoding for specific values of \arg{ContentType}. This predicate is
multifile and can thus be extended by the user.

    \predicate[det]{http_set_authorization}{2}{+URL, +Authorization}
Set user/password to supply with URLs that have \arg{URL} as prefix.
If \arg{Authorization} is the atom \verb$-$, possibly defined
authorization is cleared. For example:

\begin{code}
?- http_set_authorization('http://www.example.com/private/',
                          basic('John', 'Secret'))
\end{code}

\begin{tags}
    \tag{To be done}
Move to a separate module, so \predref{http_get}{3}, etc. can use this
too.
\end{tags}

\qpredicate[semidet,multifile]{iostream}{open_hook}{6}{+Spec, +Mode, -Stream, -Close, +Options0, -Options}Hook implementation that makes \predref{open_any}{5} support \const{http} and
\const{https} URLs for \verb$Mode == read$.

    \predicate[det]{http_close_keep_alive}{1}{+Address}
Close all keep-alive connections matching \arg{Address}. \arg{Address} is of
the form Host:Port. In particular, \verb$http_close_keep_alive(_)$
closes all currently known keep-alive connections.

\qpredicate[nondet,multifile]{http}{open_options}{2}{+Parts, -Options}This hook is used by the HTTP client library to define default
options based on the the broken-down request-URL. The following
example redirects all trafic, except for localhost over a proxy:

\begin{code}
:- multifile
    http:open_options/2.

http:open_options(Parts, Options) :-
    option(host(Host), Parts),
    Host \== localhost,
    Options = [proxy('proxy.local', 3128)].
\end{code}

This hook may return multiple solutions. The returned options
are combined using \predref{merge_options}{3} where earlier solutions
overrule later solutions.

\qpredicate[semidet,multifile]{http}{write_cookies}{3}{+Out, +Parts, +Options}Emit a \verb$Cookie:$ header for the current connection. \arg{Out} is an
open stream to the HTTP server, \arg{Parts} is the broken-down request
(see \predref{uri_components}{2}) and \arg{Options} is the list of options passed
to http_open. The predicate is called as if using \predref{ignore}{1}.

\begin{tags}
\mtag{See also}- complements \qpredref{http}{update_cookies}{3}. \\- \verb$library(http/http_cookie)$ implements cookie handling on
top of these hooks.
\end{tags}

\qpredicate[semidet,multifile]{http}{update_cookies}{3}{+CookieData, +Parts, +Options}Update the cookie database. \arg{CookieData} is the value of the
\verb$Set-Cookie$ field, \arg{Parts} is the broken-down request (see
\predref{uri_components}{2}) and \arg{Options} is the list of options passed to
http_open.

\begin{tags}
\mtag{See also}- complements http:write_cookies \\- \verb$library(http/http_cookies)$ implements cookie handling on
top of these hooks.
\end{tags}
\end{description}

