IRC Services Manual - 5. Importing and exporting databases

# IRC Services Manual

## 5. Importing and exporting databases

5-1. Exporting Services databases in XML format
5-2. Importing (merging) data from an XML file
5-3. Converting data from other Services-like programs
5-3-1. Using the `convert-db` program
5-3-2. Notes on converting databases from particular programs

Table of Contents

### 5-1. Exporting Services databases in XML format

In order to facilitate processing of the Services databases by other programs, Services provides the ability to export the databases in XML format. This is accomplished through the `misc/xml-export` module and, optionally, the HTTP server built into Services.

The `misc/xml-export` module adds an extra command-line option, `-export`, to Services. Since the command line is parsed before Services connects to the network, the module must be specified in your `ircservices.conf` file to enable this option. The option has the format "`-export`" or "`-export=filename`", where `filename` is the name of the file to write the XML data to; if it does not begin with a slash, it is taken to be relative to the Services data directory (the directory containing the database and log files). If `filename` is not given or is a single hyphen ("`-`"), the data will be printed on standard output. When the export completes, Services will terminate, and will not connect to the IRC network. This action can be performed while another copy of Services is running, but the output will not include changes made online since the last time the databases were saved to disk. (It is also possible for the running copy of Services to report a "data directory locked" error if it tries to save the databases while the export operation is in progress. This is harmless; you can force the databases to be written after the export completes by using the OperServ `UPDATE` command, but do not use the `FORCE` option while the export is still running as this may result in corrupted database files.)

Additionally, if the `misc/xml-export` module is loaded into Services, the HTTP database access module (`httpd/dbaccess`) will display an "XML database download" link. Clicking on the link will display the XML data in your browser, which you can then save (for Netscape and Internet Explorer, select "Save as..." from the browser's File menu). Alternatively, you can download the link directly without displaying it in your browser; in Netscape, hold shift while clicking on the link, and in Internet Explorer, right-click on the link and select "Save target as..." from the pop-up menu.

The format of the exported XML data is described in Appendix B. No DTD is currently available.

Notes: Services will not respond to any network activity while you are downloading the database. If you have a large database, this can make Services appear to "freeze" with respect to the IRC network, and it may even get disconnected from the network in extreme cases. Additionally, be aware that the data may change after you have downloaded it; if you want to avoid this, for example when exporting data to move it to another machine or network, use the command-line option `-export` (after shutting Services down) rather than the HTTP server interface.

Note to Internet Explorer users: Some versions of Internet Explorer have a bug which can prevent the XML data from being displayed or downloaded properly. If you have this problem, use another browser, or add the following key to your registry using the Registry Editor (`regedit`), as described in Microsoft Knowledge Base article Q239750 [`support.microsoft.com`]:

Key: |  | `HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings`
Value name: |  | `IsTextPlainHonored`
Value type: |  | `DWORD`
Value data: |  | `HEX 0x1`

Improper use of the Registry Editor can cause your system to malfunction, so be careful when making any changes. You may also need to upgrade your system files as described in the Knowledge Base article linked to above if you are using Windows 95, Windows 98, or Windows NT 4.0.

Back to top

### 5-2. Importing (merging) data from an XML file

Just as it can export data in XML format, Services can also read in XML data and import (merge) it into its databases. Importing is handled by the `misc/xml-import` module. Unlike exporting, however, importing cannot be done via the HTTP server; it is always done through the command line. For this reason, the `misc/xml-import` module must be loaded at startup (loading it via an OperServ `REHASH` or a `SIGHUP` signal will have no effect).

To import a set of data stored in XML format into Services, start Services with the command-line option "`-import=filename`", where `filename` is the name of the file containing the data; if it does not begin with a slash, it is taken to be relative to the Services data directory. Services will parse the data file, and if no errors are found, the data will be merged into Services' databases and the program will terminate. If problems are found, they will be printed on standard output and the databases will not be modified.

Normally, if nickname or channel "collisions" occur—that is, if a nickname or channel in the input data is already registered in Services' databases—Services will skip over that nickname group or channel, respectively. (For nicknames, the entire nickname group is skipped to prevent cases where, for example, a user's secondary nick is available but the primary nick is registered to someone else.) This behavior can be modified by the `OnNicknameCollision` and `OnChannelCollision` options in `modules.conf`. Additionally, Services can be made to output a list of all data imported by setting the `VerboseImport` option (which is set in the example `modules.conf` provided with Services).

Note that the maximum user count, OperServ `SU` password, and news items will not be imported. Also, any autokills, session limit exceptions, or S-lines which collide with those in the database will be automatically skipped.

Back to top

### 5-3. Converting data from other Services-like programs

#### 5-3-1. Using the `convert-db` program

For users of other Services-like programs who want to switch to IRC Services, a program called `convert-db` is supplied to convert databases used by such programs into XML data files, which can then be imported into Services as described in section 5-2 above. This program is installed into the Services data directory (`/usr/lib/ircservices` for binary distributions, `/usr/local/lib/ircservices` by default for source distributions).

In its simplest form, `convert-db` takes the pathname of the directory containing the database files to convert on the command line, and writes the converted XML data to standard output. A typical invocation might look like this:

> `convert-db /usr/local/lib/someprogram > someprogram.xml`

`convert-db` will look at the files in the directory given and attempt to determine automatically what program created them; if this succeeds, it will proceed to read in the database files and convert them to XML. If `convert-db` cannot determine what program created the files, you can tell it explicitly by adding a `+program-name` option in front of the path name (see the full syntax description below for valid values for `program-name`).

Note that not all features of all programs are supported by Services, and some settings may be lost when you import data. See section 5-3-2 below for information on particular programs, and section 5-2 for notes on importing the converted data into Services.

The full syntax for `convert-db` is as follows:

> `convert-db [-v] [+program-name] [options` . . . `] pathname`

The options and parameters are:

- `-v`: Display progress information as the data is converted.
- `+program-name`: Explicitly indicates what program was used to create the data files. A list of possible `program-name` values is given in table 5-1 below. `program-name` values are not case sensitive.
- `options`: Additional options specific to each type of data file controlling details of the data conversion; see below.
- `pathname`: Specifies the pathname of the directory containing the database files.

`program-name` | Name of program | Supported versions
--- | --- | ---
`anope` | Anope Services [`www.anope.org`] | All versions
`auspice` | Auspice Services | 2.5 and later
`bolivia` | Bolivia IRC Services | 1.2.0
`cygnus` | Cygnus [`www.habber.net`] | 0.2.0
`daylight` | Daylight | 12
`epona` | Epona IRC Services | 1.3.0 and later
`hybserv` | HybServ | All 1.x versions
`ircs-1.2` | Internet Relay Chat Services (IRCS) | 1.2, 1.3
`magick-1.4` | Magick IRC Services (Magick I) | 1.4
`ptlink` | PTlink Services [`sourceforge.net`] | 2.13.0 through 2.26-eol.1
`sirv` | SirvNET Services [`www.sirv.net`] | 2.9.0 and earlier
`trircd-4.26` | trircd IRC Services | 4.26
`wrecked-1.2` | WreckedNet IRC Services | 1.2.0

Table 5-1. Values for `+program-name` parameter to `convert-db`

Additional options are available for the following data file types:

`cygnus`

- `-tzfile=filename`: Specifies an alternative time zone file to use when loading nickname time zone data. `convert-db` defaults to the deata from the `cygnus.zone` file distributed with Cygnus 0.2.0; if you have modified this file or created your own time zone list, you will need to use this option to get nickname time zones converted correctly.
- `-no-timezones`: Causes all nickname time zone settings to be reset to default (the time zone Services runs in).
- `-reset-memo-limits`: Causes all nickname memo limits to be reset to default; see the notes on Cygnus database conversion.

`hybserv`

- `-crypt`: Indicates that all passwords are encrypted (without this option, all passwords are assumed to be in plain text). Note that HybServ uses a different type of encryption that IRC Services, and `convert-db` will currently abort if you use this option. Support for encrypted passwords will be added in version 5.1.

Back to top

#### 5-3-2. Notes on converting databases from particular programs

Note that this is not a comprehensive list of differences; only the major differences and types of data that cannot be converted are listed.

All programs

- The "`NOOP`" and "`NEVEROP`" nickname flags used in some programs to mean "do not auto-op this nickname regardless of channel access level" are not supported. Any nicknames with these flags set will still be opped by Services if they have the appropriate access level in a channel.
- The setter and reason for a forbidden nickname or channel will not be imported.
- Channel privilege level settings (the equivalent of the ChanServ `LEVELS` command), on programs that support them, will not be imported; all channels will be reset to the default values.
- Services does not record the setter and last used time for channel access entries is not supported; this data will be lost on conversion for programs that do record it.
- Services does not support "bots" or a "BotServ", and all such data will be lost on conversion.

Anope

- For version 1.7.11 and later, databases must be stored using the standard ("FFF") format; MySQL databases are not supported.
- Anope (through at least version 1.7.14) contains the MD5 encryption bug present in versions of IRC Services before 4.5; as a result, encrypted passwords will not work by default. As a workaround, you can enable the `EnableAnopeWorkaround` configuration option for the `encryption/md5` module in `modules.conf`, but be aware that this may reduce the security of your passwords. There are also rare cases in which passwords may be truncated in the databases; if you encounter such a case, the password will have to be changed manually by a Services administrator.
- Services does not store the following information; it will be lost when the databases are converted:
  - ICQ number for nicknames
  - last used time for channel access entries
- Services does not allow multiple users to have Services super-user privileges (except temporarily with the `SU` command); all users with "root" privileges in Anope will be converted to Services administrators.
- Services does not distinguish between using the ChanServ `OP`, `VOICE`, and related commands on oneself or on other users.
- Services does not support the ChanServ `SECUREFOUNDER`, `SIGNKICK`, or `XOP` settings. (`XOP`-style commands can be enabled for all channels by loading the `chanserv/access-xop` module.)
- Nickname requests (nicknames which have been registered but not confirmed via E-mail) are discarded; users with pending nickname requests will need to re-register.
- By default, only basic channel modes (`+i`, `+k`, `+l`, `+m`, `+n`, `+p`, `+s`, and `+t`) will be imported for channel mode locks. To import extended modes provided by particular IRC servers, use the `-ircd=type` command-line option, where `type` one of the following (for Anope 1.7.6 and later, use the same name as in the `IRCDModule` configuration directive):
  - `bahamut` for Bahamut
  - `dreamforge` for Dreamforge
  - `hybrid` for IRCD-Hybrid
  - `ptlink` for PTlink IRCd
  - `ultimate2` for UltimateIRCD 2.x
  - `ultimate3` for UltimateIRCD 3.x
  - `unreal` for UnrealIRCd
  - `viagra` for Viagra IRCd

Auspice

- The ChanServ `TIMEOP` feature in version 2.6 and later is not supported. Nicknames on a channel's `TIMEOP` list will not be added to the channel's access list for Services, and thus will have no privileges on the channel.
- Services does not support "root" and "master" privilege levels as in Auspice. There can only be one root nickname, which is set in `modules.conf`; all root/master settings in the data files will be discarded. (Admin/oper settings will be retained.) Note that Services admins can perform any function (except modifying the admin list), including modifying Services settings and shutting down Services.
- NickServ `AUTH`, `COMMENT`, `NOTE`, `LOCK`, `MARK`, `SET KILL HIGH`, `SETFLAG AUTOJOIN`, and `SETMLOCK`, as well as ChanServ `BADWORDS`, `FREEZE`, and `MARK`, are not supported. Nickname notes will be converted to non-expiring memos with the user themselves as the sender; other data will be lost.
- Services uses a different news system than Auspice; network and channel news items will not be converted.
- Aside from the above, Services does not store the following information; it will be lost when the databases are converted:
  - UIN, age, and gender for nicknames
  - last `GETPASS` user and `HOLD` setter and reason for channels
- E-mailing of log files is not supported.

Bolivia

- Channel memos are implemented differently (they are stored separately from nickname memos), and any user may send a memo to a channel.
- Limiting channels to priviliged users (the `LEVEL` command) is not supported, and such restrictions will be discarded when converting the databases. However, some IRC servers (such as Unreal) have channel modes allowing channels to be limited to IRC operators or administrators only, and Services allows those modes to be locked on with the `SET MLOCK` command.
- ChanServ `FREEZE` is not supported.
- ChanServ `HOLD` setter will be lost when converting.
- E-mailing of log files is not supported.

Cygnus

- Services does not store the UIN, real name, age, sex, or location for nicknames; this information will be lost when the databases are converted.
- Services does not send replies using `PRIVMSG`; the `NOTICE` and `PRIVMSG` nickname settings will be ignored.
- The following options/features are not supported:
  - NickServ `NEVEROP`, `NOOP`, and `NOSUCCESSOR`
  - ChanServ `LIMITED`
  - MemoServ `RECEIPTS`
  - RootServ `MARK`
- Nicknames designated as CSOPs will be converted to Services operators. Note that the privileges granted by IRC Services to Services operators are different from those granted by Cygnus to CSOPs.
- Nicknames will retain their memo limit settings in the converted data; however, they will not be affected by changes to the `MSMaxMemos` setting. (Memo limits can be reset to the default, which follows the `MSMaxMemos` setting, with the `-reset-memo-limits` option.)
- If you have modified the time zone list used by Cygnus (`cygnus.zone`), nickname time zones will not be transferred correctly by default. Use the `-tzfile=` option to ensure that time zones are loaded from your time zone file, or use the `-no-timezones` option to reset all time zone settings to default.
- Authentication for channels is not supported; authentication codes and "verify" information will be discarded.
- Channel access list entries in IRC Services must always be registered nicknames; `user@host` entries, and any entries not matching a registered nickname, will be deleted.
- While the channel topic-lock setting will be retained, the lock level (founder/SOP/AOP/HOP/VOP) will be reset to the IRC Services default (AOP). If you use the `LEVELS` command (in the `chanserv/access-levels` submodule), you can change the level of the `TOPIC` command to set which users are allowed to change the topic.
- The ChanServ VOPALL setting is not preserved; however, the `LEVELS` command can be used to set the AUTOVOICE level to 0, causing all users to be voiced on entering the channel.
- Channel memos are stored with the channel, and are not sent to individual users on the channel. Anyone can send a memo to a channel; only users with SOP (by default) access can read or delete the memos.
- While not directly related to the databases, note that Services uses the `AUTH` command, not the `REGISTER` or `SET EMAIL` commands, to authenticate nicknames.

Daylight

- Services does not support the `XMANAGEMENT` channel setting.

Epona

- Epona contains the MD5 encryption bug present in versions of IRC Services before 4.5; as a result, encrypted passwords will not work by default. As a workaround, you can enable the `EnableAnopeWorkaround` configuration option for the `encryption/md5` module in `modules.conf`, but be aware that this may reduce the security of your passwords. There are also rare cases in which passwords may be truncated in the databases; if you encounter such a case, the password will have to be changed manually by a Services administrator.
- Services does not store the following information; it will be lost when the databases are converted:
  - ICQ number for nicknames
  - last used time for channel access entries
- Services does not distinguish between using the ChanServ `OP`, `VOICE`, and related commands on oneself or on other users.
- By default, only basic channel modes (`+i`, `+k`, `+l`, `+m`, `+n`, `+p`, `+s`, and `+t`) will be imported for channel mode locks. To import extended modes provided by particular IRC servers, use one of the following command-line options:
  - `-ircd=bahamut` for Bahamut
  - `-ircd=ultimate2` for UltimateIRCD
  - `-ircd=unreal` for UnrealIRCd

HybServ

- Services does not store the following information; it will be lost when the databases are converted:
  - GSM, phone, or ICQ number for nicknames
- Services does not have the concept of "permanent passwords", as in HybServ 1.6.1+UniBG, and such passwords will be discarded on import.
- Services does not support the NickServ `AUTOMASK`, `HIDE URL`, `NOREGISTER`, or `NOCHANOPS` settings, or the ChanServ `GUARD` or `VERBOSE` settings.
- The NickServ `HIDE ALL` setting is treated as a combination of `HIDE EMAIL`, `HIDE USERMASK`, and `HIDE QUIT`. The nickname URL and options cannot be hidden.
- Temporary forbids are imported as permanent (Services does not support the notion of a "temporary" or "timed" forbid).
- "Forgotten" channels are not supported, and will not be imported.
- Services does not have a system of "operator modes" like HybServ, and no operator mode information will be imported.
- Statistics information (maximum number of users/channels/etc.) will not be imported.
- The founder of a channel will be removed from the channel's access list (founders always have full channel privileges and do not need to be added to the access list).
- Channel privilege levels will be reset to the Services defaults on import. Channel access levels will also be modified to fit within the -999...999 range used by Services, as follows:

  HybServ level | Services level
  --- | ---
  -10000 | -999
  -9999...-10 | -999...-1
  -9...-1 | -1
  0...5 | 0...30
  5...10 | 30...50
  10...15 | 50...100
  15...20 | 100...110
  20...200 | 110...200
  200...1000 | 200...300
  1000...9000 | 300...900
  9000...9999 | 900...999
  10000 | 999

  WARNING: As a result of differences in the default level settings between HybServ and Services, some users will have access to commands which they did not previously have access to. In particular, the `ACCESS` command is available to all users with access level 40 (HybServ level 8) or above, and the `CLEAR` command is available to all users with access level 100 (HybServ level 15) or above. Conversely, the `INVITE` command is available only to users with access level 50 (HybServ level 10) or above; auto-voice users cannot use it.

IRCS

- By default, IRCS supports channels with names longer than 63 characters (including the leading "#"). Services does not support these by default, and such channels will not be imported.
- Channel autoop/autovoice cannot be set for individual nicknames. All nicks with appropriate access will be opped even if the NickServ `NOOP` setting was set or ChanServ `SET AUTOOP OFF` was used for the nick.
- The numeric access level for `SOP` is different from that used in Services, so some access levels will be altered to match the default `SOP` level in Services. (Channel privileges will remain unchanged.)
- The "senior", "junior", and "helper" OperServ access levels do not exist in Services. Users on the "senior" list will be moved to the Services admin list; users on other lists will be moved to the Services operator list. WARNING: As a result of the above, users on the "senior" list will be given access to the `JUPE` and `LISTIGNORE` commands.

Magick I 1.4

- Services does not send replies using `PRIVMSG`; the `NI_PRIVMSG` nickname setting will be ignored.
- Services does not have a "channel news" system like Magick, and all channel news items will be discarded.

PTlink

- PTlink supports several encryption methods; passwords encrypted with a method other than MD5 cannot be used with Services, and `convert-db` will refuse to convert such databases.
- Services does not store the following information; it will be lost when the databases are converted:
  - ICQ number, location, birthdate, time of last identify, and authentication code (2.22.0 and later) for nicknames
  - maximum number of users and time of maximum for channels
- An apparent bug in PTlink causes the successor field of some channels to be corrupted; this will result in warnings when loading the channel database.

SirvNET

- Channel memos are implemented differently (they are stored separately from nickname memos), and any user may send a memo to a channel.
- Limiting channels to priviliged users (the `LEVEL` command) is not supported, and such restrictions will be discarded when converting the databases. However, some IRC servers (such as Unreal) have channel modes allowing channels to be limited to IRC operators or administrators only, and Services allows those modes to be locked on with the `SET MLOCK` command.
- NickServ `AUTH` and ChanServ `FREEZE` are not supported.
- ChanServ `HOLD` setter and reason will be lost when converting.
- E-mailing of log files is not supported.

trircd

- Forbid setter and reason information will be lost after conversion.
- `SET EXCEPTION`, `SET INVITES`, `HIDE OPTIONS`, and `HIDE DESC` channel settings are not supported.

Wrecked 1.2.0

- Services does not send replies using `PRIVMSG`; the `NI_PRIVMSG` nickname setting will be ignored.
- The `VOPALL` channel setting is not supported.

Back to top

Table of Contents | Previous section: Services command reference | Top | Next section: Adding features to Services