History
-------

29/07/2007  *** v2.32 ***

            - Uses DatLib v2.21.
            - Empty attributes are now ignored (e.g. sha1="")
            - Removed cloneof, romof and sampleof fixes from datutil -i
            - DatUtil -i, -I and -p now default the output format to CMPro
              (in addition to -g, -G and -r which already behave like that)
            - Added support for the isbios="yes" of MAME v0.117u2 onwards.

01/01/2007  *** v2.31 ***

            - Uses DatLib v2.20.
            - Added 'prune' option (-p) for removing roms, disk and/or samples
            - Added 'incorporate' option (-I) to assist in combining datafiles

17/10/2006  *** v2.30 ***

            - Uses DatLib v2.18.
            - More intelligent DTD creation when saving in ListXML format
            - Added support for game/display (MAME v0.107)
            - Added support for game/input/control (MAME v0.107)
            - Added support for emulator/build (MAME+MESS v0.101u4 upwards)
            - Added support for driver/savestate (MAME v0.101u1 upwards)
            - Note: The above changes affect the tab delimited format!
            - Added 'delimited.sql' as an example of loading dats into MySQL

16/07/2006  *** v2.29 ***

            - Uses DatLib v2.17.
            - Fixed bug that caused the DatUtil -g and -r options to crash

11/07/2006  *** v2.28 ***

            - Uses DatLib v2.16.
            - Fixed bug that messed up internal indices (affected crc matching)
            - More intelligent directory scanner (regarding samples and CHDs)
            - Removed 64MB file limit from directory scanner (reads 1MB blocks)

13/01/2006  *** v2.27 ***

            - Uses DatLib v2.15.
            - Last release would not load data files less than 4KB in size!

12/01/2006  *** v2.26 ***

            - Uses DatLib v2.14.
            - Added a save-only format to imitate MAME's gamelist.txt file.
            - File identification is now done by loading just the first 4KB.
            - Added a list of supported save-formats to the command line help.

06/12/2005  *** v2.25 ***

            - Uses DatLib v2.13.
            - DatLib v2.12 broke the RomCenter drivers (now fixed).

05/12/2005  *** v2.24 ***

            - Uses DatLib v2.12.
            - Fixed a rare crash (if the longest line in the dat was the last).
            - Fixed a problem loading RomCenter data files containing [].

13/11/2005  *** v2.23 ***

            - Uses DatLib v2.11.
            - Added support for 'board', 'colordeep', 'credits' and 'archive'
              (fully supported by ListInfo, ListXML and Delimited formats).
            - M1 XML loader now retains the 'board' information (see above).
            - M1 XML will not use hex for ROM sizes after all... changed back!
            - Improved the way that the Nebula driver handles Neo-Geo ROMs.
            - Fixed Nebula Jukebox driver for Neo-Geo ROMs (broken in v2.7).

31/10/2005  *** v2.22 ***

            - Uses DatLib v2.10.
            - Fixed identification of ListXML for MAME+MESS v0.101u4 onwards.
            - Added the '-q' option (to save Listinfo with quotes throughout).

28/10/2005  *** v2.21 ***

            - Uses DatLib v2.9.
            - M1 XML will use hex for ROM sizes. Changed driver accordingly.
            - Now reports duplicate game descriptions and CRCs (i.e. ROMs).

05/10/2005  *** v2.20 ***

            - Uses DatLib v2.8.
            - Improved the logerror() function (used for reporting ZIP errors).
            - Added support for M1's XML format (when it becomes available).

17/09/2005  *** v2.19 ***

            - Uses DatLib v2.7.
            - Compiled with zlib 1.3.
            - Improved Neo-Geo handling for the Nebula driver.
            - Improved 'romof' handling for RomCenter 2 driver (during loading).

31/08/2005  *** v2.18 ***

            - Uses DatLib v2.6.
            - Fixed a 'hang' that occurred with circular cloneof relationships.

10/04/2005  *** v2.17 ***

            - Uses DatLib v2.5.
            - Excludes disks from RomCenter data files (-k will include them).

25/03/2005  *** v2.16 ***

            - Uses DatLib v2.4.
            - Fixed another crash with datutil -X and 'forcemerging full'.
            - Changed the delimited format to use \N for NULLs.
            - Changed the delimited format so that all types are 'game_*'.
            - Changed the delimited format so that games include 'runnable'.
            - Added save support for a DISK section in the RomCenter format.
            - Added several aliases for the output formats (see notes above).

07/03/2005  *** v2.15 ***

            - Uses DatLib v2.3.
            - Fixed a problem with counting parents (happened with DatUtil -X).
            - This also avoids a crash with datutil -X and 'forcemerging full'.

28/02/2005  *** v2.14 ***

            - Uses DatLib v2.2.
            - Added support for tab delimited files ('-f delimited').
            - Added save-support for RomCenter sublists ('-f sublist').
            - Added \t support to the listinfo driver (and tab delimited).
            - Fixed problem of unrecognised escape sequences looping forever!

22/02/2005  *** v2.13 ***

            - Uses DatLib v2.1.
            - Added a test option (-t) where nothing is actually saved.
            - Log file is now automatically deleted if it is empty.
            - Made selections more flexible (can now use commas as a delimiter).
            - Added warnings for invalid game/sourcefile selections (-g and -G).
            - Added warning messages for duplicated games (i.e. same game name).
            - Can identify listinfo files even if they only contain resources.
            - The 'remove clones' option will retain the original dat format.
            - Fixed a crash caused by empty selection files (the '@' syntax).
            - Saving a RomCenter datafile will now report an error if the '¬'
              symbol is used in the game or ROM details (i.e. the dat is bad).
            - Changed the error messages that are given when files are missing.
            - Changed the command line help a little bit (explanation of -!).

13/02/2005  *** v2.12 ***

            - Uses DatLib v2.0.
            - Compiled with GCC 3.4.2 (previously used GCC 3.2.3).
            - Rewrote the manual, please read it from start to end. ;)
            - Changed the summary of switches produced by DatUtil -?.
            - Added the -k option to keep all details (chips, dips, etc).
            - Added the -G option to select games from a specified sourcefile.
            - Added the -! option so that -g and -G can be used as exclusions.
            - Added the -X option to switch off 'fix merging' (use with -g).
            - Added the -D option to switch off 'remove duplicates' (as above).
            - Removed the -z and -y options as they are redundant now.

            - DatLib v2.0 (enhancements relating to DatUtil):
            - Added 'sourcefile' support to the listinfo and listxml drivers.
            - Supports all known details in listinfo and listxml (-k option).
            - N.b. ROM region is not loaded unless using the datutil -k option.
            - Supports 'history' entries (when using the -k option).
            - Added support for \\, \n, \xXX and \" in ListInfo (e.g. history).
            - Improved entity handling (e.g. &amp; &#X;) in the ListXML parser.
            - Supports 2 additional MESS objects (devices and extensions).
            - DatLib can now save data in listxml format (MAME and MESS).
            - Complemented CRCs are reported (should use flags nodump instead).
            - Added checks for MD5 and SHA1 conflicts (CRCs already checked).
            - Added support for external selection files with the @ symbol.
            - Removed the hack made in DatLib v1.13 for Calice drivers.
            - Removed funny size and missing year checks (pretty useless now).

16/01/2005  *** v2.11 ***

            - Uses DatLib v1.14.
            - Added support for the GNGEO romrc file.
            - Increased maximum ROM size to 64MB (used when scanning folders).

04/01/2005  *** v2.10 ***

            - Uses DatLib v1.13.
            - Added support for Calice drivers.

25/11/2004  *** v2.9 ***

            - Uses DatLib v1.12.
            - Improved the handling of CRC conflicts within the 'fix merging'.

26/10/2004  *** v2.8 ***

            - Uses DatLib v1.11.
            - Added a parser for the ZiNc --list-sets.

18/08/2004  *** v2.7 ***

            - Uses DatLib v1.10.
            - Better handling of 'clones of clones' when loading data files.

30/07/2004  *** v2.6 ***

            - Uses DatLib v1.9.
            - Added full support for disk 'merge' and 'flags' information.

22/07/2004  *** v2.5 ***

            - Uses DatLib v1.8.
            - Removed the need for xml2info by adding my own MAME XML parser.
            - A nice side effect is that names like "Alien³: The Gun" work now!

21/07/2004  *** v2.4 ***

            - Uses DatLib v1.7
            - Added support for MESS XML.
            - Directory scanner now supports CHDs (when in a game directory).
            - Improved 'fix merging' (where clone ROMs are not in the parent).
            - Enhanced 'fix merging' to work for disks (SHA1/MD5 are copied).
            - Added check for SHA1/MD5 conflicts of disks across clones.

17/07/2004  *** v2.3 ***

            - Uses DatLib v1.5
            - Added the -j option for Nebula Jukebox (Z80 and Samples only).

11/07/2004  *** v2.2 ***

            - Uses DatLib v1.4:
            - Added support for comments between games in CMPro data files.
            - Added an option to ignore missing year information (-y).
            - Added an option to select MD5 usage rather than SHA1 (-m).
              Note that '-m' was previously used for the 'fix merging' option!
            - Added an option to include missing info from another dat (-i).
              Games: cloneof, romof, sampleof (good after using the -g option).
              ROMs: size, crc, md5, sha1, flags (good for adding SHA1, etc).

04/07/2004  *** v2.1 ***

            - Added a message to say that the -m option is not required now.
            - Uses DatLib v1.1:
            - ClrMamePro/RomCenter header is now ignored when using OPTION_GAME.
            - Removed a warning when you try using 'datutil -g <resource> -c'.
            - Added debug information to identify spurious warnings like above.
            - Added the -x option to calculate SHA1/MD5 when scanning ZIPs.
            - n.b. SHA1/MD5 used to be calculated by default in DatUtil v2.0.

02/07/2004  *** v2.0 ***

            Wow, I never thought I'd get around to this rewrite but here it is!

-------------------------------------------------------------------------------

Uses DatLib - The new library for handling dats of many formats
===========

- A generic dat structure with various loading and cleansing routines.

- Far better memory handling. No longer uses huge blocks of fixed size memory.

  Info: DatUtil used to allocate almost 45MB for a dat when it was loaded.
        This was without MD5/SHA1 support and for a limited number of games!
        DatLib now allocates only what it really requires (which is usually
        about twice the size of the dat file).

- Multi-phase processing (pre-parse, parse, cleanse, save):

  - Pre-parser translates dat into an intermediate dat format.
  - Parser interprets the intermediate format to populate the dat structure.
  - Cleanser does things like 'lowercase', 'remove dups', 'fix merging', etc.
  - Saving outputs the cleansed dat structure in the required output format.

  Note: The main difference is the addition of the pre-parser (which removes
        common operations from the different dat parsers) and the separation
        of the cleansing operations from the loader (they used to be embedded
        in the various parsers, particularly painful for RomCenter format!).

- No limit on the number of games, ROMs, disks, samples etc.

- No limit on the length of file names, game titles, manufacturer names etc.

- Improved size and CRC handling (now numeric, not text based).

- Supports 'rebuildto' information for games.

- Supports alternative checksums (MD5 and SHA1).
  N.B. The directory scanner also generates MD5 and SHA1 checksums for files.

- Supports 'region' information of ROMs.

- Improved driver system for developers adding new dat formats.

- Fully supports the latest RomCenter dat format (2.50).

- Improved RomCenter parser is not case sensitive like the old one was.

- All cleansing features can now be used with all supported formats.

- Improved 'fix merging'. Now operates across clones, not just parent/clone.
  Also fixes bad parent references and prevents clones of clones.
  The feature is run by default when a data file is loaded into memory.
  A summary of fixes is shown in datutil.log, use -v option for full details.

- Duplicate ROMs are now removed when the data file is loaded into memory.

- Added debugging information (to help me track down problems).

- Removed alternate ROM support. It was only used by Shark and made data file
  comparisons unnecessarily complicated!

- Removed code that fixed the bad parent 'pc' in MESS dats (the default
  'fix merging' code does the same job now but without being hard coded).

- ClrMamePro log parser now supports disk images and alternative checksums.

- Nebula parser is now much better. With the default 'fix merging' behaviour,
  the data file that is generated should be almost usable now!

- Improved baddump and nodump handling.
  Also supports the syntax of MAME v0.68 (where the word 'flags' was missing).

- Improved logging. Added a verbose (-v) option for even more detail!

- You are given a warning if you try to convert a RomCenter data file where
  the plugin is not 'arcade.dll'. It is likely that such a conversion will
  not work correctly in ClrMamePro.

- The -g and -c options can now be used on resources. You can therefore
  extract games for neogeo, zn1, etc. (e.g. 'datutil -g cpzn1 -c mame.dat').

- Removed the dat comparison facility. Use MAMEDiff instead!

- Added load support for MAME ListXML format (uses the xml2info tool).

- Added a warning summary to datutil.log.
  Warnings are for missing details, funny ROM sizes and CRC conflicts.
  For full details of the warnings, use the DatUtil -v (verbose) option.

- Added details of all information that is lost when saving in a different
  format (e.g. Years and Manufacturers are lost when using RomCenter format).


Unsupported formats - Not included in the rewrite as they are of little use!
===================

Load and Save:

  - Calice DRV (drivers)

Load only:

  - Callus gamelist
  - CPSE data (drivers)
  - Mimic HWC (drivers)
  - System16 GCS (drivers)
  - Sparcade RAT (drivers)
  - Retrocade Romlist
  - JAS romlist
  - JFF text

-------------------------------------------------------------------------------

v1.x History
------------

unreleased  *** v1.33 ***
            - Added support for comments in Nebula drivers.

26/02/2004  *** v1.32 ***
            - Finally added disk support for MAME!

29/01/2004  *** v1.31 ***
            - When saving data files in listinfo format, 'crc 00000000' is now
              written out as 'flags nodump'.

24/01/2004  *** v1.30 ***
            - Increased the maximum number of games from 5000 to 6000.
            - Removed support for the old RomCenter 1 format...
              The old format is inadequate for modern day MAME releases.
            - Removed knowledge of the MAME resource names. RomCenter 2 data
              files must therefore have a [RESOURCES] section from now on.

20/01/2004  *** v1.29 ***
            - Minor adjustment to the routine that decides if a ROM lives in a
              resource. This fixes the handling of shootgal/sg-01-0.

31/12/2003  *** v1.28 ***
            - Added knowledge of the atpsx resource.

28/12/2003  *** v1.27 ***
            - Added knowledge of the two new resources added in MAME v0.78.

14/09/2003  *** v1.26 ***
            - Changed names of Neo-Geo BIOS ROMs in the Nebula driver reader.

17/08/2003  *** v1.25 ***
            - Added knowledge of the seven new resources added in MAME v0.72.

07/06/2003  *** v1.24 ***
            - Added support for '[RESOURCES]' in RomCenter2 data files.

19/05/2003  *** v1.23 ***
            - Added a fix for the 'nodump' syntax of MAME v0.68.

06/04/2003  *** v1.22 ***
            - Fixed CMPro log scanner to support the syntax 'bad dump'.
            - Added knowledge of konamigx resource. 

18/01/2003  *** v1.21 ***
            - Removed 'cloneof neogeo' from 'neocd' in Nebula driver.
            - Added knowledge of skns and stvbios resources.

20/05/2002  *** v1.20 ***
            - New resource ROMs in MAME listinfo are automatically detected.
            - ClrMamePro log and Nebula driver loaders now use the STORE_ROM
              macro (this makes -g work correctly, amongst other things).
            - Improved -m (fix merging) behaviour for resource ROMs.
            - Nebula drivers now output the correct 'romof' for Neo-Geo clones.

22/04/2002  *** v1.19 ***
            - Added support for the PGM resource.

20/04/2002  *** v1.18 ***
            - Increased maximum number of samples to 10,000 (for PinMAME).
            - Exceeding the sample limit now results in a graceful exit.

04/04/2002  *** v1.17 ***
            - Made 'fix-merging' option check the CRC as well as the name.
            - Fix merging option warns of CRC conflicts it finds. Useful test!
            - Strips trailing spaces from Nebula ROM names (for convenience).
            - Executable is called 'datutil.exe' again.

28/03/2002  *** v1.16 ***
            - Added Nebula driver support (useful for data file validation).
            - Supports the new '0x' syntax of CMPro logs.
            - ZIP scanner now removes '.Zip' from filenames (case sensitive).
            - Increased maximum line length to 16384 bytes (for MAME history).
            - Increased maximum number of ROMs to 75000.
            - Exceeding the game or ROM limits now results in a graceful exit.

10/02/2002  *** v1.15 ***
            - Enhanced Calice support to cope with Neo-Geo sound ROMs.
            - Modified source to compile easily with CygWin (and FreeBSD?).

11/11/2001  *** v1.14 ***
            - Added knowledge of ng-lo.rom.

27/09/2001  *** v1.13 ***
            - Supports CPSE driver file.
            - Added 'ignore funny size' option (-z) for non-arcade dats.
            - Added 'make lower case' option (-l). Affects names and CRCs.
            - Puts "s around empty CMPro header items.
            - Fixed directory/zip scanner for ROM names containing spaces.
            - Fixed zip scanner to strip uppercase '.ZIP' from game names.
            - Improved the 'funny size' warning relating to RomCenter 2 dats.

28/08/2001  *** v1.12 ***
            - Supports new ROM loading introduced in Calice v0.3.8 to v0.4.4

28/06/2001  *** v1.11 ***
            - Added some warnings for bad quality RomCenter dats (wrong version
              number in header and missing ROM sizes)

23/06/2001  *** v1.10 ***
            - Fixed detection of RAINE32 dat format.
            - Improved ZIP scanner (not case sensitive when searching for zips).

08/06/2001  *** v1.9 ***
            - Added directory scanning feature (reads ZIPs and subdirectories).
            - Calice loader now uses the main driver code, not 'verifyroms'.
            - RomCenter 2 loader fixes missing 'romof' details automatically.

18/05/2001  *** v1.8 ***
            - Fixed a minor bug in the '-m' feature (bug introduced in v1.7).
            - Now compiled with MinGW (no long filename problems on NT now).

26/04/2001  *** v1.7 ***
            - Added handling of decocass resource in MAME dats.
            - Added load/save support for CMPro element 'clrmamepro' and
              ROMCenter headers.
            - Added preliminary support for Calice Driver files.
            - Added load support for ClrMamePro log files (i.e. to convert into
              a dat format).
            - Further improved the '-m' feature ('fix merging' option).
            - RomCenter2 save routine strips redundant 'merge' information from
              non-clones.
            - DJGPP compilation now uses -Wall option.
            - Can now be compiled with Microsoft Visual C++.

21/02/2001  *** v1.6 ***
            - Added option to improve MAME's merging information ('-m').

06/02/2001  *** v1.5 ***
            - Some handling of the playch10 and cvs resources in MAME dats.
            - RomCenter 2 is now the default save format when converting CMPro
              dats.

18/12/2000  *** v1.4 ***
            - Mimic HWC support.
            - New format to list game titles ('-f titlelist').
            - Year/Manufacturer support for formats that can include them.
            - Can specify values for ROMCenter/ROMCenter2 headers.
            - Remove clones factility ('-r').
            - GWC loader now understands '.speech' tags (System16 Beta).
            - Callus95 Patch renamed 'directory' to 'parent'. Now supported.
            - Auto sorts when converting to RomCenter format or a Title List.
            - Improved sorting (ensures each parent precedes its clones).
            - RAINE and MESS files are converted to CMPro format by default.
            - Fixed a problem with the '-g' option and RAINE gameinfo files.
            - Increased maximum ROM size to 9 characters (~1GB) for U64Emu.
            - Increased number of lines scanned when identifying CMPro dats.
            - Source: Added a makefile and included a SAVE_ONLY_FORMAT.
            - Source: Changed parameter passing (uses a structure now).
            - Preliminary: 'Verify Merging' facility (currently disabled).

19/05/2000  *** v1.3 ***
            - RomCenter 2 support ('-f romcenter2').
            - New method of sorting (useful for RomCenter).
            - Two EXEs. This is because the long ROM name version eats memory
              and is slow on machines without much physical memory.
            - Improved handling of invalid parents.
            - Warns of funny ROM sizes.
            - Generates log of warnings during conversion.

28/04/2000  *** v1.2 ***
            - Increased game/rom name length to 80 characters.
              (DatUtil now requires even more memory for the dats!)
            - Minor string handling improvements. yawn...

08/04/2000  *** v1.1 ***
            - Added support for JFF.TXT, Sparcade RAT and JAS Romlist.
            - Increased maximum number of games from 4096 to 5000.
            - Increased game/rom name length to 50 characters.
              (DatUtil now requires a HUGE block of memory to hold the data)
            - Improved listinfo parser so that it can handle odd formatting.
            - Other minor modifications.

31/03/2000  *** v1.0 ***
