Keywords: main

Subsections

- Maximum Concurrent Jobs for Devices
- Restore from Multiple Storage Daemons
- File Deduplication using Base Jobs
- AllowCompression = yesno
- Accurate Fileset Options
- Tab-completion for Bconsole
- Pool File and Job Retention
- Read-only File Daemon using capabilities
- Bvfs API
- Testing your Tape Drive
- New Block Checksum Device Directive
- New Bat Features
  - Media List View
  - Media Information View
  - Job Information View
  - Autochanger Content View
- Bat on Windows
- New Win32 Installer
- Win64 Installer
- Linux Bare Metal Recovery USB Key
- bconsole Timeout Option
- Important Changes
  - Truncate volume after purge
  - Custom Catalog queries
  - Deprecated parts
- Misc Changes

# New Features in 5.0.0

#
Maximum Concurrent Jobs for Devices

Maximum Concurrent Jobs is a new Device directive in the Storage Daemon configuration permits setting the maximum number of Jobs that can run concurrently on a specified Device. Using this directive, it is possible to have different Jobs using multiple drives, because when the Maximum Concurrent Jobs limit is reached, the Storage Daemon will start new Jobs on any other available compatible drive. This facilitates writing to multiple drives with multiple Jobs that all use the same Pool.

This project was funded by Bacula Systems.

# Restore from Multiple Storage Daemons

Previously, you were able to restore from multiple devices in a single Storage Daemon. Now, Bacula is able to restore from multiple Storage Daemons. For example, if your full backup runs on a Storage Daemon with an autochanger, and your incremental jobs use another Storage Daemon with lots of disks, Bacula will switch automatically from one Storage Daemon to an other within the same Restore job.

You must upgrade your File Daemon to version 3.1.3 or greater to use this feature.

This project was funded by Bacula Systems with the help of Equiinet.

# File Deduplication using Base Jobs

A base job is sort of like a Full save except that you will want the FileSet to contain only files that are unlikely to change in the future (i.e. a snapshot of most of your system after installing it). After the base job has been run, when you are doing a Full save, you specify one or more Base jobs to be used. All files that have been backed up in the Base job/jobs but not modified will then be excluded from the backup. During a restore, the Base jobs will be automatically pulled in where necessary.

This is something none of the competition does, as far as we know (except perhaps BackupPC, which is a Perl program that saves to disk only). It is big win for the user, it makes Bacula stand out as offering a unique optimization that immediately saves time and money. Basically, imagine that you have 100 nearly identical Windows or Linux machine containing the OS and user files. Now for the OS part, a Base job will be backed up once, and rather than making 100 copies of the OS, there will be only one. If one or more of the systems have some files updated, no problem, they will be automatically restored.

See the Base Job Chapterbasejobs for more information.

This project was funded by Bacula Systems.

# AllowCompression = yesno

This new directive may be added to Storage resource within the Director's configuration to allow users to selectively disable the client compression for any job which writes to this storage resource.

For example:

    Storage {
      Name = UltriumTape
      Address = ultrium-tape
      Password = storage_password # Password for Storage Daemon
      Device = Ultrium
      Media Type = LTO 3
      AllowCompression = No # Tape drive has hardware compression
    }

The above example would cause any jobs running with the UltriumTape storage resource to run without compression from the client file daemons. This effectively overrides any compression settings defined at the FileSet level.

This feature is probably most useful if you have a tape drive which supports hardware compression. By setting the `AllowCompression = No` directive for your tape drive storage resource, you can avoid additional load on the file daemon and possibly speed up tape backups.

This project was funded by Collaborative Fusion, Inc.

#
Accurate Fileset Options

In previous versions, the accurate code used the file creation and modification times to determine if a file was modified or not. Now you can specify which attributes to use (time, size, checksum, permission, owner, group, ...), similar to the Verify options.

    FileSet {
      Name = Full
      Include = {
        Options {
           Accurate = mcs
           Verify   = pin5
        }
        File = /
      }
    }

  i compare the inodes
  p compare the permission bits
  n compare the number of links
  u compare the user id
  g compare the group id
  s compare the size
  a compare the access time
  m compare the modification time (st_mtime)
  c compare the change time (st_ctime)
  d report file size decreases
  5 compare the MD5 signature
  1 compare the SHA1 signature

Important note: If you decide to use checksum in Accurate jobs, the File Daemon will have to read all files even if they normally would not be saved. This increases the I/O load, but also the accuracy of the deduplication. By default, Bacula will check modification/creation time and size.

This project was funded by Bacula Systems.

#
Tab-completion for Bconsole

If you build `bconsole` with readline support, you will be able to use the new auto-completion mode. This mode supports all commands, gives help inside command, and lists resources when required. It works also in the restore mode.

To use this feature, you should have readline development package loaded on your system, and use the following option in configure.

    ./configure --with-readline=/usr/include/readline --disable-conio ...

The new bconsole won't be able to tab-complete with older directors.

This project was funded by Bacula Systems.

#
Pool File and Job Retention

We added two new Pool directives, `FileRetention` and `JobRetention`, that take precedence over Client directives of the same name. It allows you to control the Catalog pruning algorithm Pool by Pool. For example, you can decide to increase Retention times for Archive or OffSite Pool.

It seems obvious to us, but apparently not to some users, that given the definition above that the Pool File and Job Retention periods is a global override for the normal Client based prunning, which means that when the Job is prunned, the prunning will apply globally to that particular Job.

Currently, there is a bug in the implementation that causes any Pool retention periods specified to apply to all Pools for that particular Client. Thus we suggest that you avoid using these two directives until this implementation problem is corrected.

#
Read-only File Daemon using capabilities

This feature implements support of keeping ReadAll capabilities after UID/GID switch, this allows FD to keep root read but drop write permission.

It introduces new `bacula-fd` option (`-k`) specifying that ReadAll capabilities should be kept after UID/GID switch.

    root@localhost:~# bacula-fd -k -u nobody -g nobody

The code for this feature was contributed by our friends at AltLinux.

#
Bvfs API

To help developers of restore GUI interfaces, we have added new dot commands that permit browsing the catalog in a very simple way.

- `.bvfs_update [jobid=x,y,z]` This command is required to update the Bvfs cache in the catalog. You need to run it before any access to the Bvfs layer.

- `.bvfs_lsdirs jobid=x,y,z path=/path | pathid=101` This command will list all directories in the specified `path` or `pathid`. Using `pathid` avoids problems with character encoding of path/filenames.

- `.bvfs_lsfiles jobid=x,y,z path=/path | pathid=101` This command will list all files in the specified `path` or `pathid`. Using `pathid` avoids problems with character encoding.

You can use `limit=xxx` and `offset=yyy` to limit the amount of data that will be displayed.

    * .bvfs_update jobid=1,2
    * .bvfs_update
    * .bvfs_lsdir path=/ jobid=1,2

This project was funded by Bacula Systems.

#
Testing your Tape Drive

To determine the best configuration of your tape drive, you can run the new `speed` command available in the `btape` program.

This command can have the following arguments:

`file_size=n`
  Specify the Maximum File Size for this test (between 1 and 5GB). This counter is in GB.
`nb_file=n`
  Specify the number of file to be written. The amount of data should be greater than your memory ( ).
`skip_zero`
  This flag permits to skip tests with constant data.
`skip_random`
  This flag permits to skip tests with random data.
`skip_raw`
  This flag permits to skip tests with raw access.
`skip_block`
  This flag permits to skip tests with Bacula block access.

    *speed file_size=3 skip_raw
    btape.c:1078 Test with zero data and bacula block structure.
    btape.c:956 Begin writing 3 files of 3.221 GB with blocks of 129024 bytes.
    ++++++++++++++++++++++++++++++++++++++++++
    btape.c:604 Wrote 1 EOF to "Drive-0" (/dev/nst0)
    btape.c:406 Volume bytes=3.221 GB. Write rate = 44.128 MB/s
    ...
    btape.c:383 Total Volume bytes=9.664 GB. Total Write rate = 43.531 MB/s

    btape.c:1090 Test with random data, should give the minimum throughput.
    btape.c:956 Begin writing 3 files of 3.221 GB with blocks of 129024 bytes.
    +++++++++++++++++++++++++++++++++++++++++++
    btape.c:604 Wrote 1 EOF to "Drive-0" (/dev/nst0)
    btape.c:406 Volume bytes=3.221 GB. Write rate = 7.271 MB/s
    +++++++++++++++++++++++++++++++++++++++++++
    ...
    btape.c:383 Total Volume bytes=9.664 GB. Total Write rate = 7.365 MB/s

When using compression, the random test will give your the minimum throughput of your drive . The test using constant string will give you the maximum speed of your hardware chain. (cpu, memory, scsi card, cable, drive, tape).

You can change the block size in the Storage Daemon configuration file.

# New Block Checksum Device Directive

You may now turn off the Block Checksum (CRC32) code that Bacula uses when writing blocks to a Volume. This is done by adding:

    Block Checksum = no

doing so can reduce the Storage daemon CPU usage slightly. It will also permit Bacula to read a Volume that has corrupted data.

The default is yes - i.e. the checksum is computed on write and checked on read.

We do not recommend to turn this off particularly on older tape drives or for disk Volumes where doing so may allow corrupted data to go undetected.

# New Bat Features

Those new features were funded by Bacula Systems.

## Media List View

By clicking on ``Media'', you can see the list of all your volumes. You will be able to filter by Pool, Media Type, Location,...And sort the result directly in the table. The old ``Media'' view is now known as ``Pool''.

## Media Information View

By double-clicking on a volume (on the Media list, in the Autochanger content or in the Job information panel), you can access a detailed overview of your Volume. (cf 8.1.)

Figure 8.1: Media information

## Job Information View

By double-clicking on a Job record (on the Job run list or in the Media information panel), you can access a detailed overview of your Job. (cf 8.2.)

Figure 8.2: Job information

## Autochanger Content View

By double-clicking on a Storage record (on the Storage list panel), you can access a detailed overview of your Autochanger. (cf 8.2.)

Figure 8.3: Autochanger content

To use this feature, you need to use the latest mtx-changer script version. (With new `listall` and `transfer` commands)

# Bat on Windows

We have ported bat to Windows and it is now installed by default when the installer is run. It works quite well on Win32, but has not had a lot of testing there, so your feedback would be welcome. Unfortunately, eventhough it is installed by default, it does not yet work on 64 bit Windows operating systems.

# New Win32 Installer

The Win32 installer has been modified in several very important ways.

- You must deinstall any current version of the Win32 File daemon before upgrading to the new one. If you forget to do so, the new installation will fail. To correct this failure, you must manually shutdown and deinstall the old File daemon.
- All files (other than menu links) are installed in c:/Program Files/Bacula.
- The installer no longer sets this file to require administrator privileges by default. If you want to do so, please do it manually using the cacls program. For example:

      cacls "C:\Program Files\Bacula" /T /G SYSTEM:F Administrators:F

- The server daemons (Director and Storage daemon) are no longer included in the Windows installer. If you want the Windows servers, you will either need to build them yourself (note they have not been ported to 64 bits), or you can contact Bacula Systems about this.

# Win64 Installer

We have corrected a number of problems that required manual editing of the conf files. In most cases, it should now install and work. bat is by default installed in c:/Program Files/Bacula/bin32 rather than c:/Program Files/Bacula as is the case with the 32 bit Windows installer.

# Linux Bare Metal Recovery USB Key

We have made a number of significant improvements in the Bare Metal Recovery USB key. Please see the README files it the rescue release for more details.

We are working on an equivalent USB key for Windows bare metal recovery, but it will take some time to develop it (best estimate 3Q2010 or 4Q2010)

# bconsole Timeout Option

You can now use the -u option of bconsole to set a timeout in seconds for commands. This is useful with GUI programs that use bconsole to interface to the Director.

#
Important Changes

- You are now allowed to Migrate, Copy, and Virtual Full to read and write to the same Pool. The Storage daemon ensures that you do not read and write to the same Volume.
- The `Device Poll Interval` is now 5 minutes. (previously did not poll by default).
- Virtually all the features of mtx-changer have now been parameterized, which allows you to configure mtx-changer without changing it. There is a new configuration file mtx-changer.conf that contains variables that you can set to configure mtx-changer. This configuration file will not be overwritten during upgrades. We encourage you to submit any changes that are made to mtx-changer and to parameterize it all in mtx-changer.conf so that all configuration will be done by changing only mtx-changer.conf.
- The new `mtx-changer` script has two new options, `listall` and `transfer`. Please configure them as appropriate in mtx-changer.conf.
- To enhance security of the `BackupCatalog` job, we provide a new script (`make_catalog_backup.pl`) that does not expose your catalog password. If you want to use the new script, you will need to manually change the `BackupCatalog` Job definition.
- The `bconsole` `help` command now accepts an argument, which if provided produces information on that command (ex: `help run`).

### Truncate volume after purge

Note that the Truncate Volume after purge feature doesn't work as expected in 5.0.0 version. Please, don't use it before version 5.0.1.

## Custom Catalog queries

If you wish to add specialized commands that list the contents of the catalog, you can do so by adding them to the `query.sql` file. This `query.sql` file is now empty by default. The file `examples/sample-query.sql` has an a number of sample commands you might find useful.

## Deprecated parts

The following items have been deprecated for a long time, and are now removed from the code.

- Gnome console
- Support for SQLite 2

#
Misc Changes

- Updated Nagios check_bacula
- Updated man files
- Added OSX package generation script in platforms/darwin
- Added Spanish and Ukrainian Bacula translations
- Enable/disable command shows only Jobs that can change
- Added `show disabled` command to show disabled Jobs
- Many ACL improvements
- Added Level to FD status Job output
- Begin Ingres DB driver (not yet working)
- Split RedHat spec files into bacula, bat, mtx, and docs
- Reorganized the manuals (fewer separate manuals)
- Added lock/unlock order protection in lock manager
- Allow 64 bit sizes for a number of variables
- Fixed several deadlocks or potential race conditions in the SD