DNSCheck::Overview(3)

NAME
The DNSCheck System

DESCRIPTION

  Introduction
The central part of the DNSCheck system are the Perl modules that perform the actual tests. All of them should be used by creating a DNSCheck.pm object and letting it create other objects as needed and/or requested. These modules will make lots of queries to DNS servers all around the world, and sooner or later return a result in the form of a list of messages.

At this level, the libs can be used for any number of tasks. In the examples directory in the source distribution there is, for example, a script that lets one monitor the status of a domain via the Nagios monitoring system.

The system as distributed, however, contains more than the lib. There are also a couple of programs and a web interface. We will not talk about the web interface here, except to mention that it like the rest of the programs communicate with the other parts via the database tables to be mentioned later.

  The programs
The dispatcher

The program that actually runs tests is dnscheck-dispatch. It polls the table "queue" table in the database every 0.25 seconds, and forks children to run tests on what it finds there. It monitors the state of the children, and cleans up after they exit. If a child exits uncleanly, the test it was running is re-scheduled. If the same domain causes unclean exits five times or more, that domain will be blacklisted and will not be tested again until the dispatcher is restarted.

If the dispatcher receives a TERM signal, it will stop starting new tests, wait for the ones currently running to exit and then exit in good order. This may at worst take several minutes.

If the dispatcher receives a HUP signal, it will do an orderly exit as above, and then re-execute itself. If it was not originally started with a full pathname, the re-execution may fail.

The dispatcher is written to do a small number of tasks (get queue entry, start child, clean up after it) in as robust fashion as possible. As of this writing, the program appears to be very reliable as long as the database server is not too heavily loaded. Exactly what "too heavily loaded" means in practice will vary, of course, but as a rule of thumb, if a simple UPDATE takes more than a second to finish the system is almost certainly there.

It is vital that the queue table use the InnoDB storage engine. If it is MyISAM, there will be data corruption under heavy load. Other storage engines with sane transaction handling should also work, but have not been tested.

At the moment, only MySQL is supported. Patches to add support for more databases are welcome.

The dispatcher's child processes run the default tests on the given domain, write the results to the "tests" and "results" tables in the database and then exits.

The zone change detector

One of the programs that insert entries into the "queue" table is dnscheck-zonediff. This program does a zone transfer of a specified domain and looks at all the "NS" and "DS" records. For ever subdomain, it'll concatenate the textual representations of the records and calculate an MD5 sum of the resulting string. If that sum is not identical with what it was the previous time the program was run, the subdomain in question will be scheduled for testing (that is, it'll be written to the "queue" table, where the dispatcher will pick it up).

The change detector stores data between runs in a simple text file, with a zone name and its corresponding MD5 sum on each line.

At the moment, the time it takes to do the AXFR is limited by the speed at which the Perl module Net::DNS can create new records. It also uses a fair bit of RAM (100-150MB for around 750 thousand domains), so if you're planning to run it against ".com" you'll want a pretty hefty machine.

The results mailer

The dnscheck-12hourmailer script reads results of tests from the database tables "tests" and "results", aggregates them per registrar and sends an email to each registrar with a summary of serious problems found among their domains.

This is pretty specific to ".SE", particularly the bit that maps domain names to registrars and fetches contact information for the registrars. If you want to do something similar to this, you'll probably need to modify the code of the script. At the moment, the emails are also generated entirely programmatically, so changing their formatting also involves changing code.

The preflight script

dnscheck-preflight is used by the web GUI. It takes a name as a command line argument and prints the string TRUE if it's a domain and FALSE otherwise.

The hostsyntax script

dnscheck-hostsyntax is also used by the web GUI. It takes a name as a command line argument, and prints the string TRUE if it is a syntactically valid hostname and FALSE otherwise.

The CLI interface

dnscheck is a command line tool to check domains. Run it with the switch "--help" to get a short description of how to use it.

  The database
It's entirely possible to use the lib part of DNSCheck without a database. All the surrounding functionality, however, needs one. All communication between various parts of the entire system is done entirely through database tables. While this may not be the most convenient interface ever, it does mean that anything that can talk to MySQL can be a part of the system, which is useful.

There are a number of tables. Let's start looking at them from the top of, so to speak, the food chain.

The source table

Anythingt that can write to the "queue" table can be a source of domains to be tested. In order to make it possible to keep track of which source requested which test, there is the "source" table. It has only three fields, and is supposed to be mostly read-only. The fields are "id", "name" and "contact". The id is an autoincrementing integer as usual. The name is a human-readable name for the source. The contact is some sort of human-useable contact information for the source in question (email address, phone number, ham radio call sign,...).

One example of the use of source information is the mailer described above, which only mails results about tests that were scheduled for testing by the zone change detector, not once that came from (for example) the web GUI.

The queue table

This table has six fields apart from the autoincrementing "id". There is "domain", which is the name of a domain to be tested. There is "priority", which is an integer. dnscheck-dispatch will always pick the queue entry with the highest priority first (and if there are several with the same priority, the one with the lowest id). There is "inprogress", which is NULL to begin with and gets a timestamp written to it by the dispatcher when it's been scheduled for testing. There is a "tester_pid" field, where the particular child process that's running the test for the domain writes it's system PID (if we ever need to have more than one machince running tests, thre maybe a host field added here). "source_id" pints into the source table. "source_data" is any data specific to this test that is relevant for the specific source. The two source fields are simply copied over to the "tests" table when a test is started.

The tests table

This table has one row for every test run, an contains information on the domain tested, the start and finish times of the test, a summary of the results and the source information that was copied from the queue table. The fields are: "id" as usual, "domain" for the domain name, "begin" for the start time (with whole-second resolution), "end" for the finish time, "count_critical", "count_error", "count_warning", "count_notice" and "count_info" for the number of problems at the respecive severity levels, "source_id" to point into the source table and finally "source_data" to hold whatever data that source wants.

The results table

This table has a number of rows for every test. They reflect the list of results returned after a test is run, and several of the fields are rather specific to the web GUI application. The fields that can be useful to others are: "test_id", which points into the "tests" table. "timestamp" with is the time at which the log entry was entered (with microsecond resolution). "level" which is one of the all-caps strings CRITICAL, ERROR, WARNING, NOTICE or INFO. "message" which is the message tag that describes the reason the log entry was made. "arg0" to "arg9", which are arguments to the message.

The message tags are specified in YAML files.

The messages table

The messages table contains a copy of a localy YAML file. It is used by the web GUI.

The delegation_history table

This table contains information about which nameservers has historically been authoritative for domains. Some tests use it, if it's available.

The nameservers and nsp tables

The function of these tables have been obsoleted by the registrar system.

The domains table

A list of domains. More of a convenience thing than an actual part of the DNSCheck system.

Needed permissions

The various programs need different sets of permissions in the database in order to work correctly. It breaks down as follows:

•   Everything that schedules tests needs INSERT, UPDATE and SELECT on "sources" and INSERT on "queue".

•   Everything that reads test results needs SELECT on "sources", "tests" and "results".

•   Anything that runs tests needs SELECT on "delegation_history" (or no database access at all).

•   dnscheck-dispatcher needs a lot: INSERT, SELECT, UPDATE and DELETE on "queue", INSERT, SELECT and UPDATE on "tests", INSERT on "results" plus everything listed above.

In practice, the easiest reasonably secure setup is to give the web GUI one user with very limited permissions (INSERT, UPDATE, SELECT on "sources" and INSERT on "queue"), and one user with full rights on all tables for the rest of the system. Indeed, since all of the programs use the database connection info given in the config.yaml file, you will need to point out a different such file to any programs you want to use different connection information.

  Files
The system uses a number of files.

/etc/dnscheck/config.yaml

This contains configuration information in YAML format. It overrides what is in the default configuration, and is where local changes (such as database access information and TSIG keys) should be entered.

One key that is usually not in the configuration but that can be added if needed is the "disable" key. It has two levels under it, being the name a test module (converted to lowercase) and the name of a method in that module. If the value for the method is true, that method will not be run. Note that no kind of sanity checking is done, and that some tests rely on each other, so disabling one may make others give false results. Be cautious and know what you're doing. As an example, the following would disable all DNSSEC tests:

  disable:
      dnssec:
         test: 1

All test methods mentioned in the documentation for the respective modules are possible to disable this way.

It is also possible to set log message filters in the config file (see DNSCheck::Logger for more about filters). The format for this is a hash of lists of hashes, all under the top-level key "filters", so the syntax gets a bit convoluted. Here is an example:

   filters:
       'SOA:BEGIN':
           -
               args:
                   - iis.se
               level: 'WARNING'

This adds a filter that sets the log level of a SOA:BEGIN message for the "iis.se" domain to WARNING.

policy.yaml

"policy.yaml" specifies the severity of various problems. The default values represent .SE's view on best practices for DNS. If you want to change anything, do so in "/etc/dnscheck/policy.yaml".