NONE TITLE 2.html
Keywords: NONE TITLE,fml, flexible mailing list manager, not listserv style but fmlserv emulates listserv or majordomo style

[PREVIOUS CHAPTER] [______TOC_______]

2 Mail Traffic Information

    2.1	How to

    How to customize MTI ? The most important variables are

    	$MTI_BURST_SOFT_LIMIT
    	$MTI_BURST_HARD_LIMIT

    IF you change this value smaller, even small traffic can be a bomb.
    In default the limit is about 1, sequential in-coming 12-15 mails is
    the limit of a bomb or not.

    2.2	Traffic Monitor

    FML 2.2 (after 2.1B) has internal traffic monitor system called "MTI".
    It checks the traffic and detects intermittency (burst) effect of
    traffic against mail bombing. It is applicable to trap mail loops
    for a bonus:) though several loop checks must reject the loop before
    MTI routines.

    Firstly to trap mail bombing is to detect the intermittency effect /
    burst of mail traffic. This logic is effective except for the case
    burst that traffic from network errors and hosts over UUCP or DIP
    service.  We cannot determine the burst occurs from network errors or
    bombings. We should observe the burst to assume the possibility that
    we regard network errors as mail bombing.

    In all cases, how effect MTI depends on the fake level of injected
    mails.  Since SMTP has no authentication now, we cannot believe
    anything except for the time when mail kicks off fml.pl.

    2.3	Mail Traffic Information

    Mail Traffic Information (MTI) is FML built-in traffic monitor. It
    monitors the traffic and logs (via Berkeley DB) the traffic in the
    past 1 hours (default) with the date and From:, Return-Path:, Sender:
    fields et.al.

    2.4	MAIL BOMBING Evaluation Function

    Here we use TeX statements to show some mathematical concepts.

    Consider a time sequence of mails incoming to fml.pl. Let the time
    when mail m_i comes and runs fml.pl be t_i, the Date: time of the
    mail be d_i. "i" is an index number. It has the same order as the
    article sequence number ordering.

    MTI caches a set (t_i, d_i) for m_i. The key of database is addresses
    of e.g. From:, Sender:, Return-Path:. MTI discards data entries after
    $MTI_EXPIRE_UNIT (3600 sec in default).

    In default MTI evaluation function is

    	\sum 1/  | t_i - t_j + \epsilon |

    where \epsilon is against divergence. |t_i - t_j| is the difference of
    time between when fml.pl runs.  It becomes smaller if mails comes
    continuously. We sum up the inverse of it, so the sum becomes large if
    the burst occurs. If the sum is over some thresholds, we think our
    server is attacked!

    This threshold is typically 0.2. The choice of this value is sensitive.
    For example, we evaluate it in the following way

    		i=N
    	E{t} =	\sigma 1 / | t_i - t_j + \epsilon | > \sigma 1/M \sim N/M
    		i=1

    	| t_i - t_j + \epsilon | < M

    where M = 10, N = 5. It yields N/M = 0.2. It means our limit is that
    5 mails comes continuously in the time slice of 10 seconds.

    This logic has a problem. We cannot distinguish network errors and
    bombings only based on the in-coming mails burst trap.

    To distinguish network errors or bombing, we calculate a sum of
    sequences d_i (Date:) in the same way. So consider

    	E{d} =	\sigam 1 / | d_i - d_j + \epsilon |

    Usually we edit mail draft and post it on random time sequence.
    Hence the time of Date: is random. When network errors delay mail
    delivery, Date: is random (E{d} is small) and fml.pl runs in burst
    (E{t} is large). When a real mail bomb attack you, both E{d} and E{t}
    must be large.

    A bad MUA may send plural mails in the same Date: ;_; In such a case,
    E{d} >> 1 in one mail. We avoid this by limit the minimum for |d_i -
    d_j |.

    	\sigma 1 / ( | d_i - d_j | < 3 ? 3 : | d_i - d_j | )

    2.5	Evaluation Function (default)

    Let E{d} and E{t} be the same sum described above.
    We consider the following evaluation.

    1	soft limit

    	If mail bombing (as describe above)
    	E{d} > E{t} or E{d} is nearly equal to E{t}.
    	If E{d} < E{t}, network errors may occur.
    	If E{d} > E{t} and E{d} > $MTI_BURST_SOFT_LIMIT,
    	mail bomb attack comes in, so we reject them.

    2	hard limit

    	Condition 1 is not effective if the mail header is faked.
    	So we use another absolute limit.
    	If E{d} > $MTI_BURST_HARD_LIMIT or
    	   E{t} > $MTI_BURST_HARD_LIMIT,
    	we reject in-coming mails.

    3 	">" (greater than)

    	The value of sum up has some measurement errors.
    	We need some margin for ">" evaluation.

    	if ( E{d} > E{t} ) {
    		if ( E{d} > $MTI_BURST_SOFT_LIMIT) {
    			mail bomb
    		}
    	}
    	else {  # if the mail header is a fake

    		if ( E{t} > $MTI_BURST_HARD_LIMIT ) {
    			mail bomb
    		}
    		if ( E{d} > $MTI_BURST_HARD_LIMIT ) {
    			mail bomb
    		}
    	}

    2.6	MTI Configuration Variables

    	$USE_MTI

    Enables MTI function to work. If not set, MTI does not work.

    	$MTI_BURST_SOFT_LIMIT
    	$MTI_BURST_HARD_LIMIT

    described above.

    	$MTI_BURST_MINIMUM

    described above.

    	$MTI_COST_EVAL_FUNCTION

    Evaluation Function you write.

    	$MTI_COST_EVAL_HOOK

    HOOK

    2.7	Maximum Limit of Traffic

    If the evaluation function is \sigma 1, we can count the mail traffic
    in the unit time. The cost does not count the burst effect but is
    the usual average. The unit is $MTI_EXPIRE_UNIT.

    	$MTI_DISTRIBUTE_TRAFFIC_MAX

    The maximum of traffic of distribution mails from an address.

    	$MTI_COMMAND_TRAFFIC_MAX

    The maximum of traffic of distribution command mails from an address.

    2.8	Other variables

    	$MTI_EXPIRE_UNIT

    cache life time.

    	$MTI_APPEND_TO_REJECT_ADDR_LIST

    If set, we inject the address we determined as a bomber to
    $REJECT_ADDR_LIST ($DIR/spamlist). However it may be not effective
    since $REJECT_ADDR_LIST is rejection based on From: address checks.

    2.9	Files

    	$MTI_DB
    	$MTI_HI_DB

    	$MTI_DIST_DB
    	$MTI_HI_DIST_DB
    	$MTI_HI_COMMAND_DB
    	$MTI_COMMAND_DB

    cache files.

    	$MTI_MAIL_FROM_HINT_LIST

    This file ($DIR/mti_mailfrom.hint) is a list we regard as a bomber.
    It is a hint. How you apply this to your operation depends on you.

    2.10	Arguments of function

    Here is Beta test specification:)

        $fp = $MTI_COST_EVAL_FUNCTION || 'MTISimpleBomberP';
        &$fp(*e, *MTI, *HI, *addrinfo, *hostinfo);

    	%Envelope	Envelope

    	%Envelope	Envelope
    	%MTI		address => time sequence
    	%HI		host => time sequence
    	%addrinfo	address
    	%hostinfo	host evaluated from Received: fields

    2.11	perl 5 tie

    	$MTI_TIE_TYPE

    e.g. DB_File, NDBM_File, ...

    If defined, fml uses "tie" function in MTI sub system irrespective of
    dbmopen(). This is used only on perl 5. What you use depends on
    your operating system. Please see your OS manuals and perl.

    2.12	Negative cache of warning mails

    negative cache to warn the burst traffic to maintainers

    2.13	DB type

    	$MTI_TIE_TYPE

    use "tie" function in MTI sub system.

    3	several size limits

    3.1	Limit of size for a posted article

    You can restrict a posted article size. The maximum size is defined by

    		$INCOMING_MAIL_SIZE_LIMIT

    where the unit of this variable is "bytes". If

    		$NOTIFY_MAIL_SIZE_OVERFLOW (default 1)

    is set, fml notifes the rejection to the sender.

    If message/partial style mail is given, fml speculates the total size.
    fml rejects it if the speculated total size is over
    $INCOMING_MAIL_SIZE_LIMIT.

    If

    		$ANNOUNCE_MAIL_SIZE_OVERFLOW (default 0)

    is defined, announce "somebody sends too big mail to ML" to ML.

    3.2	Limit of the number of members in a ML

    	$MAX_MEMBER_LIMIT

    is the maximum number of members. It may be useful in automatic
    registration mode.

    3.3	Example: to discard an over length mail

    We discard mail with over 1000 lines. Today please use
    $INCOMING_MAIL_SIZE_LIMIT for incoming mail size upper limit.

    $START_HOOK = q#
        if ($Envelope{'nlines'} > 1000) {
    	&Warn("Discarded on the behalf of too Large Mail", &WholeMail);
    	$DO_NOTHING = 1;
        }
    #;

    3.4	$START_HOOK: limit the number of member

    A file $LIMIT_OVER_FILE is to say "Sorry for that I cannot regist you
    since this ML is over the limit of ML member".
    Ref: START_HOOK => ../hooks 3.1

    $START_HOOK = q%;

    $MAX_MEMBER = 100;

    $LIMIT_OVER_FILE = "$DIR/limit.over";

    sub WC
    {
        local($f) = @_;
        local($lines) = 0;

        open(TMP, $f) || return 0;
        while (<TMP>) {
    	next if /^\#/;
            $lines++;
        }
        close(TMP);

        $lines;
    }

    if (&WC($ACTIVE_LIST) > $MAX_MEMBER) {
        &SendFile($From_address,
    	      "Sorry, the mailing list member exceeds the limit $ML_FN",
    	      $LIMIT_OVER_FILE);
        $DO_NOTHING = 1;
    }

    %;