How to install SMTPfeed

1. Installation of SMTPfeed

  1.1  Do "./configure"
       If you want to specify directory for installed bind4, use switch:
	 --with-bind4=/usr/local/bind4
       If you want to link bind8, use switch:
	 --with-bind8
       If you also want to specify directory for installed bind8, use switch:
	 --with-bind8=/usr/local/bind8
       If you want to enable IPv6 feature, use switch:
	 --enable-ipv6
       Destination directory for installation can be specified with switch:
	 --libexecdir= (default of destination directory is /usr/libexec
	 or /usr/lib (in case /usr/libexec is missing).)
       If you want to link libraries statically, use switch:
	 --with-static (-static will be added to LDFLAGS if CC is gcc,
	 -Bstatic will be added otherwise.)
       If you want to make FD_SETSIZE lager than system default, use switch:
	 --with-fds=NN (here, NN is a number for FD_SETSIZE. See select(2)
	 for default of FD_SETSIZE. Use limit(1) to see limit of file
	 descriptors on your system.
  1.2  Edit "Makefile" (If you need)
  1.3  Do "make all"
  1.4  Do "make install"
  1.5  Configure "syslog.conf"
       For example, declare as follows:
           local5.info /var/log/smtplog
       Default facility is "local5". facility can be changed with command
       line switch of smtpfeed.
       do not forget to touch logfile and to send signal to make syslogd
       reload (kill -HUP {PID of syslogd}).
  1.6  test
       edit TEST.in and try:
           % smtpfeed -dA -lstderr < TEST.in
       switch -dA is to get all debugging messages and switch -lstderr is
       to see the messages on a terminal.


2. Compilation of sendmail

  2.1  Get a copy of sendmail.8.11.x archived package and extract files 
       from the archive.
  2.2  Apply 3.3W.patch
       You may need new version of "GNU patch", because the .patch file
       includes "diff"s to non-existent files.
  2.3  Do "sh Build"
       Do not forget to configure Makefile around DB or BIND to fit your
       site.


3. Configuration of sendmail.cf

  In this section, three ways are described:
   A. Changing current sendmail.cf directly
   B. Using m4 technique
   C. Using CF configuration tool

  3.A  Changing current sendmail.cf directly
       Do not forget to save current version of sendmail.cf.

    3.A.1 Definition of smtpf mailer
          Duplicate definition lines on smtp (or esmtp, smtp8 etc.) mailer
          and make changes as follows:
              * Change mailer name into "smtpf"
              * Change value for P= equate from P=[IPC] to 
                P=/usr/libexec/smtpfeed (location smtpfeed is installed)
              * Append flag z to F= equate (to use LMTP) and verify that
                flag m is included already.
              * Change value for A= equate to "smtpfeed" (with command
                line switches if needed)
                For example, if SMTPfeed does not recognize FQDN of your
                hostname, specify -c $j as a command line switch.
                Note that every host names appeares in right hand side
                of any MX RRs of DNS must defined with -c or -a switch
                of SMTPfeed so that loopback detection will be performed
                certainly.
                Specify -i$i to make queueID of a message be logged in
                syslog. When a delivery is splitted into multiple
                transactions because of large number of recipients,
                the queueID information will not be passed to 2nd or later
                transaction (cleared by RSET command). If you want to
                pass the queueID information to later transaction, specify
                $?{runqueue}-i$i$|-I$i$. instead (only with sendmail 8.9.1
                +3.1W or later). This means -i$i will be passed when in
                runqueue processing and -I$i will be passed when in
                immediate delivery processing.
		You can also specify -E option instead of -i/-I options
		if you trust "id" portion of Received: header line
		as a queueID inserted by sendmail. In this case, queueIDs
		for all transactions in runqueue process will be obtained.
              * line terminating character (E= equate) may be "\n" or
                "\r\n".

          (example)
          Msmtpf, P=/usr/sbin/smtpfeed, F=mDFMuXz, S=11/31, R=21, E=\n, L=990,
                  T=DNS/RFC822/SMTP, A=smtpfeed

    3.A.2 Replace smtp (or esmtp, smtp8) mailer into smtpf mailer
          (at mailer selection rules)

          When you change mailer name on rules, the host portion (after $@)
          of the mailer dispatching triple also be changed to be a constant
          character string like "LMTP" or something else. This is required
          to pass all recipients at a time.

          (example)
          R$* < @$* > $*	$#esmtp $@ $2 $: $1 < @ $2 > $3
          must be changed to
          R$* < @$* > $*	$#smtpf $@ LMTP $: $1 < @ $2 > $3
                        ^^^^^^^^ TAB


    3.A.3 disable $[ $] operators
          Comment out all rules which use $[ $] operators for address
          completion since process by $[ $] may cause delay. But this
          is not required to use SMTPfeed.

          (example)
          R$* < @ $* $~P > $*		$: $1 < @ $[ $2 $3 $] > $4
          should be changed to
          #R$* < @ $* $~P > $*		$: $1 < @ $[ $2 $3 $] > $4
                              ^^^^^^^^^^ TAB


  3.B  use of m4 technique
       use the following definitions in .mc file:

          FEATURE(nocanonify)dnl
          MAILER_DEFINITIONS
          Msmtpf, P=/usr/sbin/smtpfeed, F=mDFMuXz, S=11/31, R=21, E=\n, L=990,
                  T=DNS/RFC822/SMTP, A=smtpfeed
          MAILER(smtp)dnl
          define(`SMART_HOST', `smtpf:LMTP')dnl

       If you are using mailertable feature, specify 'smtpf:LMTP' in database.


  3.C  use of CF configuration tool
       use the following definitions in .def file:

          LOCALHACKMAILER='# smtp feed mailer\
          Msmtpf,   P=/usr/libexec/smtpfeed, F=mDFMuXz, S=11, R=21,\
                    T=DNS/RFC822/SMTP, E=\n, L=990,\
                    A=smtpfeed'
          # specify A=smtpfeed -c $j in case hostname is not a FQDN.
          DIRECT_DELIVER_DOMAINS=none
          DEFAULT_RELAY='smtpf:LMTP'

       Be sure about install path of smtpfeed described at P= equate.
       If you are using mailertable feature, specify 'smtpf:LMTP' in database.

  And test as follows with sendmail.cf created by ways of A, B or C:
       > 3,0 user@domain
       rewrite: ruleset   3   input: user @ domain
       rewrite: ruleset  96   input: user < @ domain >
       rewrite: ruleset  96 returns: user < @ domain >
       rewrite: ruleset   3 returns: user < @ domain >
       rewrite: ruleset   0   input: user < @ domain >
       rewrite: ruleset  88   input: < smtpf : LMTP > . user < @ domain >
       rewrite: ruleset  88 returns: $# smtpf $@ LMTP $: user < @ domain >
       rewrite: ruleset   0 returns: $# smtpf $@ LMTP $: user < @ domain >
  It is OK if smtpf is selected.


4. Farther configuration of sendmail.cf

  4.1 Number of recipients par transaction

   In specification of STMP (RFC822), maximum number of recipients par
   transaction is defined as 100. There is another limitation in sendmail
   caused by static buffer size for syslog, so actual maximum number is
   less than 100. When DYNAMIC_TOBUF feature is used, the maximum is
   relaxed to 100. But there is still a limitation by the specification.
   By this specification, parallelism of SMTPfeed will not be fully
   utilized. This limitation can be more relaxed locally, you can make
   sendmail pass all recipients at once to gain performance with fully
   parallelism.
   If you apply 3.4W.patch to sendmail, r= equate at mailer definition
   is available in sendmail.cf to specify maximum number of recipients
   par transaction from sendmail to SMTPfeed. If you specify r= equate
   like:

          Msmtpf, P=/usr/sbin/smtpfeed, F=mDFMuXz, S=11/31, R=21, E=\n, L=990,
                  T=DNS/RFC822/SMTP, r=1000, A=smtpfeed

   in sendmail.cf, you can relax maximum number of recipients par
   transaction to 1000. Default value is 100.
   If there is enough memory, the value for r= should be more than
   number of recipients in every mailing-lists.

   If ConnectionCacheSize in sendmail.cf is defined with 1 or more,
   the same smtpfeed process manages multiple transactions. If you
   want make new smtpfeed process for each transactions, set
   ConnectionCacheSize to 0. With 3.2W.patch, you can suppress caching
   feature on specific mailer with mailer flag ';'.

  4.2 Suppression of SMTPfeed execution for few recipients

   If recipient is only one for a message, smtpfeed should not be
   executed since execution of SMTPfeed requires one more process
   resource but no speed improvement will be performed.
   Moreover, SMTPfeed stores all information in core memory, delivery
   of a huge message is not fit to SMTPfeed.
   So, execution of smtpfeed should be controlled by number of recipients
   or size of a message. Here is a sample configuration with MULTI_MAILER
   feature provided by 3.4W.patch.
   With this MULTI_MAILER feature, you can specify multiple mailers
   in mailer dispatching rule sets as follows:
   
          R$* < @$* > $*	$#smtpf $@ LMTP $: $1 < @ $2 > $3
                                $#esmtp $@ $2.  $: $1 < @ $2 > $3
          (Be sure that there is no TAB between $3 at the end of 1st line
           and $# at the top of 2nd line. There are TABs only between
           $* and $# in 1st line.)

   By using this rule, if delivery by 1st mailer is failed with TEMPFAIL,
   delivery will be re-tried with 2nd mailer. Trial with 1st mailer can
   be suppressed by settings on least number of recipients (*= equate
   on smtpf mailer definition line (starts with M) or largest size of a
   message (M= equate). There are also 2 mailer flags to control the
   suppression: mailer flag '!' is for suppression on run-queue process,
   mailer flag '_' is not for use of following mailer if the 1st mailer
   is used. Sample full specification of the above is:

          Msmtpf, P=/usr/sbin/smtpfeed, F=mDFMuXz!_,
                  S=11/31, R=21, E=\n, L=990, *=5,
                  T=DNS/RFC822/SMTP, M=50000, A=smtpfeed

   By this definition, If there is 2nd mailer in dispatching rule and
   number of recipients is less than 5, delivery with smtpfeed will
   be skipped (If there is no 2nd mailer, definition *=5 is ignored
   and smtpfeed will be used). If M= equate is defined, when size of
   a message is larger than the size specified (50000) and 2nd mailer
   exists, delivery with smtpfeed will be skipped instead of bouncing
   the message. If mailer flag '!' is specified, the mailer will be
   used only for the 1st trial of delivery and will not be used by
   2nd or later run-queue process. If mailer flag '_' is specified,
   when the mailer is used for trial of delivery, following mailer
   will not be used in spite of result of delivery (this mailer flag
   should be used with *= or M= equates).

   Configuration to use MULTI_MAILER feature with CF may be as follows:

          LOCALHACKMAILER='# smtp feed mailer\
          Msmtpf,   P=/usr/libexec/smtpfeed, F=mDFMuXz!_, S=11, R=21,\
                    T=DNS/RFC822/SMTP, E=\n, L=990, M=50000, *=5,\
                    A=smtpfeed'
          DIRECT_DELIVER_MAILER=smtpf
          DIRECT_DELIVER_DOMAINS=all
          #DEFAULT_RELAY=
          DSPTLOCALHACK='R<smtpf:$+>.$*	$#smtpf$@LMTP$:$2$#smtp$@$1$X$:$2'
                                       ^TAB
   Be sure about install path of smtpfeed described at P= equate.
   Configuration to use MULTI_MAILER feature with m4 technique may be
   as follows:

          FEATURE(nocanonify)dnl
          MAILER_DEFINITIONS
          Msmtpf, P=/usr/sbin/smtpfeed, F=mDFMuXz!_, S=11/31, R=21, E=\n,
                  L=990, T=DNS/RFC822/SMTP, M=50000, *=5, A=smtpfeed
          MAILER(smtp)dnl
          define(`SMART_HOST', `smtpf:LMTP')dnl
          LOCAL_CONFIG
	  # These rules should be inserted at the top of S95
	  S95
	  R< smtpf : LMTP > $*<@$*>$*	$#smtpf$@LMTP$:$1<@$2>$3$#smtp$@$2$:$1<@$2>$3
          R< smtpf : $* > $*		$#smtpf$@LMTP$:$2$#smtp$@$1$:$2
          R< smtpf > $*<@$*>$*		$#smtpf$@LMTP$:$1<@$2>$3$#smtp$@$2$:$1<@$2>$3
                              ^^^^^^^^^^ TAB

  4.3 Timeout on LMTP DATA state

   When sendmail executes SMTPfeed, it waits for termination of SMTPfeed
   and gets delivery status by response of DATA command of LMTP.
   If SMTPfeed is not terminated during a period, sendmail will give up
   to wait the termination and assume deliveries for all recipient
   were not successful and save the message to queue for next trial.
   By this reason, if SMTPfeed will not be finished during timeout period,
   some recipients may be accept copies of the message twice or more.
   To avoid duplication, SMTPfeed must be finished during timeout period.

   By default, SMTPfeed will be finished within 57 minutes (If it takes
   55 minutes, SMTPfeed switches to termination process (sending QUIT
   commands) and wait for response of QUIT commands (at most 2 minutes)).
   Then, SMTPfeed will reports delivery status at the moment. So, only
   recipients to which deliveries are not completed are queued.
   Value for this timeout can be set by -t total= command line option
   of SMTPfeed. If you set 60 minutes, specify -t total=60m (will take
   62 minutes at most).

   Timeout at side of sendmail can be configured by 'O Timeout.datafinal='
   option in sendmail.cf. Default is 1 hour (1h).


5. Delivery test with new sendmail and SMTPfeed

        # newsendmail -C newsendmail.cf addresses < file

   Please verify logged messages of maillog or smtplog.


6  Replace of sendmail and sendmail.cf, and restart sendmail

   Good luck!

7  Caution

   BIND resolver implemented in SMTPfeed will utilize additional information
   section of answers sent from BIND servers. To avoid problems reported by
   "CERT Advisory CA-97.22: Topic: BIND - the Berkeley Internet Name Daemon,"
   version of BIND servers which serve for SMTPfeed must be 4.9.6/8.1.1 or
   later. There are, of couse, other problems reported by "CERT Advisory
   CA-98.05," you should use 4.9.7/8.1.2 or later.

   Your mail server will receive many many IDENT(RFC1413) request from SMTP
   peers at a time since the SMTPfeed will open much many SMTP connections
   at once. This may introduce your server into heavily loaded condition
   if "identd" executed from "inetd" with "nowait" mode. In this case,
   you may want to disable "identd" or execute daemon "identd" to avoid
   such bad condition.

   If you want to reduce number of connections requested at a time, use
   option -n of the SMTPfeed instead of reducing number as a parameter
   for r= equate of mailer definition in sendmail.cf. Reducing value at
   r= makes the SMTPfeed less effective.
