** sp4si Script package for Suck & INN **
V0.95b 08.02.1999
[ CHANGES | DEUTSCH ]
Introdution:
This script package is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. The package is distributed in the hope that it will be useful but WITHOUT ANY WARRANTY.
(c) 1998 Peter Sobisch, All rights reserved.
Because at the beginning are many elementary questions and nobody wants to read long and tedious manual, I decided to write this as a question-answer document. This text is written as WWW-Page and as regular manual (included to sp4si package).
Contens
- What is sp4si ?
- What does sp4si ?
- Why and when I need sp4si ?
- What I need to use sp4si ?
- How does it works ?
- Syntax and functions
- configuration: env, newshosts and other
- active - scripts: get.active and rebuild.active
- spool - scripts: nntp.up, nntp.down, uucp.batch and uucp.cico
- subscriptions - script
- Installation script: install.sh
- Installation
- Configuration: first steps
- Tips
- inserting of local newsgroups
- automatic update of active file
- Where to get sp4si ?
1. What is sp4si ?
sp4si is a script package, which makes the management with INN and suck much easier. It keeps the configuration level be increased count of newshosts on the same level. sp4si is originated because of misery and is designed for everybody who uses a dial-up internet connection and wants to read the news offline on his own local INN newssystem or for everybody who just wants to expand his INN newssystem.
In the beginning sp4si was just a small script for automatically subscribing the newsgroups on the local INN system and it becames to automatically multihost- and multitaskingable spool tool for INN and suck newssystems.
[Back]
2. What does sp4si ?
sp4si is able with a single script run to spool asynchronous all configured news hosts, optional spool reports will be sent with the result of the spool.
sp4si is able to download automatically the active file from each configured news host, filter the news host active files, generate a new active file and replace the old acticve file INN conform.
sp4si is able to rebuild the suck config files in dependence of each newsgroups subscribed in user .newsrc file.
sp4si is fully replaceable for get.news and put.news from suck package. It's able to spool a UUCP via TCP site too (if available).
sp4si isn't a configuration tool for INN, so INN must be configured manually, sp4si will make the daily work much easier as before.
[Back]
3. Why and when I need sp4si ?
if you often subscribe and unsubscribe often many newsgroups
if you want to have more control and synopsis about your newssystem and want to know if the last spool was successfully done.
if you want to simply expand your newssystem with more than one newshost
if your bandwidth to your newshosts is smaller as your connection speed, you can spool all hosts parallel and save you time and money.
if your news provider offers news by UUCP via TCP
[Back]
4. What I need to use sp4si ?
You will need the following packages:
It requires a configured and running INN newssystem (local configuration) and UUCP (if needed). How to very simply configure the INN system you can read in the german (sorry) manual from Carsten Voss "Einrichtung eines News-Systems unter Linux". Furthermore you must have suck package installed and the suck command must be in your PATH environment (for example you can copy this to /usr/local/bin directory).
[Back]
5. How does it work ?
sp4si consists of some shell scripts which can be divided into three parts and two configuration files: newshosts and env. The newshosts file contains the site- and hostnames which have to be spooled. The env file contains the environment variables and will be included at the runtime by most scripts of sp4si.
The first part of the sp4si serves the active file management, the second part spools the news articles and the last part takes care of automatically newsgroups subscribing.
Nearly all scripts will read the newshosts file and apply on valid entries their activities. This is an advantage, if you want to add a new newshost, you have only to insert it at the newshosts and newsfeeds (INN) file. Because of the central environment sp4si makes the management much simpler and increases the productivity of your newssystem. If you want to use the spool frontend spool.news (is recommended), all spool activities will be logged and you could optionally send the spool reports with the spool result by email or post it to a defined newsgroup.
[Back]
6. Syntax and functions
6.1. Configuration files
env - it will be created by the installation (with security question if already exists) it will be placed in the main directory of sp4si. It contains variables definitions to influence the environment and control in central point. All options are commented.
newshosts - it will be created by the installation (with security question if already exists) it will be placed in the main directory of sp4si. Each entry for site in separate line. Comments can be introduced by a '#' at begin of each line. An entry has a following syntax: <site> <newshost> [<up/down> [<packer> <gupmail> <password>]], where:
<site> has the same name as the correspondent entry in the newsfeeds file
<newshost> contains the host's DNS name to reach the host on internet
<up/down> decribes the spool type, valid entries are:
- nntp/nntp (or just nntp): upload and download of articles by NNTP
- nntp/uucp: upload by NNTP and download by UUCP
- uucp/uucp (or just uucp): upload and download by UUCP
- uucp/nntp: upload by UUCP and download by NNTP
<packer> is the packer for outgoing batches to be packed by: gzip, compress or cat (unpacked)
<gupmail> is the mail address of an account where gup(1) serves the subscription.
<password> the password for your own site, configured by your news provider, the site name will be extracted with uuname -l. Please look at UUCP-HowTo to configure your own UUCP site. All options described before are commented in newshosts file too.
localhost.active
will be created on the installation (with security question if already exists) and is placed in the site directory. Enter your local newsgroups active(5) comform. These newsgroups will be considered by the next run of rebuild.active.
localhost.newsgroups
will be created on the installation (with security question if already exists) and is placed in the site directory. Enter here the descriptions for your local newsgroups. The file will be not yet considered.
Nearly all scripts allow to invoke they with -h or --help or -? parameter to show the syntax and help.
[Back]
6.2. active scripts
get.active
Syntax: get.active [ -n ]
it downloads the active files from all newshosts which are in the newshosts file and saves them to <site.active> files in site subdirectory (see env file). Optional the newsgroups description file could be downloaded too, but the other sp4si scripts are not yet able to handle this. This can be activated by giving the -n parameter.
rebuild.active
Syntax: rebuild.active
creates a new active file from the site's own active files considering the site's subscriptions in the newsfeeds file, stops INN, moves the new active to the local active file, loads all INN's configuration files and restart them.
ATTENTION: before running these scripts you must have already configured the newsfeeds file. Because of performance (the newsfeeds file must be not interpretated for every run of subscribe) spool files will be created and placed into site/<site> directory. This files contain names of newsgroups which are to be spooled by a certain host and are required for the autosubscribing procedure.
[Back]
6.3. Spool scripts
spool.news
Syntax: spool.news [-s]
a spool frontend with multitasking and multihosting ability. Therewith will be spool scripts: nntp.up, nntp.down, uucp.batch and uucp.cico invoked automatically, the scripts output will be put together, make a report of it and sent it via email or post it to a certain newsgroup. The -s parameter turn off multitasking spool and run the spool in synchronous mode, it is slower as the default multitasking mode but it could be a little bit securer (no warranty).
nntp.down
Syntax: spool.down <site> [ logfile ]
it spools the news articles down by NNTP from the newshost using suck(1) and post it to the local host running INN using innxmit(1). nntp.down expects as parameters the <site> and resolve from it the real hostname from the newshosts file, the stdout-output could be optionally redirected to a file by giving the filename of it: <logfile>. nntp.down will be started automatically by the spool.news script, but you can start it alone too (for debuging) if you want. Normally the downloaded articles will be posted locally running innxmit(1), you could appoint it to run rnews(1) instead by changing the LOCALPOST variable in the env file.
nntp.up
Syntax: spool.up <site> [ logfile ]
it spools the articles up by NNTP to remote host running rpost(1) and applying out.filter, for online parameters have a look at nntp.down script.
uucp.batch
Syntax: uucp.batch <site> [ logfile ]
it spools all outgoing articles for the certain site using batcher(1) to rnews(1)-batches which are sent later by uucp.cico out. It applys the out.filter to filter the outgoing batches which calculate new lengths of each article and replaces the old length with it. If an alternate packer is specified in newshosts, this will be used (possible are: gzip, compress and cat). On unknown packer the default packer compress will be used. Other packer will be implemented by the author by enough user's inquiry. To the right packer choice ask your news provider. Parameter <logfile> has the same meaning as by nntp.* scripts. Because of no information running batcher(1), uulog(1) will be started and used to find informations out from the UUCP-Logfile.
uucp.cico
Syntax: uucp.cico <site> [logfile]
invokes a UUCP call to the certain site with uucico(1). The prepared baches with uucp.bach will be sent out and new batches received and executed. Parameter <logfile> has the same meaning as by uucp.batch script.
out.filter
Syntax: out.filter [<infile> [<outfile>]]
this script has a special meaning and will be used for filtering using awk(1) and absolute required for sending articles. This filter deletes any configured header lines (Path: ... will be replaced with Path: not-for-mail) and calculates in rnews(1)-mode new length of each articles after filtering. The out.filter can be configured by changing the var FILTER_STRIP in env file. Here the header lines to be deleted inserted into string and separated with blank space.
Notice: this filter will only filter header lines (until empty line), not the message body. If the first line of a message begins with '#! rnews nnn' the filter will switch into 'rnews'-mode.
[Back]
6.4 subscribe script
subscribe
Syntax: subscribe
subscribe reads all newsrc files from each user's home directories on your local system (for that reason it must be started by root) and creates by comparing the subscribed newsgroups with the site/<site>/spool files a new subscription (see below) config files. subscribe can be started before each spool run too. For the right interpretation of newsrc files, they must be in newsrc format, because of that there many newsreaders will be supported, such as: slrn, tin and similar and Netscape Navigator and Communicator too. If the newsreader uses a different file name for his newsrc-file then a link must be created. In the env config file are options for enable support for several type of newsrc-files: NEWSRC, NEWSRC_KRN and NEWSRC_XSP4SI.
NEWSRC_XSP4SI is for a GUI for sp4si that is not yet developed. In this case an additional file ~/.newsrc-xsp4si will be readed. The subscriptions are divided into 2 diffent types:
- nntp (for suck)
the current subscriptions from the newsrc files will be compared with the site/<site>/sucknewsrc created by the last spool, those entries wil be taken for old subscriptions and on new subscriptions an additional number will be put after the name of the newsgroup. This number is specified in the env file and can be involve by setting of LAST variable. Default value is -10, this mean that on new subscriptions the last 10 articles will be sucked.
- uucp (for gup)
UUCP requires a fully different handling. A file site/<site>/uucpnewsrc with last subscriptions will be created. At the next run of subscribe, the new subscriptions will be compared with the file as named above and then if this two files are different a mail will be sent to remote gup account. In this case we are sure that your news provider weren't bombed with your subscriptions mails.
[Back]
6.5. Installation script
install.sh
Syntax: install.sh
the install script ask you for some paths, copies the scripts and generates a new env and newshosts config files into directory you have entered.. They are already default paths, so you may have just to hit ENTER.
Notice: there will be two additional config files in the site directory created (if not exists): localhost.active and localhost.newsgroups which you can easy use to insert local newsgroups by changing their contents. In this case your changes will be involved by the next run of rebuild.active. There are already two local newsgroups defined: <hostname>.test (that may be useable) and <hostname>.spool (where you can let post your spool-reports). <hostname> is the hostname of the local host where sp4si was installed. Because of usage of local hostnames in the local newsgroups-names we would avoid, that there are local postings they are posted outside because of wrong configuration of newsfeeds file.
[Back]
7. Installation
ATTENTION: befor installing sp4si the INN must be already be right configured and runable, this counts for UUCP too.
UUCP installation and configuration decripsiot belongs not to this package and you have to get it from other resources (f.e.: UUCP-HowTo). On demand could the UUCP configuration be included in the future. The sp4si package consists of following parts:
- install.sh
- the install script
- get.active
- downloads active files from all configured newshosts
- rebuild.active
- generates a new active file
- subscribe
- for automatically newsgroups subscribing from .newsrc compatible files.
- spool.news
- news spool frontend
- nntp.down
- spool articles down and posts it to the local news server
- nntp.up
- spools articles up to the remote newshost
- uucp.batch
- prepares the batches of outgoing articles
- uucp.cico
- (call-in-call-out) invokes the uucp call, spools outgoing batches, receives and executes new batches
- out.filter
- just an outgoing filter for spool.up script
- sp4si_de.html
- german manual in HTML format
- sp4si_en.html
- english manual in HTML format
- README
- the small info file :-)
- CHANGES - the history file
The archive must be unpacked first by typing tar xvfz sp4si.095.tar.gz. After that, you must change in the created subdirectory sp4si by typing cd sp4si and start the install.sh script. The install.sh script expects no parameters. The innshellvars file of your INN installation will be found and parsed. Only the destination directory for the sp4si installation will be asked manually. If the innshellvars file couldn't be found, all othe required paths will be asked too.
[Back]
8. Configuration: first steps
8.1. Step 1: newshosts configuration
After the installaion is successfully finished, the newshosts could be inserted to the newshosts file (you will find this in your destination directory), it would suppose so:
newshosts (example):
# example for NNTP sites
3dfx news.3dfx.com
stardiv news.stardiv.de nntp/nntp
# example for UUCP sites
uucpsite uucpsite.domain.com uucp/uucp gzip gup@uucpsite.domain.com secret16
uucp2 uucp2.domain.com uucp cat gupaccount@uucp2.domain.com der63lfd
# example
remsite news.domain.com nntp/uucp cat gup@news.domain.com secrt.32
[Back]
8.2. Step 2: download of active file
After you have changed and saved the newshosts file, you can attempt the second step, the active files must be loaded from your configured newshosts. By starting the script below, were alls active files downloaded and stored in site/<site> subdirectories:
get.active
[Back]
8.3. Step 3: newsfeeds configuration
After all active file are successfully downloaded, you have to attempt the difficultest step of the sp4si configuration: the newsfeeds entries. You have to take a look at the (new downloaded) active files in site/<site> subdirectories and have to decide which newsgroups from which newshosts you will spool to your local newsserver. You should decide so, that each newsgroup will be spooled only by one host, you avoid therewith to propagate newsgroups between your newsfeeds. Therefore you should read the man page for newsfeeds. You can look for an example in the german manual (sorry) too: "Erweiterung von INN & suck auf mehrere Newsserver".
[Back]
8.4. Step 4: create new active file
After the newsfeeds are configured a new active file for your INN can be generated by typing:
rebuild.active
On running this, a new active file will be created (regarding the localhost.active file) and moved to the INN's active file. For that, INN will be stopped, his config files reloaded and parsed for new (the changes on newsfeeds file will be active) and finally restarted.
Ready !
So, we have finished our configuration of sp4si and you are now able to spool all your configured news hosts.
For your sysnopsis:
- enter <site> and <newshost> and maybe <up/down> <packer> <gupmail> <password> too in newshosts file
- connect to internet and let run get.active, than you can continue offline
- enter newsfeeds in newsfeeds file
- rebuild.active
[Back]
9. Tips
sp4si is developed for decrease the news administration, here will be some usefull tips presents which can help you by your daily work with sp4si.
[Back]
9.1. Inserting of local newsgroups
to add a new newsgroup f.e: my.newsgroup, it must be inserted in localhost.active file:
localhost.active (example):
localhost.test 0000000000 0000000001 y
localhost.spool 0000000000 0000000001 y
my.newsgroup 0000000000 0000000001 y
then rebuild.active must be restarted:
rebuild.active
after that all we have now 3 local newsgroups.
[Back]
9.2. Automatic active file update
The offer of newsgroups will be continously changed, new newsgroups will be created, old deleted and subgroups created. For being up to date, you could run the update of the active automatically. For this you must create a new cron-job for the user news in the crontab(5):
crontab -u news -e
and then insert following lines:
# set vars
ONLINE=/etc/ppp/ppp-on # how to go online
OFFLINE=/etc/ppp/ppp-off # how to go offline
SP4SI=/var/lib/news/sp4si # path for sp4si
0 0 1 * * root $ONLINE; $SP4SI/get.active; $OFFLINE; $SP4SI/rebuild.active
there will be every 1st of month at 0:00 o'clock active files downloaded and a new local active file created.
For more information about crontab type: man 5 crontab.
[Back]
10. Where to get sp4si ?
You can get the sp4si package an the LAMEpage in Linux/News area. Or you use one of the links listed below:
http://home.pages.de/~lamepage/linux/news/sp4si.095.tar.gz
http://members.xoom.com/LAMEpage/linux/news/sp4si.095.tar.gz
http://cruise.de/ps/sp4si.095.tar.gz
If you have found any bugs or have any suggestion to make sp4si better please write me an email to: Peter Sobisch <petersob@gmx.net>.
I want to thank Carsten Voss , whose manual has helped me to install my first INN news system and Kevin John Kimmel for correcting my english manual.
Back | Linux | Mainpage
©1998 by Peter Sobisch