<HTML>
<HEAD>
<META NAME="GENERATOR" CONTENT="SGML-Tools 1.0.9">
- <TITLE>The DXSpider Installation and Administration Manual : Filtering</TITLE>
+ <TITLE>The DXSpider Administration Manual v1.48: Databases</TITLE>
<LINK HREF="adminmanual-6.html" REL=next>
<LINK HREF="adminmanual-4.html" REL=previous>
<LINK HREF="adminmanual.html#toc5" REL=contents>
+<link rel=stylesheet href="style.css" type="text/css" title="default stylesheet">
</HEAD>
<BODY>
<A HREF="adminmanual-6.html">Next</A>
<A HREF="adminmanual-4.html">Previous</A>
<A HREF="adminmanual.html#toc5">Contents</A>
<HR>
-<H2><A NAME="s5">5. Filtering</A></H2>
+<H2><A NAME="s5">5. Databases</A></H2>
-<P>Filters can be set for spots, announcements and WWV. You will find the directories for these under /spider/filter. You will find some examples in the directories with the suffix <EM>.issue</EM>. There are two types of filter, one for incoming information and one for outgoing information. Outgoing filters are in the form <EM>CALLSIGN.pl</EM> and incoming filters are in the form <EM>in_CALLSIGN.pl</EM>. Filters can be set for both nodes and users.
+<P>Spider allows the creation of local or remote databases. It supports
+chained databases, allowing several different databases to be scanned
+with one simple command. Importing of databases is limited at present
+to the standard AK1A databases such as OBLAST and the DB0SDX QSL
+database but will expand with time.
<P>
-<P>All filters work in basically the same way. There are several elements delimited by commas.
-There can be many lines in the filter and they are read from the top by the program.
-When writing a filter you need to think carefully about just what you want to achieve. You
-are either going to write a filter to <EM>accept</EM> or to <EM>reject</EM>.
-Think of a filter as having 2 main elements. For a reject filter, you would have a line
-or multiple lines rejecting the things you do not wish to receive and then a default
-line accepting everything else that is not included in the filter. Likewise, for an
-accept filter, you would have a line or multiple lines accepting the things you wish
-to receive and a default line rejecting everthing else.
-<P>
-<P>In the example below, a user requires a filter that would only return SSB spots
-posted in Europe on the HF bands. This is achieved by first rejecting the CW section
-of each HF band and rejecting all of VHF, UHF etc based on frequency.
-Secondly, a filter rule is set based on CQ zones to only accept spots posted in
-Europe. Lastly, a default filter rule is set to reject anything outside the filter.
-<P>
-<BLOCKQUOTE><CODE>
-<PRE>
-$in = [
- [ 0, 0, 'r', # reject all CW spots
- [
- 1800.0, 1850.0,
- 3500.0, 3600.0,
- 7000.0, 7040.0,
- 14000.0, 14100.0,
- 18068.0, 18110.0,
- 21000.0, 21150.0,
- 24890.0, 24930.0,
- 28000.0, 28180.0,
- 30000.0, 49000000000.0,
- ] ,1 ],
- [ 1, 11, 'n', [ 14, 15, 16, 20, 33, ], 15 ], #accept EU
- [ 0, 0, 'd', 0, 1 ], # 1 = want, 'd' = everything else
-];
-</PRE>
-</CODE></BLOCKQUOTE>
-<P>
-<P>The actual elements of each filter are described more fully in the following sections.
-<P>
-<H2><A NAME="ss5.1">5.1 Spots</A>
+<H2><A NAME="ss5.1">5.1 Creating databases</A>
</H2>
-<P>The elements of the Spot filter are ....
+<P>Creating a database could not be more simple. All the commands are
+sent from the cluster prompt as the <EM>sysop</EM> user.
+<P>To create a database you use the command <EM>dbcreate</EM>. It can
+be used in 3 different ways like so ..
<P>
<BLOCKQUOTE><CODE>
<PRE>
-[action, field_no, sort, possible_values, hops]
+dbcreate <name>
</PRE>
</CODE></BLOCKQUOTE>
-<P>
-<P>There are 3 elements here to look at. Firstly, the action element. This is very simple and only 2 possible states exist, accept (1) or drop (0).
-<P>
-<P>The second element is the field_no. There are 13 possiblities to choose from here ....
+<P>To simply create a database locally, you just tell the command the
+name of the database. This does not create the actual database, it
+simply defines it to say that it exists.
<P>
<BLOCKQUOTE><CODE>
<PRE>
- 0 = frequency
- 1 = call
- 2 = date in unix format
- 3 = comment
- 4 = spotter
- 5 = spotted dxcc country
- 6 = spotter's dxcc country
- 7 = origin
- 8 = spotted itu
- 9 = spotted cq
- 10 = spotter's itu
- 11 = spotter's cq
- 12 = callsign of the channel on which the spot has appeared
+dbcreate <name> chain <name> [<name>...]
</PRE>
</CODE></BLOCKQUOTE>
-<P>
-<P>The third element tells us what to expect in the fourth element. There are 4 possibilities ....
+<P>This creates a chained database entry. The first database will be
+scanned, then the second, the third etc...
<P>
<BLOCKQUOTE><CODE>
<PRE>
- n - numeric list of numbers e.g. [ 1,2,3 ]
- r - ranges of pairs of numbers e.g. between 2 and 4 or 10 to 17 - [ 2,4, 10,17 ]
- a - an alphanumeric regex
- d - the default rule
+dbcreate <name> remote <name>
</PRE>
</CODE></BLOCKQUOTE>
-<P>
-<P>The fifth element is simply the hops to set in this filter. This would only be used if the filter was for a node of course and overrides the hop count in hop_table.pl.
-<P>
-<P>So, let's look at an example spot filter. It does not matter in the example who the filter is to be used for.
-So, what do we need in the filter? We need to filter the spots the user/node requires and also set a default rule for anything else outside the filter. Below is a simple filter that stops spots arriving from outside Europe.
+<P>This creates a remote entry. the first name field is the database
+name at the remote node, then the remote switch, then the actual
+node_call of the remote node, for example...
<P>
<BLOCKQUOTE><CODE>
<PRE>
-$in = [
- [ 0, 4, 'a', '^(K|N|A|W|VE|VA|J)'], # 0 = drop, 'a' = alphanumeric
- [ 1, 0, 'd', 0, 1 ], # 1 = want, 'd' = everything else
- ];
+dbcreate buckmaster remote gb7dxc
</PRE>
</CODE></BLOCKQUOTE>
+<P>Remote databases cannot be chained, however, the last database in a
+chain can be a remote database.
<P>
-<P>So the filter is wrapped in between a pair of square brackets. This tells Spider to look in between these limits. Then each line is contained within its own square brackets and ends with a comma.
-Lets look carefully at the first line. The first element is 0 (drop). Therefore anything we put on this line will not be accepted. The next element is 4. This means we are filtering by the spotter. The third element is the letter "a" which tells the program to expect an alphanumeric expression in the fourth element. The fourth element is a list of letters separated by the pipe symbol.
-<P>
-<P>What this line does is tell the program to drop any spots posted by anyone in the USA, Canada or Japan.
-<P>
-<P>The second line is the default rule for anything else. The "d" tells us this and the line simply reads... accept anything else.
-<P>
-<P>You can add as many lines as you need to complete the filter but if there are several lines of the same type it is neater to enclose them all as one line. An example of this is where specific bands are set. We could write this like this ....
-<P>
-<BLOCKQUOTE><CODE>
-<PRE>
-[ 0,0,'r',[1800.0, 2000.0], 1],
-[ 0,0,'r',[10100.0, 10150.0], 1],
-[ 0,0,'r',[14000.0, 14350.0], 1],
-[ 0,0,'r',[18000.0, 18200.0], 1],
-</PRE>
-</CODE></BLOCKQUOTE>
-<P>
-<P>But the line below achieves the same thing and is more efficient ....
-<P>
-<BLOCKQUOTE><CODE>
-<PRE>
- [ 0, 0, 'r',
- [
- 1800.0, 2000.0, # top band
- 10100.0, 10150.0, # WARC
- 14000.0, 14350.0, # 20m
- 18000.0, 18200.0, # WARC
- [ ,1 ],
-</PRE>
-</CODE></BLOCKQUOTE>
-<P>
-<P>
-<H2><A NAME="ss5.2">5.2 Announcements</A>
+<H2><A NAME="ss5.2">5.2 Importing databases</A>
</H2>
+<P>The only databases that Spider can currently import are the standard
+AK1A databases such as OBLAST or the DB0SDX qsl and address database.
+This will be added to with time.
+<P>To import such a database, first put the file somewhere useful like /tmp
+and then issue the following command ...
<P>
<BLOCKQUOTE><CODE>
<PRE>
-
-# This is an example announce or filter allowing only West EU announces
-#
-# The element list is:-
-# 0 - callsign of announcer
-# 1 - destination * = all, <callsign> = routed to the node
-# 2 - text
-# 3 - * - sysop, <some text> - special list eg 6MUK, ' ', normal announce
-# 4 - origin
-# 5 - 0 - announce, 1 - wx
-# 6 - channel callsign (the interface from which this spot came)
-
-$in = [
- [ 1, 0, 'a', '^(P[ABCDE]|DK0WCY|G|M|2|EI|F|ON)' ],
- [ 0, 0, 'd', 0 ]
-];
+dbimport oblast /tmp/OBLAST.FUL
</PRE>
</CODE></BLOCKQUOTE>
-<P>In this example, only the prefixes listed will be allowed. It is possible to be quite specific. The Dutch prefix "P" is followed by several secondary identifiers which are allowed. So, in the example, "PA" or "PE" would be ok but not "PG". It is even possible to allow information from a single callsign. In the example this is DK0WCY, to allow the posting of his Aurora Beacon.
+<P>This will update the existing local oblast database or create it if
+it does not exist.
<P>
-<H2><A NAME="ss5.3">5.3 WWV</A>
+<H2><A NAME="ss5.3">5.3 Checking available databases</A>
</H2>
+<P>Once a database is created, you will want to check that it has been
+added. To do this use the <EM>dbavail</EM> command. This will
+output the available databases. For example ...
<P>
<BLOCKQUOTE><CODE>
<PRE>
-
-# This is an example WWV filter
-#
-# The element list is:-
-# 0 - nominal unix date of spot (ie the day + hour:13)
-# 1 - the hour
-# 2 - SFI
-# 3 - K
-# 4 - I
-# 5 - text
-# 6 - spotter
-# 7 - origin
-# 8 - incoming interface callsign
-
-# this one doesn't filter, it just sets the hop count to 6 and is
-# used mainly just to override any isolation from WWV coming from
-# the internet.
-
-$in = [
- [ 1, 0, 'd', 0, 6 ]
-];
+dbavail
+DB Name Location Chain
+qsl Local
+buck GB7ADX
+hftest GB7DXM
+G0VGS de GB7MBC 3-Feb-2001 1925Z >
</PRE>
</CODE></BLOCKQUOTE>
<P>
-<P>It should be noted that the filter will start to be used only once a user/node has logged out and back in again.
-<P>I am not going to spend any more time on these filters now as they will become more "comprehensive" in the near future.
-<P>
-<H2><A NAME="ss5.4">5.4 Filtering Mail</A>
+<H2><A NAME="ss5.4">5.4 Looking up databases</A>
</H2>
-<P>In the /spider/msg directory you will find a file called badmsg.pl.issue. Rename this to badmsg.pl and edit the file. The original looks something like this ....
+<P>To look for information in a defined database, simply use the <EM>dbshow</EM>
+command, for example ...
<P>
<BLOCKQUOTE><CODE>
<PRE>
-
-# the list of regexes for messages that we won't store having
-# received them (bear in mind that we must receive them fully before
-# we can bin them)
-
-
-# The format of each line is as follows
-
-# type source pattern
-# P/B/F T/F/O/S regex
-
-# type: P - private, B - bulletin (msg), F - file (ak1a bull)
-# source: T - to field, F - from field, O - origin, S - subject
-# pattern: a perl regex on the field requested
-
-# Currently only type B and P msgs are affected by this code.
-#
-# The list is read from the top down, the first pattern that matches
-# causes the action to be taken.
-
-# The pattern can be undef or 0 in which case it will always be selected
-# for the action specified
-
-
-
-package DXMsg;
-
-@badmsg = (
-'B', 'T', 'SALE',
-'B', 'T', 'WANTED',
-'B', 'S', 'WANTED',
-'B', 'S', 'SALE',
-'B', 'S', 'WTB',
-'B', 'S', 'WTS',
-'B', 'T', 'FS',
-);
+dbshow buckmaster G0YLM
</PRE>
</CODE></BLOCKQUOTE>
-<P>
-<P>I think this is fairly self explanatory. It is simply a list of subject headers that we do not want to pass on to either the users of the cluster or the other cluster nodes that we are linked to. This is usually because of rules and regulations pertaining to items for sale etc in a particular country.
-<P>
-<H2><A NAME="ss5.5">5.5 Filtering DX callouts</A>
-</H2>
-
-<P>In the same way as mail, there are some types of spot we do not wish to pass on to users or linked cluster nodes. In the /spider/data directory you will find a file called baddx.pl.issue. Rename this to baddx.pl and edit the file. The original looks like this ....
+<P>will show the information for the callsign G0YLM from the buckmaster
+database if it exists. To make things more standard for the users
+you can add an entry in the Aliases file so that it looks like a standard
+<EM>show</EM> command like this ...
<P>
<BLOCKQUOTE><CODE>
<PRE>
-
-# the list of dx spot addresses that we don't store and don't pass on
-
-
-package DXProt;
-
-@baddx = qw
-
- FROG
- SALE
- FORSALE
- WANTED
- P1RATE
- PIRATE
- TEST
- DXTEST
- NIL
- NOCALL
-);
+'^sh\w*/buc', 'dbshow buckmaster', 'dbshow',
</PRE>
</CODE></BLOCKQUOTE>
+<P>Now you can simply use show/buckmaster or an abreviation.
<P>
-<P>Again, this is simply a list of names we do not want to see in the spotted field of a DX callout.
-<P>
-<P>
-<H2><A NAME="ss5.6">5.6 Filtering words from text fields in Announce, Talk and DX spots</A>
+<H2><A NAME="ss5.5">5.5 Removing databases</A>
</H2>
-<P>Create a file in /spider/data called <EM>badwords</EM>. The format is quite
-simple. Lines beginning with # are ignored so comments can be added. An
-example file is below ...
+<P>To delete an existing database you use the <EM>dbremove</EM> command.
+For example ...
<P>
<BLOCKQUOTE><CODE>
<PRE>
-# Below is a list of words we do not wish to see on the cluster
-grunge grunged grunging
-splodge splodger splodging
-grince
-fluffle
+dbremove oblast
</PRE>
</CODE></BLOCKQUOTE>
-<P>Multiple words can be used on the same line as shown. Obviously these
-are just examples :-)
-<P>
-<P>You can reload the file from the cluster prompt as sysop with load/badwords.
+<P>would remove the oblast database and its associated datafile from the
+system. There are no warnings or recovery possible from this command.
+If you remove a database it ceases to exist and would have to be created
+from scratch if you still required it.
<P>
<HR>
<A HREF="adminmanual-6.html">Next</A>
projects / spider.git / blobdiff