+# -*- perl -*-
=head1 NAME
-DXSpiderWeb Orthogonal Communications Protocol $Revision$
+Aranea Orthogonal Communications Protocol
+
+$Revision$
=head1 SYNOPSIS
- <Origin>,<TimeSeq>,<Hop>,<FrmUser>,<To>,<ToUser>|<Tag>,<Data>...
+ <Origin>,<Group>,<TimeSeq>,<Hop>[,<From>]|<Tag>,<Data>...
=head1 ABSTRACT
For many years DX Clusters have used a protocol which was designed
-for a non-looped tree of nodes. This has probably never, reliably,
-been achieved in practice; certainly not recently. This document
-describes a complete replacement for that protocol. It allows a
+for a non-looped tree ofL</Node>s. This environment has probably never, reliably,
+been achieved in practice; certainly not recently.
+
+There have always been loops, sometimes bringing the network to its
+knees. In modern usage, both in order to get some resilience and also
+to expedite information flow, we use internet based, deliberately
+looped networks with filtering. Whilst this works, after a fashion, there
+are all sorts of problems that the current PC protocol can never
+address.
+
+This document
+describes a complete replacement for the PC protocol. It allows a
fully looped network, is inherently extensible and should be simple
to implement (especially in perl).
-All implementations of this protocol shall B<only> use this protocol
+All implementations shall use b<only> this protocol
for inter-node communications.
=head1 DESCRIPTION
-This protocol is encoded in UTF8 with HTTP style escaping. It is
-designed to be an extensible basis for any type of one to many
+This protocol is
+designed to be an extensible basis for any type of one -> many
"instant" line-based communications tasks.
-This protocol is designed to be flood routed in a meshed network in
-as efficient a manner as possible.
+The protocol is designed to be flood routed in a meshed network in
+as efficient a manner as possible. The reason we have chosen this
+mechanism is that most L</Messages> need to be broadcast to allL</Node>s.
+
+Experience has shown thatL</Node>s will appear and (more infrequently)
+disappear without much (or any) notice.
+Therefore, the constantly changing and uncoordinated
+nature of the network doesn't lend itself to fixed routing policies. Therefore,
+whilst metrics and routing tables (more like routing hint tables) will be
+built up over time, an aggressive aging algorithm will also be employed to prevent
+a lot of stale routing information being retained.
+
+Having said that: where routes have
+been learned through past traffic, and this data is recent, then direct routing should be used.
+Those L</Messages> that could be routed (likely to be mainly single line one to
+one "talk" L</Messages>) that, anyway,
+happen sufficiently infrequently that, should they need to be flood routed
+(because no route has been learned yet), it is a small cost overall.
+
+=head1 Messages
+
+A message is a single line of UTF8 encoded and HTTP escaped text
+terminated in the standard internet manner with a <CR><LF>.
Each message consists of a L</Routing Section> and a L</Command Section>.
-The two sections are separated with the '|' character and the whole
-message is terminated in the standard RFC/Internet manner with the
-ascii <carraige return><linefeed> characters. It follows that these
-characters (as well as a small number of other reserved characters)
+The two sections are separated with the '|' character.
+It follows that these
+characters (as well as non-printable characters, <CR>, <LF> and
+a small number of other reserved characters)
can only be sent escaped. This is described further in the
-L</Command Section>.
+L</Command Section> and L</Fields>.
Most of this document is concerned with the L</Routing Section>, however
-some L</Standard Commands> which all implementation should issue and
+some L</Standard Commands> which all implementations should issue and
must accept are described.
+=head1 Applications
+
+In the past messaging applications such as DX Cluster software have maintained
+a fairly strict division between L</Node>s and L</User>s. This protocol attempts
+to get away from that by deliberately blurring (or, in some cases, removing)
+any distinction between the two.
+
+Applications that use this protocol are essentially all peers and therefore the
+only real difference between L</Node>s and L</User>s is that a node has one or more
+listeners running that will,
+potentially, allow incoming connections from other L</Node>s, L</Endpoint>s or L</User>s. These
+routable entities are called L</Terminal>s.
+
+Any application that is a sink and/or source of data; is capable of obeying
+the protocol message construction rules and understands how to deduplicate incoming messages
+correctly can operate as a routeable entity or L</Terminal> in this protocol. It is called an L</Endpoint>.
+
+An L</Endpoint> is called a L</Node> if it accepts connections from L</Endpoint>s and is
+prepared to route messages on their behalf to other L</Node>s or L</Endpoint>s. In addition it
+may provide some other, usually simpler, interface (eg simple telnet access) for direct user access. Acting
+in the protocol, on their behalf.
+
+The concept of an L</Endpoint> has been invented because modern clients are
+capable of being more intelligent than simple
+character based connections such as telnet or ax25. They also wish to be able to
+distinguish between the various classes of message, such as: DX spots,
+announces, talk, logging info etc. It is a pain to have to do it, as now,
+by trying to make sense of the (slightly different for each piece of node
+software) human readable "user" version of the output. Far better to pass on
+regular, specified, easily computer decodable versions of the message,
+(i.e. in this protocol) and leave
+the human presentation to the application.
+
+It also helps to modularise the various interfaces that may be implemented such
+as the legacy, character based connections of existing PC protocol based nodes.
+They should be treated
+as local clients, in fact as L</User>s, B<not> as peers in this protocol. It is likely that, in order
+to do this, some extra L</Tag>s will need to be defined at application level.
+
+=head1 Definitions
+
+In this document we use a number of terms that need to be defined.
+
+=head2 Terminal
+
+A L</Terminal> is a routable entity, in other words: a callsign or service that can be routed
+to, that lives at one or a few L</Node>s.
+
+=head2 User
+
+A L</User> is a connection to a L</Node> (that allows such connections)
+that does not occur in protocol. All L</User>s shall be identified with a name
+of up to 12 characters in the set [-0-9A-Z_]. All messages have to be routed via the
+L</Node> to which this L</User> is connected.
+
+=head2 Endpoint
+
+An L</Endpoint> is a connection to a L</Node> that uses the protocol. From a routing point of
+view, it is indistiguishable from a L</Node>. The L</Endpoint> is responsible for creating and decoding
+well formed protocol messages. An L</Endpoint> does not route beyond the immediate L</Node>(s) to
+which it is connected. It may also be a L</Service> connected to a L</Node> which provides some
+addressable service (such as a database) that can be queried.
+
+=head2 Node
+
+A L</Node> is connected to other L</Node>s. It is responsible for routing messages in protocol
+from other L</Node>s or L</Endpoint>s, whether directly connected or not. Optionally, a L</Node>
+may provide other interfaces, such as direct L</User> connections or legacy PC protocol speaking
+DX Clusters.
+
+=head2 Channel
+
+A L</Channel> is a L</Group> address that is not a L</Terminal>. It is (unless qualified by a L</Terminal>)
+broadcast on all a L</Node>s interfaces unless preventing by some filtering or other local policy on
+that L</Node>.
+
+=head2 Service
+
+A L</Service> is application that either plugs into or connects as an L</Endpoint> to a L</Node>. It is an
+application that, in effect, is a database. In other words: queries are sent to the L</Service> and it sends
+back a reply.
+
=head1 Routing Section
The application that implements this protocol is essentially a line
oriented message router. One line equals one message. Each line is
effectively a datagram.
-It is assumed that nodes are connected to
+It is assumed thatL</Node>s are connected to
each other using a "reliable" streaming protocol such as TCP/IP or
-AX25. Having said that: in context, messages in this protocol could be
+AX25. Having said that: in context, L</Messages> in this protocol could be
multi/broadcast, either "as is" or wrapped in some other framing
protocol.
-Because this is an unreliable, best effort, "please route my packets
-through your node" protocol, there is no guarantee that a message
-will get to the other side of a mesh of nodes. There may be a
+Although the physical transport between L</Node>s is reliable, the actual message
+is unreliable, because this is an unreliable, best effort, "please route my packets
+through your node" protocol. There is no guarantee that a message
+will get to the other side of a mesh of L</Node>s. There may be a
discontinuity either caused by outage or deliberate filtering.
-However, as it is envisaged that most messages will be flood routed or,
-in the case of directed messages (those that have L</To> and/or
-L</ToUser> fields) down some/most/all interfaces showing a route for that
-direction, it is unlikely that messages will be lost in practice.
+However, as it is envisaged that most L</Messages> will be flood routed or,
+in the case of directed L</Messages> (those that have a L</Group> containing a L</Terminal> of some kind)
+down some/most/all interfaces showing a route for that
+direction, it is unlikely that L</Messages> will be lost in practice.
-=head2 Field Description
+Assuming that there is a path between all the L</Node>s in a network, then it is guaranteed
+that a message will be delivered everywhere, eventually. It is possible (indeed likely) that
+copies of a message
+will arrive at L</Node>s more than once. L</Node>s are responsible for deduplicating those messages
+using the information in the L</Routing Section>.
-Only the first three fields in the L</Routing Section> are compulsory
-and indicate that this is a broadcast to be sent to all nodes coming
-from the L</Origin>. If the message needs to be identified as coming
-from a user on a node, then the L</FrmUser> field is added.
+=head2 Field Description
-Adding a L</To> and/or L</ToUser> field will restrict the destinations
-or recipients that receive this message.
+All fields in the L</Routing Section> are compulsory except the L</From> field. If it is missing
+so is the separating comma.
The L</Hop> field is incremented on receipt of a message on a node.
Fields are separated by the comma ',' character with the last field
required followed by the vertical bar '|' character.
-If trailing fields are missed out then superfluous commas can also
-be left out. If intervening fields are missing then no space needs
-to be left for the separating comma.
-
The characters allowed in the routing section are restricted. Any
invalid characters in any field will cause the whole message to be
silently dropped.
=item B<Origin>
This is a compulsory field. It is the name of the originating node.
-The field can contain up to 12 characters in the set [-A-Z0-9_] in
+The field can contain up to 12 characters in the set [-A-Z0-9_/] in
any order. Higher layers may restrict this further.
The field must not be changed by any other node.
+=item B<Group>
+
+This is the Group (or Channel) to be used for this data. It is compulsory.
+
+It is a string of up to 12 characters
+in the set [-A-Z0-9_/] in any order.
+
+Optionally, for extra routing to
+a particular L</Terminal> connected at a specific L</Node>, or even a
+particular L</Terminal> in a L</Group>,
+it may have another 12 character
+string in the same set, concatenated with the first string. The two strings are separated by a ':'
+character. For example:
+
+ DX # the DX group
+ GB7DJK # the node GB7DJK
+ G1TLH # the user or endpoint G1TLH
+ GB7DJK:G1TLH # the user G1TLH at GB7DJK
+ DX:G1TLH # the user G1TLH in the DX group
+
+This field can contain either a L</Terminal> or some other string which is interpreted
+as broadcastable group address. Any message that has a L</Group> that is not recognised as a L</Terminal> must
+be broadcast.
+
+This means that messages to callsigns, for whom no specific routing information is available,
+will be found by means of a broadcast. Hopefully this will cause some kind of activity o.b.o
+that callsign will allow routing tables to be gathered that narrow down the scope of any future
+message to that callsign through the network.
+
+Remember that not all L</Node>s may pass every L</Group> field, depending on local policy.
+
=item B<TimeSeq>
This is a compulsory field. It is a 10 hexadecimal digit string which
The date portion is constructed as:
- my $date = ((((gmtime)[3] < 1) | $ntpflag) < 18) | (time % 86400);
+ my $date = ((((gmtime)[3] << 1) | $ntpflag) << 18) | (time % 86400);
The sequence number is simply an unsigned short (or 16 bit) number
starting at 0.
it on to higher layers for onward processing.
Implementations may have an upper limit to this field and may
-silently drop incoming messages with a L</Hop> count greater than the
+silently drop incoming L</Messages> with a L</Hop> count greater than the
limit.
-=item B<FrmUser>
-
-This field is optional. It is the identifier of the originating
-user. If it is missing then the message is
-assumed to come from the originating node itself.
-
-It can consist of up to 12 characters in the set [-A-Z0-9_]
-in any order. Higher layers may restrict this further.
-
-=item B<To>
-
-This field is optional. It is a string of up to 12 characters
-in the set [-A-Z0-9_] in any order.
-
-This field is used either to indicate particular node destination
-or to differentiate this broadcast in some way by making this
-message as a member of a L</Channel>. Any message can be sent
-down any L</Channel>. The names of L</Channel>s and their usage
-is entirely up to the implementor.
-
-It is assumed that node names can be differentiated from user
-names and L</Channel> names.
-
-If the field is set to a particular node destination, it will
-be routed (rather than broadcast) to that node. However, any
-intervening nodes are free to duplicate the message and send
-it down more than one, likely looking, interface - depending on any
-network policies that may pertain.
-
-=item B<ToUser>
-
-This field is optional. It is a string of up to 12 characters
-in the set [-A-Z0-9_] in any order. Higher layers may restrict
-this further.
-
-Conventionally this field is used to indicate the user to whom
-this message is directed. In an ideal world the L</To> field
-will be set, by the originating node, to the identifier of the node
-on which this user resides.
-
-If the L</To> field is not set then this message will be
-broadcast. However, should a node become apparent (on route)
-then nodes are free to fill in the L</To> field and proceed
-with a more directed approach.
-
-If it becomes apparent (on route) that there may be more than
-one possible L</To> destination for a L</ToUser> then a node
-may duplicate the message (keeping the same L</TimeSeq>) and
-route it onwards. Because of the L</DeDuplication> inherent in
-the system, it is indeterminate as to which destination will
-receive the message. It is possible for all or just some
-destinations to receive the message. The tuple (L</Origin>,
-L</TimeSeq>) will determine uniqueness.
+=item B<From>
-This field can, in the case where L</To>
-is set to the name of a node, be set to a L</Channel>. If this
-is the case then this will cause this message to be sent to
-a L</Channel> on the L</To> node only.
+The L</From> field is optional. When present, it represents a L</Terminal> at
+the originating L</Node>. If it is missing then either it is not relevant or it
+is assumed to be the L</Origin>.
=back
-=head2 Channel
-
-Channels are a concept very similar to that on IRC. It is a
-way of segregating data flows in a network. In principle, subject
-to local policy or application requirements, any data (or
-L</Command Section>) can be sent down any channel.
-
-It is up to the implementation whether to use this feature or not.
-
=head2 Routing
It is assumed that nodes will be connected in a looped network with
by looking at the tuple and merging that with the L</Hop> count.
Each interface remembers the latest L</TimeSeq> with the lowest L</Hop>
for each L</Origin> that arrives on that interface. It also remembers
-the number of messages for that L</Origin> that has been received on
+the number of L</Messages> for that L</Origin> that has been received on
that interface.
Any message for onward broadcast is duplicated and sent out on all
=head2 Examples
# on link startup from GB7BAA (both sides hello)
- GB7TLH,3D02350001,0,GB7BAA|HELLO,Aranea,1.2,24.123
- GB7BAA,3D02355421,1,GB7TLH|HELLO,Aranea,1.1,23.245
+ GB7TLH,ROUTE,3D02350001,0|HELLO,Aranea,1.2,24.123
+ GB7BAA,ROUTE,3D02355421,1|HELLO,Aranea,1.1,23.245
# on user startup to GB7TLH
- GB7TLH,3D042506F2,0,G1TLH|HELLO,PClient,1.3
+ GB7TLH,ROUTE,3D042506F2,0,G1TLH|HELLO,PClient,1.3
# on user disconnection
- GB7TLH,3D9534F32D,0,G1TLH|BYE
+ GB7TLH,ROUTE,3D9534F32D,0,G1TLH|BYE
# a talk (actually 'text') message to a user (some distance away
# from the origin node)
- GB7TLH,3D03450019,3,G1TLH,GB7BAA,G8TIC|T,Hiya Mike what's happening?
+ GB7TLH,G8TIC,3D03450019,3,G1TLH|T,Hiya Mike whats happening?
- # a talk/chat/text message to a channel or group
- GB7TLH,0413525F23,2,G1TLH,VHF|T,2m is opening on MS
+ # a talk/chat/text message to a Group
+ GB7TLH,VHF,0413525F23,2,G1TLH|T,2m is opening on MS
# a ping to find the whereabouts and distance of a user from a node
# the hex number on the end is the ping ID
- GB7TLH,1512346543,0,,,G7BRN|PING,9F4D
-
- # the same from a user on GB7TLH
- GB7TLH,1512346543,0,G1TLH,,G7BRN|PING,23
+ GB7TLH,G7BRN,1512346543,0,G1TLH|PING,9F4D
# this effectively asks whether the user is on-line on a particular node
- GB7TLH,1512346543,0,G1TLH,GB7DJK,G7BRN|PING,35DE
+ GB7TLH,GB7BAA:G7BRN,1512346543,0,G1TLH|PING,35DE
# A possible reply, same ID as ping followed by the no of hops on the
- # ping that was received
- GB7DJK,1512450534,3,G7BRN,GB7TLH,G1TLH|PONG,35DE,3
+ # ping that was received thus telling you how far away it is.
+ GB7BAA,G1TLH,1512450534,3,G7BRN|PONG,35DE,3
=head1 Command Section
The L</Command Section> of the message contains the actual data being
passed. It is called the Command Section because all commands
-are identified with a L</Tag> which is implemented by
-the software using this protocol.
+are identified with a L</Tag> each of which is implemented by
+the software using this protocol. Each </Tag> (usually) is followed by one
+or more L</Fields>.
+
+=head2 Tag
-The L</Tag> is separated from its data by a comma ','. All fields
+The L</Tag> consists of string of uppercase letters and digits, starting
+with a leading, uppercase, letter. Tags should be as short as is meaningful.
+
+Valid tags would be:
+
+ DX
+ PC23
+ ANN
+
+Invalid tags include:
+
+ 1AAA
+ dx
+ Ann
+
+The L</Tag> is separated from its data L</Fields> by a comma ','.
+
+=head2 Fields
+
+All fields
in any subsequent data shall be separated by a comma ','.
All fields shall
be HTTP encoded such that reserved characters (comma ',',
percent '%',
equals '='
and non printable characters less than 127 (or %7F in hex)
-[including newline and carraige return] are tranlated to
+[including newline and carraige return] are translated to
their two hex digit equivalent preceeded by the percent '%' character.
For example:
use UTF8;
A message (or line) is terminated with <carriage return><linefeed>
-0x0d 0x0a. Incoming messages must be accepted even when terminated
+0x0d 0x0a. Incoming L</Messages> must be accepted even when terminated
with just <linefeed>.
Care must be taken to make sure that fields have any reserved characters
There is no maximum size specified for a message. It is up to each
implimentation to enforce one (if only for their own protection).
-=head2 Tag
-
-The L</Tag> consists of string of uppercase letters and digits, starting
-with a leading, uppercase, letter. Tags should be as short as is meaningful.
-
-Valid tags would be:
-
- DX
- PC23
- ANN
-
-Invalid tags include:
-
- 1AAA
- dx
- Ann
-
=head2 Standard Commands
There are a number of L</Standard Commands> which must be accepted by
=over
-=item B<HELLO>,<software name>,<version>,<build>,<comments>
+=item B<HELLO>
+
+ HELLO,<software name>,<version>,<build>,<comments>
Command sent on connection to another node. Both sides send their information
to the other. All the possible arguments are optional, although some of the
arguments should be sent in order to help diagnose problems. This command is
broadcast.
-=item B<BYE>,<comments>
+=item B<BYE>
+
+ BYE,<comments>
Command sent to all connections when the software is shutting down. This is sent
by the node just before shutdown occurs. This is really only used to help the
network prune its routing tables. It isn't a requirement. The <comment> field
is optional.
-=item B<DISC>,<node name>,<comments>
+=item B<DISC>
+
+ DISC,<node name>,<comments>
Command sent when a node has disconnected from this node. This message is sent when
an interface shuts down. It need not be sent if a L<BYE> from an interface for
The <node name> is mandatory and is the name of the interface that has just
disconnected.
-=item B<PING>,<ping id>
+=item B<PING>
+
+ PING,<user>,<ping id>
Command to send a ping to a node or user. This command is used both by the software
and users to determine a) whether a node or user exists and b) how good the path is
-between them.
+between them.
+
+The <ping id> is a unique string which is usually the hexadecimal equivalent of an
+integer that is incremented every time it is used. But it can be anything that
+will identify this ping using the tuple (L<Origin>,<ping id>) as unique.
+
+=item B<PONG>
+
+ PONG,<ping id>,<user>,<no of hops on ping>
+
+Command to reply to a ping. This is sent as a reply to an incoming ping command.
+The <ping id> is the one supplied and the <no of hops on ping> is the number of
+hops it took for the ping to arrive.
+
+=item B<T>
-=item B<PONG>,<ping id>,<no of hops on ping>
+ T,<text>
-Command to reply to a successful ping
+All implementations must be able to send "text" (encoded as specified in
+L</Fields>). There would be little point in doing all this otherwise!
=back
=head1 COPYRIGHT AND LICENSE
-Copyright 2004 by Dirk Koopman, G1TLH
+Copyright 2004-2005 by Dirk Koopman, G1TLH
This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself.
projects / spider.git / blobdiff