+20Apr24======================================================================
+1. Try to "fix" Spot deduping (again). This is probably Work In Progress...
+2. A new(er) band.pl gratefully received from Andy G4PIQ.
+13Apr24======================================================================
+1. Fix small issue with perl 5.20 (and possibly earlier) compilation.
+27Mar24======================================================================
+1. Fix dupe PC[16]1 (spot) coming in "too fast" for dupe filter to work.
+
+ Probably.
+25Mar24======================================================================
+1. Make talk work. Note: this now works as the mojo branch originally
+ expected - which means that it may (and likely will) flood route talks
+ where the callsign that you want to talk to does not have a route to
+ both the callsign and any nodes that it may be connected to.
+22Mar24======================================================================
+1. Change the processing of trailing callsigns slightly so that /1 /2 etc are
+ retained, but the /P /M /A /MM etc are removed.
+20Mar24======================================================================
+1. Has it really been so long since the last update?
+2. Since forever, mainly because filtering was done very early on in
+ DXSpider's development, clearing a filter needed a disconnect/reconnect
+ cycle from the user/node to activate. Now clearing or changing a filter is
+ immediately effective in the current session.
+
+ Also things like: 'rej/spot user_default on vhf' will immediately apply
+ to all users that do not already have some other filter in place. Users
+ can override this by setting a filter.
+
+ The Filter system will now show the actual token that a faulty command
+ dislikes.
+3. Disable the "is there a route to callsign X3XXX" test in talk. This is a
+ legacy test that should not apply at the user level. The resulting PC93 is
+ flood routed unless there is a genuine direct route to that callsign.
+25Mar23=======================================================================
+1. Changed the default of X and Y (in item 2 below) to 27 and 20 respectively.
+24Mar23=======================================================================
+1. Restore perl regex version of is_ipaddr. This *should* allow windows
+ updates to work again.
+2. Added cmd entry rate limiting. If a user sends X commmands in Y secs then
+ they are disconnected without notice. The defaults are X
+ ($DXCommandmode::maxcmdcount) = 16 and Y ($DXCommandmode::cmdinterval) = 9.
+
+ These can be changed by the startup script.
+
+ These default values are set generously deliberately to allow certain user
+ programs to get with the program and reduce the number of cmds that they
+ issue on connection down to something reasonable. For instance, I cannot
+ see why things like name, qth, lat/long/QRA (amongst several other sticky
+ user attributes that only need to be entered once) are sent on every login.
+
+ These default values WILL reduce over time.
+23Mar23=======================================================================
+1. Fix DXCIDR brokenness introduced recently, speeded up DXCIDR::sort, allow
+ sh/badip 2a05:dfc7:402a::1 45.134.225.96 159.69.38.160 to test individual
+ IP address as well as show the default (of every badip).
+2. Speed up and be more strict with input for is_ipaddr(<ip address>).
+22Mar23=======================================================================
+1. Fix warning in DXUtil::difft.
+21Mar23=======================================================================
+1. Fix DXUtil is_ipaddr function which was accidentally included in previous
+ commit.
+2. Do not extract IP addresses to create routing info as default.
+17Mar23=======================================================================
+1. Try to nail down incoming PC92s that attempt to change the type of $myalias
+ or $mycall.
+2. Do some housekeeping with the startup and finish debugging to make the
+ order of messaging more logical. Also disable some messaging caused by
+ other errors/circumstances causing various object handles disappearing.
+3. Make sure that $mycall and $myalias are upper case (and moan about it if
+ they aren't.
+10Mar23=======================================================================
+1. I am aware that there are windows nodes out there on mojo (brave, impetuous
+ that you are) and wget is not an easy option to download badip files. So,
+ through the power of mojo I give you the new 'download' command. As it
+ stands at the moment, it will download a file from a webserver and place
+ it in /spider/local_data. If it downloads successfully, it will overwrite
+ any file with that name in /spider/local_data. Needless to say this is a
+ sysop only command.
+
+24 * * * * run_cmd('download http://www.dxspider.net/download/badip.torexit')
+24 * * * * run_cmd('download http://www.dxspider.net/download/badip.torrelay')
+24 * * * * run_cmd('download http://www.dxspider.net/download/badip.global')
+25 * * * * run_cmd('load/badip')
+
+ would be an obvious use. But it can also be used to download spot, USDB
+ and other data as well. If you do use it then you will still need to
+ do a spawn_cmd in crontab for the conversion programs, but that too is
+ going to change.
+07Mar23=======================================================================
+1. Fix odd (and old) regression dx command (line ending 59+++) '++' being the
+ culprit.
+2. Allow multiple commands on one line when separated with the 2 character
+ string '\n' (sigh).
+3. Move a few init things about on startup so that more parameters are
+ scriptable.
+4. Put some debug comments before loading various caches (that might take a
+ longgggg time) and then tell people how long each cache load took. This
+ should reduce the worry that something is "wrong" when there is a delay
+ (such as searching for announces).
+06Mar23=======================================================================
+1. Fix filter error reporting, including incrementing concurrent error count
+ if there are actually any detected parse errors.
+2. Fix warnings on difft.
+3. Add the origin of any incoming SP if it is emailed to the recipient.
+4. Oops I accidently restored one path of the node being disconnected that I
+ fixed below (because of changes to filtering).
+04Mar23=======================================================================
+1. Fixed regression caused by too many command errors in (startup) script
+ files. This is caused by much stricter checking of commands entered both
+ by users AND scripts. If there more concurrent errors ie. greater than
+ $DXChannel::maxerrors (default 10) then that user will be disconnected.
+
+ Unfortunately this could include the $mycall (node) channel if there were
+ too many command users in the startup script. The example I looked at had
+ a load of HTML pasted into it.
+
+ For various reasons the only manifestation of this problem was that pings
+ no longer worked.
+
+ This has been fixed. I am grateful to Fulvio HB9DHG for letting me break
+ his node for an hour or so until I worked out what was going on.
+03Mar23=======================================================================
+1. Fix DXDebug::DXDebug not found errors in rarely used functions. This is
+ also an interaction with differing versions of Mojolicious. Clearly this
+ has never happened to me (sigh).
+2. Allow non-privileged users to use ping (no arguments) and get 'PONG nn'
+ where nn is an incrementing counter or if there is an argument it will
+ return 'PONG <argument>' - all in UPPER CASE. See help ping for more
+ details.
+02Mar23=======================================================================
+1. Fix sh/dx/30 problem. I.e. allow old style version of sh/dx 30.
+01Mar23=======================================================================
+1. Revert PC92 A D and PC92 C record flags back to defaults.
+2. Deal with users that send gibberish.
+3. Fix show/425 (default) URL.
+4. Improve Spot deduping.
+28Feb23=======================================================================
+1. Default ip addresses ON in PC92 C records, disable PC92 A D record output.
+2. Add CTY-3308 prefixes.
+3. Fix gen_usdb_data.pl (add missing use DXUtil), update current location of
+ official FCC databases.
+4. Remove the duplicate IP addresses across exits and relays from badip lists
+ during generation by create_master_badip_files.pl.
+14Feb23=======================================================================
+1. Attempt to be more M$ Windows compatible. This basically is to do with the
+ unfortunate fact that most of the Windows perl cannot do, or simulate
+ spawning (running another process in parallel) and/or Mojolicious cannot
+ handle coping with one or more versions of these simulations.
+
+ I don't have a windows perl available to me at moment so I can only
+ simulate running under windows by setting a variable.
+
+ You need to know that running any version of DXSpider on Windows will do
+ all long running commands in line. Much as the 'master' branch does. So
+ running large nodes on Windows boxes with versions of perl that do not
+ support running spawned processes continues to be contra-indicated. Much
+ as it has ALWAYS been.
+04Feb23=======================================================================
+1. Fixed sh/log so that callsigns beginning with a digit (or several) are
+ printed rather than being ignored.
+ This affects: show/ann, chat, log, rcmd and wx
+03Feb23=======================================================================
+1. Fix is_ipaddr to accept trailing '::' on IPV6 addresses.
+2. Fix and extend the TEST program 'showdx' so that it now works on the mojo
+ branch. Type 'showxd -?' for more information or read the source. This is
+ NOT a user program, but is will accept most sh/dx expressions.
+3. Fix create_master_badip_files.pl so that it does not emit IP addresses
+ that the system is_ipaddr() function fails.
+01Feb23=======================================================================
+1. Harden DXCIDR (badip stuff) against format errors in downloaded badip files
+ downloaded using wget from the crontab. If these problems persist PLEASE
+ TELL me and send me examples of the errors that end up in the badip files.
+30Jan23=======================================================================
+1. Add ip addresses to outgoing PC93 messages
+2. Get rid of (some of?) the uninitialised warnings
+3. Add the ability to set the (time) locale to UTC or LOCALTIME by adding
+ the string LOCALE = UTC or LOCALE = LOCAL(TIME)? to your local crontab
+
+ NOTE: this will affect ALL timings in both the system as well as your
+ local crontabs. Changing the LOCALE will make no difference to the system
+ crontab other than using localtime rather than UTC. Which is fine by me.
+
+ You can see what crontab is doing and which LOCALE is in use by doing a
+ 'set/debug cron' in a console and running a 'watchdbg cron' in another
+ shell.
+25Jan23=======================================================================
+1. Add a switch to enable the routing of generated PC92 A/D records and also
+ prevent the rebroadcast of incoming ones.
+
+ set/var $DXProt::pc92_ad_enable 0
+
+ to disable PC92 A/D records.
+2. Add a switch to enable the addition of IP addresses to PC92 C records.
+
+ set/var $DXProt::pc92c_ipaddr_enable 1
+
+ to enable the addition of IP addresses to outgoing PC92 C records.
+
+ ## These two features are experimental, but will become the default if
+ ## some other changes and these are successful
+24Jan23=======================================================================
+1. Optimise the PC11->PC61 promotion code, improve readability of the
+ debugging when 'set/debug pc11' is active.
+2. Remove spurious '$' character at the end of Listeners.pm (thanks Kin).
+3, Fix show/ik3qar command
+20Jan23=======================================================================
+1. Add the variable @main::localhost_names to allow other IP addresses to
+ be treated in the same way as localhost in item 1 on 19Jan23 below. NOTE
+ you must include ALL the normal localhost names + any other interface
+ names that you might want to include:
+
+ set/var @main::localhost_names qw(127.0.0.1 ::1 192.168.1.30)
+
+ using the qw() construction is easier than:
+
+ set/var @main::localhost_names ('127.0.0.1', '::1', '192.168.1.30')
+
+ but either will work. You can define as many IP addresses as you like and
+ they can be IPV4 or 6.
+
+ You do NOT need to fiddle with this unless you specifically have more
+ than just the normal definitions of localhost. So for 'normal' nodes with
+ one external interface, you DO NOT NEED TO DO ANY OF THIS.
+2. Added CTY-3304 prefix data
+3. Fix Bad IP address system so that one can carry on as normal without
+ Net::CIDR::Lite (or libnet-cidr-lite-perl on debian based systems)
+19Jan23=======================================================================
+1. Introduce aliasing for localhost in DX Spots and outgoing PC92 A records
+ on login. There are two variables which can be set with the alias to use:
+ $main::localhost_alias_ipv4
+ $main::localhost_alias_ipv6
+ These can be set in the /spider/scripts/startup, but this is only
+ necessary if the node has more than one interface, or virtual hosts. If
+ there is ONLY ONE ipv4 and/or ipv6 IP address on the node machine then
+ these variables will be automatically populated on first use. But the SAFE
+ thing to do is to set them in the startup file.
+
+ THIS FEATURE IS EXPERIMENTAL...
+18Jan23=======================================================================
+1. Make sure than *every* channel has an IP address. Thank you (I think) Kin
+ for pointing out that PC92 A records were not going out with IP addresses.
+ I'm guessing that other things (like spots) had a similar problem.
+15Jan23=======================================================================
+1. Fix strange errors for carp on missing route_*_cache files on startup.
+14Jan23=======================================================================
+1. Fixed route PC11 promotions so that a new PC61 is actually generated and
+ also sent instead of the original PC11 (to PC61 capable nodes).
+13Jan23=======================================================================
+1. Periodically store Routing tables and, if they are young enough (def: 3hrs)
+ autotically restore them on restart of the node. This will short circuit
+ the need to rebuild the routing tables from scratch on every restart -
+ which is normally for something like software update.
+2. Fix pc11 debugging stats with the correct figures. Sigh... Also move some
+ of the totals to a different place.
+3. Add show/spotstats command which gives the current spot statistics shown
+ during pc11 debugging (which means you don't need to set/deb pc11 unless
+ you really want that extra noise).
+12Jan23=======================================================================
+1. Regularise 'set/debug pc11' output to track all the routes through PC11 and
+ PC61 processing and statistics.
+11Jan23=======================================================================
+1. Improve (?) the PC11 -> PC61 upgrading process that delays incoming PC11s
+ for a very short time in the hope that a PC61 will come in to be used
+ instead. It will also upgrade a PC11 if we have an uptodate IP address
+ that has come in from the routing system PC92s. do a 'set/debug pc11' to
+ see it in action.
+2. I have chosen a definitive list of TOR exits and relays which can be
+ downloaded from http://www.dxspider.net/download/badip.torexit,
+ http://www.dxspider.net/download/badip.torrelay and finally, for those IP
+ addresses that are deemed to be 'bad':
+ http://www.dxspider.net/download/badip.global. I have added the following
+ lines to my /spider/local_cmd/crontab:
+
+24 * * * * spawn('cd /spider/local_data; wget -qN http://www.dxspider.net/download/badip.torexit')
+24 * * * * spawn('cd /spider/local_data; wget -qN http://www.dxspider.net/download/badip.torrelay')
+24 * * * * spawn('cd /spider/local_data; wget -qN http://www.dxspider.net/download/badip.global')
+25 * * * * run_cmd('load/badip')
+
+ The tor files are downloaded from https://lists.fissionrelays.net/tor/ at
+ 15 minutes past every hour, please would you use some other minute than
+ 23 or 24 to get your own local copies.
+
+ A 'set/debug badip' will show you what is being blocked.
+3. Fix set/badip so that it appends new IP addresses correctly.
+10Jan23=======================================================================
+1. Add baddx on incoming callsign in RBN.
+2. Search for all /spider/local_data/badip.* files to allow more control on
+ which IP addresses are detected. e.g. badip.torexit, badip.torrelay as well
+ as baddx.local. The suffixes, apart from .local (created by set/badip) are
+ completely arbitrary. You can use whichever suffix name you like. This is
+ a more useful arrangement for the ever increasing sources of "bad ip
+ addresses" that we need to deter.
+
+ NOTE: all badip.<suffix> are read only EXCEPT badip.local (which can be
+ altered in real time by the sysop using set/badip <ip address> ...).
+ If one uses periodic crontab jobs to update any other badip.<suffix>
+ files from web resources then don't forget to 'load/badip' afterwards.
+3. Add a /spider/data/baddx.issue file which can be copied to (or used as a
+ basis to create) /spider/local_data/baddx
+09Jan23=======================================================================
+1. Finish implemention of DXCIDR ip address filtering. This works on both
+ logins (treated the same as locked out - i.e. just disconnected) and also
+ with PC61s where these sentences are just dropped. Also attempt to prevent
+ any *following* PC11s with the same data getting through.
+
+ YOU WILL NEED either 'cpanm Net::CIDR::Lite' or debian/ubuntu based distros
+ 'apt install libnet-cidr-lite-perl'. RedHat based systems will have similar
+ packages available.
+
+2. Recognise PC18s coming from CC Clusters more nicely.
+04Jan23=======================================================================
+1. Fillout DXCIDR, attach checks in PC61 and logins. Login that fail will
+ simply disconnect, like locked out callsigns
+2. Fix qrz.com URL in stock Internet.pm.
+3. Fix DXHash issues (baddx, badnode, badspotter etc)
+30Dec22=======================================================================
+1. Add more BadWords (regex) checks.
+01Dec22=======================================================================
+1. Re-add some debugging to see which incoming PC protcol sentences are
+ being dumped because of any bad content (words or calls) if debugging
+ option 'nologchan' is set.
+2. All debugging for badwords has the word 'badword' in it. So that one can
+ do a 'grepdbg -1 badword' and one should see who said it and all of what
+ they said. Watchdbg will also work with this.
+3. My latest recommended additions to the badword list includes:
+ RU STOP SPOT DISGRACE CRM CRIM CRUD SUBHUMAN FASCIST DEGENERATE CATTLE CNT
+23Nov22=======================================================================
+1. The BadWord system has been rewritten. This change is pretty radical and
+ needs to be used with care as a word that is entered will be reduced to the
+ minimum sized string needed to match that word.
+
+ This is effective a sysop command changeable version of the file badw_regex
+ but in a much more sysop friendly form. The (un)set/badword <word>...
+ commands now update the /spider/local_data/badword.new file in real time
+ without having to mess about with editing files and running load/badword.
+ load/badword still works, but you should now never need it.
+19Nov22=======================================================================
+1. "Fix" Badword detection in spots and announces etc.
+ NOTE: setting $DXCommandmode::maxbadcount to 0 (default 3) will disable
+ autologout on swearing. But any (detected) swearing will be logged.
+ User generated swearing will not be propagated, not will incoming
+ protocol.
+2. Fix is_ipaddr so that it accepts more (most?) valid IPV6 addresses that
+ actual real IP addresses that might appear as endpoints.
+3. Add CTY-3237 prefixes.
+06Jun22=======================================================================
+1. Fix show/wcy so that it works even if no wcy broadcasts have been received
+ this month.
+01Jun22=======================================================================
+1. Fix show/(v)hfstats commands.
+2. Add CTY-3221 Prefixes.
+07Feb22=======================================================================
+1. (re)add set/seeme and show/seeme commands which don't appear to have
+ "taken".
+03Feb22=======================================================================
+1. Improve/add the help text for grepdbg and watchdbg.
+31Jan22=======================================================================
+1. Set $RBN::respottime to 3 minutes. NOTE change of name from
+ $RBN::minspottime.
+2. Allow callsigns connected to a node with an SSID to get SEEME RBN spots
+ for their base callsigns. So G1TLH-2, G1TLH-7 will get **SEEME** RBN spots
+ if G1TLH is spotted on the RBN system - even is are both connected to the
+ node and G1TLH is not. Each call is unique and each callsign (variation)
+ must have done a set/seeme + set/skimmer. .
+29Jan22=======================================================================
+1. Implement RBN set/seeme which displays any passing RBN spots for your
+ callsign in "raw" format.
+28Jan22=======================================================================
+1. Add Capabilities Line to logged in users.
+2. Make absolutely sure that all DB_Files are closed correctly.
+3. Introduce (un)set/debug rbnchan to control the visualisation of raw RBN
+ input lines.
+25Jan22=======================================================================
+1. Fixed grepdbg so that it does what -help says it does.
+24Jan22=======================================================================
+1. Change Local::pcprot() calling conventions to be the same as the rest of
+ software (see any 'sub handle_nn' function in DXProtHandle.pm.
+2. Corrected some interpolations in Messages.
+3. Tried to make users file deletes and updates more stable.
+22Jan22=======================================================================
+1. Refine spot display to retain the ':' and also reduce the likelyhood of
+ of misalignment by using the available space in the 39 characters more
+ intelligently. This is WIP.
+21Jan22=======================================================================
+1. Fix version tracking in PC92
+2. Bring the major parts of master and mojo together to make maintenance a
+ bit less challenging. This does add anything extra to mojo, but it's still
+ quite a bit of work.
+18Jan22=======================================================================
+1. fix (un)set/registered.
+09Jan22=======================================================================
+1. Add the New Year CTY-3201 prefix data.
+08Jan22=======================================================================
+1. I don't think there are any more warnings...
+06Jan22=======================================================================
+1. Fix the last of any frequent warnings caused by yesterday's modifications.
+05Jan22=======================================================================
+1. Mark nodes that send PC92 K records as spider. These will include VE7CC
+ nodes. NOTE: there appear to be user records marked as user or other sorts
+ of node, which (now) are actually spider (compatible) nodes and will be
+ marked accordingly.
+2. Adjust nodes currently marked as spider nodes, but are sending versions
+ not in the spider range of versions on PC92 A records as AK1A.
+3. Try to undo some damage where users have been autocreated with similar
+ attributes as nodes (locked out with privilege set to 1). This will
+ slowly fix this problem over time, but see item 4 for a 'big bang'
+ approach.
+4. It has come to my attention that there are a large number of users (of
+ all sorts) that have incompatible SSIDs. See 03Jan22/4 for details.
+
+ These are now being scrubbed out of the users file and also will present
+ as their normalised selves. If a -0* SSID is encountered then, if the
+ normalised version of that call is not present, it will be renamed to
+ that normalised call. If the normalised version of that user record is
+ already present, the un-normalised user record (-0*) will be removed.
+5. Make export_users do a batch clean (as in 3. above) and also get rid of
+ (default) 12+ year old unaccessed user records and (default) 2+ year old
+ "empty" records (with no qra/latlog/qth or handle).
+
+ NOTE: if you do an manual export_users (as opposed to the automatic one
+ done once a week), do not be alarmed by the number of old (i.e. more than
+ 12 years old) callsigns that it will get rid of. In my case it was about
+ ~2/5th of the users file. Still left me with over 100,000 "active" users.
+
+ In you are a bit twitchy about this, the code will copy the current
+ user_json and user_json.ooooo to user_json.keep and user_json.backstop
+ respectively. These files will never be overwritten unless you remove one
+ or both, when they will be regenerated on the next export_user.
+04Jan22=======================================================================
+1. Fix issue in the RBN (and probably other places) with callsigns that
+ contain trailing / in callsigns like: OH0K/6, K2PO/7 etc.
+2. Regard strange callsigns like DR4W-HB (seen in skimmer spots) as invalid.
+ This *should be* something like HB9/DR4W or (spit) DR4W/HB9.
+3. Fix the (probably) spurious locking out of users that are unknown to this
+ node, that come in from other nodes. These create new user records which
+ where then automatically locked.
03Jan22=======================================================================
1. Allow overrides (on modern versions of perl) with things in DXVars.pm, such
$clusterport. This is really only of use for people trying to run more than
one instance of DXSpider on the same machine.
2. Fix who command to make RBN connections as RBN and not USER.
+3. Prevent other nodes claiming that $myalias or $mycall is a different type
+ (user or node) from changing our route table and thence the user type.
+4. Normalise callsigns of incoming connections to G1TST if G1TST-0 or G1TST-00
+ amd G1TST-2 if G1TST-02. There are 800+ instances of callsigns with extra
+ 0 characters in the SSID in my users file. Allow SSIDs up to 99.
02Jan22=======================================================================
1. Fix dx bug introduced to handle dx by ip <ipaddr> for webclusters.
2. Remove _add_thingy dbg message from general view.
projects / spider.git / blobdiff