REXX Internet Utilities

Автор: Mike Ruskai

Дата: 1999

Скачать в ZIP архиве.

REXX Internet Utilities v0.13
Simple utilities using REXX sockets:
urlget.cmd  - fetch files via HTTP and FTP
ngfetch.cmd - obtain list of groups from
specified newserver.

This currently two-membered collection of REXX socket utilities has been started due to what I perceive as a lack of simple-function Internet utilities around.

These utilities require Object REXX, and will not function with SAA REXX. If you are running Warp v4, then execute "switchrx" from a command line to activate Object REXX if you've not already done so.  Object REXX is backwards compatible with SAA REXX, and offers many benefits completely independant from object-oriented programming.  If you are running Warp v3, you can download a copy of Object REXX for free from IBM.  The current URL for that download is It contains bug fixes for problems in the libraries that shipped with Warp 4, so users of that should upgrade to the latest fixpack, which includes the REXX update.


The utilities:

URL File Fetcher

This utility has a simple function - grab a file from a remote server using either FTP or HTTP.  Just run urlget.cmd with the URL of the file to grab. The protocol used is determined by the given URL.  Read urlget.txt for more information.

NNTP Group Lister

This utility also has a very simple function (a recurring theme) - grab a list of news groups from a news server.  Run ngfetch.cmd with the name of a news server as the argument.  It will fetch a list of all available news groups.  Read ngfetch.txt for more information.


Comments, questions, and problems can be directed to me, Mike Ruskai, at



02-27-98  First iteration of this package, including httpget and

02-28-98  Fixed stupid bug in ngfetch.cmd that lost groups sporadically,
          and wrote article number information instead.

03-03-98  Modified ngfetch to display number of groups received in
          status, and fixed screen display methods to work with the new
          version of Object REXX recently downloaded.  We'll call this

03-13-98  Modified httpget.cmd to display total file size (if available)
          and percentage completion.  While testing this, discovered a
          problem with files on the root directory that I created while
          spiffing it up to allow multiple URL's.  
          I also came across bad behavior on the part of Microsoft IIS 3.0.  
          I was issuing a simple GET message via HTTP 1.0, and IIS was
          replying with a 406 error, stating that there was no suitable
          reply type.  406 is not a valid return code in HTTP 1.0!  It's
          only valid in HTTP 1.1!  With HTTP 1.1, if the client doesn't send
          a header with the acceptable transfer encoding specified, the
          specs say that the server "MAY" assume any type.  Well, IIS was
          violating those specs by assuming *ONE* type, specifically
          text/html.  I've worked around this bug, which I hope is only in
          IIS, by putting in 'Accept:' and 'Accept-Encoding:' header items.  
          I've also put in a catch for 406 (even though no web server
          should be returning it to a HTTP 1.0 request), which will only
          happen with some really screwy web server that doesn't properly
          acknowledge the headers.  If you see this happening, please
          uncomment the lines marked by the 406 comments, and send the
          resulting file to, so I can see who the culprit
          server is.  

          I also reworked the break handling, since it was ignoring the
          HALT status while in an object method.  Should be able to press
          CTRL-C/BRK at any time without orphaning a socket now :)

          We'll call this one v0.04.  Working on digesting FTP specs...

04-07-98  Just a quick fix to remove boneheadedness on my part.  I
          neglected to include the comma deliminator procedure in ngfetch,
          so it doesn't work for anyone who doesn't have such a procedure
          somewhere in the path (i.e. everyone but me).  Not worthy of it,
          but this is v0.05, just to have a number.

04-26-98  Added quiet mode to httpget and ngfetch, to suppress both the
          calculations and display of transfer status.  
          Added stats line upon the end of transfer that prints
          out in quiet and non-quiet mode as final confirmation (and only
          confirmation in quiet mode).

          Added excessive error handling to httpget, since it's getting
          bulky, and I don't want to strand sockets on people's computers
          if it crashes for any reason not found during testing.  

          Altered usage display of httpget and ngfetch to be more
          clear for those not familiar with commandline argument standards,
          though you still need to know that <> denotes mandatory, []
          denotes optional, and | means "or".  

          Fixed a very obscure "bug" that probably never happened to
          anyone.  It's only by chance that I found it at all.  It could
          only happen with a *fast* connection.  What could happen is that
          the transfer of the first 10240 bytes would happen so quickly
          that the REXX timer (which has a 0.01 second granularity) would
          give an elapsed time of 0 seconds, resulting in a divide-by-zero
          error when calculating the transfer rate.  Just made any zero
          times 0.01.

          Added User-Agent: tag to the header, specifying HTTPGet and
          Object REXX with their respective versions.

          Reformated history to be more readable (so I cheated).

          Call this version 0.06.  

05-31-98  Added version checking to prevent the programming from looking
          really nasty as it crashed with SAA REXX.  Don't know why this
          wasn't done in the first place.  After receiving a few e-mails
          about it, though, I've been straightened out

08-07-98  Modified the display behavior of httpget to not show all the
          transfer status junk if the remote file doesn't exist.

          Will only display local storage name if it differs from the
          remote filename (which includes saving to a different directory

          No more messages about socket handling; no longer says "Done!",
          which is obvious by that point.

          Modified the transfer rate calculation to reflect the "current"
          rate, rather than the overall rate so far.  What's actually
          calculated is the rate of the last four packets received.  Will
          not be very accurate for wildly erratic transfer rates.

          Tokenization pushes the EA size for this sucker to about 62KB,
          pushing the limit.

          0.07 shall be the name of this creature.  Still working (slowly)
          on FTP specs.

08-12-98  Suddenly decided to implement FTP, and after a couple of hours of
          imagining just how to kill the writers of RFC959, httpget has gone
          the way of the dodo.  Replacing it is urlget, which will fetch
          files via either HTTP or FTP, as always by URL.  In the process,
          found some dormant bugs and fixed them.

          What's *actually* calculated for transfer rate is the last four
          measured intervals.  For faster connections, that will entail
          many more than four packets (between me and my server, it's
          about 1200 packets).

          Tokenization is history.  The spiffy new urlget is more than
          twice the size of httpget, running headfirst into the EA size

          Added a bit of code to gracefully fail if the programs cannot
          load the REXX Sockets functions.

          Updated and better implemented the error levels of ngfetch.cmd.

          Added errorlevels to urlget.cmd.

          This will be version 0.08, with no big ideas for other utilities
          left.  Will work on logging for next release, and tuning urlget
          up a bit.

08-13-98  Knocked off about 13KB worth of code from urlget.  I dislike a
          string of repetitive commands that are only slightly different,
          so I made an arcane FTP command class, and looped through the
          server login and configuration procedure, with catches for the
          extra work that some commands required.  In the process, I
          learned that either ::routine provides no way of exposing
          main body variables, or the way to do it is undocumented.  This
          necessitated the passing of some variables that should be global.

          Tweaked a couple other things, to (hopefully) properly handle
          cases where either the filename or directory contain spaces.

          Fixed the handling of spaces in either the filename or path.

          This will be version 0.09.

09-18-98  Made a kludge to work around a bug in NCSA HTTPd version 1.5,
          where response headers were followed by LF characters, instead
          of the CRLF pairs required by HTTP 1.0 (or 1.1, for that matter).
          Fixed a problem with unexpected failures of the STAT command,
          which some inferior FTP servers actually don't support.  Also
          modified the parsing to work with a non-standard reply of STAT
          by a server I found.

          After learning that some FTP servers will deny passive mode for
          (dubious) security reasons, added the ability to transfer
          normally, with the server initiating the data connection.  To
          make this work properly, added a firewall flag to the program,
          which is part of the new config file (with all of two entries).
          If a server refuses passive mode, and the machine is configured
          as behind a firewall, then the transfer will fail (can't be
          done).  Otherwise, a new socket is created, bound to, and
          listened on before sending the file request.  I was shocked that
          it worked the first time I tested it
.  Note:  If your
          firewall uses IP masquerading (or something like it), it may be
          possible for FTP transfers to work without passive mode.  It
          depends on whether the IP masquerader is smart enough to know
          that FTP servers initiate data connections from the port number
          immediately preceding the control connection port.  I don't know
          one way or another.  If that's the case, then make the firewall
          setting in the config file 0.

          Wanted to do logging, but have been spending my free time with
          C++ making a trivial program called the WarpAMP Playlist Editor.
          Will do logging for next release, unless I discover bugs to be
          quickly fixed.

          This is version 0.10.

10-01-98  Worked around the same bug as above for NCSA 1.5.1.

          Worked around flaky FTP servers (such as Serv-U 2.3b for Windows)
          reacting to a STAT command with parameters as if it were a STAT
          command with no parameters.  That is, it was returning a success
          code of 211 when it should properly have issued a failure code,
          since it did not process the command as requested.  

11-29-98  Rewrote the FTP handling completely.  Removed the arcane command
          class to allow for more flexibility.  As a result, the bulk of
          the program has increased, but it doesn't matter much, because
          it won't tokenize either way without using rexxc.exe to create
          a tokenized "executable".

          Added FTP status messages, so that if nothing's happening, one
          can at least see where the hangup is.

          Added resume capability for FTP receives.

          Added logging.  Put "logging=1" in the config file to enable it.

          Reworked the command line parsing.  It worked fine, but didn't
          allow parameters very easily.  Now it does.

          Added command line switches:

            /l  - explicitly enable logging, regardless of config setting.
            /r  - attempt to resume FTP transfer if the local filename.
                  exists and is smaller than the server copy.
            /f  - same as /r, but will abort the transfer if the FTP server
                  doesn't support resume, or the file size on the server
                  can't be determined.
            /p  - attempt transfer when passive mode denied, and behind a
                  firewall - probably won't work, but what the heck.
            /d# - set the delay between attempting to send and receive data
                  during FTP processing - some servers seemed to break when
                  requests were made too quickly.  Default delay is 0.10
                  seconds, maximum allowed setting is 60 seconds.
            /b# - set the transfer block size.  Generally speaking, larger
                  blocksizes make for quicker transfers, to a point.  Valid
                  values are from 512 bytes to 65535 bytes.  The default is
                  10240 bytes.
            /m# - set the maximum number of retries for failed socket
                  transfers.  The default is 1000, and the maximum is
                  999999999.  This can be increased for patient people.

01-11-99  Somewhere along the way, fixed a few argument parsing problems,
          and perhaps some other small problems.

          Seems problem-free enough to release as version 0.12.

01-23-99  Changed ngfetch.cmd to use the same rate calculations as
          urlget.cmd, so that the number would be based on the current
          rate, rather than the average rate.  Also fixed it to close the
          socket after it's done, which was removed at some unknown point
          in the past, for some unknown reason.  Also changed it so that
          any existing .newsrc file is not overwritten until the server
          looks as if it will cooperate, and send the data.

          Changed the configuration file of urlget.cmd to be an INI
          profile, instead of a text-file.  This makes it easier to deal
          with programmatically.  If it gets more complicated than three
          variables, I'll put in the ability to change them at will, one at
          a time.  As it is, if you want to change something, you'll need
          to delete the INI file and answer three questions, short of some
          hex editing, or your own programmatic solution.

          This will be version 0.13.