Notes about a new NNTP client and server Mikael "MC" Cardell 2001-08-24 (6) * Introduction These are some notes I've made in trying to design a News server and a KOM look alike client speaking NNTP, without losing the sense of a KOM conference system. I'm planning to extend the nntp backend in Gnus to support my extensions below. I want server side subscription handling. Should it be possible to invite people? The canonical way of handling server side subscriptions is with LIST SUBSCRIPTIONS, but that command is not included in the new draft-ietf-nntpext-base-13 for NNTP. The canonical answer from LIST SUBSCRIPTIONS is in the form "groupname". Perhaps we could expand this answer in the form of "groupname number-unread"? I'm thinking about keeping user data and message indexes in a local Berkeley DB and the article spool in pre-allocated cyclic files for large traffic groups and use expanding storage for groups that shouldn't expire at all. Please note that, when implementing. Don't use an int on a 64 bit machine to store local article numbers, or if you do, check the value carefully, because: Article numbers MUST lie between 1 and 4,294,967,295 inclusive. The client and server SHOULD NOT use leading zeroes in specifying article numbers, and MUST NOT use more than 16 digits. In some situations, the value zero replaces an article number to show some special situation. * Scalability and Load Balancing Use one server for incoming feeds and another for readers, as INN does? Possibly keeping them on separate machines? How does BDB handle that? Locking issues? The news feed server might only have to handle a few commands, since the standard says: MODE READER SHOULD be sent by any client that intends to use any command other than IHAVE, HEAD, STAT, LIST, LIST EXTENSIONS, or commands advertised by the server as available via LIST EXTENSIONS. Shared disk array? DNS round robin? * Administration To facilitate administration we have a number of suggestions: 1) to keep a small control program on the physical server machine that speaks with the News server through a pipe or Unix domain socket. Possibly, the control program just modifies a configuration file and then send a signal (usually SIGHUP) to the server process to coerce it to re-read it's configuration files. It is used through logging in to the machine in traditional ways, through the use of SSH or perhaps Kerberos telnet, or, 2) an HTTP server, possibly with SSL support, runs on the server machine and acts as a frontend to the above control program, running as a CGI script or possibly as a servlet. The above can be combined. Perhaps the control program in 1) should be available with a command line system, possibly reminiscient of the CMND% interface provided by TOPS-20, but if implemented it should be possible to call the the control command with only flags and arguments as well. * Server NNTP Extensions We should list these extensions when given a LIST EXTENSIONS from the client. A successful response starts with a 202 code and is followed by a list of extensions, one per line. Each line MUST begin with exactly one space followed by an extension-label and optionally one or more parameters (separated by single spaces). The extension-label and the meaning of the parameters are specified as part of the definition of the extension. The extension-label MUST be in uppercase. The server MUST NOT list the same extension twice in the response, and MUST list all supported extensions. The order in which the extensions are listed is not significant. The server need not even consistently return the same order. If the server does not support any extensions, it SHOULD return a 402 failure response but MAY return an empty list instead. 202 extension list follows (multiline) 400 service about to terminate 402 no extensions available 503 unable to list extensions Asynchronous updates from the server to show new articles being posted. Asynchronous messages from other users. These updates are enabled with the extension command: XASYNC. WHO - A command to list current users on the server. SAY - A command to send messages to another logged in user. REFCNT - A command to start using reference count commands, e.g. C: ref1 list S: ref1 555 Newsgroup list follows. et cetera. NAMES - A command to list translation from local user friendly names to transport names (compare "International Unix Gurus" and "comp.unix.gurus"). USRINF - A command to list user information for a specific user, including what article is the user's introduction. Command lines MUST NOT exceed 512 octets, which includes the terminating US-ASCII CRLF pair. Arguments MUST NOT exceed 497 octets. 1xx - Informative message 2xx - Command ok 3xx - Command ok so far, send the rest of it. 4xx - Command was correct, but couldn't be performed for some reason. 5xx - Command unimplemented, or incorrect, or a serious program error occurred. The next digit in the code indicates the function response category. x0x - Connection, setup, and miscellaneous messages March 2001 x1x - Newsgroup selection x2x - Article selection x3x - Distribution functions x4x - Posting x8x - Reserved for authentication and authorization extensions x9x - Reserved for private use (non-standard extensions) * Authorization At least plaintext password. Kerberos authorization support? MD5 challenge/response? Should be simple. * Character Handling UTF-8 is default in the revised NNTP standard description. Is it possible to use binary coded Latin 1 directly? UTF-8 is expected in the arguments for NNTP commands. No way to negotiate anything else, afaik. * Mail Handling The NNTP server will also listen for inbound (E)SMTP connections to handle import of mailing lists and for those who want to read mail in their News client. Special data in the subscription handling notes that the subscriber is external and sends incoming articles by mail. This is done a keyed index (newsgroup.db/externals) for each newsgroup with listed external recipients for speed. Mailing lists are seen as a special case of a local user. (Perhaps a special client will handle this?) A database of Message-ID (and possibly MD5 sums of the artictle text?) is held to keep duplicates out of the spool. * IP v6 Support? Should the NNTP server be able to handle requests provided via TCP over IPv6? * New Client A KOM lookalike client speaking NNTP + above extensions, with fallback to straight NNTP. It should be rather easy to do with Python and the NNTP class, I think. * References Network News Transport Protocol, Internet Draft, March 2001. Network News Transfer Protocol (RFC 977), March, 1986. Common NNTP Exenstions (RFC 2980), Usenet Article Format? (RFC 1036)