The Limitations of Usenet File Sharing

In this section, we are going to discuss the basic ideas involved in using Usenet to upload and download large binary files. As a preview of our discussion, the two tables you see below summarize the steps we follow.

If this is the first time you are reading about such topics, you may find it a bit confusing. My overall goal is to make sure you understand the reasons behind each of the various steps, no don't worry. I promise that, by the time you finish reading this section, everything in the two tables will make sense to you.

During our discussion, I am going to mention the various file formats that are used as part of the file share process: RAR, SFV, PAR2, NFO, yEnc and NZB. In the following sections, I am going to take you through all the details, one step at a time. As I do so, I will explain more about each file format; the programs you need to create and manipulate these files; and where to download these programs (for free).

Before we do that, however, I want to make sure that everything makes sense to you. To do this, I am going to discuss the inherent limitations of Usenet as a file sharing system, and how these limitations affect the procedures we follow. By the time you finish reading this section, you will understand why we need to go through so many steps whenever we upload or download large binary files.

Usenet File Sharing Procedures for Large Binary Files

Uploading
1. Compress the original file and break it into smaller files.
2. Generate a checksum file for error checking.
3. Generate error correction files for restoring lost data.
3. Collect all the files together and encode them as text.
4. Create an information file to be distributed with the data files.
5. Create an NZB (retrieval) file with a list of all the components.
6. Upload all the files, as separate articles, to a Usenet server.

Downloading
1. Download the NZB (retrieval) file from a Usenet server.
2. Using the NZB file as a guide, download all the separate articles.
3. Decode the text files back into binary files.
4. Check to see if there is lost or corrupted data.
5. If necessary, use the recovery information to restore data.
6. Uncompress all the pieces and recreate the original binary file.
7. Read the NFO file to learn about the file and how to use it.

Usenet was developed in the 1980s. At that time, the goal was to design a distributed system for passing short text messages from one computer to another. Over the years, as technology has changed, Usenet has been enhanced in a variety of ways.

Nevertheless, for the most part, we still use the basic design from the 1980s. As a result, when it comes to file sharing, Usenet has seven important limitations, which we must take into account when we upload and download. They are:

• Smaller Is Faster
• Limit on Maximum Article Size
• No Built-in Error Detection and Correction
• No Public Search Engines
• No File Descriptions
• Only One Article at a Time
• Text Only — No Binaries

Let's take a look at each of these limitations in turn, and how it affects what we must do in order to use Usenet to share large binary files.

Smaller Is Faster

As fast as Usenet can be when you use a high- speed connection and a commercial Usenet server, it is still not fast enough, especially when you are uploading or downloading large files. Thus, the less data you need to move, the better.

For this reason, in order to reduce the transmission time, you must compress all large files before you upload them. At the other end, the downloaded files must be decompressed back into their original form. The program you use to compress such files is called WinRAR.

Limit on Maximum Article Size

As we discussed earlier in this tutorial, we send data to a Usenet server in the form of "articles". All Usenet servers have a maximum size for the articles they will accept. Although the maximum acceptable size has been increased over the years, it still exists and, for reasons of efficiency, this limit is not likely to be eliminated. If you want to upload a file that is larger than the maximum allowable size, you must first break the file into smaller pieces, each of which must be uploaded as a separate article. This also means that, in order to download a large file, you must download all the pieces separately and then re-assemble them back into the original file.

The WinRAR program I mentioned above is the one that does the work for you. It is used to both compress files and then divide them into pieces. The output of WinRAR is a set of files known as RAR files. We'll talk about the details later in the tutorial.

No Built-in Error Detection and Correction

With large files, it is common to end up with tens, or even hundreds of separate pieces. When so many packages of data are transmitted over the Internet, it happens occasionally that some of the data does not survive the journey intact. In such cases, there needs to be a way to detect and restore missing data.

For this reason, it is common to send two types of extra files along with the data. First, we create and upload what is called an SFV file. The SFV file contains what is called a "checksum", that can be used later to verify that the new, re-created file is identical to the original file.

In addition, we also create and upload a group of files, called PAR2 files, that contain information which, if necessary, enables us to fix transmission errors or replace missing data. We'll talk about the details later in the tutorial.

No Free Search Engines

Unlike the Web, Usenet does not have any comprehensive, public search engines. Similarly, there is no way to find the files you want by browsing Usenet, either by topic or by name.

To fill this need, a few Usenet providers run their own Usenet search engine, which you can access for free if are a customer. In addition, the NewsBin newsreader company has a search service to which you can subscribe for a small monthly fee. These services are invaluable: without them, it can be extremely difficult to find what you want. Nevertheless, they are not nearly as comprehensive as the Web-based search engines (Google, Bing, Yahoo) you are used to.

No File Descriptions

When you download a file, it is helpful to get technical information about the file and how it should be used. Usenet itself makes no provision for including such information along with a file: when you download anything, you are on your own.

For this reason, there is a convention that, whenever you upload a large complicated file, you also include a special NFO (information) file. (There are programs available to help you create such files.) Whenever you download a large file, the first thing you should do is read the accompanying NFO file to learn about what you have just downloaded.

Only One Article at a Time

As we have discussed, large binary files must be broken into pieces, all of which are uploaded as a separate Usenet articles. In many cases, this requires you to download tens, or even hundreds of separate articles to get all the data for one large binary file. The question arises, how can you find the many smaller files that contain the data for a particular file, and how do you make sure you have got them all?

To solve this problem, before you upload a large file, you create a special file to help with the downloading. This special file, called an NZB file, contains a detailed list of all the articles you will be posting that are necessary to recreate the original file.

Modern newsreaders (Usenet programs) know how to use the information in an NZB file to figure out which articles to downloaded. Typically, all you need to do is find the NZB file for a file you want, download the NZB file, open it with your newsreader, and then wait for the newsreader to download all the pieces and put them together automatically. (It's truly an amazing system.)

Text Only — No Binaries

Usenet has always been a text-only system. Thus, before binary data can be uploaded, it must be encoded as text. The convention is to use an binary-to-text encoded format named yEnc.

Thus, all the RAR and PAR2 files you generate must be converted to yEnc files before they can be uploaded. This work is done automatically by the program that posts (uploads) the files for you in the form of separate Usenet articles. (SFV, NFO and NZB files are already text.)

Thus, in order to download a large binary file, you first download all the yEnc files, which are then converted back into binary files. These binary files are then used to re-create the original binary file. All of this is done for you by your newsreader program.

Jump to top of page