Harley Hahn's
Usenet Center

Donation?

Home Page

File Sharing   NEW 
• Tutorial
• Posting Binaries

Usenet Providers

Quick Start Guide

Hands-On Guide

Le petit guide
illustré sur Usenet

Master List
of Usenet
Newsgroups

Harley Hahn's
Usenet Tutorial

Usenet Glossary

Internet
Resources

Site Map

About Harley


Harley Hahn
Home Page

Send a Message
to Harley

  Top Usenet
  Providers
  Thundernews
  Newsdemon
  Newshosting
  Giganews
  Astraweb
  Usenext

  Top Usenet
  Newsreaders
  NewsBin


HARLEY HAHN'S USENET CENTER

File Sharing Tutorial


Understanding yEnc Files and Segments

In the last few sections of this tutorial, we have been preparing to share a 640-megabyte video file named steal-this-film-2.avi. Because this is a large binary file, we needed to perform several steps ahead of time to create the files we will need for uploading:

RAR Files

To start, we used WinRAR to create a set of compressed files, called RAR Files, to hold the data. Once a user has downloaded the RAR files, his newsreader will use them to recreate the original data file (in our example, the AVI file).

SFV File

Using QuickSFV, we then created a checksum file, called an SFV file. A user can use the SFV file to check that the new, recreated data file is an exact duplicate of the original file.

PAR2 Files

Next, we used QuickPar to create a set of special files, called PAR2 files, consisting of a PAR2 information file and a set of PAR2 data files. If there is a data transmission problem, a user can use the PAR2 files to fix corrupted data or to replace missing data.

NFO File

We then used CoolBeans NFO Creator to compose an information file, called an NFO file. The NFO file contains information to help the user understand the contents of the data file and how to use it.

NZB File

Finally, just before we post (upload) the files to our Usenet server, we must generate one more file, called an NZB file. The NZB file contains technical information that enables a newsreader program to retrieve all the other files. (We'll talk more about NZB files later in the tutorial.)

A Summary of the Files We Are Going to Share

Before we move on, I want you to take a moment to appreciate how complex the situation has become. In order to share a single AVI (video) file, we had to create a large number of separate files. In fact, as you can see from the table below, we have created 56 different files, all of which need to be shared with any user who wants our video.



HARLEY HAHN'S
FILE SHARING TUTORIAL

(New to Usenet?  Try the Usenet Tutorial)
(Are you experienced?  Quick Guide to Posting Binaries)

1. Introduction / Usenet Terminology
2. Binary Files and Text Files
3. Why Usenet File Sharing Works Well
4. Is File Sharing Legal?
5. Anonymous File Sharing
6. Limitations of Usenet File Sharing
7. Summary: Uploading/Downloading
8. Uploading Step 1: RAR Files
9. Uploading Step 2: SFV Files
10. Uploading Step 3: PAR2 Files
11. Understanding PAR2 Files
12. Uploading Step 4: NFO Files
13. Understanding yEnc Files, Segments
14. The Process of Posting a File
15. Understanding NZB Files
16. Looking Inside a Typical NZB File
17. Uploading Step 5: Preparing to Post
18. Uploading Step 6: Posting Files
 
  More to come...

The 56 Files to Be Shared

Purpose Type
of File
File Names Created
Files
 compressed
 data
 RAR  steal-this-film-2.part01.rar
 steal-this-film-2.part02.rar

 steal-this-film-2.part42.rar
 steal-this-film-2.part43.rar
43
 checksum  SFV  steal-this-film-2.sfv 1
 error
 correction
 PAR2 info  steal-this-film-2.par2 1
 PAR2 data  steal-this-film-2.vol000+01.PAR2
 steal-this-film-2.vol001+02.PAR2
 steal-this-film-2.vol003+04.PAR2
 steal-this-film-2.vol007+08.PAR2
 steal-this-film-2.vol015+13.PAR2
 steal-this-film-2.vol028+24.PAR2
 steal-this-film-2.vol052+41.PAR2
 steal-this-film-2.vol093+41.PAR2
9
 information  NFO  steal-this-film-2.nfo 1
 retrieval  NZB  steal-this-film-2.nzb 1
Total   56

What about file sizes? It happens that the original AVI file in our example is 639 megabytes. The 56 files we need to share have a total size of 705 MB. That is, we must share about 10 percent more data than the size of the original file.

Creating 56 files in order to share a single video file, seems like a lot. However, our example is going to become even more complicated: the 56 files we need to share are going to give rise to no less than 1,895 files, all of which will need to be posted to our Usenet server in order to share our original file.

Encoding Binary to Text Using yEnc

1,895 files?

Why on earth would we have to upload 1,895 different files in order to share a single video? The answer to this question is based on three important limitations we discussed earlier in the tutorial:

  • Data is stored on Usenet servers in the form of articles (often called posts or postings).
  • Usenet articles can contain only text.
  • Usenet servers place strict limits on the size of the postings they will accept.

Thus, before we can share a binary file (such as our AVI file), it must be encoded as text. The text is then split into pieces and packaged as separate articles called SEGMENTS. The segment size must be chosen so as to be acceptable to a Usenet server (that is, it must not be too large).

Once this process is complete, all the individual segments must be uploaded to a Usenet server. At the other end, a user who wants a copy of our file must download all the segments, decode them, and then use them to recreate the original data file.

As you can imagine, this is a lot of work and the details are complex. However, it is all done for us automatically: specifically, the encoding, splitting, and uploading will be done by our posting software. (In this tutorial, we will be using a program called Camelsystems Powerpost.) The downloading, decoding, and re-creation will be done by a newsreader program.

Over the years, a number of different systems have been developed to encode binary data into text data. The current encoding system is called YENC (usually spelled yEnc). yEnc was developed specifically for Usenet by a programmer named Jürgen Helbing, who released it into the public domain in 2001.

What is the origin of the name "yEnc"?

In 1997, Jürgen Helbing started work on a Usenet-related project called MyNews. A few years later, in 2001, Helbing wanted a similar name for a new binary/text encoding system he was developing. However, this type of name had become popular among Internet programmers, and "MyEncode" was already being used by someone else.

To create a new, unique name, he simply dropped the initial "M" and came up with yEncode, which he then shortened to yEnc.

It happens that the SFV file, NFO file, and NZB file are all small text files, so they can be uploaded without a problem. The RAR files and PAR2 files, however, are much larger binary files. As such, they must be encoded as text and then broken into segments.

In our example, we have 43 different RAR files, all of which are the same size (except the last one, which is smaller). As you will see later in the tutorial, when we use our posting software to encode and split the RAR files, we will generate 41 segments per file (except for the last RAR file, which requires only 27 segments.) Thus, our 43 RAR files will be transformed into (42*41) + (1*27) = 1,724 segments.

Similarly, the PAR2 files will all need to be encoded and split into segments. As you would expect, the larger the file, the more segments it generates. In our example, the small PAR2 information file will require only 1 segment, while the largest PAR2 data files will require 42 segments each.

I have summarized these numbers for you in the table below. As you can see, in order to share one AVI file, we needed to create 56 files. And to share these 56 files, we need to upload 1,895 segments, each of which must be posted as a separate Usenet article.

The 1,895 Encoded yEnc Files to Be Uploaded

Purpose Type
of File
File Names Created
Files
Encoded
Files
 compressed
 data
 RAR  steal-this-film-2.part01.rar
 steal-this-film-2.part02.rar

 steal-this-film-2.part42.rar
 steal-this-film-2.part43.rar
43 41
41

41
27
 checksum  SFV  steal-this-film-2.sfv 1 1
 error
 correction
 PAR2 info  steal-this-film-2.par2 1 1
 PAR2 data  steal-this-film-2.vol000+01.PAR2
 steal-this-film-2.vol001+02.PAR2
 steal-this-film-2.vol003+04.PAR2
 steal-this-film-2.vol007+08.PAR2
 steal-this-film-2.vol015+13.PAR2
 steal-this-film-2.vol028+24.PAR2
 steal-this-film-2.vol052+41.PAR2
 steal-this-film-2.vol093+41.PAR2
9 2
3
5
9
14
25
42
42
 information  NFO  steal-this-film-2.nfo 1 1
 retrieval  NZB  steal-this-film-2.nzb 1 1
Total   56 1,895

The numbers you see here are typical. Sharing a large binary file really does require us to generate and post thousands of separate segments, each one as a separate Usenet article. Moreover, if even one segment is lost, the entire file sharing process can fail. This raises two obvious questions:

  • How can anyone accurately organize thousands of segments?
  • At the other end, how can a user make sure he has downloaded and properly processed all the segments he needs to recreate the original file?

The answer to these questions lie in the use of a very special kind of file, the NZB file, which was invented in 2002. Since that time, the use of NZB files has created a revolution in how people around the world share files using Usenet. Indeed, as you will see, it was actually the invention of the NZB file that made modern Usenet file sharing possible.

At this point, I'd like to put our discussion on pause for a moment to show you how to share our sample file (the AVI file). To do so, we will use a "posting program" to create the segments we need and then upload them to a Usenet file sharing group. After you see how this works, we'll talk more about NZB files, at which time I will answer the two important questions I posed above.

Jump to top of page