Difference between revisions of "Freenet"

Latest revision as of 08:49, 10 November 2007

This article is inacurate in many places. It needs to be changed.

Freenet is a decentralized censorship-resistant peer-to-peer distributed data store. Freenet works by pooling the contributed bandwidth and storage space of member computers to allow users to anonymously publish or retrieve various kinds of information. Freenet uses a kind of key based routing similar to a distributed hash table to locate peers' data.

Freenet is currently under development, and a version 1.0 has not yet been released. Freenet is considered by many to be fundamentally different from other peer-to-peer networks; it is more difficult to use, slower, and does not have integrated search functionality. According to the Freenet Project group, such tradeoffs are expected since Freenet's primary goals are neither ease-of-use, nor performance. Unlike other peer-to-peer networks, Freenet is primarily intended to combat censorship and allow people to communicate with near-total anonymity.

Purpose

Although many nations censor communications to different extents, they all share one commonality in that a body must decide what information to censor and what information to allow. What may be acceptable to one group of people may be considered offensive or even dangerous to another. Freenet is a network which, putatively, removes the possibility of any group imposing their beliefs or values on any other. In essence nobody is allowed to decide what is acceptable for anybody else. Tolerance for each others' values is encouraged and failing that, the user is asked to turn a blind eye to content which opposes his or her views.

Technical design

The type of network routing method Freenet uses is key based routing. While the idea emerged independently, Freenet's routing algorithm is similar to that employed by distributed hash tables (DHTs). The main differences are that Freenet nodes do not have fixed specialisations, and the routing algorithm is heuristic in nature. Therefore, it does not guarantee that it will find a given piece of data. Freenet can also be viewed as a small world network.

The Freenet file sharing network is designed to be highly survivable, with all internal processes completely anonymized and decentralized across the network. The system has no central servers, is peer-to-peer, and is not subject to the control of any one individual or organization. Even the designers of Freenet do not have any control over the overall system. The system is designed so that information stored in the system is encrypted and replicated across a large number of continuously-changing anonymized computers around the world. It is extremely difficult for an attacker to find out which participants are hosting a given file, since the contents of each file are encrypted, and can also be broken into sections that are distributed over many different computers. Even the participants themselves don't know what they are storing.

The end goal of the Freenet network is to store documents and allow them to be retrieved later by an associated key, as is now possible with protocols such as HTTP. The network is implemented as a number of nodes that pass messages among themselves peer-to-peer. Typically, a host computer on the network will run the software that acts as a node, and it will connect to other hosts running that same software to form a large distributed network of peer nodes. Certain nodes will be end user nodes, from which documents will be requested and presented to the human user. But these nodes communicate with each other and with intermediate routing nodes identically—there are no dedicated "clients" or "servers" on the network.

The Freenet protocol is intended to be implemented on a network with a complex network topology, much like the Internet (Internet Protocol). Each node knows only about some number of other nodes that it can reach directly (its conceptual "neighbors"), but any node can be a neighbor to any other; there is no hierarchy or other structure. Each document (or other message such as a document request) in Freenet is routed through the network by passing from neighbor to neighbor until reaching its destination. As each node passes a document to its neighbor, it does not know or care whether its neighbor is just another routing node forwarding information on behalf of another, whether it is the source of the document being passed, or whether it is a user node that will present the document to an end user. This is intentional, so that anonymity of both users and publishers can be protected.

Each node maintains a data store containing documents associated with keys, and a routing table associating nodes with records of their performance in retrieving different keys.

To find a document in the network given a key, a user sends a message to a node (probably one running on the same machine as the client program) requesting the document, providing it with the key. If the document is not found in the local data store, the node then finds the node in its routing table that it thinks will be able to locate the key most quickly, and forwards the request to that node, remembering that it has done so. Note that this is a change from the behavior of earlier versions of Freenet nodes: it represents the "Next Generation Routing" protocol. The old behavior was to remember which keys were retrieved from what nodes, and to route based on which node gave us the key closest to the one we were looking for. The effect is largely the same, but NGR, as it is called, should result in better overall performance.

The node to which the request was forwarded repeats the process until either the key is found or the request passes through a set maximum of nodes, known as the "Hops To Live" value. Along the route, if a node is visited more than once (and it will know this because it remembered forwarding the request the first time) then that node cuts off the loop by sending a message to the node that sent it the second request telling it to try the next-best choice, then the next-next-best, and so on.

Eventually, either the document is found or the hop limit is exceeded. The terminal node sends a reply that works its way back to the originator along the route specified by the intermediate nodes' records of pending requests. The intermediate nodes may choose to cache the document along the way. Besides saving bandwidth, this also makes documents harder to censor as there is no one "source node."

Essentially, the same path-finding process is used to insert a document into the network: a request for the nonexistent document is made, and once it fails, the document is sent along the same path as the request. This insures that documents are inserted into the network in the same place as requests will look for it. If the initial request doesn't fail, then the data already existed, and the insert "collides."

Initially, each node has no information about the performance of the other nodes it knows about. This means that routing of requests will be essentially random. But since different nodes have different randomness, they will disagree about where to send a request, given a key. So the data in a newly-started Freenet will therefore be distributed somewhat randomly.

As more documents are inserted by the same node, they will begin to cluster with data items whose keys are similar, because the same routing rules are used for all of them. More importantly, as data items and requests from different nodes "cross paths", they will begin to share clustering information as well.

The result is that the network will self-organize into a distributed, clustered structure where nodes tend to hold data items that are close together in key space. There will probably be multiple such clusters throughout the network, any given document being replicated numerous times, depending on how much it is used. This is a kind of "spontaneous symmetry breaking", in which an initially symmetric state (all nodes being the same, with random initial keys for each other) leads to a highly asymmetric situation, with nodes coming to specialize in data that has closely related keys.

There are forces which tend to cause clustering (shared closeness data spreads throughout the network), and forces that tend to break up clusters (local caching of commonly used data). These forces will be different depending on how often data is used, so that seldom-used data will tend to be on just a few nodes which specialize in providing that data, and frequently used items will be spread widely throughout the network. This automatic mirroring counteracts the times when web traffic becomes overloaded, and due to a mature network's intelligent routing a network of size n should only require log(n) time to retrieve any given document. Freenet does not employ broadcast searches as used by Gnutella and other similar file sharing protocols.

One thing to keep in mind is that keys are hashes, hence there is no notion of semantic closeness when speaking of key closeness. Therefore there will be no correlation between key closeness and similar popularity of data as there might be if keys did exhibit some semantic meaning, thus avoiding bottlenecks caused by popular subjects.

There are two main varieties of keys in use on Freenet, the Content Hash Key (CHK) and the Signed Subspace Key (SSK).

A CHK is an SHA-1 hash of a document and thus a node can check that the document returned is correct by hashing it and checking the digest against the key. This key contains the meat of the data on freenet. It carries all the binary data building blocks for the content to be delivered to the client for reassembly and decryption. The CHK is unique by nature and provides tamperproof content. A hostile node altering the data under a CHK will immediately be detected by the next node or the client. CHKs also reduce the redundancy of data since the same data will have the same CHK.

SSKs are based on public-key cryptography. Currently Freenet uses the DSA system as its public key infrastructure. Documents inserted under SSKs are signed by the inserter, and this signature can be verified by every node to ensure that the data is not tampered with. SSKs can be used to establish a verifiable pseudonymous identity on Freenet, and allow for documents to be updated securely by the person who inserted them. A subtype of the SSK is the Keyword Signed Key, or KSK, in which the key pair is generated in a standard way from a simple human-readable string. Inserting a document using a KSK allows the document to be retrieved and decrypted if and only if the requester knows the human-readable string; this allows for more convenient (but less secure) URIs for users to refer to.

Scalability

A P2P network is said to be scalable if the performance of the network does not deteriorate even for very large network sizes. The scalability of Freenet is being evaluated, but similar architectures have been shown to scale logarithmically [1].

History

Freenet is an enhanced open source implementation of the system described by Ian Clarke's July 1999 paper "A distributed decentralized information storage and retrieval system", written while Clarke was a student at the University of Edinburgh. Shortly after the publication of this paper, Clarke and a small number of volunteers began work on what became Freenet. By March 2000 version 0.1 of Freenet was ready for release. Since March 2000 Freenet has been extensively reported on in the press, albeit primarily due to its implications for copyright rather than for its wider aim of freedom of communication.

Freenet has been developed via a collaborative, open source, methodology. Clarke, originally from Ireland, worked with individuals from many other countries, in a distributed manner utilizing the Internet. Not only is Freenet itself distributed, decentralized Internet software, but the method used to develop it is also distributed and decentralized.

Reports of Freenet's use in authoritarian nations is difficult to track due to the very nature of Freenet's goals. One group, Freenet-China, has translated the Freenet software to Chinese and is distributing it within China on CD and floppy disk.

According to Citeseer, Ian Clarke's "Freenet: A Distributed Anonymous Information Storage and Retrieval System" was the most cited computer science paper of 2000.

An analysis^[2] of Freenet files conducted in the year 2000 claims that the top 3 types of files contained in Freenet were text (37%), audio (21%), and images (14%). 59% of all the text files were drug-related, 71% of all audio files were rock music, and 89% of all images were pornographic. It is important to note the fundamental design of Freenet makes accurate analysis of its content difficult. This analysis was done several years ago from within the United States, and the network has been vastly changed and expanded since it was published.

The program is currently undergoing a massive re-write, which changes the fundamental way the routing takes place, deviating massively from Ian's original paper. The project is attempting to model a series of linked Darknets, to emulate the larger small world effect. This is an attempt to eliminate the scaling problems which have plagued the project, and to improve anonymity.

There were discussions to move the project to the i2p network, but these did not pan out, due to immaturity of the i2p code, and doubts about their security model.

The current development version (0.7) is considered a complete re-engineering, and a complete re-write.

Controversy

The same technology which allows the oppressed to communicate with a large group, without either the publisher or the readers' identities being revealed, can also allow controversial information such as child pornography to be made available to anyone. Freenet's founders believe that only with true anonymity comes true freedom of speech, and that what they view as the beneficial uses of Freenet outweigh its negative uses. However, due to the nature of Freenet a typical user may unknowingly host this sort of information, which may potentially make them subject to severe civil and criminal penalties. Freenet attempts to prevent this through "plausible deniability", preventing the user himself from knowing what's on his own node and making it difficult to determine if a piece of information is in any given node without causing the distribution of that piece of information throughout the network to change in the process. No court cases have tested any of this to date.

Some anonymous friend-to-friend (F2F) networks do allow you to control what kind of files your friends exchange with your node in order to stop them from exchanging files you disapprove of. But Freenet's "deniability" defence wouldn't apply to users of these systems.

However F2F prevents random people from proving that your IP address can effectively be used to get some controversial files. Freenet does not offer this protection because for efficiency reasons (path shortening) some random nodes are allowed to connect directly to your node, thus exchanging files faster, but thus knowing your IP and thus being able to prove that specific files can be gotten from your computer. However, due to Freenet's "plausible deniability" and the way in which Freenet redistributes files among nodes, one cannot prove that those files were placed there by the node owner or that the node owner knows what they are.

Ways to view a freesite

A freesite (which is a commonly accepted name for a site on Freenet) can be set up in one of the following ways:

1. Installing one's own node. It must be noted that since Freenet is written in java one must have Java Runtime Environment installed prior to the installation. Also Freenet must use a portion of the hard drive for datastore to function propperly.
2. Viewing throuth the open remote fproxy. Fproxy can be set up in such a way as to allow the viewing of freesites from the different computer. The downfall of viewing of freesites this way that it compromises anonymity.

Setting up an open fproxy

First it must be noted that setting up an open fproxy may be against the rules of the ISP (when it is disallowed to run any servers or file sharing applications) or even illegal in some countries (where transmission of banned material is illegal whether or not the party performing the transmission is aware of the content).

To set up an open fproxy so that sertain internet users are allowed to view the freesites through your node one must create a web tunnel that reads from the local fproxy. The port to be read is shown in freenet.conf

# The port to listen for local FCP (Freenet Client Protocol) connections on. XXXX represents four digit port number.
clientPort=XXXX

If the web server running the tunnel is set up on the machine other than the one that is running a freenode, the refference to that machine must be added in the following place in freenet.conf

# A comma-separated list of hosts that may connect to the FCP port
# (clientPort).  If left blank, only the localhost will be allowed. If you set this, make sure
 localhost is included in the list or access won't be allowed from the local machine.
# May be given as IP addresses or host names.
fcpHosts=127.0.0.1,localhost,domainname.org

Here is the piece of code in Java that will read the content of the freesite and output it to the person requesting it (note this is not the whole program):

// assuming that the node runs on the same machine and listening to port 1234
// this will not parse the links to be linkable from your tunnel, nor will it display images propperly
// written by VolodyA! V Anarhist, public domain (since there ain't nothing interesting here)
String sskString = request.getParameter("ssk");

if(sskString==null)// || urlString.length()==0)
{
   out.println("<html><body>");
   out.println("<form><input name=\"url\" type=\"text\" value=\"\"><input type=\"submit\"></form>");
   out.println("</body></html>");

   out.close();
   return;
}

sskString = java.net.URLDecoder.decode(sskString, "UTF-8");

URL url = new URL("http://127.0.0.1:1234/" + sskString);

String          line = null;
URLConnection   conn = null;
BufferedReader  webContent = null;

try {
   conn = url.openConnection();
   conn.connect();
   while ((line = webContent.readLine() ) != null) // while there is content coming in
   {
      out.println(line);
   }
   webContent.close();
}
catch (Exception ex) {
   ex.printStackTrace();
}
finally {
   try {
      webContent.close();
   }
   catch (Exception ex) {
      ex.printStackTrace();
   }
}
out.close();

See also: Freenet help

Related tools

Freesite Insertion Wizard (FIW)

FIW is one of the most popular of all the Freenet tools. It is currently the best maintained insertion tool for freesites. It is written in Java and provides both a terminal and GUI interface. It supports the latest Freenet technologies such as containers as well.

Fishtools

Another extremely popular tool for inserting freesites. However, due to an argument, its main developer has broken off with the Freenet community, and it is no longer maintained. It is written in Python and it is still widely used and fairly up to date.

Freenet Utility for Queued Inserts and Downloads (FUQID)

This is a Windows-only tool, typically used to retrieve large splitfiles and to insert non-Freesite content such as binaries, audio, and archives. It is written in Delphi, and it is maintained sporadically. It is generally regarded as an excellent complement to either FIW or Fishtools.

FCPTools

The FCPTools are command-line driven programs for inserting and retrieving files with Freenet. Included separately is FCPLib, the Freenet Client Protocol Library. The FCPTools are linked against FCPLib and serve as nice examples for using the library in Freenet client programs.

FCPLib (Freenet Client Protocol Library) aims to be a platform independent but natively compiled set of C-based functions for storing and retrieving information to and from Freenet. There are routines for storing documents to Freenet from the local disk, and other routines for moving data in memory to and from Freenet.

Everything is released under the GNU General Public License.

FCPLib is now routinely compiled on the following platforms: Microsoft Windows (NT/2K/XP), Debian GNU/Linux, BSD, Solaris, and Mac OS X.

Freenet Tools

The Freenet Tools perform roughly the same tasks as FCPTools, however it does not include a client library for use in other projects. It is written in ANSI C, and only runs under *nix currently.

Frost

Main article: Frost

Frost is a well-maintained and popular message board system for Freenet. It uses a web of trust concept as well as public and private keys to prevent excess spam. It is very popular in the area of file sharing and is the most widely used Freenet messaging application. For a long time many people preferred Freenet Message Board (FMB) but it has fallen into disuse. It is written in Java and therefore works on any platform where a JVM is available, such as Windows, Mac OS X, Linux and other operating systems.

Freenet Message Board (FMB)

The Freenet Message Board is the most branched message board software for Freenet ever coded. It is written in Java, and the original author does not maintain it. However, the community continues to provide several branches which are generally up to date. It works on both *nix and Windows.

See also

• ANts_P2P
• Crypto-anarchism
• Cypherpunk
• Anonymous P2P
• Friend-to-friend networks
• Entropy

External links

• http://freenetproject.org/
• http://www.freenethelp.org/
• http://freenet-china.org/
• wiki devoted to freenet, suitable for technical discussions/content
• The blog of Ian Clarke, the founder of Freenet
• An analysis of the types of files contained in Freenet
• An online debate between Ian Clarke, Freenet's creator, and Matt Oppenheim, the RIAA's senior vice president of business and legal affairs