********************
Repository Structure
********************

.. _Code-overview:

``src`` Overview
----------------
.. to ../repo.rst

This section gives a brief overview of the GNUnet source code.
Specifically, we sketch the function of each of the subdirectories in
the ``gnunet/src/`` directory. The order given is roughly bottom-up (in
terms of the layers of the system).

.. todo:: Reorganise this list

``util/`` --- libgnunetutil
   Library with general utility functions, all GNUnet binaries link
   against this library. Anything from memory allocation and data
   structures to cryptography and inter-process communication. The goal
   is to provide an OS-independent interface and more 'secure' or
   convenient implementations of commonly used primitives. The API is
   spread over more than a dozen headers, developers should study those
   closely to avoid duplicating existing functions. see
   `libgnunetutil <#libgnunetutil>`__.

``hello/`` --- libgnunethello
   HELLO messages are used to describe under which addresses a peer can
   be reached (for example, protocol, IP, port). This library manages
   parsing and generating of HELLO messages.

``block/`` --- libgnunetblock
   The DHT and other components of GNUnet store information in units
   called 'blocks'. Each block has a type and the type defines a
   particular format and how that binary format is to be linked to a
   hash code (the key for the DHT and for databases). The block library
   is a wrapper around block plugins which provide the necessary
   functions for each block type.

``statistics/`` --- statistics service
   The statistics service enables associating values (of type uint64_t)
   with a component name and a string. The main uses is debugging
   (counting events), performance tracking and user entertainment (what
   did my peer do today?).

``arm/`` --- Automatic Restart Manager (ARM)
   The automatic-restart-manager (ARM) service is the GNUnet master
   service. Its role is to start gnunet-services, to re-start them when
   they crashed and finally to shut down the system when requested.

``peerinfo/`` --- peerinfo service
   The peerinfo service keeps track of which peers are known to the
   local peer and also tracks the validated addresses for each peer (in
   the form of a HELLO message) for each of those peers. The peer is not
   necessarily connected to all peers known to the peerinfo service.
   Peerinfo provides persistent storage for peer identities --- peers
   are not forgotten just because of a system restart.

``datacache/`` --- libgnunetdatacache
   The datacache library provides (temporary) block storage for the DHT.
   Existing plugins can store blocks in Sqlite, Postgres or MySQL
   databases. All data stored in the cache is lost when the peer is
   stopped or restarted (datacache uses temporary tables).

``datastore/`` --- datastore service
   The datastore service stores file-sharing blocks in databases for
   extended periods of time. In contrast to the datacache, data is not
   lost when peers restart. However, quota restrictions may still cause
   old, expired or low-priority data to be eventually discarded.
   Existing plugins can store blocks in Sqlite, Postgres or MySQL
   databases.

``template/`` --- service template
   Template for writing a new service. Does nothing.

``ats/`` --- Automatic Transport Selection
   The automatic transport selection (ATS) service is responsible for
   deciding which address (i.e. which transport plugin) should be used
   for communication with other peers, and at what bandwidth.

``nat/`` --- libgnunetnat
   Library that provides basic functions for NAT traversal. The library
   supports NAT traversal with manual hole-punching by the user, UPnP
   and ICMP-based autonomous NAT traversal. The library also includes an
   API for testing if the current configuration works and the
   ``gnunet-nat-server`` which provides an external service to test the
   local configuration.

``fragmentation/`` --- libgnunetfragmentation
   Some transports (UDP and WLAN, mostly) have restrictions on the
   maximum transfer unit (MTU) for packets. The fragmentation library
   can be used to break larger packets into chunks of at most 1k and
   transmit the resulting fragments reliably (with acknowledgment,
   retransmission, timeouts, etc.).

``transport/`` --- transport service
   The transport service is responsible for managing the basic P2P
   communication. It uses plugins to support P2P communication over TCP,
   UDP, HTTP, HTTPS and other protocols. The transport service validates
   peer addresses, enforces bandwidth restrictions, limits the total
   number of connections and enforces connectivity restrictions (e.g.
   friends-only).

``peerinfo-tool/`` --- gnunet-peerinfo
   This directory contains the gnunet-peerinfo binary which can be used
   to inspect the peers and HELLOs known to the peerinfo service.

``core/``
   The core service is responsible for establishing encrypted,
   authenticated connections with other peers, encrypting and decrypting
   messages and forwarding messages to higher-level services that are
   interested in them.

``testing/`` --- libgnunettesting
   The testing library allows starting (and stopping) peers for writing
   testcases. It also supports automatic generation of configurations
   for peers ensuring that the ports and paths are disjoint.
   libgnunettesting is also the foundation for the testbed service

``testbed/`` --- testbed service
   The testbed service is used for creating small or large scale
   deployments of GNUnet peers for evaluation of protocols. It
   facilitates peer deployments on multiple hosts (for example, in a
   cluster) and establishing various network topologies (both underlay
   and overlay).

``nse/`` --- Network Size Estimation
   The network size estimation (NSE) service implements a protocol for
   (securely) estimating the current size of the P2P network.

``dht/`` --- distributed hash table
   The distributed hash table (DHT) service provides a distributed
   implementation of a hash table to store blocks under hash keys in the
   P2P network.

``hostlist/`` --- hostlist service
   The hostlist service allows learning about other peers in the network
   by downloading HELLO messages from an HTTP server, can be configured
   to run such an HTTP server and also implements a P2P protocol to
   advertise and automatically learn about other peers that offer a
   public hostlist server.

``topology/`` --- topology service
   The topology service is responsible for maintaining the mesh
   topology. It tries to maintain connections to friends (depending on
   the configuration) and also tries to ensure that the peer has a
   decent number of active connections at all times. If necessary, new
   connections are added. All peers should run the topology service,
   otherwise they may end up not being connected to any other peer
   (unless some other service ensures that core establishes the required
   connections). The topology service also tells the transport service
   which connections are permitted (for friend-to-friend networking)

``fs/`` --- file-sharing
   The file-sharing (FS) service implements GNUnet's file-sharing
   application. Both anonymous file-sharing (using gap) and
   non-anonymous file-sharing (using dht) are supported.

``cadet/`` --- cadet service
   The CADET service provides a general-purpose routing abstraction to
   create end-to-end encrypted tunnels in mesh networks. We wrote a
   paper documenting key aspects of the design.

``tun/`` --- libgnunettun
   Library for building IPv4, IPv6 packets and creating checksums for
   UDP, TCP and ICMP packets. The header defines C structs for common
   Internet packet formats and in particular structs for interacting
   with TUN (virtual network) interfaces.

``mysql/`` --- libgnunetmysql
   Library for creating and executing prepared MySQL statements and to
   manage the connection to the MySQL database. Essentially a
   lightweight wrapper for the interaction between GNUnet components and
   libmysqlclient.

``dns/``
   Service that allows intercepting and modifying DNS requests of the
   local machine. Currently used for IPv4-IPv6 protocol translation
   (DNS-ALG) as implemented by \"pt/\" and for the GNUnet naming system.
   The service can also be configured to offer an exit service for DNS
   traffic.

``vpn/`` --- VPN service
   The virtual public network (VPN) service provides a virtual tunnel
   interface (VTUN) for IP routing over GNUnet. Needs some other peers
   to run an \"exit\" service to work. Can be activated using the
   \"gnunet-vpn\" tool or integrated with DNS using the \"pt\" daemon.

``exit/``
   Daemon to allow traffic from the VPN to exit this peer to the
   Internet or to specific IP-based services of the local peer.
   Currently, an exit service can only be restricted to IPv4 or IPv6,
   not to specific ports and or IP address ranges. If this is not
   acceptable, additional firewall rules must be added manually. exit
   currently only works for normal UDP, TCP and ICMP traffic; DNS
   queries need to leave the system via a DNS service.

``pt/``
   protocol translation daemon. This daemon enables 4-to-6, 6-to-4,
   4-over-6 or 6-over-4 transitions for the local system. It essentially
   uses \"DNS\" to intercept DNS replies and then maps results to those
   offered by the VPN, which then sends them using mesh to some daemon
   offering an appropriate exit service.

``identity/``
   Management of egos (alter egos) of a user; identities are essentially
   named ECC private keys and used for zones in the GNU name system and
   for namespaces in file-sharing, but might find other uses later

``revocation/``
   Key revocation service, can be used to revoke the private key of an
   identity if it has been compromised

``namecache/``
   Cache for resolution results for the GNU name system; data is
   encrypted and can be shared among users, loss of the data should
   ideally only result in a performance degradation (persistence not
   required)

``namestore/``
   Database for the GNU name system with per-user private information,
   persistence required

``gns/``
   GNU name system, a GNU approach to DNS and PKI.

``dv/``
   A plugin for distance-vector (DV)-based routing. DV consists of a
   service and a transport plugin to provide peers with the illusion of
   a direct P2P connection for connections that use multiple (typically
   up to 3) hops in the actual underlay network.

``regex/``
   Service for the (distributed) evaluation of regular expressions.

``scalarproduct/``
   The scalar product service offers an API to perform a secure
   multiparty computation which calculates a scalar product between two
   peers without exposing the private input vectors of the peers to each
   other.

``consensus/``
   The consensus service will allow a set of peers to agree on a set of
   values via a distributed set union computation.

``reclaim/``
   A decentralized personal data sharing service used to realize a
   decentralized identity provider. Supports OpenID Connect. See also
   https://reclaim.gnunet.org.

``rest/``
   The rest API allows access to GNUnet services using RESTful
   interaction. The services provide plugins that can exposed by the
   rest server.

.. FIXME: Where did this disappear to?
.. ``experimentation/``
      The experimentation daemon coordinates distributed experimentation 
      to evaluate transport and ATS properties.
