Original release 20160919

[fex.git] / doc / concept

diff --git a/doc/concept b/doc/concept
index ead53878c1dff9d91c1033cc703f8a790c21925e..6203c4321cf8d104ceec62cd7aee4879af6edb10 100644 (file)
--- a/doc/concept
+++ b/doc/concept
@@ -17,7 +17,7 @@ Main features of F*EX
   * full-users can create subusers, who can send only to this full-user
   * full-users can create groups, an analogy to mailing lists, but for files
   * self registration possible for internal and external users
   * full-users can create subusers, who can send only to this full-user
   * full-users can create groups, an analogy to mailing lists, but for files
   * self registration possible for internal and external users

-  * maintenance-free: no admin interaction necessary 
+  * maintenance-free: no admin interaction necessary

   * sending to multiple recipients needs storage on the server only once
   * quotas for sending and receiving
   * F*EX is a HTTP web-service and needs no firewall-tunnels
   * sending to multiple recipients needs storage on the server only once
   * quotas for sending and receiving
   * F*EX is a HTTP web-service and needs no firewall-tunnels


@@ -40,7 +40,7 @@ Main features of F*EX
 
 The end user normally uses F*EX with his web browser and the URLs
 http://YOURFEXSERVER/fup (file upload) and http://YOURFEXSERVER/fop (file
 
 The end user normally uses F*EX with his web browser and the URLs
 http://YOURFEXSERVER/fup (file upload) and http://YOURFEXSERVER/fop (file

-download). 
+download).

 
 
 F*EX is not an anonymous service (exception: "public upload" and
 
 
 F*EX is not an anonymous service (exception: "public upload" and


@@ -51,8 +51,8 @@ program "fac" (F*EX admin control) or http://YOURFEXSERVER/fac
 
 Alternativly the users can register theirselves with http://YOURFEXSERVER/fur
 (F*EX user registration), if the admin allows them to do so. This is done by
 
 Alternativly the users can register theirselves with http://YOURFEXSERVER/fur
 (F*EX user registration), if the admin allows them to do so. This is done by

-setting the variables @local_domains and @local_hosts in FEXHOME/lib/fex.ph
-Example: 
+setting the variables @local_domains and @local_hosts in $FEXHOME/lib/fex.ph
+Example:

 
        @local_hosts = qw(127.0.0.1 10.10.100.0-10.10.255.255);
        @local_domains = qw(rus.uni-stuttgart.de flupp.org);
 
        @local_hosts = qw(127.0.0.1 10.10.100.0-10.10.255.255);
        @local_domains = qw(rus.uni-stuttgart.de flupp.org);


@@ -75,12 +75,12 @@ the local recipient domains. For example:
 
 
 F*EX full users can create one time upload URLs with
 
 
 F*EX full users can create one time upload URLs with

-http://YOURFEXSERVER/fuc 
+http://YOURFEXSERVER/fuc

 
 With such a URL a foreign user can send this F*EX full user a single file.
 
 F*EX full users can theirselves register "subusers" with
 
 With such a URL a foreign user can send this F*EX full user a single file.
 
 F*EX full users can theirselves register "subusers" with

-http://YOURFEXSERVER/fuc 
+http://YOURFEXSERVER/fuc

 
 Subusers can only fex to their full-user, not to any others, and they
 cannot create other subusers.
 
 Subusers can only fex to their full-user, not to any others, and they
 cannot create other subusers.


@@ -95,7 +95,7 @@ he cannot change anymore his default settings (KEEP, AUTODELETE, etc)
 
 
 The F*EX user is the same as the "sender" in the fup CGI and the "from"
 
 
 The F*EX user is the same as the "sender" in the fup CGI and the "from"

-parameter in the F*EX URL. 
+parameter in the F*EX URL.

 
 The (confusing) naming scheme is historically based :-)
 
 
 The (confusing) naming scheme is historically based :-)
 


@@ -106,7 +106,7 @@ email.
 
 
 You do not need to build F*EX URLs manually, they are generated by the
 
 
 You do not need to build F*EX URLs manually, they are generated by the

-F*EX programs. 
+F*EX programs.

 
 
 A F*EX session is based on unique URL parameters or POST variables (FROM,
 
 
 A F*EX session is based on unique URL parameters or POST variables (FROM,


@@ -121,19 +121,18 @@ A SKEY is made of md5_hex("$mainuser:$subuser:$subuserid")
 A GKEY is made of md5_hex("$mainuser:$groupname:$groupuser:$groupuserid")
 
 Note: the AKEY, SKEY and GKEY always can be stolen by a network sniffer!
 A GKEY is made of md5_hex("$mainuser:$groupname:$groupuser:$groupuserid")
 
 Note: the AKEY, SKEY and GKEY always can be stolen by a network sniffer!

-If you need true security, then you have to use https (SSL) instead of
-http! 
+If you need true security, then you have to use https instead of http!

 
 After download the file will be deleted after a grace time of 1 minute.
 This grace time allows a recipient to get the file again if he had
 
 After download the file will be deleted after a grace time of 1 minute.
 This grace time allows a recipient to get the file again if he had

-problems in saving it. 
+problems in saving it.

 
 With the fexsend client the sender can change this behavior:
 
 option -D means "delay autodelete": do not delete the the file directly
 after download, but with the nightly fex_cleanup cronjob. More downloads
 are possible only from the same client (identified by cookie or ip
 
 With the fexsend client the sender can change this behavior:
 
 option -D means "delay autodelete": do not delete the the file directly
 after download, but with the nightly fex_cleanup cronjob. More downloads
 are possible only from the same client (identified by cookie or ip

-address). 
+address).

 
 option -K means "keep file": do not delete the file after download, but
 only after expiration date (normally 5 days). More downloads are possible
 
 option -K means "keep file": do not delete the file after download, but
 only after expiration date (normally 5 days). More downloads are possible


@@ -147,7 +146,7 @@ and more than once (until expiration date).
 If you want "delay autodelete" to be the default behaviour for all users
 and each transfer then set
        $autodelete = 'DELAY'; # or 'NO' for no autodelete
 If you want "delay autodelete" to be the default behaviour for all users
 and each transfer then set
        $autodelete = 'DELAY'; # or 'NO' for no autodelete

-in FEXHOME/lib/fex.ph
+in $FEXHOME/lib/fex.ph

 
 In addition, you can add to the "Recipient(s)" field of the fup CGI:
 ":autodelete=delay" or ":autodelete=no" or ":keep=x" (where x is the number
 
 In addition, you can add to the "Recipient(s)" field of the fup CGI:
 ":autodelete=delay" or ":autodelete=no" or ":keep=x" (where x is the number


@@ -165,7 +164,7 @@ requires (HTTP) authorization. The credentials are the F*EX user email
 and auth-ID.
 See extra documentation SSL for using https.
 
 and auth-ID.
 See extra documentation SSL for using https.
 

-If you want to have encrypted emails, then you need a GPG public key for 
+If you want to have encrypted emails, then you need a GPG public key for

 the user fex. Create it with "gpg --gen-key" (use fex@YOURFEXSERVER as the
 user or the $bcc address from fex.ph if you have set it, see below). Next,
 the user has to upload his public key via webinterface "user config &
 the user fex. Create it with "gpg --gen-key" (use fex@YOURFEXSERVER as the
 user or the $bcc address from fex.ph if you have set it, see below). Next,
 the user has to upload his public key via webinterface "user config &


@@ -194,27 +193,27 @@ The administrator can also forbid a user to fex to any recipient address,
 but the allowed ones with: fac -r USER
 
 
 but the allowed ones with: fac -r USER
 
 

-By standard installation the base directory FEXHOME is the same as the
-login HOME of user fex, but you can move it if you want. FEXHOME is
+By standard installation the base directory $FEXHOME is the same as the
+login HOME of user fex, but you can move it if you want. $FEXHOME is

 determined by the full path of fexsrv as configured in
 determined by the full path of fexsrv as configured in

-/etc/xinetd.d/fex . Change this when you move FEXHOME!
+/etc/xinetd.d/fex . Change this when you move $FEXHOME!

 
 You can also add (name based) virtual hosts with fac.
 
 
 You can also add (name based) virtual hosts with fac.
 

-Do not give write permission to any other user to any file in FEXHOME or
+Do not give write permission to any other user to any file in $FEXHOME or

 below!
 
 
 below!
 
 

-FEXHOME contains:
+$FEXHOME contains:

 
 

-       spool/                          spool directory and user data 
+       spool/                          spool directory and user data

         htdocs/                                directory for generic download files
         bin/                           directory for programs
         cgi-bin/                       directory for CGI programs
        lib/                            directory for library and config files
         doc/                           additional documentation
         locale/                                alternative language installation
         htdocs/                                directory for generic download files
         bin/                           directory for programs
         cgi-bin/                       directory for CGI programs
        lib/                            directory for library and config files
         doc/                           additional documentation
         locale/                                alternative language installation

-       
+

 
 Files in spool:
 
 
 Files in spool:
 


@@ -241,12 +240,13 @@ Files in spool:
        $user/@KEEP                     keep default
        $user/@LOCALE                   locale default
        $user/@CAPTIVE                  user must not change his settings
        $user/@KEEP                     keep default
        $user/@LOCALE                   locale default
        $user/@CAPTIVE                  user must not change his settings

-       $user/@FEXYOURSELF              user can only fex to himself via 
+       $user/@FEXYOURSELF              user can only fex to himself via

                                         web interface
        $to/$from/$file/upload          file data in upload progress
        $to/$from/$file/filename        original file name
        $to/$from/$file/size            original file size
        $to/$from/$file/useragent       HTTP header User-Agent
                                         web interface
        $to/$from/$file/upload          file data in upload progress
        $to/$from/$file/filename        original file name
        $to/$from/$file/size            original file size
        $to/$from/$file/useragent       HTTP header User-Agent

+       $to/$from/$file/uurl            upload URL

        $to/$from/$file/data            file data after complete upload
        $to/$from/$file/keep            keep time (autoexpire) in days
        $to/$from/$file/autodelete      autodelete option: YES NO or DELAY
        $to/$from/$file/data            file data after complete upload
        $to/$from/$file/keep            keep time (autoexpire) in days
        $to/$from/$file/autodelete      autodelete option: YES NO or DELAY


@@ -273,7 +273,7 @@ A registered full F*EX user is identified by the file $spooldir/$from/@
 Only if this file contains his auth-ID this user is able to send files to
 others. Otherwise he is just an unpriviledged recipient.
 
 Only if this file contains his auth-ID this user is able to send files to
 others. Otherwise he is just an unpriviledged recipient.
 

-You can customize the upload CGI fup by editing FEXHOME/lib/fup.pl
+You can customize the upload CGI fup by editing $FEXHOME/lib/fup.pl

 
 Additional directories in spool:
 
 
 Additional directories in spool:
 


@@ -294,7 +294,7 @@ This is done via UKEY, a unique upload identifier.
 The download key (DKEY) is a unique identifier for - guess what -
 downloading. It also prevents an attacker to get the file, because only
 the recipient knows the DKEY as part of the download URL from the
 The download key (DKEY) is a unique identifier for - guess what -
 downloading. It also prevents an attacker to get the file, because only
 the recipient knows the DKEY as part of the download URL from the

-notification email. 
+notification email.

 
 XKEY is an optional extra download key to have a short download URL in
 shape http://YOURFEXSERVER//XKEY
 
 XKEY is an optional extra download key to have a short download URL in
 shape http://YOURFEXSERVER//XKEY


@@ -302,7 +302,7 @@ The XKEY always starts with // and is provided as a COMMENT by the client.
 Example: fexsend schwuppdiwupp.jpg //
 The user has to realize that such URLs have very low security.
 
 Example: fexsend schwuppdiwupp.jpg //
 The user has to realize that such URLs have very low security.
 

-If you need to trace a F*EX request, then set 
+If you need to trace a F*EX request, then set

        $debug = 1;
 in fex.ph and look in ~/spool/.debug/ for the correspondening files.
 
        $debug = 1;
 in fex.ph and look in ~/spool/.debug/ for the correspondening files.
 


@@ -323,7 +323,7 @@ In this case "From: $admin" will be used in the notification emails.
 
 F*EX comes with its own web server: fexsrv
 Standard web servers like apache have been proven problematic, either in
 
 F*EX comes with its own web server: fexsrv
 Standard web servers like apache have been proven problematic, either in

-speed or because of a 2-GB-limit. 
+speed or because of a 2-GB-limit.

 
 It is not possible to use the F*EX CGIs with an alternative web server,
 because the F*EX CGIs need special fexsrv features.
 
 It is not possible to use the F*EX CGIs with an alternative web server,
 because the F*EX CGIs need special fexsrv features.


@@ -331,7 +331,7 @@ because the F*EX CGIs need special fexsrv features.
 xinetd starts fexsrv for every new connection, which then executes the
 CGIs fup (file upload), fop (file output), fuc (fex user control), foc
 (fex operation control), fac (fex admin control), fur (fex user
 xinetd starts fexsrv for every new connection, which then executes the
 CGIs fup (file upload), fop (file output), fuc (fex user control), foc
 (fex operation control), fac (fex admin control), fur (fex user

-registration) and sex (stream exchange). 
+registration) and sex (stream exchange).

 
 
 SEX has the opposite authorization model of FEX: The sender does not
 
 
 SEX has the opposite authorization model of FEX: The sender does not


@@ -342,7 +342,7 @@ otherwise. The client programs for sexing are sexsend and sexget. You can
 call them also fuckyou and fuckme :-)
 
 
 call them also fuckyou and fuckme :-)
 
 

-With the F*EX client fexsend you can have a streaming file transfer with 
+With the F*EX client fexsend you can have a streaming file transfer with

 spooling: on client side there is no temporary buffer file (archive), but
 the data is sent directly to the F*EX server. This saves time and disk
 space on the client.
 spooling: on client side there is no temporary buffer file (archive), but
 the data is sent directly to the F*EX server. This saves time and disk
 space on the client.


@@ -362,13 +362,13 @@ For streaming receiving you can use "fexget -s-" or "wget -O-".
 
 
 fexsrv also can do generic document output (via dop) like a normal web
 
 
 fexsrv also can do generic document output (via dop) like a normal web

-server. For this, your files must be under FEXHOME/htdocs and they must
-not have the same name as the CGIs under FEXHOME/cgi-bin, because the CGIs
+server. For this, your files must be under $FEXHOME/htdocs and they must
+not have the same name as the CGIs under $FEXHOME/cgi-bin, because the CGIs

 have priority.
 
 For security reasons, documents to be delivered by dop:
 - the file must be readable by group or world
 have priority.
 
 For security reasons, documents to be delivered by dop:
 - the file must be readable by group or world

-- the file must be in FEXHOME/htdocs or a directory specified by @doc_dirs
+- the file must be in $FEXHOME/htdocs or a directory specified by @doc_dirs

 - the filename must not start with a "."
 - the filename must not contain a "@"
 - the filename must not end with "~"
 - the filename must not start with a "."
 - the filename must not contain a "@"
 - the filename must not end with "~"


@@ -383,17 +383,27 @@ directory. You can name single IPs, also as IP ranges (example:
 129.69.0.0-129.69.255.255)
 
 dop is not a regular CGI program (like fup or fop), but a sub-program of
 129.69.0.0-129.69.255.255)
 
 dop is not a regular CGI program (like fup or fop), but a sub-program of

-fexsrv. 
+fexsrv.

 
 *.html files may contain $VARIABLES$ which will be substituted with the
 value of the corresponding environment variable. See example
 
 *.html files may contain $VARIABLES$ which will be substituted with the
 value of the corresponding environment variable. See example

-$SERVER_ADMIN$ in FEXHOME/htdocs/index.html
+$SERVER_ADMIN$ in $FEXHOME/htdocs/index.html

 
 *.html files may contain <<perl-code>> (even multiline) which will be
 
 *.html files may contain <<perl-code>> (even multiline) which will be

-evaluated. See example FEXHOME/htdocs/dynamic.html
-This perl-code must not contain '>>' strings itself!
+evaluated. The output from print and printf statements will be placed
+in. If the perl-code does not end with a ";" then its return value is also
+added to the output.
+
+Same goes for <<<perl-code>>> but without output catching.
+
+This perl-code must not contain ">>" strings itself!
+
+See example $FEXHOME/htdocs/dynamic.html
+
+To chainload external perl-code do not use "require" but "do" statement.

 Pay attention: do not place security relevant data inside << >> because it
 Pay attention: do not place security relevant data inside << >> because it

-will be delivered to the client if the URL ends with '!'! See example:
+will be delivered to the client if the URL ends with a "!" character, see
+example:

 http://fex.rus.uni-stuttgart.de/index.html
 http://fex.rus.uni-stuttgart.de/index.html!
 
 http://fex.rus.uni-stuttgart.de/index.html
 http://fex.rus.uni-stuttgart.de/index.html!
 


@@ -462,7 +472,7 @@ Or to delete inactive users you can put in fex.ph:
 $account_expire = "365:delete";
 
 This deletes user accounts automatically which have been inactive for 365
 $account_expire = "365:delete";
 
 This deletes user accounts automatically which have been inactive for 365

-days. 
+days.

 
 Of course you can use any number of days.
 
 
 Of course you can use any number of days.
 


@@ -489,7 +499,7 @@ because most proxies do not support persistant tcp sessions.
 See fex-client_2.pdf for the F*EX protocol specification.
 
 To understand and trace the F*EX protocol you can use fexsend with the -v
 See fex-client_2.pdf for the F*EX protocol specification.
 
 To understand and trace the F*EX protocol you can use fexsend with the -v

-option. 
+option.

 
 Example (--> means send to server, <-- means receive from server):
 
 
 Example (--> means send to server, <-- means receive from server):
 


@@ -504,47 +514,47 @@ TCPCONNECT to fex.rus.uni-stuttgart.de
 <-- X-SID: 8p2Y2qa2
 <-- X-Timeout: 30
 <-- Content-Length: 0
 <-- X-SID: 8p2Y2qa2
 <-- X-Timeout: 30
 <-- Content-Length: 0

-<-- 
+<--

 --> HEAD /fop/framstag@flupp.org/framstag@fex.rus.uni-stuttgart.de/X.png??ID=MD5H:226e896d0adab86892957aa8158b37ba HTTP/1.1
 -->
 <-- HTTP/1.1 200 OK
 <-- Content-Length: 0
 <-- X-Size: 0
 <-- X-Features: SID,CHECKRECIPIENT,GROUPS,QUOTA,FILEID,MULTIPOST,XKEY
 --> HEAD /fop/framstag@flupp.org/framstag@fex.rus.uni-stuttgart.de/X.png??ID=MD5H:226e896d0adab86892957aa8158b37ba HTTP/1.1
 -->
 <-- HTTP/1.1 200 OK
 <-- Content-Length: 0
 <-- X-Size: 0
 <-- X-Features: SID,CHECKRECIPIENT,GROUPS,QUOTA,FILEID,MULTIPOST,XKEY

-<-- 
+<--

 --> POST /fup HTTP/1.1
 --> Host: fex.rus.uni-stuttgart.de
 --> User-Agent: fexsend (Ubuntu 8.04.4 LTS)
 --> Content-Length: 149935162
 --> Content-Type: multipart/form-data; boundary=JhUOtQ3sgV5ZcHJzrTny523nBFqgUNvSAOUHoRMTdZfGpAjs
 --> POST /fup HTTP/1.1
 --> Host: fex.rus.uni-stuttgart.de
 --> User-Agent: fexsend (Ubuntu 8.04.4 LTS)
 --> Content-Length: 149935162
 --> Content-Type: multipart/form-data; boundary=JhUOtQ3sgV5ZcHJzrTny523nBFqgUNvSAOUHoRMTdZfGpAjs

---> 
+-->

 --> --JhUOtQ3sgV5ZcHJzrTny523nBFqgUNvSAOUHoRMTdZfGpAjs
 --> Content-Disposition: form-data; name="FROM"
 --> --JhUOtQ3sgV5ZcHJzrTny523nBFqgUNvSAOUHoRMTdZfGpAjs
 --> Content-Disposition: form-data; name="FROM"

---> 
+-->

 --> framstag@fex.rus.uni-stuttgart.de
 --> --JhUOtQ3sgV5ZcHJzrTny523nBFqgUNvSAOUHoRMTdZfGpAjs
 --> Content-Disposition: form-data; name="TO"
 --> framstag@fex.rus.uni-stuttgart.de
 --> --JhUOtQ3sgV5ZcHJzrTny523nBFqgUNvSAOUHoRMTdZfGpAjs
 --> Content-Disposition: form-data; name="TO"

---> 
+-->

 --> framstag@flupp.org
 --> --JhUOtQ3sgV5ZcHJzrTny523nBFqgUNvSAOUHoRMTdZfGpAjs
 --> Content-Disposition: form-data; name="ID"
 --> framstag@flupp.org
 --> --JhUOtQ3sgV5ZcHJzrTny523nBFqgUNvSAOUHoRMTdZfGpAjs
 --> Content-Disposition: form-data; name="ID"

---> 
+-->

 --> MD5H:226e896d0adab86892957aa8158b37ba
 --> --JhUOtQ3sgV5ZcHJzrTny523nBFqgUNvSAOUHoRMTdZfGpAjs
 --> Content-Disposition: form-data; name="FILESIZE"
 --> MD5H:226e896d0adab86892957aa8158b37ba
 --> --JhUOtQ3sgV5ZcHJzrTny523nBFqgUNvSAOUHoRMTdZfGpAjs
 --> Content-Disposition: form-data; name="FILESIZE"

---> 
+-->

 --> 149934400
 --> --JhUOtQ3sgV5ZcHJzrTny523nBFqgUNvSAOUHoRMTdZfGpAjs
 --> Content-Disposition: form-data; name="FILE"; filename="X.png"
 --> Content-Type: application/octet-stream
 --> Content-Length: 149934400
 --> X-File-ID: 1283077463
 --> 149934400
 --> --JhUOtQ3sgV5ZcHJzrTny523nBFqgUNvSAOUHoRMTdZfGpAjs
 --> Content-Disposition: form-data; name="FILE"; filename="X.png"
 --> Content-Type: application/octet-stream
 --> Content-Length: 149934400
 --> X-File-ID: 1283077463

---> 
+-->

 --> (file content)
 --> --JhUOtQ3sgV5ZcHJzrTny523nBFqgUNvSAOUHoRMTdZfGpAjs--
 <-- Location: http://fex.rus.uni-stuttgart.de/fop/CoVhikzk/X.png
 <-- X-Recipient: framstag@flupp.org (autodelete=YES,keep=5)
 --> (file content)
 --> --JhUOtQ3sgV5ZcHJzrTny523nBFqgUNvSAOUHoRMTdZfGpAjs--
 <-- Location: http://fex.rus.uni-stuttgart.de/fop/CoVhikzk/X.png
 <-- X-Recipient: framstag@flupp.org (autodelete=YES,keep=5)

-<-- 
+<--

 
 Comment on the HEAD request above:
 The client fexsend sends it to request whether the file has been sent
 
 Comment on the HEAD request above:
 The client fexsend sends it to request whether the file has been sent


@@ -560,7 +570,7 @@ CHECKRECIPIENT is a HTTP POST request to check whether the recpient's
 email address is valid to the server.
 GET ADDRESS_BOOK is a HTTP GET request to check if the recipient with a
 short address (= address without @) is an alias in the senders server
 email address is valid to the server.
 GET ADDRESS_BOOK is a HTTP GET request to check if the recipient with a
 short address (= address without @) is an alias in the senders server

-address book. 
+address book.

 Please use "fexsend -v" by yourself to see the whole protocol dialoge.
 
 Many HTTP proxies have a POST limit, which is often at 2 GB, but sometimes
 Please use "fexsend -v" by yourself to see the whole protocol dialoge.
 
 Many HTTP proxies have a POST limit, which is often at 2 GB, but sometimes


@@ -690,6 +700,12 @@ Important for programmers: The perl variables of the F*EX CGI's (fup, fop,
 etc) have their UTF-8 flag turned off. This means, they contain UTF-8 data
 in binary representation.
 
 etc) have their UTF-8 flag turned off. This means, they contain UTF-8 data
 in binary representation.
 

+F*EX has support for many languanges. For every language a translated
+version will be installed in $HOME/locale/
+The switch is done at runtime via locale cookie (by fexsrv).
+Some common functions from fex.pp must be present at any time for any
+locale in any language, eg notification functions. Therfore there is
+$HOME/lib/lf.pl (extracted functions from fex.pp).

 
 I have additional authentication modules for RADIUS, LDAP, mailman and
 POP. Please email me if you are interested.
 
 I have additional authentication modules for RADIUS, LDAP, mailman and
 POP. Please email me if you are interested.