@c Maintenance notes:
@c 1. Pay attention to @FIXME{}s and @UNREVISED{}s
@c 2. Before creating final variant:
-@c 1.1. Run `make check-options' to make sure all options are properly
+@c 2.1. Run `make check-options' to make sure all options are properly
@c documented;
-@c 2.1. Run `make master-menu' (see comment before the master menu).
+@c 2.2. Run `make master-menu' (see comment before the master menu).
@include rendition.texi
@include value.texi
* Changes::
* Configuring Help Summary::
-* Genfile::
* Tar Internals::
+* Genfile::
* Free Software Needs Free Documentation::
* Copying This Manual::
* Index of Command Line Options::
Options Used by @option{--create}
+* override:: Overriding File Metadata.
* Ignore Failed Read::
Options Used by @option{--extract}
* posix:: @acronym{POSIX} archives
* Checksumming:: Checksumming Problems
* Large or Negative Values:: Large files, negative time stamps, etc.
+* Other Tars:: How to Extract GNU-Specific Data Using
+ Other @command{tar} Implementations
@GNUTAR{} and @acronym{POSIX} @command{tar}
* PAX keywords:: Controlling Extended Header Keywords.
+How to Extract GNU-Specific Data Using Other @command{tar} Implementations
+
+* Split Recovery:: Members Split Between Volumes
+* Sparse Recovery:: Sparse Members
+
Using Less Space through Compression
* gzip:: Creating and Reading Compressed Archives
* Tarcat:: Concatenate Volumes into a Single Archive
-Genfile
-
-* Generate Mode:: File Generation Mode.
-* Status Mode:: File Status Mode.
-* Exec Mode:: Synchronous Execution mode.
-
Tar Internals
* Standard:: Basic Tar Format
* Extensions:: @acronym{GNU} Extensions to the Archive Format
+* Sparse Formats:: Storing Sparse Files
* Snapshot Files::
* Dumpdir::
+Storing Sparse Files
+
+* Old GNU Format::
+* PAX 0:: PAX Format, Versions 0.0 and 0.1
+* PAX 1:: PAX Format, Version 1.0
+
+Genfile
+
+* Generate Mode:: File Generation Mode.
+* Status Mode:: File Status Mode.
+* Exec Mode:: Synchronous Execution mode.
+
Copying This Manual
* GNU Free Documentation License:: License for copying this manual
You can use @command{tar} archives in many ways. We want to stress a few
of them: storage, backup, and transportation.
-@FIXME{the following table entries need a bit of work..}
+@FIXME{the following table entries need a bit of work.}
@table @asis
@item Storage
Often, @command{tar} archives are used to store related files for
easiest to understand (and we encourage you to do the same when you use
@command{tar}, to avoid errors).
-Note that the part of the command which says,
+Note that the sequence
@option{--file=@-collection.tar} is considered to be @emph{one} argument.
If you substituted any other string of characters for
@kbd{collection.tar}, then that string would become the name of the
to note these differences, and only use the option style(s) which
makes the most sense to you until you feel comfortable with the others.
-Some options @emph{may} take an argument (currently, there are
-two such options: @option{--backup} and @option{--occurrence}). Such
-options may have at most long and short forms, they do not have old style
-equivalent. The rules for specifying an argument for such options
-are stricter than those for specifying mandatory arguments. Please,
-pay special attention to them.
+Some options @emph{may} take an argument. Such options may have at
+most long and short forms, they do not have old style equivalent. The
+rules for specifying an argument for such options are stricter than
+those for specifying mandatory arguments. Please, pay special
+attention to them.
@menu
* Long Options:: Long Option Style
Files added to the @command{tar} archive will have a group id of @var{group},
rather than the group from the source file. @var{group} is first decoded
as a group symbolic name, but if this interpretation fails, it has to be
-a decimal numeric group ID. @FIXME-xref{}
+a decimal numeric group ID. @xref{override}.
Also see the comments for the @option{--owner=@var{user}} option.
When adding files to an archive, @command{tar} will use
@var{permissions} for the archive members, rather than the permissions
-from the files. The program @command{chmod} and this @command{tar}
-option share the same syntax for what @var{permissions} might be.
-@xref{File permissions, Permissions, File permissions, fileutils,
-@acronym{GNU} file utilities}. This reference also has useful
-information for those not being overly familiar with the Unix
-permission system.
-
-Of course, @var{permissions} might be plainly specified as an octal number.
-However, by using generic symbolic modifications to mode bits, this allows
-more flexibility. For example, the value @samp{a+rw} adds read and write
-permissions for everybody, while retaining executable bits on directories
-or on any other file already marked as executable.
+from the files. @var{permissions} can be specified either as an octal
+number or as symbolic permissions, like with
+@command{chmod}. @xref{override}.
+
+@opsummary{mtime}
+@item --mtime=@var{date}
+
+When adding files to an archive, @command{tar} will use @var{date} as
+the modification time of members when creating archives, instead of
+their actual modification times. The value of @var{date} can be
+either a textual date representation (@pxref{Date input formats}) or a
+name of the existing file, starting with @samp{/} or @samp{.}. In the
+latter case, the modification time of that file is used. @xref{override}.
@opsummary{multi-volume}
@item --multi-volume
Like @option{--newer}, but add only files whose
contents have changed (as opposed to just @option{--newer}, which will
-also back up files for which any status information has changed).
+also back up files for which any status information has
+changed). @xref{after}.
@opsummary{no-anchored}
@item --no-anchored
@xref{Attributes}.
@item -o
-When extracting files, this option is a synonym for
+The function of this option depends on the action @command{tar} is
+performing. When extracting files, @option{-o} is a synonym for
@option{--no-same-owner}, i.e. it prevents @command{tar} from
restoring ownership of files being extracted.
-When creating an archive, @option{-o} is a synonym for
+When creating an archive, it is a synonym for
@option{--old-archive}. This behavior is for compatibility
with previous versions of @GNUTAR{}, and will be
removed in the future releases.
@end smallexample
@noindent
-will extract the first occurrence of @file{filename} from @file{archive.tar}
+will extract the first occurrence of the member @file{filename} from @file{archive.tar}
and will terminate without scanning to the end of the archive.
@opsummary{old-archive}
when creating archives, instead of the user associated with the source
file. @var{user} is first decoded as a user symbolic name, but if
this interpretation fails, it has to be a decimal numeric user ID.
-@FIXME-xref{}
-
-There is no value indicating a missing number, and @samp{0} usually means
-@code{root}. Some people like to force @samp{0} as the value to offer in
-their distributions for the owner of files, because the @code{root} user is
-anonymous anyway, so that might as well be the owner of anonymous archives.
+@xref{override}.
This option does not affect extraction from archives.
Invokes a @acronym{GNU} extension when adding files to an archive that handles
sparse files efficiently. @xref{sparse}.
+@opsummary{sparse-version}
+@item --sparse-version=@var{version}
+
+Specified the @dfn{format version} to use when archiving sparse
+files. Implies @option{--sparse}. @xref{sparse}. For the description
+of the supported sparse formats, @xref{Sparse Formats}.
+
@opsummary{starting-file}
@item --starting-file=@var{name}
@itemx -K @var{name}
@smallexample
tar (GNU tar) @value{VERSION}
Copyright (C) 2006 Free Software Foundation, Inc.
-This is free software. You may redistribute copies of it under the terms of
-the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
+This is free software. You may redistribute copies of it under the terms
+of the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.
Written by John Gilmore and Jay Fenlason.
ready. In the meantime, programs not being localizable for dates
should prefer international dates, that's really the way to go.
-Look up @url{http://www.ft.uni-erlangen.de/@/~mskuhn/@/iso-time.html} if you
+Look up @url{http://www.cl.cam.ac.uk/@/~mgk25/@/iso-time.html} if you
are curious, it contains a detailed explanation of the ISO 8601 standard.
@end table
@samp{collection.tar} and @samp{music.tar}.
We will also use the archive files @samp{afiles.tar} and
-@samp{bfiles.tar}. @samp{afiles.tar} contains the members @samp{apple},
-@samp{angst}, and @samp{aspic}. @samp{bfiles.tar} contains the members
+@samp{bfiles.tar}. The archive @samp{afiles.tar} contains the members @samp{apple},
+@samp{angst}, and @samp{aspic}; @samp{bfiles.tar} contains the members
@samp{./birds}, @samp{baboon}, and @samp{./box}.
Unless we state otherwise, all practicing you do and examples you follow
@option{--create}.
@menu
+* override:: Overriding File Metadata.
* Ignore Failed Read::
@end menu
+@node override
+@subsection Overriding File Metadata
+
+As described above, a @command{tar} archive keeps, for each member it contains,
+its @dfn{metadata}, such as modification time, mode and ownership of
+the file. @GNUTAR{} allows to replace these data with other values
+when adding files to the archive. The options described in this
+section affect creation of archives of any type. For POSIX archives,
+see also @ref{PAX keywords}, for additional ways of controlling
+metadata, stored in the archive.
+
+@table @option
+@opindex mode
+@item --mode=@var{permissions}
+
+When adding files to an archive, @command{tar} will use
+@var{permissions} for the archive members, rather than the permissions
+from the files. @var{permissions} can be specified either as an octal
+number or as symbolic permissions, like with
+@command{chmod} (@xref{File permissions, Permissions, File
+permissions, fileutils, @acronym{GNU} file utilities}. This reference
+also has useful information for those not being overly familiar with
+the UNIX permission system). Using latter syntax allows for
+more flexibility. For example, the value @samp{a+rw} adds read and write
+permissions for everybody, while retaining executable bits on directories
+or on any other file already marked as executable:
+
+@smallexample
+$ @kbd{tar -c -f archive.tar --mode='a+rw' .}
+@end smallexample
+
+@item --mtime=@var{date}
+@opindex mtime
+
+When adding files to an archive, @command{tar} will use @var{date} as
+the modification time of members when creating archives, instead of
+their actual modification times. The argument @var{date} can be
+either a textual date representation in almost arbitrary format
+(@pxref{Date input formats}) or a name of the existing file, starting
+with @samp{/} or @samp{.}. In the latter case, the modification time
+of that file will be used.
+
+The following example will set the modification date to 00:00:00 UTC,
+January 1, 1970:
+
+@smallexample
+$ @kbd{tar -c -f archive.tar --mtime='1970-01-01' .}
+@end smallexample
+
+@noindent
+When used with @option{--verbose} (@pxref{verbose tutorial}) @GNUTAR{}
+will try to convert the specified date back to its textual
+representation and compare it with the one given with
+@option{--mtime} options. If the two dates differ, @command{tar} will
+print a warning saying what date it will use. This is to help user
+ensure he is using the right date.
+
+For example:
+
+@smallexample
+$ @kbd{tar -c -f archive.tar -v --mtime=yesterday .}
+tar: Option --mtime: Treating date `yesterday' as 2006-06-20
+13:06:29.152478
+@dots{}
+@end smallexample
+
+@item --owner=@var{user}
+@opindex owner
+
+Specifies that @command{tar} should use @var{user} as the owner of members
+when creating archives, instead of the user associated with the source
+file. The argument @var{user} can be either an existing user symbolic
+name, or a decimal numeric user ID.
+
+There is no value indicating a missing number, and @samp{0} usually means
+@code{root}. Some people like to force @samp{0} as the value to offer in
+their distributions for the owner of files, because the @code{root} user is
+anonymous anyway, so that might as well be the owner of anonymous
+archives. For example:
+
+@smallexample
+@group
+$ @kbd{tar -c -f archive.tar --owner=0 .}
+# @r{Or:}
+$ @kbd{tar -c -f archive.tar --owner=root .}
+@end group
+@end smallexample
+
+@item --group=@var{group}
+@opindex group
+
+Files added to the @command{tar} archive will have a group id of @var{group},
+rather than the group from the source file. The argument @var{group}
+can be either an existing group symbolic name, or a decimal numeric group ID.
+@end table
+
@node Ignore Failed Read
@subsection Ignore Fail Read
@table @option
@item --ignore-failed-read
+@opindex ignore-failed-read
Do not exit with nonzero on unreadable files or directories.
@end table
$ @kbd{tar -cf foo.tar --newer-mtime '2 days ago'}
@end smallexample
+When any of these options is used with the option @option{--verbose}
+(@pxref{verbose tutorial}) @GNUTAR{} will try to convert the specified
+date back to its textual representation and compare that with the
+one given with the option. If the two dates differ, @command{tar} will
+print a warning saying what date it will use. This is to help user
+ensure he is using the right date. For example:
+
+@smallexample
+@group
+$ @kbd{tar -c -f archive.tar --after-date='10 days ago' .}
+tar: Option --after-date: Treating date `10 days ago' as 2006-06-11
+13:19:37.232434
+@end group
+@end smallexample
+
@quotation
@strong{Please Note:} @option{--after-date} and @option{--newer-mtime}
should not be used for incremental backups. @xref{Incremental Dumps},
switch to @samp{posix}.
@menu
-* Portability:: Making @command{tar} Archives More Portable
* Compression:: Using Less Space through Compression
* Attributes:: Handling File Attributes
+* Portability:: Making @command{tar} Archives More Portable
* cpio:: Comparison of @command{tar} and @command{cpio}
@end menu
-@node Portability
-@section Making @command{tar} Archives More Portable
-
-Creating a @command{tar} archive on a particular system that is meant to be
-useful later on many other machines and with other versions of @command{tar}
-is more challenging than you might think. @command{tar} archive formats
-have been evolving since the first versions of Unix. Many such formats
-are around, and are not always compatible with each other. This section
-discusses a few problems, and gives some advice about making @command{tar}
-archives more portable.
-
-One golden rule is simplicity. For example, limit your @command{tar}
-archives to contain only regular files and directories, avoiding
-other kind of special files. Do not attempt to save sparse files or
-contiguous files as such. Let's discuss a few more problems, in turn.
-
-@FIXME{Discuss GNU extensions (incremental backups, multi-volume
-archives and archive labels) in GNU and PAX formats.}
+@node Compression
+@section Using Less Space through Compression
@menu
-* Portable Names:: Portable Names
-* dereference:: Symbolic Links
-* old:: Old V7 Archives
-* ustar:: Ustar Archives
-* gnu:: GNU and old GNU format archives.
-* posix:: @acronym{POSIX} archives
-* Checksumming:: Checksumming Problems
-* Large or Negative Values:: Large files, negative time stamps, etc.
+* gzip:: Creating and Reading Compressed Archives
+* sparse:: Archiving Sparse Files
@end menu
-@node Portable Names
-@subsection Portable Names
+@node gzip
+@subsection Creating and Reading Compressed Archives
+@cindex Compressed archives
+@cindex Storing archives in compressed format
-Use portable file and member names. A name is portable if it contains
-only ASCII letters and digits, @samp{/}, @samp{.}, @samp{_}, and
-@samp{-}; it cannot be empty, start with @samp{-} or @samp{//}, or
-contain @samp{/-}. Avoid deep directory nesting. For portability to
-old Unix hosts, limit your file name components to 14 characters or
-less.
+@GNUTAR{} is able to create and read compressed archives. It supports
+@command{gzip} and @command{bzip2} compression programs. For backward
+compatibilty, it also supports @command{compress} command, although
+we strongly recommend against using it, since there is a patent
+covering the algorithm it uses and you could be sued for patent
+infringement merely by running @command{compress}! Besides, it is less
+effective than @command{gzip} and @command{bzip2}.
-If you intend to have your @command{tar} archives to be read under
-MSDOS, you should not rely on case distinction for file names, and you
-might use the @acronym{GNU} @command{doschk} program for helping you
-further diagnosing illegal MSDOS names, which are even more limited
-than System V's.
+Creating a compressed archive is simple: you just specify a
+@dfn{compression option} along with the usual archive creation
+commands. The compression option is @option{-z} (@option{--gzip}) to
+create a @command{gzip} compressed archive, @option{-j}
+(@option{--bzip2}) to create a @command{bzip2} compressed archive, and
+@option{-Z} (@option{--compress}) to use @command{compress} program.
+For example:
-@node dereference
-@subsection Symbolic Links
-@cindex File names, using symbolic links
-@cindex Symbolic link as file name
+@smallexample
+$ @kbd{tar cfz archive.tar.gz .}
+@end smallexample
-@opindex dereference
-Normally, when @command{tar} archives a symbolic link, it writes a
-block to the archive naming the target of the link. In that way, the
-@command{tar} archive is a faithful record of the file system contents.
-@option{--dereference} (@option{-h}) is used with @option{--create} (@option{-c}), and causes
-@command{tar} to archive the files symbolic links point to, instead of
-the links themselves. When this option is used, when @command{tar}
-encounters a symbolic link, it will archive the linked-to file,
-instead of simply recording the presence of a symbolic link.
+Reading compressed archive is even simpler: you don't need to specify
+any additional options as @GNUTAR{} recognizes its format
+automatically. Thus, the following commands will list and extract the
+archive created in previous example:
-The name under which the file is stored in the file system is not
-recorded in the archive. To record both the symbolic link name and
-the file name in the system, archive the file under both names. If
-all links were recorded automatically by @command{tar}, an extracted file
-might be linked to a file name that no longer exists in the file
-system.
+@smallexample
+# List the compressed archive
+$ @kbd{tar tf archive.tar.gz}
+# Extract the compressed archive
+$ @kbd{tar xf archive.tar.gz}
+@end smallexample
-If a linked-to file is encountered again by @command{tar} while creating
-the same archive, an entire second copy of it will be stored. (This
-@emph{might} be considered a bug.)
+The only case when you have to specify a decompression option while
+reading the archive is when reading from a pipe or from a tape drive
+that does not support random access. However, in this case @GNUTAR{}
+will indicate which option you should use. For example:
-So, for portable archives, do not archive symbolic links as such,
-and use @option{--dereference} (@option{-h}): many systems do not support
-symbolic links, and moreover, your distribution might be unusable if
-it contains unresolved symbolic links.
+@smallexample
+$ @kbd{cat archive.tar.gz | tar tf -}
+tar: Archive is compressed. Use -z option
+tar: Error is not recoverable: exiting now
+@end smallexample
-@node old
-@subsection Old V7 Archives
-@cindex Format, old style
-@cindex Old style format
-@cindex Old style archives
-@cindex v7 archive format
+If you see such diagnostics, just add the suggested option to the
+invocation of @GNUTAR{}:
-Certain old versions of @command{tar} cannot handle additional
-information recorded by newer @command{tar} programs. To create an
-archive in V7 format (not ANSI), which can be read by these old
-versions, specify the @option{--format=v7} option in
-conjunction with the @option{--create} (@option{-c}) (@command{tar} also
-accepts @option{--portability} or @samp{op-old-archive} for this
-option). When you specify it,
-@command{tar} leaves out information about directories, pipes, fifos,
-contiguous files, and device files, and specifies file ownership by
-group and user IDs instead of group and user names.
+@smallexample
+$ @kbd{cat archive.tar.gz | tar tfz -}
+@end smallexample
-When updating an archive, do not use @option{--format=v7}
-unless the archive was created using this option.
+Notice also, that there are several restrictions on operations on
+compressed archives. First of all, compressed archives cannot be
+modified, i.e., you cannot update (@option{--update} (@option{-u})) them or delete
+(@option{--delete}) members from them. Likewise, you cannot append
+another @command{tar} archive to a compressed archive using
+@option{--append} (@option{-r})). Secondly, multi-volume archives cannot be
+compressed.
-In most cases, a @emph{new} format archive can be read by an @emph{old}
-@command{tar} program without serious trouble, so this option should
-seldom be needed. On the other hand, most modern @command{tar}s are
-able to read old format archives, so it might be safer for you to
-always use @option{--format=v7} for your distributions.
+The following table summarizes compression options used by @GNUTAR{}.
-@node ustar
-@subsection Ustar Archive Format
+@table @option
+@opindex gzip
+@opindex ungzip
+@item -z
+@itemx --gzip
+@itemx --ungzip
+Filter the archive through @command{gzip}.
-@cindex ustar archive format
-Archive format defined by @acronym{POSIX}.1-1988 specification is called
-@code{ustar}. Although it is more flexible than the V7 format, it
-still has many restrictions (@xref{Formats,ustar}, for the detailed
-description of @code{ustar} format). Along with V7 format,
-@code{ustar} format is a good choice for archives intended to be read
-with other implementations of @command{tar}.
+You can use @option{--gzip} and @option{--gunzip} on physical devices
+(tape drives, etc.) and remote files as well as on normal files; data
+to or from such devices or remote files is reblocked by another copy
+of the @command{tar} program to enforce the specified (or default) record
+size. The default compression parameters are used; if you need to
+override them, set @env{GZIP} environment variable, e.g.:
-To create archive in @code{ustar} format, use @option{--format=ustar}
-option in conjunction with the @option{--create} (@option{-c}).
+@smallexample
+$ @kbd{GZIP=--best tar cfz archive.tar.gz subdir}
+@end smallexample
-@node gnu
-@subsection @acronym{GNU} and old @GNUTAR{} format
+@noindent
+Another way would be to avoid the @option{--gzip} (@option{--gunzip}, @option{--ungzip}, @option{-z}) option and run
+@command{gzip} explicitly:
+
+@smallexample
+$ @kbd{tar cf - subdir | gzip --best -c - > archive.tar.gz}
+@end smallexample
+
+@cindex corrupted archives
+About corrupted compressed archives: @command{gzip}'ed files have no
+redundancy, for maximum compression. The adaptive nature of the
+compression scheme means that the compression tables are implicitly
+spread all over the archive. If you lose a few blocks, the dynamic
+construction of the compression tables becomes unsynchronized, and there
+is little chance that you could recover later in the archive.
+
+There are pending suggestions for having a per-volume or per-file
+compression in @GNUTAR{}. This would allow for viewing the
+contents without decompression, and for resynchronizing decompression at
+every volume or file, in case of corrupted archives. Doing so, we might
+lose some compressibility. But this would have make recovering easier.
+So, there are pros and cons. We'll see!
+
+@opindex bzip2
+@item -j
+@itemx --bzip2
+Filter the archive through @code{bzip2}. Otherwise like @option{--gzip}.
+
+@opindex compress
+@opindex uncompress
+@item -Z
+@itemx --compress
+@itemx --uncompress
+Filter the archive through @command{compress}. Otherwise like @option{--gzip}.
+
+The @acronym{GNU} Project recommends you not use
+@command{compress}, because there is a patent covering the algorithm it
+uses. You could be sued for patent infringement merely by running
+@command{compress}.
+
+@opindex use-compress-program
+@item --use-compress-program=@var{prog}
+Use external compression program @var{prog}. Use this option if you
+have a compression program that @GNUTAR{} does not support. There
+are two requirements to which @var{prog} should comply:
+
+First, when called without options, it should read data from standard
+input, compress it and output it on standard output.
+
+Secondly, if called with @option{-d} argument, it should do exactly
+the opposite, i.e., read the compressed data from the standard input
+and produce uncompressed data on the standard output.
+@end table
+
+@cindex gpg, using with tar
+@cindex gnupg, using with tar
+@cindex Using encrypted archives
+The @option{--use-compress-program} option, in particular, lets you
+implement your own filters, not necessarily dealing with
+compression/decomression. For example, suppose you wish to implement
+PGP encryption on top of compression, using @command{gpg} (@pxref{Top,
+gpg, gpg ---- encryption and signing tool, gpg, GNU Privacy Guard
+Manual}). The following script does that:
+
+@smallexample
+@group
+#! /bin/sh
+case $1 in
+-d) gpg --decrypt - | gzip -d -c;;
+'') gzip -c | gpg -s ;;
+*) echo "Unknown option $1">&2; exit 1;;
+esac
+@end group
+@end smallexample
+
+Suppose you name it @file{gpgz} and save it somewhere in your
+@env{PATH}. Then the following command will create a commpressed
+archive signed with your private key:
+
+@smallexample
+$ @kbd{tar -cf foo.tar.gpgz --use-compress=gpgz .}
+@end smallexample
+
+@noindent
+Likewise, the following command will list its contents:
+
+@smallexample
+$ @kbd{tar -tf foo.tar.gpgz --use-compress=gpgz .}
+@end smallexample
+
+@ignore
+The above is based on the following discussion:
+
+ I have one question, or maybe it's a suggestion if there isn't a way
+ to do it now. I would like to use @option{--gzip}, but I'd also like
+ the output to be fed through a program like @acronym{GNU}
+ @command{ecc} (actually, right now that's @samp{exactly} what I'd like
+ to use :-)), basically adding ECC protection on top of compression.
+ It seems as if this should be quite easy to do, but I can't work out
+ exactly how to go about it. Of course, I can pipe the standard output
+ of @command{tar} through @command{ecc}, but then I lose (though I
+ haven't started using it yet, I confess) the ability to have
+ @command{tar} use @command{rmt} for it's I/O (I think).
+
+ I think the most straightforward thing would be to let me specify a
+ general set of filters outboard of compression (preferably ordered,
+ so the order can be automatically reversed on input operations, and
+ with the options they require specifiable), but beggars shouldn't be
+ choosers and anything you decide on would be fine with me.
+
+ By the way, I like @command{ecc} but if (as the comments say) it can't
+ deal with loss of block sync, I'm tempted to throw some time at adding
+ that capability. Supposing I were to actually do such a thing and
+ get it (apparently) working, do you accept contributed changes to
+ utilities like that? (Leigh Clayton @file{loc@@soliton.com}, May 1995).
+
+ Isn't that exactly the role of the
+ @option{--use-compress-prog=@var{program}} option?
+ I never tried it myself, but I suspect you may want to write a
+ @var{prog} script or program able to filter stdin to stdout to
+ way you want. It should recognize the @option{-d} option, for when
+ extraction is needed rather than creation.
+
+ It has been reported that if one writes compressed data (through the
+ @option{--gzip} or @option{--compress} options) to a DLT and tries to use
+ the DLT compression mode, the data will actually get bigger and one will
+ end up with less space on the tape.
+@end ignore
+
+@node sparse
+@subsection Archiving Sparse Files
+@cindex Sparse Files
+
+Files in the file system occasionally have @dfn{holes}. A @dfn{hole}
+in a file is a section of the file's contents which was never written.
+The contents of a hole reads as all zeros. On many operating systems,
+actual disk storage is not allocated for holes, but they are counted
+in the length of the file. If you archive such a file, @command{tar}
+could create an archive longer than the original. To have @command{tar}
+attempt to recognize the holes in a file, use @option{--sparse}
+(@option{-S}). When you use this option, then, for any file using
+less disk space than would be expected from its length, @command{tar}
+searches the file for consecutive stretches of zeros. It then records
+in the archive for the file where the consecutive stretches of zeros
+are, and only archives the ``real contents'' of the file. On
+extraction (using @option{--sparse} is not needed on extraction) any
+such files have holes created wherever the continuous stretches of zeros
+were found. Thus, if you use @option{--sparse}, @command{tar} archives
+won't take more space than the original.
+
+@table @option
+@opindex sparse
+@item -S
+@itemx --sparse
+This option istructs @command{tar} to test each file for sparseness
+before attempting to archive it. If the file is found to be sparse it
+is treated specially, thus allowing to decrease the amount of space
+used by its image in the archive.
+
+This option is meaningful only when creating or updating archives. It
+has no effect on extraction.
+@end table
+
+Consider using @option{--sparse} when performing file system backups,
+to avoid archiving the expanded forms of files stored sparsely in the
+system.
+
+Even if your system has no sparse files currently, some may be
+created in the future. If you use @option{--sparse} while making file
+system backups as a matter of course, you can be assured the archive
+will never take more space on the media than the files take on disk
+(otherwise, archiving a disk filled with sparse files might take
+hundreds of tapes). @xref{Incremental Dumps}.
+
+However, be aware that @option{--sparse} option presents a serious
+drawback. Namely, in order to determine if the file is sparse
+@command{tar} has to read it before trying to archive it, so in total
+the file is read @strong{twice}. So, always bear in mind that the
+time needed to process all files with this option is roughly twice
+the time needed to archive them without it.
+@FIXME{A technical note:
+
+Programs like @command{dump} do not have to read the entire file; by
+examining the file system directly, they can determine in advance
+exactly where the holes are and thus avoid reading through them. The
+only data it need read are the actual allocated data blocks.
+@GNUTAR{} uses a more portable and straightforward
+archiving approach, it would be fairly difficult that it does
+otherwise. Elizabeth Zwicky writes to @file{comp.unix.internals}, on
+1990-12-10:
+
+@quotation
+What I did say is that you cannot tell the difference between a hole and an
+equivalent number of nulls without reading raw blocks. @code{st_blocks} at
+best tells you how many holes there are; it doesn't tell you @emph{where}.
+Just as programs may, conceivably, care what @code{st_blocks} is (care
+to name one that does?), they may also care where the holes are (I have
+no examples of this one either, but it's equally imaginable).
+
+I conclude from this that good archivers are not portable. One can
+arguably conclude that if you want a portable program, you can in good
+conscience restore files with as many holes as possible, since you can't
+get it right.
+@end quotation
+}
+
+@cindex sparse formats, defined
+When using @samp{POSIX} archive format, @GNUTAR{} is able to store
+sparse files using in three distinct ways, called @dfn{sparse
+formats}. A sparse format is identified by its @dfn{number},
+consisting, as usual of two decimal numbers, delimited by a dot. By
+default, format @samp{1.0} is used. If, for some reason, you wish to
+use an earlier format, you can select it using
+@option{--sparse-version} option.
+
+@table @option
+@opindex sparse-version
+@item --sparse-version=@var{version}
+
+Select the format to store sparse files in. Valid @var{version} values
+are: @samp{0.0}, @samp{0.1} and @samp{1.0}. @xref{Sparse Formats},
+for a detailed description of each format.
+@end table
+
+Using @option{--sparse-format} option implies @option{--sparse}.
+
+@node Attributes
+@section Handling File Attributes
+@UNREVISED
+
+When @command{tar} reads files, it updates their access times. To
+avoid this, use the @option{--atime-preserve[=METHOD]} option, which can either
+reset the access time retroactively or avoid changing it in the first
+place.
+
+Handling of file attributes
+
+@table @option
+@opindex atime-preserve
+@item --atime-preserve
+@itemx --atime-preserve=replace
+@itemx --atime-preserve=system
+Preserve the access times of files that are read. This works only for
+files that you own, unless you have superuser privileges.
+
+@option{--atime-preserve=replace} works on most systems, but it also
+restores the data modification time and updates the status change
+time. Hence it doesn't interact with incremental dumps nicely
+(@pxref{Incremental Dumps}), and it can set access or data modification times
+incorrectly if other programs access the file while @command{tar} is
+running.
+
+@option{--atime-preserve=system} avoids changing the access time in
+the first place, if the operating system supports this.
+Unfortunately, this may or may not work on any given operating system
+or file system. If @command{tar} knows for sure it won't work, it
+complains right away.
+
+Currently @option{--atime-preserve} with no operand defaults to
+@option{--atime-preserve=replace}, but this is intended to change to
+@option{--atime-preserve=system} when the latter is better-supported.
+
+@opindex touch
+@item -m
+@itemx --touch
+Do not extract data modification time.
+
+When this option is used, @command{tar} leaves the data modification times
+of the files it extracts as the times when the files were extracted,
+instead of setting it to the times recorded in the archive.
+
+This option is meaningless with @option{--list} (@option{-t}).
+
+@opindex same-owner
+@item --same-owner
+Create extracted files with the same ownership they have in the
+archive.
+
+This is the default behavior for the superuser,
+so this option is meaningful only for non-root users, when @command{tar}
+is executed on those systems able to give files away. This is
+considered as a security flaw by many people, at least because it
+makes quite difficult to correctly account users for the disk space
+they occupy. Also, the @code{suid} or @code{sgid} attributes of
+files are easily and silently lost when files are given away.
+
+When writing an archive, @command{tar} writes the user id and user name
+separately. If it can't find a user name (because the user id is not
+in @file{/etc/passwd}), then it does not write one. When restoring,
+it tries to look the name (if one was written) up in
+@file{/etc/passwd}. If it fails, then it uses the user id stored in
+the archive instead.
+
+@opindex no-same-owner
+@item --no-same-owner
+@itemx -o
+Do not attempt to restore ownership when extracting. This is the
+default behavior for ordinary users, so this option has an effect
+only for the superuser.
+
+@opindex numeric-owner
+@item --numeric-owner
+The @option{--numeric-owner} option allows (ANSI) archives to be written
+without user/group name information or such information to be ignored
+when extracting. It effectively disables the generation and/or use
+of user/group name information. This option forces extraction using
+the numeric ids from the archive, ignoring the names.
+
+This is useful in certain circumstances, when restoring a backup from
+an emergency floppy with different passwd/group files for example.
+It is otherwise impossible to extract files with the right ownerships
+if the password file in use during the extraction does not match the
+one belonging to the file system(s) being extracted. This occurs,
+for example, if you are restoring your files after a major crash and
+had booted from an emergency floppy with no password file or put your
+disk into another machine to do the restore.
+
+The numeric ids are @emph{always} saved into @command{tar} archives.
+The identifying names are added at create time when provided by the
+system, unless @option{--old-archive} (@option{-o}) is used. Numeric ids could be
+used when moving archives between a collection of machines using
+a centralized management for attribution of numeric ids to users
+and groups. This is often made through using the NIS capabilities.
+
+When making a @command{tar} file for distribution to other sites, it
+is sometimes cleaner to use a single owner for all files in the
+distribution, and nicer to specify the write permission bits of the
+files as stored in the archive independently of their actual value on
+the file system. The way to prepare a clean distribution is usually
+to have some Makefile rule creating a directory, copying all needed
+files in that directory, then setting ownership and permissions as
+wanted (there are a lot of possible schemes), and only then making a
+@command{tar} archive out of this directory, before cleaning
+everything out. Of course, we could add a lot of options to
+@GNUTAR{} for fine tuning permissions and ownership.
+This is not the good way, I think. @GNUTAR{} is
+already crowded with options and moreover, the approach just explained
+gives you a great deal of control already.
+
+@xopindex{same-permissions, short description}
+@xopindex{preserve-permissions, short description}
+@item -p
+@itemx --same-permissions
+@itemx --preserve-permissions
+Extract all protection information.
+
+This option causes @command{tar} to set the modes (access permissions) of
+extracted files exactly as recorded in the archive. If this option
+is not used, the current @code{umask} setting limits the permissions
+on extracted files. This option is by default enabled when
+@command{tar} is executed by a superuser.
+
+
+This option is meaningless with @option{--list} (@option{-t}).
+
+@opindex preserve
+@item --preserve
+Same as both @option{--same-permissions} and @option{--same-order}.
+
+The @option{--preserve} option has no equivalent short option name.
+It is equivalent to @option{--same-permissions} plus @option{--same-order}.
+
+@FIXME{I do not see the purpose of such an option. (Neither I. FP.)
+Neither do I. --Sergey}
+
+@end table
+
+@node Portability
+@section Making @command{tar} Archives More Portable
+
+Creating a @command{tar} archive on a particular system that is meant to be
+useful later on many other machines and with other versions of @command{tar}
+is more challenging than you might think. @command{tar} archive formats
+have been evolving since the first versions of Unix. Many such formats
+are around, and are not always compatible with each other. This section
+discusses a few problems, and gives some advice about making @command{tar}
+archives more portable.
+
+One golden rule is simplicity. For example, limit your @command{tar}
+archives to contain only regular files and directories, avoiding
+other kind of special files. Do not attempt to save sparse files or
+contiguous files as such. Let's discuss a few more problems, in turn.
+
+@FIXME{Discuss GNU extensions (incremental backups, multi-volume
+archives and archive labels) in GNU and PAX formats.}
+
+@menu
+* Portable Names:: Portable Names
+* dereference:: Symbolic Links
+* old:: Old V7 Archives
+* ustar:: Ustar Archives
+* gnu:: GNU and old GNU format archives.
+* posix:: @acronym{POSIX} archives
+* Checksumming:: Checksumming Problems
+* Large or Negative Values:: Large files, negative time stamps, etc.
+* Other Tars:: How to Extract GNU-Specific Data Using
+ Other @command{tar} Implementations
+@end menu
+
+@node Portable Names
+@subsection Portable Names
+
+Use portable file and member names. A name is portable if it contains
+only ASCII letters and digits, @samp{/}, @samp{.}, @samp{_}, and
+@samp{-}; it cannot be empty, start with @samp{-} or @samp{//}, or
+contain @samp{/-}. Avoid deep directory nesting. For portability to
+old Unix hosts, limit your file name components to 14 characters or
+less.
+
+If you intend to have your @command{tar} archives to be read under
+MSDOS, you should not rely on case distinction for file names, and you
+might use the @acronym{GNU} @command{doschk} program for helping you
+further diagnosing illegal MSDOS names, which are even more limited
+than System V's.
+
+@node dereference
+@subsection Symbolic Links
+@cindex File names, using symbolic links
+@cindex Symbolic link as file name
+
+@opindex dereference
+Normally, when @command{tar} archives a symbolic link, it writes a
+block to the archive naming the target of the link. In that way, the
+@command{tar} archive is a faithful record of the file system contents.
+@option{--dereference} (@option{-h}) is used with @option{--create} (@option{-c}), and causes
+@command{tar} to archive the files symbolic links point to, instead of
+the links themselves. When this option is used, when @command{tar}
+encounters a symbolic link, it will archive the linked-to file,
+instead of simply recording the presence of a symbolic link.
+
+The name under which the file is stored in the file system is not
+recorded in the archive. To record both the symbolic link name and
+the file name in the system, archive the file under both names. If
+all links were recorded automatically by @command{tar}, an extracted file
+might be linked to a file name that no longer exists in the file
+system.
+
+If a linked-to file is encountered again by @command{tar} while creating
+the same archive, an entire second copy of it will be stored. (This
+@emph{might} be considered a bug.)
+
+So, for portable archives, do not archive symbolic links as such,
+and use @option{--dereference} (@option{-h}): many systems do not support
+symbolic links, and moreover, your distribution might be unusable if
+it contains unresolved symbolic links.
+
+@node old
+@subsection Old V7 Archives
+@cindex Format, old style
+@cindex Old style format
+@cindex Old style archives
+@cindex v7 archive format
+
+Certain old versions of @command{tar} cannot handle additional
+information recorded by newer @command{tar} programs. To create an
+archive in V7 format (not ANSI), which can be read by these old
+versions, specify the @option{--format=v7} option in
+conjunction with the @option{--create} (@option{-c}) (@command{tar} also
+accepts @option{--portability} or @option{--old-archive} for this
+option). When you specify it,
+@command{tar} leaves out information about directories, pipes, fifos,
+contiguous files, and device files, and specifies file ownership by
+group and user IDs instead of group and user names.
+
+When updating an archive, do not use @option{--format=v7}
+unless the archive was created using this option.
+
+In most cases, a @emph{new} format archive can be read by an @emph{old}
+@command{tar} program without serious trouble, so this option should
+seldom be needed. On the other hand, most modern @command{tar}s are
+able to read old format archives, so it might be safer for you to
+always use @option{--format=v7} for your distributions. Notice,
+however, that @samp{ustar} format is a better alternative, as it is
+free from many of @samp{v7}'s drawbacks.
+
+@node ustar
+@subsection Ustar Archive Format
+
+@cindex ustar archive format
+Archive format defined by @acronym{POSIX}.1-1988 specification is called
+@code{ustar}. Although it is more flexible than the V7 format, it
+still has many restrictions (@xref{Formats,ustar}, for the detailed
+description of @code{ustar} format). Along with V7 format,
+@code{ustar} format is a good choice for archives intended to be read
+with other implementations of @command{tar}.
+
+To create archive in @code{ustar} format, use @option{--format=ustar}
+option in conjunction with the @option{--create} (@option{-c}).
+
+@node gnu
+@subsection @acronym{GNU} and old @GNUTAR{} format
@cindex GNU archive format
@cindex Old GNU archive format
In the majority of cases, @command{tar} will be configured to create
this format by default. This will change in the future releases, since
-we plan to make @samp{posix} format the default.
+we plan to make @samp{POSIX} format the default.
To force creation a @GNUTAR{} archive, use option
@option{--format=gnu}.
@FIXME{Describe how @acronym{POSIX} archives are extracted by non
POSIX-aware tars.}
-@node Compression
-@section Using Less Space through Compression
-
-@menu
-* gzip:: Creating and Reading Compressed Archives
-* sparse:: Archiving Sparse Files
-@end menu
-
-@node gzip
-@subsection Creating and Reading Compressed Archives
-@cindex Compressed archives
-@cindex Storing archives in compressed format
-
-@GNUTAR{} is able to create and read compressed archives. It supports
-@command{gzip} and @command{bzip2} compression programs. For backward
-compatibilty, it also supports @command{compress} command, although
-we strongly recommend against using it, since there is a patent
-covering the algorithm it uses and you could be sued for patent
-infringement merely by running @command{compress}! Besides, it is less
-effective than @command{gzip} and @command{bzip2}.
-
-Creating a compressed archive is simple: you just specify a
-@dfn{compression option} along with the usual archive creation
-commands. The compression option is @option{-z} (@option{--gzip}) to
-create a @command{gzip} compressed archive, @option{-j}
-(@option{--bzip2}) to create a @command{bzip2} compressed archive, and
-@option{-Z} (@option{--compress}) to use @command{compress} program.
-For example:
-
-@smallexample
-$ @kbd{tar cfz archive.tar.gz .}
-@end smallexample
-
-Reading compressed archive is even simpler: you don't need to specify
-any additional options as @GNUTAR{} recognizes its format
-automatically. Thus, the following commands will list and extract the
-archive created in previous example:
-
-@smallexample
-# List the compressed archive
-$ @kbd{tar tf archive.tar.gz}
-# Extract the compressed archive
-$ @kbd{tar xf archive.tar.gz}
-@end smallexample
-
-The only case when you have to specify a decompression option while
-reading the archive is when reading from a pipe or from a tape drive
-that does not support random access. However, in this case @GNUTAR{}
-will indicate which option you should use. For example:
-
-@smallexample
-$ @kbd{cat archive.tar.gz | tar tf -}
-tar: Archive is compressed. Use -z option
-tar: Error is not recoverable: exiting now
-@end smallexample
-
-If you see such diagnostics, just add the suggested option to the
-invocation of @GNUTAR{}:
-
-@smallexample
-$ @kbd{cat archive.tar.gz | tar tfz -}
-@end smallexample
-
-Notice also, that there are several restrictions on operations on
-compressed archives. First of all, compressed archives cannot be
-modified, i.e., you cannot update (@option{--update} (@option{-u})) them or delete
-(@option{--delete}) members from them. Likewise, you cannot append
-another @command{tar} archive to a compressed archive using
-@option{--append} (@option{-r})). Secondly, multi-volume archives cannot be
-compressed.
-
-The following table summarizes compression options used by @GNUTAR{}.
-
-@table @option
-@opindex gzip
-@opindex ungzip
-@item -z
-@itemx --gzip
-@itemx --ungzip
-Filter the archive through @command{gzip}.
-
-You can use @option{--gzip} and @option{--gunzip} on physical devices
-(tape drives, etc.) and remote files as well as on normal files; data
-to or from such devices or remote files is reblocked by another copy
-of the @command{tar} program to enforce the specified (or default) record
-size. The default compression parameters are used; if you need to
-override them, set @env{GZIP} environment variable, e.g.:
-
-@smallexample
-$ @kbd{GZIP=--best tar cfz archive.tar.gz subdir}
-@end smallexample
-
-@noindent
-Another way would be to avoid the @option{--gzip} (@option{--gunzip}, @option{--ungzip}, @option{-z}) option and run
-@command{gzip} explicitly:
-
-@smallexample
-$ @kbd{tar cf - subdir | gzip --best -c - > archive.tar.gz}
-@end smallexample
-
-@cindex corrupted archives
-About corrupted compressed archives: @command{gzip}'ed files have no
-redundancy, for maximum compression. The adaptive nature of the
-compression scheme means that the compression tables are implicitly
-spread all over the archive. If you lose a few blocks, the dynamic
-construction of the compression tables becomes unsynchronized, and there
-is little chance that you could recover later in the archive.
-
-There are pending suggestions for having a per-volume or per-file
-compression in @GNUTAR{}. This would allow for viewing the
-contents without decompression, and for resynchronizing decompression at
-every volume or file, in case of corrupted archives. Doing so, we might
-lose some compressibility. But this would have make recovering easier.
-So, there are pros and cons. We'll see!
-
-@opindex bzip2
-@item -j
-@itemx --bzip2
-Filter the archive through @code{bzip2}. Otherwise like @option{--gzip}.
-
-@opindex compress
-@opindex uncompress
-@item -Z
-@itemx --compress
-@itemx --uncompress
-Filter the archive through @command{compress}. Otherwise like @option{--gzip}.
-
-The @acronym{GNU} Project recommends you not use
-@command{compress}, because there is a patent covering the algorithm it
-uses. You could be sued for patent infringement merely by running
-@command{compress}.
-
-@opindex use-compress-program
-@item --use-compress-program=@var{prog}
-Use external compression program @var{prog}. Use this option if you
-have a compression program that @GNUTAR{} does not support. There
-are two requirements to which @var{prog} should comply:
-
-First, when called without options, it should read data from standard
-input, compress it and output it on standard output.
-
-Secondly, if called with @option{-d} argument, it should do exactly
-the opposite, i.e., read the compressed data from the standard input
-and produce uncompressed data on the standard output.
-@end table
-
-@cindex gpg, using with tar
-@cindex gnupg, using with tar
-@cindex Using encrypted archives
-The @option{--use-compress-program} option, in particular, lets you
-implement your own filters, not necessarily dealing with
-compression/decomression. For example, suppose you wish to implement
-PGP encryption on top of compression, using @command{gpg} (@pxref{Top,
-gpg, gpg ---- encryption and signing tool, gpg, GNU Privacy Guard
-Manual}). The following script does that:
-
-@smallexample
-@group
-#! /bin/sh
-case $1 in
--d) gpg --decrypt - | gzip -d -c;;
-'') gzip -c | gpg -s ;;
-*) echo "Unknown option $1">&2; exit 1;;
-esac
-@end group
-@end smallexample
+@node Other Tars
+@subsection How to Extract GNU-Specific Data Using Other @command{tar} Implementations
-Suppose you name it @file{gpgz} and save it somewhere in your
-@env{PATH}. Then the following command will create a commpressed
-archive signed with your private key:
+In previous sections you became acquainted with various quircks
+necessary to make your archives portable. Sometimes you may need to
+extract archives containing GNU-specific members using some
+third-party @command{tar} implementation or an older version of
+@GNUTAR{}. Of course your best bet is to have @GNUTAR{} installed,
+but if it is for some reason impossible, this section will explain
+how to cope without it.
-@smallexample
-$ @kbd{tar -cf foo.tar.gpgz --use-compress=gpgz .}
-@end smallexample
+When we speak about @dfn{GNU-specific} members we mean two classes of
+them: members split between the volumes of a multi-volume archive and
+sparse members. You will be able to always recover such members if
+the archive is in PAX format. In addition split members can be
+recovered from archives in old GNU format. The following subsections
+describe the required procedures in detail.
-@noindent
-Likewise, the following command will list its contents:
+@menu
+* Split Recovery:: Members Split Between Volumes
+* Sparse Recovery:: Sparse Members
+@end menu
-@smallexample
-$ @kbd{tar -tf foo.tar.gpgz --use-compress=gpgz .}
-@end smallexample
+@node Split Recovery
+@subsubsection Extracting Members Split Between Volumes
-@ignore
-The above is based on the following discussion:
+@cindex Mutli-volume archives, extracting using non-GNU tars
+If a member is split between several volumes of an old GNU format archive
+most third party @command{tar} implementation will fail to extract
+it. To extract it, use @command{tarcat} program (@pxref{Tarcat}).
+This program is available from
+@uref{http://www.gnu.org/@/software/@/tar/@/utils/@/tarcat.html, @GNUTAR{}
+home page}. It concatenates several archive volumes into a single
+valid archive. For example, if you have three volumes named from
+@file{vol-1.tar} to @file{vol-2.tar}, you can do the following to
+extract them using a third-party @command{tar}:
- I have one question, or maybe it's a suggestion if there isn't a way
- to do it now. I would like to use @option{--gzip}, but I'd also like
- the output to be fed through a program like @acronym{GNU}
- @command{ecc} (actually, right now that's @samp{exactly} what I'd like
- to use :-)), basically adding ECC protection on top of compression.
- It seems as if this should be quite easy to do, but I can't work out
- exactly how to go about it. Of course, I can pipe the standard output
- of @command{tar} through @command{ecc}, but then I lose (though I
- haven't started using it yet, I confess) the ability to have
- @command{tar} use @command{rmt} for it's I/O (I think).
+@smallexample
+$ @kbd{tarcat vol-1.tar vol-2.tar vol-3.tar | tar xf -}
+@end smallexample
- I think the most straightforward thing would be to let me specify a
- general set of filters outboard of compression (preferably ordered,
- so the order can be automatically reversed on input operations, and
- with the options they require specifiable), but beggars shouldn't be
- choosers and anything you decide on would be fine with me.
+@cindex Mutli-volume archives in PAX format, extracting using non-GNU tars
+You could use this approach for many (although not all) PAX
+format archives as well. However, extracting split members from a PAX
+archive is a much easier task, because PAX volumes are constructed in
+such a way that each part of a split member is extracted as a
+different file by @command{tar} implementations that are not aware of
+GNU extensions. More specifically, the very first part retains its
+original name, and all subsequent parts are named using the pattern:
- By the way, I like @command{ecc} but if (as the comments say) it can't
- deal with loss of block sync, I'm tempted to throw some time at adding
- that capability. Supposing I were to actually do such a thing and
- get it (apparently) working, do you accept contributed changes to
- utilities like that? (Leigh Clayton @file{loc@@soliton.com}, May 1995).
-
- Isn't that exactly the role of the
- @option{--use-compress-prog=@var{program}} option?
- I never tried it myself, but I suspect you may want to write a
- @var{prog} script or program able to filter stdin to stdout to
- way you want. It should recognize the @option{-d} option, for when
- extraction is needed rather than creation.
+@smallexample
+%d/GNUFileParts.%p/%f.%n
+@end smallexample
- It has been reported that if one writes compressed data (through the
- @option{--gzip} or @option{--compress} options) to a DLT and tries to use
- the DLT compression mode, the data will actually get bigger and one will
- end up with less space on the tape.
-@end ignore
+@noindent
+where symbols preceeded by @samp{%} are @dfn{macro characters} that
+have the following meaning:
-@node sparse
-@subsection Archiving Sparse Files
-@cindex Sparse Files
-@UNREVISED
+@multitable @columnfractions .25 .55
+@headitem Meta-character @tab Replaced By
+@item %d @tab The directory name of the file, equivalent to the
+result of the @command{dirname} utility on its full name.
+@item %f @tab The file name of the file, equivalent to the result
+of the @command{basename} utility on its full name.
+@item %p @tab The process ID of the @command{tar} process that
+created the archive.
+@item %n @tab Ordinal number of this particular part.
+@end multitable
-@table @option
-@opindex sparse
-@item -S
-@itemx --sparse
-Handle sparse files efficiently.
-@end table
+For example, if, a file @file{var/longfile} was split during archive
+creation between three volumes, and the creator @command{tar} process
+had process ID @samp{27962}, then the member names will be:
-This option causes all files to be put in the archive to be tested for
-sparseness, and handled specially if they are. The @option{--sparse}
-(@option{-S}) option is useful when many @code{dbm} files, for example, are being
-backed up. Using this option dramatically decreases the amount of
-space needed to store such a file.
+@smallexample
+var/longfile
+var/GNUFileParts.27962/longfile.1
+var/GNUFileParts.27962/longfile.2
+@end smallexample
-In later versions, this option may be removed, and the testing and
-treatment of sparse files may be done automatically with any special
-@acronym{GNU} options. For now, it is an option needing to be specified on
-the command line with the creation or updating of an archive.
+When you extract your archive using a third-party @command{tar}, these
+files will be created on your disk, and the only thing you will need
+to do to restore your file in its original form is concatenate them in
+the proper order, for example:
-Files in the file system occasionally have ``holes.'' A hole in a file
-is a section of the file's contents which was never written. The
-contents of a hole read as all zeros. On many operating systems,
-actual disk storage is not allocated for holes, but they are counted
-in the length of the file. If you archive such a file, @command{tar}
-could create an archive longer than the original. To have @command{tar}
-attempt to recognize the holes in a file, use @option{--sparse} (@option{-S}). When
-you use this option, then, for any file using less disk space than
-would be expected from its length, @command{tar} searches the file for
-consecutive stretches of zeros. It then records in the archive for
-the file where the consecutive stretches of zeros are, and only
-archives the ``real contents'' of the file. On extraction (using
-@option{--sparse} is not needed on extraction) any such
-files have holes created wherever the continuous stretches of zeros
-were found. Thus, if you use @option{--sparse}, @command{tar} archives
-won't take more space than the original.
+@smallexample
+@group
+$ @kbd{cd var}
+$ @kbd{cat GNUFileParts.27962/longfile.1 \
+ GNUFileParts.27962/longfile.2 >> longfile}
+$ rm -f GNUFileParts.27962
+@end group
+@end smallexample
-A file is sparse if it contains blocks of zeros whose existence is
-recorded, but that have no space allocated on disk. When you specify
-the @option{--sparse} option in conjunction with the @option{--create}
-(@option{-c}) operation, @command{tar} tests all files for sparseness
-while archiving. If @command{tar} finds a file to be sparse, it uses a
-sparse representation of the file in the archive. @xref{create}, for
-more information about creating archives.
+Notice, that if the @command{tar} implementation you use supports PAX
+format archives, it will probably emit warnings about unknown keywords
+during extraction. They will lool like this:
-@option{--sparse} is useful when archiving files, such as dbm files,
-likely to contain many nulls. This option dramatically
-decreases the amount of space needed to store such an archive.
+@smallexample
+@group
+Tar file too small
+Unknown extended header keyword 'GNU.volume.filename' ignored.
+Unknown extended header keyword 'GNU.volume.size' ignored.
+Unknown extended header keyword 'GNU.volume.offset' ignored.
+@end group
+@end smallexample
-@quotation
-@strong{Please Note:} Always use @option{--sparse} when performing file
-system backups, to avoid archiving the expanded forms of files stored
-sparsely in the system.
+@noindent
+You can safely ignore these warnings.
-Even if your system has no sparse files currently, some may be
-created in the future. If you use @option{--sparse} while making file
-system backups as a matter of course, you can be assured the archive
-will never take more space on the media than the files take on disk
-(otherwise, archiving a disk filled with sparse files might take
-hundreds of tapes). @xref{Incremental Dumps}.
-@end quotation
+If your @command{tar} implementation is not PAX-aware, you will get
+more warnigns and more files generated on your disk, e.g.:
-@command{tar} ignores the @option{--sparse} option when reading an archive.
+@smallexample
+@group
+$ @kbd{tar xf vol-1.tar}
+var/PaxHeaders.27962/longfile: Unknown file type 'x', extracted as
+normal file
+Unexpected EOF in archive
+$ @kbd{tar xf vol-2.tar}
+tmp/GlobalHead.27962.1: Unknown file type 'g', extracted as normal file
+GNUFileParts.27962/PaxHeaders.27962/sparsefile.1: Unknown file type
+'x', extracted as normal file
+@end group
+@end smallexample
-@table @option
-@item --sparse
-@itemx -S
-Files stored sparsely in the file system are represented sparsely in
-the archive. Use in conjunction with write operations.
-@end table
+Ignore these warnings. The @file{PaxHeaders.*} directories created
+will contain files with @dfn{extended header keywords} describing the
+extracted files. You can delete them, unless they describe sparse
+members. Read further to learn more about them.
-However, users should be well aware that at archive creation time,
-@GNUTAR{} still has to read whole disk file to
-locate the @dfn{holes}, and so, even if sparse files use little space
-on disk and in the archive, they may sometimes require inordinate
-amount of time for reading and examining all-zero blocks of a file.
-Although it works, it's painfully slow for a large (sparse) file, even
-though the resulting tar archive may be small. (One user reports that
-dumping a @file{core} file of over 400 megabytes, but with only about
-3 megabytes of actual data, took about 9 minutes on a Sun Sparcstation
-ELC, with full CPU utilization.)
-
-This reading is required in all cases and is not related to the fact
-the @option{--sparse} option is used or not, so by merely @emph{not}
-using the option, you are not saving time@footnote{Well! We should say
-the whole truth, here. When @option{--sparse} is selected while creating
-an archive, the current @command{tar} algorithm requires sparse files to be
-read twice, not once. We hope to develop a new archive format for saving
-sparse files in which one pass will be sufficient.}.
+@node Sparse Recovery
+@subsubsection Extracting Sparse Members
-Programs like @command{dump} do not have to read the entire file; by
-examining the file system directly, they can determine in advance
-exactly where the holes are and thus avoid reading through them. The
-only data it need read are the actual allocated data blocks.
-@GNUTAR{} uses a more portable and straightforward
-archiving approach, it would be fairly difficult that it does
-otherwise. Elizabeth Zwicky writes to @file{comp.unix.internals}, on
-1990-12-10:
+@cindex sparse files, extracting with non-GNU tars
+Any @command{tar} implementation will be able to extract sparse members from a
+PAX archive. However, the extracted files will be @dfn{condensed},
+i.e. any zero blocks will be removed from them. When we restore such
+a condensed file to its original form, by adding zero bloks (or
+@dfn{holes}) back to their original locations, we call this process
+@dfn{expanding} a compressed sparse file.
-@quotation
-What I did say is that you cannot tell the difference between a hole and an
-equivalent number of nulls without reading raw blocks. @code{st_blocks} at
-best tells you how many holes there are; it doesn't tell you @emph{where}.
-Just as programs may, conceivably, care what @code{st_blocks} is (care
-to name one that does?), they may also care where the holes are (I have
-no examples of this one either, but it's equally imaginable).
+@pindex xsparse
+To expand a file, you will need a simple auxiliary program called
+@command{xsparse}. It is available in source form from
+@uref{http://www.gnu.org/@/software/@/tar/@/utils/@/xsparse.html, @GNUTAR{}
+home page}.
-I conclude from this that good archivers are not portable. One can
-arguably conclude that if you want a portable program, you can in good
-conscience restore files with as many holes as possible, since you can't
-get it right.
-@end quotation
+@cindex sparse files v.1.0, extracting with non-GNU tars
+Let's begin with archive members in @dfn{sparse format
+version 1.0}@footnote{@xref{PAX 1}.}, which are the easiest to expand.
+The condensed file will contain both file map and file data, so no
+additional data will be needed to restore it. If the original file
+name was @file{@var{dir}/@var{name}}, then the condensed file will be
+named @file{@var{dir}/@/GNUSparseFile.@var{n}/@/@var{name}}, where
+@var{n} is a decimal number@footnote{technically speaking, @var{n} is a
+@dfn{process ID} of the @command{tar} process which created the
+archive (@pxref{PAX keywords}).}.
-@node Attributes
-@section Handling File Attributes
-@UNREVISED
+To expand a version 1.0 file, run @command{xsparse} as follows:
-When @command{tar} reads files, it updates their access times. To
-avoid this, use the @option{--atime-preserve[=METHOD]} option, which can either
-reset the access time retroactively or avoid changing it in the first
-place.
+@smallexample
+$ @kbd{xsparse @file{cond-file}}
+@end smallexample
-Handling of file attributes
+@noindent
+where @file{cond-file} is the name of the condensed file. The utility
+will deduce the name for the resulting expanded file using the
+following algorithm:
+
+@enumerate 1
+@item If @file{cond-file} does not contain any directories,
+@file{../cond-file} will be used;
+
+@item If @file{cond-file} has the form
+@file{@var{dir}/@var{t}/@var{name}}, where both @var{t} and @var{name}
+are simple names, with no @samp{/} characters in them, the output file
+name will be @file{@var{dir}/@var{name}}.
+
+@item Otherwise, if @file{cond-file} has the form
+@file{@var{dir}/@var{name}}, the output file name will be
+@file{@var{name}}.
+@end enumerate
-@table @option
-@opindex atime-preserve
-@item --atime-preserve
-@itemx --atime-preserve=replace
-@itemx --atime-preserve=system
-Preserve the access times of files that are read. This works only for
-files that you own, unless you have superuser privileges.
+In the unlikely case when this algorithm does not suite your needs,
+you can explicitely specify output file name as a second argument to
+the command:
-@option{--atime-preserve=replace} works on most systems, but it also
-restores the data modification time and updates the status change
-time. Hence it doesn't interact with incremental dumps nicely
-(@pxref{Backups}), and it can set access or data modification times
-incorrectly if other programs access the file while @command{tar} is
-running.
+@smallexample
+$ @kbd{xsparse @file{cond-file}}
+@end smallexample
-@option{--atime-preserve=system} avoids changing the access time in
-the first place, if the operating system supports this.
-Unfortunately, this may or may not work on any given operating system
-or file system. If @command{tar} knows for sure it won't work, it
-complains right away.
+It is often a good idea to run @command{xsparse} in @dfn{dry run} mode
+first. In this mode, the command does not actually expand the file,
+but verbosely lists all actions it would be taking to do so. The dry
+run mode is enabled by @option{-n} command line argument:
-Currently @option{--atime-preserve} with no operand defaults to
-@option{--atime-preserve=replace}, but this is intended to change to
-@option{--atime-preserve=system} when the latter is better-supported.
+@smallexample
+@group
+$ @kbd{xsparse -n /home/gray/GNUSparseFile.6058/sparsefile}
+Reading v.1.0 sparse map
+Expanding file `/home/gray/GNUSparseFile.6058/sparsefile' to
+`/home/gray/sparsefile'
+Finished dry run
+@end group
+@end smallexample
-@opindex touch
-@item -m
-@itemx --touch
-Do not extract data modification time.
+To actually expand the file, you would run:
-When this option is used, @command{tar} leaves the data modification times
-of the files it extracts as the times when the files were extracted,
-instead of setting it to the times recorded in the archive.
+@smallexample
+$ @kbd{xsparse /home/gray/GNUSparseFile.6058/sparsefile}
+@end smallexample
-This option is meaningless with @option{--list} (@option{-t}).
+@noindent
+The program behaves the same way all UNIX utilities do: it will keep
+quiet unless it has simething important to tell you (e.g. an error
+condition or something). If you wish it to produce verbose output,
+similar to that from the dry run mode, give it @option{-v} option:
-@opindex same-owner
-@item --same-owner
-Create extracted files with the same ownership they have in the
-archive.
+@smallexample
+@group
+$ @kbd{xsparse -v /home/gray/GNUSparseFile.6058/sparsefile}
+Reading v.1.0 sparse map
+Expanding file `/home/gray/GNUSparseFile.6058/sparsefile' to
+`/home/gray/sparsefile'
+Done
+@end group
+@end smallexample
-This is the default behavior for the superuser,
-so this option is meaningful only for non-root users, when @command{tar}
-is executed on those systems able to give files away. This is
-considered as a security flaw by many people, at least because it
-makes quite difficult to correctly account users for the disk space
-they occupy. Also, the @code{suid} or @code{sgid} attributes of
-files are easily and silently lost when files are given away.
+Additionally, if your @command{tar} implementation has extracted the
+@dfn{extended headers} for this file, you can instruct @command{xstar}
+to use them in order to verify the integrity of the expanded file.
+The option @option{-x} sets the name of the extended header file to
+use. Continuing our example:
-When writing an archive, @command{tar} writes the user id and user name
-separately. If it can't find a user name (because the user id is not
-in @file{/etc/passwd}), then it does not write one. When restoring,
-it tries to look the name (if one was written) up in
-@file{/etc/passwd}. If it fails, then it uses the user id stored in
-the archive instead.
+@smallexample
+@group
+$ @kbd{xsparse -v -x /home/gray/PaxHeaders.6058/sparsefile \
+ /home/gray/GNUSparseFile.6058/sparsefile}
+Reading extended header file
+Found variable GNU.sparse.major = 1
+Found variable GNU.sparse.minor = 0
+Found variable GNU.sparse.name = sparsefile
+Found variable GNU.sparse.realsize = 217481216
+Reading v.1.0 sparse map
+Expanding file `/home/gray/GNUSparseFile.6058/sparsefile' to
+`/home/gray/sparsefile'
+Done
+@end group
+@end smallexample
-@opindex no-same-owner
-@item --no-same-owner
-@itemx -o
-Do not attempt to restore ownership when extracting. This is the
-default behavior for ordinary users, so this option has an effect
-only for the superuser.
+@anchor{extracting sparse v.0.x}
+@cindex sparse files v.0.1, extracting with non-GNU tars
+@cindex sparse files v.0.0, extracting with non-GNU tars
+An @dfn{extended header} is a special @command{tar} archive header
+that precedes an archive member and contains a set of
+@dfn{variables}, describing the member properties that cannot be
+stored in the standard @code{ustar} header. While optional for
+expanding sparse version 1.0 members, use of extended headers is
+mandatory when expanding sparse members in older sparse formats: v.0.0
+and v.0.1 (The sparse formats are described in detail in @pxref{Sparse
+Formats}). So, for this format, the question is: how to obtain
+extended headers from the archive?
+
+If you use a @command{tar} implementation that does not support PAX
+format, extended headers for each member will be extracted as a
+separate file. If we represent the member name as
+@file{@var{dir}/@var{name}}, then the extended header file will be
+named @file{@var{dir}/@/PaxHeaders.@var{n}/@/@var{name}}, where
+@var{n} is an integer number.
+
+Things become more difficult if your @command{tar} implementation
+does support PAX headers, because in this case you will have to
+manually extract the headers. We recommend the following algorithm:
+
+@enumerate 1
+@item
+Consult the documentation for your @command{tar} implementation for an
+option that will print @dfn{block numbers} along with the archive
+listing (analogous to @GNUTAR{}'s @option{-R} option). For example,
+@command{star} has @option{-block-number}.
-@opindex numeric-owner
-@item --numeric-owner
-The @option{--numeric-owner} option allows (ANSI) archives to be written
-without user/group name information or such information to be ignored
-when extracting. It effectively disables the generation and/or use
-of user/group name information. This option forces extraction using
-the numeric ids from the archive, ignoring the names.
+@item
+Obtain the verbose listing using the @samp{block number} option, and
+find the position of the sparse member in question and the member
+immediately following it. For example, running @command{star} on our
+archive we obtain:
-This is useful in certain circumstances, when restoring a backup from
-an emergency floppy with different passwd/group files for example.
-It is otherwise impossible to extract files with the right ownerships
-if the password file in use during the extraction does not match the
-one belonging to the file system(s) being extracted. This occurs,
-for example, if you are restoring your files after a major crash and
-had booted from an emergency floppy with no password file or put your
-disk into another machine to do the restore.
+@smallexample
+@group
+$ @kbd{star -t -v -block-number -f arc.tar}
+@dots{}
+star: Unknown extended header keyword 'GNU.sparse.size' ignored.
+star: Unknown extended header keyword 'GNU.sparse.numblocks' ignored.
+star: Unknown extended header keyword 'GNU.sparse.name' ignored.
+star: Unknown extended header keyword 'GNU.sparse.map' ignored.
+block 56: 425984 -rw-r--r-- gray/users Jun 25 14:46 2006 GNUSparseFile.28124/sparsefile
+block 897: 65391 -rw-r--r-- gray/users Jun 24 20:06 2006 README
+@dots{}
+@end group
+@end smallexample
-The numeric ids are @emph{always} saved into @command{tar} archives.
-The identifying names are added at create time when provided by the
-system, unless @option{--old-archive} (@option{-o}) is used. Numeric ids could be
-used when moving archives between a collection of machines using
-a centralized management for attribution of numeric ids to users
-and groups. This is often made through using the NIS capabilities.
+@noindent
+(as usual, ignore the warnings about unknown keywords.)
-When making a @command{tar} file for distribution to other sites, it
-is sometimes cleaner to use a single owner for all files in the
-distribution, and nicer to specify the write permission bits of the
-files as stored in the archive independently of their actual value on
-the file system. The way to prepare a clean distribution is usually
-to have some Makefile rule creating a directory, copying all needed
-files in that directory, then setting ownership and permissions as
-wanted (there are a lot of possible schemes), and only then making a
-@command{tar} archive out of this directory, before cleaning
-everything out. Of course, we could add a lot of options to
-@GNUTAR{} for fine tuning permissions and ownership.
-This is not the good way, I think. @GNUTAR{} is
-already crowded with options and moreover, the approach just explained
-gives you a great deal of control already.
+@item
+Let @var{size} be the size of the sparse member, @var{Bs} be its block number
+and @var{Bn} be the block number of the next member.
+Compute:
-@xopindex{same-permissions, short description}
-@xopindex{preserve-permissions, short description}
-@item -p
-@itemx --same-permissions
-@itemx --preserve-permissions
-Extract all protection information.
+@smallexample
+@var{N} = @var{Bs} - @var{Bn} - @var{size}/512 - 2
+@end smallexample
-This option causes @command{tar} to set the modes (access permissions) of
-extracted files exactly as recorded in the archive. If this option
-is not used, the current @code{umask} setting limits the permissions
-on extracted files. This option is by default enabled when
-@command{tar} is executed by a superuser.
+@noindent
+This number gives the size of the extended header part in tar @dfn{blocks}.
+In our example, this formula gives: @code{897 - 56 - 425984 / 512 - 2
+= 7}.
+@item
+Use @command{dd} to extract the headers:
-This option is meaningless with @option{--list} (@option{-t}).
+@smallexample
+@kbd{dd if=@var{archive} of=@var{hname} bs=512 skip=@var{Bs} count=@var{N}}
+@end smallexample
-@opindex preserve
-@item --preserve
-Same as both @option{--same-permissions} and @option{--same-order}.
+@noindent
+where @var{archive} is the archive name, @var{hname} is a name of the
+file to store the extended header in, @var{Bs} and @var{N} are
+computed in previous steps.
-The @option{--preserve} option has no equivalent short option name.
-It is equivalent to @option{--same-permissions} plus @option{--same-order}.
+In our example, this command will be
-@FIXME{I do not see the purpose of such an option. (Neither I. FP.)
-Neither do I. --Sergey}
+@smallexample
+$ @kbd{dd if=arc.tar of=xhdr bs=512 skip=56 count=7}
+@end smallexample
+@end enumerate
-@end table
+Finally, you can expand the condensed file, using the obtained header:
+
+@smallexample
+@group
+$ @kbd{xsparse -v -x xhdr GNUSparseFile.6058/sparsefile}
+Reading extended header file
+Found variable GNU.sparse.size = 217481216
+Found variable GNU.sparse.numblocks = 208
+Found variable GNU.sparse.name = sparsefile
+Found variable GNU.sparse.map = 0,2048,1050624,2048,@dots{}
+Expanding file `GNUSparseFile.28124/sparsefile' to `sparsefile'
+Done
+@end group
+@end smallexample
@node cpio
@section Comparison of @command{tar} and @command{cpio}
@option{--label=@var{archive-label}} again in conjunction with the
@option{--append}, @option{--update} or @option{--concatenate} operation.
-@FIXME{This is no longer true: Multivolume archives in @samp{POSIX}
-format can be extracted using any posix-compliant tar
-implementation. The split members can then be recreated from parts
-using a simple shell script. Provide more information about it:}
-Beware that there is @emph{no} real standard about the proper way, for
-a @command{tar} archive, to span volume boundaries. If you have a
-multi-volume created by some vendor's @command{tar}, there is almost
-no chance you could read all the volumes with @GNUTAR{}.
-The converse is also true: you may not expect
-multi-volume archives created by @GNUTAR{} to be
-fully recovered by vendor's @command{tar}. Since there is little
-chance that, in mixed system configurations, some vendor's
-@command{tar} will work on another vendor's machine, and there is a
-great chance that @GNUTAR{} will work on most of
-them, your best bet is to install @GNUTAR{} on all
-machines between which you know exchange of files is possible.
+Notice that multi-volume support is a GNU extension and the archives
+created in this mode should be read only using @GNUTAR{}. If you
+absolutely have to process such archives using a third-party @command{tar}
+implementation, read @ref{Split Recovery}.
@node Tape Files
@subsection Tape Files
Right margin of the text output. Used for wrapping.
@end deftypevr
-@node Genfile
-@appendix Genfile
-@include genfile.texi
-
@node Tar Internals
@appendix Tar Internals
@include intern.texi
+@node Genfile
+@appendix Genfile
+@include genfile.texi
+
@node Free Software Needs Free Documentation
@appendix Free Software Needs Free Documentation
@include freemanuals.texi
projects / chaz / tar / blobdiff