[-]
[+]
|
Changed |
baculafs.changes
|
|
[-]
[+]
|
Changed |
baculafs.spec
^
|
|
|
Deleted |
BaculaFS-0.1.4.tar.gz
^
|
[-]
[+]
|
Changed |
BaculaFS-0.1.7.tar.gz/BaculaFS.egg-info/PKG-INFO
^
|
@@ -1,6 +1,6 @@
Metadata-Version: 1.0
Name: BaculaFS
-Version: 0.1.0
+Version: 0.1.7
Summary: Bacula Filesystem in USErspace
Home-page: http://code.google.com/p/baculafs
Author: Avi Rozen
@@ -10,64 +10,67 @@
BaculaFS
========
- BaculaFS_ - Exposes the Bacula_ catalog and storage as a Filesystem in
+ BaculaFS_ - Exposes the Bacula_ [*]_ catalog and storage as a Filesystem in
USErspace (FUSE_).
.. _BaculaFS: http://code.google.com/p/baculafs
.. _Bacula: http://www.bacula.org
.. _FUSE: http://fuse.sourceforge.net/
- Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
+ Copyright |(C)| 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
- INTRODUCTION
+ .. contents::
+
+ Introduction
------------
- **BaculaFS** is a tool, independent of Bacula, that represents the
- Bacula catalog and backup storage media as a read-only filesystem in
- userspace.
+ **BaculaFS** is a tool, developed independently of Bacula, that
+ represents the Bacula catalog and backup storage media as a read-only
+ filesystem in userspace.
**BaculaFS** is specifically designed to cater for the following
use-cases:
- maintaining a remote snapshot of the files in the backup storage
- using `rsync`_
+ using `rsync`_ or `duplicity`_
- auditing the contents of backup jobs, without resorting to SQL
- queries
+ queries
- comparing backup jobs (using several mount points)
- Note that **BaculaFS** is a maintenance tool - it's operation may
+ Note that **BaculaFS** is a maintenance tool - its operation may
interfere with the normal operation of a live Bacula setup (see the
- LIMITATIONS_ section below).
+ Limitations_ section below).
.. _rsync: http://samba.anu.edu.au/rsync/
+ .. _duplicity: http://www.nongnu.org/duplicity/
- REQUIREMENTS
+ Requirements
------------
**BaculaFS** has been tested with the following set of required
software packages:
- + Bacula 3.0.2 with one of the following database backends:
-
- * SQLite_ 3.6.21
- * MySQL_ 5.4.1
- * PostgreSQL_ 8.3.9
+ + Bacula 5.0.2 with one of the following database backends:
+ * SQLite_ 3.7.3
+ * MySQL_ 5.1.49
+ * PostgreSQL_ 8.4.5
+
+ FUSE_ components:
- * Python FUSE 0.2
- * FUSE library 2.8.1
- * fusermount 2.8.1
- * FUSE kernel interface 7.12
-
- + Python_ 2.5.4, with the following additional libraries:
-
- * MySQLdb_ 1.2.2
- * psycopg2_ 2.0.13
- * pexpect_ 2.3
-
+ * Python FUSE 0.2.1
+ * FUSE library 2.8.4
+ * fusermount 2.8.4
+ * FUSE kernel interface 7.12
+
+ + Python_ 2.6.6, with the following additional libraries:
+
+ * MySQLdb_ 1.2.2
+ * psycopg2_ 2.2.1
+ * pexpect_ 2.3
+
+ attr_ extended attributes utilities 2.4.44
**BaculaFS** requires the following in order to function:
@@ -75,7 +78,7 @@
+ access to the database that's used to store the Bacula catalog
+ access to the Bacula storage device
+ access to the Bacula ``bextract`` utility (bundled with the Bacula
- storage daemon installation)
+ storage daemon installation)
.. _SQLite: http://www.sqlite.org/
.. _MySQL: http://www.mysql.com/
@@ -88,7 +91,7 @@
.. _attr: http://savannah.nongnu.org/projects/attr
- INSTALLATION
+ Installation
------------
Extract the source code archive to a temporary directory, ``cd`` to
@@ -96,117 +99,136 @@
::
- python setup.py install
+ python setup.py install
- USAGE
+ Usage
-----
::
- baculafs [mountpoint] [options]
-
- Options:
- --version show program's version number and exit
- -h, --help show this help message and exit
- -o opt,[opt...] mount options
- -o driver=mysql|postgresql|sqlite|sqlite3
- database driver [default: sqlite3]
- -o host=HOST database server address [default: localhost]
- -o port=PORT database server port
- -o database=PATH database name [default: bacula]
- -o username=USERNAME database user name [default: bacula]
- -o password=PASSWORD database password
- -o conf=PATH storage daemon configuration file [default:
- /etc/bacula/bacula-sd.conf]
- -o client=CLIENT file daemon name
- -o fileset=FILESET backup fileset
- -o device=DEVICE storage device name [default: FileStorage]
- -o datetime='YYYY-MM-DD hh:mm:ss'
- snapshot date/time [default: now]
- -o recent_job select contents of most recent job only [default:
- False]
- -o joblist='JOBID1 JOBID2 ...'
- select contents of specified list of jobs
- -o cleanup clean cache directory upon umount [default: False]
- -o move_root make absolute path symlinks point to path under
- mount point [default: False]
- -o prefetch_attrs read and parse attributes for all files upon
- filesystem initialization [default: False]
- -o prefetch_symlinks extract all symbolic links upon filesystem
- initialization (implies prefetch_attrs) [default:
- False]
- -o prefetch_regex=REGEX
- extract all objects that match REGEX upon
- filesystem initialization (implies prefetch_attrs)
- -o prefetch_recent extract contents of most recent non-full job upon
- filesystem initialization (implies
- prefetch_symlinks) [default: False]
- -o prefetch_diff=PATH extract files that do not match files at PATH
- (hint: speeds up rsync; implies prefetch_symlinks)
- -o prefetch_everything
- extract everything upon filesystem initialization
- (complete restore to cache) [default: False]
- -o user_cache_path=PATH
- user specified cache path (hint: combine this with
- one of the prefetch options) [default: none]
- -o logging=debug|info|warning|critical|error
- logging level [default: warning]
- -o syslog log to both syslog and console [default: False]
-
- FUSE options:
- -d -o debug enable debug output (implies -f)
- -f foreground operation
- -s disable multi-threaded operation
-
- -o allow_other allow access to other users
- -o allow_root allow access to root
- -o nonempty allow mounts over non-empty file/dir
- -o default_permissions enable permission checking by kernel
- -o fsname=NAME set filesystem name
- -o subtype=NAME set filesystem type
- -o large_read issue large read requests (2.4 only)
- -o max_read=N set maximum size of read requests
-
- -o hard_remove immediate removal (don't hide files)
- -o use_ino let filesystem set inode numbers
- -o readdir_ino try to fill in d_ino in readdir
- -o direct_io use direct I/O
- -o kernel_cache cache files in kernel
- -o [no]auto_cache enable caching based on modification times (off)
- -o umask=M set file permissions (octal)
- -o uid=N set file owner
- -o gid=N set file group
- -o entry_timeout=T cache timeout for names (1.0s)
- -o negative_timeout=T cache timeout for deleted names (0.0s)
- -o attr_timeout=T cache timeout for attributes (1.0s)
- -o ac_attr_timeout=T auto cache timeout for attributes (attr_timeout)
- -o intr allow requests to be interrupted
- -o intr_signal=NUM signal to send on interrupt (10)
- -o modules=M1[:M2...] names of modules to push onto filesystem stack
-
- -o max_write=N set maximum size of write requests
- -o max_readahead=N set maximum readahead
- -o async_read perform reads asynchronously (default)
- -o sync_read perform reads synchronously
- -o atomic_o_trunc enable atomic open+truncate support
- -o big_writes enable larger than 4kB writes
- -o no_remote_lock disable remote file locking
-
- Module options:
-
- [subdir]
- -o subdir=DIR prepend this directory to all paths (mandatory)
- -o [no]rellinks transform absolute symlinks to relative
-
- [iconv]
- -o from_code=CHARSET original encoding of file names (default: UTF-8)
- -o to_code=CHARSET new encoding of the file names (default: UTF-8)
-
+ baculafs [mountpoint] [options]
+
+ Options:
+ --version show program's version number and exit
+ -h, --help show this help message and exit
+ -o opt,[opt...] mount options
+ -o driver=mysql|postgresql|sqlite3
+ database driver [default: sqlite3]
+ -o host=HOST database server address [default: localhost]
+ -o port=PORT database server port
+ -o database=PATH database name [default: bacula]
+ -o username=USERNAME database user name [default: bacula]
+ -o password=PASSWORD database password (use '-o password= ' to get a
+ password prompt; if not provided, the password is
+ read from the DATABASE_PASSWORD environment
+ variable)
+ -o conf=PATH storage daemon configuration file [default:
+ /etc/bacula/bacula-sd.conf]
+ -o client=CLIENT file daemon name
+ -o fileset=FILESET backup fileset
+ -o device=DEVICE storage device name [default: FileStorage]
+ -o datetime='YYYY-MM-DD hh:mm:ss'
+ snapshot date/time [default: now]
+ -o recent_job select contents of most recent job only [default:
+ False]
+ -o joblist='JOBID1 JOBID2 ...'
+ select contents of specified list of jobs
+ -o cleanup clean cache directory upon umount [default: False]
+ -o move_root make absolute path symlinks point to path under
+ mount point [default: False]
+ -o prefetch_attrs read and parse attributes for all files upon
+ filesystem initialization [default: False]
+ -o prefetch_symlinks extract all symbolic links upon filesystem
+ initialization (implies prefetch_attrs) [default:
+ False]
+ -o prefetch_regex=REGEX
+ extract all objects that match REGEX upon
+ filesystem initialization (implies prefetch_attrs)
+ -o prefetch_recent extract contents of most recent non-full job upon
+ filesystem initialization (implies
+ prefetch_symlinks) [default: False]
+ -o prefetch_diff=PATH extract files that do not match files at PATH
+ (hint: speeds up rsync; implies prefetch_symlinks)
+ -o prefetch_difflist=DIFFLIST
+ extract files that do not match files in DIFFLIST
+ (list line format: 'Day Mon DD hh:mm:ss YYYY PATH';
+ use '-' to read from standard input; hint: format
+ matches output of 'duplicity list-current-files -v0
+ target_url'; implies prefetch_symlinks)
+ -o prefetch_list=LIST extract files that match files in LIST (list should
+ contains one absolute file path per line; use '-'
+ to read from standard input; implies
+ prefetch_symlinks)
+ -o prefetch_everything
+ extract everything upon filesystem initialization
+ (complete restore to cache) [default: False]
+ -o batch_list list files to be prefetched and exit [default:
+ False]
+ -o batch_bsr dump contnets of bsr file for extracting prefetched
+ files and exit [default: False]
+ -o batch_extract extract prefetched files to mount point and exit
+ [default: False]
+ -o user_cache_path=PATH
+ user specified cache path (hint: combine this with
+ one of the prefetch options) [default: none]
+ -o logging=debug|info|warning|critical|error
+ logging level [default: info]
+ -o syslog log to both syslog and console [default: False]
+
+ FUSE options:
+ -d -o debug enable debug output (implies -f)
+ -f foreground operation
+ -s disable multi-threaded operation
+
+ -o allow_other allow access to other users
+ -o allow_root allow access to root
+ -o nonempty allow mounts over non-empty file/dir
+ -o default_permissions enable permission checking by kernel
+ -o fsname=NAME set filesystem name
+ -o subtype=NAME set filesystem type
+ -o large_read issue large read requests (2.4 only)
+ -o max_read=N set maximum size of read requests
+
+ -o hard_remove immediate removal (don't hide files)
+ -o use_ino let filesystem set inode numbers
+ -o readdir_ino try to fill in d_ino in readdir
+ -o direct_io use direct I/O
+ -o kernel_cache cache files in kernel
+ -o [no]auto_cache enable caching based on modification times (off)
+ -o umask=M set file permissions (octal)
+ -o uid=N set file owner
+ -o gid=N set file group
+ -o entry_timeout=T cache timeout for names (1.0s)
+ -o negative_timeout=T cache timeout for deleted names (0.0s)
+ -o attr_timeout=T cache timeout for attributes (1.0s)
+ -o ac_attr_timeout=T auto cache timeout for attributes (attr_timeout)
+ -o intr allow requests to be interrupted
+ -o intr_signal=NUM signal to send on interrupt (10)
+ -o modules=M1[:M2...] names of modules to push onto filesystem stack
+
+ -o max_write=N set maximum size of write requests
+ -o max_readahead=N set maximum readahead
+ -o async_read perform reads asynchronously (default)
+ -o sync_read perform reads synchronously
+ -o atomic_o_trunc enable atomic open+truncate support
+ -o big_writes enable larger than 4kB writes
+ -o no_remote_lock disable remote file locking
+
+ Module options:
+
+ [subdir]
+ -o subdir=DIR prepend this directory to all paths (mandatory)
+ -o [no]rellinks transform absolute symlinks to relative
+
+ [iconv]
+ -o from_code=CHARSET original encoding of file names (default: UTF-8)
+ -o to_code=CHARSET new encoding of the file names (default: UTF-8)
+
- OPERATION
+ Operation
---------
- INITIALIZATION
+ Initialization
~~~~~~~~~~~~~~
**BaculaFS** starts by running several SQL queries against the Bacula
@@ -223,7 +245,7 @@
At this point the filesystem is ready.
- CACHE
+ Cache
~~~~~
Opening a file for reading causes **BaculaFS** to run ``bextract`` in
@@ -239,9 +261,9 @@
For example:
- use ``-o prefetch_attrs`` for storage space usage analysis
- (e.g. with Baobab_)
+ (e.g. with Baobab_)
- use ``-o prefetch_symlinks`` for any manual filesystem traversal
- with command line or GUI tools (``find``, ``mc``, etc.)
+ with command line or GUI tools (``find``, ``mc``, etc.)
- use ``-o prefetch_diff`` with ``rsync``
.. _Baobab: http://www.marzocca.net/linux/baobab/
@@ -250,7 +272,24 @@
filesystem, with ``-o cleanup``. It may also be reused between mount
operations with ``-o user_cache_path``.
- EXTENDED ATTRIBUTES
+ Batch Mode
+ ~~~~~~~~~~
+
+ **BaculaFS** may be used in *batch mode* with ``-o batch_extract`` in
+ order to extract files from the Bacula storage device and then exit
+ without mounting the filesystem. The mountpoint specified at the
+ command line is then treated as the destination directory for
+ extracted files.
+
+ The list of files, to be extracted in batch mode, is determined by the
+ various cache prefetch options, and may be dumped with ``-o
+ batch_list``.
+
+ The bootstrap file that is generated, in order to extract the files,
+ can also be dumped to standard output with ``-o batch_bsr``.
+
+
+ Extended Attributes
~~~~~~~~~~~~~~~~~~~
**BaculaFS** uses extended file attributes to expose Bacula specific
@@ -259,35 +298,39 @@
::
- user.baculafs.FileIndex
- user.baculafs.JobId
- user.baculafs.LStat
- user.baculafs.MD5
+ user.baculafs.FileIndex
+ user.baculafs.JobId
+ user.baculafs.LStat
+ user.baculafs.MD5
+
+ Note that ``user.baculafs.MD5`` shows whatever digest Bacula was
+ configured to calculate for the file, be it MD5, SHA1, SHA256 or
+ SHA512.
The root directory has several more attributes, that expose filesystem
instance-specific information:
::
- user.baculafs.cache_prefix
- user.baculafs.client
- user.baculafs.datetime
- user.baculafs.fileset
- user.baculafs.joblist
+ user.baculafs.cache_prefix
+ user.baculafs.client
+ user.baculafs.datetime
+ user.baculafs.fileset
+ user.baculafs.joblist
and several more attributes for monitoring the file extraction
process:
-
+
::
- user.baculafs.bextract.failures
- user.baculafs.bextract.path
- user.baculafs.bextract.pending
- user.baculafs.bextract.retries
- user.baculafs.bextract.state
- user.baculafs.bextract.volume
+ user.baculafs.bextract.failures
+ user.baculafs.bextract.path
+ user.baculafs.bextract.pending
+ user.baculafs.bextract.retries
+ user.baculafs.bextract.state
+ user.baculafs.bextract.volume
- MISSING VOLUMES
+ Missing Volumes
~~~~~~~~~~~~~~~
If the storage device is a tape drive then it's possible that
@@ -301,16 +344,98 @@
::
- attr -s baculafs.bextract.state -V run <mount-point>
+ attr -s baculafs.bextract.state -V run <mount-point>
Please note that this feature has undergone only rudimentary
testing. Expect breakage.
- LIMITATIONS
+ Examples
+ ~~~~~~~~
+
+ Mount the most recent backup snapshot for Bacula client ``client-fd``
+ and fileset ``client-fileset``; SQLite database backend (default):
+
+ ::
+
+ baculafs -o client=client-fd,fileset=client-fileset /path/to/mount/point
+
+ Mount the contents of the specified list of backup jobs; MySQL database
+ backend; prompt for password and prefetch to cache all symbolic links:
+
+ ::
+
+ baculafs -o joblist='1001 1003',client=client-fd,fileset=client-fileset \
+ -o driver=mysql,password=,prefetch_symlinks /path/to/mount/point
+
+ Mount the contents of the most recent backup job only:
+
+ ::
+
+ baculafs -o recent_job,client=client-fd,fileset=client-fileset /path/to/mount/point
+
+ Mount the contents of the fileset snapshot before the specified
+ date/time:
+
+ ::
+
+ baculafs -o datetime='2009-05-23 00:00:00' \
+ -o client=client-fd,fileset=client-fileset /path/to/mount/point
+
+ Allow other users to access filesystem, set logging level to ``debug``
+ and stay in foreground, so that ``bextract`` messages may be examined;
+ assume a single Bacula fileset is defined for client ``client-fd``:
+
+ ::
+
+ baculafs -f -o allow_other,client=client-fd,logging=debug /path/to/mount/point
+
+ Prefetch to cache files that do not exist, or whose modification time
+ or size differ from those at the specified directory (i.e. files that
+ would be sent to that directory using ``rsync``); and cleanup cache after
+ un-mounting:
+
+ ::
+
+ baculafs -o prefetch_diff=/path/to/rsync/target,cleanup \
+ -o client=client-fd /path/to/mount/point
+
+ Batch update a snapshot of the current files in backup (the filesystem
+ is *not* mounted):
+
+ ::
+
+ baculafs -o batch_extract,prefetch_diff=/path/to/snapshot,cleanup \
+ -o client=client-fd /path/to/snapshot/
+
+ (this is not as accurate as mounting the filesystem, like in the
+ previous example, and then updating the snapshot with ``rsync`` - but
+ it is faster).
+
+
+ Limitations
-----------
- LOCKING
+ Common Problems
+ ~~~~~~~~~~~~~~~
+
+ **BaculaFS** will abort if no backup job was run for the
+ client/fileset combination selected by the user.
+
+ **BaculaFS** will abort if the target mount point is in use. Note,
+ however, that mount point availability is checked by FUSE only *after*
+ the (potentially lengthy) initialization of **BaculaFS**.
+
+ Browsing a **BaculaFS** mount point with a file manager like Nautilus_
+ is liable to be very slow. This is because the file manager reads data
+ from each file being listed, in order to determine its type, generate
+ a thumbnail etc. This, in turn, triggers **BaculaFS** to extract the
+ files, one at a time, from the Bacula storage device to its cache
+ directory.
+
+ .. _Nautilus: http://live.gnome.org/Nautilus
+
+ Locking
~~~~~~~
Access to the storage device by different instances of **BaculaFS** is
@@ -323,28 +448,84 @@
should note:
- the lock is *advisory*, meaning that it does not prevent the Bacula
- storage daemon itself from accessing the storage device while in use
- by **BaculaFS**
+ storage daemon itself from accessing the storage device while in use
+ by **BaculaFS**
- depending on your setup, the lock may not work if the storage daemon
- configuration file is accessed via NFS
+ configuration file is accessed via NFS
+
+
+ Unsupported Features
+ ~~~~~~~~~~~~~~~~~~~~
+
+ **BaculaFS** depends on ``bextract`` and thus inherits its
+ limitations:
+
+ - **BaculaFS** can be used with Windows filesets, but it does not
+ reproduce any Windows specific file attributes
+ - Encrypted backup files are not supported
+ - **BaculaFS** must be started with enough permissions (typically as
+ ``root``) in order to allow ``bextract`` to extract files from the
+ Bacula storage
+ Changelog
+ ---------
+ **Version 0.1.7 (2010-12-30)**
+
+ - fixed: compatibility issues with bacula v2.4.4 and FUSE 7.8
+ - fixed: synthesize missing inode numbers with -o use_ino
+ - modified: decode value of user.baculafs.MD5 extended file attribute
+ - added: cache prefetch specified list of files
+ - added: changelog to README
+ - added: batch extract mode
+
+ **Version 0.1.6 (2010-09-19)**
+
+ - fixed cache prefetch by regex
+
+ **Version 0.1.5 (2010-07-06)**
+
+ - fixed: removed reference to obsolete db field Copy
+ - fixed: recent_job option with MySQL
+ - added: read database password from environment variable DATABASE_PASSWORD
+ - added: usage examples to README
+
+ **Version 0.1.4 (2010-02-07)**
+
+ - added: cache prefetch based on duplicity file listing
+ **Version 0.1.3 (2010-01-13)**
- WINDOWS FILESETS
- ~~~~~~~~~~~~~~~~
+ - fixed: (again) prefetch restore of files split between volumes
+ - fixed: missing import sys
+ - fixed: spurious linebreaks in debug log
- **BaculaFS** can be used with Windows backup filesets, but it does not
- reproduce any Windows specific file attributes. This is because
- ``bextract`` does not extract Windows specific file attributes on
- Linux.
+ **Version 0.1.2 (2010-01-13)**
- BUGS
+ - fixed: prefetch_recent when joblist contains a single non full job
+ - fixed: prefetch restore of files split between volumes
+ - added: copyright, trademark and license blurbs
+
+ **Version 0.1.1 (2010-01-07)**
+
+ - workaround: subtle extraction bug (fix forthcoming)
+
+ **Version 0.1.1 (2010-01-06)**
+
+ - initial public release
+
+ Bugs
----
Please report problems via the **BaculaFS** issue tracking system:
`<http://code.google.com/p/baculafs/issues/list>`_
- LICENSE
+ Credits
+ -------
+
+ **BaculaFS** contains SQL queries that were adapted from Bacula,
+ Copyright |(C)| 2000-2010 Free Software Foundation Europe e.V.
+
+ License
-------
**BaculaFS** is free software: you can redistribute it and/or modify
@@ -361,11 +542,14 @@
along with this program. If not, see
`<http://www.gnu.org/licenses/>`_.
+ .. [*] Bacula is a registered trademark of Kern Sibbald.
+ .. |(C)| unicode:: 0xA9 .. copyright sign
Platform: Linux
Classifier: Development Status :: 3 - Alpha
+Classifier: Topic :: System :: Filesystems
Classifier: Topic :: System :: Archiving :: Backup
Classifier: Intended Audience :: System Administrators
Classifier: Environment :: No Input/Output (Daemon)
|
[-]
[+]
|
Changed |
BaculaFS-0.1.7.tar.gz/BaculaFS.egg-info/SOURCES.txt
^
|
@@ -16,4 +16,5 @@
baculafs/LogFile.py
baculafs/SQL.py
baculafs/__init__.py
-baculafs/baculafs
\ No newline at end of file
+baculafs/baculafs
+baculafs/junk.py
\ No newline at end of file
|
[-]
[+]
|
Changed |
BaculaFS-0.1.7.tar.gz/PKG-INFO
^
|
@@ -1,6 +1,6 @@
Metadata-Version: 1.0
Name: BaculaFS
-Version: 0.1.0
+Version: 0.1.7
Summary: Bacula Filesystem in USErspace
Home-page: http://code.google.com/p/baculafs
Author: Avi Rozen
@@ -10,64 +10,67 @@
BaculaFS
========
- BaculaFS_ - Exposes the Bacula_ catalog and storage as a Filesystem in
+ BaculaFS_ - Exposes the Bacula_ [*]_ catalog and storage as a Filesystem in
USErspace (FUSE_).
.. _BaculaFS: http://code.google.com/p/baculafs
.. _Bacula: http://www.bacula.org
.. _FUSE: http://fuse.sourceforge.net/
- Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
+ Copyright |(C)| 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
- INTRODUCTION
+ .. contents::
+
+ Introduction
------------
- **BaculaFS** is a tool, independent of Bacula, that represents the
- Bacula catalog and backup storage media as a read-only filesystem in
- userspace.
+ **BaculaFS** is a tool, developed independently of Bacula, that
+ represents the Bacula catalog and backup storage media as a read-only
+ filesystem in userspace.
**BaculaFS** is specifically designed to cater for the following
use-cases:
- maintaining a remote snapshot of the files in the backup storage
- using `rsync`_
+ using `rsync`_ or `duplicity`_
- auditing the contents of backup jobs, without resorting to SQL
- queries
+ queries
- comparing backup jobs (using several mount points)
- Note that **BaculaFS** is a maintenance tool - it's operation may
+ Note that **BaculaFS** is a maintenance tool - its operation may
interfere with the normal operation of a live Bacula setup (see the
- LIMITATIONS_ section below).
+ Limitations_ section below).
.. _rsync: http://samba.anu.edu.au/rsync/
+ .. _duplicity: http://www.nongnu.org/duplicity/
- REQUIREMENTS
+ Requirements
------------
**BaculaFS** has been tested with the following set of required
software packages:
- + Bacula 3.0.2 with one of the following database backends:
-
- * SQLite_ 3.6.21
- * MySQL_ 5.4.1
- * PostgreSQL_ 8.3.9
+ + Bacula 5.0.2 with one of the following database backends:
+ * SQLite_ 3.7.3
+ * MySQL_ 5.1.49
+ * PostgreSQL_ 8.4.5
+
+ FUSE_ components:
- * Python FUSE 0.2
- * FUSE library 2.8.1
- * fusermount 2.8.1
- * FUSE kernel interface 7.12
-
- + Python_ 2.5.4, with the following additional libraries:
-
- * MySQLdb_ 1.2.2
- * psycopg2_ 2.0.13
- * pexpect_ 2.3
-
+ * Python FUSE 0.2.1
+ * FUSE library 2.8.4
+ * fusermount 2.8.4
+ * FUSE kernel interface 7.12
+
+ + Python_ 2.6.6, with the following additional libraries:
+
+ * MySQLdb_ 1.2.2
+ * psycopg2_ 2.2.1
+ * pexpect_ 2.3
+
+ attr_ extended attributes utilities 2.4.44
**BaculaFS** requires the following in order to function:
@@ -75,7 +78,7 @@
+ access to the database that's used to store the Bacula catalog
+ access to the Bacula storage device
+ access to the Bacula ``bextract`` utility (bundled with the Bacula
- storage daemon installation)
+ storage daemon installation)
.. _SQLite: http://www.sqlite.org/
.. _MySQL: http://www.mysql.com/
@@ -88,7 +91,7 @@
.. _attr: http://savannah.nongnu.org/projects/attr
- INSTALLATION
+ Installation
------------
Extract the source code archive to a temporary directory, ``cd`` to
@@ -96,117 +99,136 @@
::
- python setup.py install
+ python setup.py install
- USAGE
+ Usage
-----
::
- baculafs [mountpoint] [options]
-
- Options:
- --version show program's version number and exit
- -h, --help show this help message and exit
- -o opt,[opt...] mount options
- -o driver=mysql|postgresql|sqlite|sqlite3
- database driver [default: sqlite3]
- -o host=HOST database server address [default: localhost]
- -o port=PORT database server port
- -o database=PATH database name [default: bacula]
- -o username=USERNAME database user name [default: bacula]
- -o password=PASSWORD database password
- -o conf=PATH storage daemon configuration file [default:
- /etc/bacula/bacula-sd.conf]
- -o client=CLIENT file daemon name
- -o fileset=FILESET backup fileset
- -o device=DEVICE storage device name [default: FileStorage]
- -o datetime='YYYY-MM-DD hh:mm:ss'
- snapshot date/time [default: now]
- -o recent_job select contents of most recent job only [default:
- False]
- -o joblist='JOBID1 JOBID2 ...'
- select contents of specified list of jobs
- -o cleanup clean cache directory upon umount [default: False]
- -o move_root make absolute path symlinks point to path under
- mount point [default: False]
- -o prefetch_attrs read and parse attributes for all files upon
- filesystem initialization [default: False]
- -o prefetch_symlinks extract all symbolic links upon filesystem
- initialization (implies prefetch_attrs) [default:
- False]
- -o prefetch_regex=REGEX
- extract all objects that match REGEX upon
- filesystem initialization (implies prefetch_attrs)
- -o prefetch_recent extract contents of most recent non-full job upon
- filesystem initialization (implies
- prefetch_symlinks) [default: False]
- -o prefetch_diff=PATH extract files that do not match files at PATH
- (hint: speeds up rsync; implies prefetch_symlinks)
- -o prefetch_everything
- extract everything upon filesystem initialization
- (complete restore to cache) [default: False]
- -o user_cache_path=PATH
- user specified cache path (hint: combine this with
- one of the prefetch options) [default: none]
- -o logging=debug|info|warning|critical|error
- logging level [default: warning]
- -o syslog log to both syslog and console [default: False]
-
- FUSE options:
- -d -o debug enable debug output (implies -f)
- -f foreground operation
- -s disable multi-threaded operation
-
- -o allow_other allow access to other users
- -o allow_root allow access to root
- -o nonempty allow mounts over non-empty file/dir
- -o default_permissions enable permission checking by kernel
- -o fsname=NAME set filesystem name
- -o subtype=NAME set filesystem type
- -o large_read issue large read requests (2.4 only)
- -o max_read=N set maximum size of read requests
-
- -o hard_remove immediate removal (don't hide files)
- -o use_ino let filesystem set inode numbers
- -o readdir_ino try to fill in d_ino in readdir
- -o direct_io use direct I/O
- -o kernel_cache cache files in kernel
- -o [no]auto_cache enable caching based on modification times (off)
- -o umask=M set file permissions (octal)
- -o uid=N set file owner
- -o gid=N set file group
- -o entry_timeout=T cache timeout for names (1.0s)
- -o negative_timeout=T cache timeout for deleted names (0.0s)
- -o attr_timeout=T cache timeout for attributes (1.0s)
- -o ac_attr_timeout=T auto cache timeout for attributes (attr_timeout)
- -o intr allow requests to be interrupted
- -o intr_signal=NUM signal to send on interrupt (10)
- -o modules=M1[:M2...] names of modules to push onto filesystem stack
-
- -o max_write=N set maximum size of write requests
- -o max_readahead=N set maximum readahead
- -o async_read perform reads asynchronously (default)
- -o sync_read perform reads synchronously
- -o atomic_o_trunc enable atomic open+truncate support
- -o big_writes enable larger than 4kB writes
- -o no_remote_lock disable remote file locking
-
- Module options:
-
- [subdir]
- -o subdir=DIR prepend this directory to all paths (mandatory)
- -o [no]rellinks transform absolute symlinks to relative
-
- [iconv]
- -o from_code=CHARSET original encoding of file names (default: UTF-8)
- -o to_code=CHARSET new encoding of the file names (default: UTF-8)
-
+ baculafs [mountpoint] [options]
+
+ Options:
+ --version show program's version number and exit
+ -h, --help show this help message and exit
+ -o opt,[opt...] mount options
+ -o driver=mysql|postgresql|sqlite3
+ database driver [default: sqlite3]
+ -o host=HOST database server address [default: localhost]
+ -o port=PORT database server port
+ -o database=PATH database name [default: bacula]
+ -o username=USERNAME database user name [default: bacula]
+ -o password=PASSWORD database password (use '-o password= ' to get a
+ password prompt; if not provided, the password is
+ read from the DATABASE_PASSWORD environment
+ variable)
+ -o conf=PATH storage daemon configuration file [default:
+ /etc/bacula/bacula-sd.conf]
+ -o client=CLIENT file daemon name
+ -o fileset=FILESET backup fileset
+ -o device=DEVICE storage device name [default: FileStorage]
+ -o datetime='YYYY-MM-DD hh:mm:ss'
+ snapshot date/time [default: now]
+ -o recent_job select contents of most recent job only [default:
+ False]
+ -o joblist='JOBID1 JOBID2 ...'
+ select contents of specified list of jobs
+ -o cleanup clean cache directory upon umount [default: False]
+ -o move_root make absolute path symlinks point to path under
+ mount point [default: False]
+ -o prefetch_attrs read and parse attributes for all files upon
+ filesystem initialization [default: False]
+ -o prefetch_symlinks extract all symbolic links upon filesystem
+ initialization (implies prefetch_attrs) [default:
+ False]
+ -o prefetch_regex=REGEX
+ extract all objects that match REGEX upon
+ filesystem initialization (implies prefetch_attrs)
+ -o prefetch_recent extract contents of most recent non-full job upon
+ filesystem initialization (implies
+ prefetch_symlinks) [default: False]
+ -o prefetch_diff=PATH extract files that do not match files at PATH
+ (hint: speeds up rsync; implies prefetch_symlinks)
+ -o prefetch_difflist=DIFFLIST
+ extract files that do not match files in DIFFLIST
+ (list line format: 'Day Mon DD hh:mm:ss YYYY PATH';
+ use '-' to read from standard input; hint: format
+ matches output of 'duplicity list-current-files -v0
+ target_url'; implies prefetch_symlinks)
+ -o prefetch_list=LIST extract files that match files in LIST (list should
+ contains one absolute file path per line; use '-'
+ to read from standard input; implies
+ prefetch_symlinks)
+ -o prefetch_everything
+ extract everything upon filesystem initialization
+ (complete restore to cache) [default: False]
+ -o batch_list list files to be prefetched and exit [default:
+ False]
+ -o batch_bsr dump contnets of bsr file for extracting prefetched
+ files and exit [default: False]
+ -o batch_extract extract prefetched files to mount point and exit
+ [default: False]
+ -o user_cache_path=PATH
+ user specified cache path (hint: combine this with
+ one of the prefetch options) [default: none]
+ -o logging=debug|info|warning|critical|error
+ logging level [default: info]
+ -o syslog log to both syslog and console [default: False]
+
+ FUSE options:
+ -d -o debug enable debug output (implies -f)
+ -f foreground operation
+ -s disable multi-threaded operation
+
+ -o allow_other allow access to other users
+ -o allow_root allow access to root
+ -o nonempty allow mounts over non-empty file/dir
+ -o default_permissions enable permission checking by kernel
+ -o fsname=NAME set filesystem name
+ -o subtype=NAME set filesystem type
+ -o large_read issue large read requests (2.4 only)
+ -o max_read=N set maximum size of read requests
+
+ -o hard_remove immediate removal (don't hide files)
+ -o use_ino let filesystem set inode numbers
+ -o readdir_ino try to fill in d_ino in readdir
+ -o direct_io use direct I/O
+ -o kernel_cache cache files in kernel
+ -o [no]auto_cache enable caching based on modification times (off)
+ -o umask=M set file permissions (octal)
+ -o uid=N set file owner
+ -o gid=N set file group
+ -o entry_timeout=T cache timeout for names (1.0s)
+ -o negative_timeout=T cache timeout for deleted names (0.0s)
+ -o attr_timeout=T cache timeout for attributes (1.0s)
+ -o ac_attr_timeout=T auto cache timeout for attributes (attr_timeout)
+ -o intr allow requests to be interrupted
+ -o intr_signal=NUM signal to send on interrupt (10)
+ -o modules=M1[:M2...] names of modules to push onto filesystem stack
+
+ -o max_write=N set maximum size of write requests
+ -o max_readahead=N set maximum readahead
+ -o async_read perform reads asynchronously (default)
+ -o sync_read perform reads synchronously
+ -o atomic_o_trunc enable atomic open+truncate support
+ -o big_writes enable larger than 4kB writes
+ -o no_remote_lock disable remote file locking
+
+ Module options:
+
+ [subdir]
+ -o subdir=DIR prepend this directory to all paths (mandatory)
+ -o [no]rellinks transform absolute symlinks to relative
+
+ [iconv]
+ -o from_code=CHARSET original encoding of file names (default: UTF-8)
+ -o to_code=CHARSET new encoding of the file names (default: UTF-8)
+
- OPERATION
+ Operation
---------
- INITIALIZATION
+ Initialization
~~~~~~~~~~~~~~
**BaculaFS** starts by running several SQL queries against the Bacula
@@ -223,7 +245,7 @@
At this point the filesystem is ready.
- CACHE
+ Cache
~~~~~
Opening a file for reading causes **BaculaFS** to run ``bextract`` in
@@ -239,9 +261,9 @@
For example:
- use ``-o prefetch_attrs`` for storage space usage analysis
- (e.g. with Baobab_)
+ (e.g. with Baobab_)
- use ``-o prefetch_symlinks`` for any manual filesystem traversal
- with command line or GUI tools (``find``, ``mc``, etc.)
+ with command line or GUI tools (``find``, ``mc``, etc.)
- use ``-o prefetch_diff`` with ``rsync``
.. _Baobab: http://www.marzocca.net/linux/baobab/
@@ -250,7 +272,24 @@
filesystem, with ``-o cleanup``. It may also be reused between mount
operations with ``-o user_cache_path``.
- EXTENDED ATTRIBUTES
+ Batch Mode
+ ~~~~~~~~~~
+
+ **BaculaFS** may be used in *batch mode* with ``-o batch_extract`` in
+ order to extract files from the Bacula storage device and then exit
+ without mounting the filesystem. The mountpoint specified at the
+ command line is then treated as the destination directory for
+ extracted files.
+
+ The list of files, to be extracted in batch mode, is determined by the
+ various cache prefetch options, and may be dumped with ``-o
+ batch_list``.
+
+ The bootstrap file that is generated, in order to extract the files,
+ can also be dumped to standard output with ``-o batch_bsr``.
+
+
+ Extended Attributes
~~~~~~~~~~~~~~~~~~~
**BaculaFS** uses extended file attributes to expose Bacula specific
@@ -259,35 +298,39 @@
::
- user.baculafs.FileIndex
- user.baculafs.JobId
- user.baculafs.LStat
- user.baculafs.MD5
+ user.baculafs.FileIndex
+ user.baculafs.JobId
+ user.baculafs.LStat
+ user.baculafs.MD5
+
+ Note that ``user.baculafs.MD5`` shows whatever digest Bacula was
+ configured to calculate for the file, be it MD5, SHA1, SHA256 or
+ SHA512.
The root directory has several more attributes, that expose filesystem
instance-specific information:
::
- user.baculafs.cache_prefix
- user.baculafs.client
- user.baculafs.datetime
- user.baculafs.fileset
- user.baculafs.joblist
+ user.baculafs.cache_prefix
+ user.baculafs.client
+ user.baculafs.datetime
+ user.baculafs.fileset
+ user.baculafs.joblist
and several more attributes for monitoring the file extraction
process:
-
+
::
- user.baculafs.bextract.failures
- user.baculafs.bextract.path
- user.baculafs.bextract.pending
- user.baculafs.bextract.retries
- user.baculafs.bextract.state
- user.baculafs.bextract.volume
+ user.baculafs.bextract.failures
+ user.baculafs.bextract.path
+ user.baculafs.bextract.pending
+ user.baculafs.bextract.retries
+ user.baculafs.bextract.state
+ user.baculafs.bextract.volume
- MISSING VOLUMES
+ Missing Volumes
~~~~~~~~~~~~~~~
If the storage device is a tape drive then it's possible that
@@ -301,16 +344,98 @@
::
- attr -s baculafs.bextract.state -V run <mount-point>
+ attr -s baculafs.bextract.state -V run <mount-point>
Please note that this feature has undergone only rudimentary
testing. Expect breakage.
- LIMITATIONS
+ Examples
+ ~~~~~~~~
+
+ Mount the most recent backup snapshot for Bacula client ``client-fd``
+ and fileset ``client-fileset``; SQLite database backend (default):
+
+ ::
+
+ baculafs -o client=client-fd,fileset=client-fileset /path/to/mount/point
+
+ Mount the contents of the specified list of backup jobs; MySQL database
+ backend; prompt for password and prefetch to cache all symbolic links:
+
+ ::
+
+ baculafs -o joblist='1001 1003',client=client-fd,fileset=client-fileset \
+ -o driver=mysql,password=,prefetch_symlinks /path/to/mount/point
+
+ Mount the contents of the most recent backup job only:
+
+ ::
+
+ baculafs -o recent_job,client=client-fd,fileset=client-fileset /path/to/mount/point
+
+ Mount the contents of the fileset snapshot before the specified
+ date/time:
+
+ ::
+
+ baculafs -o datetime='2009-05-23 00:00:00' \
+ -o client=client-fd,fileset=client-fileset /path/to/mount/point
+
+ Allow other users to access filesystem, set logging level to ``debug``
+ and stay in foreground, so that ``bextract`` messages may be examined;
+ assume a single Bacula fileset is defined for client ``client-fd``:
+
+ ::
+
+ baculafs -f -o allow_other,client=client-fd,logging=debug /path/to/mount/point
+
+ Prefetch to cache files that do not exist, or whose modification time
+ or size differ from those at the specified directory (i.e. files that
+ would be sent to that directory using ``rsync``); and cleanup cache after
+ un-mounting:
+
+ ::
+
+ baculafs -o prefetch_diff=/path/to/rsync/target,cleanup \
+ -o client=client-fd /path/to/mount/point
+
+ Batch update a snapshot of the current files in backup (the filesystem
+ is *not* mounted):
+
+ ::
+
+ baculafs -o batch_extract,prefetch_diff=/path/to/snapshot,cleanup \
+ -o client=client-fd /path/to/snapshot/
+
+ (this is not as accurate as mounting the filesystem, like in the
+ previous example, and then updating the snapshot with ``rsync`` - but
+ it is faster).
+
+
+ Limitations
-----------
- LOCKING
+ Common Problems
+ ~~~~~~~~~~~~~~~
+
+ **BaculaFS** will abort if no backup job was run for the
+ client/fileset combination selected by the user.
+
+ **BaculaFS** will abort if the target mount point is in use. Note,
+ however, that mount point availability is checked by FUSE only *after*
+ the (potentially lengthy) initialization of **BaculaFS**.
+
+ Browsing a **BaculaFS** mount point with a file manager like Nautilus_
+ is liable to be very slow. This is because the file manager reads data
+ from each file being listed, in order to determine its type, generate
+ a thumbnail etc. This, in turn, triggers **BaculaFS** to extract the
+ files, one at a time, from the Bacula storage device to its cache
+ directory.
+
+ .. _Nautilus: http://live.gnome.org/Nautilus
+
+ Locking
~~~~~~~
Access to the storage device by different instances of **BaculaFS** is
@@ -323,28 +448,84 @@
should note:
- the lock is *advisory*, meaning that it does not prevent the Bacula
- storage daemon itself from accessing the storage device while in use
- by **BaculaFS**
+ storage daemon itself from accessing the storage device while in use
+ by **BaculaFS**
- depending on your setup, the lock may not work if the storage daemon
- configuration file is accessed via NFS
+ configuration file is accessed via NFS
+
+
+ Unsupported Features
+ ~~~~~~~~~~~~~~~~~~~~
+
+ **BaculaFS** depends on ``bextract`` and thus inherits its
+ limitations:
+
+ - **BaculaFS** can be used with Windows filesets, but it does not
+ reproduce any Windows specific file attributes
+ - Encrypted backup files are not supported
+ - **BaculaFS** must be started with enough permissions (typically as
+ ``root``) in order to allow ``bextract`` to extract files from the
+ Bacula storage
+ Changelog
+ ---------
+ **Version 0.1.7 (2010-12-30)**
+
+ - fixed: compatibility issues with bacula v2.4.4 and FUSE 7.8
+ - fixed: synthesize missing inode numbers with -o use_ino
+ - modified: decode value of user.baculafs.MD5 extended file attribute
+ - added: cache prefetch specified list of files
+ - added: changelog to README
+ - added: batch extract mode
+
+ **Version 0.1.6 (2010-09-19)**
+
+ - fixed cache prefetch by regex
+
+ **Version 0.1.5 (2010-07-06)**
+
+ - fixed: removed reference to obsolete db field Copy
+ - fixed: recent_job option with MySQL
+ - added: read database password from environment variable DATABASE_PASSWORD
+ - added: usage examples to README
+
+ **Version 0.1.4 (2010-02-07)**
+
+ - added: cache prefetch based on duplicity file listing
+ **Version 0.1.3 (2010-01-13)**
- WINDOWS FILESETS
- ~~~~~~~~~~~~~~~~
+ - fixed: (again) prefetch restore of files split between volumes
+ - fixed: missing import sys
+ - fixed: spurious linebreaks in debug log
- **BaculaFS** can be used with Windows backup filesets, but it does not
- reproduce any Windows specific file attributes. This is because
- ``bextract`` does not extract Windows specific file attributes on
- Linux.
+ **Version 0.1.2 (2010-01-13)**
- BUGS
+ - fixed: prefetch_recent when joblist contains a single non full job
+ - fixed: prefetch restore of files split between volumes
+ - added: copyright, trademark and license blurbs
+
+ **Version 0.1.1 (2010-01-07)**
+
+ - workaround: subtle extraction bug (fix forthcoming)
+
+ **Version 0.1.1 (2010-01-06)**
+
+ - initial public release
+
+ Bugs
----
Please report problems via the **BaculaFS** issue tracking system:
`<http://code.google.com/p/baculafs/issues/list>`_
- LICENSE
+ Credits
+ -------
+
+ **BaculaFS** contains SQL queries that were adapted from Bacula,
+ Copyright |(C)| 2000-2010 Free Software Foundation Europe e.V.
+
+ License
-------
**BaculaFS** is free software: you can redistribute it and/or modify
@@ -361,11 +542,14 @@
along with this program. If not, see
`<http://www.gnu.org/licenses/>`_.
+ .. [*] Bacula is a registered trademark of Kern Sibbald.
+ .. |(C)| unicode:: 0xA9 .. copyright sign
Platform: Linux
Classifier: Development Status :: 3 - Alpha
+Classifier: Topic :: System :: Filesystems
Classifier: Topic :: System :: Archiving :: Backup
Classifier: Intended Audience :: System Administrators
Classifier: Environment :: No Input/Output (Daemon)
|
[-]
[+]
|
Changed |
BaculaFS-0.1.7.tar.gz/README.rst
^
|
@@ -2,63 +2,66 @@
BaculaFS
========
-BaculaFS_ - Exposes the Bacula_ catalog and storage as a Filesystem in
+BaculaFS_ - Exposes the Bacula_ [*]_ catalog and storage as a Filesystem in
USErspace (FUSE_).
.. _BaculaFS: http://code.google.com/p/baculafs
.. _Bacula: http://www.bacula.org
.. _FUSE: http://fuse.sourceforge.net/
-Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
+Copyright |(C)| 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
-INTRODUCTION
+.. contents::
+
+Introduction
------------
-**BaculaFS** is a tool, independent of Bacula, that represents the
-Bacula catalog and backup storage media as a read-only filesystem in
-userspace.
+**BaculaFS** is a tool, developed independently of Bacula, that
+represents the Bacula catalog and backup storage media as a read-only
+filesystem in userspace.
**BaculaFS** is specifically designed to cater for the following
use-cases:
- maintaining a remote snapshot of the files in the backup storage
- using `rsync`_
+ using `rsync`_ or `duplicity`_
- auditing the contents of backup jobs, without resorting to SQL
queries
- comparing backup jobs (using several mount points)
-Note that **BaculaFS** is a maintenance tool - it's operation may
+Note that **BaculaFS** is a maintenance tool - its operation may
interfere with the normal operation of a live Bacula setup (see the
-LIMITATIONS_ section below).
+Limitations_ section below).
.. _rsync: http://samba.anu.edu.au/rsync/
+.. _duplicity: http://www.nongnu.org/duplicity/
-REQUIREMENTS
+Requirements
------------
**BaculaFS** has been tested with the following set of required
software packages:
-+ Bacula 3.0.2 with one of the following database backends:
++ Bacula 5.0.2 with one of the following database backends:
- * SQLite_ 3.6.21
- * MySQL_ 5.4.1
- * PostgreSQL_ 8.3.9
+ * SQLite_ 3.7.3
+ * MySQL_ 5.1.49
+ * PostgreSQL_ 8.4.5
+ FUSE_ components:
- * Python FUSE 0.2
- * FUSE library 2.8.1
- * fusermount 2.8.1
+ * Python FUSE 0.2.1
+ * FUSE library 2.8.4
+ * fusermount 2.8.4
* FUSE kernel interface 7.12
-+ Python_ 2.5.4, with the following additional libraries:
++ Python_ 2.6.6, with the following additional libraries:
* MySQLdb_ 1.2.2
- * psycopg2_ 2.0.13
- * pexpect_ 2.3
+ * psycopg2_ 2.2.1
+ * pexpect_ 2.3
+ attr_ extended attributes utilities 2.4.44
@@ -80,7 +83,7 @@
.. _attr: http://savannah.nongnu.org/projects/attr
-INSTALLATION
+Installation
------------
Extract the source code archive to a temporary directory, ``cd`` to
@@ -90,7 +93,7 @@
python setup.py install
-USAGE
+Usage
-----
::
@@ -101,13 +104,16 @@
--version show program's version number and exit
-h, --help show this help message and exit
-o opt,[opt...] mount options
- -o driver=mysql|postgresql|sqlite|sqlite3
+ -o driver=mysql|postgresql|sqlite3
database driver [default: sqlite3]
-o host=HOST database server address [default: localhost]
-o port=PORT database server port
-o database=PATH database name [default: bacula]
-o username=USERNAME database user name [default: bacula]
- -o password=PASSWORD database password
+ -o password=PASSWORD database password (use '-o password= ' to get a
+ password prompt; if not provided, the password is
+ read from the DATABASE_PASSWORD environment
+ variable)
-o conf=PATH storage daemon configuration file [default:
/etc/bacula/bacula-sd.conf]
-o client=CLIENT file daemon name
@@ -135,14 +141,30 @@
prefetch_symlinks) [default: False]
-o prefetch_diff=PATH extract files that do not match files at PATH
(hint: speeds up rsync; implies prefetch_symlinks)
+ -o prefetch_difflist=DIFFLIST
+ extract files that do not match files in DIFFLIST
+ (list line format: 'Day Mon DD hh:mm:ss YYYY PATH';
+ use '-' to read from standard input; hint: format
+ matches output of 'duplicity list-current-files -v0
+ target_url'; implies prefetch_symlinks)
+ -o prefetch_list=LIST extract files that match files in LIST (list should
+ contains one absolute file path per line; use '-'
+ to read from standard input; implies
+ prefetch_symlinks)
-o prefetch_everything
extract everything upon filesystem initialization
(complete restore to cache) [default: False]
+ -o batch_list list files to be prefetched and exit [default:
+ False]
+ -o batch_bsr dump contnets of bsr file for extracting prefetched
+ files and exit [default: False]
+ -o batch_extract extract prefetched files to mount point and exit
+ [default: False]
-o user_cache_path=PATH
user specified cache path (hint: combine this with
one of the prefetch options) [default: none]
-o logging=debug|info|warning|critical|error
- logging level [default: warning]
+ logging level [default: info]
-o syslog log to both syslog and console [default: False]
FUSE options:
@@ -195,10 +217,10 @@
-o to_code=CHARSET new encoding of the file names (default: UTF-8)
-OPERATION
+Operation
---------
-INITIALIZATION
+Initialization
~~~~~~~~~~~~~~
**BaculaFS** starts by running several SQL queries against the Bacula
@@ -215,7 +237,7 @@
At this point the filesystem is ready.
-CACHE
+Cache
~~~~~
Opening a file for reading causes **BaculaFS** to run ``bextract`` in
@@ -242,7 +264,24 @@
filesystem, with ``-o cleanup``. It may also be reused between mount
operations with ``-o user_cache_path``.
-EXTENDED ATTRIBUTES
+Batch Mode
+~~~~~~~~~~
+
+**BaculaFS** may be used in *batch mode* with ``-o batch_extract`` in
+order to extract files from the Bacula storage device and then exit
+without mounting the filesystem. The mountpoint specified at the
+command line is then treated as the destination directory for
+extracted files.
+
+The list of files, to be extracted in batch mode, is determined by the
+various cache prefetch options, and may be dumped with ``-o
+batch_list``.
+
+The bootstrap file that is generated, in order to extract the files,
+can also be dumped to standard output with ``-o batch_bsr``.
+
+
+Extended Attributes
~~~~~~~~~~~~~~~~~~~
**BaculaFS** uses extended file attributes to expose Bacula specific
@@ -256,6 +295,10 @@
user.baculafs.LStat
user.baculafs.MD5
+Note that ``user.baculafs.MD5`` shows whatever digest Bacula was
+configured to calculate for the file, be it MD5, SHA1, SHA256 or
+SHA512.
+
The root directory has several more attributes, that expose filesystem
instance-specific information:
@@ -279,7 +322,7 @@
user.baculafs.bextract.state
user.baculafs.bextract.volume
-MISSING VOLUMES
+Missing Volumes
~~~~~~~~~~~~~~~
If the storage device is a tape drive then it's possible that
@@ -299,10 +342,92 @@
testing. Expect breakage.
-LIMITATIONS
+Examples
+~~~~~~~~
+
+Mount the most recent backup snapshot for Bacula client ``client-fd``
+and fileset ``client-fileset``; SQLite database backend (default):
+
+::
+
+ baculafs -o client=client-fd,fileset=client-fileset /path/to/mount/point
+
+Mount the contents of the specified list of backup jobs; MySQL database
+backend; prompt for password and prefetch to cache all symbolic links:
+
+::
+
+ baculafs -o joblist='1001 1003',client=client-fd,fileset=client-fileset \
+ -o driver=mysql,password=,prefetch_symlinks /path/to/mount/point
+
+Mount the contents of the most recent backup job only:
+
+::
+
+ baculafs -o recent_job,client=client-fd,fileset=client-fileset /path/to/mount/point
+
+Mount the contents of the fileset snapshot before the specified
+date/time:
+
+::
+
+ baculafs -o datetime='2009-05-23 00:00:00' \
+ -o client=client-fd,fileset=client-fileset /path/to/mount/point
+
+Allow other users to access filesystem, set logging level to ``debug``
+and stay in foreground, so that ``bextract`` messages may be examined;
+assume a single Bacula fileset is defined for client ``client-fd``:
+
+::
+
+ baculafs -f -o allow_other,client=client-fd,logging=debug /path/to/mount/point
+
+Prefetch to cache files that do not exist, or whose modification time
+or size differ from those at the specified directory (i.e. files that
+would be sent to that directory using ``rsync``); and cleanup cache after
+un-mounting:
+
+::
+
+ baculafs -o prefetch_diff=/path/to/rsync/target,cleanup \
+ -o client=client-fd /path/to/mount/point
+
+Batch update a snapshot of the current files in backup (the filesystem
+is *not* mounted):
+
+::
+
+ baculafs -o batch_extract,prefetch_diff=/path/to/snapshot,cleanup \
+ -o client=client-fd /path/to/snapshot/
+
+(this is not as accurate as mounting the filesystem, like in the
+previous example, and then updating the snapshot with ``rsync`` - but
+it is faster).
+
+
+Limitations
-----------
-LOCKING
+Common Problems
+~~~~~~~~~~~~~~~
+
+**BaculaFS** will abort if no backup job was run for the
+client/fileset combination selected by the user.
+
+**BaculaFS** will abort if the target mount point is in use. Note,
+however, that mount point availability is checked by FUSE only *after*
+the (potentially lengthy) initialization of **BaculaFS**.
+
+Browsing a **BaculaFS** mount point with a file manager like Nautilus_
+is liable to be very slow. This is because the file manager reads data
+from each file being listed, in order to determine its type, generate
+a thumbnail etc. This, in turn, triggers **BaculaFS** to extract the
+files, one at a time, from the Bacula storage device to its cache
+directory.
+
+.. _Nautilus: http://live.gnome.org/Nautilus
+
+Locking
~~~~~~~
Access to the storage device by different instances of **BaculaFS** is
@@ -321,22 +446,78 @@
configuration file is accessed via NFS
+Unsupported Features
+~~~~~~~~~~~~~~~~~~~~
+
+**BaculaFS** depends on ``bextract`` and thus inherits its
+limitations:
+
+- **BaculaFS** can be used with Windows filesets, but it does not
+ reproduce any Windows specific file attributes
+- Encrypted backup files are not supported
+- **BaculaFS** must be started with enough permissions (typically as
+ ``root``) in order to allow ``bextract`` to extract files from the
+ Bacula storage
+
+Changelog
+---------
+**Version 0.1.7 (2010-12-30)**
+
+- fixed: compatibility issues with bacula v2.4.4 and FUSE 7.8
+- fixed: synthesize missing inode numbers with -o use_ino
+- modified: decode value of user.baculafs.MD5 extended file attribute
+- added: cache prefetch specified list of files
+- added: changelog to README
+- added: batch extract mode
-WINDOWS FILESETS
-~~~~~~~~~~~~~~~~
+**Version 0.1.6 (2010-09-19)**
-**BaculaFS** can be used with Windows backup filesets, but it does not
-reproduce any Windows specific file attributes. This is because
-``bextract`` does not extract Windows specific file attributes on
-Linux.
+- fixed cache prefetch by regex
-BUGS
+**Version 0.1.5 (2010-07-06)**
+
+- fixed: removed reference to obsolete db field Copy
+- fixed: recent_job option with MySQL
+- added: read database password from environment variable DATABASE_PASSWORD
+- added: usage examples to README
+
+**Version 0.1.4 (2010-02-07)**
+
+- added: cache prefetch based on duplicity file listing
+
+**Version 0.1.3 (2010-01-13)**
+
+- fixed: (again) prefetch restore of files split between volumes
+- fixed: missing import sys
+- fixed: spurious linebreaks in debug log
+
+**Version 0.1.2 (2010-01-13)**
+
+- fixed: prefetch_recent when joblist contains a single non full job
+- fixed: prefetch restore of files split between volumes
+- added: copyright, trademark and license blurbs
+
+**Version 0.1.1 (2010-01-07)**
+
+- workaround: subtle extraction bug (fix forthcoming)
+
+**Version 0.1.1 (2010-01-06)**
+
+- initial public release
+
+Bugs
----
Please report problems via the **BaculaFS** issue tracking system:
`<http://code.google.com/p/baculafs/issues/list>`_
-LICENSE
+Credits
+-------
+
+**BaculaFS** contains SQL queries that were adapted from Bacula,
+Copyright |(C)| 2000-2010 Free Software Foundation Europe e.V.
+
+License
-------
**BaculaFS** is free software: you can redistribute it and/or modify
@@ -353,5 +534,7 @@
along with this program. If not, see
`<http://www.gnu.org/licenses/>`_.
+.. [*] Bacula is a registered trademark of Kern Sibbald.
+.. |(C)| unicode:: 0xA9 .. copyright sign
|
[-]
[+]
|
Changed |
BaculaFS-0.1.7.tar.gz/baculafs/Base64.py
^
|
@@ -1,3 +1,27 @@
+
+# BaculaFS - Bacula Filesystem in USErspace
+# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
+#
+# BaculaFS contains SQL queries that were adapted from Bacula
+# Copyright (C) 2000-2010 Free Software Foundation Europe e.V.
+#
+# This file is part of BaculaFS.
+#
+# BaculaFS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+# Bacula is a registered trademark of Kern Sibbald.
+
class Base64 :
'''
Bacula specific implementation of a base64 decoder
|
[-]
[+]
|
Changed |
BaculaFS-0.1.7.tar.gz/baculafs/Catalog.py
^
|
@@ -1,3 +1,27 @@
+
+# BaculaFS - Bacula Filesystem in USErspace
+# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
+#
+# BaculaFS contains SQL queries that were adapted from Bacula
+# Copyright (C) 2000-2010 Free Software Foundation Europe e.V.
+#
+# This file is part of BaculaFS.
+#
+# BaculaFS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+# Bacula is a registered trademark of Kern Sibbald.
+
from datetime import datetime
from SQL import *
@@ -74,7 +98,7 @@
diff_jobs = self.db.query(SQL.temp)
self.db.query(SQL.incr_jobs_temp % (diff_jobs[-1][1], self.datetime, self.client_id, self.fileset[1]), fetch=False)
jobs = self.db.query(SQL.jobs)
- self.most_recent_jobid = jobs[-1][0] if len(jobs) > 1 else -1
+ self.most_recent_jobid = jobs[-1][0] if jobs[-1][1] != 'F' else -1
# select files from the most recent job only
if select_recent_job and self.most_recent_jobid > 0 :
jobs = [jobs[-1]]
|
[-]
[+]
|
Changed |
BaculaFS-0.1.7.tar.gz/baculafs/Database.py
^
|
@@ -1,4 +1,29 @@
+
+# BaculaFS - Bacula Filesystem in USErspace
+# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
+#
+# BaculaFS contains SQL queries that were adapted from Bacula
+# Copyright (C) 2000-2010 Free Software Foundation Europe e.V.
+#
+# This file is part of BaculaFS.
+#
+# BaculaFS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+# Bacula is a registered trademark of Kern Sibbald.
+
import os
+from getpass import getpass
from Base64 import *
from SQL import *
@@ -8,12 +33,11 @@
This class shields the rest of the code from the pesky details of
actually accessing one of the supported databases.
'''
- drivers = [SQL.MYSQL, SQL.POSTGRESQL, SQL.SQLITE, SQL.SQLITE3]
+ drivers = [SQL.MYSQL, SQL.POSTGRESQL, SQL.SQLITE3]
default_database = {
SQL.MYSQL: 'bacula',
SQL.POSTGRESQL: 'bacula',
- SQL.SQLITE: '/var/lib/bacula/bacula.db',
SQL.SQLITE3: '/var/lib/bacula/bacula.db'
}
@@ -28,18 +52,21 @@
self.driver = driver
if not database :
database = Database.default_database[self.driver]
+ # user supplied empty password -> prompt
+ if password == '' :
+ password = getpass()
+ # user supplied no password -> query environment or use empty string
+ if password == None :
+ try:
+ password = os.environ['DATABASE_PASSWORD']
+ except KeyError:
+ password = ''
if self.driver == SQL.MYSQL :
from MySQLdb import connect
self.connection = connect(host=host, port=port, user=username, passwd=password, db=database)
elif self.driver == SQL.POSTGRESQL :
from psycopg2 import connect
self.connection = connect(host=host, port=port, user=username, password=password, database=database)
- elif self.driver == SQL.SQLITE :
- from sqlite import connect
- database = os.path.expanduser(database)
- if not os.path.isfile(database) or not os.access(database, os.R_OK) :
- raise RuntimeError, 'cannot read from file %s' % database
- self.connection = connect(database)
elif self.driver == SQL.SQLITE3 :
from sqlite3 import connect
database = os.path.expanduser(database)
@@ -64,10 +91,10 @@
def query(self, sql, fetch = True) :
'''
- Execute SQL and fetch all results
+ Execute SQL and fetch all results as a list
'''
self.logger.debug(sql)
self.cursor.execute(sql)
if fetch :
- return self.cursor.fetchall()
+ return list(self.cursor.fetchall())
|
[-]
[+]
|
Changed |
BaculaFS-0.1.7.tar.gz/baculafs/FileSystem.py
^
|
@@ -1,6 +1,31 @@
-__version__ = '0.1.0'
+
+# BaculaFS - Bacula Filesystem in USErspace
+# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
+#
+# BaculaFS contains SQL queries that were adapted from Bacula
+# Copyright (C) 2000-2010 Free Software Foundation Europe e.V.
+#
+# This file is part of BaculaFS.
+#
+# BaculaFS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+# Bacula is a registered trademark of Kern Sibbald.
+
+__version__ = '0.1.7'
import os
+import sys
import stat
import errno
import copy
@@ -10,10 +35,14 @@
import traceback
import pexpect
import fcntl
+import time
+import re
+import binascii
from LogFile import *
from Database import *
from Catalog import *
+from SQL import *
# pull in some spaghetti to make this stuff work without fuse-py being installed
try:
@@ -71,7 +100,18 @@
class FileSystem(Fuse) :
- null_stat = fuse.Stat(st_mode = stat.S_IFDIR | 0755, st_nlink = 2, st_ino = -1)
+ null_stat = fuse.Stat(st_mode=stat.S_IFDIR | 0755,
+ st_ino=0,
+ st_dev=0,
+ st_nlink=2,
+ st_uid=0,
+ st_gid=0,
+ st_size=0,
+ st_atime=0,
+ st_mtime=0,
+ st_ctime=0,
+ st_blksize=0,
+ st_rdev=0)
bacula_stat_fields = ['st_dev',
'st_ino',
@@ -90,7 +130,7 @@
'st_flags',
'st_streamid']
- fuse_stat_fields = dir(fuse.Stat())
+ fuse_stat_fields = [attr for attr in dir(null_stat) if attr.startswith('st_')]
xattr_prefix = 'user.baculafs.'
xattr_fields = ['FileIndex', 'JobId', 'LStat', 'MD5']
@@ -115,14 +155,14 @@
self._initialized = False
# default option values
- self.logging = 'warning'
+ self.logging = 'info'
self.syslog = False
- self.driver = 'sqlite3'
+ self.driver = SQL.SQLITE3
self.database = None
self.host = 'localhost'
self.port = 0
self.username = 'bacula'
- self.password = ''
+ self.password = None
self.conf = '/etc/bacula/bacula-sd.conf'
self.client = ''
self.fileset = None
@@ -130,6 +170,7 @@
self.datetime = None
self.recent_job = False
self.joblist = None
+ self.cache_prefix = None
self.user_cache_path = None
self.cleanup = False
self.move_root = False
@@ -138,7 +179,15 @@
self.prefetch_symlinks = False
self.prefetch_recent = False
self.prefetch_diff = None
+ self.prefetch_difflist = None
+ self.prefetch_list = None
self.prefetch_everything = False
+ self.batch_mode = False
+ self.batch_list = False
+ self.batch_bsr = False
+ self.batch_extract = False
+ self.use_ino = False
+ self.max_ino = 0
self.dirs = { '/': { '': (FileSystem.null_stat,) } }
self._bextract_status = copy.deepcopy(FileSystem.bextract_done)
@@ -188,6 +237,19 @@
self.dirs[head][tail] = (FileSystem.null_stat,)
self._add_parent_dirs(head)
+ def _update_inodes(self, head) :
+ '''
+ generate unique st_ino for each missing st_ino
+ '''
+ for tail in self.dirs[head] :
+ if self.dirs[head][tail][-1].st_ino == 0 :
+ if len(self.dirs[head][tail]) == 1:
+ self.dirs[head][tail] = (copy.deepcopy(FileSystem.null_stat),)
+ self.max_ino += 1
+ self.dirs[head][tail][-1].st_ino = self.max_ino
+ subdir = '%s%s/' % (head, tail)
+ if subdir in self.dirs :
+ self._update_inodes(subdir)
def _extract(self, path_list) :
'''
@@ -227,9 +289,7 @@
rc, sig = self._bextract(items)
# it seems that bextract does not restore mtime for symlinks
# so we create a normal file with same mtime as stored symlink
- # (note that we only use that file if the cache path was
- # supplied by the user)
- if rc == 0 :
+ if rc == 0 and not self.batch_mode :
for item in items :
if item[0] :
symlinkfile = item[0][0]
@@ -279,19 +339,14 @@
is_symlink = stat.S_ISLNK(bs.st_mode)
found = False
if os.path.exists(realpath) or os.path.lexists(realpath) :
- # do not trust user supplied cache path:
# make sure that stat info of realpath matches path
- if self.user_cache_path :
- s = os.lstat(realpath)
- conds = [getattr(s, attr) == getattr(bs, attr)
- for attr in ['st_mode', 'st_uid', 'st_gid', 'st_size', 'st_mtime']]
- if is_symlink :
- conds[-1] = (os.path.exists(symlinkpath) and
- bs.st_mtime == os.stat(symlinkpath).st_mtime)
- found = all(conds)
- else :
- found = True
- if found :
+ s = os.lstat(realpath)
+ conds = [getattr(s, attr) == getattr(bs, attr)
+ for attr in ['st_mode', 'st_uid', 'st_gid', 'st_size', 'st_mtime']]
+ if is_symlink :
+ conds[-1] = (os.path.exists(symlinkpath) and
+ bs.st_mtime == os.stat(symlinkpath).st_mtime)
+ if all(conds) :
return realpath, None, None
# generate list of volumes for path
fileindex, jobid = self.dirs[head][tail][0:2]
@@ -300,11 +355,12 @@
volumes = [[volume[1], # 0-Volume
volume[2], # 1-MediaType
self.device, # 2-Device
- jobs[0][1], # 3-VolSessionId
- jobs[0][2], # 4-VolSessionTime
- (volume[5] << 32) | volume[7], # 5-VolAddr: StartAddr
- (volume[6] << 32) | volume[8], # 6-VolAddr: EndAddr
- fileindex] # 7-FileIndex
+ jobs[0][0], # 3-JobId
+ jobs[0][1], # 4-VolSessionId
+ jobs[0][2], # 5-VolSessionTime
+ (volume[5] << 32) | volume[7], # 6-VolAddr: StartAddr
+ (volume[6] << 32) | volume[8], # 7-VolAddr: EndAddr
+ fileindex] # 8-FileIndex
for volume in self.catalog.volumes
if (volume[0] == jobid and
volume[3] <= fileindex and
@@ -369,8 +425,31 @@
'''
extract list of items from Bacula storage device
'''
+ if self.batch_list :
+ for item in items :
+ print item[1]
+ if (not self.batch_bsr and
+ not self.batch_extract) :
+ return (0, 0)
+
bsrpath = self._write_bsr(items)
- cmd = 'bextract -b "%s" -c "%s" "%s" "%s"' % (bsrpath, self.conf, self.device, self.cache_path)
+
+ if self.batch_bsr :
+ bsrfile = open(bsrpath, 'rt')
+ for line in bsrfile :
+ sys.stdout.write(line)
+ sys.stdout.flush()
+ bsrfile.close()
+ if not self.batch_extract :
+ return (0, 0)
+
+ if self.batch_extract :
+ makedirs(self.fuse_args.mountpoint)
+
+ cmd = 'bextract -b "%s" -c "%s" "%s" "%s"' % (bsrpath,
+ self.conf,
+ self.device,
+ self.cache_path if not self.batch_extract else self.fuse_args.mountpoint)
self.logger.debug(cmd)
self._bextract_set_status({'path': items[0][1],
@@ -393,6 +472,7 @@
i = child.expect([self.fail_pattern, pexpect.EOF],
timeout=None,
searchwindowsize=200)
+ self.logfile.flush(flush_tail = True)
if i == 0 :
# count retries
if missing == child.match.groups()[0] :
@@ -434,21 +514,69 @@
self._bextract_increment_counter('failures', 1)
return (child.exitstatus, child.signalstatus)
+ def _group_by_volume(self, items) :
+ '''
+ return items grouped by volume
+ '''
+ # group volumes
+ volumes = []
+ for item in items :
+ for v in item[-1] :
+ found = False
+ findex = v[-1]
+ for vindex in xrange(0,len(volumes)) :
+ volume = volumes[vindex]
+ if not any(map(cmp, v[:-1], volume[:-1])) :
+ volume[-1].append(findex)
+ found = True
+ break
+ if not found :
+ volumes.append(v[:-1] + [[v[-1]]])
+
+ # compact list of file indices
+ for volume in volumes :
+ volume[-1] = list(set(volume[-1]))
+ volume[-1].sort()
+ l = len(volume[-1])
+ findex = volume[-1][0]
+ findices = [(findex, findex)]
+ for idx in volume[-1][1:] :
+ next_idx = findices[-1][-1] + 1
+ if idx == next_idx :
+ findices[-1] = (findices[-1][0], idx)
+ else :
+ findices.append((idx,idx))
+ volume[-1] = findices
+ volume.append(l)
+
+ # reorder volumes to ensure correct handling of
+ # files spanning multiple volumes
+ volumes.sort(cmp = lambda a,b : \
+ (cmp(a[3],b[3]) or
+ cmp(a[8][-1][-1], b[8][0][0])))
+
+ return volumes
+
def _write_bsr(self, items) :
'''
generate bsr for items to be extracted
'''
bsrfd, bsrpath = tempfile.mkstemp(suffix='.bsr', dir=self.cache_bsrpath, text=True)
- for item in items :
- for volume in item[-1] :
- os.write(bsrfd, 'Volume="%s"\n' % volume[0])
- os.write(bsrfd, 'MediaType="%s"\n' % volume[1])
- os.write(bsrfd, 'Device="%s"\n' % volume[2])
- os.write(bsrfd, 'VolSessionId=%d\n' % volume[3])
- os.write(bsrfd, 'VolSessionTime=%d\n' % volume[4])
- os.write(bsrfd, 'VolAddr=%d-%d\n' % (volume[5],volume[6]))
- os.write(bsrfd, 'FileIndex=%d\n' % volume[7])
- os.write(bsrfd, 'Count=1\n')
+ volumes = self._group_by_volume(items)
+ for volume in volumes :
+ os.write(bsrfd, 'Volume="%s"\n' % volume[0])
+ os.write(bsrfd, 'MediaType="%s"\n' % volume[1])
+ os.write(bsrfd, 'Device="%s"\n' % volume[2])
+ os.write(bsrfd, 'VolSessionId=%d\n' % volume[4])
+ os.write(bsrfd, 'VolSessionTime=%d\n' % volume[5])
+ if not self.bsr_compat :
+ os.write(bsrfd, 'VolAddr=%d-%d\n' % (volume[6],volume[7]))
+ for findex in volume[8] :
+ if findex[0] == findex[1] :
+ os.write(bsrfd, 'FileIndex=%d\n' % findex[0])
+ else :
+ os.write(bsrfd, 'FileIndex=%d-%d\n' % findex)
+ os.write(bsrfd, 'Count=%d\n' % volume[9])
os.close(bsrfd)
return bsrpath
@@ -489,14 +617,22 @@
self.logger.warning(traceback.format_exc())
self.logfile = LogFile(self.logger, logging.DEBUG)
-
- def initialize(self):
+ def initialize(self, version) :
'''
initialize database, catalog
'''
self._setup_logging()
+ # batch mode
+ self.batch_mode = (self.batch_list or
+ self.batch_bsr or
+ self.batch_extract)
+ # disable INFO level logging in batch mode
+ if self.batch_mode and self.loglevel == logging.INFO :
+ self.loglevel = logging.WARNING
+ self.logger.setLevel(self.loglevel)
+
self.logger.info('Populating file system ... ')
# setup cache
@@ -511,6 +647,10 @@
self.cache_symlinks = os.path.normpath(self.cache_prefix + '/symlinks')
makedirs(self.cache_symlinks)
+ # test for old version (2.x) of bacula
+ self.bsr_compat = int(version[0]) < 3
+ if self.bsr_compat :
+ self.logger.debug('Detected old Bacula: %s' % version)
# test access to sd conf file
open(self.conf, 'r').close()
# init bextract failure pattren
@@ -534,12 +674,15 @@
self.db.close()
prefetches = []
+ difflist = {}
# validate prefetch conditions
if self.prefetch_everything :
self.prefetch_recent = False
self.prefetch_regex = None
self.prefetch_diff = None
+ self.prefetch_difflist = None
+ self.prefetch_list = None
self.prefetch_symlinks = True
if self.prefetch_regex :
try :
@@ -560,10 +703,37 @@
# can't access target directory: show traceback and ignore
self.logger.warning(traceback.format_exc())
self.prefetch_diff = None
+ if self.prefetch_difflist :
+ self.prefetch_difflist = os.path.normpath(os.path.expanduser(self.prefetch_difflist))
+ try :
+ difflistfile = sys.stdin if self.prefetch_difflist == '-' else open(self.prefetch_difflist, 'rt')
+ for line in difflistfile.readlines():
+ date = ' '.join(line.split()[:5])
+ difflist[line[(len(date) + 1):].strip()] = time.strptime(date, '%a %b %d %H:%M:%S %Y')
+ difflistfile.close()
+ self.prefetch_symlinks = True
+ except :
+ # can't access/parse difflist: show traceback and ignore
+ self.logger.warning(traceback.format_exc())
+ self.prefetch_difflist = None
+ if self.prefetch_list :
+ self.prefetch_list = os.path.normpath(os.path.expanduser(self.prefetch_list))
+ try :
+ listfile = sys.stdin if self.prefetch_list == '-' else open(self.prefetch_list, 'rt')
+ matchlist = [line.strip() for line in listfile.readlines()]
+ listfile.close()
+ self.prefetch_symlinks = True
+ except :
+ # can't access/parse list: show traceback and ignore
+ self.logger.warning(traceback.format_exc())
+ self.prefetch_list = None
if self.prefetch_recent :
self.prefetch_symlinks = True
if self.prefetch_symlinks :
self.prefetch_attrs = True
+ if 'use_ino' in self.fuse_args.optlist:
+ self.use_ino = True
+ self.prefetch_attrs = True # must figure out max st_ino
for file in files :
head = file[0]
@@ -574,6 +744,10 @@
# make file entry
if self.prefetch_attrs :
entry = file[2:] + self._bacula_stat(file[-2])
+ # find max st_ino
+ if self.use_ino:
+ if entry[-1].st_ino > self.max_ino :
+ self.max_ino = entry[-1].st_ino
# detemine if we need to prefetch this entry
filepath = head + tail
if (not stat.S_ISDIR(entry[-1].st_mode) and
@@ -581,9 +755,14 @@
(self.prefetch_recent and
file[3] == self.catalog.most_recent_jobid) or
(self.prefetch_regex and
- regex.match(filepath)) or
+ regex.search(filepath)) or
(self.prefetch_diff and
- not self._match_stat(self.prefetch_diff + filepath, entry[-1])) or
+ not self._match_stat(self.prefetch_diff + filepath, entry[-1])) or
+ (self.prefetch_difflist and
+ (filepath[1:] not in difflist or
+ difflist[filepath[1:]][:-1] != time.localtime(entry[-1].st_mtime)[:-1])) or
+ (self.prefetch_list and
+ filepath in matchlist) or
(self.prefetch_symlinks and
stat.S_ISLNK(entry[-1].st_mode)))) :
prefetches.append(filepath)
@@ -600,13 +779,17 @@
# and finally
self.dirs[head][tail] = entry
+ # fix st_ino
+ if self.use_ino:
+ self._update_inodes('/')
+
npf = len(prefetches)
if npf > 0 :
self.logger.info('Prefetching %d objects ... ' % npf)
self._extract(prefetches)
- self.logger.info('Cache directory is: %s' % self.cache_prefix)
+ self.logger.debug('Cache directory is: %s' % self.cache_prefix)
self.joblist = ' '.join([str(job[0]) for job in self.catalog.jobs])
- self.logger.info('Jobs in file system: %s' % self.joblist)
+ self.logger.debug('Job ids in file system: %s' % self.joblist)
self.logger.info('BaculaFS ready (%d files).' % len(files))
self._initialized = True
@@ -615,7 +798,7 @@
'''
remove cache directory if required
'''
- if self.cleanup and not self.user_cache_path :
+ if self.cleanup and not self.user_cache_path and self.cache_prefix :
self.logger.info('removing cache directory: %s' % self.cache_prefix)
shutil.rmtree(self.cache_prefix, ignore_errors = True)
@@ -653,6 +836,9 @@
len(self.dirs[head][tail]) != 1 and
n in FileSystem.xattr_fields) :
val = str(self.dirs[head][tail][FileSystem.xattr_fields.index(n)])
+ if n == 'MD5' and val != '0':
+ l = len(val)
+ val = binascii.b2a_hex(binascii.a2b_base64(val+'='*((l*3+8)/3-l)+'\n')) # padding
# attribute not found
if val == None :
return -errno.ENODATA
@@ -719,8 +905,11 @@
yield fuse.Direntry(key)
for key in self.dirs[path].keys() :
if len(key) > 0:
- bs = self.getattr(path + key)
- ino = bs.st_ino if bs.st_ino != 0 else -1
+ if self.use_ino:
+ bs = self.getattr(path + key)
+ ino = bs.st_ino
+ else :
+ ino = 0
yield fuse.Direntry(key, ino=ino)
def readlink(self, path):
@@ -797,7 +986,7 @@
server.parser.add_option(mountopt="username", metavar="USERNAME", default=server.username,
help="database user name [default: %default]")
server.parser.add_option(mountopt="password", metavar="PASSWORD", default=server.password,
- help="database password")
+ help="database password (use '-o password= ' to get a password prompt; if not provided, the password is read from the DATABASE_PASSWORD environment variable)")
server.parser.add_option(mountopt="conf", metavar="PATH", default=server.conf,
help="storage daemon configuration file [default: %default]")
server.parser.add_option(mountopt="client", metavar="CLIENT", default=server.client,
@@ -826,8 +1015,18 @@
help="extract contents of most recent non-full job upon filesystem initialization (implies prefetch_symlinks) [default: %default]")
server.parser.add_option(mountopt="prefetch_diff", metavar="PATH", default=server.prefetch_diff,
help="extract files that do not match files at PATH (hint: speeds up rsync; implies prefetch_symlinks)")
+ server.parser.add_option(mountopt="prefetch_difflist", metavar="DIFFLIST", default=server.prefetch_difflist,
+ help="extract files that do not match files in DIFFLIST (list line format: 'Day Mon DD hh:mm:ss YYYY PATH'; use '-' to read from standard input; hint: format matches output of 'duplicity list-current-files -v0 target_url'; implies prefetch_symlinks)")
+ server.parser.add_option(mountopt="prefetch_list", metavar="LIST", default=server.prefetch_list,
+ help="extract files that match files in LIST (list should contains one absolute file path per line; use '-' to read from standard input; implies prefetch_symlinks)")
server.parser.add_option(mountopt="prefetch_everything", action="store_true", default=server.prefetch_everything,
help="extract everything upon filesystem initialization (complete restore to cache) [default: %default]")
+ server.parser.add_option(mountopt="batch_list", action="store_true", default=server.batch_list,
+ help="list files to be prefetched and exit [default: %default]")
+ server.parser.add_option(mountopt="batch_bsr", action="store_true", default=server.batch_bsr,
+ help="dump contnets of bsr file for extracting prefetched files and exit [default: %default]")
+ server.parser.add_option(mountopt="batch_extract", action="store_true", default=server.batch_extract,
+ help="extract prefetched files to mount point and exit [default: %default]")
server.parser.add_option(mountopt="user_cache_path", metavar="PATH", default=server.user_cache_path,
help="user specified cache path (hint: combine this with one of the prefetch options) [default: %default]")
server.parser.add_option(mountopt="logging", choices=LOGGING_LEVELS.keys(), metavar='|'.join(LOGGING_LEVELS.keys()), default=server.logging,
@@ -844,13 +1043,13 @@
# we initialize before main (i.e. not in fsinit) so that
# any failure here aborts the mount
try :
- server.initialize()
+ server.initialize(bacula_version)
except :
- traceback.print_exc()
server.shutdown()
raise
- server.main()
+ if not server.batch_mode :
+ server.main()
# we shutdown after main, i.e. not in fsshutdown, because
# calling fsshutdown with multithreaded==True seems to cause
|
[-]
[+]
|
Changed |
BaculaFS-0.1.7.tar.gz/baculafs/LogFile.py
^
|
@@ -1,3 +1,27 @@
+
+# BaculaFS - Bacula Filesystem in USErspace
+# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
+#
+# BaculaFS contains SQL queries that were adapted from Bacula
+# Copyright (C) 2000-2010 Free Software Foundation Europe e.V.
+#
+# This file is part of BaculaFS.
+#
+# BaculaFS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+# Bacula is a registered trademark of Kern Sibbald.
+
import logging
import logging.handlers
@@ -14,12 +38,25 @@
def __init__(self, logger, level) :
self.logger = logger
self.level = level
+ self.tail = ''
def write(self, message) :
- for line in message.splitlines() :
+ lines = (self.tail + message).splitlines()
+ for line in lines[:-1] :
self.logger.log(self.level, line)
+ if message.endswith('\n') :
+ self.logger.log(self.level, lines[-1])
+ self.tail = ''
+ elif len(lines) > 0 :
+ self.tail = lines[-1]
+ else :
+ self.tail = ''
+
- def flush(self) :
+ def flush(self, flush_tail = False) :
+ if flush_tail and self.tail :
+ self.logger.log(self.level, self.tail)
+ self.tail = ''
for handler in self.logger.handlers :
handler.flush()
|
[-]
[+]
|
Changed |
BaculaFS-0.1.7.tar.gz/baculafs/SQL.py
^
|
@@ -1,3 +1,27 @@
+
+# BaculaFS - Bacula Filesystem in USErspace
+# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
+#
+# BaculaFS contains SQL queries that were adapted from Bacula
+# Copyright (C) 2000-2010 Free Software Foundation Europe e.V.
+#
+# This file is part of BaculaFS.
+#
+# BaculaFS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+# Bacula is a registered trademark of Kern Sibbald.
+
class SQL :
'''
Holds all SQL statements used by baculafs.
@@ -6,7 +30,6 @@
MYSQL = 'mysql'
POSTGRESQL = 'postgresql'
- SQLITE = 'sqlite'
SQLITE3 = 'sqlite3'
clients = 'SELECT Client.Name,ClientId FROM Client'
@@ -54,21 +77,6 @@
VolSessionTime INTEGER)
''',
- SQLITE: '''
- CREATE TEMPORARY TABLE temp (
- JobId INTEGER UNSIGNED NOT NULL,
- JobTDate BIGINT UNSIGNED,
- ClientId INTEGER UNSIGNED,
- Level CHAR,
- JobFiles INTEGER UNSIGNED,
- JobBytes BIGINT UNSIGNED,
- StartTime TEXT,
- VolumeName TEXT,
- StartFile INTEGER UNSIGNED,
- VolSessionId INTEGER UNSIGNED,
- VolSessionTime INTEGER UNSIGNED)
- ''',
-
SQLITE3: '''
CREATE TEMPORARY TABLE temp (
JobId INTEGER UNSIGNED NOT NULL,
@@ -97,12 +105,6 @@
JobTDate BIGINT)
''',
- SQLITE: '''
- CREATE TEMPORARY TABLE temp1 (
- JobId INTEGER UNSIGNED NOT NULL,
- JobTDate BIGINT UNSIGNED)
- ''',
-
SQLITE3: '''
CREATE TEMPORARY TABLE temp1 (
JobId INTEGER UNSIGNED NOT NULL,
@@ -191,7 +193,7 @@
AND FileSet.FileSet='%s'
'''
- jobs = 'SELECT DISTINCT JobId,StartTime FROM temp ORDER BY StartTime ASC'
+ jobs = 'SELECT DISTINCT JobId,Level,StartTime FROM temp ORDER BY StartTime ASC'
base_jobs = '''
SELECT DISTINCT BaseJobId
@@ -257,32 +259,6 @@
-- dummy comment for chomping extra parameter: %s
''',
- SQLITE: '''
- SELECT FileId, Job.JobId AS JobId, FileIndex, File.PathId AS PathId,
- File.FilenameId AS FilenameId, LStat, MD5
- FROM Job, File, (
- SELECT MAX(JobTDate) AS JobTDate, PathId, FilenameId
- FROM (
- SELECT JobTDate, PathId, FilenameId
- FROM File JOIN Job USING (JobId)
- WHERE File.JobId IN (%s)
- UNION ALL
- SELECT JobTDate, PathId, FilenameId
- FROM BaseFiles
- JOIN File USING (FileId)
- JOIN Job ON (BaseJobId = Job.JobId)
- WHERE BaseFiles.JobId IN (%s)
- ) AS tmp GROUP BY PathId, FilenameId
- ) AS T1
- WHERE (Job.JobId IN (
- SELECT DISTINCT BaseJobId FROM BaseFiles WHERE JobId IN (%s))
- OR Job.JobId IN (%s))
- AND T1.JobTDate = Job.JobTDate
- AND Job.JobId = File.JobId
- AND T1.PathId = File.PathId
- AND T1.FilenameId = File.FilenameId
- ''',
-
SQLITE3: '''
SELECT FileId, Job.JobId AS JobId, FileIndex, File.PathId AS PathId,
File.FilenameId AS FilenameId, LStat, MD5
@@ -313,13 +289,13 @@
SELECT JobId,VolSessionId,VolSessionTime,
PoolId,StartTime,EndTime,JobFiles,JobBytes,JobTDate,Job,JobStatus,
Type,Level,ClientId,Name,PriorJobId,RealEndTime,FileSetId,
- SchedTime,RealEndTime,ReadBytes,HasBase
+ SchedTime,RealEndTime,HasBase
FROM Job WHERE JobId IN (%s)
'''
volumes = '''
SELECT JobMedia.JobId,VolumeName,MediaType,FirstIndex,LastIndex,StartFile,
- JobMedia.EndFile,StartBlock,JobMedia.EndBlock,Copy,
+ JobMedia.EndFile,StartBlock,JobMedia.EndBlock,
Slot,StorageId,InChanger
FROM JobMedia,Media WHERE JobMedia.JobId IN (%s)
AND JobMedia.MediaId=Media.MediaId ORDER BY JobMedia.JobId,VolIndex,JobMediaId
|
[-]
[+]
|
Changed |
BaculaFS-0.1.7.tar.gz/baculafs/__init__.py
^
|
@@ -1,7 +1,31 @@
+
+# BaculaFS - Bacula Filesystem in USErspace
+# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
+#
+# BaculaFS contains SQL queries that were adapted from Bacula
+# Copyright (C) 2000-2010 Free Software Foundation Europe e.V.
+#
+# This file is part of BaculaFS.
+#
+# BaculaFS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+# Bacula is a registered trademark of Kern Sibbald.
+
"""Expose the Bacula catalog as a FUSE file system.
"""
from FileSystem import __version__, main
-__all__ = ["FileSystem", "Database", "Catalog", "Base64", "SQL"]
+__all__ = ["FileSystem", "Database", "Catalog", "Base64", "SQL", "LogFile"]
|
[-]
[+]
|
Changed |
BaculaFS-0.1.7.tar.gz/baculafs/baculafs
^
|
@@ -1,5 +1,28 @@
#! /usr/bin/env python
+# BaculaFS - Bacula Filesystem in USErspace
+# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
+#
+# BaculaFS contains SQL queries that were adapted from Bacula
+# Copyright (C) 2000-2010 Free Software Foundation Europe e.V.
+#
+# This file is part of BaculaFS.
+#
+# BaculaFS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+# Bacula is a registered trademark of Kern Sibbald.
+
'''BaculaFS mount script
'''
|
[-]
[+]
|
Added |
BaculaFS-0.1.7.tar.gz/baculafs/junk.py
^
|
@@ -0,0 +1,6 @@
+import commands
+baculafs_mount_points = [tokens[2] for tokens in
+ [line.split() for line in commands.getoutput('mount -v').split('\n')]
+ if tokens[4].endswith('baculafs')]
+if self.fuse_args.mountpoint in baculafs_mount_points :
+ raise RuntimeError, 'mountpoint already in use by BaculaFS'
|
[-]
[+]
|
Changed |
BaculaFS-0.1.7.tar.gz/setup.py
^
|
@@ -1,5 +1,28 @@
#!/usr/bin/python
+# BaculaFS - Bacula Filesystem in USErspace
+# Copyright (C) 2009, 2010 Avi Rozen <avi.rozen@gmail.com>
+#
+# BaculaFS contains SQL queries that were adapted from Bacula
+# Copyright (C) 2000-2010 Free Software Foundation Europe e.V.
+#
+# This file is part of BaculaFS.
+#
+# BaculaFS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+# Bacula is a registered trademark of Kern Sibbald.
+
from setuptools import setup, find_packages
from baculafs import __version__
@@ -23,6 +46,7 @@
install_requires=['fuse-python>=0.2','pexpect>=2.3','MySQL-python>=1.2.2','psycopg2>=2.0.13'],
classifiers = [
"Development Status :: 3 - Alpha",
+ "Topic :: System :: Filesystems",
"Topic :: System :: Archiving :: Backup",
"Intended Audience :: System Administrators",
"Environment :: No Input/Output (Daemon)",
|