Autodir HOWTO

Venkat Ramana Enaganti

2004-06-11

Revision History
Revision 0.102004-06-24VRE
second draft
Revision 0.92004-06-11VRE
first draft

Abstract

This HOWTO is about Autodir installation, configuration and other issues related to Autodir.


Table of Contents

Introduction
Copyright and License
Disclaimer
Feedback
Before going to details...
Where it can be used
What it is not
Differences between Autodir and Autofs
How it works
Some definitions
Directory organization under real base directory
Virtual directory expiration
Backup support
Backup program requirements
Getting it
Autodir options
Managing Home directories
Directories for autohome
Directory organization
Misc options for autohome
Managing group directories
Backup options
Examples
RPM specific
Further Information

Introduction

This document explains about how to create directories on demand using Autodir in a transparent way to the applications. This document also explains using transparent backup feature that is possible with Autodir without bringing system down for backup purpose for all directories managed by Autodir.

Copyright and License

This document, Autodir HOWTO, is copyrighted (c) 2004 by Venkat Ramana Enaganti. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is available at http://www.gnu.org/copyleft/fdl.html.

Linux is a registered trademark of Linus Torvalds.

Disclaimer

No liability for the contents of this document can be accepted. Use the concepts, examples and information at your own risk. There may be errors and inaccuracies, that could be damaging to your system. Proceed with caution, and although this is highly unlikely, the author(s) do not take any responsibility.

All copyrights are held by their by their respective owners, unless specifically noted otherwise. Use of a term in this document should not be regarded as affecting the validity of any trademark or service mark. Naming of particular products or brands should not be seen as endorsements.

Feedback

Feedback is most certainly welcome for this document. Send your additions, comments and criticisms to the following email address : .

Before going to details...

After releasing intraperson beta, I started working on a administration guide that deals with administration aspects of intraPerson. For more details check http://www.intraperson.com. But I was stuck with one simple thing. It is easy to create users in ldap -- at least I think so;-). But how to create home directories for those users in ldap where ever those ldap accounts are imported?

Creating manually is one crude solution. It is equivalent to creating each and every account on every system! Scripts? These are not standard and have their own problems. Undeterred I searched net further. I found some solutions But I was not satisfied as every solution has serious drawback attached with it.

One of the suggestions I received was, there is no need to create home directories as some servers can be configured without home directories. Good suggestion by Tony Earnshaw on ldap mailing list. But what is the flip side?

As an administrator and programmer I know how hard it is to setup and maintenance of these servers. Not to mention the geek documents one has to chew for this kind of setup. And its not over yet. Even then the configuration is different for different servers!

After going through autofs documents and hacking little bit, I arrived at conclusion that autofs protocol may offer solution.

The result is Autodir based on autofs protocol.

Where it can be used

  • Where user accounts reside in centralized databases like ldap, SQL, NIS, NIS+ or other databases, from which user and groups are imported to other systems. To create, for example home, group directories in these systems which import these accounts from centralized databases, on demand in sync with database additions.

  • It can be also used even when accounts are in local system, to hide some extent what accounts exist in /home directory, for example.

  • To exploit its transparent backup feature for 24*7 online systems.

What it is not

Autodir can create directories but it can not remove them after user,group database deletions. And there may be some more limitations with modules used with Autodir. Check appropriate sections.

Differences between Autodir and Autofs

Now the important issue arises as there is already autofs package to handle mounts and Autodir is in similar line with autofs package.

Autofs main purpose is to deal with network mounts on demand instead of mounting all at same time which results in preserving system resources. Though there is some support in autofs package to mount home directories on demand but the requirement is that these home directories must exist already.

But on the other side, Autodir specializes 'only' in local directory creation and mounting on demand.

Not only this, Autodir can create real directories in disk file system such that they do not reside in one single flat base directory. This is how utilities like useradd create by default. In standard file system setup, all home directories reside in base /home directory. For file systems like ext2,ext3 performance will degrade if large number of home directories exist in single base directory.

But for applications accessing these directories Autodir presents all directories for them in a single autofs mounted base directory on demand. But actual directories are created in subdirectories of some other directory in hierarchical style.

For example real home for user with uid user1 will be created like /autohome/u/us/user1 if configured that way but mounted in /home on demand for applications accessing home directory in /home/user1.

Permissions for real base directory where actual home directories are kept, for example /autohome in the above example, kept such a way that /autohome can not be accessed by anyone except by root.

This mounting directories on demand and unmounting when not in use, presents interesting opportunity. Able to tell when the a directory is in use and when it is not in use. Which simply means a program like backup can be started when a directory unmounted.

Autodir exploits this capability by starting command line mentioned backup when ever a directory becomes unused.

There is one more important issue to be presented if you are an administrator reading this document. Autodir does not call external programs mount and umount, as this is the case with autofs package. Rather it uses system calls directly. As a side effect it is faster, more reliable but mtab is not updated. I felt this is not necessary as all mounts and unmounts are local directories.

Adding all this functionality to autofs package will divert it from its main goal.

Another minor difference is that Autodir is completely multi threaded. Autofs is also expected to be multi threaded in future versions.

How it works

Autodir uses modules to get specific functionality. The core Autodir implements generic functionality on which modules can exploit and add specific functionality of their own.

At any movement only one module can be added to Autodir. If there are two modules, for example autohome, autogroup, only one module can be attached to Autodir. If both services are needed, two instances of Autodir should be made so that each instance will have required modules attached to it.

For further explanation I choose autohome module which handles home directory requests to simplify explanation.

Note

  • It is assumed user accounts exists but not their home directories. Either because these accounts created with -M option with useradd or else these accounts imported from ldap, NIS or some other external database for which home directories are not created.
  • It also assumed for this explanation only that all user home directories are in /home directory according to user account information. Though actual directories do not exist at this movement.

Note

Some fine details are intentionally kept aside to make explanation easy to understand.

First autofs file system is mounted on /home directory by Autodir. And this is informed to the Linux kernel that /home is managed by user space application Autodir from now on.

Note

Do not bother too much about autofs file system if you do not understand about it. Just think some special kind of file system something in similarity with memory based file system but with some additional special properties.

                                +----------------+
                                |  Linux Kernel  |
                                +----------------+
                                  /        \
                                 /          \
                                /            \
                               /              \
   +-------------+       +--------+       +------------+        +-----------------+
   | Application |------>| /home  |<----->|  Autodir   |<------>| autohome module |
   +-------------+       +--------+       +------------+        +-----------------+
                              \                                          /
                               \  +----------------+                    /
                                +-|   /autohome    |<------------------+
                                  +----------------+

Whenever an application or daemon needs access to user's home directory, for example wu-imapd, they directly enter to /home to access it. Kernel which notices this request and informs to Autodir if that directory does not exist already in /home.

Autodir, in turn, passes this request to autohome module. Autohome module does not touch /home directory. Instead it manages real home directories some where else like in /autohome as shown in the above figure.

Autohome module creates real home directory if it does not exist already in /autohome directory. After it is successfully created or failed to created, whatever the outcome, it is reported back to Autodir along with the path to real home directory -- if successful.

If autohome module reports success Autodir creates another blank home directory under /home and mounts real home directory from /autohome on it. At the end Autodir informs this to the kernel whether this whole operation successful or failure. Accordingly kernel allows application to enter the directory or reports that no such directory exists back to the application.

Some definitions

Before going further it is better to understand the following terms to simplify explanation.

Virtual directories

These do not exist on disk based file system. Instead these are created and deleted on demand. If system reboots all these directories vanish. In the previous figure, all directories under /home are virtual directories.

Virtual base directory

This is the directory that holds Virtual directories. This directory does exist in disk based file system. In the previous figure /home is virtual base directory.

Real directories

These are the directories that actually reside on the disk. Even after reboot of the system these remain intact. In the previous figure all directories created under /autohome are real directories.

Real base directory

This is the directory that holds real directories. In the above figure /autohome is real base directory.

Each virtual directory is mapped to real directory. Whatever written or modified to virtual directory is actually sent to real directory.

On reboot of the system real directories remain intact. But virtual directories are created on demand as exactly as they were before.

Virtual directories are removed if these are not used for some time And created again if the need arises.

Important

Applications should access only virtual directories. Real directories are hidden from applications except for root. But there is one exception. Backup programs should always access only real directories.

Directory organization under real base directory

Why special organization under real base directory? If we just create real directories as exactly that corresponds to virtual directories there could be performance penalty when there are large number of directories to be created under one single base directory. File systems like ext2/ext3 are not optimized for this kind simple organization.

It would be much better if real base directory is divided into more subdirectories or even dividing these subdirectories again into more subdirectories. And in the final subdirectories actual home directories are kept!

There are three types of directory organization.

level 0

Actually no organization. All home directories are created directly under real base directory.

level 1

Real base directory is divided into more subdirectories. These subdirectories names are derived from first letter of the actual directory to be created. For example, if user1 directory is to be created, first a directory named 'u' is created under real base directory. Then in that subdirectory actual directory user1 created as /<real base directory>/u/user1.

level 2

Same as level 1 organization but after first level of subdirectories, second level subdirectories also created. Name for which is derived form starting two letters of the actual directory to be created. For example, for user user1 as in the above example, /<real base directory>>/u/us/user1 directory created.

Virtual directory expiration

When an application tries to access virtual directory in virtual base directory, Autodir creates virtual directory in it if it does not exist already and mounts the real directory on it from real base directory. But once this happens and if this virtual directory is not accessed from virtual base directory for a specified time by any application, this directory is removed and accordingly that corresponding real home directory in real base directory is marked for backup.

The time period to wait for expiration can be given through command line option to Autodir.

Backup support

Autodir supports backup program launching when a virtual directory expires. The expiring virtual directory is itself means that that particular directory is not accessed by any other program.

Like there is wait duration for expiring virtual directory, for backup also Autodir waits some more time, after virutal directory expiration, for starting backup. This time period can be configured through command line option to Autodir.

A separate backup process for each real directory is used. The backup program can be given arguments of real directory on which to operate.

Important

Backup programs should never access virtual directory or virtual base directory.

Caution

Backup feature is not much useful if virtual directories are accessed all the time by applications.

Backup program requirements

Autodir demands some extra requirements from backup program used. The reason for this is that when backup is working on an expired virtual directory and that specific virtual directory is requested again by an application while backup is running, backup is killed. First SIGTERM is sent to gracefully stop it. But if it does not shutdown in time -- one second at this movement but it may change in future, SIGKILL will be sent which is guaranteed to stop the backup.

Note

When and only when backup stopped, application is given access to the virtual directory requested.

Important

Whatever backup is used, it should be able to recover from this signal gracefully, not causing unrecoverable side effects.

One more important issue is that the environment under which it is run. All backup programs are run as root user. But at the same time all unnecessary root privileges are taken away using POSIX capabilities. In other words these backups can read any file or directory that belongs to any user on the system and nothing more than that. Other then that it is like ordinary user level process.

Getting it

At this movement Autodir available in tar,rpm formats. More information can be found at http://www.intraperson.com/autodir.html.

If source is downloaded, follow these simple steps to install it.

  • Unpack the source.

    $ tar zxvf <tar file name>

  • Move to the expanded directory and execute the following.

    $ ./configure

    $ make

    # make install

Note

configure script check for required libraries. If these are not present it will stop from proceeding.

Autodir options

In this section some of the options to Autodir are explained. Backup options are explained in backup section.

-d

For specifying virtual base directory. If this path does not exist, it will be created. Absolute path is expected for this option.

-t

Expiration timeout for virtual directories. For more details refer to virtual directory expiration.

-m

Module to be used with Autodir. Currently autohome and autogroup are available. Full path to the module expected.

-o

All options that are to be passed to module are given here. This option passing syntax is similar to mount with its -o option.

-f

Stay foreground and log all messages to the console.

-l

This option expects path name to a filename to which Autodir will write its process id.

-h

Help about all options.

-v

Version information.

Managing Home directories

This section will explain how to configure Autodir so that user home directories are created on demand. For this purpose autohome is used which deals with specifics of home directory creation.

To load autohome module, use option -m like, for example, -m /usr/lib/autodir/autohome.so.

Note

autohome does not deal with creating user accounts on local systems or in ldap or in any other database. It only deals with creating home directories once these accounts exist or imported to local system from databases like ldap, NIS.

Important

It is worth mentioning one limitation with autohome module. It expects that user name and home directory are related to each other. For example, for user user1 the home directory should be /home/user1 or /some/directory/name/user1 but not /some/directory/name/userhome1. This can be supported but it will be burden on system resources as each password db entry has to be examined from first to last.

Note

If the existing user password database is such that user home directories are distributed under different base directories, for example /home/class1/user1,/home/class2/user2332, then autohome configuration becomes complicated and it is not recommended.

Directories for autohome

What is virtual base directory and what is real base directory in the context of autohome module?

It all depends on how unix accounts are created. If an unix account created for user name user1 with home directory /home/user1 then /home will become Virtual Base Directory.

Then what is real base directory? It can be any directory. Only thing that has to be kept in mind is, there should be enough space as all actual files are stored here instead of in virtual base directory.

In most server configurations /home is a separate partition mounted on it. But if /home is made virtual base directory files are not stored in that partition! The solution is, do not mount partition on /home but instead mount it under somewhere else and make it real base directory.

Autodir option -d is used to specify virtual base directory. It is little tricky to specify real base directory. If the real base directory is /var/autohome then it is specified with option -o as -o realpath=/var/autohome.

Directory organization

Please refer to directory organization under real base directory.

autohome does support this kind of organization. The option used to specify directory organization desired, is with level sub option. For example, -o level=2.

Misc options for autohome

Option skel can be used if skeleton path is not default value /etc/skel like -o skel=/some/other/dir.

Option noskel can be used with -o to indicate not to copy skeleton files to home directories created.

Managing group directories

autogroup module is for creating directories on demand for common unix group access. It can be used with Samba, for example, to dynamically create shared directories for group of people.

It is just like autohome module but unlike autohome, virtual base directory can be placed anywhere.

The module autogroup is loaded with Autodir with option -m. For example, -m /usr/lib/autodir/autogroup.so.

All options explained in managing home directories are same for autogroup except skel,noskel as these are meaningless for autogroup. But there are two other options specific for autogroup. These are given below.

Tip

autogroup can be used to create home directories as well! Provided that there exists user private groups for each user. This way all group and home directories can be created at one place with one module. But no skeleton files are copied and the autogroup option nopriv should not be used.

nopriv

Some Linux installations use user private groups. If directories for these groups are not to be created, then use this option.

nosetgid

By default setgid is set on group directories created. Use this option to disable this feature.

Backup options

These options are passed to Autodir to request backup services.

-b

This is the main option to specify backup program path and arguments to it. The path given should be absolute path otherwise Autodir does not accept it.

-w

Whenever a virtual directory is not used for a period of time, it is assumed inactive and it is unmounted. After unmounting directory, whether to launch backup immediately or to wait some more time is decided with this option. It takes arguments in seconds. It is the minimum time to wait before starting backup. It should not exceed more then one day.

-p

This is the priority to be given to backup process. This is in the range of 1 to 40 inclusive. Lower value mean higher priority and vice versa. Default value is 30.

-c

This restricts maximum number of backup process at any given time. Default is 150.

Caution

Argument for -b is inclusive of absolute backup program path as well as its own arguments. Therefore it is recommended to use single quotes around this argument

Option -b takes path to executable file as well as arguments to it. But the arguments to it are interpreted for %x character sequences and replaced with predefined strings as follows.

%N

Replaced with virtual directory name.

%L

Replaced with absolute path to real directory.

%K

Replaced with host name.

Others

Others are fed to strftime. See man page for strftime for more information.

Examples

# autodir -d /home 						\
		-m /usr/lib/autodir/autohome.so 		\
		-t 1 						\
		-f 						\
		-o 'realpath=/autohome,level=1,skel=/etc/skel' 	\
		-l /var/lock/autodir
# autodir -d /home						\
		-m /usr/lib/autodir/autohome.so			\
		-t 300						\
		-b '/bin/tar cf /tmp/%N%F.tar %L'		\
		-w 600						\
		-o 'realpath=/tmp/autohome,level=2,noskel'	\
		-l /var/lock/autodir
# autodir -d /var/abase/ 					\
		-m /usr/lib/autodir/autogroup.so		\
		-t 300						\
		-b '/bin/tar cf /tmp/%N%F.tar %L'		\
		-w 86400					\
		-o 'nopriv,nosetgid,realpath=/var/realbase,level=0' 

RPM specific

When installed from rpms, two startup scripts are provided namely /etc/rc.d/init.d/autohome and /etc/rc.d/init.d/autogroup. One for starting Autodir with autohome module and another for starting with autogroup module.

Script configuration files /etc/sysconfig/autohome, /etc/sysconfig/autogroup can be used to specify what options can be passed to Autodir.

Further Information

Mailing list for autodir http://lists.sourceforge.net/mailman/listinfo/intraperson-autodir.

Official website is at http://www.intraperson.com/autodir.html.

Autofs mailing list http://linux.kernel.org/mailman/listinfo/autofs.

Autofs Hacking http://www.goop.org/~jeremy/autofs.