hda1 / 2GB reiserfs hda2 /var 900Mb reiserfs hda3 /home 3.1GB reiserfs
AUTHOR: Archaic <archaic@remove-this.indy.rr.com> AUTHOR: Oliver Brakmann <obrakmann@gmx.net>
DATE: 2003-09-13
DESCRIPTION: A read-only root file system has many advantages over read-write when the computer unexpectedly powers off. However, some parts of the file system still need read-write access. This hint will show how to split the file system into multiple partitions to achieve data stability.
PREREQUISITES: The kernel must be compiled with tmpfs (virtual memory file system) support.
HINT: Warning! Serious ruin of your system may occur if you mistype something. I offer no warranty or guarantee that this will work and your data will be safe. It is recommended that you backup (and test the backup) all data before beginning. If you use tar, don't forget the -p switch. I prefer to boot with a boot CD and then manipulate the hard drive ensuring all needed data is in a static state. Also, you cannot read just the tabbed commands and succeed. You must read the paragraphs as well.
This hint assumes reiser file systems. I chose reiser for its stability and journaling capabilities. Ext2/3 file systems will work as well. For you uptime junkies, you will not like the many reboots in this hint, but that's the only true way to test the boot process. Now let's begin…
First, we need to partition according to our needs. The bare minimum to run a read-only rootfs is / and /var. You may want /tmp as well, but this hint assumes /tmp is on tmpfs. Optionally, /usr can be separated, but unless you are sharing /usr or want it to be read-write, then it can stay on /. Additionally, most will want a partition for /home and possibly /root. Partition size is very subjective so I will not cover it here. This hint assumes you have the ability to add at least 1 extra partition for /var. Here is my partitioning scheme:
hda1 / 2GB reiserfs hda2 /var 900Mb reiserfs hda3 /home 3.1GB reiserfs
Now it's time to start splitting the file systems. You have two choices currently; static dev or devfs (when the 2.6 series kernel stabilizes, devfs will be deprecated and a new virtual /dev will be available). While devfs has many potentially good qualities, I do not like the workarounds needed, the lessened stability, and the soon to be extinct nature of it. If you want devfs then read the devfs and/or devfsd hints and get it working on your system first. Then continue with this hint, skipping the "Dev" section.
Here, we make the transition from a hard drive-based /dev tree to a virtual one. Start by making sure /dev/pts and /dev/shm are _not_ mounted. Also switch to single user mode and make sure all daemons are shut off. If your boot scripts and symlinks are in order you can do this by simply typing 'telinit 1'. We also want to copy the device files and directories to a new location.
#umount /dev/shm #umount /dev/pts #telinit 1 #cd / #cp -a dev .dev
Next, remove all device files and directories from /dev except console, then mount a tmpfs on /dev and copy the original device files to the new /dev.
#mount dev /dev -t tmpfs #cp -a .dev/* dev/
Verify that everything is in order by catting /proc/mounts or using df to see if a tmpfs is mounted on /dev, and ls /dev to make sure the files are there.
Now it's time to make /dev mount at boot time. Since some of the device files are needed by init, we need to mount /dev before init runs. Create the following script:
#cat >/sbin/tmpdev <<EOF #!/bin/sh mount -n dev /dev -t tmpfs cp -a .dev/* /dev exec /sbin/init EOF #chmod 0700 /sbin/tmpdev
To make the kernel run this script, you have to change your boot loader's config file by adding init=/sbin/tmpdev.
Next, we modify the mountfs boot script so that /dev gets written properly to mtab. We do this by adding mount -f /dev to the script. I will show the relevant part of the boot script with the change being underlined like this. ^^^^ echo "Recording existing mounts in /etc/mtab…" > /etc/mtab mount -f / && mount -f /proc && mount -f /dev ^^^^^^^^^^^^^^^^
Lastly, we modify fstab to include the mount. Though not strictly necessary (as /dev is mounted before fstab is read), it's included for a sense of completeness. Note the 'noauto' option. That is to keep the mountfs script from trying to mount it again.
#cat >>/etc/fstab <<EOF dev /dev tmpfs defaults,noauto 0 0 EOF
Now it's time to test it out by rebooting your computer. Use /proc/mounts to verify /dev is mounted. If you use /dev/pts, make sure it is mounted as well. If all is well, it's time to separate var.
Here we will separate /var from /. Again, we need to make sure that no daemons are running. This process is pretty much the same as the Dev section. We will start with mounting the soon-to-be var partition on a temporary mount point and copying all the files in /var to this mount point, then unmounting and remounting the new partition over /var. Don't forget to substitute partition numbers for your setup.
#telinit 1 #cd / #mount /dev/hda2 /mnt #cp -a var/* /mnt #umount /mnt #mount /dev/hda2 /var
Now check to make sure var is properly mounted and contains the directory structure it's supposed to have. Cat /proc/mounts instead of using mtab for maximum reliability. If it's all good, edit fstab to mount /var upon reboot.
#cat >>/etc/fstab <<EOF /dev/hda2 /var reiserfs defaults 0 0 EOF
Now it's time to test it out by rebooting your computer. Use /proc/mounts to verify /var is mounted. If all is well, we can delete the stuff in the original /var. You might want to md5sum the stuff first, to be extra safe.
#telinit 1 #umount -n /var #rm -rf /var/* #mount -n /var
Follow the same steps as in the "Var" section. Telinit'ing to runlevel 1 is not necessary unless by some weird chance you have a daemon writing log files there.
Follow the same basic idea as in the "Dev" section copying the files in /tmp to another location, then mounting tmpfs over /tmp and copying the files back. Make sure to telinit to runlevel 1 in case something is using /tmp. Note: Some people think /tmp should be wiped at reboot and /var/tmp should be used for tmpfiles that need to last across reboots. Others vehemently object. If you are the latter, get used to using /var/tmp. ;) Tmpfs is virtual.
#telinit 1 #cd / #cp -a tmp orig_tmp #mount tmp /tmp -t tmpfs #cp -a orig_tmp/* /tmp
Verify that everything is in order by catting /proc/mounts or using df to see if a tmpfs is mounted on /tmp, and ls /tmp to make sure any files that should be there are there. If it's all good, edit fstab to mount /tmp upon reboot.
#cat >>/etc/fstab <<EOF tmp /tmp tmpfs defaults 0 0 EOF
Now it's time to test it out by rebooting your computer. Use /proc/mounts to verify /tmp is mounted. If all is well, we can delete the orig_tmp directory and the stuff still in /tmp on the root partition.
#rm -rf /orig_tmp #telinit 1 #umount /tmp #rm -rf /tmp/* #mount /tmp #telinit 3 # Or your runlevel of choice.
Note, there may be some hidden dirs or files that still need to be deleted. If someone has a command to catch them, please email me.
Now it is time to make a decision. If you prefer having a writable mtab, skip the "Symlink Mtab" section and read the "Writable Mtab" section. If you think /proc/mounts is good enough, do the opposite. If you can't decide, search the lfs-dev mailing list archives for arguments for and against each.
Here we delete mtab and make the link to /proc/mounts.
#rm /etc/mtab #ln -s /proc/mounts /etc/mtab
Next we modify the mountfs boot script to mount / read-only. Comment out the following lines:
echo "Remounting root file system in read-write mode..." mount -n -o remount,rw / evaluate_retval
Finally, we edit fstab to include the ro option for our root fs.
Since the kernel command line option for reiserfs is generally set to mount read-write, you also must change your boot loader config file to mount / ro instead of rw.
Now it's time to test it out by rebooting your computer. Cat /etc/mtab (which is now /proc/mounts) to verify everything is mounted properly. As an alternative, you can just type 'mount' to get a listing. You should see something like this:
rootfs / rootfs rw 0 0 /dev/root / reiserfs ro 0 0 #<- The "ro" is what you are looking for dev /dev tmpfs rw 0 0 proc /proc proc rw 0 0 devpts /dev/pts devpts rw 0 0 /dev/hda2 /var reiserfs rw 0 0
You can disregard rootfs. Some kernels show this, but it's supposed to be a hidden, virtual fs that only the kernel uses.
If everything looks good then congratulations! You are finished. Read the section "Other Concerns" to prepare for any (minor) gotcha's your new read-only system may throw at you.
Note: With the exception of a few edits, the following text covering a writable mtab was taken directly from the now-obsolete mtab hint written by Oliver Brakmann.
OK, so we want to have a read-only root partition, but /etc/mtab needs to be writable. The simple solution is to put /etc/mtab somewhere else! I will use /var/lib/misc/mtab in this hint, as that is the most logical place as I read the FHS. The downside to this approach is that you have to recompile glibc, util-linux and possibly other software packages. To be compatible with scripts and other (broken) pieces of software, we re-create /etc/mtab as a link to our new mtab file.
Enough introduction, now we'll get our hands dirty. Start by switching to runlevel 1 (single-user mode).
#telinit 1
We need to unpack glibc and linuxthreads, as well as apply any patches necessary per the lfs book instructions. The following instructions assume you are in glibc's top directory. The sed makes newly compiled programs look for the mtab file in /var/lib/misc/mtab.
#cp sysdeps/unix/sysv/linux/paths.h{,.orig}
#sed 's%/etc/mtab%/var/lib/misc/mtab%' \
sysdeps/unix/sysv/linux/paths.h.orig >sysdeps/unix/sysv/linux/paths.h
Next we will rebuild util-linux so that mount, umount and friends are aware of mtab's new location. There are no patches required, just look up the instructions in the LFS Book.
Note: Other packages you might want to rebuild are the GNU fileutils (df) and other programs that might access /etc/mtab. As long as these programs do not write to that file, it is not absolutely required, though. That is why we will set up that symlink.
The next step is to move /etc/mtab to /var/lib/misc/mtab then re-create /etc/mtab as a link to /var/lib/misc/mtab (for compatibility reasons).
#mv /etc/mtab /var/lib/misc/ #chmod 644 /var/lib/misc/mtab # Just to make sure #ln -s ../var/lib/misc/mtab /etc/mtab
Now we need to create /var/lib/misc/mtab _on your root partition_. We need this so that mount and umount always know what is mounted regardless of whether or not /var (where the real mtab is) is mounted, otherwise file systems may not get unmounted properly and disk corruption may occur. We will make this a symlink to /proc/mounts so we can have dynamic information on a static (read-only) root file system. We must do this by first unmounting /var.
#umount -n /var #mkdir -p /var/lib/misc #ln -s /proc/mounts /var/lib/misc/mtab #mount -n /var
Now we have to modify the mountfs boot script. We need to comment out the lines that remount / in read-write mode. Also, we need to explicitly mount /var so we can clear the writable mtab and start recording the mounts. Here is what it should look like:
echo "Remounting root file system in read-write mode…" mount -n -o remount,rw / # evaluate_retval
echo "Recording existing mounts in /var/lib/misc/mtab..." mount -n /var && > /var/lib/misc/mtab && mount -f / && mount -f /proc && mount -f /dev && mount -f /var evaluate_retval
While we are editing mountfs we need to modify the umount section by adding the -n switch. What happens is umount sees a writable mtab (not a symlink), unmounts /var, and then tries to write a lock file to /var/lib/misc. This won't work due to being a read-only file system. (If it wasn't, I wonder if it would/could write to /proc/mounts by way of the symlink.) Anyway, here is the relevant part of the script:
echo "Unmounting all other currently mounted file systems..."
umount -n -a -r
^^ <- Note the addition of -n
Next, modify your /etc/fstab so that /, /proc, and /var do not get mounted automatically (as they will already be mounted). The key word in the following example is 'noauto'. Note that with the lfs-boot scripts, my testing shows that this is not really necessary, but it makes things nice and tidy. We will also add the 'ro' option for your root file system. It's not absolutely necessary as the checkfs script explicitly mounts root as ro and we commented out the remounting rw in the mountfs script. But if you leave it out, mount will mistakenly say it is rw (/proc/mounts will say ro, though).
/dev/hda1 / reiserfs defaults,noauto,ro 0 0 /dev/hda2 /var reiserfs defaults,noauto 0 0 proc /proc proc defaults,noauto 0 0
Finally, if you are on an ext2/3 root file system (or any other that initially boots up in read-only), you are ready to go. If you use reiser, however, by default it initially boots in read-write. You will have to change the config file of your boot loader to mount ro.
You can now remount root read-only since we are not changing any more files. If you have not unmounted any partitions other than /var during the process, all those listed in /etc/mtab should still be mounted. Check out /proc/mounts and be sure that it reports the same mounted partitions as /var/lib/misc/mtab. If it checks out ok, you can switch to runlevel 3 prior to rebooting. The reason for this is to fire up the daemons so they will be using /var. This will put the system into real-life use and may cause a problem to surface that might not be detected until you reboot the next time.
#mount -n -o remount,ro / #telinit 3
Now it's time to test it out by rebooting your computer. Cat /proc/mounts to verify everything is mounted properly. You should see something like this:
rootfs / rootfs rw 0 0 /dev/root / reiserfs ro 0 0 #<- The "ro" is what you are looking for dev /dev tmpfs rw 0 0 proc /proc proc rw 0 0 devpts /dev/pts devpts rw 0 0 /dev/hda2 /var reiserfs rw 0 0
You can disregard rootfs. Some kernels show this, but it's supposed to be a hidden, virtual fs that only the kernel uses. Also, make sure you compare /proc/mounts with /etc/mtab (by catting mtab or typing 'mount') to make sure they agree. /proc/mounts will lack some detail, but what it does have needs to be the same on mtab. Make sure all mounting options are correct.
If everything looks good then congratulations! You are finished. Read the section "Other Concerns" to prepare for any (minor) gotcha's your new read-only system may throw at you.
Here I list a few of caveats you will find with your new system. The main problems I see are installing/uninstalling software, changing passwords, and creating/removing devices (if you use the static /dev method), but there are surely others.
To install something, build as usual (in /home or /var or somewhere else of your choosing as long as it is writable). Keep in mind the space requirements of the package while compiling. Glibc-2.5.x needed around 400MB during compilation but needed much less installed. I keep /home extra large for just that reason. Then when you switch to root to install, just remount root read-write and install (making sure to run ldconfig if necessary). Then remount back to read-only. That's a small price to pay for data security. Also, for the people who think it's redundant to use reiserfs when you're just going to mount read-only, I ask, "What if you lose power while installing software?" I want my data safe. Period.
Initially, the easy way to change passwords is also to remount root read-write, but this can be tedious if you have more than a couple of users, as you will have to be present whenever they change them. I haven't yet explored other options, but the simplest would seem to be symlinking passwd, shadow, group, etc. to /var. This may not be so cut-and-dry however. Shadow may need patching. If anyone tries the latter two, I would be interested in knowing how it comes out. Also anyone using pam with shadow (if that makes any difference).
This is just a reminder that you must create/remove devices in the /.dev directory for the change to last across a reboot. If you don't want to reboot for the changes to apply (who does?) then simply repeat the creation/removal of the device in the virtual /dev as well.
Now that you are running a multi-partition system, you can tweak the mounting options used for each file system. Here is a sample fstab to give you an idea of some of the things that are possible:
/dev/hda1 / reiserfs ro,noauto,nodev 0 0 /dev/hda2 /var reiserfs noauto,nodev,noexec,nosuid 0 0 /dev/hda3 /home reiserfs nodev,nosuid 0 0 /dev/hda4 /tarballs reiserfs nodev,noexec,nosuid,users 0 0 dev /dev tmpfs noauto,noexec,nosuid,size=50k 0 0 proc /proc proc noauto 0 0 tmp /tmp tmpfs nodev,nosuid,size=80m,nr_inodes=20k 0 0 devpts /dev/pts devpts gid=4,mode=620 0 0 devshm /dev/shm tmpfs defaults 0 0
Read 'man 8 mount' to understand the options. The options for tmpfs can be found in the kernel source under Documentation/filesystems/tmpfs.txt.
ACKNOWLEDGEMENTS: I would like to thank Richard Lightman for the initial idea of mounting /dev before exec'ing init (I really hate devfs ;)) and Oliver Brakmann for the mtab hint (I also hate symlinking mtab to /proc/mounts ;)). I would also like to thank Oliver for allowing me to absorb his hint into mine. It would not have been possible to have a complete hint without his hint.
Oliver Brakmann would like to give credit to Seth W. Klein for teaching him through the process in the first place and further corrections and suggestions. Also thanks to DJ Lucas, Jesse Tie-Ten-Quee, Richard Lightman and Wouter Vanwalleghem for additional feedback.
CHANGELOG: [2003-09-13] * Initial version
documented on: 2005.09.12
Newsgroups: gmane.linux.debian.user Date: 2008-04-13
> Is it possible to have /etc on a separate partition from / (root) so > that root can be read-only while /etc is read-write?
The need to have the root filesystem mounted r/w annoys me too. My ideal scenario is to be able to have everything read-only except /home (for user file modification) and /var (for all files the system wants to modify). It's great for security (esp if you have hardware write protect on your hard drives) and simplifying backups (no need to look at the read-only ones).
The hard part is that the root filesystem is supposed to have everything necessary for running the system, particularly during the early boot process… (for example. /etc/fstab is needed identify the other filesystems that need to be mounted, so its no good having that on another filesystem).
On my current system I boot with everything read-only except root, var and home. Ie /usr, /usr/local are all read-only. I have /tmp as a sym link to /var/tmp so that the root filesystem rarely gets written to. (There is a /var/tmp on my root filesystem so that /tmp is usable even before the var filesystem gets mounted on top of it).
All that really remains to allow me to keep the root filesystem read-only is to identify all the files in /etc that are user modifiable (such as /etc/passwd and /etc/shadow) and come up with a way to move them - such as with a symlink to /var/etc…
The remaining files are only modified by the super user, who can be expected to do a mount -o remount as required..
DigbyT
> Why exactly do you need things read-only? > > Try the package flashybrid for a different approach to a similar > problem.
I don't *need* things read-only. I would just rather not *need* to have my root filesystem read write.
I gave some reasons above for why I would like to be able to crontrol if and when the root filesystem is subject to writes..
Running a flash based system is certainly one application that would be made easier by this.
flashybrid looks interesting for embedded applications. But I would really like to be able to flick on the hardware write protect switch on my root filesystem on a regular server and know that nothing will break.
DigbyT
> However, consider: as things stand now, only root can alter files which > don't have write permissions for others. Sure, if the filesystem were > mounted ro then root couldn't write to the files either (or delete > files). However, root could always remount / rw. Therefore there is no > security in a system once root is compromised whatever you do. If root > is not compromised, then standard unix permission scheme will provide > the security.
The trouble is that isn't really true. As long as you have standard utilities like 'passwd' and 'chsh' normal users can cause the root filesystem to be modified any time they want..
And in the examples I gave (running root off a DVD or drive with hardware write protect), a remount rw will only succeed in getting write failures logged….
But it isn't just security. It is another file system needing regular backup, and fewer writes means less likelihood of corruption eg if power goes off at the wrong instant..
The files that are a problem are the ones where either a change can result from user activity (passwrd/shadow) or where they are changed by demons, such as resolv.conf. I don't mind explicit changes by the administrator, who can take care or write-protects or reburning media.
DigbyT
> No. The user isn't modifying anything really, its the suid utility > which is. User's don't have write permission on the /etc/passwd file. > The only security concern is if the suid utility is replaced by another; > in other words, again root is compromised.
Well sure, when a user modifies somethign it always boils down to a progrma doing it on the users behalf. The important point is that the user can invoke such a change at any time. The suid file only restricts the nature of the chance.
That means that in a standard config the root filesystem cannot be made read-only without braeaking things, preventing one possible security enhancing stategy.
> > And in the examples I gave (running root off a DVD or drive with > > hardware write protect), a remount rw will only succeed in getting > > write failures logged.... > > So root turns off logging to. If we're talking about running off a DVD > then this is a LiveCD scenario with union mounting.
so the worst case is a remount gains an infiltrator nothing if the filesystem non-writeability is enforced via hardware.
And yes, I think a LiveCD is a very good example of the sort of hoops you have to jump through to have some of the root filesystem content run off read-only media.
> > But it isn't just security. It is another file system needing regular > > backup, and fewer writes means less likelihood of corruption eg if power > > goes off at the wrong instant.. > > Well sure, that makes sense. However, the only part that needs the > backup is /etc/ anyway, which would need backup if it was separate, so > no gain there.
The /etc on a separate filesystem was the suggestion of the original poster. Its not a solution that achieves my ideal of having only one system and one user filesystem that have to be read/write.
> As for e.g. corruption, I'd suggest having a duplicate root filesystem > taken care of by a script (which checks somehow that all is well) which > rsyncs them. This second root fs would be on its own partition with its > own entry in the boot loader. This suggests that /boot is on its own > partition and it is very easy to have /boot ro.
Exactly what I am doing now on my bsd system:
skaro:/usr/home/digbyt> mount /dev/wd0a on / (NFS exported, local) /dev/wd0h on /usr (NFS exported, local, read-only) /dev/wd0g on /var (local) /dev/wd0f on /usr/local (NFS exported, local, nodev, read-only) /dev/wd0e on /usr/home (NFS exported, local) /dev/wd0d on /user1 (local, nodev, nosuid, read-only) /dev/wd1h on /backup/usr (local, nodev, nosuid, read-only) /dev/wd1f on /backup/local (local, nodev, nosuid, read-only) /dev/wd1d on /backup/user1 (local, nodev, nosuid, read-only) localhost:/cfs/null on /cfs/crypt /dev/wd1a on /backup/root (local, nodev, nosuid, read-only) /dev/wd1g on /backup/var (local, nodev, nosuid, read-only) /dev/wd1e on /backup/home (local, nodev, nosuid, read-only)
Note that all my live partitions are rsync'd with identical partitions on the backup disk every night, and by default all except home, var and root are read-only. The backup scripts know that they only have to backup live partitions that are read-write, and to remount the backup filesystems read-write during the procedure. If I do need to chance something on, for example, /usr/local, I remount it and leave it read/write. The backup scripts will see that it is r/w, back it up, and then make it read-only again when they finish so that it won't be backed up again till I repeat the process. User1 lets me arhive static user files in a way that leaves them accessible without making work for the backup scripts.
If it were not for root, then there would be no writeable filesystem with suid and dev enabled. And of course if root could be mounted read-only, that would be one less filesystem that needed to be scanned for differences every night.
There is also a big saving in boot time if there is a crash, because only filesystems mounted r/w will be dirty and need preening.
> > The files that are a problem are the ones where either a change can > > result from user activity (passwrd/shadow) or where they are changed > > by demons, such as resolv.conf. I don't mind explicit changes by the > > administrator, who can take care or write-protects or reburning media. > > I'd suggest to approach it as a live CD thingy, its a well tried path. > Anything else is frought with dragons.
Sure. I didn't mean to hijack the topic. Its just something I have thought about and experimented with, so wanted to add my 2p worth to the original posters suggestion/query.
DigbyT
> Is it possible to have /etc on a separate partition from / (root) so > that root can be read-only while /etc is read-write?
Making a read-only root-fs is quite complex. You didn't write why you do want to do this. If you have a medium that may break by to many writes (like a flash-disk), you can try mounting another (tmpfs?) over it with unionfs or aufs (I got better results with that). If you want to use that as a security feature, it won't work. If you really want read-only-root, you should have a log at grml-live [1]. Grml is a debian-based live-cd aimed for advanced users with a focus on text-tools (no kde, no ooo), and grml-live is its automated build-tool.
Timo
[1] http://www.grml.org/grml-live/
last updated on 28 January 2004 by William Scully, Senior Systems Programmer, Computer Associates.
This is a simpler procedure to follow to create a base-volume/guest-volume Linux for zSeries server which exploits the bind command. The procedure here was developed on a SUSE SLES7 2.4 kernel and the specific implementation here has been tested with the SUSE SLES8 2.4 kernel.
The plan is to create a two-pack system for each Linux system.
The first pack is the so-called base volume, or basevol. Among these materials are those which are rarely changed and can be shared read-only among several servers simultaneously. This device is owned by the so-called base-server, or, in the examples below, userid LINXBASE.
The second volume is the so-called guest volume or guestvol. These are the materials, which are unique to each Linux server and in directories which must be read/write. The userid used in the examples below is LNXGUEST.
The goal is to place as many programs (both the kernel code and vendor-supplied packages) as possible in the base volume so that it is leveraged over and over again. The remaining guest volume read/write disk space given to a typical Linux server can be far smaller as a result. Also, because fixes applied to packages installed onto the read-only disk are used by all servers, the maintenance burden is substantially decreased.
To implement this scheme requires only a modest change in the way a Linux server is created. Overall, the approach is to prevent the operating system from mounting the base volume materials in read/write mode during the boot process. This is easily accomplished because:
When the Linux boot process starts all directories are in read-only mode.
As the boot process continues, vendor-supplied scripts remount Linux directories in read/write mode, as needed.
It is easy to change the vendor-supplied boot-time scripts to avoid the read/write mounting of the Linux materials. It is also easy to create new boot scripts which mount in read/write mode only those directories which must be read/write. Thus we can easily pick and choose which directories (and subdirectories) are read-only and which are read/write. It is the use of bind-type mount commands which allow specific (sub-)directories to be R/O or R/W. The Redbook reference above includes a useful description of how bind mounts work. In particular you may want to read Sections 8.1 through 8.8.
Note that an objective is to hide, as much as possible, the fact that the server is using bind mounts. Using this implementation users who issue mount -l or cat /etc/mtab will see the same results as if the R/O server was booted with R/W directories. Only if the user issues a cat /proc/mounts will they see the details of what directories are mounted using the bind option.