Adding more persistent disk images

[Capricorny]

In Knoppix 6.X, the persistent image is a file, knoppix-data.img, which at boot gets loop-mounted on KNOPPIX-DATA and merged with the cloop-mounted Knoppix image in unionfs.
If something has gone wrong with this image, you may just mount the media (for instance fram another running version of Knoppix), and delete the file. (Don't try to delete a mounted image). On next booting from this device, you will be asked to create a new persistent image.

This image is very tightly integrated with the Knoppix version you are running, and on FAT32 it is limited to 4GB. I find I need more space, I'd like not to have it integrated with the current Knoppix version, and I want to store it on USB-sticks etc.
Typically, I want to install Eclipse, Groovy, Grails and several other Java packages on the stick independent of the running Knoppix. I also want to use USB sticks without all the limitations of the FAT32 file system.
One way to accomplish this, is adding more persistent disk images. I use a 16GB stick, with ca 1 GB for Knoppix 6.2 from CD, 4GB for standard persistent disk image, and two more persistent images, which will be mounted at boot time, but not merged with KNOPPIX in unionfs.

I choose to put the images together with the standard image inside the /knoppix folder.
First step is to create the file, which is done with dd.
Second is to setup a loop device on the file, with losetup
Third is creating an ext2 file system on the image.
Next, create a mounting point and test-mount the loop device. After testing, unset the loop connection with losetup -d
Now, the new image is ready for mounting with mount -o loop <image> <mount point>.

root@Microknoppix:/media/sdd1/knoppix# dd if=/dev/zero of=knoppix-data2.img bs=512 count=8000000
8000000+0 records in
8000000+0 records out
4096000000 bytes (4.1 GB) copied, 336.48 s, 12.2 MB/s
root@Microknoppix:/media/sdd1/knoppix# losetup /dev/loop6 knoppix-data2.img 
root@Microknoppix:/media/sdd1/knoppix# mkfs -t ext2 /dev/loop6
mke2fs 1.41.3 (12-Oct-2008)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
250480 inodes, 1000000 blocks
50000 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1027604480
31 block groups
32768 blocks per group, 32768 fragments per group
8080 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736

root@Microknoppix:/media/sdd1/knoppix# mkdir /mnt/knxdata2 && mount /dev/loop6 /mnt/knxdata2
root@Microknoppix:/media/sdd1/knoppix# umount /mnt/knxdata2 && losetup -d /dev/loop6
root@Microknoppix:/media/sdd1/knoppix# mount -o loop knoppix-data3.img /mnt/knxdata2

After this setup is performed, necessary mounting code may be added to /etc/rc.local. I prefer to stay entirely outside the hard-coded directory system, which eliminates a lot of otherwise natural choices. I use store as a common mounting point for extra "partitions", and mount the development tools volume as /store/share and the user projects as /store/varl.

Edit 20100220:
It seems to be possible to make a backup of the standard persistent image, compress it, and have it copied back at boot time if something has gone wrong with the running image. From Klaus' comments to his list of 6.2.0 cheat codes:

If you place an update*.zip or update*tar.gz file on the medium holding
the KNOPPIX data, it will be unpacked onto the overlayed filesystem
before starting "init", thus allowing quick reconfiguration of the
system.

One consequence of this, is that it may be good practice to back up the persistent image now and then. Accessing it when it is not mounted. For instance, put a KNOPPIX copy somewhere else, and either use that with the fromhd= cheat code, or (g)zipping the knoppix-data.img file from it.

[klaus2008]

Edit 20100220:
It seems to be possible to make a backup of the standard persistent image, compress it, and have it copied back at boot time if something has gone wrong with the running image. ...

I looked inside /init (Knoppix 6.2.1) and found the code for updating the system:

# Check for updates on-disk, install them if necessary.
ls /mnt-system/KNOPPIX/update*.zip /mnt-system/KNOPPIX/update*.tar.gz /mnt-system/KNOPPIX/update*.taz 2>/dev/null | while read update; do
 if [ -r "$update" ]; then
  message -e "\r${CRE}${GREEN}${UPDATING} ${YELLOW}$update${NORMAL}"
  case "$update" in
   *.zip) ( cd / ; unzip -o "$update" >/dev/null 2>&1 ) ;;
   *.tar.gz|*.taz) ( cd / ; tar -zxf "$update" >/dev/null 2>&1 ) ;;
  esac
 fi
done

As far as I understand the code, one has to create an ordinary zip or tar.gz archive inside the running system and place this archive in the KNOPPIX directory. The archive will be extracted into /.

For example, I could run KNOPPIX from the live cd and make all the necessary changes. Then I would create the update-special.tar.gz archive with my configuration files which is copied to a flash device with an installed KNOPPIX on it. I would delete an existing persistent image and boot the system from that flash device. After the creation of a new persistent image I would get a KNOPPIX with my special configuration. (I tried this with a backup of may /home/knoppix and it worked fine.)

[Capricorny]

As far as I understand the code, one has to create an ordinary zip or tar.gz archive inside the running system and place this archive in the KNOPPIX directory. The archive will be extracted into /.

For example, I could run KNOPPIX from the live cd and make all the necessary changes. Then I would create the update-special.tar.gz archive with my configuration files which is copied to a flash device with an installed KNOPPIX on it. I would delete an existing persistent image and boot the system from that flash device. After the creation of a new persistent image I would get a KNOPPIX with my special configuration. (I tried this with a backup of may /home/knoppix and it worked fine.)

Sure. But if you run it on the living system, my understanding is that you will necesarily copy the whole UNIONFS structure, wherever you pick directories that are "mixed". (But of course I may be wrong.) How will that work out on updating? If you mount the image, non-running, and do the zipping you will at least get a somewhat smaller thing.. But will that blank out the UNIONFS part from the compressed image when it is unzipped? Haven't tried.

[garaden]

I think the best thing to do is to copy the KNOPPIX-DATA directory, since that neatly isolates all the personal files. Just be careful: there are a lot of sneaky dot-files in there that probably shouldn't be tarred. I haven't had a chance to explore that further.

All I know is I tried to use Klaus's method and it only partially, I suspect when it hit one of the weird-looking aufs dot-files. So I tried a different tar without them (I did this by using two computers, one with the old image and one with the new one, making the old one an NFS server with /KNOPPIX-DATA exported, then copying all of it to a mount point on the computer running the new image, then copying that into my home directory as "knoppix" which prevented me from copying files that shouldn't be copied. In retrospect it probably would have worked just as well if I'd just moved the data on the old image to the old image's home directory [which would skip the problematic files], tarred it up and transferred it to the new image). But instead of using Klaus's method I used the "-b debug" cheatcodes and executed the command myself to see what would happen. There were no errors and the files were moved properly. I'll try it again with some of the problem files in the .tar and without them using Klaus's method when I have some time to spend testing.

[proctor]

i also have been working on getting KNOPPIX-DATA onto a dvd. i have tested garaden's solution above and can verify that it works properly once all the "dot-wh-dot" (.wh.*) files have been removed from the tarred KNOPPIX-DATA directory.

i also created a hacked version of the knoppix initialization script in the initramfs file minirt.gz which i will document a bit further at my original post http://www.knoppix.net/forum/threads...t-image-on-dvd

sincerely,
proctor

[kl522]

i also have been working on getting KNOPPIX-DATA onto a dvd. i have tested garaden's solution above and can verify that it works properly once all the "dot-wh-dot" (.wh.*) files have been removed from the tarred KNOPPIX-DATA directory.

Oh no, I wonder how people come to this conclusion about those dot-wh-dot files. In my view, if you want to copy a consistent copy of "file sets" ( ie collection of files ), you should rather copy the /UNIONFS counterpart, and not copy /KNOPPIX-DATA and omitting the dot-wh-dot.

A package is basically a set of files. Very often when you update a package (using synaptics for example), that might result in removing some older files on the read-only file system which does not have it's counterpart in the updated package. To mark that these read-only files as deleted, thereby there will be dot-wh-dot files created on the read-write file system.

The mechanism maybe too complicated to be explained here. But all in all, if you copy files on the read-write file system ( and omitting the dot-wh-dot files ), you will likely end up with having inconsistent packages.

[proctor]

by "copy the /UNIONFS counterpart" do you mean the entire system as opposed to "only the changes?" on my test system /UNIONFS consists of around 2GB data, whereas /KNOPPIX-DATA is only 140MB....

is there a better way to do what i am trying to do (use the persistent overlay image knoppix-data.img on a read-only media)?

sincerely,
proctor

[kl522]

by "copy the /UNIONFS counterpart" do you mean the entire system as opposed to "only the changes?" on my test system /UNIONFS consists of around 2GB data, whereas /KNOPPIX-DATA is only 140MB....

is there a better way to do what i am trying to do (use the persistent overlay image knoppix-data.img on a read-only media)?

sincerely,
proctor

Sorry it wasn't clear to me what you are trying to accomplish. But I am only saying that if you copy /KNOPPIX-DATA but omitting the dot-wh-dot files, you will have inconsistent packages.

[proctor]

well this is very valuable information to have!

what happens when the dot-wh-dot files are left intact is that when the init system tries to mount the tarred structure into the existing overlay, the dot-wh-dot files conflict/interfere with their already existing counterparts in the UNIONFS system.

do you know of a workaround for this issue?

proctor

[proctor]

hi,

after looking in more detail at aufs it seems to me that using the tarred /KNOPPIX-DATA (really a tarred mounted /KNOPPIX/knoppix-data.img) with removed dot-wh-dot files should not disrupt an aufs file system because the tarred system is not overlaid into the UNION and so therefore not merged in the same fashion that aufs uses: they are simply copied into the UNION, into the aufs.

it seems to me that this is actually the preferred method, and not my patch (at http://www.knoppix.net/forum/threads...t-image-on-dvd).

would anyone disagree with this?

proctor