-
Senior Member
registered user
Squashfs-ed knoppix
As I learn more about knoppix, I am beginning to form this idea of using squashfs instead of cloop. This idea is not new. Apparently there is a distro already doing it, finnix. But why use squashfs instead of cloop :-
1. Squashfs is already in the kernel. There is no need to maintain a separate cloop module source code, which can potentially be incompatible with newer kernels.
2. I ***THINK*** squashfs dynamically uses a memory chunk for decompression, whereis cloop allocates a huge chunk ( currently at least 1G ) of memory to store the uncompressed image. According to finnix, it needs only 192MB, so squashfs is more memory efficient.
I guess I will try out this idea of one of these days.
-
Senior Member
registered user
Read this for the claim about knoppix needing at least 1 GB of memory :-
http://www.finnix.org/Frequently_Ask...and_Knoppix.3F
While at it, I am still very uncertain if the claim made of finnix auther is true. Nevertheless I found that knoppix 6.2 mounted a 1G tmpfs, but the tmpfs is only used if there is no persistent store. When using persistent store, the tmpfs is not used, but it's remained mounted.
-
I suppose for people with lots of processor cycles to spare, it might make sense. Squashfs IS used, AFAIK, in the CD version of Knoppix. But for the many who have less CPU and more disk space (for swap), it's probably better to use cloop. Let us know what you determine experientially!
Cheers!
Krishna
-
Senior Member
registered user
I think the claim that cloop needs at least I G memory is likely to be over-stated. A quick check on cloop source code revealed that it also decompresses the file system on-the-fly basis.
The 1G of /ramdisk mounted, used when one does not have persistent store, is something which the actual usage will grow as it is used. The 1G /ramdisk is mounted, but not used when there is a persistent store. And therefore even though that's a "bug", it does no harm.
In terms of CPU cycles, there is no apparent difference between squashfs and cloop, because both uses the same "on-the-fly" decompression and buffered blocked strategies, and also same compression algorithm.
Therefore the only different between squashfs and cloop is that squashfs has long been accepted into the kernel, it needs no extra maintenance. Cloop is currently maintained separatelly by the author of KNOPPIX.
-
Originally Posted by
kl522
I think the claim that cloop needs at least I G memory is likely to be over-stated. A quick check on cloop source code revealed that it also decompresses the file system on-the-fly basis.
The 1G of /ramdisk mounted, used when one does not have persistent store, is something which the actual usage will grow as it is used. The 1G /ramdisk is mounted, but not used when there is a persistent store. And therefore even though that's a "bug", it does no harm.
In terms of CPU cycles, there is no apparent difference between squashfs and cloop, because both uses the same "on-the-fly" decompression and buffered blocked strategies, and also same compression algorithm.
Therefore the only different between squashfs and cloop is that squashfs has long been accepted into the kernel, it needs no extra maintenance. Cloop is currently maintained separatelly by the author of KNOPPIX.
I don't think Klaus would have become the maintainer of cloop for anything less than a substantial reason; too much work for no real benefit. I don't know what that reason is, but I know it must exist. The "phantom GB of ramdisk" aspect of cloop is probably intentional, perhaps to make possible very fast growth of the actual filesystem, or some such. Regardless of all that, if one had some reason to avoid using the compressed filesystems (perhaps speed?), how difficult would it be to leave that out in a re-master? Just curious, and I thought one of y'all might have the answer.
Cheers!
Krishna
-
Senior Member
registered user
Originally Posted by
krishna.murphy
I don't think Klaus would have become the maintainer of cloop for anything less than a substantial reason; too much work for no real benefit. I don't know what that reason is, but I know it must exist.
That's what we call "belief", it's not based on facts.
I think it's all historical. Cloop existed long time ago. Klaus is not the creator. He is just the maintainer. He needed it at that moment. So now nobody else except him has "vested" interest in it. He is using it simply because he is familiar with it. Kind of sentimental value thingie. As a matter of fact, it's a easy swap to use squashfs.
The "phantom GB of ramdisk" aspect of cloop is probably intentional, perhaps to make possible very fast growth of the actual filesystem, or some such.
I believe that you said this because you did not look at the code. The ram disk has nothing to do with cloop. The ram disk is created for unionsfs with cloop, to allow users to have write access to the otherwise read-only cloop. And it's left mounted unused when the user has a persistent store. If you look at the code you will agree with me, it's a bug.
-
Senior Member
registered user
cloop and squashfs comparison
Below is a comparison of some of the interesting aspects of squashfs vs cloop when running on the ***SAME*** machine with 4G physical memory. Some details have been snipped for clarity :-
When using cloop kernel driver :-
Code:
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 94G 33G 61G 36% /mnt-system
/dev/cloop 4.0G 4.0G 0 100% /KNOPPIX
/dev/loop0 2.4G 940M 1.5G 39% /KNOPPIX-DATA
....
# cat /proc/meminfo
MemTotal: 3588864 kB
MemFree: 3005164 kB
Buffers: 112860 kB
Cached: 303956 kB
SwapCached: 0 kB
Active: 220000 kB
Inactive: 294008 kB
....
# ls -al /mnt-system/KNOPPIX/KNOPPIX
-rwxrwxrwx 1 root root 1546036015 2010-06-16 22:36 /mnt-system/KNOPPIX/KNOPPIX
Here is the corresponding information when using squashfs :-
Code:
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 94G 33G 61G 36% /mnt-system
/dev/loop0 1.4G 1.4G 0 100% /KNOPPIX
/dev/loop1 2.4G 939M 1.5G 39% /KNOPPIX-DATA
....
# cat /proc/meminfo
MemTotal: 3588864 kB
MemFree: 2927872 kB
Buffers: 178880 kB
Cached: 329248 kB
SwapCached: 0 kB
Active: 187076 kB
Inactive: 414508 kB
....
# ls -al /mnt-system/KNOPPIX/KNOPPIX.sq
-rwxrwxrwx 1 root root 1453662208 2010-06-16 23:09 /mnt-system/KNOPPIX/KNOPPIX.sq
Summary :-
1. cloop registers the need to use 4G of virtual memory on /dev/cloop, whereas squashfs registers the need for 1.4 G of virtual memory on /dev/loop0
2. squashfs compressed image is smaller by 100M when using the same compression algorithm and block size.
Cheers.
-
Senior Member
registered user
Following what I have posted previously, I noticed that the kernel 2.6.34 has included the LZMA support for squashfs, it will be a matter of small delay, you will also see the squashfs-tools to OFFICIALLY include LZMA support. Actually squashfs using LZMA has existed a long while ago, and has been actively used by the embedded systems ( particularly OpenWRT ) so there is nothing so new about this.
Cloop also support LZMA compression.
As for me, LZMA is not particularly useful to me, because the compression takes too long. However, for a distro or a LiveCD, it is something particularly useful, meaning for the same amount of space ( say a CDROM ), you can squeeze in additional 10-20 % of additional programs into it.
-
Senior Member
registered user
Have you made any more observations using squashfs with Knoppix? If there are no clear problems or potential drawbacks, I think, if only for the sake of uniformity, we should encourage the use of squashfs. My impression from what I have read is similar to your observation: In practice, the difference isn't that big, but having the necessary modules in the standard kernel makes things a bit easier.
I'd like to make a remastering now, with updated packages and using squasfs, to have an updated system with everything needed and a 4GB persistent image on a 8GB flash drive. Now I need 16..
-
Senior Member
registered user
As you have noticed, I tried out the idea of squashfs. Eversince, I have been using it without a single problem. There is actually no downside, but there are a few ( perhaps not very significant ) upsides. For summary, they are :-
1. stock kernel module (vs patched module ).
2. less virtual memory required.
3. slightly better compression.
To create the compress image, this is what I use :-
Code:
mksquashfs /mnt/knx/source/KNOPPIX /mnt/knx/master/KNOPPIX/KNOPPIX.sq -b 262144 -noappend
For comparison, this is what I use for cloop image :-
Code:
mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" \
-hide-rr-moved -cache-inodes -pad /mnt/knx/source/KNOPPIX \
| nice -5 /usr/bin/create_compressed_fs - 262144 > /mnt/knx/master/KNOPPIX/KNOPPIX
You notice that for both cases, I don't sort the files according to 'frequent use', but that does not mean it is not important. I just don't bother to do it.
Last but not least, I have modified the initrd.gz to mount squashfs accordingly :-
Code:
indknoppix(){
local dev
for dev in "$@"; do
[ -b "$dev" ] || continue
message -n -e "\r${CRE}${BLUE}${SEARCHINGFOR} $DISTRO in: ${MAGENTA}$dev${NORMAL} "
trymount "$dev" /mnt-system >/dev/null 2>&1 || continue
if [ -r /mnt-system/"$knoppix_dir"/KNOPPIX -o -r /mnt-system/"$knoppix_dir"/KNOPPIX.sq ]; then
message -e "\r${CRE}${GREEN}$DISTRO ${FOUNDAT}: ${MAGENTA}$dev${NORMAL} "
return 0
fi
umount /mnt-system
done
return 1
}
....
mountknoppix(){
local k dev dir count=0 RC=0
[ -b /dev/cloop ] || mknod m 644 /dev/cloop b 240 0
[ -f /mnt-system/"$knoppix_dir"/modules/squashfs.ko ] && \
insmod /mnt-system/"$knoppix_dir"/modules/squashfs.ko
[ -d /sys/module/cloop ] || insmod /mnt-system/"$knoppix_dir"/modules/cloop.ko preload=32 || return 2
[ -d /sys/module/aufs ] || insmod /mnt-system/"$knoppix_dir"/modules/aufs.ko || return 3
if checkbootparam squashfs && [ -f /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].[Ss][Qq] ]
then
mount -o loop=/dev/loop1 /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].sq /KNOPPIX
elif [ -f /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].[Ss][Qq] -a \
! -f /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx] ]
then
mount -o loop=/dev/loop1 /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx].sq /KNOPPIX
else
for k in /mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx] \
/mnt-system/"$knoppix_dir"/[Kk][Nn][Oo][Pp][Pp][Ii][Xx][0-9]; do
.....
I retained the cloop booting capability for comparison purposes. When both images are present, I use a cheatcode to select the preferred one. It was kind of elaborate but now I can safely say, it is unnecessary to retain the cloop image at all.
Have fun.
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
Motherboard AM4 A520m WIFI 4 slots Ryzen Gigabyte D3SH BEST PRICE
$50.00
ASUS H170M-PLUS Motherboard M-ATX Intel H170 LGA1151 DDR4 SATA3 HDMI DVI-D VGA
$47.99
MSI H81M-P33 Desktop Motherboard Intel MicroATX Socket LGA1150 USB 3.0 W/IO
$26.99
ASRock B550M Pro SE AM4 DDR4 SATA 6Gb/s 1 PCIe 4.0 x16 Micro ATX Motherboard
$84.99
Gigabyte GA-Q170M-D3H-GSM LGA1151 DDR4 Desktop Motherboard (No I/O Plate)
$39.99
Asrock Z390 Phantom Gaming 4S/AC Wifi 8th/9th Gen Intel 1151 Motherboard Bulk
$33.47
GIGABYTE GA-X99-UD4P MOTHERBOARD, M.2 SATA3, LGA2011-3, DDR4, & I/O Shield
$69.99
MSI PRO B650-P WiFi AMD AM5 DDR5 WiFi 6E ProSeries Motherboard
$112.95
MSI PRO B650-P WiFi AMD AM5 DDR5 WiFi 6E ProSeries Motherboard
$99.95
Supermicro MotherBoard X11SPi-TF Xeon Single CPU Socket LGA3647 w/Heatsink
$314.99