Fast update to latest version in a few megabytes?

**nickh** · 07-29-2003, 06:27 PM

If Rsync is working at it's best, only the fragments of files which are different to your local copy are transmitted across the network connection. This means you can update from one version of Knoppix to the next without downloading the whole 700Mb.

It is therefore possible to update your version of Knoppix from one version to the next by downloading only a few magabytes- just the pieces of the files which differ.

The advantages of Rsync are not available when using Rsync to synchronise the whole Knoppix ISO. This is mainly because Knoppix uses a compressed file system. The compressed file system only needs to be changed slightly for all the data to change completely. - The byte offsets change causing the data in each compressed block to be different. Rsync cannot then determine the differences which need to be sent.

Suggestion:
The entire uncompressed, unpacked file system of the current Knoppix image should be made available via anonymous rsync. The user can run a special script to unpack the local copy of Knoppix, Rsync the local copy to the remote copy, then roll the updated version back into an ISO.

This could probably be done with a simple Bash script taking care of all steps- creating a mount point, mounting the ISO, Unpacking the compressed FS, rsyncing the FS then packing the whole lot back into the latest ISO. If binaries from the Knopix ISO were used in the process, the resultant image should theoretically be identical such that an MD5SUM of the resultant ISO would match the latest Knoppix version.

I do not have enough bandwidth here to offer a mirror of the current Knoppix distribution although I could do the scripting.

Another approach would be to have a script and binary data representing the differences which could be patched against a version of Knoppix.

**beaky** · 07-29-2003, 06:39 PM

Google for xdelta

**mabhatter** · 07-29-2003, 06:40 PM

This is a good solution to a huge problem right now. Even though I have DSL at home, downloading an ISO every day is a bit much--especially for minor bug fixes. Such a scheme would require a lot of disk space, but might prove to make Knoppix even more useful.
To expand on this, rather than have to download seperate ISOs for ClusterKnoppix and STD, they could just patch their versions and put up rSyncs too? [would that actually work?] Then all the various patches, and special needs could root back to 1 base distro!

**nickh** · 08-01-2003, 09:43 AM

I have written a patching system. Please see topics:

http://www.knoppix.net/forum/viewtopic.php?t=3956
http://www.knoppix.net/forum/viewtopic.php?t=3957

The patch from 6/6/03 to 26/7/03 is 130Mb.

**beaky** · 08-01-2003, 11:39 AM

Nah, Xdelta is the answer.

**nickh** · 08-01-2003, 05:50 PM

Originally Posted by beaky

Nah, Xdelta is the answer.

I have made an xdelta between the ISO of 6/6/03 and 26/7/03.

The 27/6 ISO is 727525376 100%
The xdelta file is 668205179 91.8%
My patch file is 133789668 18.3%

My patch file is much smaller. Xdelta alone is not the answer.

**beaky** · 08-01-2003, 07:37 PM

Fair enough, you win

although I'm amazed xdellta did so badly.

**nickh** · 08-01-2003, 08:05 PM

Originally Posted by beaky

Fair enough, you win

although I'm amazed xdellta did so badly.

The filesystem is compressed. A fairly small change to the uncompressed file system will change the alignment of the data to the compressed blocks. Every compressed block will resultantly be different. The compressed representation of something slightly different tends to be substantially different. Consequently, there will be very few similarities for xdelta to take advantage of.

My system uncompresses the filesystem, but leaves the structure untouched. The whole file system is chopped into manageable chunks. xdelta is then used to create a patch against those chunks.

On reconstruction, each chunk is joined back together and re-compressed.

My system will only work if the order of the files remain substantially the same and alignment is not moved by more than a few megabytes.

I have made a proposal for xdelta. This involves creating a rolling window for the rolling checksum to allow large files to be diffed with small memory footprint. Xdelta does not currently work on most systems when the target files are gigabytes.