xdelta

Re: Xdelta 3.0p (download)

Xdelta-3.x processes a window of input at a time, using a fixed memory budget. This release raises some of the default maximums, for better large-file compression. The default memory settings for large files add up to about 100Mb. (*)

A user wrote to confirm good performance on 3.7Gb WIM files, great!

So, to answer the question in your post on Jan 28th, yes we are using xdelta3 with files larger than 1Gb and it's working well.

I'm off to try boosting the compression and see how much we can scrunch the patch down. Thank-you for an extremely useful tool that is going to hugely improve our ability to distribute updates to our build images.

At the other extreme, a developer wrote to ask about using xdelta3 in the Xen kernel. Here's an example of xdelta3 configured for a 4Kb page using 32Kb.

A user writes about code size:

I saw your post where you said you’d improved the compression in Xdelta3 and that it was now slightly faster than 1.1. Do you happen to know where the two stand without compression enabled? (We don’t use Xdelta’s compression because we compress all of our files after the fact with Rar solid compression.) Thanks too for the link to those benchmarks. Those were quite helpful, and it’s cool to see Xdelta come out on top. ;)

I will say this: One nice thing about the new Xdelta is its smaller size! Xdelta 1.1’s exe is 1Mb when not compressed via UPX, whereas Xdelta3 is 100Kb. Nice. :)

Thanks! Xdelta3 is faster and better than Xdelta1 with compression disabled (the -0 flag), thanks to VCDIFF and other improvements.

I'm looking for your ideas on recursive directory diff. One solution uses tar:

XDELTA="-s source" tar --use-compress-program=xdelta3 -cf ...

This approach is better than computing pairwise differences, since data can be copied across files/directories. Pay attention to file order and source buffer size. Microsoft developers, consider using the WIM disk image format.

Thanks for the feedback (file a new issue).

(*) Xdelta-1.x processes the whole input at once, and its data structure takes linear space, (source file size/2)+O(1). Xdelta-1.x reads the the entire source file fully two times (to generate checksums, to match the input), using at least half as much memory as the source file size.