[tex-live] jigdo

[Thomas Hühn]

karl at freefriends.org (Karl Berry) writes:

> Hi Thomas,
>
>     I read "If you are able and willing to make TeX Live also available via
>     jigdo, please email." on the web site and thought I could give it a try.
>
> Thanks!

Too soon, it didn't work. :-/

I had overseen that http://tug.org/texlive/Contents/live/texlive/ is
TL2005, of course. And http://tug.org/texlive/Contents/testlive/ which I
would have needed is empty.

>     Downloading from scratch via jigdo is quite slow
>
> Hmm.  My understanding was that jigdo is something like bittorrent, ie,
> if the image propagates around to many sites, the net download time is a
> lot faster.  (Not to mention decreasing the load on the central server.)
> Is that wrong?

As far as I understand it, it's wrong, yes. But I would like to be
contradicted.

Jigdo works like this:

I put up a template file describing which file with which checksum is
located where in the iso image.

And I put up a jigdo file, describing which servers do have those files
that are contained in the iso image.

The thing is that everyone who uses this jigdo file uses the same fixed
set of servers and, worse, any single file can be mapped to only one
server. (see this mail[1]). So there will be no real load-balancing. The
only thing possible seems to be to split the load between servers in a
fixed scheme. Something like "get files 1 thru 10000 from mirror 1,
files 10001 thru 20000 from mirror 2 and so on).

So with my jigdo file (which I have taken down in the meantime) I was
getting all the files from ftp.tug.org. So, no load advantage.

And tens of thousands of wget invocations.

>     How many files do change between Tex Live releases? 
>
> Lots, probably most.  It's not presently set up to be downloaded
> partially or in combination with older releases.

If most of the files (in http://tug.org/texlive/Contents/live/texlive/)
will change between TL2005 and TL2007, it probably isn't worth it. I'll
do it nonetheless, but it's nothing that should really be advertised,
because people won't be able to use a good part of their TL2005 DVD and
just download the changed files, if almost all the files change between
releases.

If, on the other hand, many files stay the same, fine, let people
assemble their TL2007 using TL2005.

The really interesting thing about jigdo is not load balancing, but
reusing existing files.

If you publish the files via FTP/HTTP (or whatever), as you do at
http://tug.org/texlive/Contents/live/texlive/, you do not necessarily
need to waste the space to have them once more in the iso image. You
just give people a recipe to bake their cake using those ingredients.

But even if jigdo were useful in TeX Live context (which it might be; I
could get quite a lot of files from the TL2005 directory), bittorrent is
much more important, because many people know it.

And an issue with bittorrent came up today at de.comp.text.tex:

http://tug.org/texlive says "TeX Live is now available via bittorrent,
by http or utp."

I don't have the slightest clue, what "utp" might be :-), but
"bittorrent" links to www.bittorrent.com which isn't to helpful if you
want to get the image. It should link to a torrent file (or better, a
web page listing torrent files for live and inst).

"http", on the other hand, doesn't link to a web server serving the
image, but to a torrent tracker, or so it seems:

|Can't send torrent
|
| Non-registered visitors are limited to newer torrents only.
| To download torrents older than a few days, you must register.
| 
| Registration to this website is absolutely free.
| 
| Click here to go to the registration page
| or
| Click here to go back

The registration page says "To avoid running out of resources, the
public registrations are closed. We open the registrations from time to
time, usually on fridays (GMT), so please check back later."

I suggest setting up an own tracker or using a public tracker.

Thomas

[1] https://lists.berlios.de/pipermail/jigdo-user/2007-January/000620.html