[tex-live] Automated TeX Live Integration Testing

[Pander]

On 2012-09-07 14:26, Heiko Oberdiek wrote:
> On Fri, Sep 07, 2012 at 12:50:33PM +0200, Pander wrote:
> 
>> Would be nice that the people who told me to start working on this that
>> some feedback would be given.
> 
>> On 2012-08-15 12:25, Pander wrote:
>>>
>>> Please review:
>>>   tlit - TeX Live Integration Testing
>>>   https://github.com/PanderMusubi/tlit
>>>
>>> The goal of integration testing of TeX Live is to automatically confirm
>>> that an update did not break any major functionality. Detailed module
>>> testing of individual packagese is up to the package maintaineres. This
>>> is done in much more detail and is outside of the scope of integration
>>> testing. Integration testing offered here is merely a simply check if
>>> certain often used combinations of package can play together. The goal
>>> is to prevent that simultaneous updateting of multiple packages, which
>>> all passed their individual module tests without being aware that other
>>> packages are also being updated on the same time, break TeX Live.

Heiko, thanks for having a look at it.

> * It assumes that packages have individual module tests that are run on
>   updates. Both is not true with few excaptions only.

Modules are tested by developers before uploading to CTAN, that I assume
but it is indeed unclear with which version of TeXLive this is done or
done at all. Actually I don't care, I just want to know if post upgrade
all is working. I will remove the assumption on the documentation.

> * To some degree package testing already involves other packages.
>   There is no sharp separation line for integration testing.

True. I just want to prevent what prevent situation that result in
non-functional TeX Live installation as happen (but was fixed quickly)
last time.

> * I cannot find, how this is addressed:
>   * There are quite a few TeX compilers with quite a few formats.
>   * There are lots of classes and tons of packages.
>   * Classes and packages provide none up to an endless number of options
>     with multiple values and working modes.
>   * The number of package combinations and loading order is overwhelming.

The tests currently in there are some major packages and engines I use.
If they do not work, that is a big enough signal for me. I would expand
the test cases only with incidents that happen in the future.

> * What is tested:
>   * Errors: expected or unexpected
>   * Warnings: expected or unexpected
>   * Messages: expected or unexpected
>   * Output: DVI/PS/PDF
>   * log file, aux files, ...

Simply and only the result status at this time.

> * Which system to manage the tests?

Hope some has one available.

> * How to deal with the result? But report to package authors?

To be discussed and implemented.

> * ...
> 
> Good testing is a huge task requiring lots of "(wo)man power".

My idea was to start something or give an impulse to this and have it
evolve by contributions from all. I have worked with many software
development environments that have a nightly build and subsequent
testing whenever changes have been made to the repository that day.

Best regards,

Pander

PS   If some of you want a .tar.gz, please go to
https://github.com/PanderMusubi/tlit/downloads

> Yours sincerely
>   Heiko Oberdiek
>