[tex-k] Speed of kpsewhich when querying the repertories of the TeX installation

[jfbu]

[my reply had too long lines sorry]

Le 1 mai 2015 à 01:05, Norbert Preining <preining at logic.at> a écrit :

>>> If you run kpsewhich --debug=-1 --expand-path=\$TEXMFHOME,
>>> you will see what it's doing.
>> 
>> I am sure kpsewhich does a lot, but does one need to do such a lot
>> such to tell where is TEXMFLOCAL and TEXMFHOME and TEXMFDIST ?
> 
> Yes, one has to. Otherwise 20 years of expectation would be broken.
> 
> People regularly redefine TEXMFHOME and TEXMFLOCAL, and that might even happen on multiple levels (distribution makes first change, system admin another).
> 
> So besides:
> * reading the main texmf.cnf
> * searching for other texmf.cnf
> * evaluating their contents
> there is only hard coded path, and that is a no-go.
> 

There must be at least one initial hard-coded path no? I can't
relocate kpsewhich anywhere on my system or can I ?


Are you sure kpsewhich does not do, apart from evaluating the
contents of the texmf.cnf files, do some un-needed parsing of ls-R
files, I mean unneeded when kpsewhich is invoked as kpsewhich
--expand-path $TEXMFHOME (or whatever)?

This is why I suggested providing a separate utility to provide
only that service of identifying the locations of the repertories:
where is the DIST tree?, where is the LOCAL tree?, where is the
HOME tree?

It appears that there is no issue if the user has set-up
corresponding environment variables.

But in that cas TeXLive installation's manual should be explicit
about it. I am sure this brings other complications.


> 
> Let us go back to your Makefile: how many var-values do younormally need? 2? 3? Evaluating them once and saving costs 300ms.
> 
> Evaluating a complex Makefile with rules is probably more time consuming.
> 
> We do this caching of vars in many of our scripts, and that is definitely not the bottle neck.

I did that naturally when I had the problem :

TEXMF_local = $(shell kpsewhich -var-value TEXMFLOCAL)

# as kpsewhich is very slow (.5s) I want to evaluate once only.
installlocal: xint.tds.zip
	$(eval $@_tmp := $(TEXMF_local))
	unzip xint.tds.zip -d $($@_tmp) && texhash $($@_tmp)



> 
> I would invest time in other, that is to say possible, optimizations.


This is not only a Makefile issue

If I resuscitated this issue after some months, it is because I
discovered yesterday that Emacs/AUCTeX 11.88 on Mac OS X very
lengthy launch was entirely to be explained by its nine calls to
kpsewhich.

Earlier AUCTeX 11.86 did only 2, and furthermore it happened
during initial launch of Emacs, not on (first) opening a .tex file

With 9 calls to kpsewhich, on my (comparatively quite fast, with
SSD) MBA with Mavericks and TeXLive 2014, this meant a 5 seconds
wait when first launching AUCTeX 11.88

On an older Mac machine, as the one Adam Maxwell tested my October
2014 findings about the slowness of kpsewhich --var-value
TEXMFHOME, it would have been 15 seconds.

With my machine and TeXLive 2015, we go from 5 seconds to less
than 1 second thanks to the work of Adam Maxwell into improving
the Mac OS X speed of kpsewhich.

Currently, the AUCTeX maintainers are taking into accounts these
findings (this is ongoing) and already yesterday they had reduced
from 9 to 5 the number of kpsewhich calls, thus halving the launch
time

I try to get them to make only 1 such call, querying all variables
simultaneously as is allowed by kpsewhich

see

http://lists.gnu.org/archive/html/auctex/2015-04/msg00043.html

and next messages from that thread

Let me explicitely mention to people reading this thread that this
issue does concern also Linux boxes, but may well be hidden to
most because their TeXLive installation has set-up environment
variables for TEXMFLOCAL, TEXMFHOME, etc...


best

Jean-François

> 
> Norbert
> ------------------------------------------------------------------------
> PREINING, Norbert                              http://www.preining.info
> JAIST, Japan                                 TeX Live & Debian Developer
> GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
> ------------------------------------------------------------------------
> 
> On May 1, 2015, at 6:51 AM, jfbu <jfbu at free.fr> wrote:
> 
>> 
>> Le 30 avr. 2015 à 23:42, Karl Berry <karl at freefriends.org> a écrit :
>> 
>>>   I may write a Makefile which will act after querying
>>>   the repertories locations. But one tenth of a second
>>>   for each kpsewhich call is a lot.
>>> 
>>> Save the result instead of making the same call thousands of times.  I
>>> don't agree that 100 ms is such a horrible time to do all those
>>> filesystem operations.  In fact, it seems rather quick to me.
>>> 
>>>   Could it be imagined to have some kpsewhichdir utility
>>> 
>>> 1) I fail to see how making a utility restricted to a single variable
>>> could speed anything up.  
>> 
>> 
>> I probably did not express myself precisely. I am not targeting
>> a single variable; I am targeting the variables describing the main
>> repertories of the TeXLive installation
>> 
>> 
>> 
>>> Surely what takes the time is looking for the
>>> config files in all the possible directories, and (more so) reading the
>>> ls-R file.  
>> one might need to do things to find packagefoo.sty via the 
>> parsing of the ls-R files or system calls to ls, but why would these things
>> be needed for the main installation repertories ???
>> 
>> Perhaps many distributions set up environment variables thus
>> the speed penalty is not obvious to these users.
>> 
>> 
>>> 
>>> 2) Regardless, the idea of a utility dedicated to a single variable
>>> sounds wrong in principle to me and I don't think it should be done.
>>> 
>>>   the environment variable $TEXMFHOME 
>>> 
>>> So far as I can see, if the envvar is set, its value is already (has
>>> always been) returned "instantly".  The lookups are only done if they
>>> need to be.
>>> 
>>> karl
>>> 
>> 
>> 
>> I have none such environment variable and do not see why I should
>> set some
>> 
>> Jean-François