A Bottle of Wind

y combinator

Thu, 13 Oct 2022 19:05:08 -0700

the topic starts with recursive functions; a typical example is the factorial function F(n); it is easy to implement; for example, we can implement it in python using either def or lambda:

def F(n):
    if n == 0:
        return 1
    else:
        return n * F(n-1)

F = lambda n: 1 if n == 0 else n * F(n-1)

but think about this: what if all functions are anonymous? we are not able to use F in its own function body; then how do we express recursion?

session 0: basic

actually there is a way to express anonymous recursion in lambda calculus; the magic is F = GG; here G is a higher-order function that gives F when applied to itself; we pass G to itself as an argument so that it can emulate recursion; we use the factorial example (notation, use \ for λ):

F   =      \n.(n == 0 ? 1 : n *  F(n-1))        # definition of F
GG  =      \n.(n == 0 ? 1 : n * GG(n-1))        # F=GG
GG  = ( \g.\n.(n == 0 ? 1 : n * gg(n-1)) ) G    # β-reduction
G   =   \g.\n.(n == 0 ? 1 : n * gg(n-1))        # cancel G

now we have G, so we can calculate F as GG:

def exam0():
    G = lambda g : (
        lambda n : (
            1 if n == 0 else n * g(g)(n-1)
        ))
    F = G(G)
    assert F(4) == 24

the assert statement passes; this method is working;

before we end this session, we adopt a simplified format when write deductions and proofs; we only write the recursive parts because other parts are verbatim and can be complemented whenever necessary; for example, the above example can be simplified into:

F   =      \n. F(n-1)
GG  =      \n.GG(n-1)
GG  = ( \g.\n.gg(n-1) ) G
G   =   \g.\n.gg(n-1)

session 1: fancy

this session upgrades the magic F=GG into F=GGGGGGGG; the number of G on the right side can be any integer larger than one (eight in this example); the deductions are similar to the above one, just with more steps:

F          = (                      \n.       F(n-1) )
GGGGGGGG   = (                      \n.GGGGGGGG(n-1) )      # F = GGGGGGGG
GGGGGGGG   = (                   \g.\n.GGGGGGGg(n-1) ) G    # β-reduction
GGGGGGG    = (                   \g.\n.GGGGGGGg(n-1) )      # cancel G
GGGGGGG    = (                \f.\g.\n.GGGGGGfg(n-1) ) G
GGGGGG     = (                \f.\g.\n.GGGGGGfg(n-1) )
GGGGGG     = (             \e.\f.\g.\n.GGGGGefg(n-1) ) G
GGGGG      = (             \e.\f.\g.\n.GGGGGefg(n-1) )
GGGGG      = (          \d.\e.\f.\g.\n.GGGGdefg(n-1) ) G
GGGG       = (          \d.\e.\f.\g.\n.GGGGdefg(n-1) )
GGGG       = (       \c.\d.\e.\f.\g.\n.GGGcdefg(n-1) ) G
GGG        = (       \c.\d.\e.\f.\g.\n.GGGcdefg(n-1) )
GGG        = (    \b.\c.\d.\e.\f.\g.\n.GGbcdefg(n-1) ) G
GG         = (    \b.\c.\d.\e.\f.\g.\n.GGbcdefg(n-1) )
GG         = ( \a.\b.\c.\d.\e.\f.\g.\n.aabcdefg(n-1) ) G
G          = ( \a.\b.\c.\d.\e.\f.\g.\n.aabcdefg(n-1) )

def exam1():
    G = lambda a : (
        lambda b : (
        lambda c : (
        lambda d : (
        lambda e : (
        lambda f : (
        lambda g : (
        lambda n : (
            1 if n == 0 else n * (a)(a)(b)(c)(d)(e)(f)(g)(n-1)
        ))))))))
    F = (G)(G)(G)(G)(G)(G)(G)(G)
    assert F(4) == 24

to see F defined this way is correct:

F = (G)(G)(G)(G)(G)(G)(G)(G)
  = lambda n : 1 if n == 0 else n * (G)(G)(G)(G)(G)(G)(G)(G)(n-1)   # β-reduction
  = lambda n : 1 if n == 0 else n * F(n-1)                          # definition of F

session 2: fancier

you probably have noticed, in exam1, when we define:

F = (G)(G)(G)(G)(G)(G)(G)(G)

we are setting all of a, …, g to G; so, these variables can be switched between each other; this can also be reasoned from the deductions: β-reductions in the deductions factor out one or more arbitrary G, thus choosing different G to factor out can give a different term as deduction result;

the example below gives another arbitrary combination of a to g; because it has seven variables and eight positions, at least one variable will show in two positions; it is not quite obvious how to remove this duplication; anyway, this method works;

def exam2():
    G = lambda a : (
        lambda b : (
        lambda c : (
        lambda d : (
        lambda e : (
        lambda f : (
        lambda g : (
        lambda n : (
            1 if n == 0 else n * (f)(c)(a)(d)(e)(e)(a)(g)(n-1)  # any combination
        ))))))))
    F = (G)(G)(G)(G)(G)(G)(G)(G)
    assert F(4) == 24

session 3: y combinator

the solutions given in the above sessions do work, but they require we rewrite the function definition by duplicating some variables; for example, in session 0 we have G as:

G = \g.\n.gg(n-1)   # `gg`, not `g`

this looks ugly; we wish we can get F from this beautiful G that keeps the original function definition:

G = \g.\n.g(n-1)    # `g`, not `gg`

to do so, we introduce the concept of fixed point: a fixed point of a function is a value that is mapped to itself by the function; in other words, if fix f is a fixed point of f, then:

fix f = f(fix f)

to see why fixed points are useful to our problem at hand, we currently have:

F = \n.F(n-1)       # definition of F
G = \g.\n.g(n-1)    # definition of G

we have observations in both directions:

we know F is a fixed point of G, because:

GF = (\g.\n.g(n-1)) F   # definition of G
   = \n.F(n-1)          # β-reduction
   = F                  # definition of F

for any fixed point F' of G, we have:
```
F' = GF'                # F' is fixed point
   = (\g.\n.g(n-1)) F'  # definition of G
   = \n.F'(n-1))        # β-reduction
```
but this means F' = F because F' and F have the same definition (assuming they have the same initial value);

put these together, F and fix G are exactly the same thing; to get F, we just need to get fix G; imagine we have a function Y such that Y G gives fix G, then we can use Y to get F; we do not know this Y yet, but will soon see how to make it;

because F is the fixed point of G, we know F=GF (from definition of fixed point) and F=YG (from definition of Y);

now, recall our magic F=GG; but G already has a different meaning here, so we rename our magic into F=HH; we assume there is a higher-order function H that gives F when applied to itself; then we have these deductions:

F=GF
HH=G(HH)
HH=(\x.G(xx)) H
H =(\x.G(xx))

F=HH
 =(\x.G(xx))(\x.G(xx))
 =(\g.(\x.g(xx))(\x.g(xx))) G

F=YG

YG=(\g.(\x.g(xx))(\x.g(xx))) G
Y = \g.(\x.g(xx))(\x.g(xx))     # bingo!

this is the famous y combinator;

to see it works correctly, we verify YG=G(YG):

YG=(\g.(\x.g(xx))(\x.g(xx))) G
  =    (\x.G(xx))(\x.G(xx))
  =  G((\x.G(xx))(\x.G(xx)))
  =  G(YG)

this means YG indeed gives a fixed point of G;

cool, let us use this Y to define F:

def exam3():
    Y = lambda g : (
        (lambda x : g(x(x)))
        (lambda x : g(x(x)))
    )
    G = lambda g : (
        lambda n : (
            1 if n == 0 else n * g(n-1)
        ))
    F = Y(G)

wait, it does not work; python outputs:

RecursionError: maximum recursion depth exceeded

session 4: y combinator expanded

we did not expect the failure in the previous session; in fact the y combinator we got in that session is good; the problem is with evaluation order;

there are two types of evaluation orders:

strict (aka: eager) evaluation requires all function arguments to be evaluated before the function is applied;
non-strict (aka: lazy) evaluation allows a function to return a result before all arguments are evaluated;

the y combinator only works in a lazy evaluation environment, but python is an eager evaluation environment; the python evaluation of some function arguments lead to infinite loop and exhaust the stack; to find the exact error spot, we have turned the y combinator into a def implementation:

def exam4():
    def Y(g):
        def A(x):
            return g(x(x))
        def B(x):
            return g(x(x))  # boom!
        return A(B)

    G = lambda g : (
        lambda n : (
            1 if n == 0 else n * g(n-1)
        ))
    F = Y(G)

this code is pretty easy to follow, so no explanation here; do some prints to track the control flow if they are helpful;

session 5: z combinator

the rescue to an eager evaluation environment is the z combinator;

the z combinator is very similar to the y combinator; the difference is some η -reductions:

Y=(\g.(\x.g(   xx ))(\x.g(   xx )))
Z=(\g.(\x.g(\v.xxv))(\x.g(\v.xxv)))

η-reductions do not change function return value, but delay the evaluation of function arguments (because they are now provided in an evaluated form); code below now works correctly:

def exam5():
    Z = lambda g : (
        (lambda x : g(lambda v : x(x)(v)))
        (lambda x : g(lambda v : x(x)(v)))
    )
    G = lambda g : (
        lambda n : (
            1 if n == 0 else n * g(n-1)
        ))
    F = Z(G)
    assert F(4) == 24

session 6: z combinator expanded

this session turns the z combinator into a def implementation:

def exam6():
    def Z(g):
        def A(x):
            def C(v):
                return x(x)(v)
            return g(C)         # lazy
        def B(x):
            def D(v):
                return x(x)(v)
            return g(D)         # lazy
        return A(B)
    G = lambda g : (
        lambda n : (
            1 if n == 0 else n * g(n-1)
        ))
    F = Z(G)
    assert F(4) == 24

here is a stepping trace of the evaluation of F(n) | n = 4:

F(n)                = Z(G)(n)
                    = A(B)(n)
                    = G(C)(n) | (C.x=B)
                    = 1 if n == 0 else n * C(n-1) | (C.x=B)

C(n-1) | (C.x=B)    = B(B)(n-1)
                    = G(D)(n-1) | (D.x=B)
                    = 1 if (n-1) == 0 else (n-1) * D(n-2) | (D.x=B)

D(n-2) | (D.x=B)    = B(B)(n-2)
                    = G(D)(n-2) | (D.x=B)
                    = 1 if (n-2) == 0 else (n-2) * D(n-3) | (D.x=B)

D(n-3) | (D.x=B)    = B(B)(n-3)
                    = G(D)(n-3) | (D.x=B)
                    = 1 if (n-3) == 0 else (n-3) * D(n-4) | (D.x=B)

D(n-4) | (D.x=B)    = B(B)(n-4)
                    = G(D)(n-4) | (D.x=B)
                    = 1 if (n-4) == 0 else (n-4) * D(n-5) | (D.x=B)
                    = 1

session 7: hybrid combinator (y and z)

those who have step-traced the y combinator failure may realize that only the second part needs a patch in an eager evaluation environment; this produces a hybrid combinator that looks like both y and z:

def exam7():
    YZ = lambda g : (
        (lambda x : g(x(x)))                # like y combinator
        (lambda x : g(lambda v : x(x)(v)))  # like z combinator
    )
    G = lambda g : (
        lambda n : (
            1 if n == 0 else n * g(n-1)
        ))
    F = YZ(G)
    assert F(4) == 24

session 8: y8 combinator

in this session we introduce a method to construct an infinite-sized group of fixed-point combinators; the simplest in this group is the y combinator which has been introduced above, others are more complicated; because the combinator can be quite long, we use a non-standard notation ^ for repetitive patterns;

we can construct a combinator for any size greater than 1; the example below constructs a combinator with size 8;

we have:

F = \n.F(n-1)
G = \g.\n.g(n-1)

F=GF
F=YG

we want Y such that:

YG=G(YG)

assume F=HHHHHHHH, then we have HHHHHHHH=G(HHHHHHHH); then we can make the following deductions; we can freely choose any H to factor out in each step, so there are many results other than aabcdefg here:

HHHHHHHH=(                        G(HHHHHHHH))
HHHHHHHH=(                     \g.G(HHHHHHHg)) H
HHHHHHH =(                     \g.G(HHHHHHHg))
HHHHHHH =(                  \f.\g.G(HHHHHHfg)) H
HHHHHH  =(                  \f.\g.G(HHHHHHfg))
HHHHHH  =(               \e.\f.\g.G(HHHHHefg)) H
HHHHH   =(               \e.\f.\g.G(HHHHHefg))
HHHHH   =(            \d.\e.\f.\g.G(HHHHdefg)) H
HHHH    =(            \d.\e.\f.\g.G(HHHHdefg))
HHHH    =(         \c.\d.\e.\f.\g.G(HHHcdefg)) H
HHH     =(         \c.\d.\e.\f.\g.G(HHHcdefg))
HHH     =(      \b.\c.\d.\e.\f.\g.G(HHbcdefg)) H
HH      =(      \b.\c.\d.\e.\f.\g.G(HHbcdefg))
HH      =(   \a.\b.\c.\d.\e.\f.\g.G(aabcdefg)) H
H       =(   \a.\b.\c.\d.\e.\f.\g.G(aabcdefg))

H=(\a.\b.\c.\d.\e.\f.\g.G(aabcdefg))
F=HHHHHHHH
 =H^8
 =     (\a.\b.\c.\d.\e.\f.\g.G(aabcdefg))^8
 =(\x.((\a.\b.\c.\d.\e.\f.\g.x(aabcdefg))^8)) G

F=YG

Y=(\x.((\a.\b.\c.\d.\e.\f.\g.x(aabcdefg))^8))

now we have got Y; we then verify YG=G(YG);

let _ = (\a.\b.\c.\d.\e.\f.\g.G(aabcdefg)); this _ is actually H, but we use _ to mean it does not need for the proof special properties H may have;

YG=(\x.((\a.\b.\c.\d.\e.\f.\g.x(aabcdefg))^8)) G
  =     (\a.\b.\c.\d.\e.\f.\g.G(aabcdefg))^8 = ________
  =     (\a.\b.\c.\d.\e.\f.\g.G(aabcdefg))(\a.\b.\c.\d.\e.\f.\g.G(aabcdefg))(\a.\b.\c.\d.\e.\f.\g.G(aabcdefg))^6
  =     (   \b.\c.\d.\e.\f.\g.G(__bcdefg))                                  (\a.\b.\c.\d.\e.\f.\g.G(aabcdefg))^6
  =     (      \c.\d.\e.\f.\g.G(___cdefg))                                  (\a.\b.\c.\d.\e.\f.\g.G(aabcdefg))^5
  =     (         \d.\e.\f.\g.G(____defg))                                  (\a.\b.\c.\d.\e.\f.\g.G(aabcdefg))^4
  =     (            \e.\f.\g.G(_____efg))                                  (\a.\b.\c.\d.\e.\f.\g.G(aabcdefg))^3
  =     (               \f.\g.G(______fg))                                  (\a.\b.\c.\d.\e.\f.\g.G(aabcdefg))^2
  =     (                  \g.G(_______g))                                  (\a.\b.\c.\d.\e.\f.\g.G(aabcdefg))^1
  =     (                     G(________))
  =                           G(________)
  =                           G(YG)

the simplest one in the group (ie: y combinator) can be called Y2, while this big one with eight variables can be called Y8; compare Y2 and Y8:

Y2=(\x.(  (\a.                  x(aa      ))^2  ))
Y8=(\x.(  (\a.\b.\c.\d.\e.\f.\g.x(aabcdefg))^8  ))

def exam8():
    Y8 = lambda x : (
        (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: ( x((a)(a)(b)(c)(d)(e)(f)(g))))
        (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: ( x((a)(a)(b)(c)(d)(e)(f)(g))))
        (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: ( x((a)(a)(b)(c)(d)(e)(f)(g))))
        (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: ( x((a)(a)(b)(c)(d)(e)(f)(g))))
        (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: ( x((a)(a)(b)(c)(d)(e)(f)(g))))
        (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: ( x((a)(a)(b)(c)(d)(e)(f)(g))))
        (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: ( x((a)(a)(b)(c)(d)(e)(f)(g))))
        (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: ( x((a)(a)(b)(c)(d)(e)(f)(g))))
    )
    G = lambda g : (
        lambda n : (
            1 if n == 0 else n * g(n-1)
        ))
    F = Y8(G)   # boom!

as you might have expected, this y8 combinator does not work in python, for the same reason (ie: evaluation order) the y combinator does not work in python;

session 9: y8 combinator expanded

this session gives def implementation of y8 combinator, and shows where it fails; ok, that was the plan, but when i started writing this session i felt explaining y8 would need too much effort, so i had to downgrade y8 to y4 and explained y4 instead… the explanation is pretty much the code itself;

this session also shows how to patch this y4 combinator into z4 combinator, which works with eager evaluation;

def exam9():

    def Y4(x):
        def A(a):
            def B(b):
                def C(c):
                    return x((a)(a)(b)(c))      # boom!
                return C
            return B
        return (A)(A)(A)(A)

    def Z4(x):
        def A(a):
            def B(b):
                def C(c):
                    def V(v):
                        return (a)(a)(b)(c)(v)
                    return x(V)                 # lazy
                return C
            return B
        return (A)(A)(A)(A)

    G = lambda g : (
        lambda n : (
            1 if n == 0 else n * g(n-1)
        ))
    F = Y4(G)   # boom!
    F = Z4(G)
    assert F(4) == 24

session 10: z8 combinator

this session patches y8 combinator into z8 combinator, using the same technique as in the previous session (for z4 combinator); in contrast to the previous session, combinators in this session are implemented with lambda not def;

there are three combinators in this session in addition to Y8:

Z8BRUTAL: a patch abusing η-reductions without much thought;
Z8: a patch making good (and fewer) use of η-reductions;
Z8UNROLL: Z8 unrolled;

def exam10():

    #Y8=(\x.(  (\a.\b.\c.\d.\e.\f.\g.x(aabcdefg))^8  ))
    def Y8(x):
        A = (
        lambda a : (
        lambda b : (
        lambda c : (
        lambda d : (
        lambda e : (
        lambda f : (
        lambda g : (
            x((a)(a)(b)(c)(d)(e)(f)(g))
        ))))))))
        return (A)(A)(A)(A)(A)(A)(A)(A)

    def Z8BRUTAL(x):
        A = (
        lambda a : (
        lambda b : (
        lambda c : (
        lambda d : (
        lambda e : (
        lambda f : (
        lambda g : (
            x(
                lambda vg: (
                    lambda vf: (
                        lambda ve: (
                            lambda vd: (
                                lambda vc: (
                                    lambda vb: (
                                        lambda va: (a)(a)(va)
                                    )(b)(vb)
                                )(c)(vc)
                            )(d)(vd)
                        )(e)(ve)
                    )(f)(vf)
                )(g)(vg)
            )
        ))))))))
        return (A)(A)(A)(A)(A)(A)(A)(A)

    def Z8(x):
        A = (
        lambda a : (
        lambda b : (
        lambda c : (
        lambda d : (
        lambda e : (
        lambda f : (
        lambda g : (
            x(lambda v: (a)(a)(b)(c)(d)(e)(f)(g)(v))
        ))))))))
        return (A)(A)(A)(A)(A)(A)(A)(A)

    def Z8UNROLL(x):
        return (
            (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: x(lambda v: (a)(a)(b)(c)(d)(e)(f)(g)(v)))
            (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: x(lambda v: (a)(a)(b)(c)(d)(e)(f)(g)(v)))
            (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: x(lambda v: (a)(a)(b)(c)(d)(e)(f)(g)(v)))
            (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: x(lambda v: (a)(a)(b)(c)(d)(e)(f)(g)(v)))
            (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: x(lambda v: (a)(a)(b)(c)(d)(e)(f)(g)(v)))
            (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: x(lambda v: (a)(a)(b)(c)(d)(e)(f)(g)(v)))
            (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: x(lambda v: (a)(a)(b)(c)(d)(e)(f)(g)(v)))
            (lambda a: lambda b: lambda c: lambda d: lambda e: lambda f: lambda g: x(lambda v: (a)(a)(b)(c)(d)(e)(f)(g)(v)))
        )

    G = lambda g : (
        lambda n : (
            1 if n == 0 else n * g(n-1)
        ))
#    F = Y8(G)   # boom!
#    F = Z8BRUTAL(G)
    F = Z8(G)
#    F = Z8UNROLL(G)
    assert F(4) == 24

session 11: turing combinator

in addition to the y combinator, there is another important combinator called turing combinator (here denoted by T); as its name implies, it is discovered by alan turing;

deductions (recall our magic T=SS):

Tf=f(Tf)
T=\f.f(Tf)
T=SS
SS=\f.f(SSf)
SS=(\z.\f.f(zzf)) S
S =(\z.\f.f(zzf))
T =(\z.\f.f(zzf))(\z.\f.f(zzf))
  =(\x.\y.y(xxy))(\x.\y.y(xxy)) # α-conversion

verify:

TG = (\x.\y.y(xxy))(\x.\y.y(xxy)) G
   = (   \y.y(SSy))               G # β-reduction
   =        G(SSG)                  # β-reduction
   =        G( TG)

as an exercise, we can do fancy turing combinator too if we assume T=SSSSSSSS instead of T=SS; again, the number of S can be any integer larger than one and the choice of variable to factor out in each step is arbitrary:

Tf=f(Tf)
T=\f.f(Tf)
T=SSSSSSSS
SSSSSSSS=(                     \f.f(SSSSSSSSf))
SSSSSSSS=(                  \h.\f.f(SSSSSSShf)) S
SSSSSSS =(                  \h.\f.f(SSSSSSShf))
SSSSSSS =(               \g.\h.\f.f(SSSSSSghf)) S
SSSSSS  =(               \g.\h.\f.f(SSSSSSghf))
SSSSSS  =(            \e.\g.\h.\f.f(SSSSSeghf)) S
SSSSS   =(            \e.\g.\h.\f.f(SSSSSeghf))
SSSSS   =(         \d.\e.\g.\h.\f.f(SSSSdeghf)) S
SSSS    =(         \d.\e.\g.\h.\f.f(SSSSdeghf))
SSSS    =(      \c.\d.\e.\g.\h.\f.f(SSScdeghf)) S
SSS     =(      \c.\d.\e.\g.\h.\f.f(SSScdeghf))
SSS     =(   \b.\c.\d.\e.\g.\h.\f.f(SSbcdeghf)) S
SS      =(   \b.\c.\d.\e.\g.\h.\f.f(SSbcdeghf))
SS      =(\a.\b.\c.\d.\e.\g.\h.\f.f(aabcdeghf)) S
S       =(\a.\b.\c.\d.\e.\g.\h.\f.f(aabcdeghf))
T=SSSSSSSS
 =(\a.\b.\c.\d.\e.\g.\h.\f.f(aabcdeghf))^8

the same technique applied to 26 variables gives:

T=SSSSSSSSSSSSSSSSSSSSSSSSSS
S=\a.\b.\c.\d.\e.\f.\g.\h.\i.\j.\k.\l.\m.\n.\o.\p.\q.\s.\t.\u.\v.\w.\x.\y.\z.\r.r(thisisafixedpointcombinator)

as with all turing-like combinators, the last variable (r in this example) is special; all the other 25 variables have equal status;

for similar reason why we patch y combinator into z combinator, turing-like combinators also need patch for eager evaluation:

def exam11():

    TY = (
        (lambda x : (lambda y : (y(x(x)(y)))))
        (lambda x : (lambda y : (y(x(x)(y)))))
    )

    TZ = (
        (lambda x : (lambda y : (y(lambda v: x(x)(y)(v)))))
        (lambda x : (lambda y : (y(lambda v: x(x)(y)(v)))))
    )

    G = lambda g : (
        lambda n : (
            1 if n == 0 else n * g(n-1)
        ))
#    F = TY(G)   # boom!
    F = TZ(G)
    assert F(4) == 24

session 12: summary

this session gives a summary of y combinators and z combinators;

def exam12():

    def Y2(x):
        A = (
        lambda a : (
            x((a)(a))
        ))
        return (A)(A)

    def Z2(x):
        A = (
        lambda a : (
            x(lambda v: (a)(a)(v))
        ))
        return (A)(A)

    def Y4(x):
        A = (
        lambda a : (
        lambda b : (
        lambda c : (
            x((a)(a)(b)(c))
        ))))
        return (A)(A)(A)(A)

    def Z4(x):
        A = (
        lambda a : (
        lambda b : (
        lambda c : (
            x(lambda v: (a)(a)(b)(c)(v))
        ))))
        return (A)(A)(A)(A)

    def Y8(x):
        A = (
        lambda a : (
        lambda b : (
        lambda c : (
        lambda d : (
        lambda e : (
        lambda f : (
        lambda g : (
            x((a)(a)(b)(c)(d)(e)(f)(g))
        ))))))))
        return (A)(A)(A)(A)(A)(A)(A)(A)

    def Z8(x):
        A = (
        lambda a : (
        lambda b : (
        lambda c : (
        lambda d : (
        lambda e : (
        lambda f : (
        lambda g : (
            x(lambda v: (a)(a)(b)(c)(d)(e)(f)(g)(v))
        ))))))))
        return (A)(A)(A)(A)(A)(A)(A)(A)

    G = lambda g : (
        lambda n : (
            1 if n == 0 else n * g(n-1)
        ))
#    F = Y2(G)   # boom!
#    F = Z2(G)
#    F = Y4(G)   # boom!
#    F = Z4(G)
#    F = Y8(G)   # boom!
    F = Z8(G)
    assert F(4) == 24

session 13: racket implementation

the racket language allows lazy evaluation; so we can build some examples to test y combinators that do not work in python; in addition, we test z combinators work with lazy evaluation too;

#lang lazy

;Y2=\x.(  (\a.x(aa))^2  )
(define Y2 (
    λ(x)
    (
        (λ(a) (x (a a)))
        (λ(a) (x (a a)))
    )
))

;Z2=\x.(  (\a.x(\v.aav))^2  )
(define Z2 (
    λ(x)
    (
        (λ(a) (x (λ(v) ((a a) v))))
        (λ(a) (x (λ(v) ((a a) v))))
    )
))

;Y4=\x.(  (\a.\b.\c.x(aabc))^4  )
(define Y4 (
    λ(x)
    (((
    (λ(a) (λ(b) (λ(c) (x (((a a) b) c) ))))
    (λ(a) (λ(b) (λ(c) (x (((a a) b) c) )))))
    (λ(a) (λ(b) (λ(c) (x (((a a) b) c) )))))
    (λ(a) (λ(b) (λ(c) (x (((a a) b) c) )))))
))

;Z4=\x.(  (\a.\b.\c.x(\v.aabcv))^4  )
(define Z4 (
    λ(x)
    (((
    (λ(a) (λ(b) (λ(c) (x (λ(v) ((((a a) b) c) v))))))
    (λ(a) (λ(b) (λ(c) (x (λ(v) ((((a a) b) c) v)))))))
    (λ(a) (λ(b) (λ(c) (x (λ(v) ((((a a) b) c) v)))))))
    (λ(a) (λ(b) (λ(c) (x (λ(v) ((((a a) b) c) v)))))))
))

;Y8=\x.(  (\a.\b.\c.\d.\e.\f.\g.x(aabcdefg))^8  )
(define Y8 (
    λ(x)
    (((((((
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (((((((a a) b) c) d) e) f) g)))))))))
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (((((((a a) b) c) d) e) f) g))))))))))
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (((((((a a) b) c) d) e) f) g))))))))))
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (((((((a a) b) c) d) e) f) g))))))))))
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (((((((a a) b) c) d) e) f) g))))))))))
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (((((((a a) b) c) d) e) f) g))))))))))
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (((((((a a) b) c) d) e) f) g))))))))))
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (((((((a a) b) c) d) e) f) g))))))))))
))

;Z8=\x.(  (\a.\b.\c.\d.\e.\f.\g.x(\v.aabcdefgv))^8  )
(define Z8 (
    λ(x)
    (((((((
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (λ(v) ((((((((a a) b) c) d) e) f) g) v))))))))))
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (λ(v) ((((((((a a) b) c) d) e) f) g) v)))))))))))
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (λ(v) ((((((((a a) b) c) d) e) f) g) v)))))))))))
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (λ(v) ((((((((a a) b) c) d) e) f) g) v)))))))))))
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (λ(v) ((((((((a a) b) c) d) e) f) g) v)))))))))))
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (λ(v) ((((((((a a) b) c) d) e) f) g) v)))))))))))
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (λ(v) ((((((((a a) b) c) d) e) f) g) v)))))))))))
    (λ(a) (λ(b) (λ(c) (λ(d) (λ(e) (λ(f) (λ(g) (x (λ(v) ((((((((a a) b) c) d) e) f) g) v)))))))))))
))

;main
(define Fact (
;    Y2 (λ(fact) (λ(n) (if (zero? n) 1 (* n (fact (- n 1))))))
;    Y4 (λ(fact) (λ(n) (if (zero? n) 1 (* n (fact (- n 1))))))
    Y8 (λ(fact) (λ(n) (if (zero? n) 1 (* n (fact (- n 1))))))
;    Z2 (λ(fact) (λ(n) (if (zero? n) 1 (* n (fact (- n 1))))))
;    Z4 (λ(fact) (λ(n) (if (zero? n) 1 (* n (fact (- n 1))))))
;    Z8 (λ(fact) (λ(n) (if (zero? n) 1 (* n (fact (- n 1))))))
))

(Fact 4)
(Fact 40)

session 14: build a jet

this session shows how to build a jet with fixed-point combinators; well, it actually builds a fixed-point combinator with jets; the code explains itself (on a wide screen);

(define JET
  (lambda(a)
    (lambda(b)
      (lambda(c)
        (lambda(d)
          (lambda(e)
            (lambda(f)
              (lambda(g)
                (lambda(h)
                  (lambda(i)
                    (lambda(j)
                      (lambda(k)
                        (lambda(l)
                          (lambda(m)
                            (lambda(n)
                              (lambda(o)
                                (lambda(p)
                                  (lambda(q)
                                    (lambda(s)
                                      (lambda(t)
                                        (lambda(u)
                                          (lambda(v)
                                            (lambda(w)
                                              (lambda(x)
                                                (lambda(y)
                                                  (lambda(z)
                                                    (lambda(r)
                                                      (r ((((((((((((((((((((((((((t h) i) s) i) s) a) f) i) x) e) d) p) o) i) n) t) c) o) m) b) i) n) a) t) o) r)))))))))))))))))))))))))))))


(define RUNWAY (((((((((((((((((((((((((JET JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET) JET))

i am not lying: RUNWAY is a fixed-point combinator built with racket jets;

go project structure

Sat, 01 Oct 2022 07:13:43 -0700

the package clause and import declarations in go are a bit mysterious; while it is very easy to compile and run one file, it is not obvious how to properly organize a go project with multiple files for someone with no experience; the go documentation does not have a dedicated section explaining the facts straightforward; here is a brief description of the concepts involved with some working examples; well, the description may not be accurate, but the examples should work; the examples assume we are building/running an executable;

one file

the simplest case is when we put everything in one file; this file must have main as both package name and entry function name;

//  main.go
package main

import "fmt"

func main() {
    fmt.Println("hello")
}

to build the executable, run go build on this file:

go build main.go

multiple files in one package

Go programs are organized into packages. A package is a collection of source files in the same directory that are compiled together. Functions, types, variables, and constants defined in one source file are visible to all other source files within the same package.

this basically says a package can span multiple files in the same directory; in fact go does not allow multiple packages in the same directory, so a package is kind of one-to-one mapping to a directory;

now we can add another file foo.go to the same package in the same directory as main.go; because they are in the same package, we can define something in foo.go and use it in main.go; in this example, we define a function get in foo.go and use it in main.go; the name get starts with a lowercase letter; there is no need to export it because we use it in the same package (albeit different file);

//  foo.go
package main

func get() int {
    return 1234
}

//  main.go
package main

import "fmt"

func main() {
    fmt.Println("hello", get())
}

to build executable main, run go build on all these files and put main.go at first (otherwise the executable would be foo);

go build main.go foo.go

multiple packages in one module

A repository contains one or more modules. A module is a collection of related Go packages that are released together. A Go repository typically contains only one module, located at the root of the repository. A file named go.mod there declares the module path: the import path prefix for all packages within the module. The module contains the packages in the directory containing its go.mod file as well as subdirectories of that directory, up to the next subdirectory containing another go.mod file (if any).

…

An import path is a string used to import a package. A package’s import path is its module path joined with its subdirectory within the module. For example, the module github.com/google/go-cmp contains a package in the directory cmp/. That package’s import path is github.com/google/go-cmp/cmp. Packages in the standard library do not have a module path prefix.

The PackageName is used in qualified identifiers to access exported identifiers of the package within the importing source file. It is declared in the file block. If the PackageName is omitted, it defaults to the identifier specified in the package clause of the imported package. If an explicit period (.) appears instead of a name, all the package’s exported identifiers declared in that package’s package block will be declared in the importing source file’s file block and must be accessed without a qualifier.

The interpretation of the ImportPath is implementation-dependent but it is typically a substring of the full file name of the compiled package and may be relative to a repository of installed packages.

simply speaking, a go.mod file in the top-level directory makes a module that contains all packages in its directory tree; the module path is a prefix of the import path; the import path says what to import (in an implementation-dependent way);

here we init the module with an arbitrary module path a/b/c;

go mod init a/b/c

the directory structure looks like this:

.
├── go.mod
├── lib
│   └── foo.go
└── main.go

the files are almost the same as before; we changed get to Get to export it because now we are using it in a different package:

//  go.mod
module a/b/c

go 1.18

//  lib/foo.go
package lib

func Get() int {
    return 1234
}

//  main.go
package main

import "fmt"
import "a/b/c/lib"

func main() {
    fmt.Println("hello", lib.Get())
}

to build this module, run go build in top-level directory:

go build .

it is worth noting that the package name does not have to be its directory name or one of its file names; the package name is merely like a namespace to qualify identifiers exported by the imported package; for example, we can change package name from lib to libx and things still work:

//  lib/foo.go
package libx

func Get() int {
    return 1234
}

//  main.go
package main

import "fmt"
import "a/b/c/lib"

func main() {
    fmt.Println("hello", libx.Get())
}

we are not changing import path a/b/c/lib because its relative part lib must match actual filesystem path; this should have made things clear: we import the relative path (lib) of a package prefixed by the module path (a/b/c), then refer to the imported package (in the importing package) using its package name (lib, libx, etc.); the import statement can also specify an explicit package name which overrides the default package name given in package clause of the imported package;

references

stackless coroutine

Sat, 17 Sep 2022 06:24:02 -0700

coroutines are program instructions (eg: functions) whose execution can be suspended and resumed; coroutines can have a full stack (ie: stackful) or only one stack frame (ie: stackless); typically, stackless coroutines are much more performant than stackful coroutines (like 10-100x faster), and they are the topic in this article;

stateless and stateful functions

normally, the functions we write are stateless; they are subroutines with one entry point and one exit point; the function can return a value when it exits; in fact functions can have multiple exit points, but without loss of generality we only consider those with exactly one exit point, which is usually a return statement placed at the end the function; we say such functions are “stateless” because every time we call the function, it starts running from its entry point, and runs until its exit point; the function does not remember its execution state between multiple invocations;

on the other hand, stateful functions can suspend itself in the middle of its execution (optionally yielding a value), and they remember where they suspended so that next time it is invoked it can resume from the latest suspension point;

this pseudo c example compares stateless and stateful functions:

int stateless() {
    return 0;
    return 1;
    return 2;
    return 3;
}

stateless();    //  output 0;
stateless();    //  output 0;
stateless();    //  output 0;
stateless();    //  output 0;

int stateful() {
    yield 0;
    yield 1;
    yield 2;
    yield 3;
}

stateful();     //  output 0;
stateful();     //  output 1;
stateful();     //  output 2;
stateful();     //  output 3;

while function stateless() can only return the same value each time it is called, function stateful() can suspend itself at each yield statement and resume its execution from where it left off, producing four different values;

stateful functions in real c

there are no yield statements in c; why do we give an example in pseudo c? because we will implement this yield statement in real c; we begin with an implementation of stateful functions without yield, then define yield as a macro encapsulating the mechanisms we use in implementing stateful functions;

c has a mechanism where you can declare a static variable inside a function, and that variable maintains its value across multiple calls of this function; this is how we keep function state between multiple calls; another c mechanism is goto statement which allows free-style jumps within a function; this is used to resume from the latest suspension point;

int stateful() {
    static int state = 0;
    switch (state++) {
        case 0:
            goto point0;
        case 1:
            goto point1;
        case 2:
            goto point2;
        case 3:
            goto point3;
    }
    point0: return 0;
    point1: return 1;
    point2: return 2;
    point3: return 3;
}

stateful();     //  output 0;
stateful();     //  output 1;
stateful();     //  output 2;
stateful();     //  output 3;

stackless coroutines are stateful functions

are you aware of it? we have just implemented a stackless coroutine! this is because stackless coroutines are stateful functions; there are exceptions, for example, when the coroutine is not made of a function in high-level language, but those are rare cases;

our implementation discloses the internal structure of a stackless coroutine: there is a state variable maintained across multiple function calls, which is used to jump directly to the latest suspension point; it is like the function has been cut into several segments, and each invocation runs one segment;

implement `yield` statement

we do not want to write a switch statement for every stackless coroutine we define; it would become infeasible to maintain such a statement; so we will shape our program into a more ergonomic form with the same functionality;

the first thing we do is to assign a constant to each state instead of self-incrementing the state variable; we do not always increment the state every time we call it, because multiple states may be merged into one (eg: when the function has loops); plus, this makes it easy to turn these statements into macros; note that the state is set to the next one before each suspension point, so that it has the correct value when the function is resumed;

int stateful() {
    static int state = 0;
    switch (state) {
        case 0:
            goto point0;
        case 1:
            goto point1;
        case 2:
            goto point2;
        case 3:
            goto point3;
    }
    point0: state = 1; return 0;
    point1: state = 2; return 1;
    point2: state = 3; return 2;
    point3: state = 4; return 3;
}

the second thing we do is to turn goto labels into case labels, leveraging the device invented by tom duff; this allows us to eliminate goto statements from the function by inlining the goto statement body; the compiler might give a warning about non-void function does not return a value; this does not matter in this example, because we will not call a coroutine that is done;

int stateful() {
    static int state = 0;
    switch (state) {
        case 0:
            state = 1; return 0;
        case 1:
            state = 2; return 1;
        case 2:
            state = 3; return 2;
        case 3:
            state = 4; return 3;
    }
}

the third thing we do is to combine a case label with its previous return statement, to facilitate writing the macro; we also add an extra case label after the last return statement to make things consistent; the function body starts looking hacky and wacky; do you see macros in it?

int stateful() {
    static int state = 0; switch (state) { case 0:;
        state = 1; return 0; case 1:;
        state = 2; return 1; case 2:;
        state = 3; return 2; case 3:;
        state = 4; return 3; case 4:;
    }
}

the last thing we do is to turn those highly similar statements into macros:

#define begin static int state = 0; switch (state) { case 0:
#define yield(i, n) do { state = i; return n; case i:; } while (0)
#define end }

int stateful() {
    begin;
    yield(1, 0);
    yield(2, 1);
    yield(3, 2);
    yield(4, 3);
    end;
}

the greatness about this piece of code is that everyone who has seen it cannot help calling it “HACK!” yet it compiles and gives the correct output; look, we have implemented the yield statement (as a macro) and it works in a stateful function (ie: stackless coroutine); in fact, this works even when the function gets more complicated (eg: having loops); there is a great article by simon tatham giving more complicated examples;

to summarize, this is how we transformed pseudo yield statement into a real implementation:

function variables whose state needs to be kept are declared static;
convert yield to return;
set the state variable immediately before return, to the next state;
add a case label immediately after return;
call switch at function entry to jump to case label for current state;

it is worth noting that this construction works with nested stateful functions; however, if a stateful function is used multiple times, then its state needs to be re-initialized before each use; this is not possible when there is only one copy of state (eg: in a static variable); hence we need to allocate this state on heap in an activation frame for each function invocation;

activation frame

talks about coroutines often involve a concept named activation frame; this concept is a generalization of stack frame; while a stack frame (of a normal function) is allocated on the stack, an activation frame (of a coroutine) is often allocated on the heap, because this activation frame needs to be kept across coroutine yield points; a yield in stackless coroutine is like a return, and its stack frame will be discarded, but its activation frame on the heap can be preserved; more details can be found in this article; in our example we simply used a static variable as the activation frame to preserve coroutine state, but allocating the activation frame on heap is the norm in language implementations;

async/await pattern

in languages that employ stackless coroutines, we can often see the async/await pattern; the async keyword declares a function as asynchronous; the await keyword suspends an asynchronous function if necessary, waiting for it to finish before returning its result;

an async function is basically a stackless coroutine; there is a rule about async functions that often looks puzzling: the caller of an async function must also be an async function itself; that is, a sync function cannot call an async function, but an async function can call a sync function; people frequently ask why this has to be the case; if you remember stackless coroutines are basically stateful functions, then you understand async functions are more “advanced” than sync functions because they can keep their state; when a call chain consists solely of “advanced” async functions, suspend/resume work fine because each one of them can suspend/resume; having one sync function in the chain breaks the chain, losing the advantage of being able to suspend/resume, because that sync function cannot remember its state;

the await of an async function involves a concept named future, promise or delay; this is a proxy to something that is to be resolved later; an async function itself is an example of a future: the return (not yield) value of an async function is resolved only when it returns; in other cases, a future could be the response of a network request, the blob of a database read, etc.; when an async function awaits a future, it binds itself to the future and suspends itself; when the future is later resolved, this async function is resumed, with the resolved future value as return value of the await statement;

lua pitfalls

Fri, 09 Sep 2022 02:15:29 -0700

quirks can (and will) come around when migrate from one language to another, as languages often have different interpretations and implementations; it would be helpful if prominent inconsistencies are logged to avoid going over the getting started chapters again and again; this article is a list of pitfalls for people migrating to lua (from an unknown whatever language); in spite of word “pitfall” in title, this article does not mean to be a verdict about better or worse, but just poses the differences as they are;

here the lua reference in use is Programming in Lua (first edition) that aims at lua 5.0; these days most people would be on lua 5.4, but we are getting started, right? for those who want the latest language reference visit the page Lua 5.4 Reference Manual;

code examples below are in lua 5.4; so the code may not entirely conform to the lua reference in use;

each item has a series of ref tags that can be followed to find authentic information:

(ref) a chunk can be a single line in interactive mode;
(ref) global variables do not need to be declared or deleted; they can be assigned, and they are existent iff non-nil;
(ref) lua is case-sensitive;
(ref) lua has 8 basic types: nil, boolean, number, string, userdata, function, thread, and table;

(ref) nil is different from anything but nil:

> print(nil == nil)
true
> print(nil == false)
false
> print(nil == 0)
false
> print(nil == '')
false
> print(nil == {})
false

(ref) false and nil are false, everything else is true; zero 0 is true; empty string '' is true;
(ref) lua numbers are all (double) floats; lua has no integer type, because lua floats have no rounding error upto 100,000,000,000,000;
(ref) strings are immutable;
(ref) single-quoted and double-quoted literal strings are the same;

(ref) lua has automatic conversions between numbers and strings:

> print(type("12" + 34), "12" + 34)
number  46
> print(type("12" .. 34), "12" .. 34)
string  1234

(ref) use tonumber and tostring for manual conversions between numbers and strings;
(ref) lua table is associative array (aka: dict, map);
(ref) all types except nil can be used as table key (aka: index); and key types can be mixed;
(ref) table values evaluate to nil if not initialized;
(ref) use table with integer keys to implement conventional array;
(ref) use ipairs to iterate conventional array;
(ref) lua supports functional programming; functions are first-class citizens;
(ref) relational operators always result in true or false;
(ref) lua inequality operator is ~= (not !=);

(ref) values of different types are not equal:

> print(0 == '0')
false
> print(0 == '')
false
> print(0 == {})
false
> print(0 == false)
false
> print(0 == nil)
false

(ref) lua compares tables, userdata, and functions by reference:

> a = {}
> b = {}
> c = a
> a == b
false
> a == c
true

(ref) logical operators are and, or, not;
(ref) logical operators consider false and nil as false and anything else as true;
(ref) both and and or use shortcut evaluation;
(ref) lua lacks ternary operator: a and b or c is bug;

(ref) operator not always returns true or false:

> print(not nil)
true
> print(not false)
true
> print(not 0)
false
> print(not '')
false
> print(not {})
false

(ref) one can use (not not ...) to convert any value to boolean;

(ref) operator not has higher precedence than relational operators (eg: ==):

> print(not not 0 == 0)
false
> print((not not 0) == 0)
false
> print(not not (0 == 0))
true

(ref) table constructor can mix explicit and implicit indexes:

> x = {'a', 'b', [8]='c', 'd', 'e'}
> x[1]
a
> x[2]
b
> x[3]
d
> x[4]
e
> x[8]
c

(ref) mixing explicit and implicit indexes in table constructor can cause trouble:

> x = {'a', 'b', [2]='c', 'd', 'e'}
> x[1]
a
> x[2]
b
> x[3]
d
> x[4]
e
> x[2]
b

(ref) in multiple assignment, values are evaluated then assigned; so swap works:

> x, y = 3, 8
> x, y = y, x
> print(x, y)
8       3
> a = {4, 9}
> a[1], a[2] = a[2], a[1]
> print(a[1], a[2])
9       4

(ref) in multiple assignment, values are adjusted to variables; extra variables are assigned nil; extra values are discarded;
(ref) local variables are created with keyword local;

(ref) local variables have block scope; the scope begins after the declaration and goes to end of block;

local x = 1
for i = 1, 2 do
    print(x)
    local x = i * 2
    print(x)
end

(ref) local variable declaration with assignment has same rule: extra variables are assigned nil; extra values are discarded;
(ref) local variable declaration without assignment initializes all variables with nil;
(ref) one can create an explicit block with do-end:
(ref) condition expression in if, while, etc. may be any value: false and nil are treated as false, others true;
(ref) numeric for: all three expressions are evaluated only once;
(ref) numeric for: loop variables are local variables visible only inside the loop;
(ref) numeric for: never change the value of loop variables;
(ref) generic for: loop variables are local variables visible only inside the loop;
(ref) generic for: never change the value of loop variables;
(ref) generic for: use ipairs to traverse conventional arrays; use pairs to traverse associative arrays;
(ref) there is an implicit return at the end of any function that returns nothing;
(ref) one can use colon : for object-oriented calls (aka: methods): o:foo(x) is the same as o.foo(o, x);
(ref) function parameters work like local variables, initialized with function arguments;
(ref) extra function parameters are assigned nil; extra function arguments are discarded;
(ref) lua lacks default arguments;
(ref) functions can return multiple results; the number of results depend on the circumstances of the call;
(ref) a function call used as an expression gives only the first result;
(ref) a function call gives all results only when it is the last expression in a list of expressions;
(ref) function results are appended with nil as needed;
(ref) table constructor collects all results from a call, without any adjustments;
(ref) one can force a call to return exactly one result using an extra pair of parentheses:
(ref) return f() returns all values returned by f;
(ref) return (f()) returns one single value;
(ref) unpack takes an array and returns all its elements as results (starting from 1);
(ref) three dots ... parameter collects a variable number of arguments into hidden table arg;
(ref) a function can have some fixed parameters plus a variable number of parameters;
(ref) lua lacks named arguments; simulate that by passing a table argument;

(ref) functions are anonymous:

function f (x) return x+1 end   --same
f = function (x) return x+1 end --same

(ref) local function syntactic sugar handles recursive functions, but not indirect recursive functions;

hsl and hsv

Sat, 27 Aug 2022 14:25:19 -0700

hsl and hsv are two widely used color models that align more closely with human vision perception of colors; their use can be found in imaging software, camera tools, web standards, and many other applications; there are simple formulas to convert between rgb and hsl/hsv color models, yet very few documents explaining the mathematics behind those formulas in full details; so, this article aims to be a comprehensive description of these models and conversions between relative color models;

motivation

why did people invent hsl and hsv color models? where does rgb color model fall short? imagine you are drawing a picture with a blue-ish color #0080ff; after some review you would like to adjust its lightness and fullness, while keep the same blue-ish; if you are not a color expert, you get frustrated with the color slides of rgb components because when you change them the color no longer looks the same as before; when you start using hsl/hsv color models, this job becomes much easier: you can fix hue and adjust saturation and lightness/value, which gives a variety of colors, all having the same blue-ish as before; this is an example of more intuitive color picking and mixing;

background

the discussion below is slightly complicated so we need some background as well as convention; first, we let each rgb component be a float in range [0.0, 1.0] and assume computation has arbitrary precision; second, we order rgb components as (R, G, B); third, we arrange rgb gamut in a unit cube, within a coordinate system where black is at origin and white is at (1.0, 1.0, 1.0) unless we use an alternative coordinate system; fourth, we denote the vertices of the cube by a single letter as such: black(K), red(R), green(G), blue(B), cyan(C), magenta(M), yellow(Y), white(W); fifth, we will discuss hsv before hsl as we feel hsv formula is easier to derive and understand; sixth, we omit a corner case (eg: divide by zero) at various places where it simplifies analysis so you are advised to compare this article against the wikipedia article for completeness;

hsv

hsv stands for hue, saturation, value:

hue gives the “tone” of the color (eg: reddish, yellowish, etc.);
saturation gives the “fullness” of the color; high saturation feels vibrant while low saturation feels pale;
value gives the “brightness” of the color; this is the amount of maximum component (will be defined soon);

look at these pictures to understand how they vary:

the left picture is an intermediary result (hue-chroma-value, hcv); the right picture is the final result (hue-saturation-value, hsv); as you can see, hsv arranges colors in a cylinder; as we will see, this cylinder can be converted from the rgb unit cube, with the hcv cone as an intermediary result (video);

hue and chroma

now we mathematically define the hue in hsv; we first define chroma; chroma is a value of “colorfulness” and it is closely related to saturation;

assume we have rgb color $(R, G, B)$; we define:

maximum component $M_2=M=\text{max}(R, G, B)$;
minimum component $M_0=m=\text{min}(R, G, B)$;
middle component $M_1$ as the component other than $M_2$ and $M_0$;
chroma $C=M_2-M_0$;
hue offset $H’’ =(M_1-M_0)/(M_2-M_0)$; $H’’ =0$ if $M_2=M_0$;

differences from wikipedia:

wikipedia uses two symbols $M$ and $m$, we use three symbols $M_2$, $M_1$ and $M_0$ because we need an explicit middle component;
wikipedia uses a piecewise definition for hue, we use a concise formula to compute hue offset; this formula works in all pieces; later on, we will compute hue using hue offset;
wikipedia defines $H=60^{\circ} \times H’$, we do not distinct $H$ from $H’$ because they are essentially the same thing, only with different units and ranges;

the picture below visualizes these definitions:

this is a very important picture showing the rgb-to-hsv conversion; we first tilt the rgb unit cube so that black is at origin and white is vertically above it; then we project it onto a “chromaticity plane” perpendicular to the black-white axis; the projection is a hexagon as shown; why is it a hexagon? it is easily seen from symmetry that R, G, B form an equilateral triangle; so do C, M, Y; and ROC, GOM, BOY are collinear;

this picture gives intuition about the definitions of chroma and hue:

chroma is the proportion of the distance from the origin to the edge of the hexagon; it is the same as the ratio of the radii of the two hexagons;

think chroma as “magnitude”;
hue is roughly the angle we walk counterclockwise from red ($0^{\circ}$) to the point; “roughly” means there is a very small error introduced by chords and arcs, as we shall see;

think hue as “phase”;

this picture shows an interesting property: adding or subtracting the same amount from all three of R, G, B components does not change its projection, and thus does not change chroma or hue; that is, given any point in the cube, we can always move it down vertically until it lies on one of the three lower faces of the cube; all points on these faces have at least one zero component;

this picture shows another interesting property: the projection is divided evenly into 6 areas (quadrants, but for 6, or sextants): ORY (area 0), OYG (area 1), OGC (area 2), OCB (area 3), OBM (area 4), OMR (area 5); in each area, the order of R, G, B components are uniquely determined; for example, if a point has its projection in area OCB, then we know $B \ge G \ge R$:

when we move it vertically down, we are subtracting the same amount from all three components; then it crossed face OGB, which means red component came to zero first, so the red component is the smallest;
its projection is closer to blue than green, so the point on face OGB must have a larger blue component than green component; adding the same amount of red, green and blue maintains this comparison result;

finally we can start computing chroma and hue; let the point be $(R, G, B)$; wlog we assume it is an area 3 (OCB) point; from the above observations, we know $B \ge G \ge R$; so we can use point Q at $(0, G-R, B-R)$ to compute the same chroma and hue; we know Q lies on cube face OGB;

to compute chroma $C$, we extend OQ to intersect segment BC at a point Q'; we know OP/OP' = OQ/OQ' because QP is parallel to Q'P'; if we project Q and Q' onto edge OB at points T and T' respectively, then we know OQ/OQ' = OT/OT'; but T' is exactly B, OT' has length $1$, and OT has length $B-R$ (blue component of Q); so $C=B-R=M_2-M_0$;

do this in other areas and you will find $C=M_2-M_0$ always holds;
to compute hue, we first compute hue offset $H’’$; the hue offset measures how far the input point is away from its closest primary vertex; there are three primary vertices R, G, B; when input point is in area 3, its closest primary vertex is B; so the hue offset is BP'/BC=BQ'/BC (B and C in the right hand side refer to vertices in the cube, not the plane); because B is $(0, 0, 1)$, C is $(0, 1, 1)$, Q' is $(0, (G-R)/(B-R), 1)$ (computed from Q), we have
$(0, 0, 1) + H''((0, 1, 1) - (0, 0, 1)) = (0, (G-R)/(B-R), 1) \\ (0, 0, 1) + (0, H'', 0) = (0, (G-R)/(B-R), 1) \\ (0, H'', 1) = (0, (G-R)/(B-R), 1) \\ H'' = (G-R)/(B-R) = (M_1-M_0)/(M_2-M_0) \\$
to compute hue from hue offset, we first notice the point lies in area 3, so the closest primary vertex is B and we should go clockwise (subtract); now we subtract hue offset from the angle of B ($240^{\circ}$, or $4$ sextants); in this example, $H’’ =2/3$, so we compute hue as $H=(4-2/3) \times 60^{\circ}=200^{\circ}$; the hue offset captures the amount of middle component relative to maximum component (with pure white light removed); and you can verify our hue computed from hue offset matches the wikipedia piecewise function in every piece;

finally, our computation here is based on a hexagon not a circle; and the hue offset is a ratio of chords not arcs; for the hsv color space to be a cylinder, an additional warping that maps linearly each chord into an arc turns the hexagon into a circle; this still maps a hsv tuple to the same color, but only changes the shape of the hsv color space; we will omit this detail afterwards;

value

the definition of value in hsv is very simple: it is the value of maximum component:

value $V=M_2$;

for example, for a blue-ish color (blue is the maximum component), value tells how much blue there is in it;

this definition gives a “hexcone” model: all three primary colors RGB, all three secondary colors CMY, and white color W are placed in a plane at the top of tilted cube; visually, this transformation is like opening an umbrella from top to bottom; or you can think the cube as an onion with its core at the black origin K and is peeled layer by layer from top to bottom, with each layer composed by three quadrants perpendicular to each other;

saturation

saturation is closely related to chroma $C$:

saturation $S=C/V$;

this means saturation is chroma scaled into range [0.0, 1.0]; to see this, recall $C=M_2-M_0$ and $V=M_2$, so $0 \le C \le V$; dividing $C$ by $V$ does the scale; the corner case is when $V=0$, then $S$ is defined trivially as $0$;

replacing chroma with saturation expands the bottom of the hexcone to turn into a cylinder;

hsv to rgb

here we have finished the conversion from rgb to hsv; now we talk about the other direction: hsv to rgb;

the hsv-to-rgb conversion steps are:

compute $C=V \times S$; then $M_2-M_0=C$;
compute $M_2=V$; then $M_0=V-C$;
compute hue offset $H’’$; this computation can be done by looking at the chromaticity plane to decide which of R, G, B is the closest primary vertex: use R if area 0 or 5, G if area 1 or 2, B if area 3 or 4; then the computation of $H’’$ is simply an addition or subtraction of $H$ and the angle of the primary vertex;

once we have $H’’ =(M_1-M_0)/(M_2-M_0)$, we can compute $M_1$;
now we have all three components; use $H$ to find their order (ie: map $\{M_0,M_1,M_2\}$ to $\{R, G, B\}$);

hsv to rgb (optimized)

the above algorithm is easy to comprehend, but may not be the best for machines to run; to develop an optimized algorithm for machines, we observe this fact: for fixed $S$ and $V$ and varying $H$, each RGB component range is limited to $[V-VS, V]$; to see this, let us take green component as an example:

when area is 4 or 5, green component is the minimum component $M_0$, so it is equal to $M_2-(M_2-M_0)=V-C=V-VS$;
when area is 1 or 2, green component is the maximum component $M_2$, so it is equal to $V$;
when area is 0 or 3, green component is the middle component $M_1$, so it is between $M_0$ and $M_2$, that is, in range $[V-VS, V]$;

in this case there is one more observation: $H’’$ is linear in $M_1$ because $M_0$ and $M_2$ are fixed, so $H$ is linear in $M_1$ because $H’’$ and $H$ only differ by a constant; adjacent areas are obviously continuous (when $H$ is gradually increased, the point moves continuously), so we can connect the endpoints of its two adjacent areas with a straight line (area 2 and 4 for area 3, area 5 and 1 for area 0);

now we get a function mapping $H$ to green component $G$ which has a flat top, a flat bottom, and two slopes; it looks like this (green line):

this function can be realized with clever use of $\min$ and $\max$; someone has worked it out as (plot):
$f(n)=V-VS\max(0,\min(k,4-k,1)) \\ k=(n+{\frac {H}{60^{\circ }}}){\bmod {6}} \\ (R,G,B)=(f(5),f(3),f(1)) \\$
as you can see, red, green, blue components work in the same way, only with different shifts;

hsl

hsl stands for hue, saturation, lightness:

hue gives the “tone” of the color (eg: reddish, yellowish, etc.);
saturation gives the “fullness” of the color; high saturation feels vibrant while low saturation feels pale; note that saturation in hsv and hsl have a similar meaning but are computed differently: saturation is chroma divided by max chroma in both hsv and hsl, but that max chroma (ie: divisor) is value in hsv, and a function on lightness in hsl; where necessary, we use $S_V$ for saturation in hsv, and $S_L$ for saturation in hsl;
lightness gives the “lightness” of the color; its difference from value in hsv is that, value takes the amount of one component (maximum component), while lightness takes the average of two (maximum and minimum components);

look at these pictures to understand how they vary:

the left picture is an intermediary result (hue-chroma-lightness, hcl); the right picture is the final result (hue-saturation-lightness, hsl); as you can see, hsl arranges colors in a cylinder; as we will see, this cylinder can be converted from the rgb unit cube, with the hcl bicone as an intermediary result (video);

hue and chroma

hue and chroma in hsl are defined exactly the same as in hsv;

lightness

the definition of lightness in hsl is the average of maximum and minimum components:

lightness $L=(M_2+M_0)/2$;

this definition gives a “bi-hexcone” model: all three primary colors RGB, all three secondary colors CMY, and white color W are placed in a plane at the half of tilted cube; if you divide the cube into many tiny cubes arranged in three dimensions, and calculate lightness of the outermost ones (the ones on its six faces), you will find the amount of lightness is a pyramid in the three upper faces, and an inverted pyramid in the three lower faces; when you strip these tiny cubes on the surface, you are left with a smaller tilted cube possessing a similar structure (eg: $(255, 0, 0)$ becomes $(254, 1, 1)$, $(255, 0, 255)$ becomes $(254, 1, 254)$, etc.); since $255+0=254+1$, the six corners RGBCMY of the smaller cube are on the same plane as those on the original cube; this suggests the bi-hexcone is pretty much like our earth: RGBCMY are aligned on the equator, white is the north pole and black is the south pole;

saturation

saturation is closely related to chroma $C$:

saturation $S=C/(1-|2L-1|)=C/(2\min(L,1-L))$;

this means saturation is chroma scaled into range [0.0, 1.0]; to see this, recall $C=M_2-M_0$ and $L=(M_2+M_0)/2$; the lower bound of $C$ is $0$ (equal when $M_2=M_0$); the upper bound of $C$ is $2L$ if $2L \le 1$ (equal when $M_0=0, M_2=2L$), or $2-2L$ if $2L \ge 1$ (equal when $M_0=2L-1, M_2=1$);

the corner case is when $L=0$ or $L=1$, then $S$ is defined trivially as $0$;

replacing chroma with saturation expands the top and bottom of the bi-hexcone to turn into a cylinder;

hsl to rgb

here we have finished the conversion from rgb to hsl; now we talk about the other direction: hsl to rgb;

the hsl-to-rgb conversion steps are:

compute $C=2\min(L,1-L) \times S$; then $M_2-M_0=C$;
we know $M_2+M_0=2L$; so $M_2=L+C/2$ and $M_0=L-C/2$;
compute hue offset $H’’$; this computation can be done by looking at the chromaticity plane to decide which of R, G, B is the closest primary vertex: use R if area 0 or 5, G if area 1 or 2, B if area 3 or 4; then the computation of $H’’$ is simply an addition or subtraction of $H$ and the angle of the primary vertex;

once we have $H’’ =(M_1-M_0)/(M_2-M_0)$, we can compute $M_1$;
now we have all three components; use $H$ to find their order (ie: map $\{M_0,M_1,M_2\}$ to $\{R, G, B\}$);

hsl to rgb (optimized)

the above algorithm is easy to comprehend, but may not be the best for machines to run; to develop an optimized algorithm for machines, we observe this fact: for fixed $S$ and $L$ and varying $H$, each RGB component range is limited to $[L-C/2, L+C/2]$ ($C$ can be computed using only $S$ and $L$); to see this, let us take green component as an example:

when area is 4 or 5, green component is the minimum component $M_0$, so it is equal to $M_0=L-C/2$;
when area is 1 or 2, green component is the maximum component $M_2$, so it is equal to $M_2=L+C/2$;
when area is 0 or 3, green component is the middle component $M_1$, so it is between $M_0$ and $M_2$, that is, in range $[L-C/2,L+C/2]$;

in this case there is one more observation: $H’’$ is linear in $M_1$ because $M_0$ and $M_2$ are fixed, so $H$ is linear in $M_1$ because $H’’$ and $H$ only differ by a constant; adjacent areas are obviously continuous (when $H$ is gradually increased, the point moves continuously), so we can connect the endpoints of its two adjacent areas with a straight line (area 2 and 4 for area 3, area 5 and 1 for area 0);

now we get a function mapping $H$ to green component $G$ which has a flat top, a flat bottom, and two slopes; it looks quite similar to the one in hsv above, but wikipedia does not give a plot for hsl;

this function can be realized with clever use of $\min$ and $\max$; someone has worked it out as (plot):
$f(n)=L-a\max(-1,\min(k-3,9-k,1)) \\ k=(n+{\frac {H}{30^{\circ }}}){\bmod {1}}2 \\ a=S\min(L,1-L) \\ (R,G,B)=(f(0),f(8),f(4)) \\$
as you can see, red, green, blue components work in the same way, only with different shifts;

interconversion

interconversions between hsv and hsl can be computed directly without using rgb; we will be using these facts for the interconversions:

$C_V=C_L=M_2-M_0 \\ H_V=H_L \\ V=M_2 \\ L=(M_2+M_0)/2 \\$

hsv to hsl

to compute $\{H_L, S_L, L\}$ from $\{H_V, S_V, V\}$:

$H_L=H_V \\ M_2=V \\ M_0=M_2-C_V=V-C_V=V-V \times S_V\\ L=(M_2+M_0)/2=(V+V-V \times S_V)/2=V-(V \times S_V)/2 \\ S_L=C_L/(2\min(L,1-L))=(V-L)/\min(L,1-L), S_L=0 \text{ if } L=0,1 \\$

hsl to hsv

to compute $\{H_V, S_V, V\}$ from $\{H_L, S_L, L\}$:

$H_V=H_L \\ V=M_2=L+C_L/2=L+S_L \times \min(L,1-L) \\ M_0=2L-M_2 \\ C_V=M_2-M_0=2(M_2-L)=2(V-L) \\ S_V=C_V/V=2-2L/V, S_V=0 \text{ if } V=0 \\$

3d visualization

here are amazing 3d visualizations of hsl and hsv color models, coming from a bachelor thesis; you can play with them interactively to see how the point moves simultaneously in the rgb cube and in the cone/bi-cone;

epilog

some of you might get disappointed, sadly, there are no snippets for conversion between rgb and hsl/hsv in this article; the reason is i do not want to present a possibly flawed implementation that could misbehave in production, and i have no idea what programming language you will be using; so just search online, for a mature library in your language;

another thing good to mention here is units and ranges; everything works fine without units and with values in range [0.0, 1.0], since this is merely a geometric transformation; however, you will typically see in software, hue in range [0, 360], value, lightness, saturation in range [0, 100]; the hue is in degree, all others are in percentage; these units and ranges are for convenience reasons;

it is probably good to end this article with this picture:

references

custom keyboard layouts with xkb

Mon, 13 Dec 2021 06:17:37 -0800

x keyboard extension (aka: xkb), which was made several decades ago, is still in active use today; here we show how to use xkb to alter system keyboard layout and create our own; in this example, we want to switch comma and period, for no reason but a demo;

load, change, save

xkb comes with a tool called xkbcomp, which is used to compile xkb keymap descriptions; a great feature of this “compiler” is that it can talk to the “live” x server to load and save keymap; this makes our customization an easy 3-step process:

load keymap from x server;
```
xkbcomp "$DISPLAY" custom.xkb
```

change keymap;

vim custom.xkb

assuming we are using layout us; we look for section xkb_symbols then we can find these two lines:

key <AB08> {         [           comma,            less ] };
key <AB09> {         [          period,         greater ] };

as you can guess, to switch these two keys, we simply switch their values:

key <AB08> {         [          period,         greater ] };
key <AB09> {         [           comma,            less ] };

save keymap to x server;
```
xkbcomp custom.xkb "$DISPLAY"
```

this is pretty easy and works well;

change partials

the above approach involves a monolithic keymap file, which may seem daunting; we can do it a bit more elegant, by editing partial files; to do this, we step into /usr/share/X11/xkb and we find subdirs here such as keycodes, symbols, types, etc.; we are still interested symbols files so we look into symbols subdir; for simplicity we open symbols/us, which contains this part:

default partial alphanumeric_keys modifier_keys
xkb_symbols "basic" {

    name[Group1]= "English (US)";

    key <TLDE> {    [   grave,          asciitilde      ]   };
    key <AE01> {    [   1,              exclam          ]   };
    key <AE02> {    [   2,              at              ]   };
    key <AE03> {    [   3,              numbersign      ]   };
    key <AE04> {    [   4,              dollar          ]   };
    key <AE05> {    [   5,              percent         ]   };
    key <AE06> {    [   6,              asciicircum     ]   };
    key <AE07> {    [   7,              ampersand       ]   };
    key <AE08> {    [   8,              asterisk        ]   };
    key <AE09> {    [   9,              parenleft       ]   };
    key <AE10> {    [   0,              parenright      ]   };
    key <AE11> {    [   minus,          underscore      ]   };
    key <AE12> {    [   equal,          plus            ]   };

    key <AD01> {    [   q,              Q               ]   };
    key <AD02> {    [   w,              W               ]   };
    key <AD03> {    [   e,              E               ]   };
    key <AD04> {    [   r,              R               ]   };
    key <AD05> {    [   t,              T               ]   };
    key <AD06> {    [   y,              Y               ]   };
    key <AD07> {    [   u,              U               ]   };
    key <AD08> {    [   i,              I               ]   };
    key <AD09> {    [   o,              O               ]   };
    key <AD10> {    [   p,              P               ]   };
    key <AD11> {    [   bracketleft,    braceleft       ]   };
    key <AD12> {    [   bracketright,   braceright      ]   };

    key <AC01> {    [   a,              A               ]   };
    key <AC02> {    [   s,              S               ]   };
    key <AC03> {    [   d,              D               ]   };
    key <AC04> {    [   f,              F               ]   };
    key <AC05> {    [   g,              G               ]   };
    key <AC06> {    [   h,              H               ]   };
    key <AC07> {    [   j,              J               ]   };
    key <AC08> {    [   k,              K               ]   };
    key <AC09> {    [   l,              L               ]   };
    key <AC10> {    [   semicolon,      colon           ]   };
    key <AC11> {    [   apostrophe,     quotedbl        ]   };

    key <AB01> {    [   z,              Z               ]   };
    key <AB02> {    [   x,              X               ]   };
    key <AB03> {    [   c,              C               ]   };
    key <AB04> {    [   v,              V               ]   };
    key <AB05> {    [   b,              B               ]   };
    key <AB06> {    [   n,              N               ]   };
    key <AB07> {    [   m,              M               ]   };
    key <AB08> {    [   comma,          less            ]   };
    key <AB09> {    [   period,         greater         ]   };
    key <AB10> {    [   slash,          question        ]   };

    key <BKSL> {    [   backslash,      bar             ]   };
};

it is straightforward to change this file and save as a custom copy; some articles may then tell you to add this custom copy to rules by editing some file; but we will not go that way here, cuz that needs to write under /usr/share, which should not be touched by normal users; instead, we do these steps:

save our modified symbols file in a dir that can be written by normal user:
```
mkdir -p "${xkb}/symbols"
echo "$symbols" > "${xkb}/symbols/custom"
```

print current keymap using setxkbmap:

setxkbmap -print > "${xkb}/custom.xkb"

compare this with:

xkbcomp "$DISPLAY" "${xkb}/custom.xkb"

they both generate a keymap file, but setxkbmap generates it succinct:

xkb_keymap {
    ...
    xkb_symbols   { include "pc+us"         };
    ...
};

customize curent keymap:

sed -i "s/+us/+custom/" "${xkb}/custom.xkb"

call xkbcomp to apply our change on x display:
```
xkbcomp -I"$xkb" "${xkb}/custom.xkb" "$DISPLAY"
```
we need -I because our $xkb is not a standard dir searched by xkbcomp;

print keyboard layout

when our work is done, we probably want to know how our keymap looks like; for this we can use xkbprint:

xkbprint -color "$DISPLAY"

this outputs a .ps file, which we can pipe to get a .pdf file:

xkbprint -color "$DISPLAY" - | ps2pdf - custom.pdf

now comes the problem: xkbprint may not be available in your linux distro; for example, it has been retired in fedora; seriously, i could not believe it, but we are gonna help ourselves; on fedora, since xkbprint is no longer available, i pulled the xkbcomp srpm, changed all names and versions to those of xkbprint, fixed some obvious bugs in spec file, then fetched the xkbprint source tarball from here, finally it built into a binary;

references

this is a huge topic, many not covered here; but we got these great articles:

it seems xkb configuration can only read from /usr/share, not /usr/local/share or other dirs; this sucks, really;

create local pypi repos

Thu, 02 Dec 2021 11:21:07 -0800

if you ever need to customize python packages but your updates are not accepted by upstream developers, then you can host those packages in your own local repo;

host with web server

hosting your own pypi repo is actually very simple; all you need is to build a directory structure like this (check pep-503 for naming normalization):

.
├── bar
│   └── bar-0.1.tar.gz
└── foo
    ├── Foo-1.0.tar.gz
    └── Foo-2.0.tar.gz

then serve this directory using a web server with autoindex: apache, nginx, or even twisted; then set this url as pip index url;

this serves well if all you need is this sole local repo; in real life, you may want something more; for example, you may want to fallback to pypi.org if the requested package does not exist in your own local repo; for this you need more than a simple web server;

host with pypi server

you need a pypi server;

the one behind pypi.org is warehouse, which is powerful and specialized; for personal use, however, i do prefer something simpler; the one getting into my sight is pypiserver, which implements the same interfaces as pypi.org and works with tools like pip and twine; i dont want to talk too much about it because it has a readme; i just show how to use it:

pypi-server -i 127.0.0.1 -p 8080 --fallback-url "https://pypi.org/simple/" {path}

this command starts a pypi server at http://127.0.0.1:8080, hosting packages at local filesystem path {path}; these packages can be source tarballs (sdist), binary wheels (bdist_wheel), etc.; you can install these packages using pip with a custom index url:

using cli option --index-url:

pip install --index-url "http://127.0.0.1:8080/simple/" ...

using envar PIP_INDEX_URL:

PIP_INDEX_URL="http://127.0.0.1:8080/simple/" pip install ...

using config file ~/.config/pip/pip.conf:

[global]
index-url = http://127.0.0.1:8080/simple/

if you request a package which does not exist in {path}, then pypiserver will fallback to https://pypi.org/simple/, as configured; in this way, your local packages have priority over pypi.org packages;

(do not) use `--extra-index-url`

if you run pip install --help, you may see it has a --extra-index-url option; please do not use it, unless you know what it really means; improper use of this option can lead to severe security issues (including arbitrary code execution);

simply speaking, the --extra-index-url defines a peer package index; there is no priority between the (main) index and these extra indexes; when pip installs a package, it looks for the “best” target it can find, and this often means a higher version number; a bad consequence that can result from this is:

you placed a custom package foo==1.0.0 in your own local pypi repo;
an evil placed a malicious package foo==9.9.9 on pypi.org;
you have pypi.org as the main index url and your own local pypi repo as an extra index url;
when you pip install foo, you get the malicious package, not your local package, because the former has a higher version number;

i guess now you see what i mean; it is a very bad consequence, unless you are a security expert who can get multi-$30k from this; alright, so forget about the money and stop using --extra-index-url; it is hardly useful except for mirroring and sharding;

wrap pip to use local pypi repo

you may feel tiresome typing that long index url every time you use pip; putting it in pip.conf is one solution, as shown above; there are multiple pip configuration files you can use, and this scheme indeed offers some flexibility; but what if you sometimes want the local repo, sometimes not? would you go edit that configuration file again and again?

consider using envars in this case; we can wrap the pip command inside an envar wrapper:

_wrap() {
    PIP_INDEX_URL="http://127.0.0.1:8080/simple/" "$@"
}

this allows us to use local repo with any pip:

_wrap pip install ...
_wrap pip3.9 install ...
_wrap pip3.10 install ...

this is better, right? what comes next is even better (note the trailing space):

alias wrap='_wrap '

this further allows us to use local repo with a pip alias:

alias _p='pip'
_wrap _p install ...    # fail
wrap _p install ...     # ok

this works too if the above line is still too long for you:

alias p='wrap _p'
p install ...

all of these together provide a lot of flexibility;

references

wxpython: scrolled panel magics

Fri, 26 Nov 2021 10:56:56 -0800

today is a good day to create some wxpython scrolled panels; what i had imagined is a scrolled panel with client size 256x256 and virtual size 1024x1024, using a wxpython 4.0.7 gtk2 build;

something that does not work

this is what i came up with earlier; very natural no brainer:

#!/usr/bin/env python3

'''
show a scrolled panel with client size 256x256 and virtual size 1024x1024;
'''

from wx.lib.scrolledpanel import ScrolledPanel
import wx

def OnScrollWin(evt):
    panel = evt.GetEventObject()
    frame = panel.GetParent()
    ##  expect: `frame=(..., ...), panel=(256, 256)`
    print(f'frame={frame.GetClientSize()}, panel={panel.GetClientSize()}')
    evt.Skip()

app = wx.App()

frame = wx.Frame(None)

panel = ScrolledPanel(frame)
panel.SetMinClientSize((256, 256))
panel.SetScrollbars(1, 1, 1024, 1024)
panel.Bind(wx.EVT_SCROLLWIN, OnScrollWin)

sizer = wx.BoxSizer(wx.VERTICAL)
sizer.Add(panel, 0)
frame.SetSizer(sizer)

frame.Fit()
frame.Layout()
frame.Center()
frame.Show()

app.MainLoop()

unfortunately, this does not work; as i wrote it i was looking for a client size of 256x256 when scrolling; but the program always gave wrong result;

something that does work

then i realized there must be some magics; after some trial and error, this is what i came up with later that finally works:

#!/usr/bin/env python3

'''
show a scrolled panel with client size 256x256 and virtual size 1024x1024;
'''

from wx.lib.scrolledpanel import ScrolledPanel
import wx

def OnScrollWin(evt):
    panel = evt.GetEventObject()
    frame = panel.GetParent()
    ##  expect: `frame=(..., ...), panel=(256, 256)`
    print(f'frame={frame.GetClientSize()}, panel={panel.GetClientSize()}')
    evt.Skip()

app = wx.App()

frame = wx.Frame(None)
frame.SetClientSize(frame.GetClientSize())  # magic_1

panel = ScrolledPanel(frame)
panel.SetMinClientSize((256, 256))
panel.SetScrollbars(1, 1, 1024, 1024)
panel.Bind(wx.EVT_SCROLLWIN, OnScrollWin)

sizer = wx.BoxSizer(wx.VERTICAL)
sizer.Add(panel, 0)
frame.SetSizer(sizer)

frame.Show()                                # magic_2
panel.SetMinClientSize((256, 256))          # magic_3
frame.Fit()
frame.Layout()
frame.Center()

app.MainLoop()

there are 3 magics that made it work:

call SetClientSize after frame creation;

setting-a-getting really looks like a (harmless) noop, but is actually no noop;
call Show before Fit and Layout;

i always considered early Show to be smelly code, cuz it displays partial result; but in this case i have no other way to get it work; my conjecture is that the scrollbar size is only known after the frame is shown (because it is native), but this is only my guess;
call SetMinClientSize on panel, again;

yup, just call it again after frame is shown; who knows why the first call no longer works? just call it, again;

to be honest, these are real magics; i could not explain why; one needs magics to deal with scrolled panels, really;

reduce git repo size

Sun, 14 Nov 2021 02:39:59 -0800

git repos can grow fat after long time of use; fortunately, there are some easy commands to reduce their size;

garbage collection

the primary downsize technique is garbage collection; the command we use here for garbage collection is:

git gc --aggressive --prune=now

this deletes unreachable objects in the repo in an aggressive manner; you can find more about this in git help gc;

prune reflog

if you have indeed read the doc shown by git help gc, you see it says:

git gc tries very hard not to delete objects that are referenced anywhere in your repository. In particular, it will keep not only objects referenced by your current set of branches and tags, but also objects referenced by the index, remote-tracking branches, reflogs (which may reference commits in branches that were later amended or rewound), and anything else in the refs/* namespace. Note that a note (of the kind created by git notes) attached to an object does not contribute in keeping the object alive. If you are expecting some objects to be deleted and they aren’t, check all of those locations and decide whether it makes sense in your case to remove those references.

in short, we need to remove unnecessary references to let gc actually delete the objects; as it says, there are several kinds of references:

current set of branches and tags;
index;
remote-tracking branches;
reflogs;
anything else in the refs/* namespace;

the one that is common but often ignored is reflog, which we can prune with:

git reflog expire --expire=90.days.ago --expire-unreachable=all --all

this command prunes reachable reflog entries that are 90 days or older, and all unreachable reflog entries; when used together, run this command before git gc so that git gc takes its effect:

git reflog expire --expire=90.days.ago --expire-unreachable=all --all
git gc --aggressive --prune=now

delete all commits, but one

warn: this can be destructive!

sometimes we have a lot of legacy in our git repo, and what we really want is merely its current state (ie: the last commit); when this is the case, we can remove all the other commits, which can reduce the repo size quite a bit:

git checkout --orphan newbranch
git add -A
git commit
git branch -D master
git branch -m master
git push -f origin master

this creates a new orphan branch as the master branch, then discards the old one (including all the commits only reachable from it); this technique is very potent, so use it with care; if you need to push the new branch to remote, use forced push; finally, this technique is not limited to the master branch and the name newbranch is arbitrary;

when you have done this, you can append the same downsize commands above; note that because you have freed the old branch, there are possibly many objects to delete; so this can be a huge saving (at the expense of losing commit history); as such, you may not want this in a public git repo;

references

bash bite: dynamic scoping

Wed, 18 Aug 2021 01:47:31 -0700

well-known or not, bash uses dynamic scoping; this can make some fun when manipulating variables in caller and callee functions;

a brief introduction

a great introduction to dynamic scoping is right here in man bash:

The shell uses dynamic scoping to control a variable’s visibility within functions. With dynamic scoping, visible variables and their values are a result of the sequence of function calls that caused execution to reach the current function. The value of a variable that a function sees depends on its value within its caller, if any, whether that caller is the “global” scope or another shell function. This is also the value that a local variable declaration “shadows”, and the value that is restored when the function returns.

For example, if a variable var is declared as local in function func1, and func1 calls another function func2, references to var made from within func2 will resolve to the local variable var from func1, shadowing any global variable named var.

in short, dynamic scoping means the scope depends on the call stack, not the source code;

the call stack

now let us give an example showing the call stack:

f3() {
    local l=3
    echo "n=3, l=$l"
    f2
    echo "n=3, l=$l"
}

f2() {
    local l=2
    echo "n=2, l=$l"
    f1
    echo "n=2, l=$l"
}

f1() {
    local l=1
    echo "n=1, l=$l"
    f0
    echo "n=1, l=$l"
}

f0() {
    local l=0
    echo "n=0, l=$l"
    :
    echo "n=0, l=$l"
}

f3

output:

n=3, l=3
n=2, l=2
n=1, l=1
n=0, l=0
n=0, l=0
n=1, l=1
n=2, l=2
n=3, l=3

this example defines a chain of 4 functions; when we call f3 we create a call stack of 4 frames; the frame number is shown by n; in each frame, we define a local variable l, whose value is also printed in output;

this is a pretty simple example, but serves as the beginning of our exploration;

access variable in previous scope

accessing a variable defined in a previous scope is straightforward: if we do not define a variable with the same name in current scope, then this name binds to the variable in the closest previous scope; think previous as calling or ancestor;

here is a read example:

f3() {
    local l=3
    echo "n=3, l=$l"
    f2
    echo "n=3, l=$l"
}

f2() {
    local l=2
    echo "n=2, l=$l"
    f1
    echo "n=2, l=$l"
}

f1() {
    echo "n=1, l=$l"
    f0
    echo "n=1, l=$l"
}

f0() {
    local l=0
    echo "n=0, l=$l"
    :
    echo "n=0, l=$l"
}

f3

output:

n=3, l=3
n=2, l=2
n=1, l=2
n=0, l=0
n=0, l=0
n=1, l=2
n=2, l=2
n=3, l=3

in this example, we omit the local definition of l in f1; as a result, when we access l in f1, the access is made to the l defined in its parent scope f2, the closest one that defines l;

here is a write example, which works in the same way:

f3() {
    local l=3
    echo "n=3, l=$l"
    f2
    echo "n=3, l=$l"
}

f2() {
    local l=2
    echo "n=2, l=$l"
    f1
    echo "n=2, l=$l"
}

f1() {
    l=x
    echo "n=1, l=$l"
    f0
    echo "n=1, l=$l"
}

f0() {
    local l=0
    echo "n=0, l=$l"
    :
    echo "n=0, l=$l"
}

f3

output:

n=3, l=3
n=2, l=2
n=1, l=x
n=0, l=0
n=0, l=0
n=1, l=x
n=2, l=x
n=3, l=3

in this example, we assign a letter x to variable l in f1; because f1 does not define a variable named l, this l resolves to the l defined in f2;

unset a variable

read and write are so easy; how about unset?

again, refer to man bash:

The unset builtin also acts using the same dynamic scope: if a variable is local to the current scope, unset will unset it; otherwise the unset will refer to the variable found in any calling scope as described above. If a variable at the current local scope is unset, it will remain so until it is reset in that scope or until the function returns. Once the function returns, any instance of the variable at a previous scope will become visible. If the unset acts on a variable at a previous scope, any instance of a variable with that name that had been shadowed will become visible.

here is a brief explanation:

unset a variable in the current scope will “mark” it as unset;
unset a variable in a previous scope will “remove” it and reveal the shadowed variable defined in previous-previous scope, if any;

unset in the current scope

f3() {
    local l=3
    echo "n=3, l=$l"
    f2
    echo "n=3, l=$l"
}

f2() {
    local l=2
    echo "n=2, l=$l"
    f1
    echo "n=2, l=$l"
}

f1() {
    local l=1
    unset -v l
    echo "n=1, l=$l"
    f0
    echo "n=1, l=$l"
}

f0() {
    echo "n=0, l=$l"
    :
    echo "n=0, l=$l"
}

f3

output:

n=3, l=3
n=2, l=2
n=1, l=
n=0, l=
n=0, l=
n=1, l=
n=2, l=2
n=3, l=3

in this example, we unset l in f1 without defining l in f0; the result is, both f1 and f0 see l as unset; in specific, f0 is accessing the l in f1, which is unset;

the example above also shows that unset a variable is not the same as omit its definition; this often looks counter-intuitive; compare the example above with the one below:

f3() {
    local l=3
    echo "n=3, l=$l"
    f2
    echo "n=3, l=$l"
}

f2() {
    local l=2
    echo "n=2, l=$l"
    f1
    echo "n=2, l=$l"
}

f1() {
    echo "n=1, l=$l"
    f0
    echo "n=1, l=$l"
}

f0() {
    echo "n=0, l=$l"
    :
    echo "n=0, l=$l"
}

f3

output:

n=3, l=3
n=2, l=2
n=1, l=2
n=0, l=2
n=0, l=2
n=1, l=2
n=2, l=2
n=3, l=3

in this example, both f1 and f0 are accessing the l in f2;

unset in a previous scope

unset looks more intuitive when applied to a variable in a previous scope; in this case, we can think like the variable definition has been removed in that scope;

f3() {
    local l=3
    echo "n=3, l=$l"
    f2
    echo "n=3, l=$l"
}

f2() {
    local l=2
    echo "n=2, l=$l"
    f1
    echo "n=2, l=$l"
}

f1() {
    unset -v l
    echo "n=1, l=$l"
    f0
    echo "n=1, l=$l"
}

f0() {
    echo "n=0, l=$l"
    :
    echo "n=0, l=$l"
}

f3

output:

n=3, l=3
n=2, l=2
n=1, l=3
n=0, l=3
n=0, l=3
n=1, l=3
n=2, l=3
n=3, l=3

in this example, f1 does not define a local l, so when it unsets l it is unsetting the l defined in f2; this works like removing the l defined in f2, which reveals the l defined in f3;

revisit: access variable in previous scope

you may have found out that we cannot access a variable defined in a previous scope if we have defined it locally in the current scope; and unsetting it in the current scope does not help;

the solution is using the upvar magic:

f3() {
    local l=3
    echo "n=3, l=$l"
    f2
    echo "n=3, l=$l"
}

f2() {
    local l=2
    echo "n=2, l=$l"
    f1
    echo "n=2, l=$l"
}

f1() {
    local l=1

    ##  what to do here if we want to access `l` in `f2`? without `local l=1`,
    ##  this is as simple as `echo "$l"` (read) or `l=x` (write); the answer:
    local l && upvar l x

    echo "n=1, l=$l"
    f0
    echo "n=1, l=$l"
}

f0() {
    local l=0
    echo "n=0, l=$l"
    :
    echo "n=0, l=$l"
}

f3

output:

n=3, l=3
n=2, l=2
n=1, l=x
n=0, l=0
n=0, l=0
n=1, l=x
n=2, l=x
n=3, l=3

as we have seen, upvar assigns a new value to l defined in f2, from within the scope of f1; what is the magic of upvar?

the `upvar` magic

here is the definition of upvar:

upvar() {
    if unset -v "$1"; then           # Unset & validate varname
        if (( $# == 2 )); then
            eval $1=\"\$2\"          # Return single value
        else
            eval $1=\(\"\${@:2}\"\)  # Return array
        fi
    fi
}

when we call local l && upvar l x in f1, we define a local variable l in the current scope f1, then go into the upvar scope, in which l is unset; because upvar does not define a variable l, name l in the unset resolves to the one we defined in f1; because f1 is a previous scope of the upvar scope, this unset removes the l in f1 and reveals the shadowed l in the closest previous-f1 scope (f2 in this case); thus subsequent access to l in upvar occurs on the l in f2; this includes the assignment by eval; put it together, we have assigned to l in f2;

the upvar magic is, a parent makes a child to deal with the grandparent;

the `upvars` magic

there is also upvars that works similar to upvar but in addition supports arrays; both upvar and upvars can be found on this page; below is a comparison of their usage:

the upvar magic:

f() { local b; g b; echo $b; }
g() { local "$1" && upvar $1 bar; }
f  # Ok: b=bar

the upvars magic:

f() { local a b; g a b; declare -p a b; }
g() {
    local c=( foo bar )
    local "$1" "$2" && upvars -v $1 A -a${#c[@]} $2 "${c[@]}"
}
f  # Ok: a=A, b=(foo bar)

option `localvar_unset`

before we end this article, we mention that bash-5.0-beta2 introduces a shopt option localvar_unset:

If set, calling unset on local variables in previous function scopes marks them so subsequent lookups find them unset until that function returns. This is identical to the behavior of unsetting local variables at the current function scope.

the upvar and upvars magics work only when this option is disabled:

shopt -u localvar_unset

A Bottle of Wind

y combinator

session 0: basic

session 1: fancy

session 2: fancier

session 3: y combinator

session 4: y combinator expanded

session 5: z combinator

session 6: z combinator expanded

session 7: hybrid combinator (y and z)

session 8: y8 combinator

session 9: y8 combinator expanded

session 10: z8 combinator

session 11: turing combinator

session 12: summary

session 13: racket implementation

session 14: build a jet

go project structure

one file

multiple files in one package

multiple packages in one module

references

stackless coroutine

stateless and stateful functions

stateful functions in real c

stackless coroutines are stateful functions

implement yield statement

activation frame

async/await pattern

lua pitfalls

hsl and hsv

motivation

background

hsv

hue and chroma

value

saturation

hsv to rgb

hsv to rgb (optimized)

hsl

hue and chroma

lightness

saturation

hsl to rgb

hsl to rgb (optimized)

interconversion

hsv to hsl

hsl to hsv

3d visualization

epilog

references

custom keyboard layouts with xkb

load, change, save

change partials

print keyboard layout

references

create local pypi repos

host with web server

host with pypi server

(do not) use --extra-index-url

wrap pip to use local pypi repo

references

wxpython: scrolled panel magics

something that does not work

something that does work

reduce git repo size

garbage collection

prune reflog

delete all commits, but one

references

bash bite: dynamic scoping

a brief introduction

the call stack

access variable in previous scope

unset a variable

unset in the current scope

unset in a previous scope

revisit: access variable in previous scope

the upvar magic

the upvars magic

implement `yield` statement

(do not) use `--extra-index-url`

the `upvar` magic

the `upvars` magic

option `localvar_unset`