HOBiT: Programming Lenses Without Using Lens Combinators

Abstract

We propose HOBiT, a higher-order bidirectional programming language, in which users can write bidirectional programs in the familiar style of conventional functional programming, while enjoying the full expressiveness of lenses. A bidirectional transformation, or a lens, is a pair of mappings between source and view data objects, one in each direction. When the view is modified, the source is updated accordingly with respect to some laws—a pattern that is found in databases, model-driven development, compiler construction, and so on. The most common way of programming lenses is with lens combinators, which are lens-to-lens functions that compose simpler lenses to form more complex ones. Lens combinators preserve the bidirectionality of lenses and are expressive; but they compel programmers to a specialised point-free style—i.e., no naming of intermediate computation results—limiting the scalability of bidirectional programming. To address this issue, we propose a new bidirectional programming language HOBiT, in which lenses are represented as standard functions, and combinators are mapped to language constructs with binders. This design transforms bidirectional programming, enabling programmers to write bidirectional programs in a flexible functional style and at the same time access the full expressiveness of lenses. We formally define the syntax, type system, and the semantics of the language, and then show that programs in HOBiT satisfy bidirectionality. Additionally, we demonstrate HOBiT ’s programmability with examples.

1 Introduction

Transforming data from one format to another is a common task of programming: compilers transform program texts into syntax trees, manipulate the trees and then generate low-level code; database queries transform base relations into views; model transformations generate lower-level implementations from higher-level models; and so on. Very often, such transformations will benefit from being bidirectional, allowing changes to the targets to be mapped back to the sources too. For example, if one can run a compiler front-end (preprocessing, parsing, desugaring, etc.) backwards, then all sorts of program analysis tools will be able to focus on a much smaller core language, without sacrificing usability, as their outputs in term of the core language will be transformed backwards to the source language. In the same way, such needs arise in databases (the view-update problem [1, 6, 12]) and model-driven engineering (bidirectional model transformation) [28, 33, 35].

As a response to this challenge, programming language researchers have started to design languages that execute deterministically in both directions, and the lens framework is the most prominent among all. In the lens framework, a bidirectional transformation (or a lens) $\ell \in Lens \;{S}\;{V}$, consists of $get \;\ell \in S \rightarrow V$, and $put \;\ell \in S \rightarrow V\rightarrow S$ [3, 7, 8]. (When clear from the context, or unimportant, we sometimes omit the lens name and write simply $get$/$put$.) Function $get$ extracts a view from a source, and $put$ takes both an updated view and the original source as inputs to produce an updated source. The additional parameter of $put$ makes it possible to recover some of the source data that is not present in the view. In other words, $get$ needs not to be injective to have a $put$. Not all pairs of $get$/$put$ are considered correct lenses. The following round-triping laws of a lens $\ell$ are generally required to establish bidirectionality:

$$\begin{aligned}&put \;{\ell }\;s\;v = s \quad \text {if}\quad get \;{\ell }\;s = v&(\mathbf {Acceptability})\\&get \;{\ell }\;s' = v \quad \,\,\, \text{ if }\quad put \;{\ell }\;s\;v = s'&\,\,(\mathbf {Consistency}) \end{aligned}$$

for all $s$, $s'$ and $v$. (In this paper we write $e = e'$ with the assumption that neither $e$ nor $e'$ is undefined. Stronger variants of the laws enforcing totality exist elsewhere, for example in [7].) Here consistency ensures that all updates on a view are captured by the updated source, and acceptability prohibits changes to the source if no update has been made on the view. Collectively, the two laws defines well-behavedness [1, 7, 12].

The most common way of programming lenses is with lens combinators [3, 7, 8], which are basically a selection of lens-to-lens functions that compose simpler lenses to form more complex ones. This combinator-based approach follows the long history of lightweight language development in functional programming. The distinctive advantage of this approach is that by restricting the lens language to a few selected combinators, well-behavedness can be more easily preserved in programming, and therefore given well-behaved lenses as inputs, the combinators are guaranteed to produce well-behaved lenses. This idea of lens combinators is very influential academically, and various designs and implementations have been proposed [2, 3, 7,8,9, 16, 17, 27, 32] over the years.

1.1 The Challenge of Programmability

The complexity of a piece of software can be classified as either intrinsic or accidental. Intrinsic complexity reflects the inherent difficulty of the problem at hand, whereas accidental complexity arises from the particular programming language, design or tools used to implement the solution. This work aims at reducing the accidental complexity of bidirectional programming by contributing to the design of bidirectional languages. In particularly, we identify a language restriction—i.e., no naming of intermediate computation results—which complicates lens programming, and propose a new design that removes it.

As a teaser to demonstrate the problem, let us consider the list append function. In standard unidirectional programming, it can be defined simply as $append \;x \;y = \mathbf {case}~x~\mathbf {of}~\{ [\,]\rightarrow y; a:x' \rightarrow a: append \;x' \;y\}$. Astute readers may have already noticed that $append$ is defined by structural recursion on x, which can be made explicit by using $foldr$ as in $append \;x \;y = foldr \;(:) \;y \;x$.

But in a lens language based on combinators, things are more difficult. Specifically, $append$ now requires a more complicated recursion pattern, as below.

It is beyond the scope of this paper to explain how exactly the definition of $appendL$ works, as its obscurity is what this work aims to remove. Instead, we informally describe its behaviour and the various components of the code. The above code defines a lens: forwards, it behaves as the standard $append$, and backwards, it splits the updated view list, and when the length of the list changes, this definition implements (with the part) the bias of keeping the length of the first source list whenever possible (to disambiguate multiple candidate source changes). Here, $cond$, $(\mathbin {\hat{\circ }})$, etc. are lens combinators and $outListL$ and $rearr$ are auxiliary lenses, as can be seen from their types. Unlike its unidirectional counterpart, $appendL$ can no longer be defined as a structural recursion on list; instead it traverses a pair of lists with rather complex rearrangement $rearr$.

Intuitively, the additional parts is intrinsic complexity, as they are needed for directing backwards execution. However, the complicated recursion scheme, which is a direct result of the underlying limitation of lens languages, is certainly accidental. Recall that in the definition of $append$, we were able to use the variable $y$, which is bound outside of the recursion pattern, inside the body of $foldr$. But the same is not possible with lens combinators which are strictly ‘pointfree’. Moreover, even if one could name such variables (points), their usage with lens combinators will be very restricted in order to guarantee well-behavedness [21, 23]. This problem is specific to opaque non-function objects such as lenses, and goes well beyond the traditional issues associated with the pointfree programming style.

In this paper, we design a new bidirectional language HOBiT, which aims to remove much of the accidental difficulty found in combinator-based lens programming, and reduces the gap between bidirectional programming and standard functional programming. For example, the following definition in HOBiT implements the same lens as $appendL$.

As expected, the above code shares the part with the definition of $appendL$ as the two implement the same backwards behaviour. The difference is that $appendB$ uses structural recursion in the same way as the standard unidirectional $append$, greatly simplifying programming. This is made possible by the HOBiT ’s type system and semantics, allowing unrestricted use of free variables. This difference in approach is also reflected in the types: $appendB$ is a proper function (instead of the abstract lens type of $appendL$), which readily lends itself to conventional functional programming. At the same time, $appendB$ is also a proper lens, which when executed by the HOBiT interpreter behave exactly like $appendL$. A major technical challenge in the design of HOBiT is to guarantee this duality, so that functions like $appendB$ are well-behaved by construction despite the flexibility in their construction.

1.2 Contributions

As we can already see from the very simple example above, the use of HOBiT simplifies bidirectional programming by removing much of the accidental complexity. Specifically, HOBiT stands out from existing bidirectional languages in two ways:

• 1. It supports the conventional programming style that is used in unidirectional programming. As a result, a program in HOBiT can be defined in a way similar to how one would define only its $get$ component. For example, $appendB$ is defined in the same way as the unidirectional $append$.

• 2. It supports incremental improvement. Given the very often close resemblance of a bidirectional-program definition and that of its get component, it becomes possible to write an initial version of a bidirectional program almost identical to its $get$ component and then to adjust the backwards behaviour gradually, without having to significantly restructure the existing definition.

Thanks to these distinctive advantages, HOBiT for the first time allows us to construct realistically-sized bidirectional programs with relative ease. Of course, this does not mean free lunch: the ability to control backwards behaviours will not magically come without additional code (for example the part above). What HOBiT achieves is that programming effort may now focus on the productive part of specifying backwards behaviours, instead of being consumed by circumventing language restrictions.

In summary, we make the following contributions in this paper.

• We design a higher-order bidirectional programming language HOBiT, which supports convenient bidirectional programming with control of backwards behaviours (Sect. 3). We also discuss several extensions to the language (Sect. 5).

• We present the semantics of HOBiT inspired by the idea of staging [5], and prove the well-behavedness property using Kripke logical relations [18] (Sect. 4).

• We demonstrate the programmability of HOBiT with examples such as desugaring/resugaring [26] (Sect. 6). Additional examples including a bidirectional evaluator for $\lambda$-calculus [21, 23], a parser/printer for S-expressions, and bookmark extraction for Netscape [7] can be found at https://bitbucket.org/kztk/hibx together with a prototype implementation of HOBiT.

2 Overview: Bidirectional Programming Without Combinators

In this section, we informally introduce the essential constructs of HOBiT and demonstrate their use by a few small examples. Recall that, as seen in the $appendB$ example, the strength of HOBiT lies in allowing programmers to access $\lambda$-abstractions without restrictions on the use of $\lambda$-bound variables.

2.1 The $\underline{\mathbf {case}}$ Construct

The most important language construct in HOBiT is $\underline{\mathbf {case}}$ (pronounced as bidirectional case), which provides pattern matching and easy access to bidirectional branching, and also importantly, allows unrestricted use of $\lambda$-bound variables.

In general, a $\underline{\mathbf {case}}$ expression has the following form.

$$ \underline{\mathbf {case}}~e~\underline{\mathbf {of}}~\{ p_1 \rightarrow e_1 \mathrel {\underline{\mathbf {with}}}\phi _1 \mathrel {\underline{\mathbf {by}}}\rho _1; \dots ; p_n \rightarrow e_n \mathrel {\underline{\mathbf {with}}}\phi _n \mathrel {\underline{\mathbf {by}}}\rho _n \} $$

(Like Haskell, we shall omit “$\{$”, “$\}$” and “;” if they are clear from the layout.) In the type system of HOBiT, a $\underline{\mathbf {case}}$-expression has type $\varvec{\mathsf {B}}{B}$, if e and $e_i$ have types $\varvec{\mathsf {B}}{A}$ and $\varvec{\mathsf {B}}{B}$, and $\phi _i$ and $\rho _i$ have types $B \rightarrow Bool$ and $A \rightarrow B \rightarrow A$, where A and B contains neither ($\rightarrow$) nor $\varvec{\mathsf {B}}{}$. The type $\varvec{\mathsf {B}}{A}$ can be understood intuitively as “updatable A”. Typically, the source and view data are given such $\varvec{\mathsf {B}}{}$-types, and a function of type $\varvec{\mathsf {B}}{A} \rightarrow \varvec{\mathsf {B}}{B}$ is the HOBiT equivalent of $Lens \;{A}\;{B}$.

The pattern matching part of $\underline{\mathbf {case}}$ performs two implicit operations: it first unwraps the $\varvec{\mathsf {B}}{}$-typed value, exposing its content for normal pattern matching, and then it wraps the variables bound by the pattern matching, turning them into ‘updatable’ $\varvec{\mathsf {B}}{}$-typed values to be used in the bodies. For example, in the second branch of $appendB$, a and $x'$ can be seen as having types A and [A] in the pattern, but $\varvec{\mathsf {B}}{A}$ and $\varvec{\mathsf {B}}{[A]}$ types in the body; and the bidirectional constructor $(\mathbin {\underline{:}}) \,{:}{:}\, \varvec{\mathsf {B}}{A} \rightarrow \varvec{\mathsf {B}}{[A]} \rightarrow \varvec{\mathsf {B}}{[A]}$ combines them to produce a $\varvec{\mathsf {B}}{}$-typed list.

In addition to the standard conditional branches, $\underline{\mathbf {case}}$-expression has two unique components $\phi _i$ and $\rho _i$ called exit conditions and reconciliation functions respectively, which are used in backwards executions. Exit condition $\phi _i$ is an over-approximation of the forwards-execution results of the expressions $e_i$. In other words, if branch i is choosen, then $\phi _i\;e_i$ must evaluate to $\mathsf {True}$. This assertion is checked dynamically in HOBiT, though could be checked statically with a sophisticated type system [7]. In the backwards direction the exit condition is used for deciding branching: the branch with its exit condition satisfied by the updated view (when more than one match, the original branch used in the forwards direction has higher priority) will be picked for execution. The idea is that due to the update in the view, the branch taken in the backwards direction may be different from the one taken in the original forwards execution, a feature that is commonly supported by lens languages [7] which we call branch switching.

Branch switching is crucial to $put$’s robustness, i.e., the ability to handle a wide range of view updates (including those affect the branching decisions) without failing. We explain its working in details in the following.

Branch Switching. Being able to choose a different branch in the backwards direction only solves part of the problem. Let us consider the case where a forward execution chooses the $n^\mathrm {th}$ branch, and the backwards execution, based on the updated view, chooses the $m^\mathrm {th}$ ($m \ne n$) branch. In this case, the original value of the pattern-matched expression e, which is the reason for the $n^\mathrm {th}$ branch being chosen, is not compatible with the $put$ of the $m^\mathrm {th}$ branch.

As an example, let us consider a simple function that pattern-matches on an $Either$ structure and returns an list. Note that we have purposely omitted the reconciliation functions.

We have said that functions of type $\varvec{\mathsf {B}}{A} \rightarrow \varvec{\mathsf {B}}{B}$ are also fully functioning lenses of type $Lens \;{A}\;{B}$. In HOBiT, the above code runs as follows, where $\texttt {HOBiT>}$ is the prompt of HOBiT ’s read-eval-print loop, and :get and :put are meta-language operations to perform $get$ and $put$ respectively.

As we have explained above, exit conditions are used to decide which branch will be used in the backwards direction. For the first and second evaluations of $put$, the exit conditions corresponding to the original branches were true for the updated view. For the last evaluation of $put$, since the exit condition of the original branch was false but that of the other branch was true, branch switching is required here. However, a direct $put$-execution of f with the inputs $(\mathsf {Right} \;(1,[2,3]))$ and $[\,]$ crashes (represented by $\bot$ above), for a good reason, as the two inputs are in an inconsistent state with respect to f.

This is where reconciliation functions come into the picture. For the $\mathsf {Left}$ branch above, a sensible reconciliation function will be , which when applied turns the conflicting source $(\mathsf {Right} \;(1,[2,3]))$ into $\mathsf {Left}\;[\,]$, and consequently the $put$-execution may succeed with the new inputs and returns $\mathsf {Left}\;[\,]$. It is not difficult to verify that the “reconciled” $put$-execution still satisfies well-behavedness. Note that despite the similarity in types, reconciliation functions are not $put$; they merely provide a default source value to allow stuck $put$-executions to proceed. We visualise the effect of reconciliation functions in Fig. 1. The left-hand side is bidirectional execution without successful branch-switching, and since $\phi _m \;b_n$ is false (indicating that $b_n$ is not in the range of the $m^{th}$ branch) the execution of $put$ must (rightfully) fail in order to guarantee well-behavedness. On the right-hand side, reconciliation function $\rho _n$ produces a suitable source from $a_m$ and $b_n$ (where $\phi _n \;( get \;(\rho _n \;a_m\;b_n))$ is True), and $put$ executes with $b_n$ and the new source $\rho _n \;a_m\;b_n$. It is worth mentioning that branch switching with reconciliation functions does not compromise correctness: though the quality of the user-defined reconciliation functions affects robustness as they may or may not be able to resolve conflicts, successful $put$-executions always guarantee well-behavedness, regardless the involvement of reconciliation functions.

Fig. 1.

Reconciliation function: assuming exit conditions $\phi _m$ and $\phi _n$ where $\phi _m \;b_n = \mathsf {False}$ but $\phi _n \;b_n = \mathsf {True}$, and reconciliation functions $\rho _m$ and $\rho _n$.

Revisiting appendB. Recall $appendB$ from Sect. 1.1 (reproduced below).

The exit condition for the nil case always returns true as there is no restriction on the value of $y$, and for the cons case it requires the returned list to be non-empty. In the backwards direction, when the updated view is non-empty, both exit conditions will be true, and then the original branch will be taken. This means that since $appendB$ is defined as a recursion on x, the backwards execution will try to unroll the original recursion step by step (i.e., the cons branch will be taken for a number of times that is the same as the length of $x$) as long as the view remains non-empty. If an updated view list is shorter than $x$, then $not \circ null$ will become false before the unrolling finishes, and the nil branch will be taken (branch-switching) and the reconciliation function will be called.

The definition of $appendB$ is curried; straightforward uncurrying turns it into the standard form $\varvec{\mathsf {B}}{A} \rightarrow \varvec{\mathsf {B}}{B}$ that can be interpreted by HOBiT as a lens. The following HOBiT program is the bidirectional variant of $uncurry$.

Here, $\underline{\mathbf {let}}~p = e~\underline{\mathbf {in}}~e'$ is syntactic sugar for , in which the reconciliation function is never called as there is only one branch. Let $appendB ' = uncurryB \; appendB$, then we can run $appendB '$ as:

Difference from Lens Combinators. As mentioned above, the idea of branch switching can be traced back to lens languages. In particular, the design of $\underline{\mathbf {case}}$ is inspired by the combinator $cond$ [7]. Despite the similarities, it is important to recognise that $\underline{\mathbf {case}}$ is not only a more convenient syntax for $cond$, but also crucially supports the unrestricted use of $\lambda$-bound variables. This more fundamental difference is the reason why we could define $appendB$ in the conventional functional style as the variables $x$ and $y$ are used freely in the body of $\underline{\mathbf {case}}$. In other words, the novelty of HOBiT is its ability to combine the traditional (higher-order) functional programming and the bidirectional constructs as found in lens combinators, effectively establishing a new way of bidirectional programming.

2.2 A More Elaborate Example: $linesB$

In addition to supporting convenient programming and robustness in $put$ execution, the $\underline{\mathbf {case}}$ constructs can also be used to express intricate details of backwards behaviours. Let us consider the $lines$ function in Haskell as an example, which splits a string into a list of strings by newlines, for example, , except that the last newline character in its input is optional. For example, $lines$ returns for both and . Suppose that we want the backwards transformation of $lines$ to exhibit a behaviour that depends on the original source:

Fig. 2.

$linesB$ and $breakNLB$

This behaviour is achieved by the definition in Fig. 2, which makes good use of reconciliation functions. Note that we do not consider the contrived corner case where the string ends with duplicated newlines such as in . The function $breakNLB$ splits a string at the first newline; since $breakNLB$ is injective, its exit conditions and reconciliation functions are of little interest. The interesting part is in the definition of $linesB$, particularly its use of reconciliation functions to track the existence of a last newline character. We firstly explain the branching structure of the program. On the top level, when the first line is removed from the input, the remaining string b may contain more lines, or be the end (represented by either the empty list or the singleton list ). If the first branch is taken, the returned result will be a list of more than one element. In the second branch when it is the end of the text, b could contain a newline or simply be empty. We do not explicitly give patterns for the two cases as they have the same body $f \mathbin {\underline{:}}\underline{[\,]}$, but the reconciliation function distinguishes the two in order to preserve the original source structure in the backwards execution. Note that we intentionally use the same variable name b in the case analysis and the reconciliation function, to signify that the two represent the same source data. The use of argument b in the reconciliation functions serves the purpose of remembering the (non)existence of the last newline in the original source, which is then preserved in the new source.

It is worth noting that just like the other examples we have seen, this definition in HOBiT shares a similar structure with a definition of $lines$ in Haskell.¹ The notable difference is that a Haskell definition is likely to have a different grouping of the three cases of $lines$ into two branches, as there is no need to keep track of the last newline for backwards execution. Recall that reconciliation functions are called after branches are chosen by exit conditions; in the case of $linesB$, the reconciliation function is used to decide the reconciled value of $b'$ to be or . This, however, means that we cannot separate the pattern $b'$ into two and with copying its branch body and exit condition, because then we lose a chance to choose a reconciled value of b based on its original value.

3 Syntax and Type System of HOBiT Core

In this section, we describe the syntax and the type system of the core of HOBiT.

3.1 Syntax

The syntax of HOBiT Core is given in Fig. 3. For simplicity, we only consider booleans and lists. The syntax is almost the same as the standard $\lambda$-calculus with the fixed-point combinator ($\mathbf {fix}$), lists and booleans. For data constructors and case expressions, there are in addition bidirectional versions that are underlined. We allow the body of $\mathbf {fix}$ to be non-$\lambda$s to make our semantics simple (Sect. 4), though such a definition like $\mathbf {fix}(\lambda x.{\mathsf {True}:x})$ can diverge.

Fig. 3.

Syntax of HOBiT Core

Although in examples we used $\mathbf {case}$/$\underline{\mathbf {case}}$-expressions with an arbitrary number of branches having overlapping patterns under the first-match principle, we assume for simplicity that in HOBiT Core $\mathbf {case}$/$\underline{\mathbf {case}}$-expressions must have exactly two branches whose patterns do not overlap; extensions to support these features are straightforward. As in Haskell, we sometimes omit the braces and semicolons if they are clear from the layout.

3.2 Type System

The types in HOBiT Core are defined as follows.

We use the metavariable $\sigma , \tau , \dots$ for types that do not contain $\rightarrow$ nor $\varvec{\mathsf {B}}{}$, We call $\sigma$-types pure datatypes, which are used for sources and views of lenses. Intuitively, $\varvec{\mathsf {B}}{\sigma }$ represents “updatable $\sigma$”—data subject to update in bidirectional transformation. We keep the type system of HOBiT Core simple, though it is possible to include polymorphic types or intersection types to unify unidirectional and bidirectional constructors.

Fig. 4.

Typing rules: ${\varDelta } \vdash {p} : {\sigma }$ is similar to ${\varGamma } \vdash {p} : {A}$ but asserts that the resulting environment is actually a bidirectional environment.

The typing judgment ${\varGamma };{\varDelta } \vdash {e} : {A}$, which reads that under environments $\varGamma$ and $\varDelta$, expression e has type A, is defined by the typing rules in Fig. 4. We use two environments: $\varDelta$ (the bidirectional type environment) is for variables introduced by pattern-matching through $\underline{\mathbf {case}}$, and $\varGamma$ for everything else. It is interesting to observe that $\varDelta$ only holds pure datatypes, as the pattern variables of $\underline{\mathbf {case}}$ have pure datatypes, while $\varGamma$ holds any types. We assume that the variables in $\varGamma$ and those in $\varDelta$ are disjoint, and appropriate $\alpha$-renaming has been done to ensure this. This separation of $\varDelta$ from $\varGamma$ does not affect typeability, but is key to our semantics and correctness proof (Sect. 4). Most of the rules are standard except $\underline{\mathbf {case}}$; recall that we only use unidirectional constructors in patterns which have pure types, while the variables bound in the patterns are used as $\varvec{\mathsf {B}}{}$-typed values in branch bodies.

4 Semantics of HOBiT Core

Recall that the unique strength of HOBiT is its ability to mix higher-order uni-directional programming with bidirectional programming. A consequence of this mixture is that we can no longer specify its semantics in the same way as other first-order bidirectional languages such as [13], where two semantics—one for $get$ and the other for $put$—suffice. This is because the category of lenses is believed to have no exponential objects [27] (and thus does not permit $\lambda$s).

4.1 Basic Idea: Staging

Our solution to this problem is staging [5], which separates evaluation into two stages: the unidirectional parts is evaluated first to make way for a bidirectional semantics, which only has to deal with the residual first-order programs. As a simple example, consider the expression $(\lambda z.z) \;(x \mathbin {\underline{:}}((\lambda w.w) \;y) \mathbin {\underline{:}}\underline{[\,]})$. The first-stage evaluation, $e \Downarrow _\mathrm {U} E$, eliminates $\lambda$s from the expression as in $(\lambda z.z) \;(x \mathbin {\underline{:}}((\lambda w.w) \;y) \mathbin {\underline{:}}\underline{[\,]}) \Downarrow _\mathrm {U} x \mathbin {\underline{:}}y \mathbin {\underline{:}}\underline{[\,]}$. Then, our bidirectional semantics will be able to treat the residual expression as a lens between value environments and values, following [13, 20]. Specifically, we have the $get$ evaluation relation $\mu \vdash _\mathrm {G} E \Rightarrow v$, which computes the value v of E under environment $\mu$ as usual, and the $put$ evaluation relation $\mu \vdash _\mathrm {P} v \Leftarrow E \dashv \mu '$, which computes an updated environment $\mu '$ for E from the updated view v and the original environment $\mu$. In pseudo syntax, it can be understood as $put \;E \;\mu \;v = \mu '$, where $\mu$ represents the original source and $\mu '$ the new source.

It is worth mentioning that a complete separation of the stages is not possible due to the combination of $\mathbf {fix}$ and $\underline{\mathbf {case}}$, as an attempt to fully evaluate them in the first stage will result in divergence. Thus, we delay the unidirectional evaluation inside $\underline{\mathbf {case}}$ to allow $\mathbf {fix}$, and consequently the three evaluation relations (uni-directional, $get$, and $put$) are mutually dependent.

4.2 Three Evaluation Relations: Unidirectional, $get$ and $put$

First, we formally define the set of residual expressions:

They are treated as values in the unidirectional evaluation, and as expressions in the $get$ and $put$ evaluations. Notice that e or $e_i$ appear under $\lambda$ or $\underline{\mathbf {case}}$, meaning that their evaluations are delayed.

The set of (first-order) values is defined as below.

Accordingly, we define a (first-order) value environment $\mu$ as a finite mapping from variables to first-order values.

Unidirectional Evaluation Relation. The rules for the unidirectional evaluation relation is rather standard, as excerpted in Fig. 5. The bidirectional constructs (i.e., bidirectional constructors and $\underline{\mathbf {case}}$) are frozen, i.e., behave just like ordinary constructors in this evaluation. Notice that we can evaluate an expression containing free variables; then the resulting residual expression may contain the free variables.

Fig. 5.

Evaluation rules for unidirectional parts (excerpt)

Bidirectional $\mathbf {(}{\varvec{get}}\, \mathbf {and}\,{\varvec{put}}\mathbf {)}$ Evaluation Relations. The $get$ and $put$ evaluation relations, $\mu \vdash _\mathrm {G} E \Rightarrow v$ and $\mu \vdash _\mathrm {P} v \Leftarrow E \dashv \mu '$, are defined so that they together form a lens.

Weakening of Environment. Before we lay out the semantics, it is worth explaining a subtlety in environment handling. In conventional evaluation semantics, a larger than necessary environment does no harm, as long as there is no name clashes. For example, whether the expression x is evaluated under the environment $\left\{ x = 1\right\}$ or $\left\{ x = 1, y = 2\right\}$ does not matter. However, the same is not true for bidirectional evaluation. Let us consider a residual expression $E = x \mathbin {\underline{:}}y \mathbin {\underline{:}}\underline{[\,]}$, and a value environment $\mu = \left\{ x = 1, y = 2\right\}$ as the original source. We expect to have $\mu \vdash _\mathrm {G} E \Rightarrow 1 : 2 : [\,]$, which may be derived as:

In the $put$ direction, for an updated view say $3 : 4 : [\,]$, we expect to have $\mu \vdash _\mathrm {P} 3 : 4 : [\,] \Leftarrow E \dashv \left\{ x = 3, y = 4\right\}$ with the corresponding derivation:

What shall the environments $?_1$ and $?_2$ be? One way is to have $\mu \vdash _\mathrm {P} 3 \Leftarrow x \dashv \left\{ x = 3, y = 2\right\}$, and $\mu \vdash _\mathrm {P} 4 : [\,] \Leftarrow y \mathbin {\underline{:}}\underline{[\,]} \dashv \left\{ x = 1, y = 4\right\}$, where the variables do not appear free in the residual expression takes their values from the original source environment $\mu$. However, the evaluation will get stuck here, as there is no reasonable way to produce the expected result $\left\{ x = 3, y = 4\right\}$ from $?_1 = \left\{ x = 3, y = 2\right\}$ and $?_2 = \left\{ x = 1, y = 4\right\}$. In other words, the redundancy in environment is harmful as it may cause conflicts downstream.

Our solution to this problem, which follows from [21,22,23, 29], is to allow $put$ to return value environments containing only bindings that are relevant for the residual expressions under evaluation. For example, we have $\mu \vdash _\mathrm {P} 3 \Leftarrow x \dashv \left\{ x = 3\right\}$, and $\mu \vdash _\mathrm {P} 4 : [\,] \Leftarrow y \mathbin {\underline{:}}\underline{[\,]} \dashv \left\{ y = 4\right\}$. Then, we can merge the two value environments $?_1 = \left\{ x = 3\right\}$ and $?_2 = \left\{ y = 4\right\}$ to obtain the expected result $\left\{ x = 3, y = 4\right\}$. As a remark, this seemingly simple solution actually has a non-trivial effect on the reasoning of well-behavedness. We defer a detailed discussion on this to Sect. 4.3.

Now we are ready to define $get$ and $put$ evaluation rules for each bidirectional constructs. For variables, we just lookup or update environments. Recall that $\mu$ is a mapping (i.e., function) from variables to (first-order) values, while we use a record-like notation such as $\{ x = v\}$.

For constants $\underline{c}$ where $c = \mathsf {False}, \mathsf {True}, [\,]$, the evaluation rules are straightforward.

The above-mentioned behaviour of the bidirectional cons expression $E_1 \mathbin {\underline{:}}E_2$ is formally given as:

(Note that the variable rules guarantee that only free variables in the residual expressions end up in the resulting environments.) Here, $\mathbin {\curlyvee }$ is the merging operator defined as: $\mu \mathbin {\curlyvee }\mu ' = \mu \cup \mu ' \text { if there is no } x \text { such that } \mu (x) \ne \mu '(x).$ For example, , and , but is undefined.

The most interesting rules are for $\underline{\mathbf {case}}$. In the $get$ direction, it is not different from the ordinary $\mathbf {case}$ except that exit conditions are asserted, as shown in Fig. 6. We use the following predicate for pattern matching.

$$ match (p_k, v_0,\mu _k) ~=~ (p_k \mu _k = v_0) \wedge (\mathsf {dom}(\mu _k) = \mathsf {fv}(p_k)) $$

Here, we abuse the notation to write $p_{k}\mu _{k}$ for the value obtained from $p_{k}$ by replacing the free variables x in $p_{k}$ with $\mu _k(x)$. One might notice that we have the disjoint union in Fig. 6 where $\mu _{i}$ holds the values of the variables in $p_i$, as we assume $\alpha$-renaming of bound variables that is consistent in $get$ and $put$. Recall that $p_{1}$ and $p_{2}$ are assumed not to overlap, and hence the evaluation is deterministic. Note that the reconciliation functions $E''_{i}$ are untouched by the rule.

The $put$ evaluation rule of $\underline{\mathbf {case}}$ shown in Fig. 6 is more involved. In addition to checking which branch should be chosen by using exit conditions, we need two rules to handle the cases with and without branch switching. Basically, the branch to be taken in the backwards direction is decided first, by the $get$-evaluation of the case condition $E_{0}$ and the checking of the exit condition $E_i'$ against the updated view v. After that, the body of the chosen branch $e_{i}$ is firstly uni-directionally evaluated, and then its residual expression $E_{i}$ is $put$-evaluated. The last step is $put$-evaluation of the case-condition $E_{0}$. When branch switching happens, there is the additional step of applying the reconciliation function $E''_{j}$.

Note the use of operator $\triangleleft$ in computing the updated case condition $v_0'$.

$$ (\mu ' \triangleleft \mu )(x) = {\left\{ \begin{array}{ll} \mu '(x) &{} \text {if}~x \in \mathsf {dom}(\mu ') \\ \mu (x) &{} \text {otherwise} \end{array}\right. } $$

Recall that in the beginning of this subsection, we discussed our approach of avoiding conflicts by producing environments with only relevant variables. This means the $\mu _i'$ above contains only variables that appear free in $E_{i}$, which may or may not be all the variables in $p_{i}$. Since this is the point where these variables are introduced, we need to supplement $\mu _i'$ with $\mu _{i}$ from the original pattern matching so that $p_{i}$ can be properly instantiated.

Fig. 6.

$get$- and $put$-Evaluation of $\underline{\mathbf {case}}$: we write $\mu \mathbin {\uplus _{X,Y}} \mu '$ to ensure that $\mathsf {dom}(\mu ) \subseteq X$ and $\mathsf {dom}(\mu ') \subseteq Y$.

Construction of Lens. Let us write $\mathcal {L}_{0}[\![ E ]\!]$ for a lens between value environments and values, defined as:

Then, we can define the lens induced from e (a closed function expression), where $e \;x \Downarrow _\mathrm {U} E$ for some fresh variable x.

Actually, :get and :put in Sect. 2 are realised by and .

4.3 Correctness

We establish the correctness of HOBiT Core: is well-behaved for closed e of type $\varvec{\mathsf {B}}{\sigma } \rightarrow \varvec{\mathsf {B}}{\tau }$. Recall that $Lens \;{S}\;{V}$ is a set of lenses $\ell$, where $get \;\ell \in S \rightarrow V$ and $put \;\ell \in S \rightarrow V \rightarrow S$. We only provide proof sketches in this subsection due to space limitation.

$\varvec{\mathrel {\preceq }}$-well-behavedness. Recall that in the previous subsection, we allow environments to be weakened during $put$-evaluation. Since not all variables in a source may appear in the view, during some intermediate evaluation steps (for example within $\underline{\mathbf {case}}$-branches) the weakened environment may not be sufficient to fully construct a new source. Recall that, in $\mu \vdash _\mathrm {P} v \Leftarrow e \dashv \mu '$, $\mathsf {dom}(\mu ')$ can be smaller than $\mathsf {dom}(\mu )$, a gap that is fixed at a later stage of evaluation by merging ($\mathbin {\curlyvee }$) and defaulting ($\triangleleft$) with other environments. This technique reduces conflicts, but at the same time complicates the compositional reasoning of correctness. Specifically, due to the potentially missing information in the intermediate environments, well-behavedness may be temporally broken during evaluation. Instead, we use a variant of well-behavedness that is weakening aware, which will then be used to establish the standard well-behavedness for the final result.

Definition 1

($\mathrel {\preceq }$-well-behavedness). Let $(S, \mathrel {\preceq })$ and $(V, \mathrel {\preceq })$ be partially-ordered sets. A lens $\ell \in Lens \;{S}\;{V}$ is called $\mathrel {\preceq }$-well-behaved if it satisfies

(\textbf {Acceptability)

for any $s,s' \in S$ and $v \in V$, where s is maximal.    $\square$

We write $Lens ^\mathrm {\mathrel {\preceq }{}wb} \;S \;V$ for the set of lenses in $Lens \;{S}\;{V}$ that are $\mathrel {\preceq }$-well-behaved. In this section, we only consider the case where S and V are value environments and first-order values, where value environments are ordered by weakening ($\mu \mathrel {\preceq }\mu '$ if $\mu (x) = \mu '(x)$ for all $x \in \mathsf {dom}(\mu )$), and $(\mathrel {\preceq }) = (=)$ for first-order values. In Sect. 5.2 we consider a slightly more general situation.

The $\mathrel {\preceq }$-well-behavedness is a generalisation of the ordinary well-behavedness, as it coincides with the ordinary well-behavedness when $(\mathrel {\preceq }) = (=)$.

Theorem 1

For S and V with $(\mathrel {\preceq }) = (=)$, a lens $\ell \in Lens \;{S}\;{V}$ is $\mathrel {\preceq }$-well-behaved iff it is well-behaved.    $\square$

Kripke Logical Relation. The key step to prove the correctness of HOBiT Core is to prove that $\mathcal {L}_{0}[\![ E ]\!]$ is always $\mathrel {\preceq }$-well-behaved if E is an evaluation result of a well-typed expression e. The basic idea is to prove this by logical relation that expression e of type $\varvec{\mathsf {B}}{\sigma }$ under the context $\varDelta$ is evaluated to E, assuming termination, such that $\mathcal {L}_{0}[\![ E ]\!]$ is a $\mathrel {\preceq }$-well-behaved lens between $[\![\varDelta ]\!]$ and $[\![\sigma ]\!]$.

Usually a logical relation is defined only by induction on the type. In our case, as we need to consider $\varDelta$ in the interpretation of $\varvec{\mathsf {B}}{\sigma }$, the relation should be indexed by $\varDelta$ too. However, naive indexing does not work due to substitutions. For example, we could define a (unary) relation $\mathcal {E}_\varDelta (\varvec{\mathsf {B}}{\sigma })$ as a set of expressions that evaluate to “good” (i.e., $\mathrel {\preceq }$-well-behaved) lenses between (the semantics of) $\varDelta$ and $\sigma$, and $\mathcal {E}_\varDelta (\varvec{\mathsf {B}}{\sigma } \rightarrow \varvec{\mathsf {B}}{\tau })$ as a set of expressions that evaluate to “good” functions that map good lenses between $\varDelta$ and $\sigma$ to those between $\varDelta$ and $\tau$. This naive relation, however, does not respect substitution, which can substitute a value obtained from an expression typed under $\varDelta$ to a variable typed under $\varDelta '$ such that $\varDelta \subseteq \varDelta '$, where $\varDelta$ and $\varDelta '$ need not be the same. With the naive definition, good functions at $\varDelta$ need not be good functions at $\varDelta '$, as a good lens between $\varDelta '$ and $\sigma$ is not always a good lens between $\varDelta$ and $\sigma$.

To remedy the situation, inspired by the denotation semantics in [24], we use Kripke logical relations [18] where worlds are $\varDelta$s.

Definition 2

We define the set of expressions, the set of residual expressions, the set of values and the set of value environments as below.

Here, for a set S, $List \;S$ is inductively defined as: $[\,]\in List \;S$, and $s : t \in List \;S$ for all $s \in S$ and $t \in List \;S$.    $\square$

The notable difference from ordinary logical relations is the definition of where we consider an arbitrary $\varDelta '$ such that $\varDelta \subseteq \varDelta '$. This is the key to state if $\varDelta \subseteq \varDelta '$. Notice that for any $\varDelta$.

We have the following lemmas.

Lemma 1

If $\varDelta \subseteq \varDelta '$, implies .    $\square$

Lemma 2

for any $\varDelta$ such that $\varDelta (x) = \sigma$.    $\square$

Lemma 3

For any $\sigma$ and $\varDelta$, and .    $\square$

Lemma 4

If and , then .    $\square$

Lemma 5

Let $\sigma$ and $\tau$ be pure types and $\varDelta$ a pure type environment. Suppose that for ${\varDelta _{i}} \vdash {p_{i}} : {\sigma }$ ($i = 1,2$), and that , and . Then, .

Proof

(Sketch). The proof itself is straightforward by case analysis. The key property is that $get$ and $put$ use the same branches in both proofs of ${\mathrel {\preceq }}{\text {-}}{} \mathbf{Acceptability}$ and ${\mathrel {\preceq }}{\text {-}}{} \mathbf{Consistency}$. Slight care is required for unidirectional evaluations of $e_{1}$ and $e_{2}$, and applications of $E'_{1},E'_{2},E''_{1}$ and $E''_{2}$. However, the semantics is carefully designed so that in the proof of ${\mathrel {\preceq }}{\text {-}}{} \mathbf{Acceptability}$, unidirectional evaluations that happen in $put$ have already happened in the evaluation of $get$, and a similar discussion applies to ${\mathrel {\preceq }}{\text {-}}{} \mathbf{Consistency}$.    $\square$

As a remark, recall that we assumed $\alpha$-renaming of $p_{i}$ so that the disjoint unions ($\uplus$) in Fig. 6 succeed. This renaming depends on the $\mu$s received in $get$ and $put$ evaluations, and can be realised by using de Bruijn levels.

Lemma 6

(Fundamental Lemma). For ${\varGamma };{\varDelta } \vdash {e} : {A}$, for any $\varDelta '$ with $\varDelta \subseteq \varDelta '$ and , we have .

Proof

(Sketch). We prove the lemma by induction on typing derivation. For bidirectional constructs, we just apply the above lemmas appropriately. The other parts are rather routine.    $\square$

Now we are ready to state the correctness of our construction of lenses.

Corollary 1

If ${\varepsilon };{\varepsilon } \vdash {e} : {\varvec{\mathsf {B}}{\sigma } \rightarrow \varvec{\mathsf {B}}{\tau }}$, then .    $\square$

Lemma 7

If , (if defined) is in (and thus well-behaved by Theorem 1).    $\square$

Theorem 2

If ${\varepsilon };{\varepsilon } \vdash {e} : {\varvec{\mathsf {B}}{\sigma } \rightarrow \varvec{\mathsf {B}}{\tau }}$, then (if defined) is well-behaved.    $\square$

5 Extensions

Before presenting a larger example, we discuss a few extensions of HOBiT Core which facilitate programming.

5.1 In-Language Lens Definition

In HOBiT programming, it is still sometimes useful to allow manually defined primitive lenses (i.e., lenses constructed from independently specified $get$/$put$ functions), for backwards compatibility and also for programs with relatively simple computation logic but complicated backwards behaviours. This feature is supported by the construct $\underline{\mathbf {appLens}}\;e_{1} \;e_{2} \;e_{3}$ in HOBiT. For example, we can write to define a bidirectional increment function . Note that for simplicity we require the additional expression x (represented by $e_{3}$ in the general case) to convert between normal functions and lenses. The typing rule for $\underline{\mathbf {appLens}}\;e_{1} \;e_2 \;e_3$ is as below.

Accordingly, we add the following unidirectional evaluation rule.

Also, we add the following $get / put$ evaluation rules for $\underline{\mathbf {appLens}}$.

Notice that $\underline{\mathbf {appLens}}\;e_1 \;e_2 \;e_3$ is “good” if $e_3$ is so, i.e., if , provided that the $get$/$put$ pair $(e_1,e_2)$ is well-behaved.

5.2 Lens Combinators as Language Constructs

In this paper, we have focused on the $\underline{\mathbf {case}}$ construct, which is inspired by the $cond$ combinator [7]. Although $cond$ is certainly an important lens combinator, it is not the only one worth considering. Actually, we can obtain language constructs from a number of lens combinators including those that take care of alignment [2]. For the sake of demonstration, we outline the derivation of a simpler example . As the construction depends solely on types, we purposely leave the combinator abstract.

A naive way of lifting combinators can already be found in [21, 23]. For example, for $comb$, we might prepare the construct $\underline{\mathbf {comb}}_\mathrm {bad}$ with the following typing rule (where ${\varepsilon }$ is the empty environment):

Notice that in this version e is required to be closed so that we can turn the function directly into a lens by , and the evaluation of $\underline{\mathbf {comb}}_\mathrm {bad}$ can then be based on standard lens composition: (we omit the straightforward concrete evaluation rules), where E and $E'$ is the unidirectional evaluation results of e and $e'$ (notice that a residual expression is also an expression), and $\mathbin {\hat{\circ }}$ is the lens composition combinator [7] defined by:

The combinator preserves $\mathrel {\preceq }$-well-behavedness, and thus $\underline{\mathbf {comb}}_\mathrm {bad}$ guarantees correctness. However, as discussed extensively in the case of $\underline{\mathbf {case}}$, this “closedness” requirements prevents flexible use of variables and creates a major obstacle in programming.

So instead of the plain $comb$, we shall assume a parameterised version that allows each source to have an extra component T, which is expected to be kept track of by the combinator without modification. Here T is assumed to have a partial merging operator $(\mathbin {\curlyvee }) \in T \rightarrow T \rightarrow T$ and a minimum element, and $pcomb$ may use these facts in its definition. By using $pcomb$, we can give a corresponding language construct $\underline{\mathbf {comb}}$ with a binder, typed as follows.

We give its unidirectional evaluation rule as

We omit the $get$/$put$ evaluation rules, which are straightforwardly obtained from the following equation.

where and are lens combinators defined for any $\varDelta$ as:

Both combinators preserve $\mathrel {\preceq }$-well-behavedness, where we assume the component-wise ordering on pairs. No “closedness” requirement is imposed on e in this version. From the construct, we can construct a higher-order function $\lambda f. \lambda z. \underline{\mathbf {comb}} \;(x.f \;x) \;z : (\varvec{\mathsf {B}}{\sigma } \rightarrow \varvec{\mathsf {B}}{\tau }) \rightarrow \varvec{\mathsf {B}}{\sigma '} \rightarrow \varvec{\mathsf {B}}{\tau '}$. That is, in HOBiT, lens combinators are just higher-order functions, as long as they permit the above-mentioned parameterisation. This observation means that we are able to systematically derive language constructs from lens combinators; as a matter of fact, the semantics of $\underline{\mathbf {case}}$ is derived from a variant of the $cond$ combinator [7].

Even better, the parametrised $pcomb$ can be systematically constructed from the definition of $comb$. For $comb$, it is typical that $get \;( comb \;\ell )$ only uses $get \;\ell$, and $put \;( comb \;\ell )$ uses $put \;\ell$; that is, $comb$ essentially consists of two functions of types and . Then, we can obtain $pcomb$ of the above type merely by “monad”ifying the two functions: using the reader monad $T \rightarrow {-}$ for the former and the composition of the reader and writer monads $T \rightarrow ({-}, T)$ backwards for the latter suffice to construct $pcomb$.

A remaining issue is to ensure that $pcomb$ preserves $\mathrel {\preceq }$-well-behavedness, which ensures under the assumptions and . Currently, such a proof has to be done manually, even though $comb$ preserves well-behavedness and $pcomb$ is systematically constructed. Whether we can lift the correctness proof for $comb$ to $pcomb$ in a systematic way will be an interesting future exploration.

5.3 Guards

Guards used for branching are merely syntactic sugar in ordinary unidirectional languages such as Haskell. But interestingly, they actually increase the expressive power of HOBiT, by enabling inspection of updatable values without making the inspection functions bidirectional.

For example, Glück and Kawabe’s reversible equivalence check [10] can be implemented in HOBiT as follows.

Here, $\mathopen {\underline{(}} -,- \mathclose {\underline{)}}$ is the bidirectional version of the pair constructor. The exit condition $isRight$ checks whether a value is headed by the constructor $\mathsf {Right}$, and $isLeft$ by $\mathsf {Left}$. Notice that the backwards transformation of $eqCheck$ fails when the updated view is $\mathsf {Left}\;(v,v)$ for some v.

5.4 Syntax Sugar for Reconciliation Functions

In the general form, reconciliation functions take in two arguments for the computation of the new source. But as we have seen, very often the arguments are not used in the definition and therefore redundant. This observation motivates the following syntax sugar.

$$\begin{aligned} p \rightarrow e \mathrel {\underline{\mathbf {with}}}e' \mathrel {\underline{\mathbf {default}}}\{x_1 = e''_1; \dots ; x_n = e''_n\} \end{aligned}$$

Here, $x_1,\dots ,x_n$ are the free variables in p. This syntax sugar is translated as:

Furthermore, it is also possible to automatically derive some default values from their types. This idea can be effectively implemented if we extend HOBiT with type classes.

5.5 Inference of Exit Conditions

It is possible to infer exit conditions from their surrounding contexts; an idea that has been studied in the literature of invertible programming [11, 20], and may benefit from range analysis.

Our prototype implementation adopts a very simple inference that constructs an exit condition for each branch, where $p_e$ is the skeleton of the branch body e, constructed by replacing bidirectional constructors with the unidirectional counterparts, and non-constructor expressions with . For example, from $a \mathbin {\underline{:}} appendB \;x' \;y$, we obtain the pattern . This embarrassingly simple inference has proven to be handy for developing larger HOBiT programs as we will see in Sect. 6.

6 An Involved Example: Desugaring

In this section, we demonstrate the programmability of HOBiT using the example of bidirectional desugaring [26]. Desugaring is a standard process for most programming languages, and making it bidirectional allows information in desugared form to be propagated back to the surface programs. It is argued convincingly in [26] that such bidirectional propagation (coined resugaring) is effective in mapping reduction sequences of desugared programs into those of the surface programs.

Let us consider a small programming language that consists of $\mathbf {let}$, $\mathbf {if}$, Boolean constants, and predefined operators.

Variables are represented as de Bruijn indices.

Some operators in this language are syntactic sugar. For example, we may want to desugar

$$ \mathsf {EOp} \;\texttt {"not"} \;[e] \qquad \text {as} \qquad \mathsf {EIf} \;e \;\mathsf {EFalse} \;\mathsf {ETrue}\text {.} $$

Also, $e_1 \mathbin {\texttt {||}}e_2$ can be transformed to $\mathbf {let}~x = e_1~\mathbf {in}~\mathbf {if}~x~\mathbf {then}~x~\mathbf {else}~e_2$, which in our mini-language is the following.

$$ \mathsf {EOp} \;\texttt {"or"} \;[e_1, e_2] \qquad \text {as} \qquad \mathsf {ELet} \;e_1 \;(\mathsf {EIf} \;(\mathsf {EVar} \;0) \;(\mathsf {EVar} \;0) \;( shift \;0 \;e_2) $$

Here, $shift \;n$ is the standard shifting operator for de Brujin indexed-term that increments the variables that have indices greater than n (these variables are “free” in the given expression). We will program a bidirectional version of the above desugaring process in Figs. 7 and 8, with the particular goal of keeping the result of a backward execution as close as possible to the original sugared form (so that it is not merely a “decompilation” in the sense that the original source has to be consulted).

Fig. 7.

$composB$: a useful building block

Fig. 8.

$desugarB$: bidirectional desugring

We start with an auxiliary function $compos$ [4] in Fig. 7, which is a useful building block for defining shifting and desugaring. We have omitted the straightforward exit conditions; they will be inferred as explained in Sect. 5.5. The function $mapB$ is the bidirectional map. The reconciliation function $recE$ tries to preserves as much source structure as possible by reusing the original source e. Here, $arities \,{:}{:}\,[( Name , Int )]$ maps operator names to their arities (i.e. ). The function $shift$ is the standard uni-directional shifting function. We omit its definition as it is similar to the bidirectional version in Fig. 8. Note that $\mathrel {\underline{\mathbf {default}}}$ is syntactic sugar for reconciliation function introduced in Sect. 5.4. Here, $incB$ is the bidirectional increment function defined in Sect. 5.1. Thanks to $composB$, we only need to define the interesting parts in the definitions of $shiftB$ and $desugarB$. The reconciliation functions $recE$ and $toOp$ try to keep as much source information as possible, which enables the behaviour that the backwards execution produces “not” and “or” in the sugared form only if the original expression has the sugar.

Consider a sugared expression as a source $source$.

The following updated views may be obtained by reductions from the view.

The following are the corresponding backward transformation results.

As the AST structure of the view is changed, all of the three cases require branch-switching in the backwards executions; our program handles it with ease. For $view _2$, the top-level expression $\mathsf {EIf} \;\mathsf {EFalse} \;\mathsf {EFalse}~...$ does not have a corresponding sugared form. Our program keeps the top level unchanged, and proceeds to the subexpression with correct resugaring, a behaviour enabled by the appropriate use of reconciliation function (the first line of $recE$ for this particular case) in $composB$.

If we were to present the above results as the evaluation steps in the surface language, one may argue that the second result above does not correspond to a valid evaluation step in the surface language. In [26], AST nodes introduced in desugaring are marked with the information of the original sugared syntax, and resugaring results containing the marked nodes will be skipped, as they do not correspond to any reduction step in the surface language. The marking also makes the backwards behaviour more predictable and stable for drastic changes on the view, as the desugaring becomes injective with this change. This technique is orthogonal to our exploration here, and may be combined with our approach.

7 Related Work

Controlling Backwards Behaviour. In addition to $put \in S \rightarrow V \rightarrow S$, many lens languages [3] supply a $\textit{create} \in V \rightarrow S$ (which is in essence a right-inverse of get) to be used when the original source data is unavailable. This happens when new data is inserted in the view, which does not have any corresponding source for put to execute, or when branch-switching happens but with no reconciliation function available. Being a right-inverse, $\textit{create}$ does not fail (assuming it terminates), but since it is not guided by the original source, the results are more arbitrary. We do not include $\textit{create}$ in HOBiT, as it complicates the system without offering obvious benefits. Our branch-switching facilities are perfectly capable of handling missing source data via reconciliation functions.

Using exit conditions in branching constructs for backwards evaluation can be found in a number of related fields: bidirectional transformation [7], reversible computation [34] and program inversion [11, 20]. Our design of $\underline{\mathbf {case}}$ is inspired by the $\textit{cond}$ combinator in the lens framework [7] and the if-statement in Janus [34]. A similar combinator is $\textit{Case}$ in BiGUL [16], where a branch has a function performing a similar role as an exit condition, but taking the original source in addition. This difference makes $\textit{Case}$ more expressive than $\textit{cond}$; for example, $\textit{Case}$ can implement matching lenses [2]. Our design of $\underline{\mathbf {case}}$ follows $\textit{cond}$ for its relative simplicity, but the same underlying technique can be applied to $\textit{Case}$ as mentioned in Sect. 5.2. In the context of bidirectionalization [19, 29, 30] there is the idea of “Plug-ins” [31] that are similar to reconciliation functions in the sense that source values can be adapted to direct backwards execution.

Applicative Lenses. The applicative lens framework [21, 23] provides a way to use $\lambda$-abstraction and function application as in normal functional programming to compose lenses. Note that this use of “applicative” refers to the classical applicative (functional) programming style, and is not directly related to Applicative functor in Haskell. In this sense, it shares a similar goal to us. But crucially, applicative lens lacks HOBiT ’s ability to allow $\lambda$-bound variables to be used freely, and as a result suffers from the same limitation of lens languages. There are also a couple of technical differences between applicative lens and our work: applicative lens is based on Yoneda embedding while ours is based on separating $\varGamma$ and $\varDelta$ and having three semantics (Sect. 4); and applicative lens is implemented as an embedded DSL, while HOBiT is given as a standalone language. Embedded implementation of HOBiT is possible, but a type-correct embedding would expose the handling of environment $\varDelta$ to programmers, which is undesirable.

Lenses and Their Extensions. As mentioned in Sect. 1, the most common way to construct lenses is by using combinators [3, 7, 8], in which lenses are treated as opaque objects and composed by using lens combinators. Our goal in this paper is to enhance the programmability of lens programming, while keeping its expressive power as possible. In HOBiT, primitive lenses can be represented as functions on $\varvec{\mathsf {B}}{}$-typed values (Sect. 5.1), and lens combinators satisfying certain conditions can be represented as language construct with binders (Sect. 5.2), which is at least enough to express the original lenses in [7].

Among extensions of the lens language [2, 3, 7,8,9, 16, 17, 27, 32], there exists a few that extend the classical lens model [7], namely quotient lenses [8], symmetric lenses [14], and edit-based lenses [15]. A natural question to ask is whether our development, which is based on the classical lenses, can be extended to them. The answer depends on treatment of value environments $\mu$ in $get$ and $put$. In our semantics, we assume a non-linear system as we can use the same variable in $\mu$ any number of times. This requires us to extend the classical lens to allow merging ($\mathbin {\curlyvee }$) and defaulting ($\triangleleft$) operations in $put$ with $\mathrel {\preceq }$-well-behavedness, but makes the syntax and type system of HOBiT simple, and HOBiT free from the design issues of linear programming languages [25]. Such extension of lenses would be applicable to some kinds of lens models, including quotient lenses and symmetric lenses, but its applicability is not clear in general. Also, we want to mention that allowing duplications in bidirectional transformation is still open, as it essentially entails multiple views and the synchronization among them.

8 Conclusion

We have designed HOBiT, a higher-order bidirectional programming language in which lenses are represented as functions and lens combinators are represented as language constructs with binders. The main advantage of HOBiT is that users can program in a style similar to conventional functional programming, while still enjoying the benefits of lenses (i.e., the expressive power and well-behavedness guarantee). This has allowed us to program realistic examples with relative ease.

HOBiT for the first time introduces a truly “functional” way of constructing bidirectional programs, which opens up a new area of future explorations. Particularly, we have just started to look at programming techniques in HOBiT. Moreover, given the resemblance of HOBiT code to that in conventional languages, the application of existing programming tools becomes plausible.