Prototext and JSON PB support

This release introduces a more comprehensive implementation of protobuf, supporting textproto and JSON protobuf mappings, and paving the way for binary protobuf support as well.

This release required some significant changes to the cue command logic.

A hallmark property of protobuf data formats is that they cannot be parsed without a schema (unlike JSON, for instance). This is true to various degrees for the different formats, but it holds true in some shape or form for all of these. This required some changes in the CUE tooling to make this possible. It also required some backwards incompatible changes.

Protobuf changes

Filetype *.textproto

CUE now supports interpreting and writing text proto files. Text proto files cannot be interpreted, or even parsed, without a schema. This means that .textproto files can only be read with an explicit schema using the --schema/-d flag. Moreover, the schema must have @protobuf for certain types, such as maps, to be interpreted properly.

Note that the available .textproto parsing libraries are incredibly buggy. So there will be some rough edges that are kind out of CUE’s hands.

JSON conversion: package jsonpb and json+pb

The Protobuf documentation has a recommendation on how Proto messages should map to JSON. Package jsonpb now supports this mapping in both directions. It implements this by rewriting a CUE AST based on a given schema, a cue.Value. This allows this mapping to also be combined in conjunction with Yaml, for instance.

On the command line this feature can be used through the pb “interpretation”, for instance as json+pb or yaml+pb. Both input and output are supported.

Note that interpretations to CUE require a schema for the interpretation. This can be an explicitly specified schema using the --schema/-d flag or an implicit only by unifying it with a schema value or playing it within a schema using the placement flags (see cue help flags).

Backwards incompatible changes

@protobuf tag

The protobuf previously had only one required argument: the numeric value of the enum. The type was optional and only included if different in name from the CUE type. As it turned out, though, the CUE type was not always sufficient information to be able to represent proto values. Most notably, integer values are encoded differently in JSON depending on the type of integer in the proto specification (this is not a typo!!).

The new format includes the type unconditionally as the second argument. CUE does its best to recognize old formats for backwards compatibility purposes, but this may cause issues.

@protobuf(<tag num>,<type>, ...options)

Protobuf options are still represented as string literals, allowing CUE options, such as alternative names (name=x) to be represented as the usual <key>=<value> format.

Another change is the representation for map types. Previously, the Protobuf map type was included verbatim (map<T,U>). This was somewhat inconvenient for parsing attributes, though: angular brackets are, unlike (), [], and {} not matched. So the comma in the map type commanded some escaping. To avoid this, maps are now represented as map[T]U. We contemplated using [T]:U, but opted for the map prefix for clarity and future extendibility.

JSON mappings

An initial design decision for the Proto to CUE mapping was to have CUE follow the Proto to JSON mapping. By hindsight, though, this did not make much sense. The inconsistency of having integers represented as strings does not make sense for a language that supports expressions on such integers (unless we want to give up typing, perhaps).

The stance now is to take the representation that makes sense, and have a protobuf-specific CUE to/from JSON converters. This is akin to the JSON schema and OpenAPI converters, which map CUE to some data format, using JSON or YAML, for instance, as a transport layer.

Luckily, the Protobuf to CUE mapping already deviated from the recommended mapping, always defaulting to int for integer types. So there are no changes there, other than that there is now support for following the recommended mappings for import and export.

Enum mappings

The most noteworthy backwards incompatible change is how enum types are mapped. Previously, CUE followed the recommended JSON mapping of using strings. This turned out to be a bad idea for various reasons. Firstly, the source of truth are integer values, really, not the names. There may be multiple names per number, making comparing names somewhat meaningless. Finally, most other languages used integers as representations, making the CUE representation not interoperate well with such languages.

Although the old mapping is still supported, the integer-based mapping is now recommended.

The default enum representation when using cue import proto is now to represent enums as integers in CUE. The --proto_enum=json flag can be used to invoke the old conversion behavior.

The API will keep converting using the JSON format, but now has a Config.EnumMode option for selecting the alternative behavior.

The Protobuf to JSON interpretation (filetype json+pb or package jsonpb) supports converting back and forth in either format.

Changelog

aaf6e84 cmd/cue/cmd: compute schema before values
0b5084a cmd/cue/cmd: hook up protobuf encodings types
8e5eeab cmd/cue/cmd: parseArgs: split values and schemas early
f228236 cmd/cue/cmd: preparse command line path expressions
75d0180 cmd/cue/cmd: simplify parseArgs in preparation of proto support
dea3c5d cue/build: organized Instance fields
660b090 cue/build: remove support for file lists
38ad7c3 cue: add ReferencePath
af509c6 cue: eliminate context type
52db572 cue: get rid of internal index type
c0fe9ce cue: refactor index
362e3a5 encoding/protobuf/jsonpb: add encoder
4a288d5 encoding/protobuf/textproto: add decoder implementation
a0035de encoding/protobuf/textproto: add encoder
3bdfa5d encoding/protobuf: always include type as second argument
dcfff00 encoding/protobuf: support integer enums
8ba98ee internal/encoding: pass schema to config
80a0a6e internal: move DebugStr to new astinternal package
22abdad internal: replace internal.CoreValue with more type-safe variant

A couple of API bug fixes

Not a big release, just a couple of API bug fixes.

Changelog

c8b7fe0 internal/core/export: export floats as float literals
f1d3d81 internal/core/export: fix decl export printing and extraction

Bug fixes and API work

This release contains several bug fixes, API additions, some API deprecations, and a language addition. One of the bug fixes may lead to unexpected behavior (see below).

Backwards Incompatible Bug Fix

One common use case of cmd/cue involves combining CUE and non-CUE files. Consider the following example:

// check.cue
package check

map: {for n in nodes {"\(n.name)": n}}

# data.yaml
nodes:
- name: bar
  parent: foo

- name: baz
  parent: foo

With v0.3.0 and earlier, cue eval would happily combine these two files:

$ cue eval data.yaml check.cue
map: {
    bar: {
        name:   "bar"
        parent: "foo"
    }
    baz: {
        name:   "baz"
        parent: "foo"
    }
}
nodes: [{
    name:   "bar"
    parent: "foo"
}, {
    name:   "baz"
    parent: "foo"
}]

But cue vet would complain:

$ cue vet data.yaml check.cue
map: reference "nodes" not found:
    ./check.cue:3:16

cue vet is actually correct here. Identifier resolution should only happen across files that belong to the same package: non-CUE files are equivalent to a CUE file without a package clause or an anonymous package clause (package _), hence cue eval should really fail in this instance.

v0.3.1 makes cue eval and other commands consistent with cue vet. This was deemed a sufficiently breaking change to warrant its own release.

The fix in this case is to define nodes in the scope of package check:

// check.cue

package check

nodes: _
map: {for n in nodes {"\(n.name)": n}}

at which point cue eval succeeds as before.

An upcoming change will also ensure this condition of identifier resolution is satisfied for all non-package CUE file arguments.

API changes

The cue.Selector model has been extended to allow querying optional fields and arbitrary elements.

Deprecations

The following methods of Value have now been deprecated:
FieldByName
Elem
Template

Really, you should only need LookupPath for your lookup needs. For the most part, these old methods have been reimplemented using the new API.

Also, updated the doc that one really, really should not use Merge.

If you are a user of the API we strongly recommend running staticcheck: it will raise errors where you are using deprecated parts of the API.

Language Additions

CUE now allows parenthesized expressions as labels. This is not (yet) published in the spec, but part of the query proposal. So "\(x)": foo, where x is a string, may now be written as (x): foo.

Changelog

a8ae7d1 cmd/cue/cmd: allow "exec" in commands for TestX
460357b cue/ast: allow parentheses as labels
e70a1db cue/build: remove unused Scope field
f0adb4e cue: deprecate Value.Elem and Value.Template
f063a61 cue: deprecate Value.FieldByName
6a1ae9c cue: move LookupPath to query
e440183 cue: remove error type aliases
4459434 cue: resolve identifiers in an ast.Expr for FillPath
c505c19 cue: separate closedness for lists and structs
957003c cue: support optional field lookup in LookupPath
ad4d1a1 cue: update Merge comment to REALLY NOT USE IT
5c2b281 internal/core: don't resolve values into anonymous packages
72e8fb4 internal/diff: fix printing and top-level diffs

First large step towards backwards compatibility guarantee

After a long road, we have now an official v0.3.0 release. Aside from implementing some important changes to the language (see release notes), it prepares CUE for exciting new features such as the query and policy extensions and other language extensions as well as much better performance.

The focus will now be to get to language stability and a v1.0.0 release.

With this release will also come a different approach to versioning leading up to the v1.0.0 release. The goal is to get to the stability guarantee for the language first, then the API and tooling. Up to v1.0.0, version increments will have the following meanings:

• minor (_.x._)
  • backwards incompatible changes
  • bug fixes with notable backwards incompatible changes
  • large new features
• patch (_._.x)
  • bug and spec conformance fixes
  • new features

We plan to get the few small remaining backwards incompatible languages chances out as soon as possible. To make the transition as simple as possible, we aim to put possible impactful bug fixes in small releases, so that users get a chance to get used to them. And, of course, we plan to make any necessary transitions as smooth as possible using cue fix aids when possible.

Note that to smooth out any transition, users can register their repos at https://github.com/cue-sh/unity. This is used to ensure that their CUE evaluations will not unexpectedly stop working.

API

encoding/protobuf/jsonpb

Protobuf defines its own, somewhat peculiar, mapping to JSON. This package provides a way to interpret JSON for such mappings, given a CUE schema. This is a first step towards broader Protobuf value support.

cue.Selector

Several new features have been added to Selector, including the possibility to query whether a selector is a regular string field and the ability to create a Selector form an ast.Label.

Value.Attributes

Value has been extended with a more extensive API to retrieve attributes.

Value.Subsume

Now allows raw (CUE-native) subsumption, instead of just the "Schema" or "Data" interpretations.

Value.FillPath

For completeness, note that beta.8 already introduced FillPath.

Changelog

4476060 cmd/cue/cmd: load import dependencies of tool files
276e164 cue/literal: expose some internal data
c24a281 cue/load: drop out-of-date comment about type of value in Overlay map
792da39 cue: add function for Value.Attributes()
1ea47e0 cue: allow raw subsumption, taking defaults into account
20a4878 cue: expose some path utilities
48f2a22 doc/cmd: refer people to cue help for now
aa99414 doc: fix various typos
2115955 encoding/openapi: correctly extract type in the presence of null
a1903ca encoding/openapi: dedup conjuncts and disjuncts
3701bef encoding/protobuf/jsonpb: add Rewrite* for interpreting JSON in PB terms
96e84eb tools/flow/testdata: add test with package dependency

v0.3.0-beta.8 the real beta.7

This release really is what beta.7 should have been as we actually forgot to include what was indicated as the most important change in beta.7 🤦‍♂️.

In the meantime there are a few more fixes. If nothing serious comes in, we’ll cut v0.3.0 in about a week!

Changelog

2e934c0 ci: do not unset Code-Review label
b00c91f cmd/cue/cmd: disallow commands in non-tool files
27d305c cmd: fix -h handling
f9164c6 cue/ast: sort CommentGroups by position when adding
7ca3968 cue: add FillPath
4ca1a5d cue: allow hidden fields in Path
ccca558 cue: allow spaces around attribute args
efd22f6 cue: allow string value as first element in ParsePath
a1551b0 doc/ref: fix HTML anchor link in spec
6195171 doc: fix running Kubernetes test with CUE_UPDATE=1
cbda0d3 internal/core/adt: fix omission of closedness check
9b263eb internal/core/adt: track more incomplete errors
a6e1627 internal/core/adt: update comments for Subsume
e3e11e3 internal: store Body in Attr type
b1730b6 tools/trim: optional fields should not remove non-optional

Bug fixes, de facto release candidate

This release contains several bug fixes.

The most important bug fix, perhaps, is that it fixes a long-standing regression where commands for cue cmd were allowed to be specified in non-tool files. The current implementation will require a command to be define in at least one *_tool.cue file.

Performance fixes have been put on hold for the moment to allow some stabilization before the final v0.3.0 release.

Changelog

35e19b2 cmd/cue: fix bug in -o, which would not write files for some commands
7c5d28e cmd/cue: fix race in testscript tests that use go/packages
81aa8cd cmd/export: fix help
d4cb2c8 cue: add locations for disallowed field
eb7b60d doc/ref/spec.md: fix ListLit production
fec7a9c encoding/openapi: fix title handling regression
bbe68e1 internal/core/adt: allow pattern constraints alongside fields
5049ab7 internal/core/adt: fix nil interface panic
1f9300d internal/core/adt: handle error properly for failed pattern constraint
2a937a9 internal/core/adt: improve error message
dd5083f tools/flow: add clarifications in the API comments

More bug fixes

This release has some important bug fixes, fixing some regressions in disjunctions and trim, mostly, among some other bug fixes.

Some of the regressions in this release slipped in because a swat of tests was accidentally disabled in the move to the new evaluator. All former tests are now reenabled and a few more corresponding issues have been resolved.

There are some performance issues remaining. We are considering leaving those in and do a v0.3.0 release, addressing these in a subsequent release. In this case, we are getting fairly close to a v0.3.0.

There will still be a beta.7 release, which addresses a regression, allowing commands in non-tool files, that we would like to address in a separate release.

Changelog

91abe0d ci: fix goreleaser config
a113048 ci: move to using go1.16
f014c14 ci: set a "trybot" tag for trybot repository dispatch flow
5d3afa9 cmd/cue/cmd: correct documentation for -d flag
de1f80c cmd/cue/cmd: remove skip tests
304a3e0 cmd/cue/cmd: use unify semantics merging across packages
ae43812 cmd/cue: make get go operate on a best-efforts basis
cf5c48e cmd/cue: only ignore type and parse errors with get go
7326cfa cue: Value.Equal should not only consider values
830cfbc cue: remove build.Context reference
17ea1d5 doc/ref/spec.md: a few fixes
f5a19ef doc/ref/spec.md: fix escaping
3ca8680 doc/ref/spec.md: fix int_lit 0
e4d0d13 doc/ref/spec.md: remove faulty use of \x in double-quoted string
7df9fa4 doc/tutorial/basics: fix example
b51368e doc/tutorial/kubernetes: add required go mod init step
430c3dd encoding/jsonschema: fix dropping of struct for additionalProperties: true
1347dd4 general: add .unity-bin to gitignore
a0e1970 general: add unity configuration to codereview.cfg
351a77e internal/core/adt: add more error positions
79644c3 internal/core/adt: allow hidden fields alongside embedded lists
70e2f2e internal/core/adt: discard errors in field verification
452c7bd internal/core/adt: don't count optional fields as proof of struct
3d926af internal/core/adt: don't eliminate incomplete disjuncts prematurely
1589a07 internal/core/adt: get default when comparing ot bottom
e12e26e internal/core/compile: fix let resolution bug
77e6b51 internal/core/compile: reenable disabled tests
2c86835 internal/core/eval: minor performance improvements for disjunctions
e1e7031 internal/core/runtime: decouple cue API from Runtime
c6b7ab2 internal/cuetest: add support for issue-based conditions/checkers
38f0f63 pkg/internal: fix decimal and string list errors
af3c9dc pkg/list: fix list.Unique dedupping
dcf932f tools/trim: fix too aggressive elimination for disjunctions

v0.3.0-beta.5

This release has some important bug fixes. It also is a bit stricter in considering some errors that were previously evaluation-time errors to be compile time errors.

There is also a slight change in default handling, which should not affect most configurations.

New compile-time errors

The expression >10 <20 is syntactically valid CUE, but can never meaningfully evaluate to something. It parses as >(10 < 20) and thus evaluates to >true, which is undefined. The error messages associated with this were often confusing to the user. Also, leaving this to be detected during runtime would sometimes hide the error away.

Consistent with the “fail early” mantra, these cases are now detected and reported by the compiler.

Backwards compatibility

Defaults

Previously, for

#def: {
   *{} | {a: int} | {b: int}
   *{} | {c: int} | {d: int}
}
x: #def & {a: int}

would officially resolve to

x: {a: int} | {a: int, c: int} | {a: int, d: int}

Thanks to a hack in the current v0.3 implementation, the first disjunct was made the default and all resolved fine. We have now slightly changed the definition of defaults to handle this case.
Basically, now if during a unification all terms of a disjunction that are marked as default are eliminated, the remaining terms will behave as if they originated from an unmarked disjunction.
In practice this means defaults behave as users typically expect them to behave. This may also cover cases that were not previously handed

Note that v0.2 solved things differently: if in the result of a disjunction one term subsumes another, then the latter could be dropped. So in v0.2, the answer would be x: {a: int}, even in the absence of defaults. The problem with this approach is that it could lead to confusing answers and that the default could not always be accurately determined. A goal of v0.3 was to have a simpler story for disambiguation and let simplifications be merely a matter of optimization.

There is still a slight discrepancy in the current implementation and the spec that may cause unexpected results. We expect this to be sufficiently rare and we intend to address this in a rework of disjunction computation, aimed at further performance optimizations.

Changelog

1b03c9f ci: move to non-Docker version of goreleaser
ae51469 ci: skip known flakey test on go1.15 windows
500e431 doc/ref/spec.md: add missing binary_digits
89cada6 doc/tutorial: address cuelang/cuelang.org#126
77b475f doc: fix install instructions in main README and contributing
25ba1d9 internal/core/adt: comment out leak test
b25147c internal/core/adt: control fixes for if clauses
267379b internal/core/adt: fix trim bug
48d255a internal/core/adt: some optimizations for pattern constraints
87d43b7 internal/core/adt: strip "notDefault" status for disjunctions with unused defaults
760aa11 internal/core/compile: make >10 <20 a compile error
b20ac5a internal/core/convert: fix conversion from float
bcd752a internal/core/eval: handle embedded top correctly
a5daa16 internal/core/export: fix quoting of definitions
cb4fae9 internal/cuetest: consolidate testscript/cuetxtar helpers
bf6c3fc internal/cuetest: move Kubernetes related helper to where they are used
15934d9 tools/trim: add tracer
1d9aa17 tools/trim: don't trim shared nodes
af5ea97 tools/trim: fix defaults bug

Performance improvements and bug fixes

This release has some important bug fixes related to performance, disjunctions, and performance of disjunctions.

Backwards compatibility

As a result of the disjunction rework, we introduced another internal workaround for dealing with the fact that there is no good way to represent proto oneOf fields in CUE. Right now this is done as

*{} | { a: int } | { b: int }

or

*{} | { a?: int } | { b?: int }

The need for the default is because one cannot say that a field must exist. So the default is used to disambiguate. This works well for one set of oneOfs, but not in the presence of multiple oneOfs.

A previous hack would unmark a disjunction as having a default if all but one of the defaults remains. After the critical performance improvements, this still worked, but only because these changes introduced a bug. Now this is fixed we can no longer rely on it. To keep protobuf working, there is now another hack that recognizes this pattern and adjusts defaults in a similar manner.

This is an unacceptable solution, of course. The idea is to allow specifying required fields instead, somewhat like this

{} | { a!: int } | { b!: int }

and transition to this model.

The required field specification has many benefits and solves several modeling limitations of CUE we’ve encountered since its inception. It can also almost, if not entirely, eliminate the need for ?, provides a solution for some limitations of the proposed query syntax, and generally provides better alternatives for cases where one finds the need to abuse defaults.

We plan to officially introduce this construct in a proposal soon.

Changelog

09cd07b all: add tests for the GitHub workflow process
f0d1fa4 ci: add first version of copybara workflow
10a48e9 ci: drop go1.13 from build matrix; add go1.16rc1
e097927 ci: fix broken build
34333ce ci: fix job dependency in test workflow
726d93e ci: fix start message of CI jobs for CL builds
9b7fc4b ci: fix workflow definitions
3088513 ci: only go test -race on latest Go+linux, and master
5b3d551 ci: refactor CI setup
2338b59 ci: rename repository dispatch workflow
ef59f61 ci: update version of Go for go generate to 1.14.14
257486c cmd/cue/cmd: allow fmt to pass with errors
dfce25c cmd/cue: add goproxytest instance for script tests
b8d9972 cmd/cue: allow CUE_UPDATE to update cmd/cue/cmd testscript tests
a616925 cmd/cue: embed packages required for get go checking
22aafc2 cmd/cue: fix typo in cmd documentation headline
37c16cc cmd/cue: use gotooltest within testscript tests
d9d0487 cue/scanner: fix JSON conformance
ba8eb37 cue: allow LookupDef on embedded scalars
c6f1287 cue: deprecate Merge
f73e78f doc/../43_schema.txt: remove unnecessary word
6c3be7f doc: fix typo in types tutorial (closed -> close)
2df92cb doc: remove duplicate command from Kubernetes tutorial
b360fe4 encoding/gocode: use FieldByName in generated code
7110020 general: move from godoc.org to pkg.go.dev for documentation
d5d5fd1 general: remove junk files
63104c8 internal/core/adt: "fix" disjunction resolution
2ca6c5e internal/core/adt: cache evaluation results for dynamic fields
c866a51 internal/core/adt: don't check cycles for inlined arcs
6b96001 internal/core/adt: fix benign memory leak
b8b4892 internal/core/adt: fix bottom cycle error
c091274 internal/core/adt: fix bug that left errors of incompleted expressions unreported
9c5489f internal/core/adt: fix memory leak
ee78ec3 internal/core/adt: fix nil panic
8d537aa internal/core/adt: fix regressions in disjunct elimination
e6a2fb0 internal/core/adt: fix special cycle condition
bedb047 internal/core/adt: more fixes to cycle handling
8334a9e internal/core/adt: more merging of adt and former eval
d0124e2 internal/core/adt: refactor node lookup
0ad02fd internal/core/adt: restrict notification
d2aa995 internal/core/export: don't embed proper conjuncts
76a72c4 internal/core/export: fix optional output bug
fdccfbc internal/core/export: retain top in embedded scalars
34587b9 internal/core/export: support package and struct attributes
dfaea59 internal/filetypes: fix test breakage
10180b6 pkg: consistently use 0666 as perm when creating regular files
f55f17e tools/fix: eliminate x & _ and unnecessary parens
611d622 tools/flow: replace package doc example with real example

v0.3.0-beta.3

This release addresses some important bug fixes, including a performance regression. It also sports a complete reimplementation of trim, utilizing the possibilities of the data structures of the new evaluator.

cue trim

The biggest new feature of trim is that definitions now can remove fields of non-definitions. Other than that, it mostly is just bug fixes. The data structures of the new evaluator make it considerably easier to implement, leaving less than a quarter of the code.

encoding/protobuf

This package now links in the full set of builtins, obviating the need for API users to have to do this.

Changelog

fdc7010 cmd/cue/cmd: fix cue cmd crash
a8de61c cue/testdata/cycle: fix another cycle bug
28cfa74 cue: exempt hidden fields also in Unify and UnifyAccept
ca5d2db doc/tutorial/kubernetes: update files of Kubernetes demo
f0c025c encoding/protobuf: enforce loading builting packages
08e814f internal/core/adt: add support info for trim
8e6e795 internal/core/adt: change closeInfo.embedded to a flag field
0e401d6 internal/core/adt: fix performance issue with disjunctions
82bda84 internal/core/adt: treat files as belonging to same struct
c58e0ff internal/core/eval: strip arcs for nodes with structural cycles
0f570a9 internal/core/validate: pick up non-concrete values in defaults
f03928d tools/fix: allow fixing non-essential simplification
6898897 tools/trim: implement using adt