Typeclass experiments refactored #20061

odersky · 2024-04-01T17:35:20Z

A refactoring of the typeclass-experiments branch according to topics.

DISCLAIMER: This is the same as the previous typeclass-experiments Pre-pre SIP. It's just that the commits are now in a more logical order instead of the chronological order of the first PR. Some part of this is currently under consideration as SIP-64. Other parts might be proposed as Pre-SIPs in the future.

The order of exposition described in the docs of this PR is different from the planned proposals of SIPs. I concentrate here not on how to sequence details, but instead want to present a vision of what is possible. For instance, the docs in this PR start with Self types and is syntax, which have turned out to be controversial and that will probably be proposed only late in the sequence of SIPs.

The PR needs a minor release since it adds experimental language imports, which did not exist before. Everything covered is under experimental. Baseline Scala is not affected.

docs/_docs/reference/experimental/typeclasses.md

bjornregnell · 2024-04-04T09:09:11Z

docs/_docs/reference/experimental/modularity.md

+If we assume `tracked` for parameter `x` (which is implicitly a `val`),
+then `foo` would get inferred type `Foo { val x: 1 }`, so it could not
+be reassigned to a value of type `Foo { val x: 2 }` on the next line.
+


But if the compiler detects all reassigments of this kind, then it could be tracked by default if no reassignments are detected? And if they are detected the compiler can explain in an error message why it is not tracked (and explain that other stuff like dependent typing inference will not work). With a tracked-by-default-if-no-dangers rule we will not force the coder to write tracked unnecessarily, as it is probably tracked the coder wants if not doing reassignments, and it will be easier to work with dependent typing as you do not always need to bother with tracked. Would that work?

The problem is the ordering. At the time where we have to decide whether something needs to be tracked or not, we cannot see what the body of the class requires. Also, tracked implies a parameter is turned into a val, which means it can potentially clash with members defined in base classes. I believe this means we need to make it explicit. But we can certainly suggest to add tracked in many error situations. Basically, whenever an error mentions a skolem like ?1.Out, it's likely a missing tracked and we can point out where to add one.

docs/_docs/reference/experimental/modularity.md

bjornregnell · 2024-04-04T09:12:13Z

docs/_docs/reference/experimental/modularity.md

+```scala
+class D extends C { type T = Int; def m(): 22 }: // error
+  def next(): D
+```


Perhaps explain why this must be illegal?

bjornregnell · 2024-04-04T09:14:15Z

docs/_docs/reference/changed-features/implicit-resolution.md

-**9.** The following change is currently enabled in `-source future`:
+
+**9.** Given disambiguation has changed. When comparing two givens that both match an expected type, we used to pick the most specific one, in alignment with
+overloading resolution. From Scala 3.5 on, we pick the most general one instead. Compiling with Scala 3.5-migration will print a warning in all cases where the preference has changed. Example:


Explanation of this would be good: Why is this change needed? (And what migration actions can be done in code that rely on the old behavior, so that it still works with the new behavior?)

This is a separate PR: #19300

It is rolled into this PR because this PR contains examples where we have triangles such as in the parser combinators, where two type classes extend Combinator. The new rules are needed to resolve these better. And they seem not to break anything, empirically! #19300 contains a discussion in the comments.

bjornregnell · 2024-04-04T09:22:03Z

docs/_docs/reference/experimental/typeclasses.md

+
+
+## Auxiliary Type Alias `is`
+


I think I like is better than forms so good that you changed back! (I think forms is esoteric lingo; I could also perhaps learn to like implements but is is shorter and does not stick in the eye as much, and is almost as silent as a colon but still carries reasonable semantics).

I think is indeed is less esoteric lingo. But it is semantically wrong how Scala is used today. If you look at upickle, play-json, circe, cats, zio-prelude, akka, etc. then you have type classes like Format, Reader, Writer, Encoder, Decoder, Monoid, Ordering, Marhsaller and similar. For all these things you would say has and not is.

If you want parties to migrate to Scala 3 you need to take into consideration what is in production today. And hypothetical type classes like Collection and Readable that are not used in real-world libraries (in Scala) should not take preference.

I know this is bike-shedding, but I don't think the reputation Scala currently has helps if these type of changes are made without broad consensus and support.

So from here you have implicit val decodeFoo: Decoder[Thing] = new Decoder[Thing]{...} which would be

given Thing is Decoder: ...

and you would like

given Thing has Decoder: ...

We could have both as it is just a Predef type alias.

But anyone could also if is is too itchy:

type Decoded = Decoder given Thing is Decoded: ...

and it reads a bit better.

Maybe it's because I'm a non-native English speaker that I'm more pragmatic about

given Thing is Decoder

I'm fine with it because it kind of is a Decoder in the sense that Thing now can do decoding, which is what the Decoder is doing for it behind the scenes.

We could have both as it is just a Predef type alias.

Sure, but I do think has is a much better default than is if you look at popular type classes in Scala 2 today. Also it is IMO better to not give too much choice and stick with the most logical choice looking at how Scala is used.

given Thing is Decoder

IMO if you make Thing concrete for actual types like Int, String and Person the is stands completely out of place. Now again, I know this is nitpicking. BUt I had the idea that there was much more consensus for has the last time this discussion was held and it is ignored (again).

(Aside: this thread comments a commit that is part of early experimentation even before a subsequent improvement proposal is submitted. No comments will be "ignored" as we will continue to discuss all trade-offs. All previous discussions is input to the SIP committee as we come to decision-making about this so I don't think it is fair to say "ignored (again)" here; this thread is open to the whole planet earth to read and comment and the SIP committee's intention is to be benevolent of all factual input and try to make good trade-offs based on what is included both in written proposals and in pointers to open discussions etc.)

Anyways, back to the factual matters: The proposal regards new type classes that use the new Self-type-member scheme; the old style wont go away as type parameters and given-using wont go away. So the naming of type-param-based type classes can stay the same. For new type-member-based type classes the naming can take the is-type-alias into account. And the nice thing with is being just a type alias in Predef is that anyone can define has in a similar way locally.

I actually think this makes sense:

given Int is Decoder

if you think of it as the Int class is given the ability to be a Decoder of serialized integer values. You can read it as "given that Int is a Decoder then we can use Int to decode serialized Int values".

The proposal regards new type classes that use the new Self-type-member scheme; the old style wont go away as type parameters and given-using wont go away.

That will mean that Scala will have 2 different ways of defining and using typeclasses. And in reality you can add implicit to that mix as well for as long as it is still being supported.

Well, that's a price to pay when embracing both innovation, stability and ease of migration. We should consider and balance trade-offs between stability and innovation in each case. And, after these experiments are turned into a proposal the SIP committee will take all input into account and vote to try to make the best decision. But this is still early days and I think it is good if these experiments that try to leverage the recent abilities of the type system can get out there in a reasonably stable "best-possible-for-now"-version for everyone to play with under a flag to give feedback and surface unanticipated problems etc.

bjornregnell · 2024-04-04T09:48:30Z

docs/_docs/reference/experimental/typeclasses.md

+
+**Proposal:** Introduce a new way to implement a given definition in a trait like this:
+```scala
+given T = deferred


Why not call it abstract instead of the perhaps more cryptic deferred? After all it is called an "abstract given" in the text...

abstract given T

Abstract givens are different. Here we have a concrete given from a users point of view. It's just that the implementation is spread out over subclasses. But you see that it is concrete because if you want to implement it yourself in a subclass you need an override modifier.

Note that to write an implementation with override, one need to use the exact generated name, which might not be easy to guess (even though there are clear rules for these names).

trait Ord: type Self trait Sorted: type Element: Ord object Scoped: given (Int is Ord)() class SortedIntCorrect extends Sorted: type Element = Int class SortedIntCorrect2 extends Sorted: type Element = Int override given (Int is Ord)() as given_Ord_Element class SortedIntWrong1 extends Sorted: type Element = Int override given (Element is Ord)() // error class SortedIntWrong2 extends Sorted: type Element = Int override given (Int is Ord)() // error

so I would expect reasonable code to be like SortedIntCorrect instead of SortedIntCorrect2.

bjornregnell · 2024-04-04T09:55:08Z

docs/_docs/reference/experimental/typeclasses.md

+given String is Ord:
+  def compare(x: String, y: String) = ...
+
+given [A : Ord] => List[A] is Ord:


I'd like an explanation here of what the syntax with the rocket means. I could not find any example with rockets in the other existing doc pages on given instances etc. There are a lot of symbols here, colon and brackets and arrows and what not so an explanation will help the reader a lot (at least me :) ).

This currently looks very weird to me, as I am used to think about givens as special vals or defs. (Anyway, probably not the place and time to discuss the syntax.)

bjornregnell · 2024-04-04T09:59:27Z

docs/_docs/reference/experimental/typeclasses.md

+starts migrating.
+
+Migration to the new syntax is straightforward, and can be supported by automatic rewrites. For a transition period we can support both the old and the new syntax. It would be a good idea to backport the new given syntax to the LTS version of Scala so that code written in this version can already use it. The current LTS would then support old and new-style givens indefinitely, whereas new Scala 3.x versions would phase out the old syntax over time.
+


I think this reasoning makes sense. In 10 years I think we will find that this was the right decision even if the change-it-again-after-recent-change-scenario is painful.

bjornregnell · 2024-04-04T10:01:27Z

docs/_docs/reference/experimental/typeclasses.md

+    def pure[A](x: A): List[A] =
+      List(x)
+
+  type Reader[Ctx] = [X] =>> Ctx => X


This looks hairy. Perhaps explain with a comment what it means, and refer to relevant doc pages.

docs/_docs/reference/experimental/typeclasses.md

odersky · 2024-04-04T16:27:00Z

About naming "is/Self" vs "forms/Something else": Note that the is syntax will only work for type classes that are defined in terms of Self. So the argument that existing type classes like Reader don't follow this naming convention is moot. They are not defined in terms of Self (not should they be), and they cannot be summoned with is. They can still be used in context bounds, as is the case today, of course.

The reasoning for "is/Self" is as follows. With this PR, we use context bounds X: B very much like type ascriptions x: T. They both mean similar things, that's why they have the same syntax. For instance, if m is a member of a type T and x: T, then you can write x.m. The same holds for members m of a type class C appearing in a context bound X: C: you can write X.m. Examples are X.unit or X.Element.

Now, if x: T, we pronounce that as x is (a) T. If you are a stickler, you might insist that it should be isa, not is. But is is much more common, for instance in type tests x is T in other languages. So analogously, if X: C we would pronounce that X is C. E.g. X is Monoidal, X is Numeric, X is Typeable, and so on. It's not the "is" in the meaning of identity, but the "is" in the meaning of an adjective, such as "the Sky is blue". Type classes characterize types, and in that sense play the linguistic role of adjectives. That's why, despite all objections I find is to work much better than forms or other alternatives.

In light of the adjective analogy, maybe we we should tweak the names of our example type classes some more. Should be Monoidal instead of Monoid, or Monadic instead of Monad? Maybe that's overthinking it. List is Monad is slightly off, but still quite common and easily understandable.

sideeffffect · 2024-04-04T18:20:47Z

If you excuse a well meant comment from the peanut gallery 🙏

If we want to go ahead with using is, we should be thorough and consistent. When we have e.g.

  given Int is Ord:

we should also use is like this

  def minimum[T is Ord](xs: List[T]) =

or

  given [T is Ord] => List[T] is Ord:

I think retiring : from context bounds is desirable. It is confusing, because it's yet another role the character : plays (besides type annotations and new block demarcations). Using is for this purpose would also be much more regular and thus easier for the newcomers to learn.

We can keep : in context bounds for backward compatibility, but all these new features should only work with is.

Note that my proposal doesn't really care whether it is is or forms or whatever else gets picked, the point is to use it everywhere, including the places where : is currently used.

(And we could even drop the [/], but that would be for another discussion)

  given T is Ord => List[T] is Ord:

bishabosha · 2024-04-05T08:22:50Z

I think retiring : from context bounds is desirable. It is confusing, because it's yet another role the character : plays (besides type annotations and new block demarcations).

I think this is focusing too much on small details - T: Ord is very much in spirit with xs: List[T] - X conforms to some constraint Y.

Using is for this purpose would also be much more regular and thus easier for the newcomers to learn.

But then you are using is as both a keyword to introduce a context bound to a type parameter declaration (T is Ord), and also as an infix-type at use-site (List[T] is Ord). This is surely more confusing

sideeffffect · 2024-04-05T13:11:05Z

X conforms to some constraint Y

It's this vagueness which people will find confusing.

using is as both a keyword to introduce [...] and also as an infix-type at

I am certain that is not how people will think of it. And not just the beginners, but ordinary Scala developers. They wouldn't think about the first par as using keywords and the other part as using infix types. For them, it would be the same thing.

If anything, this shows that if we really want to improve the Type Class situation in Scala, it would be desirable in the part which is currently "infix-type at use-site (List[T] is Ord)" to turn the is there into keyword as well. That should make it more robust. I'm worried that using only infix types is fragile and that the error messages won't be friendly.

odersky · 2024-04-05T19:54:34Z

One new thing over the previous PR is that we fix is the handling of Singleton. X <: Singleton is wrong and unsound in the presence of unions. We have 1 <: Singleton and 2 <: Singleton and therefore by the laws of subtyping union types, 1 | 2 <: Singleton. But that makes no sense. The proper approach is to treat Singleton as a type class. It should be X: Singleton to indicate that instances of X are all singletons. The problem is how to get there, yet keep the familiar name Singleton. Self-based type classes afre a solution. We now define singleton as

trait Singleton extends Any:
  type Self

That's compatible with the previous subtyping constraints, but also allows the new context bound syntax. Furthermore, we treat Singleton as an erased class, and that means that no code will generated for witness arguments.

odersky · 2024-04-07T13:39:49Z

We now also implement precise type inference using a Precise typeclass trait. After the change to Singleton, this was relatively straightforward except we hit some complications with constant folding, where precise type variables were constrained from below with the unfolded underlying type instead of the folded constant type.

soronpo · 2024-04-14T14:45:27Z

Regarding the Precise type class, does it also keep tuples precise?
I think it would good to look at #15765 for test cases. It has pretty good coverage for feature interaction.

odersky · 2024-04-14T16:12:34Z

I believe the answer will be "it depends". A precise type variable gets instantiated without widening of singletons or unions at the top-level.vA variable is precise if it is declared with a Precise context bound, or if it has another precise type variable as a part of its upper bound.

Great idea to check behavior and expressiveness using the test cases of #15765.

bishabosha · 2024-04-30T12:10:10Z

library/src/scala/runtime/stdLibPatches/Predef.scala

   */
-  @experimental
  infix type is[A <: AnyKind, B <: Any{type Self <: AnyKind}] = B { type Self = A }


does it still appear experimental in TASTy? otherwise I guess we'd have to make this PR needs-minor-release to avoid leaking to prior compilers in same minor

It will hopefully get into 3.5

It will hopefully get into 3.5

"self" and "is" both will be added in scala 3.5?

We already reduce `R { type A = T } # A` to `T` in most situations when we create types. We now also reduce `R { val x: S } # x` to `S` if `S` is a singleton type. This will simplify types as we go to more term-dependent typing. As a concrete benefit, it will avoid several test-pickling failures due to pickling differences when using dependent types.

If a context bound type `T` for type parameter `A` does not have type parameters, demand evidence of type `T { type Self = A }` instead.

Allow to constrain type variables to be singletons by a context bound [X: Singleton] instead of an unsound supertype [X <: Singleton]. This fixes the soundness hole of singletons.

Now: 3.5: old scheme but warn if there are changes in the future 3.6-migration: new scheme, warn if prioritization has changed 3.6: new scheme, no warning

If they are illegally used as values, we need to return an error tree, not a tree with a symbol that can't be pickled.

Error number changed

odersky force-pushed the typeclass-experiments-refactored branch 4 times, most recently from 5a65e4b to f38d3c7 Compare April 2, 2024 21:29

odersky mentioned this pull request Apr 4, 2024

Pre-SIP: Improvements to Type Classes #19395

Closed

bishabosha reviewed Apr 4, 2024

View reviewed changes

docs/_docs/reference/experimental/typeclasses.md Show resolved Hide resolved

bjornregnell reviewed Apr 4, 2024

View reviewed changes

odersky force-pushed the typeclass-experiments-refactored branch from 76d519c to fccba2b Compare April 5, 2024 18:21

odersky marked this pull request as ready for review April 5, 2024 18:54

odersky added the needs-minor-release This PR cannot be merged until the next minor release label Apr 5, 2024

odersky force-pushed the typeclass-experiments-refactored branch 3 times, most recently from 4f1fc94 to 0be5237 Compare April 7, 2024 12:43

odersky mentioned this pull request Apr 7, 2024

Pre SIP: Improvements to Modularity #18958

Closed

odersky force-pushed the typeclass-experiments-refactored branch 2 times, most recently from f72b090 to 706bcd1 Compare April 14, 2024 14:23

odersky force-pushed the typeclass-experiments-refactored branch from 706bcd1 to 7887fe1 Compare April 15, 2024 10:35

odersky self-assigned this Apr 27, 2024

odersky force-pushed the typeclass-experiments-refactored branch 3 times, most recently from c6f7c74 to cf0a97e Compare April 30, 2024 12:03

bishabosha reviewed Apr 30, 2024

View reviewed changes

odersky added 18 commits May 7, 2024 14:48

Implement context bound companions

ce09ef3

Allow contecxt bounds with abstract Self types

c6388c2

If a context bound type `T` for type parameter `A` does not have type parameters, demand evidence of type `T { type Self = A }` instead.

Add a doc page

f444b46

Fix Singleton

f713652

Allow to constrain type variables to be singletons by a context bound [X: Singleton] instead of an unsound supertype [X <: Singleton]. This fixes the soundness hole of singletons.

Tweaks to doc pages

1f2e735

Add Precise type class for precise type inference

94bc6fe

Fix rebase breakage

887fbc4

Delay roll-out of new prioritization scheme:

1e72282

Now: 3.5: old scheme but warn if there are changes in the future 3.6-migration: new scheme, warn if prioritization has changed 3.6: new scheme, no warning

Fix rebase breakage again

9d0ca20

Make best effort compilation work with context bound companions

fd072dc

If they are illegally used as values, we need to return an error tree, not a tree with a symbol that can't be pickled.

Tweaks after review

21f5e67

Fix rebase breakage

d3e6a95

Make Singleton an erased class only under modularity import

b2f0791

Address review comments

4f28d8a

Adress review comments with changed docs and new tests

0dddcb7

Update warn check files

62e0244

Error number changed

Update InlayHints

9959f28

odersky force-pushed the typeclass-experiments-refactored branch from d060838 to e9bddd5 Compare May 7, 2024 13:10

Fix typo

3c78ada

odersky force-pushed the typeclass-experiments-refactored branch from e9bddd5 to 3c78ada Compare May 7, 2024 13:16

odersky merged commit 1f3c652 into scala:main May 7, 2024
19 checks passed

odersky deleted the typeclass-experiments-refactored branch May 7, 2024 19:22

odersky added the backport:nominated If we agree to backport this PR, replace this tag with "backport:accepted", otherwise delete it. label May 7, 2024

odersky modified the milestones: mbovel, 3.5.0 May 7, 2024

WojciechMazur added backport:done This PR was successfully backported. and removed backport:nominated If we agree to backport this PR, replace this tag with "backport:accepted", otherwise delete it. labels Jun 20, 2024

e45lee mentioned this pull request Jun 20, 2024

Modularity + TASTy serialization issue when compiling modified standard library #20707

Open

farnoy mentioned this pull request Oct 8, 2024

Associated types become opaque with indirection #21416

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Typeclass experiments refactored #20061

Typeclass experiments refactored #20061

odersky commented Apr 1, 2024 •

edited

Loading

bjornregnell Apr 4, 2024 •

edited

Loading

odersky Apr 5, 2024

bjornregnell Apr 4, 2024

bjornregnell Apr 4, 2024 •

edited

Loading

odersky Apr 5, 2024

bjornregnell Apr 4, 2024 •

edited

Loading

bmeesters Apr 4, 2024 •

edited

Loading

bjornregnell Apr 4, 2024 •

edited

Loading

bmeesters Apr 5, 2024

bjornregnell Apr 5, 2024

Jasper-M Apr 5, 2024

bjornregnell Apr 6, 2024

bjornregnell Apr 4, 2024

odersky Apr 5, 2024

mbovel May 6, 2024

bjornregnell Apr 4, 2024

mbovel May 6, 2024 •

edited

Loading

bjornregnell Apr 4, 2024

bjornregnell Apr 4, 2024 •

edited

Loading

odersky commented Apr 4, 2024 •

edited

Loading

sideeffffect commented Apr 4, 2024 •

edited

Loading

bishabosha commented Apr 5, 2024 •

edited

Loading

sideeffffect commented Apr 5, 2024

odersky commented Apr 5, 2024 •

edited

Loading

odersky commented Apr 7, 2024 •

edited

Loading

soronpo commented Apr 14, 2024 •

edited

Loading

odersky commented Apr 14, 2024

bishabosha Apr 30, 2024

odersky May 7, 2024

flysky9981 May 13, 2024

		starts migrating.

		Migration to the new syntax is straightforward, and can be supported by automatic rewrites. For a transition period we can support both the old and the new syntax. It would be a good idea to backport the new given syntax to the LTS version of Scala so that code written in this version can already use it. The current LTS would then support old and new-style givens indefinitely, whereas new Scala 3.x versions would phase out the old syntax over time.

Typeclass experiments refactored #20061

Typeclass experiments refactored #20061

Conversation

odersky commented Apr 1, 2024 • edited Loading

bjornregnell Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bjornregnell Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bjornregnell Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

bmeesters Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

bjornregnell Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbovel May 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bjornregnell Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

odersky commented Apr 4, 2024 • edited Loading

sideeffffect commented Apr 4, 2024 • edited Loading

bishabosha commented Apr 5, 2024 • edited Loading

sideeffffect commented Apr 5, 2024

odersky commented Apr 5, 2024 • edited Loading

odersky commented Apr 7, 2024 • edited Loading

soronpo commented Apr 14, 2024 • edited Loading

odersky commented Apr 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

odersky commented Apr 1, 2024 •

edited

Loading

bjornregnell Apr 4, 2024 •

edited

Loading

bjornregnell Apr 4, 2024 •

edited

Loading

bjornregnell Apr 4, 2024 •

edited

Loading

bmeesters Apr 4, 2024 •

edited

Loading

bjornregnell Apr 4, 2024 •

edited

Loading

mbovel May 6, 2024 •

edited

Loading

bjornregnell Apr 4, 2024 •

edited

Loading

odersky commented Apr 4, 2024 •

edited

Loading

sideeffffect commented Apr 4, 2024 •

edited

Loading

bishabosha commented Apr 5, 2024 •

edited

Loading

odersky commented Apr 5, 2024 •

edited

Loading

odersky commented Apr 7, 2024 •

edited

Loading

soronpo commented Apr 14, 2024 •

edited

Loading