Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom ResourceReader implementation in Golang fails to decode read data #656

Open
barahona42 opened this issue Sep 22, 2024 · 3 comments
Open

Comments

@barahona42
Copy link

context

Hi all, first time submitting a bug report ever. Here's the info:

  • What are you trying to achieve?
    • Implement a way for pkl to retrieve configuration values from AWS SSM Parameters when read("ssm:/foo/SomeParameterPath") is assigned to a pkl resource.
    • Working language: go1.23.0
  • What's the impact of this bug?
    • Unmarshaling/resolving values from a module is failing.

outputs

  • pkl --version: Pkl 0.26.3 (macOS 14.3.1, native)
  • complete error message:
    java.lang.InterruptedException
    
    –– Pkl Error ––
    None (cause has no message)
    
    1 | SomeParameter = read("ssm:ParameterName")
        ^^^^^^^^^^^^^^^^
    at config#SomeParameter (file:///etc/server/config.pkl)
    
    Pkl 0.26.3 (Linux 5.15.0-1057-aws, native)
    
    java.lang.InterruptedException
            at [email protected]/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:386)
            at [email protected]/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
            at org.pkl.server.UtilsKt.getUnderlying(Utils.kt:64)
            at org.pkl.server.ClientResourceReader.doRead(ClientResourceReader.kt:104)
            at org.pkl.server.ClientResourceReader.read(ClientResourceReader.kt:46)
            at org.pkl.core.runtime.ResourceManager.lambda$read$1(ResourceManager.java:91)
            at [email protected]/java.util.HashMap.computeIfAbsent(HashMap.java:1220)
            at org.pkl.core.runtime.ResourceManager.read(ResourceManager.java:72)
            at org.pkl.core.ast.expression.unary.AbstractReadNode.doRead(AbstractReadNode.java:55)
            at org.pkl.core.ast.expression.unary.ReadNode.read(ReadNode.java:33)
            at org.pkl.core.ast.expression.unary.ReadNodeGen.executeAndSpecialize(ReadNodeGen.java:75)
            at org.pkl.core.ast.expression.unary.ReadNodeGen.executeGeneric(ReadNodeGen.java:66)
            at org.pkl.core.ast.PklRootNode.executeBody(PklRootNode.java:41)
            at org.pkl.core.ast.MemberNode.executeBody(MemberNode.java:53)
            at org.pkl.core.ast.member.TypeCheckedPropertyNode.evalTypedObjectCached(TypeCheckedPropertyNode.java:54)
            at org.pkl.core.ast.member.TypeCheckedPropertyNodeGen.executeAndSpecialize(TypeCheckedPropertyNodeGen.java:136)
            at org.pkl.core.ast.member.TypeCheckedPropertyNodeGen.execute(TypeCheckedPropertyNodeGen.java:101)
            at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.executeRootNode(OptimizedCallTarget.java:718)
            at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.profiledPERoot(OptimizedCallTarget.java:641)
            at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callBoundary(OptimizedCallTarget.java:574)
            at org.graalvm.nativeimage.builder/com.oracle.svm.truffle.api.SubstrateOptimizedCallTarget.invokeCallBoundary(SubstrateOptimizedCallTarget.java:115)
            at com.oracle.svm.svm_enterprise/com.oracle.svm.enterprise.truffle.SubstrateEnterpriseOptimizedCallTarget.a(stripped:284)
            at com.oracle.svm.svm_enterprise/com.oracle.svm.enterprise.truffle.SubstrateEnterpriseOptimizedCallTarget.doInvoke(stripped:250)
            at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callIndirect(OptimizedCallTarget.java:486)
            at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.call(OptimizedCallTarget.java:467)
            at org.graalvm.truffle/com.oracle.truffle.api.nodes.IndirectCallNode$1.call(IndirectCallNode.java:91)
            at org.pkl.core.runtime.VmUtils.doReadMember(VmUtils.java:305)
            at org.pkl.core.runtime.VmUtils.doReadMember(VmUtils.java:218)
            at org.pkl.core.runtime.VmObject.force(VmObject.java:177)
            at org.pkl.core.runtime.VmObject.force(VmObject.java:197)
            at org.pkl.core.runtime.VmValue.force(VmValue.java:67)
            at org.pkl.server.BinaryEvaluator.evaluate$lambda-2(BinaryEvaluator.kt:62)
            at org.pkl.core.EvaluatorImpl.lambda$doEvaluate$13(EvaluatorImpl.java:353)
            at org.pkl.core.EvaluatorImpl.doEvaluate(EvaluatorImpl.java:301)
            at org.pkl.core.EvaluatorImpl.doEvaluate(EvaluatorImpl.java:349)
            at org.pkl.server.BinaryEvaluator.evaluate(BinaryEvaluator.kt:58)
            at org.pkl.server.Server.handleEvaluate$lambda-0(Server.kt:101)
            at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
            at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
            at [email protected]/java.lang.Thread.run(Thread.java:842)
            at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:807)
            at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:210)
    
    
    goroutine 1 [running]:
    cdr-getter-api/configuration.AppConfig()
            /app/configuration/configuration.go:51 +0x1f6
    cdr-getter-api/controllers.init()
            /app/controllers/globals.go:24 +0x17
    bash-5.2# An unexpected error has occurred. Would you mind filing a bug report?
    
    java.lang.NullPointerException
            at org.pkl.server.ClientResourceReader$doRead$1$1$1.invoke(ClientResourceReader.kt:94)
            at org.pkl.server.ClientResourceReader$doRead$1$1$1.invoke(ClientResourceReader.kt:88)
            at org.pkl.server.MessageTransports$AbstractMessageTransport.accept(MessageTransports.kt:98)
            at org.pkl.server.MessageTransports$EncodingMessageTransport.doStart(MessageTransports.kt:50)
            at org.pkl.server.MessageTransports$AbstractMessageTransport.start(MessageTransports.kt:110)
            at org.pkl.server.Server.start(Server.kt:42)
            at org.pkl.cli.CliServer.doRun(CliServer.kt:29)
            at org.pkl.commons.cli.CliCommand.run(CliCommand.kt:44)
            at org.pkl.cli.commands.ServerCommand.run(ServerCommand.kt:30)
            at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:198)
            at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:211)
            at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:18)
            at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:400)
            at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:397)
            at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:415)
            at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:440)
            at org.pkl.cli.Main$main$1.invoke(Main.kt:39)
            at org.pkl.cli.Main$main$1.invoke(Main.kt:27)
            at org.pkl.commons.cli.CliMainKt.cliMain(CliMain.kt:33)
            at org.pkl.cli.Main.main(Main.kt:27)
    
  • Reproducing:
    • contents of .pkl file:
      SomeParameterValue = read("ssm:/foo/SomeParameterPath")
      
    • implement a a custom evaluator whose Read method returns a byte array as shown below:
      // from the pkl.ResourceReader requirement of pkl.Reader interface
      func (r PklResourceReader) Read(url url.URL) ([]byte, error) {
          // v is type *string 
          v, err := GetDecryptedParameter(context.Background(), url.Path)
          if err != nil {
              return nil, err
          }
          if v == nil {
              return make([]byte, 0), nil
          }
          var b []byte
          // if err := msgpack.NewEncoder(bytes.NewBuffer(b)).EncodeString(*v); err != nil {
          //     return nil, err
          // }
          return b, nil
      }
    • load the file using the pkl module's .Load() method.
    • note that the contents of b do make a difference:
      • on an empty array, the error above is shown.
      • on a non-empty array like []byte(*v), []byte("foo"), the following error returns:
        msgpack: invalid code=94 decoding string/bytes length
        

Any pointers/insight on this would be appreciated. Thank you.

@HT154
Copy link
Contributor

HT154 commented Sep 22, 2024

The issue with returning an empty/nil byte slice is definitely a bug! The read contents are assumed to be non-null even though an empty byte slice on the go side is encoded as nil. A fix similar to #480 is needed for this, which should be pretty straightforward. I'll probably have a PR up for this later today or tomorrow.

@HT154
Copy link
Contributor

HT154 commented Sep 22, 2024

The other issue is not as clear. Can you share some of the exact code that produced that error?

You definitely don't need to do the msgpack encoding yourself, just return the string produced by your resource directly, eg. return []byte("foo"), nil

PS: great bug report! ❤️

HT154 added a commit to HT154/pkl that referenced this issue Sep 22, 2024
In messages "Read Resource Response" and "Read Module Response", if `contents` and `error` are both null, default to an empty byte array/string.

This resolves one of the issues reported in apple#656
@barahona42
Copy link
Author

Thanks for the response. Here's the implementation of the resource and the code that gets the ssm parameter value.

// wraps ssm call
func GetDecryptedParameter(ctx context.Context, name string) (*string, error) {
	// do some service initialization
	input := ssm.GetParameterInput{Name: aws.String(name), WithDecryption: aws.Bool(true)} 
        // call to aws sdk method
	output, err := svc.GetParameter(ctx, &input)
	if err != nil {
		return nil, err
	}
	return output.Parameter.Value, nil
} 


// pkl.ResourceReader implementation
type PklResourceReader struct{}

// Scheme: pkl.Reader implementation
func (r PklResourceReader) Scheme() string {
	return "ssm"
}

// IsGlobbable: pkl.Reader implementation
func (r PklResourceReader) IsGlobbable() bool {
	return false
}

// HasHierarchicalUris: pkl.Reader implementation
func (r PklResourceReader) HasHierarchicalUris() bool {
	return false
}

// ListElements: pkl.Reader implementation
func (r PklResourceReader) ListElements(url url.URL) ([]pkl.PathElement, error) {
	return nil, nil
}

// Read: pkl.Reader implementation
func (r PklResourceReader) Read(url url.URL) ([]byte, error) {
	v, err := GetDecryptedParameter(context.Background(), url.Path)
	if err != nil {
		return nil, err
	}
	if v == nil {
		return make([]byte, 0), nil
	}
	return []byte(*v), nil
}

bioball pushed a commit that referenced this issue Oct 1, 2024
In messages "Read Resource Response" and "Read Module Response", if `contents` and `error` are both null, default to an empty byte array/string.

This resolves one of the issues reported in #656
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants