beam-scala-examples

Apache Beam examples with Scala using official Java API, and it's different from using Scio API.

NOTE: Scala 2.13.1 (lots of changes for collection migration) + Apache Beam 2.15 (certain methods in the examples using master branch are patched via implicit)

Progress

Word Count

• Minimal Word Count
• Word Count
• Debugging Word Count
• Windowed Word Count

Complete

• AutoComplete
• StreamingWordExtract
• TfIdf
• TopWikipediaSessions
• TrafficMaxLaneFlow
• TrafficRoutes

Complete/Game

• UserScore
• HourlyTeamScore
• StatefulTeamScore
• LeaderBoard
• GameStats

Cookbook

• BigQueryTornadoes
• CombinePerKeyExamples
• DistinctExample
• FilterExamples
• JoinExamples
• MaxPerKeyExamples
• TriggerExample

Subprocess

• ExampleEchoPipeline

Build

Clean up code formatting, treat warnings as errors and build the Assembly Jar

$ ./gradlew licenseFormatMain spotlessApply assemblyJar

Run

Mimic mvn run

$ ./gradlew exec \
    -PmainClass=org.apache.beam.examples.scala.WordCount \
    --args='--output=/tmp/wc/wc'

Run using jars

$ ./gradlew execJar \
    -PmainClass=org.apache.beam.examples.scala.WordCount \
    --args='--output=/tmp/wc/wc --inputFile=/tmp/tagdata.txt'

Local Dev

Per work space settings with coc.nvim and scalameta/metals LSP

• plugins {} and version properties doesn't work with gradle 5.3.1 that metals was bundled with for bootstrapping
• Metals .jvmopts and .sbtopts per workspace doesn't work nicely with coc.nvim. Have to modify server settings directly

Both of the above changes requires modification for global settings. However, coc.nvim provide configuration file resolved per workspace. Create .vim folder with coc-settings.json file for per workspace configuration (point to latest gradle version + scalafmtConfigPath) without polluting global settings.

{
  "languageserver": {
    "metals": {
      "command": "metals-vim",
      "rootPatterns": ["build.sbt", "build.sc", "build.gradle"],
      "filetypes": ["scala", "sbt", "gradle"],
      "settings": {
        "metals": {
          "gradleScript": "/usr/bin/gradle or /home/user/.sdkman/...",
          "scalafmtConfigPath": "codequality/scalafmt.conf"
        }
      }
    }
  }
}

License

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0