Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GOBBLIN-1914] Configurable Gobblin Application Master class for Yarn #3781

Merged
merged 2 commits into from
Sep 21, 2023
Merged

[GOBBLIN-1914] Configurable Gobblin Application Master class for Yarn #3781

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
471a96e
[GOBBLIN-1914] Configurable Gobblin Application Master class for Yarn
homatthew Sep 20, 2023
2a5ff84
Configurable logs
homatthew Sep 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -801,23 +801,34 @@ private void addJobConfPackage(String jobConfPackagePath, Path destDir, Map<Stri

@VisibleForTesting
protected String buildApplicationMasterCommand(String applicationId, int memoryMbs) {
String appMasterClassName = GobblinApplicationMaster.class.getSimpleName();
Class appMasterClass;
try {
String appMasterClassName = ConfigUtils.getString(
config, GobblinYarnConfigurationKeys.APP_MASTER_CLASS, GobblinYarnConfigurationKeys.DEFAULT_APP_MASTER_CLASS);
appMasterClass = Class.forName(appMasterClassName);
} catch (ClassNotFoundException e) {
throw new RuntimeException(e);
}

String logFileName = ConfigUtils.getString(config,
GobblinYarnConfigurationKeys.APP_MASTER_LOG_FILE_NAME, appMasterClass.getSimpleName());

return new StringBuilder()
.append(ApplicationConstants.Environment.JAVA_HOME.$()).append("/bin/java")
.append(" -Xmx").append((int) (memoryMbs * this.jvmMemoryXmxRatio) - this.jvmMemoryOverheadMbs).append("M")
.append(" -D").append(GobblinYarnConfigurationKeys.JVM_USER_TIMEZONE_CONFIG).append("=").append(this.containerTimezone)
.append(" -D").append(GobblinYarnConfigurationKeys.GOBBLIN_YARN_CONTAINER_LOG_DIR_NAME).append("=").append(ApplicationConstants.LOG_DIR_EXPANSION_VAR)
.append(" -D").append(GobblinYarnConfigurationKeys.GOBBLIN_YARN_CONTAINER_LOG_FILE_NAME).append("=").append(appMasterClassName).append(".").append(ApplicationConstants.STDOUT)
.append(" -D").append(GobblinYarnConfigurationKeys.GOBBLIN_YARN_CONTAINER_LOG_FILE_NAME).append("=").append(logFileName).append(".").append(ApplicationConstants.STDOUT)
.append(" ").append(JvmUtils.formatJvmArguments(this.appMasterJvmArgs))
.append(" ").append(GobblinApplicationMaster.class.getName())
.append(" ").append(appMasterClass.getName())
.append(" --").append(GobblinClusterConfigurationKeys.APPLICATION_NAME_OPTION_NAME)
.append(" ").append(this.applicationName)
.append(" --").append(GobblinClusterConfigurationKeys.APPLICATION_ID_OPTION_NAME)
.append(" ").append(applicationId)
.append(" 1>").append(ApplicationConstants.LOG_DIR_EXPANSION_VAR).append(File.separator).append(
appMasterClassName).append(".").append(ApplicationConstants.STDOUT)
logFileName).append(".").append(ApplicationConstants.STDOUT)
homatthew marked this conversation as resolved.
Show resolved Hide resolved
.append(" 2>").append(ApplicationConstants.LOG_DIR_EXPANSION_VAR).append(File.separator).append(
appMasterClassName).append(".").append(ApplicationConstants.STDERR)
logFileName).append(".").append(ApplicationConstants.STDERR)
.toString();
}

Expand Down Expand Up @@ -1071,4 +1082,4 @@ public void run() {

gobblinYarnAppLauncher.launch();
}
}
}
3 changes: 3 additions & 0 deletions gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnConfigurationKeys.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ public class GobblinYarnConfigurationKeys {
public static final String GOBBLIN_YARN_PREFIX = "gobblin.yarn.";

// General Gobblin Yarn application configuration properties.
public static final String APP_MASTER_CLASS = GOBBLIN_YARN_PREFIX + "app.master.class";
public static final String DEFAULT_APP_MASTER_CLASS = GobblinApplicationMaster.class.getName();
Copy link
Contributor

@phet phet Sep 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are in the same package, so potentially OK... but just noting this introduces a circular dependency between the two classes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth calling out. I can't think of a scenario where this would be a problem but open to input

public static final String APP_MASTER_LOG_FILE_NAME = GOBBLIN_YARN_PREFIX + "app.master.log.file.name";
public static final String APPLICATION_NAME_KEY = GOBBLIN_YARN_PREFIX + "app.name";
public static final String APP_QUEUE_KEY = GOBBLIN_YARN_PREFIX + "app.queue";
public static final String APP_REPORT_INTERVAL_MINUTES_KEY = GOBBLIN_YARN_PREFIX + "app.report.interval.minutes";
Expand Down
3 changes: 3 additions & 0 deletions gobblin-yarn/src/test/java/org/apache/gobblin/yarn/GobblinYarnAppLauncherTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,9 @@ public void testBuildApplicationMasterCommand() {

// 41 is from 64 * 0.8 - 10
Assert.assertTrue(command.contains("-Xmx41"));
Assert.assertTrue(command.contains("org.apache.gobblin.yarn.GobblinApplicationMaster"));
Assert.assertTrue(command.contains("GobblinApplicationMaster.stdout"));
Assert.assertTrue(command.contains("GobblinApplicationMaster.stderr"));
}

@Test
Expand Down
Loading