Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Edge Caching Mechanism in InitializeEdgesWorker #2827

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@
@AttributeDefinition(name = "Number of Threads", description = "Pool-Size: the maximum number of concurrent connections")
int pgConnectionPoolSize() default 40;

@AttributeDefinition(name = "Cache Refresh Interval", description = "Interval in minutes to refresh the Edges cache; 0 to disable refreshing")
int cacheRefreshInterval() default 20;

@AttributeDefinition(name = "Debug Mode", description = "Activates the debug mode")
DebugMode debugMode() default DebugMode.OFF;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
import java.sql.SQLException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
Expand All @@ -15,48 +17,75 @@
import io.openems.backend.metadata.odoo.Field.EdgeDevice;
import io.openems.common.utils.ThreadPoolUtils;

public class InitializeEdgesWorker {
public class EdgeCachingWorker {

private final Logger log = LoggerFactory.getLogger(InitializeEdgesWorker.class);
private final Logger log = LoggerFactory.getLogger(EdgeCachingWorker.class);

Check warning on line 22 in io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java#L22

Added line #L22 was not covered by tests
protected final PostgresHandler parent;
private final HikariDataSource dataSource;
private final Runnable onFinished;
private final ScheduledExecutorService scheduledExecutor = Executors.newScheduledThreadPool(1);
private boolean isMarkAllEdgesAsOfflineCalled = false;

Check warning on line 27 in io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java#L26-L27

Added lines #L26 - L27 were not covered by tests

/**
* Executor for subscriptions task.
*/
private final ExecutorService executor = Executors.newSingleThreadExecutor();

public InitializeEdgesWorker(PostgresHandler parent, HikariDataSource dataSource, Runnable onFinished) {
public EdgeCachingWorker(PostgresHandler parent, HikariDataSource dataSource, Runnable onFinished) {

Check warning on line 34 in io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java#L34

Added line #L34 was not covered by tests
this.parent = parent;
this.dataSource = dataSource;
this.onFinished = onFinished;
}

/**
* Starts the {@link InitializeEdgesWorker}.
* Starts the {@link EdgeCachingWorker}.
*/
public synchronized void start() {
this.executor.execute(() -> {
try (var con = this.dataSource.getConnection()) {
this.parent.logInfo(this.log, "Caching Edges from Postgres [started]");
this.markAllEdgesAsOffline(con);
this.readAllEdgesFromPostgres(con);
this.parent.logInfo(this.log, "Caching Edges from Postgres [finished]");
// First Execution Immediately
this.runCachingEdgesTask(con);

Check warning on line 47 in io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java#L47

Added line #L47 was not covered by tests

// Configure Refresh-Time
int refreshTimeInMinutes = this.parent.getRefreshTime();

Check warning on line 50 in io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java#L50

Added line #L50 was not covered by tests

// Schedule Execution only if refreshTime is greater than 0
if (refreshTimeInMinutes > 0) {
this.scheduledExecutor.scheduleAtFixedRate(() -> {
try (var newCon = this.dataSource.getConnection()) {
this.runCachingEdgesTask(newCon);
} catch (SQLException e) {
this.logError("Error while re-connecting to Postgres dataSource.", e);
}
}, refreshTimeInMinutes, refreshTimeInMinutes, TimeUnit.MINUTES);

Check warning on line 60 in io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java#L54-L60

Added lines #L54 - L60 were not covered by tests
} else {
this.parent.logInfo(this.log, "Refresh time is set to 0. Caching logic is deactivated.");

Check warning on line 62 in io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java#L62

Added line #L62 was not covered by tests
}
} catch (SQLException e) {
this.parent.logWarn(this.log, "Caching Edges from Postgres [canceled]");
this.logError("Unable to connect do dataSource. ", e);
this.logError("Unable to connect to dataSource.", e);

Check warning on line 65 in io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java#L65

Added line #L65 was not covered by tests
}
this.onFinished.run();
});
}

private void runCachingEdgesTask(Connection con) {
this.parent.logInfo(this.log, "Caching Edges from Postgres [started]");

Check warning on line 72 in io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java#L72

Added line #L72 was not covered by tests
// Check if markAllEdgesAsOffline has already been called
if (!this.isMarkAllEdgesAsOfflineCalled) {
this.markAllEdgesAsOffline(con);
this.isMarkAllEdgesAsOfflineCalled = true;

Check warning on line 76 in io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java#L75-L76

Added lines #L75 - L76 were not covered by tests
}
this.readAllEdgesFromPostgres(con);
this.parent.logInfo(this.log, "Caching Edges from Postgres [finished]");
}

Check warning on line 80 in io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java#L78-L80

Added lines #L78 - L80 were not covered by tests

/**
* Stops the {@link InitializeEdgesWorker}.
* Stops the {@link EdgeCachingWorker}.
*/
public synchronized void stop() {
// Shutdown executor
ThreadPoolUtils.shutdownAndAwaitTermination(this.executor, 5);
ThreadPoolUtils.shutdownAndAwaitTermination(this.scheduledExecutor, 5);

Check warning on line 88 in io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/EdgeCachingWorker.java#L88

Added line #L88 was not covered by tests
}

private void markAllEdgesAsOffline(Connection con) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,10 @@

protected final EdgeCache edgeCache;

private final Config config;
private final MetadataOdoo parent;
private final HikariDataSource dataSource;
private final InitializeEdgesWorker initializeEdgesWorker;
private final EdgeCachingWorker initializeEdgesWorker;
private final PeriodicWriteWorker periodicWriteWorker;

public PostgresHandler(MetadataOdoo parent, EdgeCache edgeCache, Config config, Runnable onInitialized)
Expand All @@ -32,7 +33,8 @@
this.edgeCache = edgeCache;
this.dataSource = this.getDataSource(config);
this.edge = new PgEdgeHandler(this.dataSource);
this.initializeEdgesWorker = new InitializeEdgesWorker(this, this.dataSource, () -> {
this.config = config;
this.initializeEdgesWorker = new EdgeCachingWorker(this, this.dataSource, () -> {

Check warning on line 37 in io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/PostgresHandler.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/PostgresHandler.java#L36-L37

Added lines #L36 - L37 were not covered by tests
onInitialized.run();
});
this.initializeEdgesWorker.start();
Expand Down Expand Up @@ -87,6 +89,10 @@
return result;
}

public int getRefreshTime() {
return this.config.cacheRefreshInterval();

Check warning on line 93 in io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/PostgresHandler.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.metadata.odoo/src/io/openems/backend/metadata/odoo/postgres/PostgresHandler.java#L93

Added line #L93 was not covered by tests
}

protected Connection getConnection() throws SQLException {
return this.dataSource.getConnection();
}
Expand Down