Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DSM] PEPPER-1402 PEPPER-1397 refactoring DSM event transmission into DSS and retrying failed events #2874

Merged
merged 36 commits into from
Jun 6, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
b75f783
PEPPER-1402 WIP some refactoring and test working now
pegahtah Apr 11, 2024
8f7ce29
Merge remote-tracking branch 'origin/develop' into PEPPER-1402_PEPPER…
pegahtah Apr 12, 2024
864f014
Merge branch 'develop' into PEPPER-1402_PEPPER-1397_trigger_ddp
pegahtah Apr 24, 2024
4e39668
PEPPER-1402 PEPPER-1397
pegahtah Apr 24, 2024
962e9f6
PEPPER-1402 PEPPER-1397
pegahtah Apr 24, 2024
d0d09f8
PEPPER-1402 PEPPER-1397
pegahtah Apr 24, 2024
ef86513
PEPPER-1402 PEPPER-1397
pegahtah Apr 26, 2024
5aca459
PEPPER-1402 PEPPER-1397
pegahtah Apr 26, 2024
b9aba34
PEPPER-1402 merged develop in and added changes
pegahtah May 9, 2024
a91eb51
PEPPER-1402 checkstyle fix
pegahtah May 9, 2024
2011496
Merge branch 'develop' into PEPPER-1402_PEPPER-1397_trigger_ddp
pegahtah May 21, 2024
c392262
PEPPER-1402 more refactoring, changed EventUtil to EventService
pegahtah May 21, 2024
73192f6
PEPPER-1402 PEPPER-1397 added tests for all scenarios, changed the cl…
pegahtah May 23, 2024
5c40fa7
PEPPER-1402 PEPPER-1397 checkstyle fix
pegahtah May 23, 2024
b483a7f
Merge remote-tracking branch 'origin/develop' into PEPPER-1402_PEPPER…
pegahtah May 23, 2024
2d31cc2
PEPPER-1402 PEPPER-1397 deleting unused classes
pegahtah May 23, 2024
4114d03
PEPPER-1402 PEPPER-1397 deleting unused classes
pegahtah May 23, 2024
71acc96
PEPPER-1402 PEPPER-1397 small clean ups
pegahtah May 23, 2024
a68831a
PEPPER-1402 PEPPER-1397 fixed a bug in TestHelper and moved dto and d…
pegahtah May 23, 2024
9a33f71
PEPPER-1402 PEPPER-1397 fixed java doc comment
pegahtah May 23, 2024
90db874
PEPPER-1402 PEPPER-1397 small letters
pegahtah May 23, 2024
f41e052
PEPPER-1402 PEPPER-1397 small letters
pegahtah May 23, 2024
991aad6
PEPPER-1402 changes to add jitters using IntervalFunction and Retry, …
pegahtah May 24, 2024
b060d15
PEPPER-1402 Adding logging
pegahtah May 28, 2024
0389076
PEPPER-1402 Moved DsmNotificationEventType.java and DsmNotificationPa…
pegahtah May 28, 2024
4a64816
Merge remote-tracking branch 'origin/develop' into PEPPER-1402_PEPPER…
pegahtah May 28, 2024
3b72df0
PEPPER-1402 based on PR comments I increased the initial wait time fo…
pegahtah Jun 3, 2024
4c27130
Merge remote-tracking branch 'origin/develop' into PEPPER-1402_PEPPER…
pegahtah Jun 3, 2024
5158814
PEPPER-1402 cleaned up code
pegahtah Jun 3, 2024
81336d8
PEPPER-1402 cleaned up code
pegahtah Jun 3, 2024
66308da
PEPPER-1402 cleaned up code
pegahtah Jun 3, 2024
f5834d3
PEPPER-1402 made BaseKitTestUtil an abstract class
pegahtah Jun 3, 2024
d8dd384
PEPPER-1402 changed BaseKitTestUtil into KitTestUtil and removed Pepp…
pegahtah Jun 5, 2024
c3dbf1c
Merge branch 'develop' into PEPPER-1402_PEPPER-1397_trigger_ddp
pegahtah Jun 6, 2024
66ddffb
Merge remote-tracking branch 'origin/develop' into PEPPER-1402_PEPPER…
pegahtah Jun 6, 2024
dad9982
PEPPER-1402 resolving conflicts
pegahtah Jun 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ public class KitRequestShipping extends KitRequest implements HasDdpInstanceId {
private static final String ERROR = "error";
private static final String SENT = "sent";
private static final String RECEIVED = "received";
private static final String OVERVIEW = "overview";
public static final String OVERVIEW = "overview";
private static final String DEACTIVATED = "deactivated";
private static final String TRIGGERED = "triggered";
private static final String WAITING = "waiting";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ protected static boolean triggerDssByKitEvent(KitDDPNotification kitDDPNotificat
@VisibleForTesting
protected static boolean triggerDssWithEvent(@NonNull String eventType, DDPInstance ddpInstance, long eventDate,
@NotNull String ddpParticipantId, @NotNull String eventInfo, KitReasonType reason) {
final long initialInterval = 100; // base delay in milliseconds
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I increased the initial interval

final long initialInterval = 500; // base delay in milliseconds
final double multiplier = 2.0; // exponential backoff multiplier

IntervalFunction intervalFn = IntervalFunction.ofExponentialBackoff(initialInterval, multiplier);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
import org.broadinstitute.dsm.db.dto.ddp.participant.ParticipantDto;
import org.broadinstitute.dsm.db.dto.mercury.ClinicalOrderDto;
import org.broadinstitute.dsm.db.dto.mercury.MercuryOrderDto;
import org.broadinstitute.dsm.kits.TestKitUtil;
import org.broadinstitute.dsm.kits.PepperKitUtil;
import org.broadinstitute.dsm.util.MedicalRecordUtil;
import org.broadinstitute.dsm.util.MercuryOrderTestUtil;
import org.broadinstitute.dsm.util.SampleIdTestUtil;
Expand Down Expand Up @@ -64,7 +64,7 @@ public class ClinicalOrderDaoTest extends DbTxnBaseTest {

private static int oncHistoryDetailId = -1;

public static TestKitUtil testKitUtil;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's basically the only change in this class, and below is only the name changed for pepperKitUtil

public static PepperKitUtil pepperKitUtil;

private static int ddpInstanceId = -1;

Expand All @@ -81,13 +81,13 @@ public static void setup() {
studyInstanceName = "ClinOrdTest" + nameAppend;
String studyGroup = "ClinOrdTest" + nameAppend;

testKitUtil = new TestKitUtil(studyInstanceName, studyInstanceName, "CinOrdTest", studyGroup,
"SALIVA", null, null);
testKitUtil.setupInstanceAndSettings();
userId = testKitUtil.adminUtil.createTestUser(generateUserEmail(), Arrays.asList(KIT_SHIPPING, PT_LIST_VIEW));
pepperKitUtil = new PepperKitUtil(studyInstanceName, studyInstanceName, "CinOrdTest", studyGroup,
"SALIVA", "SALIVA", null, null);
pepperKitUtil.setupInstanceAndSettings();
userId = pepperKitUtil.getAdminUtil().createTestUser(generateUserEmail(), Arrays.asList(KIT_SHIPPING, PT_LIST_VIEW));

ParticipantDto participantDto =
new ParticipantDto.Builder(testKitUtil.ddpInstanceId, System.currentTimeMillis())
new ParticipantDto.Builder(pepperKitUtil.getDdpInstanceId(), System.currentTimeMillis())
.withDdpParticipantId(ddpParticipantId)
.withLastVersion(0L)
.withLastVersionDate("")
Expand Down Expand Up @@ -118,7 +118,7 @@ public static void setup() {
});
oncHistoryDetailId = OncHistoryDetail.createOncHistoryDetail(medicalRecordId, "tester");
log.info("Created onc history detail {}", oncHistoryDetailId);
ddpInstanceId = testKitUtil.adminUtil.getDdpInstanceId();
ddpInstanceId = pepperKitUtil.getDdpInstanceId();
}

@Test
Expand Down Expand Up @@ -153,7 +153,7 @@ public void testGetClinicalOrdersForTissuesWhenATissueHasOrders() {
String order1Barcode = "testtestBarcode";

MercuryOrderDto orderDto = mercuryOrderTestUtil.createMercuryOrder(ddpParticipantId, order1Barcode,
testKitUtil.kitTypeId, ddpInstanceId, tissueId);
pepperKitUtil.getKitTypeId(), ddpInstanceId, tissueId);

Map<Integer, Collection<ClinicalOrderDto>> ordersByTissue =
clinicalOrderDao.getClinicalOrdersForTissueIds(tissueIds);
Expand All @@ -176,16 +176,16 @@ public void testGetClinicalOrdersForTissuesWhenSomeTissuesHaveOrdersAndSomeDoNot

String order1Barcode = "TestBarcode1";

ddpInstanceId = testKitUtil.adminUtil.getDdpInstanceId();
ddpInstanceId = pepperKitUtil.getAdminUtil().getDdpInstanceId();
MercuryOrderDto orderDto = mercuryOrderTestUtil.createMercuryOrder(ddpParticipantId, order1Barcode,
testKitUtil.kitTypeId, ddpInstanceId, tissueWithOrder);
pepperKitUtil.getKitTypeId(), ddpInstanceId, tissueWithOrder);

log.info("Created order {} for tissue {} and sample {}", orderDto.getMercurySequencingId(), tissueWithOrder,
sampleWithOrder);

String order2Barcode = "TestBarcode2";
MercuryOrderDto order2Dto = mercuryOrderTestUtil.createMercuryOrder(ddpParticipantId, order2Barcode,
testKitUtil.kitTypeId, ddpInstanceId, tissueWithOrder);
pepperKitUtil.getKitTypeId(), ddpInstanceId, tissueWithOrder);
log.info("Created order {} for tissue {} and sample {}", order2Dto.getMercurySequencingId(), tissueWithOrder,
sampleWithOrder);

Expand Down Expand Up @@ -233,6 +233,6 @@ public static void deleteTestingData() {
log.info("Deleted institution {}", institutionDto.getInstitutionId());
participantDao.delete(participantId);
log.info("Deleted participant {}", participantId);
testKitUtil.deleteGeneratedData();
pepperKitUtil.deleteGeneratedData();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
import org.broadinstitute.dsm.model.nonpepperkit.NonPepperStatusKitService;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the changes here are only name changes from JuniperTestSetupUtil to JuniperKitUtil

import org.broadinstitute.dsm.statics.DBConstants;
import org.broadinstitute.dsm.util.EasyPostUtil;
import org.broadinstitute.dsm.util.NotificationUtil;
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.Before;
Expand Down Expand Up @@ -70,22 +71,22 @@ public class JuniperKitCreationStatusTest extends DbTxnBaseTest {
Tracker mockShipmentTracker = mock(Tracker.class);
NonPepperKitCreationService nonPepperKitCreationService = new NonPepperKitCreationService();
NonPepperStatusKitService nonPepperStatusKitService = new NonPepperStatusKitService();
private static JuniperTestSetupUtil juniperTestSetupUtil;
private static JuniperKitUtil juniperKitUtil;

@BeforeClass
public static void setupJuniperBefore() {

juniperTestSetupUtil =
new JuniperTestSetupUtil(instanceName, instanceGuid, "Juniper-Test", "JuniperTestProject", "Juniper-Group");
juniperTestSetupUtil.setupJuniperInstanceAndSettings();
juniperKitUtil =
new JuniperKitUtil(instanceName, instanceGuid, "Juniper-Test", "JuniperTestProject", "Juniper-Group");
juniperKitUtil.setupJuniperInstanceAndSettings();
ddpInstance = DDPInstance.getDDPInstanceWithRoleByStudyGuid(instanceGuid, DBConstants.JUNIPER_STUDY_INSTANCE_ROLE);

}

@AfterClass
public static void deleteJuniperInstance() {
juniperTestSetupUtil.deleteKitsArray(createdKitIds);
juniperTestSetupUtil.deleteInstanceAndSettings();
juniperKitUtil.deleteKitsArray(createdKitIds);
juniperKitUtil.deleteInstanceAndSettings();
}

@Before
Expand All @@ -101,16 +102,18 @@ public void testCurrentStatusField() {
createNonPepperTestKit(juniperTestKit);
KitResponse kitResponse = nonPepperStatusKitService.getKitsBasedOnJuniperKitId(juniperTestKit.getJuniperKitId());
verifyStatusKitResponse(kitResponse, juniperTestKit, rand, KitCurrentStatus.KIT_WITHOUT_LABEL.getValue());
juniperTestSetupUtil.changeKitToQueue(juniperTestKit, mockEasyPostUtil);
juniperKitUtil.changeKitToQueue(juniperTestKit.getJuniperParticipantID(), mockEasyPostUtil);
kitResponse = nonPepperStatusKitService.getKitsBasedOnJuniperKitId(juniperTestKit.getJuniperKitId());
verifyStatusKitResponse(kitResponse, juniperTestKit, rand, KitCurrentStatus.QUEUE.getValue());
juniperTestKit.setDdpLabel(kitResponse.getKits().get(0).getDsmShippingLabel());
List<ScanResult> scanResultList = juniperTestSetupUtil.changeKitToSent(juniperTestKit);
List<ScanResult> scanResultList = juniperKitUtil.changeKitToSent(juniperTestKit.getDdpLabel(),
"SOME_RANDOM_KIT_LABEL");
Assert.assertFalse(
scanResultList.stream().filter(scanError -> scanError.hasError()).findAny().isPresent());
kitResponse = nonPepperStatusKitService.getKitsBasedOnJuniperKitId(juniperTestKit.getJuniperKitId());
verifyStatusKitResponse(kitResponse, juniperTestKit, rand, KitCurrentStatus.SENT.getValue());
JuniperTestSetupUtil.changeKitToReceived();
NotificationUtil notificationUtil = mock(NotificationUtil.class);
juniperKitUtil.changeKitToReceived(notificationUtil, "SOME_RANDOM_KIT_LABEL");
kitResponse = nonPepperStatusKitService.getKitsBasedOnJuniperKitId(juniperTestKit.getJuniperKitId());
verifyStatusKitResponse(kitResponse, juniperTestKit, rand, KitCurrentStatus.RECEIVED.getValue());

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
package org.broadinstitute.dsm.juniperkits;

import static org.broadinstitute.ddp.db.TransactionWrapper.inTransaction;
import static org.broadinstitute.dsm.service.admin.UserAdminService.PEPPER_ADMIN_ROLE;
import static org.broadinstitute.dsm.statics.DBConstants.KIT_SHIPPING;

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.Arrays;
import java.util.Collections;

import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.broadinstitute.dsm.kits.BaseKitTestUtil;
import org.broadinstitute.lddp.db.SimpleResult;

/**
* This class has methods to set up a study in DSM database that has the needed configuration to send out kits
* It creates the instance, group, instance_role, kit_type, carrier,
* kit_dimensions, kit_return and ddp_kit_request_settings for the
* newly created study.
* It also contains methods to delete what was set up after tests are complete
* </p>
* <p>
* The usage is by first creating an instance of this class by declaring the desired instance name and study guid,
* display name and collaborator prefix.
* Then call setupJuniperInstanceAndSettings() for initiating all the config in database.
*When done, call deleteJuniperInstanceAndSettings() to delete everything
* </p>
*/
@Slf4j
@Getter
public class JuniperKitUtil extends BaseKitTestUtil {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class has the config setup and what else that is only related to Juniper kits

private Integer instanceRoleId;
private Integer ddpInstanceRoleId;
private String displayName;
private String collaboratorPrefix;
private static String esIndex = null;

public JuniperKitUtil(String instanceName, String studyGuid, String displayName, String collaboratorPrefix, String groupName) {
super(instanceName, studyGuid, groupName);
this.displayName = displayName;
this.collaboratorPrefix = collaboratorPrefix;
}

public void setupJuniperInstanceAndSettings() {
//everything should get inserted in one transaction
SimpleResult results = inTransaction((conn) -> {
SimpleResult simpleResult = new SimpleResult();
try {
adminUtil.createRealmAndStudyGroup(instanceName, studyGuid, collaboratorPrefix, groupName, esIndex);
ddpInstanceId = adminUtil.getDdpInstanceId();
ddpGroupId = adminUtil.getStudyGroupId();
instanceRoleId = getJuniperStudyInstanceRole(conn);
ddpInstanceRoleId = createDdpInstanceRole(conn, instanceRoleId);
kitTypeId = getKitTypeId(conn);
kitDimensionId = createKitDimension(conn);
kitReturnId = createKitReturnInformation(conn);
carrierId = createCarrierInformation(conn);
ddpKitRequestSettingsId = createKitRequestSettingsInformation(conn);
adminUtil.setStudyAdminAndRoles(generateUserEmail(), PEPPER_ADMIN_ROLE,
Arrays.asList(KIT_SHIPPING));

userWithKitShippingAccess = Integer.toString(adminUtil.createTestUser(generateUserEmail(),
Collections.singletonList(KIT_SHIPPING)));
} catch (SQLException e) {
simpleResult.resultException = e;
}
return simpleResult;
});
if (results.resultException != null) {
log.error("Error creating data ", results.resultException);
deleteInstanceAndSettings();
}
}

public void deleteInstanceAndSettings() {
SimpleResult results = inTransaction((conn) -> {
SimpleResult dbVals = new SimpleResult();
try {
delete(conn, "kit_dimension", "kit_dimension_id", kitDimensionId);
delete(conn, "kit_return_information", "kit_return_id", kitReturnId);
delete(conn, "carrier_service", "carrier_service_id", carrierId);
delete(conn, "ddp_kit_request_settings", "ddp_kit_request_settings_id", ddpKitRequestSettingsId);
delete(conn, "ddp_instance_role", "ddp_instance_role_id", ddpInstanceRoleId);
delete(conn, "ddp_instance_group", "instance_group_id", ddpInstanceGroupId);
delete(conn, "ddp_instance", "ddp_instance_id", ddpInstanceId);
delete(conn, "ddp_group", "group_id", ddpGroupId);
adminUtil.deleteGeneratedData();
} catch (Exception e) {
dbVals.resultException = e;
}
return dbVals;
});

}

private String generateUserEmail() {
return "JuniperTest-" + System.currentTimeMillis() + "@broad.dev";
}

private Integer getJuniperStudyInstanceRole(Connection conn) throws SQLException {
if (instanceRoleId != null) {
return instanceRoleId;
}
PreparedStatement stmt = conn.prepareStatement(SELECT_INSTANCE_ROLE);
stmt.setString(1, "juniper_study");
ResultSet rs = stmt.executeQuery();
return getPrimaryKey(rs, "instance_role");
}




}
Loading