Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: update test memory leaks #2394

Merged
merged 18 commits into from
Feb 27, 2024
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,6 @@
"jest": "^29.7.0",
"npm-run-all2": "^5.0.2",
"prettier": "^3.2.4",
"ts-jest": "^29.1.2",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we use @swc/jest instead, not needed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BlairCurrey was about to merge until I saw this, fixed now 👍

"tunnelmole": "^2.2.13",
"typescript": "^5.3.3",
"uuid": "^9.0.1"
Expand Down
3 changes: 3 additions & 0 deletions packages/backend/jest.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ const baseConfig = require('../../jest.config.base.js')
// eslint-disable-next-line @typescript-eslint/no-var-requires
const packageName = require('./package.json').name

process.env.LOG_LEVEL = 'silent'

module.exports = {
...baseConfig,
clearMocks: true,
Expand All @@ -12,6 +14,7 @@ module.exports = {
globalSetup: `<rootDir>/packages/${packageName}/jest.setup.ts`,
globalTeardown: `<rootDir>/packages/${packageName}/jest.teardown.js`,
testRegex: `(packages/${packageName}/.*/__tests__/.*|\\.(test|spec))\\.tsx?$`,
testEnvironment: `<rootDir>/packages/${packageName}/jest.custom-environment.ts`,
moduleDirectories: [
`node_modules`,
`packages/${packageName}/node_modules`,
Expand Down
9 changes: 9 additions & 0 deletions packages/backend/jest.custom-environment.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
import { TestEnvironment } from 'jest-environment-node'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main change. Described here: nock/nock#2358

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess these nock changes are needed even after this fix? nock/nock#2572 It looks like we are on the latest and have this fix. I suppose there were two issues:

  • nock leaking from simply being imported (fixed above)
  • us not cleaning it up properly which still require these changes

import nock from 'nock'

export default class CustomEnvironment extends TestEnvironment {
constructor(config, context) {
super(config, context)
this.global.nock = nock
}
}
2 changes: 0 additions & 2 deletions packages/backend/jest.setup.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
require('ts-node/register')

import { knex } from 'knex'
import { GenericContainer, Wait } from 'testcontainers'

Expand Down
21 changes: 21 additions & 0 deletions packages/backend/jest.tigerbeetle-environment.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
import { startTigerbeetleContainer } from './src/tests/tigerbeetle'
import { StartedTestContainer } from 'testcontainers'

import CustomTestEnvironment from './jest.custom-environment'

export default class TigerbeetleEnvironment extends CustomTestEnvironment {
private tbContainer: StartedTestContainer | undefined

public async setup(): Promise<void> {
await super.setup()
const tbContainer = await startTigerbeetleContainer()

this.tbContainer = tbContainer.container
this.global.tigerbeetlePort = tbContainer.port
}

public async teardown(): Promise<void> {
await super.teardown()
await this.tbContainer?.stop()
}
}
5 changes: 2 additions & 3 deletions packages/backend/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -31,16 +31,15 @@
"@types/uuid": "^9.0.8",
"cross-fetch": "^4.0.0",
"ilp-protocol-stream": "^2.7.2-alpha.2",
"jest-environment-node": "^29.7.0",
"jest-openapi": "^0.14.2",
"nock": "^13.5.1",
"node-mocks-http": "^1.14.1",
"openapi-types": "^12.1.3",
"pino-pretty": "^10.3.1",
"react": "~18.2.0",
"rosie": "^2.1.1",
"testcontainers": "^10.6.0",
"tmp": "^0.2.1",
"ts-node": "^10.9.2"
"tmp": "^0.2.1"
},
"dependencies": {
"@adonisjs/fold": "^8.2.0",
Expand Down
24 changes: 15 additions & 9 deletions packages/backend/src/accounting/tigerbeetle/service.test.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
/**
* @jest-environment ./packages/backend/jest.tigerbeetle-environment.ts
*/
Comment on lines +1 to +3
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

specifies specific environment for this test suite


import assert from 'assert'
import { CreateAccountError as CreateTbAccountError } from 'tigerbeetle-node'
import { v4 as uuid } from 'uuid'
Expand All @@ -9,7 +13,6 @@ import { IocContract } from '@adonisjs/fold'
import { initIocContainer } from '../../'
import { AppServices } from '../../app'
import { truncateTables } from '../../tests/tableManager'
import { startTigerbeetleContainer } from '../../tests/tigerbeetle'
import { AccountFactory, FactoryAccount } from '../../tests/accountFactory'
import { isTransferError, TransferError } from '../errors'
import {
Expand All @@ -20,7 +23,7 @@ import {
Withdrawal
} from '../service'

describe('Accounting Service', (): void => {
describe('Tigerbeetle Accounting Service', (): void => {
let deps: IocContract<AppServices>
let appContainer: TestContainer
let accountingService: AccountingService
Expand All @@ -33,10 +36,14 @@ describe('Accounting Service', (): void => {
}

beforeAll(async (): Promise<void> => {
const { port } = await startTigerbeetleContainer()
Config.tigerbeetleReplicaAddresses = [port.toString()]
const tigerbeetlePort = (global as unknown as { tigerbeetlePort: number })
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gets injected via the tigerbeetle test environment

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like this suggestion applies here as well: https://github.com/interledger/rafiki/pull/2394/files#r1480529987

.tigerbeetlePort

deps = await initIocContainer({ ...Config, useTigerbeetle: true })
deps = initIocContainer({
...Config,
tigerbeetleReplicaAddresses: [tigerbeetlePort.toString()],
useTigerbeetle: true
})
appContainer = await createTestApp(deps)
accountingService = await deps.use('accountingService')
accountFactory = new AccountFactory(accountingService, newLedger)
Expand All @@ -48,7 +55,6 @@ describe('Accounting Service', (): void => {

afterAll(async (): Promise<void> => {
await appContainer.shutdown()
// TODO: find a way to gracefully stop TB container without running into a thread panic
})

describe('Create Liquidity Account', (): void => {
Expand Down Expand Up @@ -83,11 +89,11 @@ describe('Accounting Service', (): void => {
},
LiquidityAccountType.ASSET
)
).rejects.toThrowError('unable to create account, invalid id')
).rejects.toThrow('unable to create account, invalid id')
})

test('Create throws on error', async (): Promise<void> => {
const tigerbeetle = await deps.use('tigerbeetle')
const tigerbeetle = await deps.use('tigerbeetle')!
jest.spyOn(tigerbeetle, 'createAccounts').mockResolvedValueOnce([
{
index: 0,
Expand All @@ -106,7 +112,7 @@ describe('Accounting Service', (): void => {
},
LiquidityAccountType.ASSET
)
).rejects.toThrowError(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using toThrowError gave me a deprecated warning

).rejects.toThrow(
new TigerbeetleCreateAccountError(
CreateTbAccountError.exists_with_different_ledger
)
Expand Down
6 changes: 3 additions & 3 deletions packages/backend/src/app.ts
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ export interface AppServices {
autoPeeringService: Promise<AutoPeeringService>
autoPeeringRoutes: Promise<AutoPeeringRoutes>
connectorApp: Promise<ConnectorApp>
tigerbeetle: Promise<TigerbeetleClient>
tigerbeetle?: Promise<TigerbeetleClient>
paymentMethodHandlerService: Promise<PaymentMethodHandlerService>
ilpPaymentService: Promise<IlpPaymentService>
}
Expand Down Expand Up @@ -611,8 +611,8 @@ export class App {
if (this.openPaymentsServer) {
await this.stopServer(this.openPaymentsServer)
}
if (this.adminServer) {
await this.stopServer(this.adminServer)
if (this.apolloServer) {
await this.apolloServer.stop()
}
Comment on lines +618 to 620
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we do plugins: [ApolloServerPluginDrainHttpServer({ httpServer })] when creating the apollo server, the httpServer will be automatically stopped/connections will be drained. However, before we weren't closing the actual apollo server that does other operations as well: https://www.apollographql.com/docs/apollo-server/api/apollo-server/#stop

if (this.ilpConnectorService) {
await this.stopServer(this.ilpConnectorService)
Expand Down
3 changes: 2 additions & 1 deletion packages/backend/src/graphql/resolvers/auto-peering.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import { faker } from '@faker-js/faker'
import { gql } from '@apollo/client'
import assert from 'assert'
import nock from 'nock'

import { createTestApp, TestContainer } from '../../tests/app'
import { IocContract } from '@adonisjs/fold'
Expand All @@ -20,6 +19,8 @@ import { CreateOrUpdatePeerByUrlInput } from '../generated/graphql'
import { AutoPeeringService } from '../../payment-method/ilp/auto-peering/service'
import { v4 as uuid } from 'uuid'

const nock = (global as unknown as { nock: typeof import('nock') }).nock
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about this?

add this to a new file at rafiki/packages/backend/src/global.d.ts:

declare module globalThis {
  var nock: typeof import('nock')
}

which lets us just do:

const nock = global.nock

Copy link
Contributor

@BlairCurrey BlairCurrey Feb 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I dont love that this suggests nock is on global anywhere (including outside tests, or outside the that specific test environment) ... was just thinking maybe we could avoid doing all this casting every time we want to use it. alternatively maybe we can initialize it like this once somewhere and export it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not too fond of this long casting, but I think I prefer not mixing the shipped code vs the testing code IMO


describe('Auto Peering Resolvers', (): void => {
let deps: IocContract<AppServices>
let appContainer: TestContainer
Expand Down
37 changes: 22 additions & 15 deletions packages/backend/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,6 @@ BigInt.prototype.toJSON = function () {
return this.toString()
}

const container = initIocContainer(Config)
const app = new App(container)

export function initIocContainer(
config: typeof Config
): IocContract<AppServices> {
Expand Down Expand Up @@ -120,13 +117,7 @@ export function initIocContainer(
serverAddress: config.ilpAddress
})
})
container.singleton('tigerbeetle', async (deps) => {
const config = await deps.use('config')
return createClient({
cluster_id: BigInt(config.tigerbeetleClusterId),
replica_addresses: config.tigerbeetleReplicaAddresses
})
})

container.singleton('openApi', async () => {
const resourceServerSpec = await createOpenAPI(
path.resolve(__dirname, './openapi/resource-server.yaml')
Expand Down Expand Up @@ -185,7 +176,15 @@ export function initIocContainer(
const config = await deps.use('config')

if (config.useTigerbeetle) {
const tigerbeetle = await deps.use('tigerbeetle')
container.singleton('tigerbeetle', async (deps) => {
const config = await deps.use('config')
return createClient({
cluster_id: BigInt(config.tigerbeetleClusterId),
replica_addresses: config.tigerbeetleReplicaAddresses
})
})

const tigerbeetle = await deps.use('tigerbeetle')!

return createTigerbeetleAccountingService({
logger,
Expand Down Expand Up @@ -465,10 +464,15 @@ export const gracefulShutdown = async (
await app.shutdown()
const knex = await container.use('knex')
await knex.destroy()
const tigerbeetle = await container.use('tigerbeetle')
tigerbeetle.destroy()

const config = await container.use('config')
if (config.useTigerbeetle) {
const tigerbeetle = await container.use('tigerbeetle')
tigerbeetle?.destroy()
}

const redis = await container.use('redis')
redis.disconnect()
await redis.quit()
}

export const start = async (
Expand Down Expand Up @@ -556,7 +560,10 @@ export const start = async (
}

// If this script is run directly, start the server
if (!module.parent) {
if (require.main === module) {
const container = initIocContainer(Config)
const app = new App(container)

start(container, app).catch(async (e): Promise<void> => {
const errInfo = e && typeof e === 'object' && e.stack ? e.stack : e
const logger = await container.use('logger')
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,12 @@ describe('Redis Data Store', (): void => {
const dataStore = createRedisDataStore(redis, ttlMs)

afterEach(async () => {
jest.useRealTimers()
await redis.flushall()
})

afterAll(async () => {
redis.disconnect()
jest.useRealTimers()
await redis.quit()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems to be the preferred way to stop redis connections: redis/ioredis#1088

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about resetting the timers?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved it to the afterEach

})

describe('set', (): void => {
Expand Down
2 changes: 1 addition & 1 deletion packages/backend/src/middleware/lock/redis.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ describe('Redis Lock', (): void => {
})

afterAll(async () => {
redis.disconnect()
await redis.quit()
})

describe('acquire', () => {
Expand Down
3 changes: 2 additions & 1 deletion packages/backend/src/open_payments/auth/middleware.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import { generateKeyPairSync } from 'crypto'
import { faker } from '@faker-js/faker'
import nock from 'nock'
import { Client, ActiveTokenInfo } from 'token-introspection'
import { v4 as uuid } from 'uuid'
import {
Expand Down Expand Up @@ -30,6 +29,8 @@ import { setup } from '../wallet_address/model.test'
import { parseLimits } from '../payment/outgoing/limits'
import { AccessAction, AccessType } from '@interledger/open-payments'

const nock = (global as unknown as { nock: typeof import('nock') }).nock

type AppMiddleware = (
ctx: WalletAddressContext,
next: () => Promise<void>
Expand Down
2 changes: 1 addition & 1 deletion packages/backend/src/open_payments/grant/service.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ describe('Grant Service', (): void => {
})

afterEach(async (): Promise<void> => {
await truncateTables(knex)
jest.useRealTimers()
await truncateTables(knex)
})

afterAll(async (): Promise<void> => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,19 +16,14 @@ describe('Wallet Address Key Service', (): void => {
let deps: IocContract<AppServices>
let appContainer: TestContainer
let walletAddressKeyService: WalletAddressKeyService
const mockMessageProducer = {
send: jest.fn()
}

beforeAll(async (): Promise<void> => {
deps = await initIocContainer(Config)
deps.bind('messageProducer', async () => mockMessageProducer)
deps = initIocContainer(Config)
appContainer = await createTestApp(deps)
walletAddressKeyService = await deps.use('walletAddressKeyService')
})

afterEach(async (): Promise<void> => {
jest.useRealTimers()
await truncateTables(appContainer.knex)
})

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -413,8 +413,7 @@ describe('Open Payments Wallet Address Service', (): void => {
let delayProcessAt: Date | null = null

beforeEach((): void => {
jest.useFakeTimers()
jest.setSystemTime(new Date())
jest.useFakeTimers({ now: Date.now() })
if (withdrawalThrottleDelay !== undefined) {
delayProcessAt = new Date(Date.now() + withdrawalThrottleDelay)
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import assert from 'assert'
import { IocContract } from '@adonisjs/fold'
import nock from 'nock'
import { initIocContainer } from '../../..'
import { AppServices } from '../../../app'
import { Config, IAppConfig } from '../../../config/app'
Expand All @@ -19,6 +18,8 @@ import { PeerError } from '../peer/errors'
import { v4 as uuid } from 'uuid'
import { AccountingService } from '../../../accounting/service'

const nock = (global as unknown as { nock: typeof import('nock') }).nock

describe('Auto Peering Service', (): void => {
let deps: IocContract<AppServices>
let appContainer: TestContainer
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import { faker } from '@faker-js/faker'

export const IlpPrepareFactory = Factory.define<IlpPrepare>('IlpPrepare').attrs(
{
amount: faker.finance.amount(1, 100, 0),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this type signature is deprecated (was giving me warnings)

amount: faker.finance.amount({ min: 1, max: 100, dec: 0 }),
data: Buffer.alloc(0),
destination: 'test.rafiki.' + faker.person.firstName(),
expiresAt: new Date(Date.now() + 10 * 1000),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ describe('Stream Controller', function () {

afterAll(async () => {
await services.redis.flushdb()
await services.redis.disconnect()
await services.redis.quit()
})

test('constructs a reply for a receive account', async () => {
Expand Down
Loading
Loading