Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zonemta continues to restart poorly, something to do with mongo? #18

Open
jtagcat opened this issue Mar 19, 2022 · 2 comments
Open

Zonemta continues to restart poorly, something to do with mongo? #18

jtagcat opened this issue Mar 19, 2022 · 2 comments

Comments

@jtagcat
Copy link

jtagcat commented Mar 19, 2022

zonemta_1           | info App  ---------------------------------------------------------------
zonemta_1           | info App
zonemta_1           | info App  ███████╗ ██████╗ ███╗   ██╗███████╗███╗   ███╗████████╗ █████╗
zonemta_1           | info App  ╚══███╔╝██╔═══██╗████╗  ██║██╔════╝████╗ ████║╚══██╔══╝██╔══██╗
zonemta_1           | info App    ███╔╝ ██║   ██║██╔██╗ ██║█████╗  ██╔████╔██║   ██║   ███████║
zonemta_1           | info App   ███╔╝  ██║   ██║██║╚██╗██║██╔══╝  ██║╚██╔╝██║   ██║   ██╔══██║
zonemta_1           | info App  ███████╗╚██████╔╝██║ ╚████║███████╗██║ ╚═╝ ██║   ██║   ██║  ██║
zonemta_1           | info App  ╚══════╝ ╚═════╝ ╚═╝  ╚═══╝╚══════╝╚═╝     ╚═╝   ╚═╝   ╚═╝  ╚═╝
zonemta_1           | info App
zonemta_1           | info App  ------------------------ [email protected] -----------------------
zonemta_1           | info App
zonemta_1           | (node:8) Warning: Accessing non-existent property 'MongoError' of module exports inside circular dependency
zonemta_1           | (Use `node --trace-warnings ...` to show where the warning was created)
zonemta_1           | info SMTP/feeder SMTP feeder MTA server started listening on port 587
zonemta_1           | info QS Queue server started
zonemta_1           | info API API server started
zonemta_1           | info Redis/8 Redis ready to take connections
zonemta_1           | info mongo Setup 10 indexes
zonemta_1           | Top-level use of w, wtimeout, j, and fsync is deprecated. Use writeConcern instead.
zonemta_1           | info Queue Sending queue initialized
zonemta_1           | info Child/default/25 [c9a6879d2242b82aadd0] Spawning sender process for default
zonemta_1           | (node:19) Warning: Accessing non-existent property 'MongoError' of module exports inside circular dependency
zonemta_1           | (Use `node --trace-warnings ...` to show where the warning was created)
zonemta_1           | info Redis/19 Redis ready to take connections
...
zonemta_1           | info Redis/63 Redis ready to take connections
zonemta_1           | ERR! Queue/52 Could not initialize database: connection timed out
zonemta_1           | ERR! Child/default/52 [ccd344a2913955a5ae88] Sender process for default exited with 1
zonemta_1           | info Child/default/74 [0cc769d3c031272f47bb] Spawning sender process for default
zonemta_1           | info Child/default/81 [2a31870bab6ec98e54b9] Spawning sender process for default
zonemta_1           | info DNS/74 Loaded DNS cache
zonemta_1           | info Sender/default/74 [0cc769d3c031272f47bb] Starting sending for default
zonemta_1           | info ConnPool/74 Connection pool is ready to take connections
zonemta_1           | (node:74) Warning: Accessing non-existent property 'MongoError' of module exports inside circular dependency
zonemta_1           | (Use `node --trace-warnings ...` to show where the warning was created)
zonemta_1           | info Redis/74 Redis ready to take connections
...
zonemta_1           | ERR! Queue/63 Could not initialize database: connect ECONNREFUSED 172.25.0.3:27017
...
zonemta_1           | ERR! Queue/63 Could not initialize database: connect ECONNREFUSED 172.25.0.3:27017
...

zonemta container bursts on the CPU every time it retries, and memory allocated to the container grows by 100-300M every restart (indefinitely, stopped it at ~20G)

Using default generated config, with env HOSTNAMES=sub.domain.tld (removed [], as =[sub.domain.tld] didn't work.

Mongo seems to be getting connections:

...
mongo_1             | {"t":{"$date":"2022-03-19T10:01:09.713+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"/tmp/mongodb-27017.sock"}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:09.713+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"0.0.0.0"}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:09.713+00:00"},"s":"I",  "c":"NETWORK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:10.835+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.26.0.6:60462","uuid":"593ee7e3-9177-434f-8584-91d15
62cb54f","connectionId":1,"connectionCount":1}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:10.842+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn1","msg":"client metadata","attr":{"remote":"172.26.0.6:60462","client":"conn1","doc":{"driver":{"name":"no
dejs","version":"3.6.4"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"5.4.0-104-generic"},"platform":"'Node.js v14.16.0, LE (unified)"}}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:10.855+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.26.0.6:60464","uuid":"7da01757-e299-446b-883b-b01da
dbf6c84","connectionId":2,"connectionCount":2}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:10.856+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn2","msg":"client metadata","attr":{"remote":"172.26.0.6:60464","client":"conn2","doc":{"driver":{"name":"no
dejs","version":"3.6.4"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"5.4.0-104-generic"},"platform":"'Node.js v14.16.0, LE (unified)"}}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:11.720+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.26.0.6:60470","uuid":"94679ad4-8ed4-4734-9688-efe19
067c39e","connectionId":3,"connectionCount":3}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:11.727+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn3","msg":"client metadata","attr":{"remote":"172.26.0.6:60470","client":"conn3","doc":{"driver":{"name":"no
dejs","version":"3.6.4"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"5.4.0-104-generic"},"platform":"'Node.js v14.16.0, LE (unified)"}}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:11.909+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.26.0.7:35562","uuid":"10f78d0b-62bc-4f12-b05a-6f2f1
f813dfe","connectionId":4,"connectionCount":4}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:11.918+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn4","msg":"client metadata","attr":{"remote":"172.26.0.7:35562","client":"conn4","doc":{"driver":{"name":"no
dejs","version":"3.6.9"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"5.4.0-104-generic"},"platform":"'Node.js v14.17.1, LE (unified)"}}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:11.954+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.26.0.7:35570","uuid":"f3410b79-fc6a-4e58-9670-3af9c
f4939ce","connectionId":5,"connectionCount":5}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:11.954+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn5","msg":"client metadata","attr":{"remote":"172.26.0.7:35570","client":"conn5","doc":{"driver":{"name":"no
dejs","version":"3.6.9"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"5.4.0-104-generic"},"platform":"'Node.js v14.17.1, LE (unified)"}}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:12.440+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.26.0.8:49584","uuid":"5bba1fc0-46e1-45ec-9776-fc415
802ab06","connectionId":6,"connectionCount":6}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:12.459+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn6","msg":"client metadata","attr":{"remote":"172.26.0.8:49584","client":"conn6","doc":{"driver":{"name":"no
dejs","version":"3.6.3"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"5.4.0-104-generic"},"platform":"'Node.js v14.16.0, LE (unified)"}}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:14.074+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.26.0.8:49592","uuid":"3af84bb3-c469-4bd3-85e4-ae08f
8a4e7d2","connectionId":7,"connectionCount":7}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:14.099+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn7","msg":"client metadata","attr":{"remote":"172.26.0.8:49592","client":"conn7","doc":{"driver":{"name":"no
dejs","version":"3.6.3"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"5.4.0-104-generic"},"platform":"'Node.js v14.16.0, LE (unified)"}}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:21.364+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.26.0.6:60530","uuid":"6318701d-700c-450b-acbb-4aaff
c6478b4","connectionId":8,"connectionCount":8}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:21.365+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn8","msg":"client metadata","attr":{"remote":"172.26.0.6:60530","client":"conn8","doc":{"driver":{"name":"no
dejs","version":"3.6.4"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"5.4.0-104-generic"},"platform":"'Node.js v14.16.0, LE (unified)"}}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:22.445+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.26.0.7:35622","uuid":"d3935efe-3ee0-430a-854e-891bd
6cce7ca","connectionId":9,"connectionCount":9}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:22.448+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn9","msg":"client metadata","attr":{"remote":"172.26.0.7:35622","client":"conn9","doc":{"driver":{"name":"no
dejs","version":"3.6.9"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"5.4.0-104-generic"},"platform":"'Node.js v14.17.1, LE (unified)"}}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:22.556+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.26.0.6:60536","uuid":"4afdc1b8-4545-4274-bfdd-091ae
81863ec","connectionId":10,"connectionCount":10}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:22.561+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn10","msg":"client metadata","attr":{"remote":"172.26.0.6:60536","client":"conn10","doc":{"driver":{"name":"
nodejs","version":"3.6.4"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"5.4.0-104-generic"},"platform":"'Node.js v14.16.0, LE (unified)"}}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:23.020+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.26.0.8:49606","uuid":"d67c61d5-6163-434e-9a1a-0c8f5
a298888","connectionId":11,"connectionCount":11}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:23.026+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn11","msg":"client metadata","attr":{"remote":"172.26.0.8:49606","client":"conn11","doc":{"driver":{"name":"
nodejs","version":"3.6.3"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"5.4.0-104-generic"},"platform":"'Node.js v14.16.0, LE (unified)"}}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:24.643+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.26.0.8:49608","uuid":"1296ef44-aa49-4485-a925-fe4e0
21e23cf","connectionId":12,"connectionCount":12}}
mongo_1             | {"t":{"$date":"2022-03-19T10:01:24.646+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn12","msg":"client metadata","attr":{"remote":"172.26.0.8:49608","client":"conn12","doc":{"driver":{"name":"
nodejs","version":"3.6.3"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"5.4.0-104-generic"},"platform":"'Node.js v14.16.0, LE (unified)"}}}
...

On bringing the containers down, mongo closed ~300 open connections.

@WeebNetsu
Copy link

I also have this issue, any information on it would be appreciated

@marius-lazar
Copy link

marius-lazar commented Aug 8, 2023

turning off mongo net compression helps with the load if it's a big-ish environment
config file (github doesn't let me align it properly):
net: port: 27017 bindIp: localhost compression: compressors: disabled

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants