-
Notifications
You must be signed in to change notification settings - Fork 132
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Application doesn't exit. #17
Comments
Can you share some code example to reproduce the issue ? |
|
@vanthome will have to jump in. I haven't touched this project in a very long time. Thomas will know better. He is the one taking care of it nowadays. I suspect either the checkEsConnection function or the elasticsearch client |
I am also having this problem. Setting keepAlive to false doesn't work. However, as @jackdbernier mentioned, something like |
ok, do you have any suggestion when to close it? |
@eshao did you really try this solution ? |
@eshao actually, it works when I close the client directly, but there is a latency of 1.5 seconds at the end of the script. |
@christiansaiki can you explain the actual issue, I cannot completely follow @RTSGIT words. |
@vanthome, yes! |
OK and what exactly means existing the application - |
any progress on this? what @christiansaiki means is that if you run this in a non-server application, the ES client is keeping the connection open and thus the application never exits. |
ok, in this case, my approach would be to get the ES client via |
Would love to see a close() function. |
Tragically this makes this really unusable in production... I tried following the Waiting for Winston logs to finish, but that flow doesn't seem to work. The |
Are you sure, have you tried with the latest Winston 3.0 based release? |
Ah, the winston docs use
As is, the event never triggers, so client doesn't close, event gets logged, app keeps running. If I use the correct event name, client does close, but event never gets logged. Am I missing something? |
Does anyone have an example where the app exits?? |
Since I can't figure this out, I've switched to using the |
What means "client does close, but event never gets logged."? So if the client gets closed it should exit... |
Right it should, but it doesn't (at least for me). Does anyone have any actual code example where you can log AND have the application exit? |
can you post the output of the logger? I want to see what you makes think that the client "does close". |
good, then in that event handler, can you try this (what I have proposed earlier in this thread)?
or you provide your own ES client, hold a handle to it and do the same. |
I believe that's what my code above does; I create my own ES Client, use it to initialize the transport. And I close that client when I get the Or are you proposing something different? |
Yes, sorry, you are right, if you have changed the event name, it's exactly what I propose. And you are sure that there is nothing else going on in your application, and what happens if you add a |
Yea, that seems like a rather unfriendly approach. I mean that can't be the right approach, this just seems like a bug. Seriously is that how folks are running with this? Only in applications which don't actually end on their own? |
I also notice this issue and can't apply any of the suggested solutions. |
This works:
|
It seems the timer of the BulkWriter needs to be stopped, so the program can end. Before doing that the BulkWriter should be flushed, so all messages are sent to ES. To end the program, this works for me: // Flush the winston stream
winston.end();
await new Promise((resolve, reject) => {
winston.on('finish', resolve);
});
// Flush the winston-elasticsearch bulk-writer
while (transport.bulkWriter.bulk.length > 0) {
await transport.bulkWriter.flush();
}
transport.bulkWriter.stop();
transport.client.close(); |
I think the solution proposed by @chunfengd is better as it flushes the bulkWriter gracefully and the |
Nice - works fine! I can confirm that with the new version 0.7.6 when calling |
I'm still running into this problem with winston 3.1.0 and winston-elasticsearch 0.7.7 -- things almost work but I think the logger is terminating before the writer has a chance to finish writing everything. Typically, if I'm sending less than 15 or so log entries, everything is ok, when I try and send more I get errors.
The full stack of the error (which happens many times):
suggestions? |
A few changes to the above script:
output:
|
Here is a working version of winston3 writing to console and elasticsearch. The only safe way I was able to get working is to poll transport.bulkWriter.bulk.length and wait for it to become zero, then call logger.end(). NOTE This doesn't work, see comments below. const winston = require('winston'); // 3.1.0
const winston_es = require('winston-elasticsearch'); // 0.7.7
const util = require('./lib/util.js');
async function main() {
const transportOps = {
'clientOpts': {
'host': 'http://localhost:9200'
}
};
let transport = new winston_es(transportOps);
const logger = winston.createLogger({
transports: [
new winston.transports.Console({
format: winston.format.simple()
}),
transport
]
});
logger.on('finish', () => {
console.log("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX logger finished");
});
// synchronous work - must have all calls to
// logger done before we start polling bulkWriter
for (let i = 0; i < 50; i++) {
logger.info("writing data " + i);
}
logger.info("sleeping until all writing is done...");
// after here it is no longer safe to call logger.info()
while (transport.bulkWriter.bulk.length > 0) {
console.log('messages remaining: ' + transport.bulkWriter.bulk.length);
await util.sleep(1000);
}
logger.end(() => {
console.log('logger end done.');
});
}
main(); |
Here's a version using winston3 and the default logger. Since there is no NOTE This doesn't work, see comment below... const winston = require('winston'); // 3.1.0
const winston_es = require('winston-elasticsearch'); // 0.7.7
const util = require('./lib/util.js');
// try and use the default logger to log to elasticsearch.
var transport;
async function setupLogging() {
console.log("setup logging");
return new Promise(async (resolve, reject) => {
const transportOps = {
'clientOpts': {
'host': 'http://localhost:9200'
}
};
transport = new winston_es(transportOps);
winston.configure({
transports: [
new winston.transports.Console({format: winston.format.simple()}),
transport
]
});
transport.on('finish', () => { console.log("transport finished"); });
transport.on('error', e => { console.log('error:' + e); });
resolve();
});
}
async function doWork() {
console.log("do work");
return new Promise(async (resolve, reject) => {
for (let i = 0; i < 50; i++) {
winston.info("writing data " + i);
}
resolve();
});
}
async function shutdownLogging() {
console.log("shutdown logging");
return new Promise(async (resolve, reject) => {
while (transport.bulkWriter.bulk.length > 0) {
console.log('messages remaining: ' + transport.bulkWriter.bulk.length);
await util.sleep(1000); // sleep for a little bit to flush the bulk out
}
// winston default logging doesn't have a "logger.end()"
// so we try to use transport.end() instead.
// this works maybe 50% of the time - otherwise we get cascading
// "TypeError: cb is not a function" errors.
// transport.end();
resolve();
});
}
async function main() {
await setupLogging();
await doWork();
// after here it is no longer safe to call winston.log()
await shutdownLogging();
// since we can't stop the transport,
// we have to forcibly terminate our application.
process.exit(123);
}
main(); |
Ok, bad news: the above two samples do not work reliably. There is time between calling If you're calling process.exit you will not see the log messages. If you're calling transport.end then you'll get errors about write-after-end. Back to the drawing board ... I think the core problem is that I need to wait until all log entries have been transmitted through elasticsearch before calling logger.end() and I don't have a reliable way to know when that is. |
why not implement a close funtion like the mongo transport: https://github.com/winstonjs/winston-mongodb/blob/master/lib/winston-mongodb.js#L163 const Elasticsearch = require('winston-elasticsearch');
Elasticsearch.prototype.close = async function () {
await this.bulkWriter.flush();
this.bulkWriter.stop();
};
logger.close(); seems to do the trick for me |
Because we assume that the underlying ES client is what needs to be closed. @trasa can you try this and see whether it fixes the problem for you? |
Changed my example script around to use .close() and still get problems: https://gist.github.com/trasa/f4c93846d077a740387a630f01a1d4e8 The result is that we're still ending the transport before all the work has been done.
|
What I ended up doing is stripping the winston transport down to a minimum: no queues, just sync writes to ES. For my particular console application this is acceptable. There's lots of cases where it wouldn't be. https://gist.github.com/trasa/1efaee47267db3de079024ec99a1c823 |
I'm having a similar issue, I've added ElasticSearch.prototype.end = async function() {
await this.bulkWriter.flush();
this.bulkWriter.stop();
}; Which gets called when This does end my process, but the two issues I'm having is that I noticed that This to me signals that The only thing I can think to do is somehow, after end, when append is called, just immediatly flush to keep the event loop going. |
Edit This seems to work for continuously flushing until done ElasticSearch.prototype.end = async function() {
const bulkWriter = this.bulkWriter;
bulkWriter.stop();
await doWhilst(() => bulkWriter.flush(), () => bulkWriter.bulk.length > 0);
this.emit('finish');
}; |
@barlock Did you manage to find a newer solution? I used yours in this manner below, but the program is still not exiting.
|
Is this still an issue? |
I'm still having the issue. Although I see this MR #141 I'm using: Aditional info:
Conclusion:, I can confirm that the issue seems still to be happening on the elasticsearch transport from this library. |
ok, can you try to do the follwoing (maybe you even already know): what if you set https://github.com/vanthome/winston-elasticsearch/blob/master/index.js#L107 Maybe you can even try this? |
In a basic index.js code, I addded winston elasticsearch pointing to a server. Winston-elasticsearch trace shows successful connection, but after logging, the application doesn't exit.
Any suggestions?
The text was updated successfully, but these errors were encountered: