Ability to Run Logstash with Centralized Pipeline Management

[morgango]

I would like to be able to use stack-docker to run Logstash with Centralized Pipeline Management.

To do this we would need to either make the logstash.yml file publicly available for editing OR automatically configure the logstash instance to use centralized pipeline management and specify the name of the pipelines that could be run, something like "script-1", "script-2" in the xpack.management.pipeline.id parameter (see configuration for more details).

I think the best approach might be to have a separate Logstash instance is correctly configured with a username, password, and predefined script names. However, this instance might be commented out because it might cause confusion to have two different Logstash instances running.

I would really like to have this feature for demonstrations.

[fxdgear]

@morgango Thanks for your issue.

and I 100% completely agree with you on this.

If you take a look at this PR here: #39

You can see what' i've done is pulled the config for each service out into it's own yml file which is now part of the repo and configurable via the user (who ever clones this repo).

The next step after getting this PR merged is to remove the logstash.conf file and instead make the setup-logstash.sh script upload the logstash conf file to Elasticsearch for centralized pipeline management.

So keep your eyes peeled. It's coming I promise!

[leonrinkel]

This would be really helpful. Are there any updates?

[fxdgear]

@leonrinkel no. unfortunately. My wife and I recently had our first baby and was on parental leave. Just getting back into the swing of things today. So I'm catching up on email and issues across all the repos I help maintain.

Though I have not forgotten about this feature! :)

[leonrinkel]

Congratulations 👼!