Refactor and improve docs

[plaguss]

Refactor and improve docs.

Closes #120

This PR is a refactor of the docs settings, including new sections, ased on the ideas from diátaxis.
We now have a Learn section that contains tutorials and how-to guides (work in progress), and a technical reference that contains both concept guides and the API reference.

[alvarobartt]

Hi here @plaguss, nice idea! I like the approach, feel free to open the PR for review once you're happy with it / is more mature 👍🏻

[plaguss]

You can take a look and let me know whatever. I think all the current examples can work perfectly as user guides with some explanation of what they do, I've let an example for one of the scripts . If you think so I can split them with @ignacioct and move them there.

[ignacioct]

@plaguss I also like the structure, and happy to help and split it between you and me. I'm still learning about distilabel, so I might need a little guidance, but eager to help.

[plaguss]

@plaguss I also like the structure, and happy to help and split it between you and me. I'm still learning about distilabel, so I might need a little guidance, but eager to help.

we are together in that 😉

[dvsrepo]

For distilabel docs, I'd highly recommend a more sober, technical and concise style.

[dvsrepo]

Hi @plaguss I like the structure too and the ref you make to diataxis. The only thing is that I feel the proposal of using the current examples as the main topics for the user guides is not very aligned. They are very quick examples and not so meaningful. The line between those and the tutorial is not so clean.

Things we need to document well:

Pipeline and pipeline

Tasks: what are the current tasks, their parameters, how to use these parameters

For preference a description of the different options (UF, etc.), parameters, etc.

For text generation, how to use prompt templates, parameters, and how to use principles.

LLM: what are the current LLMs, their parameters, etc. Here for example we should help users to choose and option, to explain how llamacpp is used (which models?), how to setup vLLM with multiple GPUs (cc @gabrielmbmb )

Happy to do a call when I'm back from my holidays but I'd recommend keeping a mindset of adding parts describing key things in the most direct, descriptive, but simple language.

[dvsrepo]

@plaguss can you replace the alembic svg with this:

[dvsrepo]

good job, this is a very good version.

My only comment is that I think the noqa comments if displayed in the docs can be annoying and out of place (never seen this in other docs), don't we have a general way to exclude those code snippets from the qa analysis?

[plaguss]

good job, this is a very good version.

My only comment is that I think the noqa comments if displayed in the docs can be annoying and out of place (never seen this in other docs), don't we have a general way to exclude those code snippets from the qa analysis?

Done! my bad, I just removed the /docs folder from the ruff formatter (should have done that the first time).
I added the new alembic.svg, but I left the previous one because it's not working for the favicon (I don't know why).

[alvarobartt]

LGTM! Thanks for this awesome (and much needed) PR @plaguss 🎉 I just have some comments that you can find below, but great work so far, it's been really straight forward to go through the docs!

P.S. Also I'm unsure about the value of keeping the scripts in separate files and then injecting them in the Markdown files, could you elaborate on that decision? How's that better to keeping those within the Markdown files? Thanks in advance 👍🏻

[plaguss]

LGTM! Thanks for this awesome (and much needed) PR @plaguss 🎉 I just have some comments that you can find below, but great work so far, it's been really straight forward to go through the docs!

P.S. Also I'm unsure about the value of keeping the scripts in separate files and then injecting them in the Markdown files, could you elaborate on that decision? How's that better to keeping those within the Markdown files? Thanks in advance 👍🏻

Thank you!!

The decision for keeping the scripts in the snippets directory instead of the Markdown files because we already had it that way for a couple of examples. I would be useful if we reused the code snippets in different parts of the docs (which is not the case currently, and if that was the case we could do it for a single case specifically). In this case I think it's more a matter of preference, but I'm fine with any decision, I can place them directly in the markdown files and remove the code snippets.

[alvarobartt]

LGTM! Thanks for this awesome (and much needed) PR @plaguss 🎉 I just have some comments that you can find below, but great work so far, it's been really straight forward to go through the docs!
P.S. Also I'm unsure about the value of keeping the scripts in separate files and then injecting them in the Markdown files, could you elaborate on that decision? How's that better to keeping those within the Markdown files? Thanks in advance 👍🏻

Thank you!!

The decision for keeping the scripts in the snippets directory instead of the Markdown files because we already had it that way for a couple of examples. I would be useful if we reused the code snippets in different parts of the docs (which is not the case currently, and if that was the case we could do it for a single case specifically). In this case I think it's more a matter of preference, but I'm fine with any decision, I can place them directly in the markdown files and remove the code snippets.

Thanks for the answer! I'm happy keeping those within the docs instead, but may be opinionated, so lets double-check with @gabrielmbmb and @dvsrepo to see what are their thoughts 👍🏻

[gabrielmbmb]

Hi @plaguss! Looks good to me, I saw there are some scripts using in the docs that got the license header injected. I think we should skip adding the license in those scripts.

@alvarobartt regarding the decision on having the code snippets in different files, I think it's better to have them in a .py file because is better for the development of them (you get assistance from the IDE and they can be formatted using tools) and also you can quickly execute them to test they are working (for this to be true, they have to be fully working examples)