Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[yaml] Enhance YAML API docs #32825

Merged
merged 4 commits into from
Oct 31, 2024
Merged

[yaml] Enhance YAML API docs #32825

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4e64143
[yaml] Enhance YAML API docs
Polber Oct 16, 2024
6598f9c
use single html file
Polber Oct 24, 2024
0298b63
fix test failures
Polber Oct 25, 2024
bc08051
rebase on master
Polber Oct 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
164 changes: 138 additions & 26 deletions sdks/python/apache_beam/yaml/generate_yaml_docs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@

from apache_beam.portability.api import schema_pb2
from apache_beam.utils import subprocess_server
from apache_beam.version import __version__ as beam_version
from apache_beam.yaml import json_utils
from apache_beam.yaml import yaml_provider

Expand Down Expand Up @@ -284,42 +285,143 @@ def main():
markdown.extensions.toc.TocExtension(toc_depth=2),
'codehilite',
])
html = md.convert(markdown_out.getvalue())
pygments_style = pygments.formatters.HtmlFormatter().get_style_defs(
'.codehilite')
extra_style = '''
.nav {
height: 100%;
width: 12em;
* {
box-sizing: border-box;
}
body {
font-family: 'Roboto', sans-serif;
font-weight: normal;
color: #404040;
background: #edf0f2;
}
.body-for-nav {
background: #fcfcfc;
}
.grid-for-nav {
width: 100%;
}
.nav-side {
position: fixed;
top: 0;
left: 0;
overflow-x: hidden;
width: 300px;
height: 100%;
padding-bottom: 2em;
color: #9b9b9b;
background: #343131;
}
.nav a {
color: #333;
padding: .2em;
.nav-header {
display: block;
text-decoration: none;
width: 300px;
padding: 1em;
background-color: #2980B9;
text-align: center;
color: #fcfcfc;
}
.nav-header a {
color: #fcfcfc;
font-weight: bold;
display: inline-block;
padding: 4px 6px;
margin-bottom: 1em;
text-decoration:none;
}
.nav-header>div.version {
margin-top: -.5em;
margin-bottom: 1em;
font-weight: normal;
color: rgba(255, 255, 255, 0.3);
}
.nav a:hover {
color: #888;
.toc {
width: 300px;
text-align: left;
overflow-y: auto;
max-height: calc(100% - 4.3em);
scrollbar-width: thin;
scrollbar-color: #9b9b9b #343131;
}
.nav li {
list-style-type: none;
.toc ul {
margin: 0;
padding: 0;
list-style: none;
}
.content {
margin-left: 12em;
.toc li {
border-bottom: 1px solid #4e4a4a;
margin-left: 1em;
}
.toc a {
display: block;
line-height: 36px;
font-size: 90%;
color: #d9d9d9;
padding: .1em 0.6em;
text-decoration: none;
transition: background-color 0.3s ease, color 0.3s ease;
}
h2 {
margin-top: 2em;
.toc a:hover {
background-color: #4e4a4a;
color: #ffffff;
}
.transform-content-wrap {
margin-left: 300px;
background: #fcfcfc;
}
.transform-content {
padding: 1.5em 3em;
margin: 20px;
padding-bottom: 2em;
}
.transform-content li::marker {
display: inline-block;
width: 0.5em;
}
.transform-content h1 {
font-size: 40px;
}
.transform-content ul {
margin-left: 0.75em;
text-align: left;
list-style-type: disc;
}
hr {
color: gray;
display: block;
height: 1px;
border: 0;
border-top: 1px solid #e1e4e5;
margin-bottom: 3em;
margin-top: 3em;
padding: 0;
}
.codehilite {
background: #f5f5f5;
border: 1px solid #ccc;
border-radius: 4px;
padding: 0.2em 1em;
overflow: auto;
font-family: monospace;
font-size: 14px;
line-height: 1.5;
}
p code, li code {
white-space: nowrap;
max-width: 100%;
background: #fff;
border: solid 1px #e1e4e5;
padding: 0 5px;
font-family: monospace;
color: #404040;
font-weight: bold;
padding: 2px 5px;
}
'''

with open(options.html_file, 'w') as fout:
fout.write(
html = md.convert(markdown_out.getvalue())
with open(options.html_file, 'w') as html_out:
html_out.write(
f'''
<html>
<head>
Expand All @@ -329,13 +431,23 @@ def main():
{extra_style}
</style>
</head>
<body>
<div class="nav">
{md.toc}
</div>
<div class="content">
<h1>{title}</h1>
{html}
<body class="body-for-nav">
<div class="grid-for-nav">
<nav class="nav-side">
<div class="nav-header">
<a href=#>Beam YAML Transform Index</a>
<div class="version">
{beam_version}
</div>
</div>
{getattr(md, 'toc')}
</nav>
<section class="transform-content-wrap">
<div class="transform-content">
<h1>{title}</h1>
{html.replace('<h2', '<hr><h2')}
</div>
</section>
</div>
</body>
</html>
Expand Down
6 changes: 3 additions & 3 deletions sdks/python/apache_beam/yaml/tests/map.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ pipelines:
append: true
fields:
# TODO(https://github.com/apache/beam/issues/32832): Figure out why Java sometimes re-orders these fields.
literal_int: 10
named_field: element
literal_int: 10
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should try to get to the bottom of this... File a bug?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, this isn't super sustainable. I think we can just use #32832

literal_float: 1.5
literal_str: '"abc"'

Expand All @@ -43,5 +43,5 @@ pipelines:
- type: AssertEqual
config:
elements:
- {element: 100, literal_int: 10, named_field: 100, literal_float: 1.5, literal_str: "abc"}
- {element: 200, literal_int: 10, named_field: 200, literal_float: 1.5, literal_str: "abc"}
- {element: 100, named_field: 100, literal_int: 10, literal_float: 1.5, literal_str: "abc"}
- {element: 200, named_field: 200, literal_int: 10, literal_float: 1.5, literal_str: "abc"}
Loading