Skip to content

Commit

Permalink
Small pre-release adjustments
Browse files Browse the repository at this point in the history
  • Loading branch information
karllessard committed Feb 23, 2024
1 parent 22e370a commit b14bc39
Show file tree
Hide file tree
Showing 23 changed files with 149 additions and 58 deletions.
30 changes: 30 additions & 0 deletions .github/ISSUE_TEMPLATE/00-bug-performance-issue.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
name: Bug/Performance Issue
about: Use this template for reporting a bug or a performance issue.

---

<em>Please make sure that this is a bug. As per our [GitHub Policy](https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md), we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em>

**System information**
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04 x86\_64):
- TensorFlow installed from (source or binary):
- TensorFlow version (use command below):
- Java version (i.e., the output of `java -version`):
- Java command line flags (e.g., GC parameters):
- Python version (if transferring a model trained in Python):
- Bazel version (if compiling from source):
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version:
- GPU model and memory:

**Describe the current behavior**

**Describe the expected behavior**

**Code to reproduce the issue**
Provide a reproducible test case that is the bare minimum necessary to generate the problem.

**Other info / logs**
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
27 changes: 27 additions & 0 deletions .github/ISSUE_TEMPLATE/10-build-installation-issue.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
name: Build/Installation Issue
about: Use this template for build/installation issues

---

<em>Please make sure that this is a build/installation issue. As per our [GitHub Policy](https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md), we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template</em>

**System information**
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04 x86\_64):
- TensorFlow installed from (source or binary):
- TensorFlow version:
- Java version (i.e., the output of `java -version`):
- Java command line flags (e.g., GC parameters):
- Installed from Maven Central?:
- Bazel version (if compiling from source):
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version:
- GPU model and memory:


**Describe the problem**

**Provide the exact sequence of commands / steps that you executed before running into the problem**

**Any other info / logs**
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
17 changes: 17 additions & 0 deletions .github/ISSUE_TEMPLATE/20-documentation-issue.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
name: Documentation Issue
about: Use this template for documentation related issues

---

<em>Please make sure that this is a documentation issue. As per our [GitHub Policy](https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md), we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:doc_template</em>


**System information**
- TensorFlow version:
- Doc Link:


**Describe the documentation issue**

**We welcome contributions by users. Will you be able to update submit a PR (use the [doc style guide](https://www.tensorflow.org/community/documentation)) to fix the doc Issue?**
22 changes: 22 additions & 0 deletions .github/ISSUE_TEMPLATE/30-feature-request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
name: Feature Request
about: Use this template for raising a feature request

---

<em>Please make sure that this is a feature request. As per our [GitHub Policy](https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md), we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template</em>


**System information**
- TensorFlow version (you are using):
- Are you willing to contribute it (Yes/No):



**Describe the feature and the current behavior/state.**

**Will this change the current api? How?**

**Who will benefit with this feature?**

**Any Other info.**
13 changes: 13 additions & 0 deletions .github/ISSUE_TEMPLATE/40-other-issues.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
name: Other Issues
about: Use this template for any other non-support related issues

---

This template is for miscellaneous issues not covered by the other issue categories.

For questions on how to work with TensorFlow, or support for problems that are not verified bugs in TensorFlow, please go to [StackOverflow](https://stackoverflow.com/questions/tagged/tensorflow).

If you are reporting a vulnerability, please use the [dedicated reporting process](https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md).

For high-level discussions about TensorFlow, please post to [email protected], for questions about the development or internal workings of TensorFlow, or if you would like to know how to contribute to TensorFlow, please post to [email protected].
7 changes: 6 additions & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,12 @@ jobs:
java-version: '11'
- name: Checkout repository
uses: actions/checkout@v1
- name: Build project
run: |
java -version
mvn -version
mvn clean install -B -U -e
- name: Deploy snapshot artifacts
run: |
echo "<settings><servers><server><id>ossrh</id><username>${{ secrets.CI_DEPLOY_USERNAME }}</username><password>${{ secrets.CI_DEPLOY_PASSWORD }}</password></server></servers></settings>" > $HOME/.m2/settings.xml
mvn clean deploy -Pdeploying -B -e
mvn deploy -Pdeploying -B -e -Dmaven.test.skip=true
30 changes: 5 additions & 25 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,29 +35,12 @@ This can be done in `.mvn/jvm.config` or `MAVEN_OPTS`.
### Native Builds

By default, the build will attempt to download the existing TensorFlow binaries from the web for the platform it is running on (so you need to have an active internet connection).
If such binaries are not available for your platform, you will need to build the TensorFlow runtime library from sources, by adding the `-Dnative.build` argument to your Maven
If such binaries are not available for your platform, you will need to build the TensorFlow runtime library from sources, by appending the `-Dnative.build` argument to your Maven
command. This requires a valid environment for building TensorFlow, including the [bazel](https://bazel.build/) build tool and a few Python dependencies
(please read [TensorFlow documentation](https://www.tensorflow.org/install/source) for more details). Note that building from sources can take multiple hours on a regular laptop.

### GPU Support

Currently, due to build time constraints, the GPU binaries only support compute capacities 3.5 and 7.0.
To use with un-supported GPUs, you have to build it yourself, after changing the value [here](tensorflow-core/tensorflow-core-api/build.sh#L27),
setting the environment variable `TF_CUDA_COMPUTE_CAPABILITIES`, or configuring it in a bazel rc file (
i.e. `build --action_env TF_CUDA_COMPUTE_CAPABILITIES="6.1"`). While this is far from ideal, we are working on getting more build resources, and for
now this is the best option.

To build for GPU, pass `-Djavacpp.platform.extension=-gpu` to maven. By default, the CI options are used for the bazel build, see the above section
for more info. If you add `bazelrc` files, make sure the `TF_CUDA_COMPUTE_CAPABILITIES` value in them matches the value set elsewhere, as it will take
precedence if present.

### Apple Silicon

The TensorFlow Java project relies on [GitHub-hosted runners](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners)
to build and distribute the native binaries for TensorFlow. Unfortunately at the moment, GitHub Actions still does not support runners with a
Apple Silicon chip (such as M1). Therefore, we cannot distribute artifacts for this platform, so they must be build locally on these systems.

Please follow the present [procedure](CONTRIBUTING.md#building) for building TensorFlow Java.
To build for GPU, pass `-Djavacpp.platform.extension=-gpu` to maven. If you want to use TensorFlow Java with unsupported GPUs, set the environment variable `TF_CUDA_COMPUTE_CAPABILITIES`, or
configure it in a bazel rc file (i.e. `build --action_env TF_CUDA_COMPUTE_CAPABILITIES="6.1"`).

### Native Crashes

Expand Down Expand Up @@ -115,14 +98,12 @@ patches are found in `tensorflow-core/tensorflow-core-native/external`.
- To create a new patch or update one, you can make a copy of the TensorFlow source file to change, make your change and generate a patch using `git diff <file> <file-updated>`
- If more than one file needs to be added to the patch, it's easier to clone the [TensorFlow repository](https://github.com/tensorflow/tensorflow), apply the changes and use `git diff` at the root of the tree

Once these steps have been executed, you can run `mvn install` to build the new version but make sure to delete completely the `src/gen` directory of `tensorflow-core-api` first.

### Generating Java Bindings

After upgrading the TensorFlow library, you need to regenerate all Java bindings that depends on the native code. That includes Java protos, C API bindings (JavaCPP) and
operator classes. You can trigger the regeneration of these bindings with the Maven command `mvn clean install -Pgenerating`.

This will trigger a small Bazel build of the TensorFlow sources to regenerate the Java protos, so make sure your [environment](CONTRIBUTING.md#native-builds) is setup properly.
This will also trigger a small Bazel build of the TensorFlow sources to regenerate the Java protos, so make sure your [environment](CONTRIBUTING.md#native-builds) is setup properly.

#### Operations Classification

Expand Down Expand Up @@ -182,5 +163,4 @@ org.tensorflow.exceptions.TensorFlowException: No gradient defined for op: ReadV
```
The description in the [linked file](https://www.tensorflow.org/code/tensorflow/cc/gradients/README.md) are accurate for adding C++ Graph gradients, which are used by our `Graph`. Examples of doing that are [tensorflow/tensorflow#46115](https://github.com/tensorflow/tensorflow/pull/46115) and [tensorflow/tensorflow#47774](https://github.com/tensorflow/tensorflow/pull/47774).

However, Tensorflow Core is in the process of migrating gradient definitions to [`c/experimental/gradients`](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/c/experimental/gradients), which will be what our eager mode uses once it has gradient support.
Anyone adding gradients is strongly encouraged to add one there as well, and eventually it should replace the legacy `cc/gradients` gradients.
You can also code and register the missing gradients in Java, using the TensorFlow Java custom gradient registration capabilities. Check at the JavaDoc of `tensorflow-core-api` for more details.
8 changes: 3 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,11 +59,9 @@ only binaries for the followings are being **supported and distributed** by this
- `linux-x86_64`: Linux platforms on Intel chips
- `linux-x86_64-gpu`: Linux platforms on Intel chips with Cuda GPU support
- `macosx-x86_64`: MacOS X platforms on Intel chips
- `macosx-arm64`: MacOS X platforms on Apple Silicon chips
- `windows-x86_64`: Windows platforms on Intel chips

*Note: No binaries are distributed to run TensorFlow Java on machines with Apple Silicon chips (`macosx-arm64`), these
should be build from sources. See [here](CONTRIBUTING.md#apple-silicon) for more details.*

For example, for building a JAR that uses TensorFlow and is targeted to be deployed only on Linux
systems with no GPU support, you should add the following dependencies:
```xml
Expand Down Expand Up @@ -98,7 +96,7 @@ native dependencies as follows:
<groupId>org.tensorflow</groupId>
<artifactId>tensorflow-core-native</artifactId>
<version>0.5.0</version>
<classifier>macosx-x86_64</classifier>
<classifier>macosx-arm64</classifier>
</dependency>
<dependency>
<groupId>org.tensorflow</groupId>
Expand All @@ -116,7 +114,7 @@ Only one dependency can be added per platform, meaning that you cannot add nativ
In some cases, it might be preferable to add a single dependency that includes transitively all the artifacts
required to run TensorFlow Java on any [supported platforms](README.md#individual-dependencies)

- `tensorflow-core-platform`: Includes supports for `linux-x86_64`, `macosx-x86_64` and `windows-x86_64`
- `tensorflow-core-platform`: Includes `tensorflow-core-api`, plus native artifacts for `linux-x86_64`, `macosx-arm64`, `macosx-x86_64` and `windows-x86_64`

For example, to run TensorFlow Java on any platform for which a binary is being distributed by this project, you can
simply add this dependency to your application:
Expand Down
2 changes: 1 addition & 1 deletion RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ for temporary staging.
2. Execute the `release.sh` script. This will deploy artifacts on OSS Sonatype. All native artifacts
previously temporarily staged by GitHub Actions will be fetched, signed and redeployed as well.
The script takes in paramater the sequence number of the staging repository created in OSSRH
The script takes in a parameter the sequence number of the staging repository created in OSSRH
by the GitHub Actions workflow. You can retrieve this ID by looking in the staging repositories
in OSSRH console directly, or check at the output of the step `Create Staging Repository` of the
`prepare` job in the workflow execution, where the ID is printed.
Expand Down
1 change: 1 addition & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@
</snapshots>
</repository>
</repositories>

<pluginRepositories>
<pluginRepository>
<id>ossrh-snapshots</id>
Expand Down
4 changes: 1 addition & 3 deletions release.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ fi
# To get a shell to poke around the maven artifacts with.
if [[ -z "${CMD}" ]]
then
CMD="mvn clean deploy -B -e --settings ./settings.xml -Pdeploying -Preleasing"
CMD="mvn clean deploy -B -e --settings ./settings.xml -Pdeploying -Preleasing -DstagingRepositoryId=${STAGING_SEQ}"
fi

export GPG_TTY=$(tty)
Expand All @@ -46,8 +46,6 @@ then
fi

docker run \
-e IN_CONTAINER="true" \
-e STAGING_SEQ="${STAGING_SEQ}" \
-e GPG_TTY="${GPG_TTY}" \
-v ${PWD}:/tensorflow-java \
-v ${HOME}/.gnupg:/root/.gnupg \
Expand Down
2 changes: 1 addition & 1 deletion tensorflow-core/tensorflow-core-api/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
<description>Platform-dependent native code and pure-Java code for the TensorFlow machine intelligence library.</description>

<properties>
<ndarray.version>0.4.0</ndarray.version>
<ndarray.version>1.0.0-rc.1</ndarray.version>
<truth.version>1.1.5</truth.version>
<test.download.skip>false</test.download.skip>
<test.download.folder>${project.build.directory}/tf-text-download/</test.download.folder>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ static TBfloat16 tensorOf(Shape shape) {
* @return the new tensor
*/
static TBfloat16 tensorOf(Shape shape, FloatDataBuffer data) {
return Tensor.of(TBfloat16.class, shape, d -> d.write(data));
return Tensor.of(TBfloat16.class, shape, d -> d.copyFrom(data));
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ static TBool tensorOf(Shape shape) {
* @return the new tensor
*/
static TBool tensorOf(Shape shape, BooleanDataBuffer data) {
return Tensor.of(TBool.class, shape, d -> d.write(data));
return Tensor.of(TBool.class, shape, d -> d.copyFrom(data));
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ static TFloat16 tensorOf(Shape shape) {
* @return the new tensor
*/
static TFloat16 tensorOf(Shape shape, FloatDataBuffer data) {
return Tensor.of(TFloat16.class, shape, d -> d.write(data));
return Tensor.of(TFloat16.class, shape, d -> d.copyFrom(data));
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ static TFloat32 tensorOf(Shape shape) {
* @return the new tensor
*/
static TFloat32 tensorOf(Shape shape, FloatDataBuffer data) {
return Tensor.of(TFloat32.class, shape, d -> d.write(data));
return Tensor.of(TFloat32.class, shape, d -> d.copyFrom(data));
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ static TFloat64 tensorOf(Shape shape) {
* @return the new tensor
*/
static TFloat64 tensorOf(Shape shape, DoubleDataBuffer data) {
return Tensor.of(TFloat64.class, shape, d -> d.write(data));
return Tensor.of(TFloat64.class, shape, d -> d.copyFrom(data));
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ static TInt32 tensorOf(Shape shape) {
* @return the new tensor
*/
static TInt32 tensorOf(Shape shape, IntDataBuffer data) {
return Tensor.of(TInt32.class, shape, d -> d.write(data));
return Tensor.of(TInt32.class, shape, d -> d.copyFrom(data));
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ static TInt64 tensorOf(Shape shape) {
* @return the new tensor
*/
static TInt64 tensorOf(Shape shape, LongDataBuffer data) {
return Tensor.of(TInt64.class, shape, d -> d.write(data));
return Tensor.of(TInt64.class, shape, d -> d.copyFrom(data));
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ static TUint16 tensorOf(Shape shape) {
* @return the new tensor
*/
static TUint16 tensorOf(Shape shape, ShortDataBuffer data) {
return Tensor.of(TUint16.class, shape, d -> d.write(data));
return Tensor.of(TUint16.class, shape, d -> d.copyFrom(data));
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ static TUint8 tensorOf(Shape shape) {
* @return the new tensor
*/
static TUint8 tensorOf(Shape shape, ByteDataBuffer data) {
return Tensor.of(TUint8.class, shape, d -> d.write(data));
return Tensor.of(TUint8.class, shape, d -> d.copyFrom(data));
}

/**
Expand Down
Loading

0 comments on commit b14bc39

Please sign in to comment.