-
Notifications
You must be signed in to change notification settings - Fork 48
Developers Console is now "Cloud Platform Console" #118
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -143,25 +143,22 @@ | |
.. _Contact us: [email protected] | ||
|
||
.. ### Google Guide Links | ||
.. _Google Developers Console Guide: https://developers.google.com/console/help/new/ | ||
.. _Google Identity Platform Guide: https://developers.google.com/identity/protocols/OAuth2 | ||
.. _Application Default Credentials: https://developers.google.com/identity/protocols/application-default-credentials | ||
|
||
.. ### Google Product Links | ||
.. _Google BigQuery: https://cloud.google.com/bigquery/ | ||
.. _Google Cloud Dataflow: https://cloud.google.com/dataflow/ | ||
.. _Google Cloud Storage: https://cloud.google.com/storage/ | ||
.. _Google Compute Engine: https://cloud.google.com/compute/ | ||
.. _Google Developers Console: https://console.developers.google.com/ | ||
.. _Google Cloud Platform Console: https://console.cloud.google.com/ | ||
.. _Google Genomics: https://cloud.google.com/genomics/ | ||
.. _Google Cloud Datalab: https://cloud.google.com/datalab/ | ||
.. _Google Cloud Dataproc: https://cloud.google.com/dataproc/ | ||
|
||
.. ### Deep links into the Developers Console | ||
.. _Project List: https://console.developers.google.com/project | ||
.. _click-to-deploy NCBI BLAST: https://console.developers.google.com/project/_/launcher/details/click-to-deploy-images/ncbiblast | ||
.. _click-to-deploy Bioconductor: https://console.developers.google.com/project/_/mc/template/bioconductor | ||
.. _Deployments: https://console.developers.google.com/project/_/deployments | ||
.. _click-to-deploy NCBI BLAST: https://console.cloud.google.com/project/_/launcher/details/click-to-deploy-images/ncbiblast | ||
.. _click-to-deploy Bioconductor: https://console.cloud.google.com/project/_/mc/template/bioconductor | ||
.. _Deployments: https://console.cloud.google.com/project/_/deployments | ||
|
||
.. ### Deep links into cloud.google.com documentation | ||
.. _Compute Engine resource quota: https://cloud.google.com/compute/docs/resource-quotas | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1 @@ | ||
If you do not yet have a cloud project, `create a Genomics and Cloud Storage enabled project via the Google Developers Console <https://console.developers.google.com/start/api?id=genomics,storage_api>`_. | ||
If you do not yet have a cloud project, `create a Genomics and Cloud Storage enabled project via the Google Cloud Platform Console <https://console.cloud.google.com/start/api?id=genomics,storage_api>`_. |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -54,9 +54,9 @@ Google Cloud Platform data locations | |
|
||
* Google Cloud Storage folders | ||
* These files were loaded into Google Genomics datasets: | ||
* `gs://genomics-public-data/1000-genomes <https://console.developers.google.com/storage/genomics-public-data/1000-genomes/>`_ | ||
* `gs://genomics-public-data/1000-genomes-phase-3 <https://console.developers.google.com/storage/genomics-public-data/1000-genomes-phase-3/>`_ | ||
* A full mirror of http://ftp-trace.ncbi.nih.gov/1000genomes/ftp/ `gs://genomics-public-data/ftp-trace.ncbi.nih.gov/1000genomes/ftp/ <https://console.developers.google.com/storage/browser/genomics-public-data/ftp-trace.ncbi.nih.gov/1000genomes/ftp/>`_. | ||
* `gs://genomics-public-data/1000-genomes <https://console.cloud.google.com/storage/genomics-public-data/1000-genomes/>`_ | ||
* `gs://genomics-public-data/1000-genomes-phase-3 <https://console.cloud.google.com/storage/genomics-public-data/1000-genomes-phase-3/>`_ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The following file still points to When one clicks on it, it will show the following:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same thing as above for the phase-3 link here: |
||
* A full mirror of http://ftp-trace.ncbi.nih.gov/1000genomes/ftp/ `gs://genomics-public-data/ftp-trace.ncbi.nih.gov/1000genomes/ftp/ <https://console.cloud.google.com/storage/browser/genomics-public-data/ftp-trace.ncbi.nih.gov/1000genomes/ftp/>`_. | ||
* Google Genomics Dataset IDs | ||
* Dataset Id `10473108253681171589 <https://developers.google.com/apis-explorer/#p/genomics/v1/genomics.datasets.get?datasetId=10473108253681171589>`_ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. s/v1/v1beta2 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. v1 is the correct version There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yup :) |
||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -31,7 +31,7 @@ This dataset comprises the three public synthetic tumor/normal pairs created for | |
|
||
Google Cloud Platform data locations | ||
------------------------------------ | ||
* Google Cloud Storage folder `gs://public-dream-data/ <https://console.developers.google.com/storage/browser/public-dream-data/>`_ | ||
* Google Cloud Storage folder `gs://public-dream-data/ <https://console.console.google.com/storage/browser/public-dream-data/>`_ | ||
* Google Genomics dataset `337315832689 <https://developers.google.com/apis-explorer/#p/genomics/v1/genomics.datasets.get?datasetId=337315832689>`_. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This one seems to still point to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. v1 has superceded v1beta2. Version v1beta2 of the API is deprecated. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh sorry, I totally missed Jonathan's post here - and thought that https://groups.google.com/forum/#!topic/google-genomics-discuss/SsuPecO29WA Thanks for the clarification. ~p |
||
|
||
Provenance | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -28,7 +28,7 @@ This dataset comprises roughly 180 Complete Genomics genomes. See the `Personal | |
Google Cloud Platform data locations | ||
------------------------------------ | ||
|
||
* Google Cloud Storage folder `gs://pgp-harvard-data-public <https://console.developers.google.com/storage/pgp-harvard-data-public>`_ | ||
* Google Cloud Storage folder `gs://pgp-harvard-data-public <https://console.cloud.google.com/storage/pgp-harvard-data-public>`_ | ||
* Google Genomics Dataset ID `9170389916365079788 <https://developers.google.com/apis-explorer/#p/genomics/v1/genomics.datasets.get?datasetId=9170389916365079788>`_ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. same thing as above - this one points to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes. We are in the process of upgrading all samples to v1 from v1beta2. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah, perfect! Thanks, |
||
* Google BigQuery Dataset IDs | ||
* `google.com:biggene:pgp_20150205.genome_calls <https://bigquery.cloud.google.com/table/google.com:biggene:pgp_20150205.genome_calls>`_ | ||
|
@@ -38,7 +38,7 @@ Provenance | |
|
||
Google Genomics variant set for dataset ``pgp_20150205``: `9170389916365079788 <https://developers.google.com/apis-explorer/#p/genomics/v1/genomics.datasets.get?datasetId=9170389916365079788>`_ contains: | ||
|
||
* the Complete Genomics datasets from `gs://pgp-harvard-data-public/**/masterVar*bz2 <https://console.developers.google.com/storage/pgp-harvard-data-public>`_ | ||
* the Complete Genomics datasets from `gs://pgp-harvard-data-public/**/masterVar*bz2 <https://console.cloud.google.com/storage/pgp-harvard-data-public>`_ | ||
|
||
Appendix | ||
-------- | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -22,7 +22,7 @@ This dataset comprises the 17 member CEPH pedigree 1463. See http://www.illumin | |
Google Cloud Platform data locations | ||
------------------------------------ | ||
|
||
* Google Cloud Storage folder `gs://genomics-public-data/platinum-genomes <https://console.developers.google.com/storage/genomics-public-data/platinum-genomes/>`_ | ||
* Google Cloud Storage folder `gs://genomics-public-data/platinum-genomes <https://console.cloud.google.com/storage/genomics-public-data/platinum-genomes/>`_ | ||
* Google Genomics Dataset ID `3049512673186936334 <https://developers.google.com/apis-explorer/#p/genomics/v1/genomics.datasets.get?datasetId=3049512673186936334>`_ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. s/v1/v1beta2 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. v1 is the correct version There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Understood :) |
||
|
||
* `ReadGroupSet IDs <https://developers.google.com/apis-explorer/#p/genomics/v1/genomics.readgroupsets.search?fields=readGroupSets(id%252Cname)&_h=5&resource=%257B%250A++%2522datasetIds%2522%253A+%250A++%255B%25223049512673186936334%2522%250A++%255D%250A%257D&>`_ | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -22,7 +22,7 @@ Reference Genomes such as GRCh37, GRCh37lite, GRCh38, hg19, hs37d5, and b37 are | |
Google Cloud Platform data locations | ||
------------------------------------ | ||
|
||
* Google Cloud Storage folder `gs://genomics-public-data/references <https://console.developers.google.com/storage/genomics-public-data/references/>`_ | ||
* Google Cloud Storage folder `gs://genomics-public-data/references <https://console.cloud.google.com/storage/genomics-public-data/references/>`_ | ||
* Google Genomics `reference sets <https://developers.google.com/apis-explorer/#p/genomics/v1/genomics.referencesets.search>`_ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. s/v1/v1beta2 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. v1 is the correct version There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yup, my mistake. Thanks, |
||
|
||
Provenance | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -122,7 +122,7 @@ This command uses an older, slower REST based API. To run using GRPC API impleme | |
|
||
For Java 7 (as opposed to 8) use *alpn-boot-7.1.3.v20150130.jar*. | ||
|
||
We use a test readset here from `genomics-test-data <https://console.developers.google.com/project/genomics-test-data/storage/browser/gatk-tools-java/>`_ project. | ||
We use a test readset here from `genomics-test-data <https://console.cloud.google.com/project/genomics-test-data/storage/browser/gatk-tools-java/>`_ project. | ||
|
||
Specifying a genomics region to use from the readset | ||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
@@ -143,7 +143,7 @@ Timing the reading speed from the cloud | |
You can run `gatk-tools-java/src/main/scripts/example.sh <https://github.com/googlegenomics/gatk-tools-java/blob/master/src/main/scripts/example.sh>`_ with and without "grpc" command line parameter to see the difference in reading speed. The timing statistics are dumped to the terminal. | ||
We benchmarked **x11** speed improvements with GRPC compared to REST, giving **~12,000 reads/second**. | ||
|
||
The tests were done on `Platinum Genomes NA12877_S1.bam dataset <https://console.developers.google.com/storage/browser/genomics-public-data/platinum-genomes/bam/?_ga=1.197206447.160385476.1431305548>`_, please see the `detailed writeup of the test procedure and results <https://docs.google.com/document/d/1Br7RMSbAChNpG6pi2teujf-YthczF-rAM1afSgDoCgQ/edit#>`_ if you want to repeat the test. | ||
The tests were done on `Platinum Genomes NA12877_S1.bam dataset <https://console.cloud.google.com/storage/browser/genomics-public-data/platinum-genomes/bam/?_ga=1.197206447.160385476.1431305548>`_, please see the `detailed writeup of the test procedure and results <https://docs.google.com/document/d/1Br7RMSbAChNpG6pi2teujf-YthczF-rAM1afSgDoCgQ/edit#>`_ if you want to repeat the test. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The Google Doc requires access permission, is that a prerequisite or should it be public? |
||
|
||
We therefore recommend running GRPC variants of command line. | ||
|
||
|
@@ -170,7 +170,7 @@ How do you find an ID of the readset from the :doc:`/use_cases/discover_public_ | |
|
||
We will do it step by step using the command line API client. | ||
|
||
* Lets say we want to use `Platinum Genomes NA12877_S1.bam readgroupset <https://console.developers.google.com/storage/browser/genomics-public-data/platinum-genomes/bam/?_ga=1.197206447.160385476.1431305548>`_ from :doc:`/use_cases/discover_public_data/1000_genomes` project. | ||
* Lets say we want to use `Platinum Genomes NA12877_S1.bam readgroupset <https://console.cloud.google.com/storage/browser/genomics-public-data/platinum-genomes/bam/?_ga=1.197206447.160385476.1431305548>`_ from :doc:`/use_cases/discover_public_data/1000_genomes` project. | ||
|
||
* The `documentation <https://cloud.google.com/genomics/data/1000-genomes?hl=en>`_ page states that the dataset id for this set of files is **10473108253681171589**. | ||
|
||
|
@@ -190,7 +190,7 @@ We will do it step by step using the command line API client. | |
Now lets suppose we are not looking for one of the readgroupsets form the genomics public data but instead want to use one from our own project. | ||
In this case we need to figure out the *dataset id* for our files first, before we can use "readgroupsets list" command to list the individual readgroupsets. | ||
|
||
* Lets say we want to figure out which dataset ids are present under `genomics test data <https://console.developers.google.com/project/genomics-test-data/storage/browser>`_ project. | ||
* Lets say we want to figure out which dataset ids are present under `genomics test data <https://console.cloud.google.com/project/genomics-test-data/storage/browser>`_ project. | ||
|
||
* First we need to set the project id for subsequent commands to be our project using | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BigQuery works publicly, but accounts seem to have access to the files on gs:// disabled. I'm getting the following error:
It might be good to have it automated so that gs:// also is accessible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi Paul - can you explain further? I see no problem accessing the tute_db bucket from either the Cloud Console, nor using "gsutil ls". Where are you receiving this error message?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi Matt,
I see the same message for both. So below are the steps of what I perform and what I see:
https://console.cloud.google.com/m/cloudstorage/b/tute_db/o/hg19allsnp.split100.ggapidb.txt.gz
The response I get from the browser is the following:
It seemed to have worked before, since I performed an analysis through R on the files, and posted it at the following location a while back:
https://groups.google.com/forum/#!topic/tute-genomics/wW5ubyPDV-Y
Thank you for helping me out.
Paul