Skip to content

Latest commit

 

History

History
1380 lines (1100 loc) · 55.8 KB

scenarios.md

File metadata and controls

1380 lines (1100 loc) · 55.8 KB
Raw
layout title description permalink customjs toc toc-min
page
How Do I … ?
Describes different data transfer scenarios, and how Globus can help.
scenarios.html
./assets/js/faq.js
false
true

Everyone who wants to use Globus has to start somewhere, and that place is here. You might have been told "Oh, you need to do X", or you might not be sure what you need to do. This page is for you!

This page contains common Globus scenarios and questions. Each answer either contains a direct answer to your question or a link to a different part of the site.

The entries are grouped into sections:

After reading here, you should also take a look at the section of the site specific to your needs: Accounts, Client, or Server. Finally, if you still have questions, you should reach out to Support.

Start Here

{% capture z1 %}

You are probably here because you've heard about Globus and you want to use it for something!

Globus is used to transfer data between two points. Those two points could be a collection of data on a server (such as SCG Lab space), or a personal endpoint running on a laptop, or a space in the Cloud.

To get started, you should create a Globus account by signing in to Globus for the first time. If you have accounts at multiple institutions, you should also learn about how Globus accounts work, and how to link identies together.

(Here is a side challenge: If you have already registered for your ORCID iD, try linking it to your Stanford Globus identity!)

Now that you have your Globus account, you need to access either the collection or the endpoint which has the data you want. Every endpoint and collection can be identified either by name, or by unique ID. You can search for both on the Endpoints search page. Be careful when searching by name, because names are not guaranteed to be unique.

If the source or destination is a server, ask the server administrator for a link to the appropriate collection. (For example, see SCG and Sherlock.) If you are the server administrator, you can install Globus Connect Server to make your data available.

If the Cloud is one end (or both ends) of the transfer, Stanford supports using Globus with a number of cloud providers! See the list of supported cloud services.

If you want to transfer to or from an endpoint (a laptop or desktop), you can install Globus Connect Personal, giving you access to your endpoint's storage via Globus. Globus Connect Personal is smart enough to pause transfers when you laptop goes to sleep, and is able to cope with unstable network connections.

Once you have identified both ends of your transfer, you can move data! The Globus instructions are very helpful here.

{% endcapture %}

{% include panel.html parent="started" id="z1" scenario="How do I get started?" content=z1 %}

Accounts and Globus ID

Open All
{% capture a1 %}

Creating a Globus account is easy! If you have never used Globus before, just go to Create a Globus Account and follow the instructions there.

If you have used Globus before (for example, at another institution), then you should first log in to Globus using that institution's credentials, and link your Stanford identity.

{% endcapture %}

{% include panel.html parent="scenario" id="a1" scenario="How do I get access to Globus?" content=a1 %}

{% capture a2 %}

Although Globus recognizes that a person can have logins at multiple institutions, Globus only allows one email address per person per instutition. If you have an email address at another institution, you can Link Identities. As for your Stanford identity, whichever email address is your primary email address is what Globus will see.

See also How do I change my email address in Globus?.

{% endcapture %} {% include panel.html parent="scenario" id="a2" scenario="How do I add a new email address to Globus?" content=a2 %}

{% capture a3 %}

If you change your Stanford email address (in Accounts or StanfordYou), Globus will get the new email address a few days after the change, once you log back in to the Globus web site.

NOTE: Within Globus, people can be discovered by their identity (which, for Stanford, is always [email protected]) or by their email address. So, even if you change your email address to something else, you will still be discoverable in Globus by your SUNetID.

{% endcapture %} {% include panel.html parent="scenario" id="a3" scenario="How do I change my email address in Globus?" content=a3 %}

{% capture a4 %}

Globus ID is a form of identity—similar to your Stanford identity, but for entities (like labs and departments) and for users who can not get an identity through other means. At Stanford, a Globus ID is only needed when you want to run a Globus Connect Server, because Globus Connect Server instances can only be owned by entities (like your group), not by people. Globus IDs are also used at Stanford by developers, so that their Globus-using applications are not tied to individuals.

Getting a Globus ID for Globus Connect Server use is covered in the Globus Accounts section, on the Globus ID page.

{% endcapture %} {% include panel.html parent="scenario" id="a4" scenario="How do I get a Globus ID? Do I even need one?" content=a4 %}

The Globus Web Site

Open All
{% capture w1 %}

There are several ways to refer to an endpoint: By short name, by full name, or by UUID. You can look up an endpoint on the Globus web site using each of these identifiers.

  • You may have been given an endpoint UUID. For example, db57ddde-6d04-11e5-ba46-22000b92c6ec is the UUID (unique ID) of the ESnet test endpoint in Sunnyvale. With this unique ID, you can construct bookmarkable URLs. For example, if you take the following template…
    https://www.globus.org/app/transfer?destination_id=UUID

    … and replace UUID with the unique ID of the endpoint, you will have a URL that takes you directly to the transfer screen, with that endpoint filled in.

  • Your contact may have given you a short name (like srcc#oak), or a full name (like Stanford SRCC Oak). In both cases, you can go to the endpoints search page. Paste in the name you were given, and the endpoint should appear in the results.
  • If the endpoint is a shared endpoint, your contact may have given you explicit access to the endpoint. In that case, the endpoint will appear in the endpoints shared with me list.

Once you have located an endpoint, and you click on its name, if you check the URL of the page you are on, it will look like this…

https://www.globus.org/app/endpoints/db57ddde-6d04-11e5-ba46-22000b92c6ec/overview

In the above URL, db57ddde-6d04-11e5-ba46-22000b92c6ec is the endpoint's unique ID. You should make a note of it, as that unique ID will not change (unless the endpoint is deleted).

{% endcapture %} {% include panel.html parent="web" id="w1" scenario="Someone gave me access to a Globus endpoint. How do I access it?" content=w1 %}

{% capture w2 %}

If you are getting a permission denied error when trying to login to the endpoint, then you should check which credentials you are using. For example, endpoints normally only accept logins through their institution. So, if you have accounts at Stanford and UC Santa Cruz, your UC credentials will not let you in to a Stanford endpoint.

(If you have multiple logins to Globus, you might be logged in with the wrong one. You should Link Identities to keep that from happening again.)

If your credentials are being accepted; but you are getting permission denied when you try to do a directory listing, or when you are trying to transfer data from the endpoint, then the endpoint's configuration is not allowing access to that directory. For Globus Connect Server and shared endpoints, you need to reach out to the endpoint administrator. For Globus Connect Personal endpoints, you need to make sure you have configured access to the directory. See the Globus Connect Personal installation instructions (specifically, the Add Allowed Paths section).

If you get permission denied errors when transferring data to an endpoint, then you probably only have read-only access to the directory. You either need to talk to the endpoint administrator (for Shared endpoints and Globus Connect Server endpoints), or check your endpoint configuration (for Globus Connect Personal endpoints).

{% endcapture %} {% include panel.html parent="web" id="w2" scenario="I am getting Permission Denied errors when I try accessing an endpoint. How do I get access?" content=w2 %}

{% capture w5 %}

Shared endpoints do not exist on their own. They require a host endpoint, which is either a Globus Connect Personal endpoint or a Globus Connect Server endpoint.

If the host endpoint is a Globus Connect Personal endpoint, then you (the owner of that endpoint) must have access to Globus Plus, which is an optional Globus feature that is included in Stanford's campus subscription. Read more about Globus Plus.

If your host endpoint is a Globus Connect Server endpoint, the endpoint administrator may have disabled sharing. Or, the endpoint might not be a managed endpoint (which requires access to the Stanford campus Globus subscription). In the latter case, the endpoint administrator should read our guides for Globus Connect Server, specifically the pages on Pre-Installation Planning and Finishing Installation.

The other possible reason is, sharing might not be allowed for the directory you are trying to share. For Globus Connect Personal-hosted endpoints, you must explicitly give Globus Connect Personal access to the directory you wish to share (or to a parent directory), and sharing must be explicitly enabled. For Globus Connect Server-hosted endpoints, the default configuration is to allow sharing for any directory users can access, but this can be changed by the Globus Connect Server administrator.

So, if the host endpoint is running Globus Connect Personal, you need to make sure that you have Globus Plus, and that sharing is explicitly enabled. And if your host endpoint is running Globus Connect Server, you need to talk to the administrator of your host endpoint.

{% endcapture %} {% include panel.html parent="web" id="w5" scenario="I am getting Permission Denied errors when I try making a shared endpoint. How do I fix this?" content=w5 %}

{% capture w3 %}

Transfers between endpoints can be paused by endpoint administrators. If you got an email saying your transfer has been paused—and you didn't pause the transfer yourself—then an endpoint administrator (at one or both ends) has either paused your specific transfer, or has paused all transfers to/from their endpoint.

Transfers are often paused when endpoint maintenance is taking place. This is an alternative to simply letting connections fail (for example, during an endpoint restart).

Once the pause is lifted, your transfer will resume. If that does not happen, you should reach out to the administrator of the endpoint who paused your transfer.

{% endcapture %} {% include panel.html parent="web" id="w3" scenario="I got an email saying my transfer has been paused. How do I restart it?" content=w3 %}

{% capture w4 %}

In order to transfer files, Globus needs your credentials at both ends of the transfer. Giving Globus these credentials is called activating an endpoint.

For Globus Connect Personal endpoints, and for shared endpoints, activation is permanent; as long as you have access to the endpoint, no additional credentials are required. Globus Connect Server activations are time-limited. If your transfer involves a Globus Connect Server endpoint, and the activation expires during a transfer, your transfer will be paused and you will get this email.

To solve this problem, view your in-use endpoints, and look for any with expired credentials. For each endpoint with expired credentials, click on the endpoint to reactivate it. Once reactivated, your transfer will automatically resume.

Transfers already started will continue to attempt to run for up to a day before being terminated.

{% endcapture %} {% include panel.html parent="web" id="w4" scenario="I got an email saying my credentials have expired. How do I fix it?" content=w4 %}

The Globus Client (Globus Connect Personal) and Globus Plus

Open All
{% capture c1 %}

Globus Connect Personal (the proper name for the Globus client software) is how you can use Globus to transfer files to/from your laptop/desktop. The software runs on your system in the background, and gives you access to transfer files between your system and other Globus endpoints.

Read more about Globus Connect Personal .

{% endcapture %} {% include panel.html parent="client" id="c1" scenario="Someone told me that I need the Globus client. What even is it? How do I get it?" content=c1 %}

{% capture c8 %}

There is no ready-made, pre-approved solution for using Globus with High Risk data. Stanford does not have a BAA with Globus or the University of Chicago, and Stanford's use of Globus has not undergone a Data Risk Assessment.

Your best route forward may be to get your data into a form that is Low or Moderate Risk, and onto systems that are also Low or Moderate Risk, and install Globus Connect Personal (or Globus Connect Server) there. You should reach out to your IT people to assist, or engage with us.

If you are set on using Globus with High Risk data, or on a system that is exposed to High Risk data, you should expect to bear substantial costs in both money and time. Reach out if you are still interested!

{% endcapture %}
{% include panel.html
   parent="client"
   id="c8"
   scenario="My laptop or desktop handles High Risk (including PCI / PHI) data.  Can I use Globus on it?"
   content=c8

%}

{% capture c5 %}

No, but if you firewall outbound connection, then some ports will need to be opened. Globus Connect Personal does not need to accept connections from the outside, except in some cases when performing a transfer between two Globus Connect Personal endpoints.

Read more in the installation instructions.

If you are unable to get your firewall changed, then your last-resort option is to use the Stanford VPN, but you must use Full Traffic (non-split-tunnel) mode, which sends all traffic (even non-Stanford traffic) through the VPN.

{% endcapture %} {% include panel.html parent="client" id="c5" scenario="Do I need to open ports for Globus Connect Personal to work?" content=c5 %}

{% capture c2 %}

Globus Connect Personal already works to transfer data as quickly as possible, without making your Internet connection unusable. That being said, there are a few things you can do to help make things go faster.

  • Leave your laptop plugged in. Laptops will agressively power-save when it detects that you aren't using it. Plugging in your laptop typically reduces the level of power-saving.
  • Use a wired connection. With a wireless network, your traffic has to contend with all of the other laptops, phones, and other devices in the local area, along with other miscellaneous forms of interference (such as walls and microwaves). Using a wired connections removes these forms of interference.
  • Have good upstream bandwidth. For example, let's say your desktop is connected to a small Belkin Gigabit switch, shared with five other lab members, with one Ethernet cable to the wall. In that case, you are sharing a single Gigabit connection. Ideally, your desktop should be plugged directly into a network port on the wall, which goes back to a building switch, which then has at least a 10 Gbps link back to the core network. If you are in a residence that does not have ports on the wall (for example, because you have a cable modem), you should plug in the router.

If you routinely need to perform large (multi-TB) data transfers, you should consider taking advantage of the Stanford Research Network.

{% endcapture %} {% include panel.html parent="client" id="c2" scenario="I am transferring data to/from my laptop/desktop. How do I make my transfer go faster?" content=c2 %}

{% capture c4 %}

relay.globusonline.org is the link between Globus Connect Personal and Globus. All transfer operations—except for the actual data being transferred—goes through this connection.

Globus Connect Personal must be able to make outbound connections to relay.globusonline.org on TCP port 2223. To fix this, you will need to reach out to your IT person. For students (undergrads and grads), reach out to Student Technical Support at the Lathrop Learning Hub. For everyone else, go to your LNA.

If you are unable to get your firewall changed, then your last-resort option is to use the Stanford VPN, but you must use Full Traffic (non-split-tunnel) mode, which sends all traffic (even non-Stanford traffic) through the VPN.

Transfers already started will continue to attempt to run for up to a day before being terminated.

{% endcapture %} {% include panel.html parent="client" id="c4" scenario="I am getting errors connecting to relay.globusonline.org. How do I fix this?" content=c4 %}

{% capture c3 %}

First, try transferring a file from the ESNet Sunnyvale test endpoint to your system. If the transfer is successful, then your system is probably OK.

If ESnet transfers are also reporting 'connection failed', then the most likely problem is a network-level block. Globus Connect Personal needs to be able to make outbound connections to TCP ports 50000 through 51000. If that is blocked, then you will be able to start transfers, but they will not run. To fix this, you will need to reach out to your IT person. For students (undergrads and grads), reach out to Student Technical Support at the Lathrop Learning Hub. For everyone else, go to your LNA.

If you are unable to get your firewall changed, then your last-resort option is to use the Stanford VPN, but you must use Full Traffic (non-split-tunnel) mode, which sends all traffic (even non-Stanford traffic) through the VPN.

Your transfer will continue to attempt to run for up to a day before being terminated.

{% endcapture %} {% include panel.html parent="client" id="c3" scenario="All of my transfers are saying 'connection failed'. How do I fix this?" content=c3 %}

{% capture c6 %}

When you transfer files between a Globus Connect Personal endpoint and a Globus Connect Server endpoint, regardless of the direction of the transfer, the Globus Connect Personal endpoint is always the one that opens the connection (so that all connections are outbound from your endpoint). This is done because Globus Connect Server administrators make sure that inbound connections are allowed.

When both ends of a transfer are running Globus Connect Personal, one end has to allow an incoming connection, and that is not always possible. For example, if one laptop is on a home network (behind a router), and another laptop is on a different home network (behind a different router), the two routers will block inbound connections.

In a case like this, both ends of the connection should be brought onto a common network, in order for the data transfer to succeed. One way of doing this is by putting both endpoints onto the Stanford VPN. If that still does not work, try using the Full Traffic (non-split-tunnel) mode, which routes all network traffic through the VPN.

{% endcapture %}
{% include panel.html
   parent="client"
   id="c6"
   scenario="I am having problems transferring files between two Globus Connect Personal endpoints.  How do I fix this?"
   content=c6
%}

{% capture c7 %}

If you have a full-service (or full-sponsored) SUNetID (in other words, if you have Stanford email service), you can have the Globus Plus feature enabled on your account. If you already have a Globus Connect Personal endpoint, Globus Plus allows you to share part of it with others.

Read about Globus Plus.

{% endcapture %}
{% include panel.html
   parent="client"
   id="c7"
   scenario="How do I allow sharing?"
   content=c7
%}

The Globus Server (aka Globus Connect Server)

Open All

{% capture s1 %}

You will need to install Globus Connect Server onto a system that has access to the data your users wish to share. As you follow the process, make sure to watch out for sections that talk about enabling sharing. In the end, you will have an endpoint name, or a UUID.

Once your users have the endpoint information, they should go to the Globus Transfer Files page, where they can search for your endpoint, activate it (by authenticating), and transfer files.

See also the question Someone gave me access to a Globus endpoint. How do I access it?.

If your users want to transfer files to/from another environment (like, for example, another lab's storage), then that other environment will also need Globus Connect Server; you should direct the lab's IT person to this question. Once a Globus Connect Server is up and running there, your users will need that server's information (a name, a UUID, etc., just like with your endpoint).

If your users want to transfer files to/from their own systems (a laptop or a desktop), they will need Globus Connect Personal. You should point your users to this page, and the question Someone told me that I need the Globus client. What even is it? How do I get it?.

{% endcapture %} {% include panel.html parent="server" id="s1" scenario="I want my users to be able to transfer files using Globus. How do I do this?" content=s1 %}

{% capture s21 %}

This can be done, but it has to be done in two steps:

  1. First, install Globus Connect Server onto a system that has access to the data your users wish to share. As you follow the process, make sure to watch out for sections that talk about enabling sharing. In the end, you will have an endpoint URL. It will look something like this: 
    https://www.globus.org/app/endpoints/96a13ae8-1fb5-11e7-bc36-22000b9a448b/overview

    Give that URL to your users.

  2. Next, each user who wishes to share must access the endpoint using the URL provided (or, see the question Someone gave me access to a Globus endpoint. How do I access it? for alternative ways of discovering your endpoint). The endpoint's web page will have a My Shares tab; where users can authenticate, create a share, and give others access (read-only or read-write) to the share.
{% endcapture %}

{% include panel.html parent="server" id="s21" scenario="I want my users to be able to share files with others (who do not have local accounts). How do I do this?" content=s21 %}

{% capture s8 %}

Globus Connect Server works fine with storage that is accessed via NFS. With NFSv3, no changes are needed. With NFSv4, you will need to use sec=sys. The Kerberos security model does not work with Globus, because Globus has no way of getting Kerberos credentials for users.

{% endcapture %} {% include panel.html parent="server" id="s8" scenario="How can I use Globus Connect Server with NFS?" content=s8 %}

{% capture s11 %}

To meet the requirements for Moderate Risk data, simply follow the normal installation instructions on the Globus Connect Server pages, but be sure to use SUNetID Auth with CILogon. Other than that, the installation instructions will mention the settings which need to be in place for Moderate Risk environments.

{% endcapture %} {% include panel.html parent="server" id="s11" scenario="How do I use Globus for Moderate Risk data?" content=s11 %}

{% capture s12 %}

There is no ready-made, pre-approved solution for using Globus with High Risk data. Stanford does not have a BAA with Globus or the University of Chicago, and Stanford's use of Globus has not undergone a Data Risk Assessment.

Your best route forward may be to get your data into a form that is Low or Moderate Risk, and onto systems that are also Low or Moderate Risk, and do your transfers from there. You should reach out to your IT people to assist, or engage with us.

If you are set on using Globus with High Risk data, you should expect to bear substantial costs in both money and time. Reach out if you are still interested!

{% endcapture %} {% include panel.html parent="server" id="s12" scenario="How do I use Globus for High Risk (including PCI / PHI) data?" content=s12 %}

{% capture s2 %}

Beyond the documentation on the Pre-Installation Planning page, there is a simple checklist for determining which authentication method to use:

  1. In your environment, are your users using SUNetIDs as usernames? Then use CILogon.
  2. If your data are Moderate Risk, and your users are not using SUNetIDs as usernames, then you will need to set up some form of username mapping—or change everyone's usernames—and then use CILogon.
  3. If your data are Low Risk, and your users are not using SUNetIDs as usernames, then you can set up MyProxy OAuth authentication.
  4. If you get pushback about opening a web server to the world (which is required for MyProxy OAuth authentication), and you trust Globus enough to handle your user's credentials, then use legacy MyProxy.

{% endcapture %} {% include panel.html parent="server" id="s2" scenario="How do I know which authentication method to use?" content=s2 %}

{% capture s3 %}

In general, if you follow MinSec, using MyProxy OAuth is more secure than using legacy MyProxy. It all has to do with where your password goes.

With legacy MyProxy, your username and password are sent to Globus, which passes it on to the MyProxy service running on your endpoint. Although each connection (from you to Globus, and from Globus to MyProxy) uses TLS, it still means that your username and password are being held (albeit temporarily) by a third-party (Globus).

With MyProxy OAuth, your username and password are sent directly to the OAuth server running on your endpoint. Globus then receives a client cert, the same as if you were authenticating using CILogon.

Although MyProxy OAuth does require that you have a web server (HTTPS only) open to the world, as long as you follow proper installation and patching procedures, you should remain reasonably secure.

{% endcapture %} {% include panel.html parent="server" id="s3" scenario="How is MyProxy OAuth better than legacy MyProxy if they both use MyProxy?" content=s3 %}

{% capture s4 %}

Right now, it is not possible to handle Moderate Risk data in Globus without using CILogon. The reason is, MinSec for Applications for Moderate Risk applications requires that Duo two-step authentication be used for all interactive user logins, and many end-user Globus activities are interactive.

Unfortunately, none of the MyProxy-based authentication methods (legacy MyProxy or MyProxy OAuth) support any form of multi-factor authentication, due to the limitations of the MyProxy protocol.

{% endcapture %} {% include panel.html parent="server" id="s4" scenario="How can I handle Moderate Risk data without using CILogon?" content=s4 %}

{% capture s5 %}

Not easily, no. At least, not without a sizeable amount of effort.

{% endcapture %} {% include panel.html parent="server" id="s5" scenario="Can I use CILogon without using SUNetIDs?" content=s5 %}

{% capture s6 %}

OK, yes, it is possible, but it will either require constant maintenance and setup work, or will require software development.

When using CILogon, Globus uses client certs to authenticate to GridFTP as a person. That client cert includes a Distinguished Name (or DN) to identify the user, as well as an eduPersonPrincipalName (or EPPN) attribute that contains the user's "scoped username" (that is, their username in the context of the institution), which for Stanford is always [email protected].

For example, to see your CILogon distinguished name, go to CILogon and log in with your Stanford University credentials. The "Certificate Subject" is your DN.

The first way to convert a client cert into a username is to use a gridmap file. This is a mapping from DN to local username, and is described in Grid Security documentation Section 5.1. You will need to have each user go to CILogon, get their DN, and send it to you for inclusion in the gridmap file.

The second way is to write a callout. This is a shared library containing code that takes in various parameters (like a client cert) and returns a local username. The file format and calling convention is documented in Grid Security documentation Section 5.2 and is also explained on CILogon's site. You can see the code for Globus' EPPN callout on GitHub.

Unfortunately, due to the complexity involved in this process, must Stanford-based support groups are unable to provide free support for custom gridmaps. Your first call for support should be the Globus "Developer Discuss" forum on Google Groups, which you can join using your Stanford Google account.

{% endcapture %} {% include panel.html parent="server" id="s6" scenario="Can I really not use CILogon without SUNetIDs?" content=s6 %}

{% capture s7 %}

Happily, in this scenario, nothing else needs to be done! That is because, with CILogon, the authentication and authorization functions are separate.

With CILogon, authentication is handled by Stanford SAML. Globus passes the results of this successful authentication on to your endpoint, which extracts the SUNetID and looks for a matching user. So, no local authentication (password or otherwise) is needed.

It is important to note, CILogon only handles authentication. The authorization function (which determines what the user can access) is still handled by the operating system.

{% endcapture %} {% include panel.html parent="server" id="s7" scenario="My users coincidentally use SUNetIDs already, but not for passwords. How can I use CILogon?" content=s7 %}

{% capture s9 %}

Globus Connect Server works even when users are not able to log in directly. The server just needs to be able to identify users, and rely on the OS' authorization mechanisms to verify permissions.

See also the question How do I allow sharing when my users don't have home directories?

{% endcapture %} {% include panel.html parent="server" id="s9" scenario="How can I run Globus Connect Server on a server where users cannot log in?" content=s9 %}

{% capture s10 %}

Files uploaded to a Globus Connect Server endpoint are owned by the user who activated the endpoint. Files uploaded to a shared endpoint are owned by the user who created the shared endpoint. The file's group will be set to the owning user's primary group.

File permissions are set based on the GridFTP server's umask, and its default permissions. GridFTP's default permissions are 644 (read/write for owner; read-only for everyone else). To change GridFTP's default permissions, edit /etc/gridftp.conf, adding the following line:

perms 664

(Change 664 to whatever value is appropriate for your situation.)

GridFTP's umask is set by your system's service-management software (normally either SysV init, or systemd). It is normally set to 022. This umask means group and world write access will always be disabled, regardless of the permissions you set.

If you want to change the umask, you will have to modify the /etc/init.d/globus-gridftp-server script in two ways:

  1. First, look for this block of code in the script…
    lsb=""
if [ -f /lib/lsb/init-functions ]; then
    lsb=_lsb
    lsb_ok=1
    . /lib/lsb/init-functions
    if [ -f /etc/redhat-release ] && lsb_release -v | grep -q 'core-[123]'; then
        unset lsb_ok
    fi

fi

    In that block, change the string lsb=_lsb to lsb="". This is needed to stop LSB-specific startup scripts from running, as they will override the umask in all cases.

  2. Next, in the start() block, locate this line:
            $gridftpd -S -c $conf -C $confdir -pidfile "${pidfile}"

    Immediately before that line, add a new line, where you set the umask to be what you want.

This work is required because GridFTP does not have a built-in way to change its umask. Once the changes are made, restart GridFTP, and test.

{% endcapture %} {% include panel.html parent="server" id="s10" scenario="How do I control permissions for uploaded files?" content=s10 %}

{% capture s13 %}

When the 'encrypt transfer' box is checked, at the start of a transfer, Globus generates a 2048-bit RSA key, and a temporary SSL certificate valid for two days. The key and cert are given to both endpoints involved in the transfer.

When one endpoint connects to the other, it immediately opens a TLS connection, using the key and certificate provided. The default cipher list is the OpenSSL HIGH cipher list, which means the endpoints will generatlly use Elliptic-Curve Diffie-Hellman Ephemeral (ECDHE) symmetric-key negotiation (falling back to RSA key negotiation if needed), and some form of AES symmetric encryption.

If necessary (for example, for transfers which take more than two days to complete), Globus will automatically generate a new key and cert.

{% endcapture %} {% include panel.html parent="server" id="s13" scenario="How does Globus encrypt data traffic when encryption is requested?" content=s13 %}

{% capture s14 %}

The control traffic between Globus and you (used for directory listings and to control data transfer) also uses TLS. When an endpoint is created, Globus generates a very-long-lived 2048-bit RSA key and certificate. This is used as a server cert, only for connections coming from Globus. The default cipher list is the OpenSSL HIGH cipher list, but containing only the RSA key-exchange ciphers.

The only other traffic is API traffic during endpoint setup, and when checking for a new software version. Both connections are out to services on AWS, which use TLS with normal server certs. This API traffic comes from code written in Python, which uses whatever TLS client configuration is the default for the OS-provided Python 2 ssl module.

{% endcapture %} {% include panel.html parent="server" id="s14" scenario="How does Globus encrypt other traffic?" content=s14 %}

{% capture s15 %}

First, you need to set up a managed Globus Connect Server endpoint. Start at the Pre-Installation Planning page and go from there.

In that section, you will learn about the different sharing capabilities, how to enable them, and how to limit them. That will have the information you need!

{% endcapture %} {% include panel.html parent="server" id="s15" scenario="How do I allow sharing?" content=s15 %}

{% capture s16 %}

When a user creates a shared endpoint, Globus Connect Server must write some sensitive data related to the share; this data must only be accessible by the user who creates the shared endpoint. Normally, this information is stored in the user's home directory, but if users do not have home directories, an alternative storage location must be used.

This is a delicate procedure—the layers of permissions are fairly complex—but it is doable! Read more in the GridFTP: Sharing State section of the Globus Connect Server Initial Configuration page.

{% endcapture %} {% include panel.html parent="server" id="s16" scenario="How do I allow sharing when my users don't have home directories?" content=s16 %}

{% capture s17 %}

This is the unique ID of the Stanford Globus Statistics Gateway. See the next question for more information.

{% endcapture %} {% include panel.html parent="server" id="s17" scenario="What is [email protected] ?" content=s17 %}

{% capture s18 %}

The Stanford Globus Statistics Gateway is a program that is operated by SRCC, which collects information on transfers made using Globus. It is primarily meant for gathering usage statistics, but the information can also be used to get a picture of who is transferring data (alhtough, information on what is transferred is not collected). Granting access to this program is a condition for accessing Stanford's Globus subscription.

You need to give the Stanford Globus Statistics Gateway "Activity Monitor" access to your Globus Connect Server endpoints. This gives read-only access, both to your endpoint and to shared endpoints hosted by your endpoint.

{% endcapture %} {% include panel.html parent="server" id="s18" scenario="What is the Stanford Globus Statistics Gateway? Why do I need to give it access to my stuff?" content=s18 %}

{% capture s19 %}

This is the unique ID of the old Stanford Globus Statistics Gateway, which was active during the time this site was initially being developed. It has been retired, and you can remove its access from your endpoints.

{% endcapture %} {% include panel.html parent="server" id="s19" scenario="What is [email protected]?" content=s19 %}

{% capture s20 %}

This is a complex question, because there are lots of factors that influence transfer speed, and only some of those factors can be adjusted on your end; performance depends as much on the other end as it does on you. That being said, here are some things you should consider:

  • Dedicate a modern machine. This is especially impotant when encryption is used, because it will give you access to CPUs that are able to do AES encryption operations natively. Dedicating an entire machine also ensures that the system is not being distracted with other work.
  • Have a 10 GBps or better link. A single Globus Connect Server installation, set to aggressive network use, will happily saturate a 10 Gbps link.
  • Use a modern OS. You should use the latest-available version of your chosen distribution, whichever one includes long-term support. That includes the latest RHEL or CentOS, the latest Debian, and the latest LTS Ubuntu. Using the latest-available OS ensures that you have the best-available TCP congestion-control algorithms.
  • Place your endpoint outside the firewall. Even if the rest of your environment is firewalled, your endpoint's data-transfer links should be on a non-firewalled VLAN. Bypassing the network firewall improves performance, because the network firewall increases latency, as it has to screen each connection. Also, network firewalls may be limited to 10 or 20 Gbps connections—depending on firewall model—which has to be shared with all other network firewall users.

    Note that this does require careful work to maintain security: The links outside of the network firewalls must only be used for data-transfer traffic, not regular user logins. You must also containue to run a host-based firewall, to meet MinSec requirements.

  • Tune your host. Even with the latest OS, there is additional OS and hardware tuning that can be done to extract more performance. Note, however, that some performance adjustments will actually slow down lower-bandwidth connections (like home Internet connections).

    Read more on the ESnet Host Tuning page.

  • Connect as close to the core as possible. Ideally, your data transfer node should be in a Stanford data center, where it will be able to connect as close as possible to the network core. The next-best option would be to have it in a building room with switches that connect directly back to an ECH. Either way, it is best if your endpoint bypasses normal rack switches, as most rack switches only have a 10 or 20 Gbps link back to the core. One way of doing this is connecting your host to the Stanford Research Network.
  • Consider multiple servers. If all of the above options have been exhausted, and your back-end storage still has available throughput capacity, you can add another server to your existing Globus Connect Server endpoint. When Globus sees multiple servers for a single endpoint, it will spread transfer work across the multiple servers. Besides the performance increase, this gives you some redundancy, as you can take one server down for maintenance without taking the entire endpoint offline.

You should also be monitoring your endpoint. Beyond monitoring CPU, memory, and utilization of the data-transfer interface, you should also monitor the performance of the link to your storage. You can use the ESnet Sunnyvale endpoint to test transfer performance to your storage.

{% endcapture %} {% include panel.html parent="server" id="s20" scenario="How do I make transfers to/from my endpoints go faster?" content=s20 %}

Policies & Support

Open All 
{% capture p2 %}

At this time, Stanford pays for a Globus Standard subscription, and the following Premium Storage Connectors: Box, Google Cloud Storage, Google Drive, and S3.

We have not paid for the High Assurance—which would likely be required for transferring with High Risk data—or HIPAA BAA—which would be required for transferring PHI—levels.

{% endcapture %}

{% include panel.html parent="support" id="p2" scenario="What does Stanford pay for?" content=p2 %}

{% capture p1 %}

Anyone can use the free portions of Globus (such as Globus Connect Personal), regardless of Stanford affiliation.

For the parts of Globus that Stanford pays for, the license is "campus-wide". Some notable exclusions from "campus-wide" are the Carnegie Institution, SLAC, Stanford Children's Health, and Stanford Health Care (the adult hospital). For people affiliated with those entities.

It does not matter if you are a student, staff, postdoc, or whatever: If your group is not part of one of the excluded entities, you can take advantage of our campus subscription!

For full details, see the policies page.

{% endcapture %}

{% include panel.html parent="support" id="p1" scenario="Who can use this?" content=p1 %}

{% capture p3 %}

There is only one limit on the type of data: No High Risk data may be transferred using Globus. Other than that limit, there is no limit on the types of data you can transfer. (For example, some services—not Globus—have exclusions on administrative data. That is not the case here.)

There is one caveat: If you are working with Moderate Risk data, there are certain things you have to do as part of setting up Globus. For example, you have to make sure that encryption is required for all transfers, and that TLS 1.0 is disabled. The instructions on this site take all of this into account.

Also, Globus Search only supports Low Risk data. This is OK, because it is meant for searching public data sets.

There are absolutely no limits on the quantity of data that you may transfer under our subscription. The only limits are those that are naturally imposed by your storage space, and your network connection.

{% endcapture %}

{% include panel.html parent="support" id="p3" scenario="Are there any limits on the type or quantity of data that I can transfer?" content=p3 %}

{% capture p4 %}

Yes, you can! Check out our Cloud page to see what we have licensed.

{% endcapture %}

{% include panel.html parent="support" id="p4" scenario="Can I use this with Amazon S3, GCP, or other cloud services?" content=p4 %}

{% capture p5 %}

Your best starting point is the support page. This page has pointers to various common problems, places where you can get in-person help, and pointers to local support contacts. There are also several mailing lists that are run by Globus. Finally, if your issue is particularly weird, there are several people on-campus who have direct access to Globus support.

{% endcapture %}

{% include panel.html parent="support" id="p5" scenario="Where do I go for support?" content=p5 %}

{% capture p6 %}

Funding for Globus at Stanford comes from several sources.

The Standard Subscription is currently paid for by the Stanford Research Computing Center, a joint effort of the Dean of Research and University IT; and Stanford Libraries.

The funding for the S3 connector—supporting Amazon S3, Wasabi, and other S3-compatible platforms—comes from the Genomics Bioinformatice Service Center. The funding for the Box, Google Cloud, and Google Drive connectors comes from the Box Migration project, run by the Service Strategy unit within University IT.

{% endcapture %}

{% include panel.html parent="support" id="p6" scenario="Who pays for this?" content=p6 %}