RDA News

Accessing RDA OPeNDAP endpoints with authentication

March 12, 2019
Posted by: Doug Schuster

Note: This page was originally sourced from our Blogger page: http://ncarrda.blogspot.com/2017/07/accessing-rda-opendap-endpoints-with.html

Update:
As an alternative to the steps below, it may be simpler specify the username and password within the URL to satisfy the authentication.

For example,
https://USERNAME%40DOMAIN:PASSWORD@rda.ucar.edu/thredds/dodsC/files/g/ds083.3/2016/201602/gdas1.fnl0p25.2016020218.f00.grib2'

For me, it might look like,

https://rpconroy%40ucar.edu:MySuperSecretPassword@rda.ucar.edu/thredds/dodsC/files/g/ds083.3/2016/201602/gdas1.fnl0p25.2016020218.f00.grib2'

Note the %40 in place of the '@' in the email.


All RDA OPeNDAP supported datasets found under https://rda.ucar.edu/thredds now require user authentication for access.  Here are details on how to configure your system, such that most applications including NCL, will work properly with the authentication step (these instructions are based on what is provided by the hdfeos group):

1. RDA Registration and Setting Up Cookies

NCAR RDA OPeNDAP server access requires user registration and cookies. If you do not have RDA Login, please register first and have RDA username and password ready.

Once you have RDA Login username and password, and verify that they work by logging into https://rda.ucar.edu, you need to set up cookies to access data.  We highlight the key steps here.

To set up cookies properly, you will need the following 3 files in your home directory (e.g., /home/rdahelp on Linux or /Users/rdahelp on Mac assuming that rdahelp is your UNIX system's login name).
  1. /home/rdahelp/.netrc
  2. /home/rdahelp/.rda_cookies
  3. /home/rdahelp/.dodsrc
1.1 CREATE .NETRC FILE FOR LOGIN / PASSWORD
The first file .netrc should have RDA login and password. For example, if your username is rdahelp and password is 1234abcd, the file should have the following line.
machine rda.ucar.edu login rdahelp password 1234abcd
Please edit the above file to match your username and password. Since the file contains the password, make sure that others cannot see it by changing permission.
$chmod go-rwx /home/rdahelp/.netrc
1.2 CREATE .RDA_COOKIES FILE USING CURL OR WGET
The second file, .rda_cookies, should be created automatically by either wget or curl command.
If you like to use wget, issue the following command to create the cookie file. 
$wget --load-cookies ~/.rda_cookies --save-cookies ~/.rda_cookies --auth-no-challenge=on --keep-session-cookies https://rda.ucar.edu/thredds/dodsC/files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1.dds -O test.dds
If you like to use curl, issue the following command to create the cookie file. 
$curl -n -c ~/.rda_cookies -b ~/.rda_cookies -L -g --url https://rda.ucar.edu/thredds/dodsC/files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1.dds -o test.dods
If the above command succeeds, you will get the following output when you check the content of test.dods file.
$more test.dods
Dataset {                                                        Int32 LatLon_Projection;                                    Float32 lat[lat = 181];                                          Float32 lon[lon = 360];                                                         ... }files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1
If you don't see the above output, your login/password and system is not working properly with NCAR RDA OPeNDAP server. You cannot access data until this step works. If you get the same output, you can delete the temporary output test.dods file.
1.3 CREATE .DODSRC FILE FOR .NETRC AND .RDA_COOKIES FILES
The final step is to create .dodsrc file if you don't already have one and add the following lines.
HTTP.COOKIEJAR=/home/rdahelp/.rda_cookies
HTTP.NETRC=/home/rdahelp/.netrc

2. Test NCL open and read of OPeNDAP data (NCL version 6.4.0 is required)

The following example shows an example of how to test if NCL can open and read an OPeNDAP file after the above steps have been completed.
$ ncl                                                             ncl 0> f = addfile("https://rda.ucar.edu/thredds/dodsC/files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1","r")                     ncl 1> print(f)                                               Variable: f                                                       Type: file                                                         filename:    fnl_20000201_06_00                                   path: https://rda.ucar.edu/thredds/dodsC/files/g/ds083.2/grib1/2000/2000.02/fnl_20000201_06_00.grib1                                         file global attributes:                                              Originating_or_generating_Center : US National Weather Service, National Centres for Environmental Prediction (NCEP)          Originating_or_generating_Subcenter : 0                            GRIB_table_version : 0,2                                          ... etc                         
In the above code, the url points to the remote FNL file being served by OPeNDAP. Use print command to check if the url file is opened successfully and to see what variables are available inside the file.