A multi-value Info option is used to pass multiple values for one Info option into 'dsupdt'. At lease one value must follow each multi-value option.
(Alias: -Action), Action options set in update controls and local file records to execute 'dsupdt' for processing controls and 'dsarch' for archiving data, respectively.
sets the number of days or hours the remote files have to be aged on remote servers to be valid for download. If this value is set, 'dsupdt' tries to check timestamp of a data file on remote server. If the time difference between the current date/time and the data file date/time is shorter than the age time set in RDADB by this option, 'dsupdt' will not download this file.
external commands for additional processes to build local files instead of the standard way of taring/compressing.
Options allowed passing information into the command include -LF, -RF, -SF, -DS, -ED, -EH, -SN/LN, and -LI. Each individual option is replaced by a real value at the time 'dsupdt' calls the command.
If you send email inside your build command and want the email being held until the update action of 'dsupdt' is finished successfully, you can use option -LI to pass the local file index to you command, and save a customized email into field of dlupdt.emnote in the local file record. If you want the email being held until multiple local file records being updated successfully, you can pass to your command the index value of the last local file record. The saved customized email will be sent out only when the update action is completed successfully against all the included local file indices.
used for remote file records to specify beginning times. Combined with the time interval given per Info option -TI, it is used to set the beginning time of creating multiple remote file names based on the temporal pattern of the file name in each update period. It is defaulted to 0, which means the first available time during a update period, i.e., 00 hour for hourly controlled updates and the 1st of a month for monthly updates.
Beginning time can be set as D:D:D:D:D:D:D:D:D:D:D:D for 12 offset values for setting beginning days of each month in a year, if they are not the first days of each month of a monthly controlled update.
Info Option -CC (-CarbonCopy) :
sets additional one or multiple email addresses into update control records via Action -SC (-SetControl) to send Cc'd email notification of the update results. This option can also provided on command line to do the same thing. For DSS specialists, login user names themselves are acceptable; otherwise full email addresses are required for email domains other than 'ucar.edu'.
To Make A Carbon Copy Email To 'Schuster@Ucar.Edu' For Update results of 'ds337.0', provide Info option as '-CC schuster'
dsupdt | 337.0 UF -MU -IE -CC schuster |
update control indices for set update control records. A single Control index can be provided for updates of multiple local file records. Provide one Update Control index at a time for any update actions.
sets in RDADB for a local file record, and used later to clean the temporary data files during data update.
Info Option -CO (-ControlOffset) :
time offsets, i.e., 2D10H, 3H15N, work with control frequency to set next update control time. For a monthly control frequency 1M and the data update is finished for the current month 2011-09, without the control time offset, the update control time for next month is set to 2011-10-01 00:00.00. If an offset is 2D10H30N, update control time for next month is set to 2011-10-03 10:30.00 instead.
Info Option -CT (-ControlTime) :
update control times, in form of YYY-MM-DD HH:NN:SS, for update control records to be due for processing. The control times are reset to next update control periods according to both the control frequencies and control offsets after successful update processes in the current periods.
command to download a file on a remote server, copy a file locally somewhere else, or process data and generate a remote file. It can be set in both local and remote file records. This command is used for Action of download remote file, -DR (-DownloadRemote). A command provided on command line at execution is considered first; otherwise, the command specified in the remote file record, if exists, is used. If the command is not set in the remote file record either, the one set for the local file record is used.
turns on debug mode with specified information. This option provides upto 3 values, they are Debug Level, debug log file path and debug log file name. The debug level is mandatory for this option. It can be a single integer value, for example, 1000 means to log debug messages for debug levels 1 to 1000; or a range of values, for example, 200-1000 means to log debug messages from debug levels 200 to 1000. The default debug file path is '$DSSHOME}/dssdb/log' and the default debug file name is 'mydss.dbg'. Provides the second and third values for this option to override the default ones respectively.
for file descriptions of data files. Multiple lines are allowed for a description if it is passed in from an input file specified by Info option -IF (-InputFile).
Info Option -DO (-DownloadOrder) :
the download order indices for remote file records for situation that a single remote file is available on multiple remote servers. The location with download index of 0 is tried first. If missed, the second location, index 1, is tried and so on.
times, in form of YYY-MM-DD HH:NN:SS, for data being updated upto. They are used for dependencies of the control records. The dependencies of update controls are set via Info option -PI (-ParentIndex). If parent index of an update control record is not empty, the control record for an update period will be blocked until the data time of the parent control record shows it is done for that period.
Info Option -EC (-ErrorControl) :
a single letter option for error controls to set in update control records:
for data end date of next data update period.
for end hour of next data update period if hourly controlled.
The dependencies of local indices are set via Info option -PI (-ParentIndex). If parent index of a local file record is not empty, the local file record for end date, and hour if hourly update, will be blocked until the end data date and hour of the parent local file record shows it is done for that end date and hour.
specifies the end time of a given update period determined from the given update frequency per -FQ (-Frequency). Default value of 0 means that the end time should be the end of update period, i.e., for march of 2007 of a monthly update data record, the end time should be exactly '2007-03-31'.
used for remote file records to specify ending times. Combined with the time interval given per Info option -TI, it is used to set the ending time of creating multiple remote file names based on the temporal pattern of the file name in each update period. It is defaulted to 0, which means the last available time during a update period.
Ending time can be set as D:D:D:D:D:D:D:D:D:D:D:D for 12 offset values for setting ending days of each month in a year, if they are not the last days of each month of a monthly controlled update.
data update and control frequencies are time internals in format of NU, N - integer value and U - a single letter unit measure, i.e., 1Y, 1M, 1W, 5D, 6H. Frequency of fraction of a month is supported in format of NU/F. For example, 1M/3 means the update frequency as one third of a month, so that 10 days for the first two periods (1 - 10 and 11 - 20) of each month, and 8 to 11 days for the third period (21 - End Of Month) depending on how many days in a month.
specifies an archived file name for data on RDA Server. Archived file names are normally created automatically, but they can be specified otherwise per this option.
generic pattern option for
pattern values to be provided on command line. The first value is for
replacing pattern '
If 2 patterns are required for matching,
host machine names to set into update control records. One or multiple host names can be set into an update control record. The hostname information in update control records limits the associated update actions can, or cannot, be processed on the specified machines.
Set Update Control Record Of Control Index 1 Of Ds277.0 To control its associated update actions to be processed on bross and evans only
dsupdt | SC -CI 1 -HN bross:evans |
Info Option -HO (-HourOffset) :
works with Mode option -CN (-CheckNew) to provide a time zone offset value in hours to compare the timestamp of a server file to its local copy. For example, 2 means the server time zone is 2 hours ahead of the local time; while -2 means the server time zone is 2 hours behind the local time. It can be set into update control record via this Info option for action -SC (-SetControl).
Info Option -ID (-ControlID) :
a unique string to identify a Update control record.
Info Option -IF (-InputFile) :
for input file names; one or multiple file names may be given on command line. A input file name must starts with dataset number as in format of 'dsnnn.n.*' and the dataset number must match the dataset number given per option -DS (-Dataset). This restriction prevents specialists archiving data files accidentally into a wrong dataset. Input files are used to hold valid options and the associated values of Info options that need to be passed in for execution of 'dsupdt'.
In a input file, lines start with sign '#' are considered as comments; Option Names can be given either short, long or alias names. Action and Mode options are given in format of OptionName. Single value Assignment is given in format of OptionName<=>OptionValue. One option is given on each line. Different setting sign of Action and Mode options can be provided by Info option -AO (-ActOption, default to ); and different equal sign of single value assignment can be provided by Info option -ES, (-EqualSign, default to '<=>'). Multi-value assignments can be given in columns delimited with separator specified per option -SP (-Separator, default to '<:>'). It starts with a column title line for multi-value option names and the rest holds values corresponding to each column titles. The value information stops at the end of the file or when a new column name line or another single value assignment appears. If the last column is a multi-line value field, an additional separator must be appended for each line, including the column title line to end lines properly.
single letter options set in update control records for keeping files on local disks after data update finished:
local file names for set update local file records and are used later for building up local data files and archiving them onto RDA Server. If there is any temporal patterns in the given file names, the patterns are replaced by the times evaluated from the end date/hour of the update periods.
A different local file name can be provided at the update run time if a local file index is provided.
indices for set update local file records. Locfile index can be provided for update of individual local file record.
Info Option -MC (-EMailControl) :
a single letter option for email controls set in update control records:
Info Option -MR (-MissRemote) :
if set to 'Y', allows missing one of some remote files of the multiple remote files needed to build a single local file. Value 'Y' of this option means differently if the value of option -VI (-ValidInterval) is not empty; that download effort is not given up for the remote data files until they are out of the valid periods defined by the given valid intervals.
the data due interval between data end date/hour and the date/hour the data file file ready for update, which is set in local file record. The next update date/hour of a local file is evaluated by adding the due interval to the data end date/hour.
a string, upto 128 characters, passed to 'dsarch' for additional Info and Mode options to archiving data files. Group index or group can be passed to 'dsarch'. Temporal patterns can be setup into the group index or name values to dynamically identify group information for the data files being updated.
pattern delimiters, default to ["<", ">"],
are used to hold temporal or generic patterns in file names and any other
text fields. Normally the data end date/hour are used for the temporal
patterns, but current date/hour will be used if the pattern is both started
and ended with letter 'C', as in <CYYYY.MM.DD.HHC> for example. Beginning
date/hour of update period are used if the pattern is both started and ended
with letter 'B', as in <BYYYY.MM.DD.HHB> for example. If fractional month is
used for update frequency, pattern as
Generic patterns can be specified per option -GP (-GenericPattern) on command
line while execute 'dsupdt' for update actions. For a given local file name
as '
Info Option -PI (-ParentIndex) :
parent update control or local indices for data update dependencies. Set parent indices for update control records, if the control records need the parent controls be processed first; set parent local index for local file records, if the the local file records need the parent local files be build and archived first.
Info Option -PR (-ProcessRemote) :
external command specified by individual specialists for additional processes to validate and convert remote files after they are successfully downloaded.
Options allowed passing information into the command include -LF, -RF, -SF, -DS, -ED, -EH, -SN/LN, and -LI. Each individual option is replaced by a real value at the time 'dsupdt' calls the command.
If you send email inside your process command and want the email being held until the update action of 'dsupdt' is finished successfully, you can use option -LI to pass the local file index to you command, and save a customized email into field of dlupdt.emnote in the local file record. If you want the email being held until multiple local file records being updated successfully, you can pass to your command the local file index value of the last local file record. The saved customized email will be sent out only the update action is completed successfully for all local file records.
(Alias: -PBSOptions), specifies options to execute dsupdt as a batch job via qsub on PBS nodes. The qsub options must be quoted when prsented on command line, such as, -QS '-l walltime=12:00:00'.
Info Option -RF (-RemoteFile) :
for remote file names to set update remote file records and are used later for download remote data files and building local files. If there is any temporal patterns in the given file names, the patterns are replaced by the times evaluated from the end date/hour of the update periods. Local file names are used if remote file names are not specified.
For simple situation that a remote file is the same as local file, information of the local file record is enough to process data update. A remote file update record is only required if the remote file is different from local file, including situations that multiple remote files are downloaded and tarred for a single local file, or a single remote file is available from multiple servers.
Multiple remote files can be provided in a single remote file record by joining them with separator '::', as Rfile1::Rfile2::Rfile3 for example.
For remote file name with a leading '!' it is treated as an executable command and its returned string is used as a dynamically created remote file name.
Info Option -RI (-RetryInterval) :
set the number of days or hours to retry the update control process when first try failed.
when present, resets the execute orders of the given local file lists as the orders the update records are given for actions -SL (-SetLocalFile). Another way of reordering the local files is to provide explicitly order index values per Info option -XO (-ExecOrder).
(Alias: -SlurmOptions), specifies options to execute dsupdt as a batch job via sbatch on SLURM nodes. The sbatch options must be quoted when prsented on command line, such as, -QS '-t 24:00:00'.
Info Option -SF (-ServerFile) :
only used to set file names on Remote servers when they are different from the remote file names.
sets login names of specialists into RDADB for update records. It defaults to the login name of a specialist if not proved explicitly. At run time of a data update, specialist login name is validated against saved specialist names and the update process is blocked if them are not match.
used for remote file records to specify a time increase step. The interval is used to create multiple remote file names based on the temporal pattern of the file name in each update period; starting from the beginning time given per Info option -BT (-BeginTime) to the data end date/hour.
Info Option -UC (-UpdateControl) :
a string to hold multiple single letter options setting in update control records:
Info Option -VI (-ValidInterval) :
sets the number of days or hours the remote files are valid on remote servers. If the time difference between the current date/time and the data date/time is longer than the time internal set in RDADB by this option, 'dsupdt' will not download.
specifies a working directory to hold the temporary data files for the download/archive activities. The leading path of the working directory can be set as an environment variable $UPDTWKP. It is defaulted to '/glade/data02/dsswork' on machines that can write to /glade/data02. A working directory '$UPDTWKP/zji/icoads' means that the temporary downloaded data files are staged in '/glade/data02/dsswork/zji/icoads' during the update. Specify it differently or use a different environment variable name if you want to store the temporary data files some where else. Be aware that the default value for $UPDTWKP is used if you use a different environment variable name without defining it.
external command for additional process after successful update action for a specified update control configuration.
for executing order indices of local file records of a given a dataset. This Info option is ignored if Mode option -RO (-ResetOrder) is present.