3.3.1 - Update Data Files

      Action Option -UF (-UpdateFile) :

      comprehensive action to download/copy server files, validate/convert the server files into remote files, build up local files from available remote files, archive local files onto RDA server, and clean the temporary data files generated during the update procedure.

      dsupdt[[-(DS|dataset)] dsnnn.n] -(UF|UpdateFile) [Mode Options]
        [-(CI|ControlIndex) UpdateControlIndex]
        [-(LI|LocalIndex) LocalFileIndices]
        [-(LF|LocalFile) LocalFileNames]
        [-(XO|ExecOrder) ExecOrderIndex]
        [-(RF|RemoteFile) RemoteFileNames]
        [-(SF|ServerFile) ServerFileNames]
        [-(DO|DownloadOrder) DownloadOrderIndices]
        [-(DC|DownloadCommand) DownloadCommand]
        [-(ED|EndDate) NextDataEndDate]
        [-(EH|EndHour) NextDataEndHour]
        [-(GP|GenericPattern) GenericPatterns]
        [-(WD|WorkDir) WorkingDirectory]
        [-(MR|MissRemote) AllowMissRemoteFile]
        [-(FQ|Frequency) DataFrequency]
        [-(PR|ProcessRemote) AdditionalRemoteFilePrecess]
        [-(BC|BuildCommand) CommandBuildLocalFile]
        [-(CD|CurrentDate) CurrentDate]
        [-(CH|CurrentHour) CurrentHour]
        [-(VS|ValidSize) MinSizeForValidFile]
        [-(PL|ProcessLimit) MaxNumberOfChildProcesses]
        [-(HO|HourOffset) TimeZoneHourOffset]
        [-(SB|SbatchOptions) SlurmBatchOptions]
        [-(QS|QsubOptions) PBSBatchOptions]
        [-(BP|BatchProcess) [BatchProcessHosts]]
        [-(CC|CarbonCopy) Cc'dEmailAddresses]
        [-(DB|Debug) DebugModeInfo]

      Mode options that can be specified for updating file Action:

      Dataset number is provided per Info option -DS (-Dataset) to process data update of a single dataset. Some Info options as listed can be provided at the update running time to identify specified update records and others to override the update information saved in RDADB. Two special Info options -CD (-CurrentDate) and -CH (-CurrentHour) can only be proved at run time to specify current date/hour rather than the default current time.

      Multiple update periods may be processed if more than one update frequency periods exist between the data end date/hour and the current date/hour and the Mode option -MU (-MultipleUpdate) is present at the run time of update actions.

      As mentioned in the introduction section, an individual data file update is interrupted if any error encountered during the update process. Error of download a remote file is, however, allowed if value 'Y' is set for the update record in RDADB via option -MR (-MissRemote) for situation that a local file is built from multiple remote files. That means a partial update is allowed. The partial update behaviors differently if a valid period is set for a update record in RDADB via option -VI (-ValidInterval). The partial update will be put on hold until the the remote data file are still not available beyond the valid period.

      If Info Option -PL is present with a value larger than 1, the update actions of 'dsupdt' forks multiple child processes to process one update record in each process. This helps dataset update with many time consuming independent update records.

      Additional one or multiple email addresses for notification of update result can be specified by the Info option -CC (-CarbonCopy). For DSS specialist, login user names themselves are acceptable, '-CC zji schuster' for example.

      This action, -UF (-UpdateFile) can be executed on command line or per cron jobs to update files configured in the local file records on a specified computer. The local file records can also be linked to update control record for this action to be executed via a centralized daemon 'dscheck'. Instead of adding Mode options at run time for complicated update actions against local file records, most of the Mode options can be pre-configured into update control records.

      Other than this comprehensive update action, simple actions, -DR (-DownloadRemote), -BL (-BuildLocal), -AF(-ArchiveFile) or -CF (-CleanFile) can be used to do step by step for a download/archive procedure. Their command line syntaxes are similar to Action -UF(-UpdateFile) and are described following.



      Back to Top
      -(BG|BackGround)background process to turn off screen display for both
      -(CN|CheckNewer)for server file is locally, checks if the file is changed
      -(CP|CurrentPeriod)allows update of the end date/hour beyond the current
      -(EE-ErrorEmail)sends email only when error happens during file updates
      -(FU|ForceUpdate)if present, force update at lease for one end data/time,
      -(HU|HourlyUpdate)forces update of time to hours after successful file
      -(IE|IgnoreError)works with Mode option -MU (-MultipleUpdate) to skip
      -(KR|KeepRemote)keep remote file on local disk by copying it to local
      -(KS|KeepServer)keep server file on local disk by copying it to remove
      -(LO|LogOn)turn detail logging on if option -PL is present
      -(MO|MissedOnly)update data file only if it is not archived yet
      -(MU|MultipleUpdate)allows multiple updates if multiple data update
      -(NE|NoEmail)does not send email to the specialist after update
      -(NY|NoLeapYear)skips February 29 for leap years
      -(QE|QuitError)quits processing update of a dataset when an error
      -(RA|RetryArchive)force archive of local files by passing option -RA
      -(RD|RetryDownload)retry download the remote file if it is local already
      -(RE|ResetEndTime)if present, reset end datae/hour according to the local file
      -(SE|SummaryEmail)send a summary email to the specialist after update
      -(UB|UseBeginTime)when present, use the beginning time of update period
      -UT|UpdateTime)forces to update the data end time and next due or
      -UT|UpdateTime)forces to update the data end time and next due or update times