.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "examples/suvi/get_suvi_data_from_nodd.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_examples_suvi_get_suvi_data_from_nodd.py: Get SUVI data from NODD at a specific cadence ============================================= Purpose: Download SUVI L1b data at a specific cadence from the NOAA Open Data Dissemination (`NODD `__) AWS S3 buckets using the boto3 library. .. GENERATED FROM PYTHON SOURCE LINES 9-19 The NODD AWS S3 buckets contain almost all SUVI L1b data for GOES-16 through GOES-19 (some early mission data is not available). The data is publicly available at s3://noaa-goes19/ (replace the 19 with 16, 17, or 18 for the earlier missions). The script below is intended to download data in a specific wavelength channel at a requested cadence. For example, using this script, you could download a SUVI 171 Å file from GOES-19 at a cadence of one hour, instead of having to download all data on the S3 bucket and discarding what you don't need. How this script operates: the code will create reference timestamps in the requested period and at the requested cadence, and then look for the file that is closest to each reference timestamp. If a file is found within the defined tolerance, it is accepted, otherwise rejected. .. GENERATED FROM PYTHON SOURCE LINES 21-23 Let's first import the necessary Python libraries. The "UNSIGNED" and "Config" from botocore are imported so we don't need a specific username/token for the NODD S3 buckets. .. GENERATED FROM PYTHON SOURCE LINES 23-31 .. code-block:: Python __author__ = "cbethge" import datetime import numpy as np import os import boto3 from botocore import UNSIGNED from botocore.config import Config .. GENERATED FROM PYTHON SOURCE LINES 32-34 Now we define the start and end of the period that the code is supposed to consider. The format for those dates is YYYY-MM-DDThh:mm:ss (ISOT without the milliseconds). .. GENERATED FROM PYTHON SOURCE LINES 34-37 .. code-block:: Python start_date = '2026-05-15T12:20:00' end_date = '2026-05-15T23:30:00' .. GENERATED FROM PYTHON SOURCE LINES 38-43 .. note:: The code has to parse *all* files in the requested period! A period of about one month can be handled, but it will be fairly slow. Anything beyond about one month should be broken up into one-month chunks, e.g. with an enclosing for-loop. .. GENERATED FROM PYTHON SOURCE LINES 45-48 Here are some pretty self-explanatory settings: which satellite to get the data from (16, 17, 18, or 19), which wavelength (94, 131, 171, 195, 284, or 304), which file ending (.fits for FITS files or .nc for netCDF), where to download the data to, the cadence in minutes, and the tolerance in minutes. .. GENERATED FROM PYTHON SOURCE LINES 48-55 .. code-block:: Python satellite = 19 # which GOES satellite (16, 17, 18, 19) wavelength = 171 # which SUVI wavelength channel (94, 131, 171, 195, 284, 304) file_ending = '.fits' # which file type (.fits or .nc) outpath = './' # where to download the data cadence = 60. # requested cadence in minutes tolerance = 4. # tolerance in minutes .. GENERATED FROM PYTHON SOURCE LINES 56-72 How the tolerance works: in this example script, we requested 171 Å data from GOES-19 at a cadence of one hour, starting at '2026-05-15T12:20:00'. The reference timestamps created in this case would be: | 2026-05-15T12:20:00 | 2026-05-15T13:20:00 | 2026-05-15T14:20:00 | 2026-05-15T15:20:00 | | and so on. | | Let's say the closest file to the second reference timestamp is | "OR_SUVI-L1b-Fe171_G19_s20261351318239_e20261351318249_c20261351318474.fits". | The start time for this file is "s20261351318239", or "2026-05-15T13:18:23.9" in ISOT format. In | other words, the difference to the respective reference timestamp is: minus 1 minute, 36 seconds, | and 1 millisecond. Since the tolerance was set to 4 minutes, this file would be accepted. If the | tolerance had been set to 1 minute, the file would be rejected. .. GENERATED FROM PYTHON SOURCE LINES 74-76 Now on to some more settings, they are all Boolean (True or False). The first one is: .. GENERATED FROM PYTHON SOURCE LINES 76-77 .. code-block:: Python enforce_requested_period = False .. GENERATED FROM PYTHON SOURCE LINES 78-85 | This is best explained with another example. | The file closest to the first reference timestamp mentioned above is | "OR_SUVI-L1b-Fe171_G19_s20261351218237_e20261351218247_c20261351218479.fits". | The timestamp for this file is "s20261351218237", or "2026-05-15T12:18:23.7" in ISOT format. | Note that this time is *before* the requested period. If enforce_requested_period is set to False, the | the file would be accepted anyway. If it is set to True, all files outside of the requested period are | rejected. .. GENERATED FROM PYTHON SOURCE LINES 87-88 The second True/False setting is: .. GENERATED FROM PYTHON SOURCE LINES 88-89 .. code-block:: Python short_exp = False .. GENERATED FROM PYTHON SOURCE LINES 90-95 By default, the code will download SUVI L1b long exposures. If short_exp is set to True, short exposures will be downloaded instead. Note that for the 94 and 131 Å channels, the code cannot distinguish between SUVI short exposures and SUVI short flare exposures based on their filenames, so it will download whichever of the two is closest in time to the reference timestamp in this case! .. GENERATED FROM PYTHON SOURCE LINES 97-98 Moving on with the True/False settings: .. GENERATED FROM PYTHON SOURCE LINES 98-99 .. code-block:: Python no_sub_dirs = False .. GENERATED FROM PYTHON SOURCE LINES 100-103 By default, the code will create subdirectories for satellite/year/month/day/wavelength/. If no_sub_dirs is set to True, no subdirectories will be created and the files are downloaded directly to the "outpath" folder defined above. .. GENERATED FROM PYTHON SOURCE LINES 105-106 And finally: .. GENERATED FROM PYTHON SOURCE LINES 106-107 .. code-block:: Python dryrun = True .. GENERATED FROM PYTHON SOURCE LINES 108-110 If dryrun is set to True, the filenames of the accepted files will only be printed out, but the files will not be downloaded. This is the default setting for this website example script. .. GENERATED FROM PYTHON SOURCE LINES 112-114 Now that we are done with the settings, we do some really basic sanity checks. First, we define a function to make sure the submitted dates are in the right format: .. GENERATED FROM PYTHON SOURCE LINES 114-126 .. code-block:: Python def is_valid_date_string(date_string): # Defining the format from above for datetime date_format = "%Y-%m-%dT%H:%M:%S" try: # Attempt to parse the string datetime.datetime.strptime(date_string, date_format) return True except ValueError: # If a ValueError is raised, the format or the date itself is invalid return False .. GENERATED FROM PYTHON SOURCE LINES 127-128 Check that the settings from above are valid inputs: .. GENERATED FROM PYTHON SOURCE LINES 128-171 .. code-block:: Python valid_satellites = [16, 17, 18, 19] assert satellite in valid_satellites, f'satellite must be in {valid_satellites}' valid_wavelengths = [94, 131, 171, 195, 284, 304] assert wavelength in valid_wavelengths, f'wavelength must be in {valid_wavelengths}' assert is_valid_date_string(start_date), 'Start date must have the form YYYY-MM-DDThh:mm:ss' syear = int(start_date[0:4]) smonth = int(start_date[5:7]) sday = int(start_date[8:10]) shour = int(start_date[11:13]) sminute = int(start_date[14:16]) ssecond = int(start_date[17:19]) assert syear >= 2019, 'Start year must be >= 2017.' assert 1 <= smonth <= 12, 'Start month must be between 1 and 12.' assert 1 <= sday <= 31, 'Start day must be between 1 and 31.' assert 0 <= shour <= 23, 'Start hour must be between 0 and 23.' assert 0 <= sminute <= 59, 'Start minute must be between 0 and 59.' assert 0 <= ssecond <= 59, 'Start second must be between 0 and 59.' start_date = datetime.datetime.fromisoformat(start_date).replace(tzinfo=datetime.timezone.utc) assert is_valid_date_string(end_date), 'End date must have the form YYYY-MM-DDThh:mm:ss' eyear = int(end_date[0:4]) emonth = int(end_date[5:7]) eday = int(end_date[8:10]) ehour = int(end_date[11:13]) eminute = int(end_date[14:16]) esecond = int(end_date[17:19]) assert eyear >= 2017, 'End year must be >= 2017.' assert 1 <= emonth <= 12, 'End month must be between 1 and 12.' assert 1 <= eday <= 31, 'End day must be between 1 and 31.' assert 0 <= ehour <= 23, 'End hour must be between 0 and 23.' assert 0 <= eminute <= 59, 'End minute must be between 0 and 59.' assert 0 <= esecond <= 59, 'End second must be between 0 and 59.' end_date = datetime.datetime.fromisoformat(end_date).replace(tzinfo=datetime.timezone.utc) assert (end_date-start_date).total_seconds() > 0, 'End date must be after start date.' .. GENERATED FROM PYTHON SOURCE LINES 172-173 The main algorithm: .. GENERATED FROM PYTHON SOURCE LINES 173-310 .. code-block:: Python # Start processing... satellite_name = 'G'+str(satellite) satellite_name_long = 'goes'+str(satellite) # Initialize connection with S3 bucket suvi_session = boto3.session.Session() s3 = boto3.resource('s3', config=Config(signature_version=UNSIGNED)) suvi_bucket = s3.Bucket('noaa-'+satellite_name_long) # These are the SUVI wavelength subfolders on the NODD S3 buckets wl_folders = { 94: 'SUVI-L1b-Fe093', 131: 'SUVI-L1b-Fe131', 171: 'SUVI-L1b-Fe171', 195: 'SUVI-L1b-Fe195', 284: 'SUVI-L1b-Fe284', 304: 'SUVI-L1b-He303'} # We have to parse through all days being considered to make a list of all # available files on the S3 print('Making list of files on S3 bucket...') tmp_valid_files = [] current_date = start_date while current_date <= end_date: date = current_date.strftime("%Y%m%d") day_of_year = current_date.strftime('%j') year = int(date[0:4]) month = int(date[4:6]) day = int(date[6:8]) print('Checking', date, 'on S3 bucket...') # Check if there is data on the server for the requested day. # If not, skip this day. number_of_obj = 0 # Need to loop through the hours for the NODD S3 buckets for h in range(0,24): hstr = '{:02d}'.format(h) this_prefix = wl_folders[wavelength]+'/'+'{:04d}'.format(year)+'/'+day_of_year+'/'+hstr+'/' for obj in suvi_bucket.objects.filter(Prefix=this_prefix).all(): if obj.key.endswith(file_ending): number_of_obj += 1 tmp_valid_files.append(obj.key) if number_of_obj == 0: warn_msg = f'No files found on S3 bucket for {date}. Skipping...' print(warn_msg) current_date = current_date + datetime.timedelta(days=1) continue current_date = current_date + datetime.timedelta(days=1) # Get the start and end times from the filenames on the S3 bucket valid_files = [] if len(tmp_valid_files) == 0: err_msg = 'No valid files could be selected.' raise Exception(err_msg) else: print('Found', str(len(tmp_valid_files)), 'valid files on the S3.') tmp_start_times = [] tmp_end_times = [] for tmp_file in tmp_valid_files: file_base = os.path.basename(tmp_file) file_base_split = file_base.split('_') stime = file_base_split[3][1:] etime = file_base_split[4][1:] tmp_start_time = stime[0:4]+':'+stime[4:7]+':'+stime[7:9]+':'+stime[9:11]+':'+stime[11:13] tmp_end_time = etime[0:4]+':'+etime[4:7]+':'+etime[7:9]+':'+etime[9:11]+':'+etime[11:13] # Exclude files outside of the requested period if requested: if enforce_requested_period: if ((datetime.datetime.strptime(tmp_start_time, "%Y:%j:%H:%M:%S").replace(tzinfo=datetime.timezone.utc)-start_date).total_seconds() >= 0) and \ ((datetime.datetime.strptime(tmp_end_time, "%Y:%j:%H:%M:%S").replace(tzinfo=datetime.timezone.utc)-end_date).total_seconds() <= 0): valid_files.append(tmp_file) tmp_start_times.append(datetime.datetime.strptime(tmp_start_time, "%Y:%j:%H:%M:%S").replace(tzinfo=datetime.timezone.utc)) tmp_end_times.append(datetime.datetime.strptime(tmp_end_time, "%Y:%j:%H:%M:%S").replace(tzinfo=datetime.timezone.utc)) else: valid_files.append(tmp_file) tmp_start_times.append(datetime.datetime.strptime(tmp_start_time, "%Y:%j:%H:%M:%S").replace(tzinfo=datetime.timezone.utc)) tmp_end_times.append(datetime.datetime.strptime(tmp_end_time, "%Y:%j:%H:%M:%S").replace(tzinfo=datetime.timezone.utc)) if len(valid_files) == 0: err_msg = 'No valid files found on the NODD S3 buckets.' raise Exception(err_msg) # Make lists of start times, end times, and exposure times start_time = [] end_time = [] exposure_time = [] for counter, tmp_start_time in enumerate(tmp_start_times): start_time.append(tmp_start_time) end_time.append(tmp_end_times[counter]) exposure_time.append((tmp_end_times[counter]-tmp_start_time).total_seconds()) exposure_time = np.array(exposure_time) # Select long or short exposures, depending on what was requested if short_exp: chosen_exposures = np.where(np.around(exposure_time) == 0.) else: chosen_exposures = np.where(np.around(exposure_time) == 1.) chosen_exposure_files = list(np.array(valid_files)[chosen_exposures]) # Make the reference timestamps at the requested cadence reference_timestamps = [] current_timestamp = start_date while current_timestamp <= end_date: reference_timestamps.append(current_timestamp) current_timestamp = current_timestamp + datetime.timedelta(minutes=cadence) selected_files = [] for ttime in reference_timestamps: delta_t = np.zeros(len(chosen_exposures[0])) for i in range(0,len(chosen_exposures[0])): delta_t[i] = (ttime-start_time[chosen_exposures[0][i]]).total_seconds() # Only accept files within the given tolerance: if (np.abs(delta_t).min()/60.) <= tolerance: which_file = np.abs(delta_t).argmin() # Avoid adding the same file multiple times if chosen_exposure_files[which_file] not in selected_files: selected_files.append(chosen_exposure_files[which_file]) if len(selected_files) == 0: err_msg = 'No valid files could be selected.' raise Exception(err_msg) else: if no_sub_dirs: store_loc = outpath else: store_loc = outpath+satellite_name_long+'/'+date[0:4]+'/'+date[4:6]+'/'+'/'+date[6:8]+'/'+str(wavelength)+'/' if not dryrun: # Create folders for downloaded data if they do not exist if not os.path.exists(store_loc): os.makedirs(store_loc) for sf in selected_files: print('Downloading', os.path.basename(sf)) if not dryrun: suvi_bucket.download_file(sf, store_loc+os.path.basename(sf)) .. rst-class:: sphx-glr-script-out .. code-block:: none Making list of files on S3 bucket... Checking 20260515 on S3 bucket... Found 720 valid files on the S3. Downloading OR_SUVI-L1b-Fe171_G19_s20261351218237_e20261351218247_c20261351218479.fits Downloading OR_SUVI-L1b-Fe171_G19_s20261351318239_e20261351318249_c20261351318474.fits Downloading OR_SUVI-L1b-Fe171_G19_s20261351418240_e20261351418250_c20261351418478.fits Downloading OR_SUVI-L1b-Fe171_G19_s20261351518241_e20261351518251_c20261351518479.fits Downloading OR_SUVI-L1b-Fe171_G19_s20261351618243_e20261351618253_c20261351618472.fits Downloading OR_SUVI-L1b-Fe171_G19_s20261351718244_e20261351718254_c20261351718482.fits Downloading OR_SUVI-L1b-Fe171_G19_s20261351818245_e20261351818255_c20261351818474.fits Downloading OR_SUVI-L1b-Fe171_G19_s20261351918246_e20261351918256_c20261351918485.fits Downloading OR_SUVI-L1b-Fe171_G19_s20261352018248_e20261352018258_c20261352018476.fits Downloading OR_SUVI-L1b-Fe171_G19_s20261352118249_e20261352118259_c20261352118477.fits Downloading OR_SUVI-L1b-Fe171_G19_s20261352218250_e20261352218260_c20261352218475.fits Downloading OR_SUVI-L1b-Fe171_G19_s20261352318252_e20261352318262_c20261352318484.fits .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 1.611 seconds) .. _sphx_glr_download_examples_suvi_get_suvi_data_from_nodd.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: get_suvi_data_from_nodd.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: get_suvi_data_from_nodd.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: get_suvi_data_from_nodd.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_