Python Data Selection Function: Criteria Based on Zero Value Percentage

This function selects data from specific directories based on certain criteria. The function takes no input parameters.

The function first sets the paths to the directories containing 'dBZ', 'KDP', and 'ZDR' data. It also sets the path to the file where the selected data will be written.

def select_data():
    dbz_dir_path = 'data/NJU_CPOL_update2308/dBZ/3.0km/'
    kdp_dir_path = 'data/NJU_CPOL_update2308/KDP/3.0km/'
    zdr_dir_path = 'data/NJU_CPOL_update2308/ZDR/3.0km/'
    data_select = 'data/select.txt'
    data_select_file = open(data_select, mode='a')
    dir_list = os.listdir(dbz_dir_path)
    for dir in dir_list:
        dbz_data_path = dbz_dir_path + dir
        dbz_list = os.listdir(dbz_data_path)
        i = 0
        data_list = []
        for i in range(len(dbz_list)):
            data_list.append(np.load(dbz_data_path + '/' + dbz_list[i]))
            if (len(dbz_list) - i) < (20 - len(data_list)):
                break
            if len(data_list) == 20:
                data_list = np.array(data_list)
                if (np.sum(data_list == 0)) < 0.8 * 20 * 256 * 256:
                    # print(np.sum(data_list==0))
                    data_select_file.write('
' + '3.0km' + ',' + dir + ',' + dbz_list[i + 1 - 20])
                    print('dir:', dir, 'frame:', dbz_list[i + 1 - 20])
                    data_list = []
                    continue
                else:
                    i = i - 19
                    data_list = []
                    continue

The function then opens the data_select_file in 'append' mode.

Next, the function retrieves a list of directories within the dBZ directory. For each directory, the function retrieves a list of files within that directory.

The function initializes a counter variable i to 0, and creates an empty list called data_list.

The function then loops through the files in the current directory and appends the loaded data to the data_list. If the difference between the total number of files and the current index is less than the desired number of frames (20) minus the current size of the data_list, the loop is broken.

If the size of the data_list reaches 20, it is converted to a numpy array. If the percentage of zero values in the array is less than 80%, the data is considered selected. The details of the selected data (directory name and frame name) are written to the data_select_file. The selected data is then printed.

If the percentage of zero values in the array is greater than or equal to 80%, the counter variable i is decreased by 19 (to move back to the start of the current frame) and the data_list is cleared.

The function continues this process until all directories and files have been processed.