Option (Mac) | Alt (Win) -clicking a preset on a panel (Brush Presets, Swatches, Tool Presets etc.), will delete the preset (the icon swaps to a pair of scissors).
In order to reorder presets, select Edit > Preset > Presets Manager. Under Preset Type, select Brushes, then drag and drop the brush icons into the desired order. Note: this shortcut also works with any of the preset lists in the Preset Manager (Swatches, Gradients, Styles etc.).