The 2-D median filter, one of the oldest and most well-established image-filtering techniques, still sees widespread use throughout computer vision. Despite its relative algorithmic simplicity, accelerating the 2-D median filter via a hardware implementation becomes increasingly challenging as the window size increases, since the resources required grow quadratically with the window size. Previous works, in a non-FPGA context, have shown that applying a sequence of multiple directional median filters to an image yields results that are competitive with, and in some cases even better than, those of a classic 2-D window median. Inspired by these approaches, we propose a novel way of substituting a 2-D median filter on an FPGA with a sequence of directional median filters, in our case in the pursuit of an FPGA implementation that achieves better scalability and hardware efficiency without sacrificing accuracy. We empirically show that the combination of three particular directional filters, in any order, achieves this, whilst requiring quadratically fewer resources on the FPGA. Our approach allows for much higher throughput and is easier to implement as a pipeline.