I ain't no expert, but I wonder: "fitness for what?"
Right now I'm pretty "running fit" but I can tell you that going back on the bike in March/April (when I will be at the top of my running fitness) will be painful, and I will not be climbing those hills the way I climbed them at the crest of my cycling fitness last summer... In "fitness" there's "fit," as in "fitting (with something)," in my view.
I would probably get fatigued long before a trained "fitness walker" if we went walking together for a long time...
Now there might be a few "universal" markers of fitness, one of them being feeling energetic and happy. Then any aerobic sport can help towards that!