I'd say, combo of things, already mentioned above. Wheels (maybe) but more likely the tires and maybe the bottom bracket. Try another wheelset that has different tires on it. I had Kenda Negevals and they suck. Most versions are thick heavy tires. Yea they stick but they are like riding a bike with thick strips of sorbathane wrapped around the wheels. It's not the weight of the tires, that's a small factor. It's the materials/design. Those tires just suck. I'm riding Tioga Psycho genius tires now, not light weight but they roll really well. Wheels can also just be loose. Squeeze a few spokes together and compare the tension to other wheelsets. A really loose wheelset will be VERY different.
You have already eliminated the hub bearings, as they spin down well, and heavy brake drag. So check that bottom bracket, it's easy to compare two bikes with the chains removed. The crank should spin without crunchy noises and smoothly.
If everything checks, and different wheelset makes no difference, I'd say sell it and move on, the geometry just isn't working for you.
Tzvia- rollin' slow...
Specialized Ruby Expert/mens Bontrager Inform RXL
Specialized SWorks Safire/mens Bontrager Inform RL
Giant Anthem-W XT-XTR/mens Bontrager Inform RXL
Fuji Newest 3 commuter/mens Bontrager Inform RL
Novara E.T.A commuter/mens Bontrager Inform RL