Yep, you probably should just leave that the way it is, as long as the chimney bricks are okay. The vent pipe would be good if you had to rebuild the chimney, because then you could just use wood siding.
I can foresee it being a bit difficult getting that damper box on top of the firebrick. You'll have to fix it up there, then install the last bricks with some mortar on top to seal it to the bricks.
You also have to go from the herring bone pattern to straight horizontal at some point, or do a lot of fancy cuts on the top.