You'll have troubles with the camera height map when correcting for unit z heights (walking up terrain) since the camera map uses a smoother version of the terrain map (so there is no jerkiness while panning). I did find a work-around though. If you generate your terrain heightmap with terrain deformations on load, the camera height map will be flat and it will not update to the terrain deformations.