Have been doing something similar to this using image to image translation (XYZ rendered images to RGB space domain). Most of the information is contained in the Z-axis which gives you the height information, e.g. helps to distinguish the grass and buildings color. However I am skeptical if the X and Y is noise and how much spatial information it provides during Conv blocks. Anyone who had previous experience on this?
https://github.com/f0ti/thesis