FYI, not all buildings have height information. I would think it’s only a small portion of the buildings that have a height specified. Perhaps building:levels is more common.
Can’t help you with the best format. I just read that the free shape files from Geofabrik do not contain all information, but there paid versions contain more.
If you only need a small area, you could download the buildings via e.g. OverpassTurbo.
Otherwise a planet dump, or derived country dump is probably the way to go.