The late night host said the "alleged insiders" simply "don’t know what they’re talking about" when it comes to Stephen Colbert's program.