I will describe our present understanding and current modelling of the internal structure and evolution of brown dwarfs and Extra-solar planets. I will also address the case of short-period, strongly irradiated planets and discuss some of the physical mechanisms which have been suggested to explain anomalously large observed radius. I will finally discuss the early evolution of planets versus brown dwarfs and I will critically analyse the suggestion that young planets should be fainter than young brown dwarfs.