Hybrid optical communication is an attractive candidate for the wireless network in the near future which utilizes both wired and wireless communication. Optical communication meant for long haul communication has been intended for an eight channel Wavelength Division Multiplexing (WDM) system followed by a Free Space Optics (FSO) called wireless technology. In this paper, hybrid optical communication for an eight channel WDM is performed and different modulation format is analysed for the same link. Dispersion is the major effect present in the optical fiber which causes broadening of pulses and leads to Inter Symbol Interference (ISI). In this paper, dispersion is also compensated by utilizing Optical Phase Conjugation (OPC) which utilizes Four Wave Mixing (FWM) as the non-linear degrading effect. It has been found that the apposite modulation format for the transmitter part of the optical system is Modified Duo-binary Return-to-Zero (MDRZ) which is good for both linear as well as non-linear effects when compared it with Return-to-Zero (RZ), Non-Return-to-Zero (NRZ), Carrier Suppressed Return-to-Zero (CSRZ) and Duo-binary modulation format. The feat of the intended hybrid optical system is scrutinized in terms of Eye Opening, Bit Error Rate (BER), and Q-value etc. Better BER is achieved.