Forum Moderators: open
I would say that having videos on your site is not the cause of your Google woes.
The original question was about optimising video, and was answered well by Bill.
The questioner then asked "how to optimize or rank a site having videos?".
The answer is "the same way you optimise any site for ranking".
Imagine for one moment that the question was "how to optimize or rank a site having images" and you will get the idea - in the same way that you would use descriptive ALT tags for images you can use descriptive text in the embedding markup for the video, but other aspects of the site markup (such as titles, descriptions and text) will still need to be good enough to interest search engines.
Unless, like YouTube, you have a gazillion incoming links...