Biến đổi:
$\cos^2 x=(a\sin x+b\cos x)(\sin x+\sqrt{3}\cos x)+c(\sin^2x+\cos^2x )$
$=(a+c)\sin^2x+(a \sqrt{3}+b)\sin x\cos x+(b \sqrt{3}+c)\cos ^2x $
Đồng nhất đẳng thứa ta có:
$\begin{cases}a+c=0 \\ a \sqrt{3}+b =0 \\b \sqrt{3}+c=1 \end{cases} \Leftrightarrow \begin{cases}a=-\frac{1}{4} \\ b=\frac{\sqrt{3} }{4}\\c=\frac{1}{4} \end{cases} $
Khi đó:
$I=\int\limits \frac{(-\frac{1}{4}\sin x+\frac{\sqrt{3} }{4} \cos x)(\sin x+\sqrt{3} \cos x)+\frac{1}{4} }{\sin x+\sqrt{3}\cos x } dx $
$=\int\limits (-\frac{1}{4}\sin x+ \frac{\sqrt{3} }{4}\cos x)dx+\frac{1}{4}\int\limits \frac{dx}{\sin x+\sqrt{3}\cos x } $
$=\frac{1}{2}\int\limits \cos (x+\frac{\pi}{6})dx+\frac{1}{8}\int\limits \frac{dx}{\sin (x+\frac{\pi}{3} )} $
$=\frac{1}{2}\int\limits \cos (x+\frac{\pi}{6})d(x+\frac{\pi}{6})+ \frac{1}{8}\int\limits \frac{dx}{2\sin(\frac{x}{2} +\frac{\pi}{6})\cos(\frac{x}{2} +\frac{\pi}{6})} $
$=\frac{1}{2}\sin(x+\frac{\pi}{6})+\frac{1}{8}\int\limits \frac{d[\tan(\frac{x}{2} +\frac{\pi}{6})]}{\tan(\frac{x}{2} +\frac{\pi}{6})} $
$=\frac{1}{2}\sin(x+\frac{\pi}{6})+\frac{1}{8}\ln|\tan(\frac{x}{2} +\frac{\pi}{6})|+C $