$\frac{1}{1+x+\sqrt{1+x^2}}=\frac{1+x-\sqrt{1+x^2}}{2x}=\frac{1}{2x}+\frac{1}{2}-\frac{x\sqrt{1+x^2}}{2x^2}$$\Rightarrow I=\int\limits_{\sqrt{3}}^{\sqrt{8}}(\frac{1}{2x}+\frac{1}{2}-\frac{x\sqrt{1+x^2}}{2x^2})dx=(\frac{1}{2}ln|x|+\frac{1}{2}x)\left| \begin{matrix} \sqrt{8}\\ \sqrt{3} \end{matrix}{} \right.-\int\limits_{a}^{b}\frac{x\sqrt{1+x^2}}{2x^2}$
$=\frac{1}{4}ln\frac{8}{3}+\frac{1}{2}(\sqrt{8}-\sqrt{3})-\int\limits_{\sqrt{3}}^{\sqrt{8}}\frac{x\sqrt{1+x^2}}{2x^2}dx$
đặt $t=\sqrt{1+x^2}\Rightarrow 2tdt=2xdx\Rightarrow tdt=xdt$
đổi cận $x=\sqrt{3}\Rightarrow t=2,x=\sqrt{8}\Rightarrow t=3$
$\Rightarrow I=\frac{1}{4}ln\frac{8}{3}+\frac{1}{2}(\sqrt{8}-\sqrt{3})-\int\limits_{2}^{3}\frac{t^2dt}{2(t^2-1)}$
ta có$\int\limits_{2}^{3}\frac{t^2}{t^2-1}=\int\limits_{2}^{3}(1+\frac{1}{2(t-1)}-\frac{1}{2(t+1)})dt=t+\frac{1}{2}ln|\frac{t-1}{t+1}|=1+\frac{1}{2}ln\frac{3}{2}$
$I=\frac{1}{2}ln\frac{4}{3}+\frac{\sqrt{8}-\sqrt{3}-1}{2}$