From the assumptions, we have: $\frac{(a^3+b^3)(a+b)}{ab}=(1-a)(1-b)$ $(1)$
Because:
$\frac{(a^3+b^3)(a+b)}{ab}=(\frac{a^2}{b}+\frac{b^2}{a})(a+b)\geq 4ab$
and:
$(1-a)(1-b)=1-(a+b)+ab\leq 1-2\sqrt{ab}+ab$
Up from (1) inferred:
$4ab\leq 1-2\sqrt{ab}+ab$
Set $t=ab\Rightarrow \left\{ \begin{array}{l} 0<t\leq \frac{1}{3}\\4t\leq (1-3t)^2 \end{array} \right.\Leftrightarrow 0<t\leq \frac{1}{9}.$
We have:
$\frac{1}{\sqrt{1+a^2}}+\frac{1}{\sqrt{1+b^2}}\leq \frac{2}{\sqrt{1+ab}}$
and:
$3ab-a^2-b^2=ab-(a-b)^2\leq ab$
So:
$F\leq \frac{2}{\sqrt{1+ab}}+ab=\frac{2}{\sqrt{1+t}}+t$
Review function with $0<t\leq \frac{1}{9}.$
get: $f(t)\leq \frac{6}{\sqrt{10}}+\frac{1}{9}$
at: $a=b=\frac{1}{b}$
Conclude...............................................