Ta có
$\sqrt{2a^4+2a^2b+5b^2}=\sqrt{(a^4+a^2b+a^2b+b^2)+a^4+4b^2}$$ \overset{AM-GM}{\ge} \sqrt{4a^2b+a^4+4b^2}=a^2+2b$
$\Rightarrow \frac{a}{\sqrt{2a^4+2a^2b+5b^2}+3} \le \frac{a}{a^2+2b+3}$
Tương tự
$\Rightarrow \sum\frac{a}{\sqrt{2a^4+2a^2b+5b^2}+3} \le \sum \frac{a}{a^2+2b+3} $
$\Leftrightarrow P \le \sum \frac{a}{a^2+2b+3} =\sum\frac{a}{(a^2+1)+2b+2} \le \sum\frac{a}{2a+2b+2}$
Ta sẽ chứng minh $\sum\frac{a}{2a+2b+2} \le \frac 12$
$\Leftrightarrow \sum\frac{a}{a+b+1} \le 1\Leftrightarrow \sum \frac{b+1}{a+b+1} \ge 2$
BĐT trên đúng do
$\sum \frac{b+1}{a+b+1}=\sum\frac{(b+1)^2}{(a+b)(a+b+1)} \ge \frac{(a+b+c+3)^2}{\sum(a+b)(a+b+1)}=2$ (khai triển k.hợp đk :D)
Vậy $P \le \frac12$