$P=\frac{3(x^2+y^2)+3(x+y)}{4}+\frac{3-(x+y)}{x+y}-(x^2+y^2)$
$=-\frac{x^2+y^2}{4}+\frac{3(x+y)}{4}+\frac{3}{x+y}-1$
$=-\frac{(x+y)^2}{4}+(x+y)-1-\frac{x+y}{4}+\frac{3}{x+y}+\frac{xy}{2}$
$=-(\frac{x+y}{2}-1)^2-\frac{x+y}{4}+\frac{3}{x+y}+\frac{3-(x+y)}{2}$
$=-(\frac{x+y}{2}-1)^2-3(\frac{x+y}{4}-\frac{1}{x+y})+\frac{3}{2}$
$=-(\frac{x+y}{2}-1)^2-3.\frac{(x+y)^2-4}{4(x+y)}+\frac{3}{2}$ (1)
Ta có: $3=xy+x+y \le \frac{(x+y)^2}{4}+x+y=(\frac{x+y}{2}+1)^2-1$
=> $(\frac{x+y}{2}+1)^2 \ge 4$
=> $x+y \ge 2$
=> $(x+y)^2 \ge 4$ (do $x+y>0$) (2)
từ (1) và (2) => $P \le \frac{3}{2}$
Dấu = xảy ra <=> $x=y=1$
Vậy $P_{max}=\frac{3}{2}$ tại $x=y=1$