5. Hệ phương trình tương đương với:
$\left\{\begin{array}{l}5(x+y)^2+3(x-y)^2+\dfrac{5}{(x+y)^2}=13\\x+y+x-y+\dfrac{1}{x+y}=1\end{array}\right.$
$\Leftrightarrow \left\{\begin{array}{l}5(x+y+\dfrac{1}{x+y})^2+3(x-y)^2=23\\x+y+\dfrac{1}{x+y}+x-y=1\end{array}\right.$
Đặt $u=x+y+\dfrac{1}{x+y};v=x-y$, hệ trở thành:
$\left\{\begin{array}{l}5u^2+3v^2=23\\u+v=1\end{array}\right.$
$\Leftrightarrow \left\{\begin{array}{l}v=1-u\\5u^2+3(1-u)^2=23\end{array}\right.$
$\Leftrightarrow \left[\begin{array}{l}\left\{\begin{array}{l}u=2\\v=-1\end{array}\right.\\\left\{\begin{array}{l}u=-\dfrac{5}{4}\\v=\dfrac{9}{4}\end{array}\right.\end{array}\right.$
Với $\left\{\begin{array}{l}u=2\\v=-1\end{array}\right.$, ta có:
$\left\{\begin{array}{l}x+y+\dfrac{1}{x+y}=2\\x-y=-1\end{array}\right.$
$\Leftrightarrow \left\{\begin{array}{l}x+y=1\\x-y=-1\end{array}\right.$
$\Leftrightarrow \left\{\begin{array}{l}x=0\\y=1\end{array}\right.$
Với $\left\{\begin{array}{l}u=-\dfrac{5}{4}\\v=\dfrac{9}{4}\end{array}\right.$, ta có:$\left\{\begin{array}{l}x+y+\dfrac{1}{x+y}=-\dfrac{5}{4}\\x-y=\dfrac{9}{4}\end{array}\right.$, vô nghiệm.