Fun read. I have done similar. The difference for me is that I didn’t try to predict “price” just direction and magnitude. I would use the magnitude as a confidence value and direction for whether I thought it was better to buy or sell. I would do this with log returns instead of regular returns and that seemed to smooth things out and make the predictions more stable.