The scariest time for me was in my early days of training.
My instructor and I were heading back to our airport. There was another plane between us and our airport about 1-2 miles ahead practicing steep turns right at our altitude. It would go left, then right, then up, the down, then turn again...