The record holder for most distant object ever observed was a gamma ray burst. GRB 090423 was picked up by the Swift satellite. It lasted about 10 seconds, and was measured to have a redshift of 8.2. Whatever caused it exploded roughly 13 billion years ago — estimated at a mere 630 million years after the Big Bang. The Universe was just a baby back then. Even the galaxies were young that long ago.
Interestingly, seeing a GRB this old implies that massive stars were indeed forming and dying in the early Universe. It was probably stars like these that created the heavier elements that would eventually form cosmic dust, planets… and us.