In this paper we will talk about the real world uses of the robots.txt file. I will show what it is and normally used for. Then I will show how you can leverage this file in your reconnaissance steps of a target. We will also look at some interesting robots.txt files that have been over time.

Who is David Young. Been in the I.T. industry for the past 17 years. Past 7 years dedicated to IT Security. Loves to think outside of the box. Avid gamer and all around true blue geek.

Back to BSides Columbus Ohio 2015 video list