EAGER: Measuring the Stability of Web Links

With every passing day, the Internet becomes more and more integrated in society. Our reliance on online web services in our professional and personal lives continuously increases. Recognizing this, attackers take advantage of the web's popularity to mount a wide range of web application attacks, compromising systems, and stealing private and financial information. Although many classes of these attacks are seemingly unrelated, in reality, many of them stem from the stateless nature of the web, where no safeguards exist to ensure that a resource that was linked to yesterday is the same resource being accessed today. As a result, enforcing link integrity holds the promise of mitigating a multitude of security attacks. However, the dynamicity of the web poses challenges to link integrity schemes, which must cope with constant benign changes while flagging the malicious ones. Pursuing this line of research requires a fundamental understanding of the security aspects of the web?s dynamic nature. This nature cannot be assessed today with the available tools, because existing web crawlers are limited to retrieving content, and generally do not gather the metadata critical to the development of link integrity schemes. This project aims to quantify the stability of web links by developing a custom web crawler infrastructure and tools that will collect security-related metadata about remote resources. The gathering and subsequent assessing of this metadata is a critical first step to demonstrate the feasibility of developing robust link integrity schemes that can be used to detect abnormal content and infrastructure changes (indicative of an ongoing attack) while keeping false positives to a minimum.