These crawls are part of an effort to archive pages as they are created and archive the pages that they refer to. That way, as the pages that are referenced are changed or taken from the web, a link to the version that was live when the page was written will be preserved.
Then the Internet Archive hopes that references to these archived pages will be put in place of a link that would be otherwise be broken, or a companion link to allow people to see what was originally intended by a page's authors.
'General Session | Agile IaaS with the OVH Dedicated Cloud'
It is common to see providers deploy cloud instances using automation, but what happens when these same principles are applied to the provisioning of the infrastructure? In this General Session attendees will gain insider information on how OVH, winner of the VMware Global Partner Network Award 2013, can make automation work to provide IaaS that scales within minutes.