A practical guide to web caching for archivists
Web caching stores copies of online resources so they can be served again without another live request to the origin server. The term covers several different systems: a browser cache, a shared proxy or content delivery network, and a preservation repository holding dated snapshots. For archivists, purpose matters. Temporary caching improves speed and reduces traffic; preservation caching creates a historical trace that can survive a broken link, redesign or withdrawn page.
A sound programme joins collection policy with reliable capture, storage, metadata, rights review and ongoing oversight. It should establish what is kept, when it was seen, how it can be replayed and which restrictions apply. Australian collections often operate between national institutions in Canberra, commercial suppliers and community sites spread across the continent. Trove and national web-preservation programmes provide a national backdrop, while local and family histories give smaller sites their particular value.
Why cached web content matters
Web pages often contain the only surviving copy of a notice, community report, campaign page or personal account. A content management system can be replaced without warning, while a dormant domain may expire and its DNS records disappear. Search indexes and social posts may still point to the old address, leaving researchers with references that no longer resolve. Cached copies preserve the response as it appeared, including linked images and stylesheets, so the record remains intelligible rather than becoming a collection of fragments.
Preservation caching also retains evidence of how a service behaved at a particular time. An archivist can use the discussion on keeping old pages accessible as a plain-language starting point for treating a dead link as a collection risk rather than a clerical annoyance. A durable capture can retain redirects, page structure, timestamps and assets while giving users a stable replay address. That matters for a resource hub that connects archived news, reference pages and community articles: the landing pages remain useful only when their internal destinations can also be reached.
Deciding what to capture
Begin with a written capture policy rather than an unrestricted crawl. Rank pages by significance, uniqueness, risk of loss, demand and relationship to other material. Decide whether to capture one page, a whole section, supporting files, linked documents or an entire host. Dynamic features may need separate handling because search boxes, comment feeds and APIs change independently of static HTML. A smaller, well-described collection is easier to defend, replay and maintain than a large set of unexplained captures.
Community material often gains meaning from relationships between stories, images and contributor names. The community-focused Many Mamas, Many Papas collection shows how a themed site can connect family memory with local identity and shared history. An archive should preserve that context while recording which links were live, which assets were incomplete and which requests failed. Where consent or cultural protocols limit access, the record can remain available to authorised users even if public replay is restricted.
Understanding cache behaviour
HTTP defines several caching controls, and archivists need to distinguish speed settings from preservation decisions. Cache-Control: max-age and s-maxage define freshness for a browser or shared cache; no-cache permits storage but requires revalidation, while no-store asks clients not to store the response. ETag and Last-Modified act as validators, allowing a server to confirm that a stored copy is unchanged. Vary matters when the same URL returns different content according to language, device or other request headers.
An archival crawler works differently from a general browser. It requests each resource, records the response body and headers in a container such as WARC, and stores request details, status codes and timestamps. A replay system later serves that record, often through a separate preservation address. Because relative links and scripts depend on the original site, the crawler must also capture dependencies or create safe redirect mappings. Domain changes, expiring registrations and renewed certificates can break this chain, making historical DNS and URL records part of the evidential context.
Recording pages with context
Every capture needs enough metadata to be found, trusted and interpreted. Record the source URL, retrieval time, HTTP status, MIME type, byte count, content digest, software version and any processing decisions. A memento-style timestamp gives researchers a stable reference to the state of a page on a chosen date. Descriptive metadata should name the site, organisation, people, places and subjects represented, while technical metadata explains whether the record is complete, partial or a redirected representation.
Relationships to material elsewhere need careful treatment because a linked page can disappear or change independently. If a page documents an off-platform account or export process, the archive can preserve a lawful copy and its relationship to the originating page. One example encountered in this kind of research is detailed account export guidance, which shows why preserving surrounding explanation matters when the host later alters or removes the original. Credentials, private messages and unnecessary personal details should be excluded, even when they appear in a public interface.
Storing and verifying archive files
Archive files need durable storage, fixity checking and geographically sensible replication. The Australian market is a mixture of global cloud platforms with Australian regions in Sydney and Melbourne, local managed-service providers and institutional systems located in Canberra and elsewhere. That gives collection managers options, but service availability, contract terms and exit arrangements must be tested rather than assumed. Copies stored in Sydney and Melbourne improve resilience for eastern states; for collections serving Perth, Hobart, Darwin or Alice Springs, latency and network failure remain real even when the data itself is sound.
Checksums such as SHA-256 provide evidence that a file has not changed. The familiar three-copy approach should be adapted for digital preservation: maintain at least one verified copy in a separate geographic location, use a second storage system or provider, and keep an offline or logically separated master where budgets allow. Replicate indexes and metadata as carefully as payloads, document restore procedures and test a sample quarterly. A backup that cannot be recovered from is only an assumption, especially during regional outages or a provider migration.
Rights, access and cultural responsibilities
Rights review belongs in the capture workflow. Australian institutions may rely on specific preservation and library-copying provisions under the Copyright Act 1968, but those provisions do not create unrestricted publication rights. A capture can be retained for legitimate archival purposes while public access is limited for privacy, contractual or cultural reasons. Terms of service, robots instructions and publisher permissions should be recorded and assessed; robots.txt is a crawler directive, not a licence or a complete legal test.
Privacy, records law and cultural responsibilities also shape access. The Privacy Act 1988 may apply to identifiable information held by covered organisations, while state records rules and contractual duties can impose separate obligations. Public availability does not automatically make sensitive material suitable for broad circulation. For First Nations material, seek the relevant community’s authority and follow its cultural and intellectual property protocols, including decisions about names, images, sacred knowledge and descendants’ access. These protocols are community-specific and should never be replaced by a generic consent form.
Testing and maintaining the cache
Testing turns a collection plan into a reliable service. Replay sample records in a clean browser without contacting the live site, then check text, images, stylesheets, downloads, forms and error pages. Record redirects and broken dependencies, test both IPv4 and IPv6 connections, and inspect captures with common Australian desktop and mobile configurations. A crawler that reports HTTP 200 has only proved that the request succeeded; it has not proved that the page is readable or complete. Include known volatile elements in acceptance tests so later changes are obvious.
Maintenance should be scheduled rather than left to the next crisis. Audit fixity, search indexes, replay links and redirect maps; review capture scope as sites are redesigned; and keep the software, format documentation and domain contact details current. A small change in a content management system can affect thousands of pages, while a quiet local site can disappear between annual crawls. Australian collection requests may arrive in relaxed language—“no worries”, “when you get a sec” or “not urgent”—so triage notes should preserve the original wording and date. Informality can conceal a strict deadline in messages travelling from a branch in Perth, Hobart or Alice Springs to a national office.
Start with a modest, well-described capture that answers a real research or community need. Define success through searchable metadata, accurate replay, documented rights and a tested recovery path. Involve the people whose material is held, especially communities that have been affected by broken or ungoverned online records. A cache is successful when it keeps context available without pretending that every page can be preserved indefinitely.
Browse the preserved reviews, community-focused articles and historical sections in the Papajohn Phillips resource hub, then use them as a model for linking archived material responsibly. Visitors can browse the maintained collections or enquire about the domain and its long-term stewardship. Preserving a stable page, adding a useful description and keeping the next generation of links alive are practical ways to strengthen Australia’s collective web memory.