A Guide To Common HTTP Status Codes For Archivists
When an archivist visits a webpage, an HTTP status code provides an immediate clue about what happened between the browser, the server and the requested resource. The visible page may look ordinary, yet the response can reveal whether the item is current, redirected, temporarily unavailable, deliberately removed or blocked from access.
Understanding these signals helps with web harvesting, link checking, catalogue maintenance and long-term stewardship. It is especially useful for a resource hub such as papajohnphillips.com, where historical articles, archived news and domain information may need to remain discoverable across changing site structures.
How Status Codes Describe A Web Request
HTTP responses are grouped into five families. Codes beginning with 2 indicate success, 3 usually describe redirection or a response that can be served from a cache, 4 point to a problem with the request or access rights, and 5 indicate a server-side failure. The first digit gives the broad meaning, while the following digits provide operational detail.
An archivist should record the complete response rather than relying on a browser’s display. Useful fields include the status code, final URL, redirect chain, response date, content type, content length and key headers. A command such as curl -I https://example.org/page can reveal these details without downloading the full document, although a HEAD request should be checked against a normal GET because some servers handle the two methods differently.
Status codes are evidence, not absolute historical truth. A 404 today does not prove that a page was absent last year, and a 200 response does not prove that the content is authentic or unchanged. Pair the response with crawl dates, screenshots, WARC files, checksums and catalogue notes.
Successful Responses And Preserved Content
The most familiar success code is 200 OK. It means the server returned a response successfully, but it says nothing about the quality, completeness or historical accuracy of the content. An archived article may return 200 while images, style sheets or embedded media fail separately, so each linked asset deserves its own check.
Code 206 Partial Content is important when a server returns only part of a file in response to a range request. Large audio, video and PDF resources may use this behaviour. A harvesting tool should combine valid segments carefully and retain the relevant headers, especially Content-Range, so that the final object can be verified.
Code 204 No Content is less common in public publishing. It confirms a successful request while returning no body. Archivists may encounter it in APIs, form submissions or tracking endpoints. It should not be treated as proof that a web page has vanished; it may simply mean that the requested operation was designed to return metadata or an empty result.
Redirects And The History Of A URL
A 301 Moved Permanently response tells clients that a resource has a new address. A 308 Permanent Redirect conveys a similar idea while preserving the HTTP method. These codes are valuable historical clues: a publication may have moved from an old folder into a news archive, or a domain may have shifted from one platform to another.
Temporary redirects, commonly 302 Found and 307 Temporary Redirect, require more caution. They can be used during migrations, campaign periods, maintenance or login flows. The final destination might change again, so an archivist should save the complete chain instead of recording only the last URL. Excessive hops can also cause crawlers to stop before reaching the content.
A 304 Not Modified response means the server believes a cached copy is still current, based on headers such as ETag or Last-Modified. It does not contain the full page. For preservation work, retain the earlier body alongside the 304 metadata. This helps establish which version was available when the cache validation occurred.
Domain ownership and naming history can add context to redirect records. When assessing an address that may be sold or repurposed, archivists can review domain pricing factors alongside its publication history, backlinks and archived reputation.
Missing Pages And Deliberate Removal
404 Not Found is the standard signal for a resource that the server cannot locate. It may result from a mistyped URL, a deleted article, a broken internal link or a site migration. Custom 404 pages often return a friendly message while still correctly using status 404, but some websites mistakenly send a 200 response for every missing page. This is called a soft 404 and can confuse search engines and archival crawlers.
410 Gone is stronger. It indicates that the resource has been intentionally removed and is unlikely to return. The distinction matters when preserving a publication history. A 410 may support a note such as “removed by publisher”, while a 404 should generally be recorded as “not found at time of capture” unless other evidence explains the absence.
Status 451 Unavailable For Legal Reasons can appear when a publisher, court or network operator restricts access because of legal requirements. Its use is not universal, and a blocked page may instead produce 403 or a generic error. In Australia, records involving defamation, privacy, copyright or court-related restrictions should be handled with care under relevant legal and institutional policies.
A missing page should trigger a search through earlier captures, sitemap files, RSS feeds, old navigation pages and external references. Australian community reporting often circulates through local Facebook groups, council newsletters and email bulletins, so a vanished article may leave traces outside the original site.
Permissions, Robots And Restricted Access
403 Forbidden means the server understood the request but refuses to provide the resource. It may reflect an access-control rule, an IP block, a missing authentication token, a firewall or a decision to disallow automated harvesting. Repeated requests are unlikely to solve the problem and may worsen the block.
401 Unauthorized usually indicates that authentication is required. The distinction is useful: 401 invites a valid authentication process, while 403 generally says that permission will not be granted under the current conditions. A 429 Too Many Requests signals rate limiting, often with a Retry-After header. Respecting that instruction is essential for responsible collecting.
Robots exclusion files and terms of use provide additional signals, though they are not the same as HTTP permissions. An archivist should identify the collecting institution, use a clear user agent where appropriate and limit concurrency. This is particularly considerate for small Australian organisations whose websites may run on modest hosting accounts in Sydney, Melbourne or Brisbane.
Privacy obligations also matter. The Australian Privacy Act 1988 and the Australian Privacy Principles may be relevant when an archive captures personal information, user comments, contact details or photographs. Public availability does not automatically remove every ethical or legal concern. Record the basis for collection, access restrictions and any redaction decisions in the preservation metadata.
Server Errors And Responsible Retry Behaviour
The 500 Internal Server Error is a general failure on the server. It may arise from faulty application code, a database problem or an unexpected configuration issue. Code 502 Bad Gateway means that a gateway or proxy received an invalid response from an upstream server, while 503 Service Unavailable commonly indicates maintenance, overload or planned downtime.
504 Gateway Timeout means that an intermediary waited too long for another server to respond. These failures can be temporary, so a single failed capture should not be treated as evidence that the content has disappeared. Schedule a later retry and compare the results. For Australian sites, maintenance may be timed around overnight periods, public holidays or local business hours rather than the schedule used by overseas crawlers.
Retrying should be measured rather than aggressive. Use increasing delays, honour Retry-After, avoid parallel requests to a fragile host and preserve the original error response. A 503 followed by a 200 is valuable operational history, especially if the site serves community information during emergencies, local events or council consultations.
Caching layers can complicate interpretation. A content delivery network may return a different status from the origin server, and a reverse proxy may conceal application errors. Record headers such as Server, Via, Age, Date and cache indicators where permitted. These fields help explain why two captures made minutes apart produced different results.
A Practical Capture Record For Archives
A useful workflow begins before the request is sent. Define the scope, identify the target URLs, note the reason for collecting them and check whether the site publishes a sitemap or feed. Capture the landing page, linked assets and redirect destinations where possible. Store timestamps in a consistent format, preferably UTC, while retaining the local Australian date and time when it helps explain publishing activity.
The response should be paired with the retrieved body and its checksum. WARC files are widely used for web preservation because they can hold request and response information, headers and payloads in a structured package. A catalogue record can then connect the WARC entry to the title, author, publication date, location, rights statement and status-code history.
File handling deserves attention too. Community archives often contain scanned newsletters, photographs and recordings created on phones or consumer cameras. Clear filenames, stable identifiers and documented transfer methods reduce confusion when materials move between volunteers, libraries and institutional storage. Guidance on faster file transfers can be relevant when large capture packages need to move across uneven home or regional internet connections.
For Australian collections, note the source location and any relevant state or territory context. A page about a bushfire in regional New South Wales, a community event in Adelaide or a planning notice in Perth may need descriptive metadata beyond its URL. Good context makes a preserved response useful to researchers long after the original server has changed.
Quick Reference For Archival Decisions
The code alone should never decide whether an item is preserved, excluded or described as missing. Compare it with the body, headers, earlier captures and the purpose of the collection. The following reference gives a practical starting point for triage.
| Status code | Common meaning | Archival response |
|---|---|---|
| 200 | Resource returned successfully | Save the body, headers, timestamp and checksum |
| 204 | Successful response with no body | Record the event; do not assume the page was deleted |
| 206 | Partial content returned | Reassemble and verify range-based downloads |
| 301 / 308 | Permanent redirect | Preserve the source URL, destination and redirect chain |
| 302 / 307 | Temporary redirect | Capture the destination but expect future changes |
| 304 | Cached copy remains valid | Link the response to the earlier stored representation |
| 401 | Authentication required | Record the restriction and avoid bypassing controls |
| 403 | Access forbidden | Log the refusal and review policy, robots and permissions |
| 404 | Resource not found | Search earlier captures and label absence by capture date |
| 410 | Resource deliberately gone | Record likely removal and retain supporting evidence |
| 429 | Too many requests | Pause, follow Retry-After and reduce request frequency |
| 451 | Unavailable for legal reasons | Document the restriction and apply rights-aware access rules |
| 500 / 502 | Server or gateway failure | Preserve the error and retry later with backoff |
| 503 / 504 | Service unavailable or timed out | Schedule a later capture and record downtime context |
Use this guide as part of a broader preservation policy. Regular link audits, redundant storage, fixity checks and clear ownership arrangements are more dependable than a single crawl. A small resource hub can remain useful for many years when each response is treated as evidence of a changing digital environment.
Explore the preserved materials, review their access notes and keep a dated record of important URLs and status changes. If you manage a historical domain, community archive or collection of web pages, contact the site team to discuss responsible stewardship and future preservation.