About the Data Set

About the Common Crawl Data Set

number of years
size
link back to website

File Locations

The entire Common Crawl data set is stored on Amazon S3 as an Public Data Set.

http://aws.amazon.com/datasets/41740

The directory structure is as follows:

Crawl #1 - s3://aws-publicdatasets/common-crawl/crawl-001/
Crawl #2 - s3://aws-publicdatasets/common-crawl/crawl-002/
Crawl #3 - s3://aws-publicdatasets/common-crawl/parse-output/

General

.

ARC Files - Raw Content

ARC files contain the full HTTP response & payload for all pages crawled.

Text Only files are saved as Hadoop SequenceFiles using GZIP compression. The key and value data types are both Text. The key in these files is the URL, and the value is the actual text content. From HTML pages, the text content includes the page title, the page meta description content, and all text content from the HTML body. They are located in the segment directories, with a file name of "textData-nnnnn". For example:

  s3://aws-publicdatasets/common-crawl/parse-output/segment/1341690169105/1341826131693_45.arc.gz

in the segment directories, with a file name of "textData-nnnnn". For example:

Text Files - Text Only

Common Crawl also produces a series of text only files. These files take content returned as HTML or RSS and parse out just the text content - making it easier for researchers to perform text-based analysis.

Text Only files are saved as Hadoop SequenceFiles using GZIP compression. The key and value data types are both Text. The key in these files is the URL, and the value is the actual text content. From HTML pages, the text content includes the page title, the page meta description content, and all text content from the HTML body. They are located in the segment directories, with a file name of "textData-nnnnn". For example:

  s3://aws-publicdatasets/common-crawl/parse-output/segment/1341690169105/textData-00112

The numbers at the end of the filename are sequentially assigned to Text Only files within the segment - they have not specific meaning.

Further, Text Only files are translated from their native character sets into UTF-8. All Text Only content (in all languages) can be read using the UTF-8 character set.

Currently, we are only producing Text Only files from HTML and RSS/Atom content. The Text Only files are on average 20% of the size of the raw content.

Metadata

In addition to content files, Common Crawl produces a series of metadata files that provide useful information about the crawled content. For each URL, the

Records in the Metadata files are in the same order and have the same file numbers as the Text Only content.

Metadata files are saved as Hadoop SequenceFiles using GZIP compression. The key and value data types are also both Text. The key in these files is the URL, and the value is a JSON structure of fields and subfields - the full structure is defined below.. Just like the Text Only files, the Metadata files are located in the segment directories, with a file name of "metadata-nnnnn". For example:

  s3://aws-publicdatasets/common-crawl/parse-output/segment/1341690169105/metadata-00112

Currently, we are only producing Metadata from HTML and RSS/Atom content.

Metadata Schema

following attributes:

Attribute Name	Attribute Description	Available
attempt_time	The time (in UNIX time format) that the crawl of this page was attempted.	Always
disposition	SUCCESS if the crawler received a successful HTTP response; FAILURE if not.	Always
failure_reason	A code representing why the crawl of this page failed.	On Failure
failure_detail	A message, if available, on why the crawl of this page failed.	On Failure
server_ip	The IP address of the server that returned the response.	On Success
http_result	The HTTP result code.	On Success
http_headers	A JSON object containing all returned HTTP headers as key/value pairs.	On Success
redirect_from
content_len	The value of the Content-Length HTTP header.	On Success
mime_type	The value of the Content-Type HTTP header (stripped of the charset).	On Success
download_size	The actual size of the downloaded content.	On Success
content_is_gzip	Optional attribute that specifies that source content was gzip'd.
gunzip_content_len	If the content was gzip'd, this is the decompressed length of the incoming content.
md5	md5 hash of the incoming content.
text_simhash	A 64 bit simhash of the text (UCS-2) content if document was a valid text type.
charset_detected	The character set Common Crawl detected for the downloaded content.
charset_detector	0 - The character set was derived from an HTTP header. 1 - The character set was derived from an HTML "meta" tag. 2 - The character set was derived from the ICU detector. 3 - The character set was derived from the Mozilla detector. 10 - The character set could not be determined. ISO-8895-1 is assumed.
parsed_as	html - Downloaded content was parsed as HTML. feed - Downloaded content was parsed as an RSS/Atom feed.
content	If the http response was 20x, as the document was parsed as html or a feed, the JSON object represent the document metadata.
content > type
content > title	The value of the HTML "title" tag.
content > meta_tags	A JSON array of objects representing each "meta" tag found by the parser. Note: If the "meta" tag uses a "property" attribute instead of a "name" attribute, "property" is used as the key.
content > links	A JSON array of objects representing each link found by the parser.
content > links > type	The HTML tag type that the link was found in. Examples: a, area, frame, iframe, script, img, link, etc.
content > links > href	The URL associated with the tag, usually from the "href" attribute.
content > links > text	The text displayed for the link. Usually the value of the link element.
content > links > *	Every attribute of the link tag is provided.

The details of the content JSON object if the document was an RSS feed:

"type": "rss-feed"
"title": "The title attribute of the feed."
"link": "A JSON object representing the link attribute of the feed."
"description": "Feed description"
"updated": "Either LastBuildDate or PubDate, depending on which one is greater."
"categories": "A JSON array representing the category names associated with the feed."
"generator": "The Feed Generator"
"ttl": "The TTL value if specified in the RSS Feed."
"items": "A JSON array of feed items."

Each RSS feed item contains the following attributes:

"title": "The title attribute of the Feed Item."
"description": "Feed Item description."
"link": "A JSON object representing the link attribute of the Feed Item."
"author": "The Feed Item's author."
"categories": "A JSON array representing the category names associated with the Feed Item."
"comments": "The url used to retrieve comments associated with this Feed."
"published": "The PubDate of this Feed Item."
"guid": "The RSS guide representing this Feed Item."
"content": "A JSON object or array of objects that contain any links embedded in the content body of the Feed Item."

The details of the content JSON object if the document was an Atom feed:

"type": "atom-feed"
"title": "The HTML stripped title attribute of the feed."
"link": "A JSON object representing the rel=alternate link attribute of the feed."
"updated": "The Update date of the Feed."
"categories": "A JSON array representing the category names associated with the feed."
"generator": "The Feed Generator"
"authors": "A JSON array of objects representing the Feed's authors."
"items": "A JSON array of Feed Items."

Each Atom feed item contains the following attributes:

"title": "The title attribute of the Feed Item."
"description": "Feed Item description."
"published": "The Published date of this Feed Item."
"updated": "The Updated date of this Feed Item."
"link": "A JSON object or array of objects representing the rel=alternate links in contained in this Feed Item."
"self": "A JSON object or array of objects representing the rel=self links in contained in this Feed Item."
"replies": "A JSON object or array of objects representing the rel=replies links in contained in this Feed Item."
"authors": "A JSON array of objects representing the Feed's authors."
"categories": "A JSON array representing the category names associated with the feed."
"content": "A JSON object that contains any links embedded in the content body of the Feed Item."
"content": "A JSON object or array of objects that contain any links embedded in the content body of the Feed Item."