- Created by Chris Stephens, last modified on Jul 14, 2012
You are viewing an old version of this page. View the current version.
Compare with Current View Page History
« Previous Version 6 Next »
About the Common Crawl Data Set
- number of years
- size
- link back to website
File Locations
The entire Common Crawl data set is stored on Amazon S3 as an Public Data Set.
http://aws.amazon.com/datasets/41740
The directory structure is as follows:
Crawl #1 - s3://aws-publicdatasets/common-crawl/crawl-001/
Crawl #2 - s3://aws-publicdatasets/common-crawl/crawl-002/
Crawl #3 - s3://aws-publicdatasets/common-crawl/parse-output/
General
.
ARC Files - Raw Content
ARC files contain the full HTTP response & payload for all pages crawled.
Text Only files are saved as Hadoop SequenceFiles using GZIP compression. The key and value data types are both Text. The key in these files is the URL, and the value is the actual text content. From HTML pages, the text content includes the page title, the page meta description content, and all text content from the HTML body. They are located in the segment directories, with a file name of "textData-nnnnn". For example:
s3://aws-publicdatasets/common-crawl/parse-output/segment/1341690169105/1341826131693_45.arc.gz
in the segment directories, with a file name of "textData-nnnnn". For example:
Text Files - Text Only
Common Crawl also produces a series of text only files. These files take content returned as HTML or RSS and parse out just the text content - making it easier for researchers to perform text-based analysis.
Text Only files are saved as Hadoop SequenceFiles using GZIP compression. The key and value data types are both Text. The key in these files is the URL, and the value is the actual text content. From HTML pages, the text content includes the page title, the page meta description content, and all text content from the HTML body. They are located in the segment directories, with a file name of "textData-nnnnn". For example:
s3://aws-publicdatasets/common-crawl/parse-output/segment/1341690169105/textData-00112
The numbers at the end of the filename are sequentially assigned to Text Only files within the segment - they have not specific meaning.
Further, Text Only files are translated from their native character sets into UTF-8. All Text Only content (in all languages) can be read using the UTF-8 character set.
Currently, we are only producing Text Only files from HTML and RSS/Atom content. The Text Only files are on average 20% of the size of the raw content.
Metadata
In addition to content files, Common Crawl produces a series of metadata files that provide useful information about the crawled content. For each URL, the
Records in the Metadata files are in the same order and have the same file numbers as the Text Only content.
Metadata files are saved as Hadoop SequenceFiles using GZIP compression. The key and value data types are also both Text. The key in these files is the URL, and the value is a JSON structure of fields and subfields - the full structure is defined below.. Just like the Text Only files, the Metadata files are located in the segment directories, with a file name of "metadata-nnnnn". For example:
s3://aws-publicdatasets/common-crawl/parse-output/segment/1341690169105/metadata-00112
Currently, we are only producing Metadata from HTML and RSS/Atom content.
Metadata Schema
following attributes:
Attribute Name | Attribute Description | Available |
---|---|---|
attempt_time | The time (in UNIX time format) that the crawl of this page was attempted. | Always |
disposition | SUCCESS if the crawler received a successful HTTP response; FAILURE if not. | Always |
failure_reason | A code representing why the crawl of this page failed. | On Failure |
failure_detail | A message, if available, on why the crawl of this page failed. | On Failure |
server_ip | The IP address of the server that returned the response. | On Success |
http_result | The HTTP result code. | On Success |
http_headers | A JSON object containing all returned HTTP headers as key/value pairs. | On Success |
redirect_from | ||
content_len | The value of the Content-Length HTTP header. | On Success |
mime_type | The value of the Content-Type HTTP header (stripped of the charset). | On Success |
download_size | The actual size of the downloaded content. | On Success |
content_is_gzip | Optional attribute that specifies that source content was gzip'd. | |
gunzip_content_len | If the content was gzip'd, this is the decompressed length of the incoming content. | |
md5 | md5 hash of the incoming content. | |
text_simhash | A 64 bit simhash of the text (UCS-2) content if document was a valid text type. | |
charset_detected | The character set Common Crawl detected for the downloaded content. | |
charset_detector | 0 - The character set was derived from an HTTP header. | |
parsed_as | html - Downloaded content was parsed as HTML. feed - Downloaded content was parsed as an RSS/Atom feed. | |
content | If the http response was 20x, as the document was parsed as html or a feed, the JSON object represent the document metadata. | |
content > type | ||
content > title | The value of the HTML "title" tag. | |
content > meta_tags | A JSON array of objects representing each "meta" tag found by the parser. Note: If the "meta" tag uses a "property" attribute instead of a "name" attribute, "property" is used as the key. | |
content > links | A JSON array of objects representing each link found by the parser. | |
content > links > type | The HTML tag type that the link was found in. Examples: a, area, frame, iframe, script, img, link, etc. | |
content > links > href | The URL associated with the tag, usually from the "href" attribute. | |
content > links > text | The text displayed for the link. Usually the value of the link element. | |
content > links > * | Every attribute of the link tag is provided. | |
The details of the content JSON object if the document was an RSS feed:
"type": "rss-feed"
"title": "The title attribute of the feed."
"link": "A JSON object representing the link attribute of the feed."
"description": "Feed description"
"updated": "Either LastBuildDate or PubDate, depending on which one is greater."
"categories": "A JSON array representing the category names associated with the feed."
"generator": "The Feed Generator"
"ttl": "The TTL value if specified in the RSS Feed."
"items": "A JSON array of feed items."
Each RSS feed item contains the following attributes:
"title": "The title attribute of the Feed Item."
"description": "Feed Item description."
"link": "A JSON object representing the link attribute of the Feed Item."
"author": "The Feed Item's author."
"categories": "A JSON array representing the category names associated with the Feed Item."
"comments": "The url used to retrieve comments associated with this Feed."
"published": "The PubDate of this Feed Item."
"guid": "The RSS guide representing this Feed Item."
"content": "A JSON object or array of objects that contain any links embedded in the content body of the Feed Item."
The details of the content JSON object if the document was an Atom feed:
"type": "atom-feed"
"title": "The HTML stripped title attribute of the feed."
"link": "A JSON object representing the rel=alternate link attribute of the feed."
"updated": "The Update date of the Feed."
"categories": "A JSON array representing the category names associated with the feed."
"generator": "The Feed Generator"
"authors": "A JSON array of objects representing the Feed's authors."
"items": "A JSON array of Feed Items."
Each Atom feed item contains the following attributes:
"title": "The title attribute of the Feed Item."
"description": "Feed Item description."
"published": "The Published date of this Feed Item."
"updated": "The Updated date of this Feed Item."
"link": "A JSON object or array of objects representing the rel=alternate links in contained in this Feed Item."
"self": "A JSON object or array of objects representing the rel=self links in contained in this Feed Item."
"replies": "A JSON object or array of objects representing the rel=replies links in contained in this Feed Item."
"authors": "A JSON array of objects representing the Feed's authors."
"categories": "A JSON array representing the category names associated with the feed."
"content": "A JSON object that contains any links embedded in the content body of the Feed Item."
"content": "A JSON object or array of objects that contain any links embedded in the content body of the Feed Item."
- No labels