658 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Felix Hamborg	3c25624706	Update README.md	2021-08-14 10:17:52 +02:00
Felix Hamborg	8490b6dd14	Update README.md	2021-08-14 10:15:42 +02:00
Felix Hamborg	7e11b1c915	Update bug_report.md	2021-08-02 15:06:12 +02:00
Felix Hamborg	cc0be8e5e2	Update commoncrawl.py	2021-06-27 21:29:05 +02:00
Felix Hamborg	1e94327a3b	Update README.md	2021-06-19 21:16:50 +02:00
Felix Hamborg	520c975926	add cchardet	2021-05-06 10:01:52 +02:00
Felix Hamborg	e6cdd0f22b	add warc file filtering	2021-04-22 11:03:45 +02:00
Felix Hamborg	cf7d78ef15	Merge pull request #210 from shangw-nvidia/shangw-nvidia/warc_files_end_date ... Support warc_files_end_date for common crawl crawler.	2021-04-22 10:55:31 +02:00
Shang Wang	1bea565ca1	Support warc_files_end_date for common crawl crawler.	2021-04-19 00:17:33 -04:00
Felix Hamborg	44a196f1eb	Update README.md	2021-04-08 13:53:32 +02:00
Felix Hamborg	84ec12a089	Update README.md	2021-03-26 14:23:59 +01:00
Felix Hamborg	6ef48710d3	Update README.md	2021-03-26 14:23:22 +01:00
Felix Hamborg	0adfb8913b	Update README.md	2021-03-23 13:39:41 +01:00
Felix Hamborg	6232b7d60f	fix #206	2021-03-16 12:04:59 +01:00
Felix Hamborg	961948477d	inc version	2021-02-24 08:53:58 +01:00
Felix Hamborg	145cb0b641	add fetch_images optionto ccnc script (default=false)	2021-02-24 08:53:24 +01:00
Felix Hamborg	14b9cef89c	Merge pull request #203 from mood-mapping-muppets/no-fetch-images-newspaper ... Add option of not fetching images using newspaper library and make default for commoncrawl	2021-02-24 08:43:52 +01:00
Felix Hamborg	af1e03531b	Merge pull request #201 from mood-mapping-muppets/empty-warc-newspaper ... Filter empty responses from WARC to avoid spurious exceptions from `newspaper`	2021-02-16 11:17:23 +01:00
Felix Hamborg	0b53affe20	Merge pull request #202 from mood-mapping-muppets/empty-publish-date-newspaper-extractor ... Handle publish_date == "" in newspaper_extractor	2021-02-11 16:32:54 +01:00
Frankie Robertson	11e6d7748c	Add option of not fetching images using newspaper library and make default for commoncrawl	2021-02-10 15:17:32 +02:00
Frankie Robertson	f9da1b7424	Handle publish_date == "" in newspaper_extractor	2021-02-10 15:15:50 +02:00
Frankie Robertson	8456ec4f89	Filter empty responses from WARC to avoid spurious exceptions from newspaper	2021-02-10 15:14:01 +02:00
Felix Hamborg	d3c3eed26b	Merge pull request #198 from mood-mapping-muppets/robust-unicode-warc ... Add option of replacing unicode decode errors in WARC/common crawl extraction	2021-02-08 16:20:04 +01:00
Frankie Robertson	5ed9b804bf	Fallback to utf-8 when document gives unknown encoding	2021-02-08 07:18:05 +02:00
Frankie Robertson	2ade640525	Add option of replacing unicode decode errors in WARC/common crawl extraction	2021-02-08 07:12:18 +02:00
Felix Hamborg	1bbef4a189	Update README.md	2021-02-07 18:18:01 +01:00
Felix Hamborg	f3e478cde3	Merge pull request #200 from mood-mapping-muppets/allow-none-fully-extracted-log ... Check for when log_pathname_fully_extracted_warcs is None and don't log in this case	2021-02-07 14:59:59 +01:00
Felix Hamborg	09754829c7	Update README.md	2021-02-06 19:39:23 +01:00
Frankie Robertson	584facb897	Check for when log_pathname_fully_extracted_warcs is None and don't log in this case	2021-02-06 16:55:10 +02:00
Felix Hamborg	9f7dfef939	Merge pull request #199 from lgov/temp_file_fix ... * newsplease/crawler/commoncrawl_crawler.py: use a system temp file	2021-02-06 15:46:21 +01:00
Felix Hamborg	7f986642bc	Merge pull request #197 from mood-mapping-muppets/continue-article-lang-detect-exception ... Continue detection when LangDetectException in <article>	2021-02-06 15:44:43 +01:00
Felix Hamborg	336c74bc95	Merge pull request #193 from mood-mapping-muppets/programmatic-filter ... Make filter_record public to enable subclasses to override	2021-02-06 15:43:48 +01:00
Frankie Robertson	41beb6d80f	Add documentation of extractor_cls argument	2021-02-06 16:19:08 +02:00
Lieven Govaerts	ae3172d652	* newsplease/crawler/commoncrawl_crawler.py: use a system temp file instead of one in the current directory, which might not be writable.	2021-02-06 13:21:57 +01:00
Frankie Robertson	292133f9f1	Continue detection when LangDetectException in <article>	2021-02-06 14:20:58 +02:00
Felix Hamborg	a5f2fb4bd1	Update setup.py	2021-02-06 12:20:43 +01:00
Felix Hamborg	f69eb3463e	Merge pull request #184 from lgov/filter_warc_files_by_exact_date ... Filter commoncrawl warc files after exact timestamp, not only per year+month	2021-02-06 12:14:44 +01:00
Felix Hamborg	6d29e8367a	Update setup.py	2021-02-05 16:06:54 +01:00
Felix Hamborg	c6163f0032	Merge pull request #195 from mood-mapping-muppets/makedir-race-condition ... Avoid race condition by using exist_ok=True for makedirs rather than checking for exists first	2021-02-05 15:58:29 +01:00
Felix Hamborg	acdd1377dd	Merge pull request #196 from mood-mapping-muppets/longest-article-max-fix ... Fix longest article detection langdetect	2021-02-05 15:57:45 +01:00
Frankie Robertson	f4b1efe06b	Fix longest article detection langdetect	2021-02-05 15:55:34 +02:00
Frankie Robertson	45bd8e5e41	Avoid race condition by using exist_ok=True for makedirs rather than checking for exists first	2021-02-04 19:04:35 +02:00
Felix Hamborg	48ab3300ac	Update setup.py	2021-02-02 14:02:25 +01:00
Felix Hamborg	6beef8bcf9	Merge pull request #194 from mood-mapping-muppets/robust-fromstring ... Fallback to bytes fromstring when lxml unicode fromstring fails	2021-02-02 14:02:04 +01:00
Frankie Robertson	1116f1ca7a	Fallback to bytes fromstring when lxml unicode fromstring fails	2021-02-02 14:42:47 +02:00
Frankie Robertson	563de2b7d7	Add extractor_cls parameter to crawl_from_commoncrawl	2021-02-02 10:30:50 +02:00
Frankie Robertson	89f51e3b6b	Make filter_record public to enable subclasses to override	2021-02-01 19:15:42 +02:00
Felix Hamborg	afc7b411de	Update README.md	2021-01-20 09:30:47 +01:00
Felix Hamborg	b7bc13659b	Update README.md	2021-01-14 15:04:32 +01:00
Felix Hamborg	506034941c	Update README.md	2021-01-14 15:03:47 +01:00