Felix Hamborg cc0be8e5e2 Update commoncrawl.py

2021-06-27 21:29:05 +02:00

..

config

Removed f-strings and used format method. Added the new config parameter into config_lib.cfg file

2020-04-28 14:04:31 +05:30

crawler

Support warc_files_end_date for common crawl crawler.

2021-04-19 00:17:33 -04:00

examples

Update commoncrawl.py

2021-06-27 21:29:05 +02:00

helper_classes

Renamed modle_util.py into class_loader.py. More specific error handling for loading custom news item module and class. Fixed a typo.

2020-05-12 08:07:24 +05:30

pipeline

Merge pull request #203 from mood-mapping-muppets/no-fetch-images-newspaper

2021-02-24 08:43:52 +01:00

__init__.py

Merge pull request #203 from mood-mapping-muppets/no-fetch-images-newspaper

2021-02-24 08:43:52 +01:00

__main__.py

Add Postgresql Storage option with init and reset of data

2020-04-05 15:18:54 +03:00

config.py

Fix config warning

2020-06-23 12:05:45 -07:00

helper.py

Changes to make Scrapy Item class customizable via configuration

2020-04-22 19:10:50 +05:30

init-postgresql-db.sql

Add Postgresql Storage option with init and reset of data

2020-04-05 15:18:54 +03:00

NewsArticle.py

continue transition from text to maintext

2020-03-20 11:24:50 +01:00

single_crawler.py

Renamed modle_util.py into class_loader.py. More specific error handling for loading custom news item module and class. Fixed a typo.

2020-05-12 08:07:24 +05:30