Removing Manually-Generated Boilerplate from Electronic Texts: Experiments with Project Gutenberg e-Books

Kaser, Owen et Lemire, Daniel (2007). « Removing Manually-Generated Boilerplate from Electronic Texts: Experiments with Project Gutenberg e-Books », dans CASCON 2007 (17th IBM Centers for Advanced Studies Annual International Conference, Toronto, 2007)

Fichier(s) associé(s) à ce document :

Prévisualisation	PDF Télécharger (125kB)

Résumé

Collaborative work on unstructured or semi-structured documents, such as in literature corpora or source code, often involves agreed upon templates containing metadata. These templates are not consistent across users and over time. Rule-based parsing of these templates is expensive to maintain and tends to fail as new documents are added. Statistical techniques based on frequent occurrences have the potential to identify automatically a large fraction of the templates, thus reducing the burden on the programmers. We investigate the case of the Project Gutenberg corpus, where most documents are in ASCII format with preambles and epilogues that are often copied and pasted or manually typed. We show that a statistical approach can solve most cases though some documents require knowledge of English. We also survey various technical solutions that make our approach applicable to large data sets.

Type:	Communication, article de congrès ou colloque
Mots-clés ou Sujets:	Digital Libraries, Automated Text Processing, Frequent items
Unité d'appartenance:	Télé-université > UER Science et Technologie
Déposé par:	Daniel Lemire
Date de dépôt:	16 juill. 2007
Dernière modification:	01 nov. 2014 02:03
Adresse URL :	http://archipel.uqam.ca/id/eprint/351

Modifier les métadonnées (propriétaire du document)

Statistiques