chapter_08

[priti-chahal]

• Collaborator: Priti

What is semanticClimate?

semanticClimate is a project which aims to convert climate related documents by the IPCC to a semantic form, understood by both humans and computers.

Why are such conversions required?

The goal is to extract useful information from the IPCC documents so that relevant information could be made accessible for everyone to read and understand. This is achieved by creating dictionaries of different types for different purposes.

WG3 (Mitigation)

We are currently using following tools:

py4ami
docanalysis
gensim(keyword extraction)

Environment and origins:

window 11
python version 3.8.0
docanalysis version 0.2.0
py4ami version 0.0.45
What we started with and our environment.

Software used: py4ami docanalysis

How we set up the software:

1. Install Anaconda
2. open Anaconda
3. Create venv
4. setup python version 3.8.0
5. run
   pip install docanalysis
docanalysis --help

py4ami has been used for:

• Converting pdf to html
• Validating whether the created dictionary is correct or not
• Annotating the dictionary, which means, one can find the word from the dictionary in the generated html file, and then go to the wikidata page for the word, just by clicking on the word.

docanalysis has been used for:

• Creating automated dictionary from a html file.
• Creating different entity dictionaries (abbreviation dictionary, organisation dictionary etc) from the same html file.

The protocol:

Download the IPCC/ar6/wg3 Chapter* pdf:

• from IPCC website
• from Peter’s github repository

Download Peter’s semanticClimate github repository:

git clone https://github.com/petermr/semanticClimate.git

What is chapter 08 about?

• Title: Urban System and other settlements.

Summary of the Chapter

• Summary of the Chapter
• fulltext.pdf

Creation of raw HTML:(using py4ami)

create pdf to html:

python -m py4ami.ami_pdf --inpath \Users\priti\Climate\semanticClimate\ipcc\ar6\wg3\Chapter08\fulltext.pdf /
--outdir \Users\priti\Climate\semanticClimate\ipcc\ar6\wg3\Chapter08\fulltext.html --maxpage 110  

fulltext.html

Creation of sections

• dict: this section contains two types of dictionaries
  • ip_3_8_urban_man.xml
  • ip_3_8_urban_abb.xml
  • keyphrase_dict(NYI)

Extraction of dictionaries

1. ip_3_8_urban_man.xml: created manually by reading the chapter.
2. ip_3_8_urban_abb.xml: automated created by docanalysis using spacy

Creation of manual dictionary from pdf:

1. ip_3_8_urban_man.xml: This is done by manually reading the pdf, and manually entering any term or abbreviaion that you do not understand, or you think is important for the chapter. The dictionary is a xml file(emissions.xml).

• Use pycharm or a similar IDE environment that can be used to

• make dictionaries(manual dictionary)
• identify the problems in the dictionary
• upload the changes made in the dictionary or other texts in Peter’s github

Creation of abbreviation dictionary from pdf

2. ip_3_8_urban_abb.xml: automated created by docanalysis using spacy.

method to create abb. dict.:
docanalysis tutorial for help

mkdir wiki_hackathon

cd wiki_hackathon

mkdir Chapter08

cd Chapter08

mkdir sections

cd sections

mkdir 0_main_body

• Place the html of your chapter inside the 0_main_body directory.

docanalysis --project_name wiki_hackathon --output dict_search_5.csv --make_json dict_search_5.json --make_ami_dict entities --extract_abb ip_3_8_urban_abb

where,

--project name – the name of the project (here, wiki_hackathon)

--output - a csv for dictionary search (not of our use, but required to be created)

--make_json - just enter this. Not of current use, but required.

--make_ami_dict – uses the entities created in the above command

--extract_abb - the abbreviation dictionary that is the output.

Validating the created dictionary:

python -m py4ami.pyamix DICT --dict <dict_path> -–validate

keywords/phrase: keyword are extracted by the help of gensim method

keywords.csv

Introduction.md

Table of contents.md

FAQs.md

Annotation of HTML using dictionaries:

• html is annotated with dict. by the help of py4ami

• method:

py4ami HTML --annotate --dict <dict_path> --inpath <html_path> --outpath <outdir_path> --color <color>

where,

• dict_path – dictionary used for annotation.

• html_path – html file to be annotated.

• outdir_path – output directory for annotated html file.

• color - color to highlight annotation (symbolic name, eg YELLOW, or RGB, eg '#ff7700').

annotatedfulltext.html

Chapter 08 google colab notebook : click here