CoNLL to JSON: Convert Named Entity Annotation Format

This code provides a solution for converting named entity annotation data from the CoNLL format to JSON format. It utilizes two functions:

• read_conll_file(filename): This function reads a CoNLL file and parses it into a list of sentences. Each sentence is represented as a list of tuples, where each tuple contains the word, part-of-speech tag, and named entity tag.
• conll_to_json(sentences): This function takes a list of sentences in the CoNLL format and converts it to a JSON list. Each sentence is represented as a dictionary containing two keys: 'words' and 'ner_tags'. 'words' holds a list of words from the sentence, while 'ner_tags' contains a list of corresponding named entity tags.

The code uses a standard approach to parse CoNLL files, iterating through lines and recognizing sentence boundaries using empty lines. It then processes each line to extract the necessary information. In the conll_to_json function, it iterates through the sentences and constructs the JSON structure by mapping the words and tags. The final JSON output is formatted for readability using the json.dumps function with indentation.

import json

def read_conll_file(filename):
    sentences = []
    with open(filename, 'r') as f:
        sentence = []
        for line in f:
            line = line.strip()
            if line == '':
                sentences.append(sentence)
                sentence = []
            else:
                tokens = line.split()
                word = tokens[0]
                pos = tokens[1]
                ner = tokens[3]
                sentence.append((word, pos, ner))
        if sentence:
            sentences.append(sentence)
    return sentences

def conll_to_json(sentences):
    data = []
    for sentence in sentences:
        words = []
        ner_tags = []
        for token in sentence:
            word = token[0]
            ner = token[2]
            words.append(word)
            ner_tags.append(ner)
        sentence_data = {'words': words, 'ner_tags': ner_tags}
        data.append(sentence_data)
    return data

conll_file = 'example.conll'
sentences = read_conll_file(conll_file)
json_data = conll_to_json(sentences)

print(json.dumps(json_data, indent=2))

This code provides a simple and effective way to convert CoNLL data into a JSON format suitable for various natural language processing tasks.