BloombergGPT: A Large Language Model for Finance

This article, originally published on the Bloomberg Engineering blog, delves into the application of natural language processing (NLP) in finance, specifically focusing on the development of BloombergGPT.

NLP plays a growing role in finance, enabling tasks like automated news summarization, sentiment analysis, and event extraction. These tasks demand robust NLP techniques to understand and process complex financial text.

BloombergGPT, a large pre-trained language model (LLM) developed at Bloomberg, aims to address these needs. It's built on OpenAI's GPT-2 and trained on a vast corpus of Bloomberg financial data. This model boasts 150 million parameters, encompassing 12 Transformer blocks and 768 hidden units. The training process leverages two tasks: masked language modeling (MLM) and next word prediction (NWP).

For wider accessibility, BloombergGPT has been open-sourced and made available on the Hugging Face model library, providing a powerful tool for researchers and developers in the financial domain.

While BloombergGPT is a powerful LLM, its training on a general corpus necessitates fine-tuning to adapt it to financial tasks.

Bloomberg utilizes its extensive financial data for this fine-tuning process, focusing on two key tasks:

• Financial Entity Recognition (FER): This task involves identifying entities within financial text, such as company names, stock tickers, and indices. Bloomberg's financial news and research reports are used for training.

• Financial Text Classification (FTC): The goal here is to categorize financial text into specific classes like company earnings reports or macroeconomic indicators. Again, Bloomberg's financial news and research reports serve as the training data.

The fine-tuning process employs the AdamW adaptive optimization algorithm along with learning rate warm-up and decay strategies for optimization.

The fine-tuned BloombergGPT model is also available on the Hugging Face model library, enabling researchers and developers to leverage its capabilities in financial applications.

In conclusion, BloombergGPT represents a significant advancement in the field of NLP for finance. Its availability as an open-sourced, fine-tuned model empowers researchers and developers to explore a wide range of financial NLP tasks. This innovative approach promises to drive further advancements in understanding and leveraging financial information.