The Regulations Challenge aims to push the boundaries of LLMs in understanding, interpreting, and applying regulatory knowledge in the finance industry. In this challenge, participants will participate in 9 tasks to explore key issues, including, but not limited to, regulatory complexity, ethical considerations, domain-specific terminology, industry standards, and interpretability. We welcome students, researchers, and practitioners who are passionate about finance and LLMs. We encourage participants to develop solutions that advance the capabilities of LLMs in addressing the challenges of financial regulations and industry standards.
These tasks assess the LLM's ability to handle different types of questions within the regulatory domain which follow:
- Abbreviation Recognition Task:
Goal: Match an abbreviation with its expanded form.
Input Template: "Expand the following acronym into its full form: {acronym}. Answer:"
- Definition Recognition Task:
Goal: Correctly define a regulatory term or phrase.
Input Template: "Define the following term: {regulatory term or phrase}. Answer:"
- Named Entity Recognition (NER) Task:
Goal: Ensure the output correctly identifies entities and places them into groups that the user specifies.
Input Template: "Given the following text, only list the following for each: specific Organizations, Legislations, Dates, Monetary Values, and Statistics: {input text}."
- Question Answering Task:
Goal: Ensure the output matches the correct answer to a detailed question about regulatory practices or laws.
Input Template: "Provide a concise answer to the following question: {detailed question}? Answer:"
- Link Retrieval Task:
Goal: Ensure the link output matches the actual law.
Input Template: "Provide a link for ____ law, Write in the format of ("{Law}: {Link}" or "{Law}: Not able to find a link for the law")"
- Certificate Question Task:
Goal: Select the correct answer choice to a question that may be based on additional context.
Input Template: "(This context is used for the question that follows: {context}). Please answer the following question with only the letter and associated description of the correct answer choice: {question and answer choices}. Answer:"
- XBRL Analytics Task:
Goal: Ensure the output strictly matches the correct answer to a detailed question about financial data extraction and application tasks via XBRL filings. These standardized digital documents contain detailed financial information.
Input Template: "Provide the exact answer to the following question: {detailed question}? Answer:"
- Common Domain Model (CDM) Task:
Goal: Deliver precise responses to questions about the Fintech Open Source Foundation's (FINOS) Common Domain Model (CDM).
Input Template: "Provide a concise answer to the following question related to Financial Industry Operating Network's (FINO) Common Domain Model (CDM): {detailed question}? Answer:"
- Model Openness Framework (MOF) Licenses Task:
Goal: Deliver precise responses to questions concerning the requirement of license under the Model Openness Framework.
Input Template: "Provide a concise answer to the following question about MOF's licensing requirements: {detailed question}? Answer:"
The final score is determined by the weighted average of metrics for 9 tasks. We assign the weight of 10% to Task 1-5 each, 20% to Task 6, and 10% to Task 7-8 each.
Registration
We welcome students, researchers, and industry practitioners who are passionate about finance and LLMs to participate in this challenge. Winners will have the chance to present their work at the FinNLP-FNP-LLMFinLegal Workshop, happening on January 19-20, 2025, during the COLING 2025 Conference.
Register: Here
Important Dates
Training Set Release: September 15, 2024
Training Data Details: Summary of Question Dataset
Validation Set Release: October 30, 2024
Systems Submission: November 7, 2024
Release of Results: November 12, 2024
Paper Submission Deadline: November 25, 2024
Notification of Acceptance: December 5, 2024
Camera-ready Paper Deadline: December 13, 2024
Workshop Date: January 19-20, 2025
Task Organizers
Keyi Wang, Columbia University, Northwestern University
Lihang (Charlie) Shen, Columbia University
Haoqiang Kang, Columbia University
Xingjian Zhao, Rensselaer Polytechnic Institute
Namir Xia, Rensselaer Polytechnic Institute
Christopher Poon, Rensselaer Polytechnic Institute
Jaisal Patel, Rensselaer Polytechnic Institute
Andy Zhu, Rensselaer Polytechnic Institute
Shengyuan Lin, Rensselaer Polytechnic Institute
Daniel Kim, Rensselaer Polytechnic Institute
Jaswanth Duddu, Rensselaer Polytechnic Institute
Matthew Tavares, Rensselaer Polytechnic Institute
Shanshan Yang, Stevens Institute of Technology
Sai Gonigeni, Stevens Institute of Technology
Kayli Gregory, Stevens Institute of Technology
Katie Ng, Stevens Institute of Technology
Andrew Thomas, Stevens Institute of Technology
Dong Li, FinAI
Supervisors
Yanglet Xiao-Yang Liu, Rensselaer Polytechnic Institute, Columbia University
Steve Yang, School of Business at Stevens Institute of Technology
Kairong Xiao, Roger F. Murray Associate Professor of Business at Columbia Business School
Matt White, Executive Director, PyTorch Foundation. GM of AI, Linux Foundation
Cailean Osborne, University of Oxford
Wes Turner, Rensselaer Center for Open Source (RCOS), Rensselaer Polytechnic Institute
Neha Keshan, Rensselaer Polytechnic Institute
Luca Borella, PM of AI Strategic Initiative, FINOS Ambassador, Linux Foundation
Karl Moll, Technical Project Advocate, FINOS, Linux Foundation
For more details, please visit https://coling2025regulations.thefin.ai/ or contact colingregchallenge2025 [at] gmail.com