IJCKG 2025 will organize evaluation challenges related to Knowledge Graph, aiming to provide researchers with a platform to test technologies, algorithms and systems. Evaluations challenge organizers of IJCKG 2025 can select the platform and evaluation plan by themselves, and we sincerely solicit evaluation challenges from researchers, research institutions and enterprises in related fields.
At IJCKG 2025, each evaluation challenge will receive one slot during the main conference, where organizers present the challenge, and participants present the solutions. At least one organizer must register and be present at the conference. Winners will receive a certificate and be invited to present their systems during the poster and demo session.
Topics
For IJCKG 2025, evaluation challenge proposals are invited for all tasks on Knowledge Graphs, including but not limited to:
Proposal Submission Guidelines
Proposals for evaluation challenges should be concise and include:
Timeline for Evaluation Challenge Organizers:
Evaluation Challenge Chair
Organizers
Description
Natural language interaction with databases in a more friendly and intuitive
way is a challenging work, which aims to translate natural language
questions into executable SQL statements. Some recent works have achieved
good performance on existing datasets, but they cannot efficiently perform
complex reasoning such as arithmetic, common sense, and hypothesis.
To this end, we propose Archer, a dataset that incorporates the above three
types of inference to make more complex and subtle queries. In addition,
we tested with both large language models and fine-tuned models.
Archer has three types of reasoning: arithmetic reasoning, commonsense
reasoning and hypothetical reasoning. Arithmetic reasoning has an important
proportion in the specific application scenarios of SQL.
Commonsense reasoning refers to the ability to reason based on implicit
commonsense knowledge, Archer contains some questions that require
understanding the database to infer missing details;
Hypothesis reasoning requires the model to have counterfactual thinking
ability, which is the ability to imagine and reason about unseen situations
based on visible facts and counterfactual hypotheses.
Website
https://sig4kg.github.io/archer-bench/
Social Media
Slack chat
https://join.slack.com/t/archer-ijckg2025/shared_invite/zt-3855g81oj-Ke0YaLuN3mAwjHrqLtUTXw
Challenge Prizes
Important Dates