International Planning Competition 2023 Classical Tracks

IPC 2023 Classical Tracks

International Planning Competition 2023 Classical Tracks

This is the website for the classical (sequential, deterministic) track of the IPC 2023. This is the 10th IPC containing classical tracks making it the oldest part of IPC.

Results

Result Presentation

The results were presented at the 33rd International Conference on Automated Planning and Scheduling on July 11 in Prague. The presentation slides of this talk contain additional details.

Optimal Track

Optimal Track Results Table
folding labyrinth quantum. recharg. ricochet. rubiks slither. SUM
ragnarok 8 8 13 14 17 10 7 77
scorpion-2023 8 5 14 14 17 10 6 74
odin 8 5 13 14 17 10 6 73
dofri 8 5 13 13 17 10 4 70
cegarplusplus 9 5 13 14 17 0 7 65
hapori-stonesoup-opt 7 2 13 14 9 11 6 62
fdss-2023-opt 7 3 13 13 12 9 4 61
hapori-mip2-opt 7 1 13 14 9 10 6 60
hapori-ibacop2-opt 6 1 13 12 15 7 4 58
hapori-greedy-opt 5 1 13 11 9 10 7 56
baseline-blind 7 1 7 12 11 8 4 50
decstar-opt 6 1 12 11 8 8 4 50
hapori-delfi-opt 5 2 12 12 8 0 2 41
complementary 5 1 12 13 3 0 3 37
decabstar 2 1 12 10 7 0 5 37
symk 3 1 9 13 4 0 7 37
fts-ms-opt 1 1 12 13 2 0 7 36
baseline-lmcut 2 1 12 8 5 0 6 34
hapori-epslr-opt 2 1 9 6 4 10 2 34
SymBD-2023-opt 2 1 9 13 1 0 6 32
dom-opt 2 1 12 6 4 0 6 31
hapori-epsdt-opt 1 0 9 6 4 7 4 31
dalai-opt 2 1 11 7 4 0 4 29
fts-sbd-opt 1 0 4 13 0 0 4 22

Satisficing Track

Satisficing Track Results Table
folding labyrinth quantum. recharg. ricochet. rubiks slither. SUM
maidu-sat 6.80 0.00 19.60 13.94 11.36 14.16 6.00 71.86
levitron-sat 8.66 0.00 19.60 13.94 11.44 14.16 4.00 71.79
fdss-2023-sat 8.95 0.00 19.49 13.80 8.51 14.13 6.00 70.88
baseline-lama 9.70 1.00 17.86 13.19 9.83 12.18 5.00 68.76
hapori-ibacop2-sat 8.69 3.91 17.07 13.46 7.37 11.22 6.00 67.72
disco-sat 8.58 0.00 17.78 13.95 9.14 11.77 5.00 66.24
cerberus-sat 6.39 0.00 18.03 10.43 12.73 10.93 6.00 64.52
spock 5.83 0.00 19.67 13.94 5.01 13.90 6.00 64.36
decstar-sat 5.40 0.00 17.33 13.29 8.41 12.42 6.00 62.85
tftm-argmax-sat 6.73 0.00 16.74 9.55 12.73 9.98 6.00 61.73
tftm-co1-sat 6.67 0.00 15.84 10.43 12.65 10.10 6.00 61.68
hapori-mip2-sat 5.64 3.91 18.86 13.28 5.04 10.00 4.00 60.74
dalai-sat 4.84 4.00 17.29 13.00 8.82 5.00 3.00 55.95
ApxNoveltyAnytime 5.00 0.00 18.31 8.00 15.32 5.00 4.00 55.63
hapori-stonesoup-sat 7.69 3.91 18.44 12.32 4.17 4.00 4.00 54.52
NoveltyFBAnytime 1.00 0.00 18.31 8.00 13.80 5.00 4.00 50.12
opcount4sat-sat 2.00 3.76 13.32 10.11 6.15 0.00 4.00 39.34
hapori-epsdt-sat 7.40 14.14 0.00 5.38 0.62 6.03 0.00 33.57
hapori-delfi-sat 3.81 0.00 12.52 8.85 2.35 4.00 1.00 32.54
FSM 3.03 0.00 6.60 8.22 3.00 6.00 3.00 29.84
powerlifted-sat 9.69 0.00 16.71 0.00 0.00 0.00 2.00 28.40
hapori-epslr-sat 1.00 0.00 0.00 3.45 0.00 11.00 5.00 20.45
hapori-greedy-sat 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Agile Track

Agile Track Results Table
folding labyrinth quantum. recharg. ricochet. rubiks slither. SUM
baseline-lama-first 3.35 0.00 16.86 3.79 0.94 13.16 2.19 40.28
decstar-agl 2.62 0.00 15.26 5.01 2.72 12.95 1.70 40.25
fdss-2023-agl 2.96 1.07 15.31 4.47 0.56 11.07 2.39 37.82
ApxNoveltyTarski 1.91 2.66 19.53 3.92 3.41 3.13 1.80 36.35
disco-agl 3.39 0.00 16.82 6.47 1.08 6.30 1.91 35.96
maidu-agl 2.52 0.28 15.57 4.41 1.57 9.64 1.93 35.93
levitron-agl 2.49 0.34 15.64 4.11 1.00 9.48 2.55 35.61
ApxNovelty 1.43 0.00 19.64 3.03 3.41 4.00 1.81 33.32
cerberus-agl 2.74 0.00 13.47 2.38 1.83 7.79 2.64 30.84
dalai-agl 3.05 0.34 13.64 6.57 3.54 2.67 1.01 30.81
hapori-stonesoup-agl 0.87 0.74 16.70 3.89 0.81 3.86 1.62 28.49
hapori-mip2-agl 0.37 0.77 13.60 2.47 0.82 7.12 2.67 27.82
tftm-argmax-agl 2.73 0.00 12.15 2.45 1.74 6.01 2.64 27.73
tftm-co1-agl 2.56 0.00 12.26 2.37 1.89 6.01 2.61 27.70
NoveltyFB 0.14 0.00 18.88 0.83 1.18 4.00 0.48 25.50
fts-ff-agl 0.00 0.00 9.61 5.19 0.55 0.00 1.97 17.31
hapori-epsdt-agl 1.84 2.65 0.00 2.82 0.51 5.20 0.00 13.03
hapori-epslr-agl 0.01 0.00 0.00 2.48 0.00 7.46 0.98 10.94
hapori-delfi-agl 0.68 0.00 5.06 2.40 1.53 0.64 0.52 10.83
opcount4sat-agl 0.00 0.04 3.04 6.28 0.00 0.00 0.74 10.10
FSM 0.00 0.00 0.82 0.95 0.24 3.17 0.09 5.26
powerlifted-agl 0.73 0.00 0.00 0.00 0.00 0.00 0.07 0.80
hapori-greedy-agl 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Domains

IPC 2023 Dataset

Repository with all competition tasks can be found here.

List of competition domains:

Using IPC 2023 planners

All participating planners can be used as Apptainer images. To build such an image, install Apptainer and clone the corresponding planner repository. You can find links to them in the list of participants. Code repositories contain two branches: ipc2023-classical contains exactly the code version that ran in the IPC, while latest can contain additional bug fixes published after the competition. We recommend using latest for all experiments. Some code repositories contain multiple Apptainer files for different planners that share a code base. Use Apptainer to build the planner from one of those recipes like this:

git clone https://github.com/ipc2023-classical/planner8.git
cd planner8
sudo apptainer build maidu_sat.sif Apptainer.maidu_sat

Some planners require CPLEX to run. To use them, you have to acquire a CPLEX license (IBM offers free academic licenses) and download the installer file cplex_studio2211.linux_x86_64.bin. Place it into a directory and use the environment variable IPC_THIRD_PARTY to identify it.

export IPC_THIRD_PARTY=/path/to/some/directory
cp cplex_studio2211.linux_x86_64.bin $IPC_THIRD_PARTY

git clone https://github.com/ipc2023-classical/planner17.git
cd planner17
sudo apptainer build ragnarok.sif Apptainer.ragnarok

All planners are also available through planutils.

Calls

Please forward the following calls to all interested parties.

Preliminary Schedule

Event Date
Call for domains August 1, 2022
Call for participation October, 2022
Domain expression of interest deadline September 30, 2022
Domain submission deadline December 9, 2022
Demo problems provided December, 2022
Planner registration January 13, 2023
Feature stop (final planner submission) March 24, 2023
Planner Abstract submission deadline April 21, 2023
Contest run May - June, 2023
Results announced July, 2023
Result analysis deadline October 31, 2023

Tracks

Optimal Track

Satisficing Track

Agile Track

PDDL Fragment

IPC 2023 will use a subset of PDDL 3.1, as done since IPC 2011. Planners must support the subset of the language involving STRIPS, action costs, negative preconditions, and conditional effects (possibly in combination with forall, as in IPC 2014 and 2018). We will also consider including domains with disjunctive preconditions and existential quantifiers, in which case we provide an automatic translation compiling these features away, and we run all planners on both variants and select the best result per domain.

Most planners in previous IPCs rely on a grounding procedure to instantiate the entire planning task prior to start solving it. In IPC 2023, we will follow in the steps of the previous IPC by including domains and problems that are hard to ground.

Participants

Optimal Track

Satisficing Track

Agile Track

Registration

To register a team, the participants need to send an e-mail with a subject containing "[Registration]" to ipc2023-classical@googlegroups.com. The e-mail must contain:

  1. names of participants,
  2. e-mail contacts,
  3. github usernames,
  4. the number of repositories (code bases) the team needs (multiple planners can be built from the same repository),
  5. a (tentative) list of tracks, where the team intends to submit their planners; if you are interested in a different track than optimal, satisficing, or agile, please, let us know too.

Based on that, we will create private repositories under the ipc2023-classical organization and add all participants as users with with write access and participants can commit to the repository as they wish until the "feature stop" deadline (March 10, 2023).

Planner Submission

As in previous editions, the competitors must submit the source code of their planners that will be run by the organizers on the actual competition domains/problems, unknown to the competitors until this time. This way no fine-tuning of the planners will be possible.

As in the previous IPC 2018, we will use the container technology Apptainer (formerly known as Singularity) to promote reproducibility and help with compilation issues that have caused problems in the past. In contrast to the previous IPC, we will host repositories of planners ourselves. The repositories will be hosted on Github under the ipc2023-classical organization, and they will be kept private until the end of the competition when we make them public, i.e., after the competition is concluded, we plan to make all planners, domains, and all related data accessible from one place.

This year, we will also allow to submit multiple planners to multiple tracks from a single repository. In each repository, we only consider the branch ipc2023-classical. Feel free to use other branches for development as you wish, but we will ignore them. Any file called Apptainer.<shortname> in the root directory of this branch defines one entry. For the <shortname>, please use the name and variant of your planner as a short identifier (a single word, up to 16 characters long, starting with a letter, using only letters, digits, and underscores). If you build different versions of your planner from the same repository, use a different <shortname> per version. A single entry can participate in multiple tracks, see "Apptainer Images" for details.

We got many registrations and in particular many registrations with multiple submissions in a team. To keep the computation time manageable, we ask that you limit your submissions to one variant per planner. If two of your submissions are substantially different ideas, it is of course fine to submit both. As a guideline, if two submissions would be described with two planner abstracts, then they are two submissions. If you would describe them in one planner abstract with a short paragraph describing the differences between them, they are variants of the same planner.

After some discussions, we decided to allow multiple variants of the same planner if you think they are likely end up on very different spots in scoring (i.e., not next to each other). In that case, you should still write two planner abstracts, but one may discuss the difference to the other submission. In that case, we ask you to write a paragraph into the second planner abstract that discusses your reasons of why you think this should be a separate submission, i.e., why you think these two submissions will not end up next to each other in the scoring. In the post-IPC analysis, we then ask you to evaluate this reasoning: if the planners indeed ended up in very different spots, what made the difference? If they ended up next to each other, why did this happen? Moreover, if the leaderboard ends up with multiple variants of the same planner in the first places, we reserve the right to merge these entries. New variants can be submitted until April 16 via pull requests.

Apptainer Images

We prepared a demo submission that showcases how to set up the repository and Apptainer scripts.

Your Apptainer recipe files have to specify the following labels:

To improve reproducibility, we require Apptainer images to be self-contained and licensed appropriately.

In addition to reproducibility and licensing issues, we ask that you make your image as small as possible using the following tricks:

Bug-fixing Policy

When a competition team registers (see above), we create a private repository (or multiple repositories if needed) and add competitors as users with write access. After the “feature stop” deadline (March 10, 2023), we allow competitors to send only a pull request with bug fixes. We will review every pull request with its accompanying description of the bug fix to make sure that no big changes or parameter tuning is committed to the repository.

To help us with the debugging process, in contrast to previous years, planner authors will be responsible for detecting if the run of their planner and our analysis of the results was successful. After the feature stop deadline, we will run all planners on all tasks and give the participants access to the results of their planners. For each run, the data will contain the log files of the planner, measured time and memory consumption, exit code, and our conclusion about what this means in terms of solving the instance. We ask participants to check their results for any errors. If an error was caused by a bug in the planner, please send a pull request on Github with a detailed description of the bug and the fix. If the error was on our side (e.g., malformed PDDL) let us know as soon as possible. We will do at least two rounds of this starting after the "feature stop" deadline (exact timeline TBD).

Planner Abstract Submission

All competitors must submit an abstract (max. 300 words) and an up to 8-page paper describing their planners. After the competition we ask the participants to analyze the results of their planner and submit an extended version of their paper. An important requirement for IPC 2023 competitors is to give the organizers the right to post their paper and the source code of their planners on the official IPC 2023 web site, and the source code of submitted planners must be released under a license allowing free non-commercial use.

Organizers

Contact us: ipc2023-classical@googlegroups.com