ISP Test Results

Diffusion 2D, Monte Carlo, and Game of Life ISP Tests

Test Name ISP Results
diff2.c



This is the Diffusion 2D example.
# Processes: 4
No Deadlocks
1 Interleaving
output
monte.c



This is the actual, unmodified, monte carlo code.

# Processes: 4
No Deadlocks
6 Interleavings
output

# Processes: 5
No Deadlocks
24 Interleavings
output

# Processes: 6
No Deadlocks
120 Interleavings
output

Game of Life (mlife)



In this example, ISP looks for memory leaks. This example was
presented at EuroPVM/MPI 2007 by Gropp and Lusk as a tutorial
on advanced MPI-2, and has 522 lines of C code. We provide the
results of running ISP on this example.

# Processes: 4
No Deadlocks
1 Interleaving
output

# Processes: 9
No Deadlocks
1 Interleaving
output

Game of Life (mlife2d)



In this example, ISP looks for memory leaks. This example was
presented at EuroPVM/MPI 2007 by Gropp and Lusk as a tutorial
on advanced MPI-2, and has 552 lines of C code. We provide the
results of running ISP on this example.

# Processes: 4
No Deadlocks
1 Interleaving
output

# Processes: 9
No Deadlocks
1 Interleaving
output

ISP, Marmot, and MPICH Comparison Results

Acknowledgements
We are grateful to Dr. Bronis R. de Supinski of the Lawrence Livermore National Laboratory for providing us the tests labeled "Umpire Tests" below.

Disclaimer
These experiments are being reported in the best interest of comparing tools on a scientific basis. No implication of the superiority of ISP over Marmot is expressed or intended. In fact, ISP is a new tool needing lot more improvements. These experiments only show that with a model-checking based search using representative interleavings (as provided by partial order reductions - our POE algorithm in particular) we seem to be able to find certain bugs always - as captured by these tests. Similar advantages of model checkers over testing tools are well known in model checking - our data is only to highlight that the same holds for MPI runtime verification also.

The table below represents the running of the Umpire test cases on three different platforms. All of the tests provide a copy-and-paste of the console session in which the tests where run. There weren't any special compile flags for Marmot and MPICH. I used the "marmotcc" and "mpicc" scripts in order to compile the tests.

The tests that end in "_mod" are slightly modified versions of the associated Umpire test. The modification is just a swap of ranks 1 and 2.

# Test Name ISP using MPICH2 1.0.7 Marmot 2.0.0 using MPICH2 1.0.7 MPICH 1.2.6
1 any_src-can-deadlock.c No Deadlock Detected
2 Interleavings
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
2 any_src-can-deadlock10.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 7 / 10 Runs
output
Deadlocked in 8 / 10 Runs
output
2.1 any_src-can-deadlock10_mod.c Deadlock Detected
2 Interleavings
output
Caught Deadlock in 5 / 10 Runs
output
Deadlocked in 3 / 10 Runs
output
3 any_src-can-deadlock11.c Deadlock Detected
2 Interleavings
output
Caught Deadlock in 5 / 10 Runs
output
Deadlocked in 3 / 10 Runs
output
4 any_src-can-deadlock2.c No Deadlock Detected
2 Interleavings
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
5 any_src-can-deadlock3.c No Deadlock Detected
2 Interleavings
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
6 any_src-can-deadlock4.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 4 / 10 Runs
output
Deadlocked in 6 / 10 Runs
output
6.1 any_src-can-deadlock4_mod.c Deadlock Detected
2 Interleavings
output
Caught Deadlock in 3 / 10 Runs
output
Deadlocked in 5 / 10 Runs
output
7 any_src-can-deadlock5.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 6 / 10 Runs
output
Deadlocked in 6 / 10 Runs
output
7.1 any_src-can-deadlock5_mod.c Deadlock Detected
2 Interleavings
output
Caught Deadlock in 6 / 10 Runs
output
Deadlocked in 4 / 10 Runs
output
8 any_src-can-deadlock6.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 6 / 10 Runs
output
Deadlocked in 9 / 10 Runs
output
8.1 any_src-can-deadlock6_mod.c Deadlock Detected
2 Interleavings
output
Caught Deadlock in 4 / 10 Runs
output
Deadlocked in 3 / 10 Runs
output
9 any_src-can-deadlock7.c Deadlock Detected
2 Interleavings
output
Caught Deadlock in 5 / 10 Runs
output
Deadlocked in 2 / 10 Runs
output
10 any_src-can-deadlock8.c Deadlock Detected
2 Interleavings
output
Caught Deadlock in 6 / 10 Runs
output
Deadlocked in 3 / 10 Runs
output
11 any_src-can-deadlock9.c Deadlock Detected
2 Interleavings
output
Caught Deadlock in 5 / 10 Runs
output
Deadlocked in 1 / 10 Runs
output
12 any_src-deadlock.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
13 any_src-waitall-deadlock.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
14 any_src-waitall-deadlock2.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
15 any_src-waitall-deadlock3.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
16 any_src-waitany-deadlock.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
17 any_src-waitany-deadlock2.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
18 any_src-wait-deadlock.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
19 any_src-wait-deadlock2.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
20 basic-deadlock.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
21 basic-deadlock-comm_create.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
22 basic-deadlock-comm_dup.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
23 basic-deadlock-comm_split.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
24 bcast-deadlock.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
25 change-send-buffer.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 10 Runs
output
No Deadlock in 30 Runs
output
26 collective-misorder.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
27 collective-misorder2.c Deadlock Detected
1 Interleaving
output
Didn't Catch Deadlock in 20 Runs
output
Didn't Deadlock in 30 Runs
output
28 collective-misorder-allreduce.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
29 comm-bcast-deadlock.c Deadlock Detected
1 Interleaving
output
Didn't Catch Deadlock in 20 Runs
output
Didn't Deadlock in 30 Runs
output
30 comm-deadlock.c Deadlock Detected
1 Interleaving
output
Didn't Catch Deadlock in 20 Runs
output
Didn't Deadlock in 30 Runs
output
31 comm-dup-no-error.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
32 comm-dup-no-free.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
33 comm-dup-no-free2.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
34 comm-simple.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
35 comm-split-no-free.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
36 comm-translate-ranks.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
37 complex-deadlock.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
38 deadlock-config.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
Deadlocked in 10 / 10 Runs
output
38.1 deadlock-config.c In ISP's default mode, it does not catch the deadlock. A big enough buffer is assumed,
so MPI_Send does not block. If the blocking MPI_Send option is enabled,
it will catch the deadlock, as seen here. It is recommended to run the program using both options.
39 dropped-req.c Deadlock Detected
1 Interleaving
output
Didn't Catch Deadlock in 15 Runs
output
Didn't Deadlock in 30 Runs
output
40 finalize-deadlock.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
41 hello.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
42 irecv-deadlock.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
43 irecv-isend-ok.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
44 irecv-isend-ok2.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
45 lost-request-waitall.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 10 Runs
output
No Deadlock in 30 Runs
output
46 no-error.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
47 no-error2.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
48 no-error3.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
49 no-error3-any_src.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
50 no-error-any_src.c No Deadlock Detected
2 Interleavings
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
51 no-error-interleaved-isend.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
52 no-error-test.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
53 no-error-testall.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
54 no-error-testany.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
55 no-error-vector-isend.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
56 no-error-wait.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
57 no-error-waitall-any_src2.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
58 no-error-waitall-any_src3.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
59 no-error-waitany.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
60 no-error-wait-any_src.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
61 no-error-wait-any_src2.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
62 no-error-wait-any_src3.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
63 no-error-wait-any_src4.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
64 no-error-waitany-any_src.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
65 partial-recv.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
66 sendrecv-deadlock.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
67 send-recv-ok.c No Deadlock Detected
1 Interleaving
output
No Deadlock in 20 Runs
output
No Deadlock in 30 Runs
output
68 waitall-deadlock.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
69 waitany-deadlock.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output
70 wait-deadlock.c Deadlock Detected
1 Interleaving
output
Caught Deadlock in 10 / 10 Runs
output
Deadlocked in 10 / 10 Runs
output