TRUSTING IPSLCM6.2_HEADLMDZ at TGCC irene-skylake


by J. Ghattas and Th. Lurton, IPSL
This trusting evaluates the technical issues with the coupled configuration IPSLCM6.2 at irene-skylake/TGCC.

It forces an extraction of the latest version (HEAD) of the trunk branch of development for LMDZ.

Each line in the grey table corresponds to a specific installation of IPSLCM6.2 and all related tests. The revision number of
modipsl is set in column 2 and the revsion numbers for the other components are listed in the end of the line.
The 7th column shows the reference revision of modipsl which is used for comparing results for the individual tests.

The different tests listed in the table below are given on following syntax:
XXXXyz : XXXX= subconfig, y=p/d compilation in prod or debug mode, z=1 or 2 for different tests with different PeriodLength.
For example
LRp1 - subconfig LR with production mode compiled execuatble test 1 with PeriodLength=5D, total lenght 10days.
LRp2 - subconfig LR with production mode compiled execuatble test 2 with PeriodLength=10D, total lenght 10days.
LRd1 - subconfig LR with debug mode compiled execuatble test 1 with PeriodLength=1D, total lenght 2days.

The tests of type LRp1 = LRp2, are validating the restartability of the model. Both tests have the same total length but different PeriodLenth. The results should be the same in the end of the run.
The RealCpu time noted for each test concerns the last executed period only.

2023-04-19T03:30

IOIPSL svn checkout -r 6273 --username icmc_users https://forge.ipsl.jussieu.fr/igcmg/svn/IOIPSL/tags/v2_2_5@6273 IOIPSL
ORCHIDEE svn checkout -r 7820 svn://forge.ipsl.jussieu.fr/orchidee/branches/ORCHIDEE_2_2/ORCHIDEE@7820 ORCHIDEE
oasis3-mct svn checkout -r 4775 --username icmc_users https://forge.ipsl.jussieu.fr/igcmg/svn/CPL/oasis3-mct/branches/OASIS3-MCT_2.0_branch@4775 oasis3-mct
LMDZ svn checkout -r HEAD https://svn.lmd.jussieu.fr/LMDZ/LMDZ6/trunk@HEAD LMDZ
IPSLCM6 svn checkout -r HEAD --username icmc_users https://forge.ipsl.jussieu.fr/igcmg/svn/CONFIG/UNIFORM/v6/IPSLCM6.2@HEAD IPSLCM6
libIGCM svn checkout -r 1581 --username icmc_users https://forge.ipsl.jussieu.fr/libigcm/svn/trunk/libIGCM@1581 libIGCM
https://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE/NEMOGCM@9455 svn checkout -r 9455 https://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE/NEMOGCM@9455
https://forge.ipsl.jussieu.fr/shaconemo/svn/trunk/ORCA1_LIM3_PISCES@307 svn checkout -r 307 https://forge.ipsl.jussieu.fr/shaconemo/svn/trunk/ORCA1_LIM3_PISCES@307
https://forge.ipsl.jussieu.fr/shaconemo/svn/trunk/ORCA025_LIM3_PISCES@307 svn checkout -r 307 https://forge.ipsl.jussieu.fr/shaconemo/svn/trunk/ORCA025_LIM3_PISCES@307
INCA svn checkout -r 1079 --username inca https://forge.ipsl.jussieu.fr/inca/svn/trunk/INCA6@1079 INCA
XIOS svn checkout -r 2439 https://forge.ipsl.jussieu.fr/ioserver/svn/XIOS2/trunk@2439 XIOS


Date modipsl revision Status Compilation Other information Trusting level Reference modipsl revision All tests => LRp1 (prod,5D) LRp2 (prod,10D) LRd1 (debug,1D) LRp1 = LRp2 LMDZORp1 (prod,5D) LMDZORp2 (prod,10D) LMDZORd1 (debug,1D) LMDZORp1 = LMDZORp2 ESMCO2p1 (prod,5D) ESMCO2p2 (prod,10D) ESMCO2d1 (debug,1D) ESMCO2p1 = ESMCO2p2 MR025p1 (prod,1D) ESMAERp1 (prod,5D) ESMAERp2 (prod,10D) ESMAERd1 (debug,1D) ESMAERp1 = ESMAERp2 . . . IOIPSL ORCHIDEE oasis3-mct LMDZ IPSLCM6 libIGCM NEMOGCM ORCA1_LIM3_PISCES ORCA025_LIM3_PISCES INCA XIOS XIOS/arch ... ..
2021-11-19T18:08 5980 FAILED OK 8 5860 Restart OK. RealCpuTime=130.050 Restart OK. RealCpuTime=203.680 Fatal Restart OK. Restart OK. RealCpuTime=71.870 Restart OK. RealCpuTime=234.590 Fatal Restart OK but err=1. Restart OK. RealCpuTime=138.800 Restart OK. RealCpuTime=230.250 Fatal Restart OK but err=2. solver.stat OK Restart OK. RealCpuTime=314.700 Restart OK. RealCpuTime=544.070 Fatal Restart OK. 5635 7086 4775 4014 5645 1545 9455 303 303 915 2029
2021-11-19T18:08 5980 FAILED OK Manually changed OK_COSP=n in -debug experiments. 8 5860 Restart OK. RealCpuTime=130.050 Restart OK. RealCpuTime=203.680 Fatal Restart OK. Restart OK. RealCpuTime=71.870 Restart OK. RealCpuTime=234.590 Fatal Restart OK but err=1. Restart OK. RealCpuTime=138.800 Restart OK. RealCpuTime=230.250 Fatal Restart OK but err=2. solver.stat OK Restart OK. RealCpuTime=314.700 Restart OK. RealCpuTime=544.070 Fatal Restart OK. 5635 7086 4775 4014 5645 1545 9455 303 303 915 2029
2021-12-20T15:38 6022 FAILED OK Error in checkout 8 5860 Restart OK. RealCpuTime=128.000 Restart OK. RealCpuTime=215.320 Fatal Restart OK. Restart OK. RealCpuTime=81.110 Restart OK. RealCpuTime=236.410 Fatal Restart OK but err=1. Restart OK. RealCpuTime=121.340 Restart OK. RealCpuTime=215.950 Fatal Restart OK but err=2. solver.stat OK Restart OK. RealCpuTime=346.090 Restart OK. RealCpuTime=542.310 Fatal Restart OK.
2022-01-21T14:24 6048 FAILED OK 8 5860 Restart OK. RealCpuTime=149.530 Restart OK. RealCpuTime=252.010 Fatal Restart OK. Restart OK. RealCpuTime=96.890 Restart OK. RealCpuTime=273.700 Fatal Restart OK but err=1. Restart differ err=1993. RealCpuTime=143.990 Restart differ err=1996. RealCpuTime=297.440 Fatal Error: Restart differ err=1600. solver.stat differ Fatal Fatal Fatal Fatal. 5635 7086 4775 4058 5645 1545 9455 303 303 915 2029
2022-03-11T10:13 6101 FAILED OK 8 5860 Restart differ err=1868. RealCpuTime=128.700 Restart differ err=1868. RealCpuTime=209.890 Restart differ err=1831. RealCpuTime=293.310 Restart OK. Restart differ err=1315. RealCpuTime=72.870 Restart differ err=1315. RealCpuTime=244.580 Restart differ err=1190. RealCpuTime=159.710 Restart OK but err=1. Restart differ err=1993. RealCpuTime=127.880 Restart differ err=1993. RealCpuTime=226.350 Restart differ err=1945. RealCpuTime=297.740 Restart OK but err=2. solver.stat differ Restart differ err=4397. RealCpuTime=312.710 Restart differ err=4397. RealCpuTime=518.640 Restart differ err=4344. RealCpuTime=908.790 Restart OK. 6058 7086 4775 4092 6096 1545 9455 303 303 915 2029
2022-06-23T10:54 6183 FAILED FAILED 8 5860 Restart differ err=655. RealCpuTime=120.630 Restart differ err=655. RealCpuTime=658.420 Restart differ err=655. RealCpuTime=282.770 Restart OK. Restart differ err=664. RealCpuTime=253.640 Restart differ err=664. RealCpuTime=245.840 Restart differ err=663. RealCpuTime=148.280 Restart OK but err=1. Fatal Fatal Fatal Fatal. Fatal Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- 6058 7613 4775 4176 6129 1552 9455 307 307 1034 2331
2022-07-12T13:03 6198 FAILED FAILED 8 5860 Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- 6058 7613 4775 4199 6129 1552 9455 307 307 1034 2331
2022-07-19T03:30 6202 FAILED FAILED Problem with compile_ipslcm6.sh not supporting newest developments of LMDZ. 8 5860 Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- 6058 7613 4775 4199 6129 1552 9455 307 307 1034 2331
2022-07-20T09:30 6203 FAILED OK With manual tweak to override suffix problem in gcm executable name. Taken as new ref. 8 5860 Restart differ err=655. RealCpuTime=116.850 Restart differ err=655. RealCpuTime=199.520 Restart differ err=655. RealCpuTime=286.590 Restart OK. Restart differ err=664. RealCpuTime=76.970 Restart differ err=664. RealCpuTime=135.640 Restart differ err=663. RealCpuTime=149.620 Restart OK but err=1. Fatal Fatal Fatal Fatal. Fatal Restart differ err=3181. RealCpuTime=277.420 Restart differ err=3181. RealCpuTime=515.810 Restart differ err=3170. RealCpuTime=909.610 Restart OK. 6058 7613 4775 4199 6129 1552 9455 307 307 1034 2331
2022-07-22T20:44 6208 FAILED OK Testing new version of compile_ipslcm6.sh, courtesy of L. Fairhead. 8 6203 Restart OK. RealCpuTime=120.270 Restart OK. RealCpuTime=201.100 Restart OK. RealCpuTime=295.120 Restart OK. Restart OK. RealCpuTime=82.530 Restart OK. RealCpuTime=137.320 Fatal Restart OK but err=1. Fatal Fatal Fatal Fatal. Fatal Fatal Fatal Fatal Fatal. 6058 7613 4775 4224 6129 1552 9455 307 307 1034 2331
2022-08-19T03:30 6231 FAILED FAILED Compilation failed due to obsolete version of compile_ipslcm6.sh. 8 6203 Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- 6058 7613 4775 4229 6129 1552 9455 307 307 1034 2331
2022-09-01T15:10 6234 FAILED OK Now with corrected version of compile_ipslcm6.sh. 8 6203 Fatal Fatal Fatal Fatal. Fatal Fatal Fatal Fatal. Fatal Fatal Fatal Fatal. Fatal Fatal Fatal Fatal Fatal. 6058 7613 4775 4234 6212 1552 9455 307 307 1034 2331
2022-09-19T03:30 6239 FAILED OK 8 6203 Restart OK. RealCpuTime=120.030 Restart OK. RealCpuTime=212.060 Restart OK. RealCpuTime=285.560 Restart OK. Restart differ err=1316. RealCpuTime=80.120 Restart differ err=1316. RealCpuTime=136.760 Restart OK. RealCpuTime=155.340 Restart OK but err=1. Fatal Fatal Fatal Fatal. solver.stat OK Fatal Fatal Fatal Fatal. 6058 7613 4775 4250 6212 1552 9455 307 307 1034 2331
2022-10-19T03:30 6260 FAILED OK Taken as new ref. 8 6203 Restart OK. RealCpuTime=122.340 Restart OK. RealCpuTime=207.330 Restart OK. RealCpuTime=391.000 Restart OK. Restart differ err=1316. RealCpuTime=72.510 Restart differ err=1316. RealCpuTime=135.750 Restart OK. RealCpuTime=152.270 Restart OK but err=1. Restart differ err=300. RealCpuTime=117.110 Restart differ err=300. RealCpuTime=192.370 Restart differ err=300. RealCpuTime=292.270 Restart OK but err=2. solver.stat OK Fatal Fatal Fatal Fatal. 6058 7613 4775 4298 6212 1552 9455 307 307 1034 2331
2022-11-19T03:30 6267 FAILED FAILED 8 6260 Run not started Fatal Fatal Fatal. Fatal Fatal Fatal Fatal. Fatal Fatal Fatal Fatal. Fatal Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- 6058 7613 4775 4351 6263 1552 9455 307 307 1034 2331
2022-12-19T03:30 6287 FAILED FAILED 8 6260 Restart OK. RealCpuTime=121.520 Restart OK. RealCpuTime=202.580 Restart OK. RealCpuTime=282.370 Restart OK. Restart differ err=300. RealCpuTime=77.170 Restart differ err=300. RealCpuTime=148.180 Restart differ err=300. RealCpuTime=149.400 Restart OK but err=1. Restart differ err=1977. RealCpuTime=112.880 Restart differ err=1977. RealCpuTime=191.890 Restart differ err=1931. RealCpuTime=286.440 Restart OK but err=2. solver.stat differ Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- 6273 7820 4775 4374 6276 1565 9455 307 307 1057 2379
2023-01-19T03:30 6311 FAILED FAILED 8 6260 Fatal Fatal Fatal Fatal. Restart differ err=300. RealCpuTime=75.930 Restart differ err=300. RealCpuTime=148.840 Restart differ err=300. RealCpuTime=155.240 Restart OK but err=1. Fatal Fatal Fatal Fatal. solver.stat differ Compilation failed or not yet finished Compilation failed or not yet finished Compilation failed or not yet finished --- 6273 7820 4775 4386 6310 1565 9455 307 307 1057 2439
2023-02-19T03:30 6326 FAILED OK 8 6260 Fatal Fatal Fatal Fatal. Restart differ err=300. RealCpuTime=79.150 Restart differ err=300. RealCpuTime=147.240 Restart differ err=300. RealCpuTime=155.330 Error: Restart differ err=400. Fatal Fatal Fatal Fatal. solver.stat differ Fatal Fatal Fatal Fatal. 6273 7820 4775 4443 6316 1572 9455 307 307 1057 2439
2023-03-19T03:30 6342 FAILED OK 8 6260 Fatal Fatal Fatal Fatal. Restart differ err=300. RealCpuTime=77.390 Restart differ err=300. RealCpuTime=147.010 Restart differ err=300. RealCpuTime=157.670 Error: Restart differ err=400. Fatal Fatal Fatal Fatal. solver.stat differ Fatal Fatal Fatal Fatal. 6273 7820 4775 4473 6338 1572 9455 307 307 1057 2439
2023-04-19T03:30 6410 FAILED OK 8 6260 Fatal Fatal Fatal Fatal. Restart differ err=300. RealCpuTime=97.260 Restart differ err=300. RealCpuTime=145.590 Restart differ err=300. RealCpuTime=177.080 Error: Restart differ err=400. Fatal Fatal Fatal Fatal. solver.stat differ Fatal Fatal Fatal Fatal. 6273 7820 4775 4512 6385 1581 9455 307 307 1079 2439