1. 1 Hester van Eeren Erasmus Medical Centre, Rotterdam Halsteren, August 23, 2010

2. 2 The propensity score (1) The propensity score is “…the conditional probability of assignment to a particular treatment given a vector of observed covariates.” (Rosenbaum en Rubin, 1983: 41). Used in non-randomized studies to control for selection bias Balance observed pretreatment variables among patient Find an estimate of the average treatment effect But, treatment effect can be different within subgroups

3. 3 The propensity score (2) Univariate propensity scoreMultivariate propensity score Bartak and colleagues (2009)Spreeuwenberg and colleagues (2010) Used for 2 treatment categoriesUsed for > 2 treatment categories Propensity score used in: Matching Stratification Regression Inverse probability weight Combinations of …

4. Methods in this study To find a treatment effect within subgroups, if the propensity score is applied: Method 1: Regression analysis with propensity score, subgroups and interaction with treatment assignment; Method 2: Weighted regression analysis with inverse of the propensity score (to weight observations), subgroups and interaction with treatment assignment; Method 3: Propensity score applied for groups defined on treatment assignment and subgroups; then, regression analysis with propensity score and dummies for groups Two treatment categories and two subgroups are used in this study 4

5. Variable selection for propensity score Does the variable for subgroups has to be included? Discussion about variable selection for propensity score; Only variables related to outcome? Only variables related to treatment assignment? Both variables…? In this study; 8 different propensity scores (PS) formulated, based on: Variables related to outcome, with and without subgroup Variables related to treatment assigment, with and without subgroup Both variables…, with and without subgroup Only variables related to both outcome and treatment assignment, with and without subgroup 5

6. How to test? (1) Real dataset not useful because effects unknown beforehand;  You cannot test whether the effect found is accurate Monte Carlo simulation study to test methods and different propensity scores: Simulate data with known treatment effects Estimate different propensity scores for this data Apply different methods for different propensity scores, for this data  Repeat this process 1000 times 6

7. How to test? (2) What do you want to know? If the treatment effect estimated is (almost) equal to the treatment effect you used to simulate the data Bias of estimator: difference between estimated treatment effect and the true value of parameter Want to have an unbiased estimate; Less bias indicates a more accurate estimate of the treatment effect Bias is estimated for overall treatment effect and for the treatment effect within subgroups 7

8. Results 8 N=250;ρ=0N=250;ρ=0.3N=250; ρ=0.7N=500;ρ=0N=500;ρ=0.3N=500; ρ=0.7N=1000;ρ=0N=1000;ρ=0.3N=1000; ρ=0.7 RowMethodBiasMSEBiasMSEBiasMSEBiasMSEBiasMSEBiasMSEBiasMSEBiasMSEBiasMSE Regression 1PS1, func 1.0736.0243.0706.0265.0797.0304.0844.0160.0763.0159.08030.172.0812.0109.0786.0113.0834.0135 2PS1, func 2.0728.0237.0724.0258.0785.0288.0853.0156.0774.0156.0807.0169.0801.0104.0779.0110.0825.0130 3PS1, func 3 -.0060.0312-.0050.0336-.0014.0345.0042.0149-.0025.0162.0006.0168-.0001.0071-.0053.0080.0029.0092 4PS1, subgr. -.0035.0820-.0056.0762-.0008.0719.0030.0414.0007.0382.0000.0398.0006.0218.0085.0193-.0014.0178 5PS2, func 1.0746.0262.0710.0290.0798.0317.0860.0175.0745.0171.0804.0180.0813.0116.0787.0119.0838.0139 6PS2, func 2.0740.0257.0730.0281.0785.0298.0864.0171.0755.0166.0807.0175.0804.0112.0778.0116.0830.0135 7PS2, func 3 -.0049.0343-.0055.0371-.0023.0368.0070.0169-.0053.0183.0012.0178.0002.0083-.0061.0088.0033.0098 8PS2, subgr. -.0030.0901-.0025.0892.0020.0765-.0015.0467.0028.0448-.0014.0433.0006.0243.0105.0220-.0009.0196 9PS3, func 1.0727.0237.0723.0258.0787.0289.0852.0156.0774.0156.0807.0169.0802.0104.0779.0110.0824.0130 10PS3, func 2.0740.02370730.0258.0785.0289.0864.0156.0755.0156.0807.0169.0804.0104.0778.0110.0830.0130 11PS3, func 3 -.0062.0312-.0053.0337-.0017.0346.0039.0149-.0028.0162.0005.0168-.0001.0071-.0053.0080.0027.0091 12PS3, subgr. -.0030.0819-.0047.0763.0001.0719.0034.0414.0012.0383.0003.0399.0008.0218.0088.0190-.0011.0178 13PS4, func 1.0738.0257.0731.0281.0782.0299.0863.0170.0756.0166.0808.0175.0805.0112.0778.0116.0830.0135 14PS4, func 2.0739.0257.0730.0281.0782.0299.0862.0170.0756.0166.0807.0175.0805.0112.0779.0116.0830.0135 15PS4, func 3 -.0051.0343-.0056.0372-.0029.0370.0068.0169-.0053.0183.0011.0178.0002.0083-.0061.0088.0030.0098 16PS4, subgr. -.0027.0901-.0020.0894.0027.0766-.0012.0466.0033.0449-.0012.0433.0007.0243.0107.0220-.0006.0197 17PS5, func 1.0838.0345.0855.0383.0859.0357.0809.0201.0785.0213.0779.0196.0827.0138.0768..0126.0812.0130 18PS5, func 2.0836.0332.0850.0370.0872.0345.0806.0193.0769.0202.0778.0189.0824.0136.0766.0123.0800.0126 19PS5, func 3.0047.0407.0027.0440.0129.0396-.0018.0209-.0056.0209-.0026.0183-.0002.0104-.0002.0100.0000.0085 20PS5, subgr. -.0023.0898.0061.0895-.0135.0868.0064.0481.0066.0421.0013.0381.0067.0224-.0081.0220-.0006.0184 21PS6, func 1.0836.0332.0849.0369.0871.0346.0805.0193.0769.0202.0778.0188.0823.0136.0765.0123.0800.0126 22PS6, func 2.0836.0332.0850.0370.0871.0345.0806.0193.0768.0202.0778.0189.0823.0136.0765.0123.0801.0126 23PS6, func 3.0046.0407.0023.0439.0126.0395-.0019.0210-.0059.0209-.0027.0183-.0003.0104-.0004.0100-.0001.0085 24PS6, subgr. -.0021.0898.0069.0834-.0127.0869.0067.0481.0070.0421.0016.0381.0068.0224-.0079.0220-.0003.0184 25PS7, func 1.0839.0320.1611.0528.1663.0544.0791.0180.1561.0380.1584.0377.0829.0130.1561.0303.1632.0328 26PS7, func 2.0836.0307.1603.0518.1678.0536.0784.0173.1550.0369.1587.0373.0827.0129.1560.0301.1621.0322 27PS7, func 3.0045.0362.0790.0460.0948.0459-.0037.0180.0724.0242.0786.0235.0017.0093.0794.0154.0827.0152 28PS7, subgr. -.0013.0819.0037.0849-.0162.0828.0055.0435.0068.0402.0005.0379.0029.0207-.0085.0205-.0022.0185 29PS8, func 1.0837.0307.1602.0517.1679.0537.0783.0173.1551.0369.1587.0373.0827.0129.1560.0301.1621.0323 30PS8, func 2.0836.0307.1603.0517.1680.0538.0784.0173.1550.0369.1587.0373.0827.0129.1560.0301.1621.0323 31PS8, func 3.0043.0362.0787.0458.0947.0459-.0039.0181.0722.0242.0785.0235.0016.0093.0793.0154.0826.0152 32PS8, subgr. -.0008.0819.0046.0848.0059.0436.0073.0402.0009.0379.0009.0379.0031.0207-.0083.0205-.0019.0185 N=250;ρ=0N=250;ρ=0.3N=250; ρ=0.7N=500;ρ=0N=500;ρ=0.3N=500; ρ=0.7N=1000;ρ=0N=1000;ρ=0.3N=1000; ρ=0.7 RowMethodBiasMSEBiasMSEBiasMSEBiasMSEBiasMSEBiasMSEBiasMSEBiasMSEBiasMSE Regression 1PS1, func 1.0812.0272.1017.0301.0987.0321.0874.0173.1043.0208.0960.0208.0796.0113.1043.0158.1019.0160 2PS1, func 2.0780.0258.0735.0247.0715.0274.0851.0163.0746.0149.0687.0156.0796.0110.0751.0104.0763.0114 3PS1, func 3 -.0011.0322-.0022.0296-.0131.0358.0094.0157-.0040.0148-.0138.0175-.0002.0080-.0041.0075-.0109.0087 4PS1, subgr..0025.0806-.0087.0797.0253.0913-.0103.0426-.0009.0375.0186.0441-.0005.0202.0000.0184.0306.0218 5PS2, func 1.0824.0289.0792.0280.0762.0289.0866.0185.0799.0175.0741.0180.0781.0118.0780.0119.0802.0125 6PS2, func 2.0805.0273.0813.0280.0763.0293.0846.0175.0767.0170.0726.0169.0780.0115.0802.0117.0801.0124 7PS2, func 3 -.0009.0341.0035.0347-.0086.0380.0094.0179.0011.0170-.0102.0185-.0006.0092.0000.0086-.0072.0098 8PS2, subgr..0030.0863-.0047.0968.0248.0989-.0112.0475-.0012.0451.0183.0477-.0031.0224.0020.0208.0294.0246 9PS3, func 1.0798.0258.0790.0254.0769.0280.0852.0163.0733.0158.0735.0163.0795.0110.0803.0112.0807.0121 10PS3, func 2.0805.0258.0813.0253.0763.0279.0846.0163.0767.0158.0726.0162.0780.0110.0802.0112.0801.0120 11PS3, func 3 -.0014.0322.0018.0296-.0117.0359.0093.0157-.0002.0148-.0129.0175-.0003.0080-.0003.0075-.0105.0088 12PS3, subgr..0033.0806-.0050.0806.0352.0930-.0098.0426.0027.0377.0286.0451-.0002.0202.0034.0186.0400.0228 13PS4, func 1.0804.0273.0810.0277.0764.0289.0847.0175.0798.0170.0727.0170.0780.0115.0801.0117.0801.0124 14PS4, func 2.0805.0273.0810.0276.0760.0287.0847.0175.0797.0170.0728.0169.0780.0115.0800.0117.0800.0124 15PS4, func 3 -.0010.0342.0030.0347-.0088.0376.0093.0179.0010.0170-.0102.0185-0007.0092-.0003.0085-.0073.0097 16PS4, subgr..0033.0863-.0042.0972.0252.0992-.0109.0476-.0009.0450.0189.0478-.0029.0224.0021.0209.0294.0247 17PS5, func 1.0851.0343.0769.0326.0800.0330.0875.0213.0784.0201.0752.0191.0830.0139.0787.0134.0729.0118 18PS5, func 2.0845.0329.0758.0317.0786.0321.0873.0204.0789.0195.0744.0184.0830.0136.0779.0129.0743.0116 19PS5, func 3.0062.0410-.0024.0430-.0029.0403.0078.0201-.0016.0208-.0123.0185.0009.0097-.0038.0105-.0092.0095 20PS5, subgr. -.0046.0945-.0029.0899.0144.0955-.0010.0465.0031.0442.0288.0450.0054.0243.0063.0252.0189.0236 21PS6, func 1.0844.0329.0756.0317.0786.0322.0872.0204.0791.0196.0743.0184.0830.0136.0781.0129.0742.0116 22PS6, func 2.0844.0330.0756.0317.0783.0321.0872.0204.0790.0195.0741.0183.0830.0136.0780.0129.0741.0115 23PS6, func 3.0059.0410-.0030.0430-.0034.0405.0075.0201-.0016.0208-.0128.0185.0009.0098-.0039.0105-.0091.0095 24PS6, subgr. -.0041.0934-.0019.0899.0153.0958-.0006.0465.0035.0442.0296.0451.0056.0243.0065.0252.0185.0234 25PS7, func 1.0831.0303.0187.0592.1980.0650.0875.0199.1881.0478.1941.0505.0816.0128.1882.0420.1939.0439 26PS7, func 2.0819.0289.1426.0437.1426.0447.0875.0192.1461.0332.1389.0318.0815.0125.1448.0272.1406.0256 27PS7, func 3.0046.0359.0677.0433.0672.0412.0080.0185.0677.0231.0585.0210-.0006.0087.0645.0137.0640.0131 28PS7, subgr. -.0068.0852-.0101.0820.0002.0883-.0010.0437-.0018.0410.0136.0434.0053.0220.0031.0229.0026.0220 29PS8, func 1.0818.0289.1428.0438.1437.0450.0876.0192.1464.0333.1402.0322.0815.0124.1451.0273.1419.0259 30PS8, func 2.0818.0289.1429.0438.1434.0450.0876.0192.1464.0333.1398.0320.0815.0125.1451.0273.1416.0258 31PS8, func 3.0043.0358.0660.0433.0626.0409.0078.0185.0665.0230.0537.0206-.0006.0087.0631.0135.0595.0126 32PS8, subgr. -.0062.0853-.0051.0821.0146.0892-.0005.0437.0022.0411.0284.0444.0055.0220.0071.0230.0165.0224 N=250;ρ=0N=250;ρ=0.3N=250; ρ=0.7N=500;ρ=0N=500;ρ=0.3N=500; ρ=0.7N=1000;ρ=0N=1000;ρ=0.3N=1000; ρ=0.7 RowMethodBiasMSEBiasMSEBiasMSEBiasMSEBiasMSEBiasMSEBiasMSEBiasMSEBiasMSE Inverse PS 1PS1, func 1.0831.0298.11160377.1402.0712.0892.0181.1054.0244.1127.0409.0810.0118.1041.0173.1102.0311 2PS1, func 2.0835.0283.0714.0295.0738.0490.0881.0173.0611.0165.0478.0257.0816.0117.0597.0101.0483.0175 3PS1, func 3.0049.0380-.0027.0492-.0034.0781.0120.0198-.0171.0267-.0311.0473.0027.0099-.0191.0136-.0387.0313 4PS1, subgr..0015.1177-.0018.1667.0181.2239-.0068.0634.0032.0881.0136.1394-.0016.0307.0008.0411.0279.0811 5PS2, func 1.0921.0380.1012.0524.1384.0801.0940.0236.0872.0356.1012.0504.0831.0147.0838.0207.0933.0471 6PS2, func 2.0922.0354.1058.0467.1239.0646.0940.0227.0889.0303.0959.0376.0837.0143.0856.0193.0917.0338 7PS2, func 3.0120.0494.0362.0667.0450.0866.0182.0282.0120.0376.0154.0556.0032.0140.0044.0225.0011.0474 8PS2, subgr..0076.1496-.0095.2144.0219.2375-.0058.0818.0038.1280.0181.1561.0038.0418.0086.0596.0391.1124 9PS3, func 1.0818.0285.0903.0329.1257.0649.0871.0171.0821.0196.1008.0360.0811.0116.0810.0130.1007.0269 10PS3, func 2.0820.0283.0905.0313.1157.0496.0875.0172.0823.0187.0963.0286.0812.0116.0812.0127.0961.0214 11PS3, func 3.0033.0384.0159.0476.0391.0701.0111.0198.0043.0251.0120.0392.0023.0099.0029.0124.0117.0246 12PS3, subgr..0014.1187-.0026.1638.0108.2112-.0064.0635.0016.0855.0024.1268-.0015.0307-.0016.0399.0177.0716 13PS4, func 1.0895.0368.1025.0526.1384.0821.0920.0229.0868.0348.1006.0498.0830.0143.0842.0207.0931.0467 14PS4, func 2.0903.0359.1026.0455.1225.0592.0932.0227.0879.0293.0981.0360.0833.0143.0849.0187.0914.0321 15PS4, func 3.0100.0510.0327.0669.0429.0815.0171.0288.0109.0376.0180.0525.0027.0141.0035.0218.0013.0449 16PS4, subgr..0076.1534-.0092.2170.0230.2368-.0052.0828.0038.1285.0162.1545.0040.0421.0087.0595.0377.1125 17PS5, func 1.0957.0458.0994.0595.1335.0816.0918.0273.0914.0350.1102.0612.0823.0162.0830.0219.0972.0316 18PS5, func 2.0983.0440.0985.0523.1210.0651.0936.0254.0930.0312.1025.0417.0830.0158.0843.0199.0960.0249 19PS5, func 3.0298.0619.0213.0845.0455.0861.0182.0295.0075.0407.0211.0487.0012.0147.0027.0225.0172.0256 20PS5, subgr. -.0204.1722.0117.2172.0125.2358-.0063.0885.0235.1119.0204.1491.0074.0437.0092.0614.0047.0745 21PS6, func 1.0959.0446.0970.0587.1326.0812.0914.0262.0912.0350.1097.0599.0823.0160.0822.0216.0974.0308 22PS6, func 2.0972.0438.0963.0523.1221.0604.0925.0254.0914.0307.1034.0391.0826.0158.0839.0198.0934.0231 23PS6, func 3.0287.0621.0191.0861.0469.0813.0173.0296.0058.0407.0230.0450.0006.0148.0024.0223.0142.0245 24PS6, subgr. -.0208.1731.0115.2208.0111.2352-.0068.0894.0237.1130.0176.1505.0076.0439.0088.0621.0059.0749 25PS7, func 1.0843.0318.1923.0677.2242.1002.0889.0204.1925.0520.2136.0765.0809.0131.1889.0436.2088.0585 26PS7, func 2.0851.0306.1346.0473.1166.0568.0901.0198.1359.0325.1082.0370.0813.0128.1301.0245.1063.0233 27PS7, func 3.0116.0434.0576.0604.0425.0760.0114.0219.0520.0302.0337.0395-.0012.0107.0502.0160.0332.0199 28PS7, subgr. -.0114.1236.0041.1558.0087.2113-.0007.0648.0165.0792.0022.1262.0076.0325.0032.0407-.0091.0603 29PS8, func 1.0836.0304.1499.0528.1819.0787.0889.0197.1519.0376.1730.0571.0809.0127.1466.0292.1706.0418 30PS8, func 2.0840.0304.1502.0510.1746.0658.0832.0197.1515.0366.1670.0479.0810.0127.1469.0290.1646.0368 31PS8, func 3.0102.0435.0729.0610.1027.0722.0103.0220.0674.0313.0939.0400-.0016.0107.0672.0176.0932.0243 32PS8, subgr. -.0111.1241.0026.1530-.0049.1924-.0003.0651.0154.0782-.0076.1122.0079.0325.0015.400-.0174.0522

9. Results (1) Which propensity score is the most accurate within each method tested (tested with ANOVA): But, some values for bias per propensity score where not very different from each other… 9 General treatment effectTreatment effect within subgroups Method 1PS with variables related to outcome Method 2PS with variables related to outcome and variable for subgroups PS with variables only related to outcome and treatment assignment and variables for subgroups Method 3PS with variables related to outcome NA

10. Results (2) Which method is most accurate when the most accurate propensity scores are compared? Decide on partial effect size of method in ANOVA* For general treatment effect, the partial effect size is 0,028, where method 1 gives the lowest bias (followed by method 3) For treatment effect within subgroups, the partial effect size is 0,051, where method 1 gives the lowest bias too Although the effect sizes for method are not very large, regression analysis with treatment assignment, subgroup, interaction between these and the propensity score, which is estimated with variables related to outcome, seems to be the most accurate method to find treatment effects within subgroups *Effect size – 0,010 = small; 0,059 = medium; 0,138 = large (Cohen, 1988) 10

11. Discussion (1) Data simulation is done for different settings: Sample size, correlation between covariates and correlation with covariate for subgroups are changed over simulations Results for most accurate propensity score are based on sum of bias over all these settings; comparisons between methods for all propensity scores could give more in depth results The overall bias for different propensity scores was sometimes not very different Model for simulation of data was simple, linear; the relation between variables and outcome in practice can be more complicated …. 11

12. Discussion (2) Questions? 12