"The long arms are adjustable, but the lowers don't effect caster. Sorry, didn't realize you had the long arm. I thought your truck looked pretty tall for a 4" lift. If you have not had it to the alignment shop since you got rid of the shimmies, that will make a difference. I would bet you are toed out slightly causing the wander, that is where my truck is stuck because the Rusty's steering doesn't adjust in enough to set toe correct. Do you have slop in the steering?"
The lowers DEFINITELY affect caster numbers. If you want more positive caster, lengthen them in comparison to the uppers and the opposite for negative caster.
If its really a big issue, you could always consider doing a cut and turn on the knuckles. Its not the easiest process and would require fab tools, but its something to consider if its really a problem.
It sounds like you are doing this in your driveway, so it might be worth putting it on an alignment rack if you want it perfect. The lifetime Alignment from Firestone seems expensive, but if you use it 3 times its worth it. You can literally take it in before and after every trail ride if you wanted just to make sure it all stayed correct.