Current virtual try-on methods primarily enhance performance through network optimization, like coarse-to-fine structures and referenceNet for clothing information injection. However, limited sample quantity and diversity restrict their improvement. To overcome this, we present a unified mask-free virtual try-on framework. It utilizes diffusion models' inherent properties to boost each pipeline part's ability to deeply fit the target sample distribution, thus improving performance. On the input side, our proposed text-driven pseudo-input preparation approach increases the diversity of clothing-agnostic regions in person pseudo-samples. This prompts the generator to focus more on variations in these areas and improves the model's generalization ability. At the generator, we propose gated manipulation to prevent weight forgetting and cut training costs, and introduce texture-aware injection to explicitly add human-perceptible clothing texture info. For inference, our proposed refining conditional inference approach reduces Gaussian noise randomness, thus preserving identity information and clothing details in results. Extensive experiments demonstrate our method outperforms previous virtual try-on methods.