Hybridization between species can either promote or impede adaptation. But we know very little about the genetic basis of hybrid fitness, especially in nondomesticated organisms, and when populations are facing environmental stress. We made genetically variable F2 hybrid populations from two divergent Saccharomyces yeast species. We exposed populations to ten toxins and sequenced the most resilient hybrids on low coverage using ddRADseq to investigate four aspects of their genomes: 1) hybridity, 2) interspecific heterozygosity, 3) epistasis (positive or negative associations between nonhomologous chromosomes), and 4) ploidy. We used linear mixed-effect models and simulations to measure to which extent hybrid genome composition was contingent on the environment. Genomes grown in different environments varied in every aspect of hybridness measured, revealing strong genotype–environment interactions. We also found selection against heterozygosity or directional selection for one of the parental alleles, with larger fitness of genomes carrying more homozygous allelic combinations in an otherwise hybrid genomic background. In addition, individual chromosomes and chromosomal interactions showed significant species biases and pervasive aneuploidies. Against our expectations, we observed multiple beneficial, opposite-species chromosome associations, confirmed by epistasis- and selection-free computer simulations, which is surprising given the large divergence of parental genomes (∼15%). Together, these results suggest that successful, stress-resilient hybrid genomes can be assembled from the best features of both parents without paying high costs of negative epistasis. This illustrates the importance of measuring genetic trait architecture in an environmental context when determining the evolutionary potential of genetically diverse hybrid populations.