Selecting the optimal protein target structure
The optimal protein structure to use for a docking or screening simulation is identical to the protein used in the experiments, binds to a ligand very similar to the one you are trying to dock or expect to see as top-scoring binders in a screening, and comes from a high quality crystal structure.
The BLAST tool can be used to search for alternative protein structures in the PDB, using the sequence corresponding to the protein used in the experiments as input (see BLAST search against the PDB database).
The protein structure and the protein in the experiments should ideally be from the same species
If the protein structure is from the PDB, the species information can be found on pdb.org.
If the protein structure is not from the same species, or if there are introduced mutations into the protein in the experiments, it should be checked that the binding site residues are at least conserved. To do that, find the protein sequence corresponding to the protein used in the experiments, e.g. using the "Search for Sequences in UniProt" tool (see UniProt (Swiss-Prot/TrEMBL) search). Extract the protein sequence of your protein structure using the "Show Sequence" option in the Side Panel of the Molecule Project (Sequences associated with the molecules). Make a sequence alignment of the two sequences using the "Create Alignment" tool (see Create an alignment). If the sequences have high sequence identity, and all amino acids found in the binding site are conserved, the structure should be fine to use.
The protein structure resolution should be as high as possible
If the protein structure is from the PDB, the resolution information can be found on pdb.org, or using the "Search for PDB Structures at NCBI" tool in the workbench.
The general uncertainty in crystal structures with lower resolution (e.g. >2.5 Å) makes it hard to reproduce a ligand binding mode to a high accuracy (<2 Å RMSD). For some low resolution structures, atoms and whole side chains are missing. If this results in an incomplete structure in the binding site, then the protein structure is not fit to be used for docking. For other low resolution structures, the side chains have been modeled fully, even though their positions are not well determined. That may result in a poor representation of the binding site. Try representing the protein or just the binding pocket residues using one of the Atom and Bonds visualizations, and select the "Color by Temperature" color scheme (Customizing the visualization). Atom positions not well defined in the protein structure will show up in clear red color, while very well determined atom positions will be blue.
The protein structure should be in complex with a ligand in the binding site
A functioning protein can in some cases take up different overall conformations, depending on whether it binds to a substrate or not. The local conformation of side chains in the binding pocket can also depend on which type of molecule is binding. The protein structure should therefore ideally be interacting with something in the binding pocket similar to the ligand being docked. If the binding site is empty in the structure, it may be that side chain conformations are not optimal for ligand interaction.