I’ve had difficulty growing certain vegetables over the years,

According to halcha fertilizer does not need to be kosher therefore you can use all types of blood and crab fertilizer.

Explanation

You are not actually eating the fertilizer rather the fertilizer gets absorbed in the ground and thereby the ground produces minerals to cause the fruit and vegetables to grow properly.

However, regarding fertilizer which comes from something which is asur behano something which one is not allowed to have benefit from is more complicated

The Talmud teaches us that one is allowed to plant in a field which was fertilized with blood of a cow which was slaughtered for idols (g-d forbid) and so is the halcha in the Shulchan aruch.

This allowance follows the ruling that zeh vzeh gorem is allowed meaning if there is something which is being created by something kosher and something not kosher it is allowed

Example if a cow was feed food which is forbidden to derive pleasure from it together with permissible foods one is allowed to eat that cow (yet if it was only feed forbidden food poskim write that one is not to slaughter and eat it).

So too in our situation being that food grows not only because of the fertilizer but also because of the earth itself therefore one is allowed to eat the produce which grew in such a manner nonetheless poskim point out that this allowance is only bideved after the fact however lchatchila one is not to fertilize a field with such fertilizer as doing so on purpose is considered “lvtel issur lchatchila” to initially go ahead and nullify something forbidden which that is prohibited.

Sources:

גמ’ ע”ז דף מ”ט ע”א שו”ע יו”ד סי’ קמ”ב סי”א ונו”כ שם וראה שו”ע יו”ד סו”ס פ”ז