We have 3 custom fields
So the requirement is, if Development Type = "Core", the Story Points field would be displayed and if Development Type = "Custom", the Hours field would be displayed.
It is possible to set 2 validations in your transition, in order to ensure that:
To do it, you just need to use Boolean validator with math, date-time or text-string terms with the following configurations:
When "Development Type = Core" then field "Story Points" will be set and entered a value higher than 0, and field "Hours" is not set or zero:
Expression used is: %{12500} = "Core" IMPLIES ({12501} != null AND {12501} > 0 AND ({12502} = null OR {12502} = 0))
When "Development Type = Custom" then field "Hours" will be set and given a value higher than 0, and field "Story Points" is not set or zero:
Expression used is: %{12500} = "Custom" IMPLIES ({12502} != null AND {12502} > 0 AND ({12501} = null OR {12501} = 0))
Note that in the particular JIRA instance I used to build this example:
Once configured, your transition will look like this: