Isn't irradiance at surface in a specific direction just radiance? Irradiance is the amount of energy coming in from a bunch of angles at a single infinitesimally small point on the surface. If we constrain it to a specific angle, isn't this just radiance? Why is it then specified as irradiance above?

Yes, irradiance becomes radiance when (i) constraining the wedge to a single lighting direction, and (ii) removing the dependency on surface orientation.

At first, it may seem that the denominator in this BRDF function should also be just radiance, but we need to be a little careful. The BRDF function is used with our reflectance equation:

$L_{out}(\omega_o) = \int f(\omega_i,\omega_o) L(\omega_i) \cos(\theta_i) d\omega_i$

Here, $L(\omega_i) \cos(\theta_i) d\omega_i$ represents the irradiance at a surface point, where $L(\omega_i)$ is the radiance associated with an incident lighting direction $\omega_i$. In this irradiance term, the light rays are contained within a (small) wedge with solid angle $d\omega_i$, and there's a dependency on surface orientation $\cos(\theta_i)$.

So for the reflectance equation to output radiance, the BRDF needs to be defined as the ratio of outgoing radiance to surface irradiance.

There's also a post here that explains this in more detail.