3.1 On positive definiteness

Recall that a continuous function f on $\mathbb {R}$ is positive definite if for any $\{v_s\}_{s=1}^{n}\subset \mathbb {R}$ and $\{z_s\}_{s=1}^{n}\subset \mathbb {C}$ there holds

$$\begin{align*}\sum_{s,l=1}^{n}z_s\overline{z_{l}}f(v_s-v_l)\ge 0. \end{align*}$$

Bochner’s theorem states that any continuous positive definite function $f(v)$ , $f(0)=1$ , is the Fourier transform of a probability measure.

Recall that the function $e_{2r+1}(v,\lambda )$ is given in (2.4). By (2.9) and (2.10),

$$\begin{align*}e_{2r+1}(v,\lambda)=c_{\lambda}\int_{-1}^{1}(1-t^2)^{\lambda-1/2}(1+P_{2r+1}^{(\lambda-1/2)}(t))e^{-ivt}\,dt. \end{align*}$$

Then, by Remark 3.2 and Corollaries 3.3 and 3.4, we deduce the following

4.1 When the kernel is bounded by one

In this section, for the sake of completeness, we give the proof of Theorem 1.1 in the general case, following ideas from the paper [Reference Constales, De Bie and Lian10]. We stress that the proof below also allows one to deal with the case $d=1$ .

Let $d\in \mathbb {N}$ , $a>0$ , $\langle \kappa \rangle \ge 0$ , and $\eta =\lambda _{\kappa }=\langle \kappa \rangle +\frac {d-2}{2}>0$ . For $w\ge 0$ and $\tau \in [-1,1]$ , define

(4.1) $$ \begin{align} \Psi_{a}^d(w,\tau)=2^{2\eta/a}\Gamma\Bigl(\frac{2\eta+a}{a}\Bigr)\sum_{j=0}^{\infty}e^{-\frac{i\pi j}{a}}\,\frac{\eta+j}{\eta}\,w^{-2\eta/a}J_{\frac{2(\eta+j)}{a}}(w)C_{j}^{\eta}(\tau). \end{align} $$

Putting in (4.1)

(4.2) $$ \begin{align} x,y\in\mathbb{R}^d,\quad x=|x|x',\quad y=|y|y',\quad x',y'\in\mathbb{S}^{d-1},\quad w=\frac{2}{a}\,(|x||y|)^{a/2},\quad \tau=\langle x',\,y'\rangle, \end{align} $$

we obtain the function

$$ \begin{align*} K_{a}^d(x,y)=a^{2\eta/a}\Gamma\Bigl(\frac{2\eta+a}{a}\Bigr)(|x||y|)^{-\eta}\sum_{j=0}^{\infty}e^{-\frac{i\pi j}{a}}\,\frac{\eta+j}{\eta}\,J_{\frac{2(\eta+j)}{a}}\Bigl(\frac{2}{a}(|x||y|)^{a/2}\Bigr)C_{j}^{\eta}(\langle x',\,y'\rangle). \end{align*} $$

Note that $K_{a}^d(x,y)=K_{a}^d(y,x)$ and $\Psi _{a}^d(0,\tau )=K_{a}^d(0,y)=1$ .

Let $V_{\kappa }f(x)$ be the intertwining operator in the Dunkl harmonic analysis [Reference Rösler20], which is a positive operator satisfying

$$\begin{align*}V_{\kappa}f(x)=\int_{\mathbb{R}^d}f(\xi)\,d\mu_{x}^{\kappa}(\xi). \end{align*}$$

The representing measures $\mu _{x}^{\kappa }(\xi )$ are compactly supported probability measures with $\text {supp}\,\mu _{x}^{\kappa }(\xi )\subset \text {co}{}\{gx\colon g\in G\}$ [Reference Rösler22].

The kernel of the generalized Fourier transform $\mathcal {F}_{\kappa ,a}$ is given by [Reference Ben Saïd, Kobayashi and Ørsted4, Chapters 4, 5]

(4.3) $$ \begin{align} &B_{\kappa,a}(x,y)=V_{\kappa}K_{a}^d(x,|y|\cdot)(y') \notag\\ &\qquad\quad=a^{2\eta/a}\Gamma\Bigl(\frac{2\eta+a}{a}\Bigr)(|x||y|)^{-\eta}\sum_{j=0}^{\infty}e^{-\frac{i\pi j}{a}}\,\frac{\eta+j}{\eta}\,J_{\frac{2(\eta+j)}{a}}\Bigl(\frac{2}{a}(|x||y|)^{a/2}\Bigr)V_{\kappa}C_{j}^{\eta}(\langle x',{\cdot}\,\rangle)(y'). \end{align} $$

If $\langle \kappa \rangle =0$ , that is, $\eta =(d-2)/2$ , then $V_{\kappa }=\text {I}$ is the identity operator and $K_{a}^d(x,y)=B_{0,a}(x,y)$ .

Since the operator $V_{\kappa }$ is positive and $V_{\kappa }1=1$ , the proof of equality (1.2) can proceed as follows: If $|\Psi _{a}^d(w,\tau )|\le 1$ , then $|B_{\kappa ,a}(x,y)|\le 1$ .

Let further $\frac {2}{a}\in \mathbb {N}$ , $a=\frac {2}{R}$ . The inequality $|\Psi _{a}^d(w,\tau )|\le 1$ was proved in [Reference Constales, De Bie and Lian10] under the conditions $\eta>0$ and $\eta =(d-2)/2$ . We note that the proof of this estimate in [Reference Constales, De Bie and Lian10] also holds for $\eta =\langle \kappa \rangle +(d-2)/2>0$ . Hence, (1.2) is valid for $\langle \kappa \rangle +(d-2)/2>0$ . If $\eta =0$ , then $d=1$ , $\kappa =1/2$ or $d=2$ , $\kappa \equiv 0$ , and in these cases, (1.2) holds as well (see Theorem 1.1 and [Reference De Bie5]). This completes the proof of Theorem 1.1.

4.2 Positive definiteness of $\Psi _{a}^d$ . Another proof of Theorem 1.1

In [Reference Constales, De Bie and Lian10], to evaluate the kernel $B_{0,a}(x,y)$ , the authors used the Laplace transform of the function $\Psi _{a}^d$ given by (4.1). In the general case $d\ge 1$ , we give another proof of the estimate $|\Psi _{a}^d|\le 1$ based on the approach from the papers [Reference De Bie and Lian6, Reference Deleaval and Demni11], which allows one to show positive definiteness of the function $\Psi _{a}^d(w,\tau )$ with respect to w. Note also that for $a=1,2$ , the functions

$$ \begin{align*} \Psi_{1}^d(w,\tau)=j_{\lambda_{\kappa}-1/2}(w\sqrt{(1+\tau)/2}\,), \quad \Psi_{2}^d(w,\tau)=e^{-iw\tau}, \end{align*} $$

are positive definite (see [Reference Ben Saïd, Kobayashi and Ørsted4, Example 4.18], [Reference Constales, De Bie and Lian10]).

Let

$$\begin{align*}I_{\eta}(b)=\Bigl(\frac{b}{2}\Bigr)^{\eta}\,\sum_{j=0}^{\infty}\frac{1}{j!\,\Gamma(j+\eta+1)}\Bigl(\frac{b^2}{4}\Bigr)^j \end{align*}$$

be the modified Bessel function of the first kind (see [Reference Bateman, Erdélyi, Magnus, Oberhettinger and Tricomi1, Chapter 7, 7.2.2]) and

$$\begin{align*}\Phi_2^{(m)}(\beta_1,\dots,\beta_m;\gamma;x_1,\dots,x_m)= \sum_{j_1,\dots,j_m\ge 0}\frac{(\beta_1)_{j_1}\cdots (\beta_m)_{j_m}}{(\gamma)_{j_1+\cdots+j_m}}\,\frac{x_1^{j_1}}{j_{1}!}\cdots\frac{x_m^{j_m}}{j_{m}!} \end{align*}$$

be the second Humbert function of m variables (see [Reference Exton14, Chapter 2, 2.1.1.2]). In the case when $\gamma -\sum _{j=1}^m\beta _j>0$ and $\beta _j>0$ , $j=1,\dots ,m$ , the hypergeometric function $\Phi _2^{(m)}$ admits the following integral representation

(4.4) $$ \begin{align} \Phi_2^{(m)}(\beta_1,\dots,\beta_m;\gamma;x_1,\dots,x_m) =C_{\beta}^{(\gamma)}\int_{T^m}e^{\sum_{j=1}^mx_jt_j}\Bigl(1-\sum_{j=1}^{m}t_j\Bigr)^{\gamma-\sum_{j=1}^m\beta_j-1} \prod_{j=1}^{m}t_j^{\beta_j-1}\,dt_1\cdots dt_m, \end{align} $$

where

$$\begin{align*}C_{\beta}^{(\gamma)}=\frac{\Gamma(\gamma)}{\Gamma(\gamma-\sum_{j=1}^{m}\beta_j)\prod_{j=1}^{m}\Gamma(\beta_j)} \end{align*}$$

and

$$\begin{align*}T^m=\Bigl\{(t_1,\dots,t_m)\colon t_j\ge 0,\ j=1,\dots,m,\ \sum_{j=1}^{m}t_j\le 1\Bigr\} \end{align*}$$

is the unit simplex in $\mathbb {R}^m$ ([Reference Chamayou and Wesolowski9, Reference Humbert17]).

Taking into account the results from [Reference De Bie and Lian6, Reference Deleaval and Demni11], we obtain the following proposition, which gives an alternative proof of Theorem 1.1.

Proposition 4.1. Let $d\in \mathbb {N}$ , $a=\frac {2}{R}$ , $R\in \mathbb {N}$ , $R\ge 2$ , $\eta =\langle \kappa \rangle +\frac {d-2}{2}>0$ , $\tau \in [-1,1]$ , and $q=\arccos \tau $ . The function $w \mapsto \Psi _{a}^d(w,\tau )$ is a positive definite entire function of exponential type

(4.5) $$ \begin{align} \theta(a,\tau)= \begin{cases} \cos\frac{q}{R}, & R=2r\ \text{and}\ 0\le q\le\pi \ \text{or}\ R=2r+1\ \text{and}\ 0\le q\le\pi/2,\\ \cos\frac{\pi-q}{R}, & R=2r+1\ \text{and}\ \pi/2\le q\le\pi. \end{cases} \end{align} $$

Proof. Let us consider the function

$$\begin{align*}f_{R,\eta}(b,\tau)=\Gamma(R\eta+1)\Bigl(\frac{b}{2}\Bigr)^{\eta}\sum_{j=0}^{\infty} \frac{j+\eta}{\eta}\,I_{R(j+\eta)}(b)C_{j}^{\eta}(\tau), \quad R\in\mathbb{N}. \end{align*}$$

Let $\tau =\langle x',\,y'\rangle $ , $b=-iw$ , $R=\frac {2}{a}$ , $R\ge 2$ . Since $I_{\eta }(-iw)=e^{-i\frac {\pi \eta }{2}}J_{\eta }(w)$ [Reference Bateman, Erdélyi, Magnus, Oberhettinger and Tricomi1, Chapter 7, 7.2.2], then $f_{R,\eta }(b,\tau )=\Psi _{a}^d(w,\tau )$ . It is known (see [Reference De Bie and Lian6, Reference Deleaval and Demni11]) that

$$\begin{align*}f_{R,\eta}(b,\tau)=b_0\Phi_{2}^{(R-1)}(\eta,\dots,\eta;R\eta;b_1-b_0,\dots,b_{R-1}-b_0), \end{align*}$$

where $b_j=b\cos {}(\frac {q-2\pi j}{R})$ , $j=0,\dots ,R-1$ . In light of (4.4) and using $\eta>0$ , we get

(4.6) $$ \begin{align} &\Psi_{a}^d(w,\tau)=f_{R,\eta}(-iw,\tau)\notag\\ &\qquad=C_{\eta}\int_{T^{R-1}}e^{-iw\{\cos{}(\frac{q}{R})+\sum_{j=1}^{R-1}(\cos{}(\frac{q-2\pi j}{R})-\cos{}(\frac{q}{R}))t_j\}}\Bigl(1-\sum_{j=1}^{R-1}t_j\Bigr)^{\eta-1}\prod_{j=1}^{R-1}t_j^{\eta-1}\, \,dt_1\cdots dt_{R-1}, \end{align} $$

where $C_{\eta }=\Gamma (R\eta )/(\Gamma (\eta ))^R$ . For $\alpha>0$ and $\beta \in \mathbb {R}$ , the function $\alpha e^{i\beta w}$ is positive definite, hence the function $\Psi _{a}^d(w,\tau )$ is also positive definite with respect to w.

The representation (4.6) implies that $\Psi _a^d(w,\tau )$ is an entire function of exponential type with respect to w. Let us calculate its type. Since for $j=1,\dots ,R-1$ , $q=\arccos \tau $ , $\tau \in [-1,1]$ , $ \cos \frac {q}{R}-\cos \frac {q-2\pi j}{R}=2\sin \frac {\pi i}{R}\,\sin \frac {\pi j-q}{R}\ge 0, $ then for any $(t_1,\dots ,t_{R-1})\in T^{R-1}$ ,

$$\begin{align*}\min_{1\le j\le R-1}\cos\frac{q-2\pi j}{R}\le \cos\frac{q}{R}+\sum_{j=1}^{R-1}\Bigl(\cos\frac{q-2\pi j}{R}-\cos\frac{q}{R}\Bigr)t_j\le \cos\frac{q}{R}. \end{align*}$$

Since

$$ \begin{align*} \min_{1\le j\le R-1}\cos\frac{q-2\pi j}{R}= \begin{cases}-\cos\frac{q}{R}, & R=2r\ \text{or}\ R=2r+1\ \text{and}\ 0\le q\le\pi/2,\\ -\cos\frac{\pi-q}{R}, & R=2r+1\ \text{and}\ \pi/2\le q\le\pi, \end{cases} \end{align*} $$

the type of the function $\Psi _{a}^d(w,\tau )$ is given by (4.5).

Remark 4.2. Let $t,\tau \in (-1,1)$ , $\lambda =2\eta /a$ , $a=1/r$ , $r\in \mathbb {N}$ . Then the function $\Psi _{a}^{d}(w,\tau )$ has the following integral representation

(4.7) $$ \begin{align} \Psi_{a}^{d}(w,\tau)=c_{\lambda}\int_{-1}^{1}(1-t^2)^{\lambda-1/2}\sum_{j=0}^{\infty}\frac{\eta+j}{\eta}\,P_{2j/a}^{(\lambda-1/2)}(t) C_{j}^{\eta}(\tau)e^{-iwt}\,dt. \end{align} $$

Indeed, using (2.8) and (4.1) and Lemmas 2.2 and 2.6, we get

$$ \begin{align*} &\Psi_{a}^d(w,\tau)=\sum_{j=0}^{\infty}(-1)^{rj}\,\frac{\eta+j}{\eta}\,\frac{w^{2rj}}{2^{2rj}(\lambda+1)_{2rj}}\,j_{\lambda+2rj}(w) C_{j}^{\eta}(\tau)\\ &=\sum_{j=0}^{\infty}\frac{\eta+j}{\eta}\Bigl\{j_{\lambda}(w)+\sum_{s=1}^{rj-1}(-1)^s\binom{rj}{s}\frac{(\lambda+rj)_s}{(\lambda+1)_s}\,j_{\lambda+s}(w) {+}(-1)^{rj}\,\frac{(\lambda+rj+1)_{rj-1}}{(\lambda+1)_{rj-1}}\,j_{\lambda+rj}(w)\Bigr\}C_{j}^{\eta}(\tau)\\ &\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad=c_{\lambda}\int_{-1}^{1}(1-t^2)^{\lambda-1/2}\sum_{j=0}^{\infty}\frac{\eta+j}{\eta}\,P_{2rj}^{(\lambda-1/2)}(t) C_{j}^{\eta}(\tau)e^{-iwt}\,dt. \end{align*} $$

The integral representation (4.7) implies that

$$ \begin{align*} \psi_a^d(t,\tau)=\sum_{j=0}^{\infty}\frac{\eta+j}{\eta}\,P_{2rj}^{(\lambda-1/2)}(t) C_{j}^{\eta}(\tau)\ge 0,\quad t,\,\tau\in (-1,1), \end{align*} $$

is equivalent to the positive definiteness of the function $\Psi _{a}^d(w,\tau )$ . Proposition 4.1 shows that the function $\psi _{a}^d(t,\tau )$ is positive and its support as a function of t lies on the interval $I_a=[-\theta (a,\tau ),\theta (a,\tau )]$ .

In the case $a=1$ , one can easily write an explicit formula for $\psi _{1}^d(t,\tau )$ . In more detail, we have $r=1$ , $\lambda =2\eta $ , and $\theta (a,\tau )=\cos \frac {q}{2}=\sqrt {(1+\tau )/2}$ . Then using [Reference Bateman, Erdélyi, Magnus, Oberhettinger and Tricomi2, Chapter VIII, 8.7(31)], we obtain

$$ \begin{align*} \psi_{1}^d(t,\tau)&=(2\pi c_{\lambda})^{-1}(1-t^2)^{1/2-2\eta}\int_{-\infty}^{\infty}j_{\eta-1/2}(w\sqrt{(1+\tau)/2})e^{iwt}\,dw\\ &=c(\eta)(1-t^2)^{1/2-2\eta}(1+\tau)^{1/2-\eta}(1+\tau-2t^2)^{\eta-1}\chi_{I_1}(t)\\ &=c(\eta)(1-t^2)^{1/2-2\eta}(1+\tau)^{1/2-\eta}(\tau-P_2^{(-1/2)}(t))^{\eta-1}\chi_{I_1}(t), \end{align*} $$

where $\chi _{I_1}(t)$ is the characteristic function of the interval $I_1$ and $P_2^{(-1/2)}(t)= \cos {}(2\arccos t)$ .

4.3 Parameters when the kernel is not bounded by one

We will show that in some cases the uniform norm of the kernel $B_{\kappa ,a}(x,y)$ is not bounded by $1$ and can be either finite or infinite. We mention that the conditions when the norm is not finite are known only in the one-dimensional case. In particular, if $d=1$ , $0<a<2$ , and $\kappa <\frac {1}{2}-\frac {a}{4}$ , then $\|B_{\kappa ,a}\|_{\infty }=\infty $ (see Proposition 2.1).

Proof of Theorem 1.2.

Let first $d=1$ and $\lambda =\lambda _{\kappa ,a}=\frac {2\kappa -1}{a}$ . Changing variables $x=(\frac {a}{2}|v|)^{2/a}{\operatorname{sign}}\ {}v$ in (2.3), we get

(4.8) $$ \begin{align} e_{\kappa,a}(v,\lambda)=b_{\kappa,a}\Bigl(\Bigl(\frac{a}{2}\,|v|\Bigr)^{2/a}{\operatorname{sign}}{}\ v\Bigr) =j_{\lambda}(v)&+\frac{\Gamma(\lambda+1)\cos\frac{\pi}{a}}{2^{2/a}\Gamma(\lambda+1+2/a)}\, |v|^{2/a}j_{\lambda+\frac{2}{a}}(v){\operatorname{sign}}{}\ v\notag\\ &{}-i\,\frac{\Gamma(\lambda+1)\sin\frac{\pi}{a}}{2^{2/a}\Gamma(\lambda+1+2/a)}\, |v|^{2/a}j_{\lambda+\frac{2}{a}}(v){\operatorname{sign}}{}\ v. \end{align} $$

If $0<a\le 1$ and $\kappa =\frac {1}{2}-\frac {a}{4}$ , then $\lambda =-1/2$ , $j_{-1/2}(v)=\cos v$ , $j_{1/2}(v)=\frac {\sin v}{v}$ .

Assume first that $\cos \frac {\pi }{a}=0$ or $a=\frac {2}{2r+1}$ , $r\in \mathbb {N}$ . Since $\sin \frac {\pi }{a}\neq 0$ , $j_{-1/2}(2\pi )=1$ , ${j_{1/2}(2\pi )=0}$ , and $j_{2r+1/2}(2\pi )\neq 0$ (see [Reference Watson25, Chapter 15, 15.28]), then from (4.8), we get $|e_{\kappa ,a}(2\pi ,-1/2)|=|e_{2r+1}(2\pi ,-1/2)|>1$ .

Let now $\cos {}(\pi /a)\neq 0$ . In light of (4.8), there holds

$$\begin{align*}|e_{\kappa,a}(v,\lambda)|\ge \Bigl|\cos v+\frac{\Gamma(\frac{1}{2})\cos\frac{\pi}{a}}{2^{2/a}\Gamma(\frac{2}{a}+\frac{1}{2})}\, |v|^{2/a}j_{\frac{2}{a}-\frac{1}{2}}(v){\operatorname{sign}}{}\ v\Bigr|. \end{align*}$$

Since

$$\begin{align*}(2\pi s)^{2/a}j_{\frac{2}{a}-\frac{1}{2}}(2\pi s)=\frac{2^{2/a}\Gamma(\frac{2}{a}+\frac{1}{2})}{\Gamma(\frac{1}{2})}\, \Bigl\{\cos\frac{\pi}{a}+O\Bigl(\frac{1}{s}\Bigr)\Bigr\} \end{align*}$$

as $s\to +\infty $ (see [Reference Watson25, Chapter VII, 7.1]), we deduce that for sufficiently large s

$$\begin{align*}|e_{\kappa,a}(2\pi s,\lambda)|\ge 1+\cos^2\frac{\pi}{a}+O\Bigl(\frac{1}{s}\Bigr)>1, \end{align*}$$

which completes the proof of (1.3). Note that similar arguments also imply the proof of (1.3) for $1<a<2$ and $\kappa =\frac {1}{2}-\frac {a}{4}$ .

Now, we consider the multivariate case. Suppose that $d\ge 1$ , $a\in (1,2)\cup (2,\infty )$ , $\langle \kappa \rangle \ge 0$ , and $\eta =\lambda _{\kappa }$ , $\lambda =\frac {2\eta }{a}=\frac {2\langle \kappa \rangle +d-2}{a}$ . Recall that $x'$ , $y'$ , w are defined in (4.2). In view of (2.2) and the proof for the case $d=1$ , we may assume that $\eta ,\lambda>-1/2$ and $\cos \frac {\pi }{a}\neq 0$ .

Note that (4.3) can be equivalently written as

$$\begin{align*}B_{\kappa,a}(x,y) =\sum_{j=0}^{\infty}e^{-\frac{i\pi j}{a}}\,\frac{\eta+j}{\eta}\,\frac{\Gamma(\lambda+1)}{2^{2j/a}\,\Gamma(\lambda+1+2j/a)}\,w^{2j/a} j_{\lambda+\frac{2j}{a}}(w)V_{\kappa}C_{j}^{\eta}(\langle x',{\cdot}\,\rangle)(y'). \end{align*}$$

For Gegenbauer polynomials (2.11), the following estimate holds

$$\begin{align*}\sup_{-1\le t\le 1}\Bigl|\frac{1}{\eta}\,C_j^{\eta}(t)\Bigr|\le \widetilde{c}(\eta)j^{2\eta-1},\quad j\ge 1, \end{align*}$$

[Reference Ben Saïd, Kobayashi and Ørsted4, Lemma 4.9]. Using (2.11), we get

$$\begin{align*}\Bigl|\frac{\eta+j}{\eta}\,V_{\kappa}C_j^{\eta}(\langle x',{\cdot}\,\rangle)(y')\Bigr|\le c(\eta)(j^{2\eta}+1). \end{align*}$$

Since $|j_{\lambda }(w)|\le 1$ , $\frac {1}{\eta }\,C_1^{\eta }(t)=2t$ , then for $0\le w\le 1$ , we have

$$ \begin{align*} \Bigl|\sum_{j=2}^{\infty}e^{-\frac{i\pi j}{a}}\,\frac{\eta+j}{\eta}\,\frac{\Gamma(\lambda+1)}{2^{2j/a}\Gamma(\lambda+1+2j/a)}\,w^{2j/a} &j_{\lambda+\frac{2j}{a}}(w)V_{\kappa}C_{j}^{\eta}(\langle x',{\cdot}\,\rangle)(y')\Bigr|\\ &{}\le c(\eta)w^{4/a}\sum_{j=2}^{\infty}\frac{\Gamma(\lambda+1)(j^{2\eta}+1)}{2^{2j/a}\Gamma(\lambda+1+2j/a)} \le c(a,\kappa)w^{4/a} \end{align*} $$

and

$$ \begin{align*} B_{\kappa,a}(x,y)=j_{\lambda}(w)&+\frac{2(\eta+1)\Gamma(\lambda+1)\cos\frac{\pi}{a}}{2^{2/a}\Gamma(\lambda+1+2/a)}\,w^{2/a} j_{\lambda+\frac{2}{a}}(w)V_{\kappa}(\langle x',{\cdot}\,\rangle)(y')\\ &{}-i\,\frac{2(\eta+1)\Gamma(\lambda+1)\sin\frac{\pi}{a}}{2^{2/a}\Gamma(\lambda+1+2/a)}\,w^{2/a} j_{\lambda+\frac{2}{a}}(w)V_{\kappa}(\langle x',{\cdot}\,\rangle)(y')+O(w^{4/a}),\quad w\to 0. \end{align*} $$

Therefore,

(4.9) $$ \begin{align} |B_{\kappa,a}(x,y)|\ge \Bigl|j_{\lambda}(w)+\frac{2(\eta+1)\Gamma(\lambda+1)\cos\frac{\pi}{a}}{2^{2/a}\Gamma(\lambda+1+2/a)}\,w^{2/a} j_{\lambda+\frac{2}{a}}(w)V_{\kappa}(\langle x',{\cdot}\,\rangle)(y')+O(w^{4/a})\Bigr|. \end{align} $$

Since the operator $V_{\kappa }$ is an isomorphism on the space of homogeneous polynomials of degree $1$ , then for suitable $x',y'\in \mathbb {S}^{d-1}$ , we deduce

(4.10) $$ \begin{align} V_{\kappa}(\langle x',{\cdot}\,\rangle)(y')=\int_{\mathbb{R}^d}\langle x',\xi\rangle\,d\mu_{y'}^{\kappa}(\xi)\neq 0,\quad {\operatorname{sign}} (V_{\kappa}(\langle x',{\cdot}\,\rangle)(y'))={\operatorname{sign}}{}\ (\cos{}(\pi/a)). \end{align} $$

We have

$$\begin{align*}j_{\lambda}(w)=1+O(w^2),\quad j_{\lambda+\frac{2}{a}}(w)=1+O(w^2) \end{align*}$$

as $w\to 0$ . This, (4.9) and (4.10) imply that, for given $x',y'\in \mathbb {S}^{d-1}$ as above and sufficiently small positive w,

$$\begin{align*}|B_{\kappa,a}(x,y)|\ge 1+\frac{2(\eta+1)\Gamma(\lambda+1)|\!\cos\frac{\pi}{a}|}{2^{2/a}\Gamma(\lambda+1+2/a)}\,w^{2/a} |V_{\kappa}(\langle x',{\cdot}\,\rangle)(y')|+O(w^{4/a})+O(w^2)>1.\\[-42pt] \end{align*}$$

5.1 The case of $a>0$

Let $d\nu _{\eta }(u)=b_{\eta }u^{2\eta +1}\,du$ , $b_{\eta }^{-1}=2^{\eta }\Gamma (\eta +1)$ , $d\nu _{\eta ,a}(u)=b_{\eta ,a}u^{2\eta +a-1}du$ , $b_{\eta ,a}^{-1}=a^{2\eta /a}\Gamma (2\eta /a+1)$ ,

$$\begin{align*}H_{\eta}(f_0)(v)=\int_{0}^{\infty}f_0(u)j_{\eta}(uv)\,d\nu_{\eta}(u),\quad f_{0}\in L^1(\mathbb{R}_+,d\nu_{\eta}),\quad \eta\ge-1/2, \end{align*}$$

be the Hankel transform and

$$\begin{align*}H_{\eta,a}(f_0)(v)=\int_{0}^{\infty}f_0(u)j_{\frac{2\eta}{a}}\Bigl(\frac{2}{a}\,(vu)^{a/2} \Bigr)\,d\nu_{\eta,a}(u),\quad 2\eta+a\ge 1, \end{align*}$$

the a-deformed Hankel transform (see [Reference Ben Saïd, Kobayashi and Ørsted4, Reference Gorbachev, Ivanov and Tikhonov15]).

Recall that $\lambda _{\kappa }=\langle \kappa \rangle +(d-2)/2$ , $\lambda =\lambda _{\kappa ,a}=2\lambda _{\kappa }/a$ .

Example 5.1. Consider $f(x)=e^{-|x|^2}\in \mathcal {S}(\mathbb {R}^d)$ , $f(x)=f_0(\rho )$ , $\rho =|x|$ . If $|y|=v$ , $\rho =(a/2)^{1/a}u^{2/a}$ , then (see [Reference Ben Saïd, Kobayashi and Ørsted4, Reference Gorbachev, Ivanov and Tikhonov15])

$$ \begin{align*} \mathcal{F}_{\kappa,a}(f)(y)=H_{\lambda_{\kappa},a}(f_0)(v)&= \int_{0}^{\infty}f_0(\rho)j_{\frac{2\lambda_{\kappa}}{a}}\Bigl(\frac{2}{a}\,(v\rho)^{a/2} \Bigr)\,d\nu_{\lambda_{\kappa},a}(\rho)\\ &= \int_{0}^{\infty}g_a(u)j_{\lambda}\Bigl(\Bigl(\frac{2}{a}\Bigr)^{1/2}\,v^{a/2}u\Bigr)\,d\nu_{\lambda}(u), \end{align*} $$

where

$$\begin{align*}g_a(u)=\exp \Bigl(-\Bigl(\frac{a}{2}\Bigr)^{2/a}u^{4/a}\Bigr),\quad u\ge 0. \end{align*}$$

Assuming that $\mathcal {F}_{\kappa ,a}(f)(y)$ is rapidly decreasing at infinity, the same is true for the function

$$\begin{align*}G_a(v)=H_{\lambda}(g_a)(v)=\int_{0}^{\infty}g_a(u) j_{\lambda}(uv)\,d\nu_{\lambda}(u). \end{align*}$$

Considering even extensions of the functions $g_a$ and $G_a$ , we get the following representation via the Dunkl transform:

$$\begin{align*}G_a(v)=\int_{-\infty}^{\infty}g_a(u)j_{\lambda}(uv)\,d\mu_{\kappa,2}(u)=\mathcal{F}_{\kappa,2}(g_a)(v),\quad v\in\mathbb{R}. \end{align*}$$

If $2/a$ is a noninteger, then either $4/a\not \in \mathbb {N}$ or $4/a=2s+1$ , $s\in \mathbb {Z}_+$ , and the function

$$\begin{align*}g_a(u)=\exp \Bigl(-\Bigl(\frac{a}{2}\Bigr)^{2/a}|u|^{4/a}\Bigr),\quad u\in \mathbb{R}, \end{align*}$$

has finite smoothness at the origin. On the other hand, since $g_a,G_a\in L^1(\mathbb {R},d\mu _{\kappa ,2})$ , we obtain

$$\begin{align*}g_a(u)=\mathcal{F}_{\kappa,2}(G_a)(u)=\int_{-\infty}^{\infty}G_a(v)j_{\lambda}(uv)\,d\mu_{\kappa,2}(v) ,\end{align*}$$

and the right-hand side is infinitely differentiable at the origin if $G_a(v)$ is fast decreasing. This contradiction shows that, for $2/a\not \in \mathbb {N}$ , $\mathcal {F}_{\kappa ,a}(f)(y)$ cannot rapidly decay at infinity and so it is not a Schwartz function.

Moreover, let us denote for convenience the derivative of f by $\partial f$ . Since

$$\begin{align*}(\partial_v^2j_{\lambda}(uv))\bigr|_{v=0}=-\frac{u^2}{2(\lambda+1)},\quad (\partial_v^4j_{\lambda}(uv))\bigr|_{v=0}=\frac{3u^4}{4(\lambda+1)(\lambda+2)}, \end{align*}$$

then

$$\begin{align*}G_a(v)=\xi_0+\xi_1v^2+O(v^4),\qquad v\to 0,\qquad \xi_1\neq 0, \end{align*}$$

and

$$\begin{align*}\mathcal{F}_{\kappa,a}(f)(y)=\xi_0+\widetilde{\xi}_1|y|^a+O(|y|^{2a}),\qquad y\to 0,\qquad \widetilde{\xi}_1\neq 0. \end{align*}$$

If a is not even, then $\mathcal {F}_{\kappa ,a}(f)(y)$ has finite smoothness at the origin.

The following statement follows from Example 5.1.

The conditions in Proposition 5.2 are also sufficient, at least in the one-dimensional case. Indeed, suppose $\lambda =(2\kappa -1)/a\ge -1/2$ and denote the even and odd parts of f, as usual, by

$$\begin{align*}f_{e}(x)=\frac{f(x)+f(-x)}{2},\quad f_{o}(x)=\frac{f(x)-f(-x)}{2}. \end{align*}$$

Then the one-dimensional generalized Fourier transform can be written as

(5.1) $$ \begin{align} \mathcal{F}_{\kappa,a}(f)(y)=2\int_{0}^{\infty}(B_{\kappa,a}(\,{\cdot}\,,y))_{e}(x) f_e(x)\,d\mu_{\kappa,a}(x)+ 2\int_{0}^{\infty}(B_{\kappa,a}(\,{\cdot}\,,y))_{o}(x) f_o(x)\,d\mu_{\kappa,a}(x). \end{align} $$

Putting in (5.1)

(5.2) $$ \begin{align} \begin{gathered} x=x(u)=\Bigl(\frac{a}{2}\Bigr)^{\frac{1}{a}}\,u^{\frac{2}{a}},\quad g(u)=Af(u)=f(x(u))=f\Bigl(\Bigl(\frac{a}{2}\Bigr)^{\frac{1}{a}}\,u^{\frac{2}{a}}\Bigr),\\ g_e(u)=Af_e(u),\quad g_o(u)=Af_o(u),\quad x,u\in\mathbb{R}_+, \end{gathered} \end{align} $$

and taking into account (2.1), we deduce

(5.3) $$ \begin{align} \mathcal{F}_{\kappa,a}(f)(y)&=\int_{0}^{\infty}Af_e(u)j_{\lambda} \Bigl(\Bigl(\frac{2}{a}\Bigr)^{1/2}|y|^{a/2}u\Bigr)\,d\nu_{\lambda}(u)\notag \\ &\quad \ {}+e^{-\frac{\pi i}{a}}\Bigl(\frac{2}{a}\Bigr)^{1/a}y\int_{0}^{\infty}u^{-2/a}Af_o(u) j_{\lambda+\frac{2}{a}}\Bigl(\Bigl(\frac{2}{a}\Bigr)^{1/2} |y|^{a/2}u\Bigr)\,d\nu_{\lambda+\frac{2}{a}}(u)\notag \\ &\qquad\qquad =H_{\lambda}(g_e)\Bigl(\Bigl(\frac{2}{a}\Bigr)^{1/2}|y|^{a/2}\Bigr)+ e^{-\frac{\pi i}{a}} \Bigl(\frac{2}{a}\Bigr)^{1/a} yH_{\lambda+\frac{2}{a}}(u^{-2/a}g_o)\Bigl(\Bigl(\frac{2}{a}\Bigr)^{1/2}|y|^{a/2}\Bigr). \end{align} $$

If $f\in \mathcal {S}(\mathbb {R})$ , then $g_e(u)$ , $u^{-2/a}g_o(u)$ decrease rapidly at infinity and there exist

$$\begin{align*}\lim_{u\to 0+0}u^{-2/a}g_o(u)=\Bigl(\frac{a}{2}\Bigr)^{\frac{1}{a}}f'(0). \end{align*}$$

Therefore, the representation (5.3) shows that, for even a, we have the embedding $\mathcal {F}_{\kappa ,a}(\mathcal {S}(\mathbb {R}))\subset C^{\infty }(\mathbb {R})$ .

Further, if $\frac {2}{a}\in \mathbb {N}$ , $f\in \mathcal {S}(\mathbb {R})$ , then the functions $g_e(u),u^{-2/a}g_o(u) \in \mathcal {S}(\mathbb {R}_+)$ as well as (see [Reference Rösler20]) the even functions

$$\begin{align*}H_{\lambda}(g_e)(v),\ H_{\lambda+\frac{2}{a}}(u^{-2/a}g_o)(v)\in \mathcal{S}(\mathbb{R}). \end{align*}$$

Hence, we prove the following

Proposition 5.4. Suppose $\frac {2}{a}\in \mathbb {N}$ , $\lambda \ge -1/2$ , $f\in \mathcal {S}(\mathbb {R})$ , then

(5.4) $$ \begin{align} \mathcal{F}_{\kappa,a}(f)(y)=F_1\bigl(|y|^{a/2}\bigr)+yF_2\bigl(|y|^{a/2}\bigr), \end{align} $$

where the even functions $F_1,F_2\in \mathcal {S}(\mathbb {R})$ .

Thus, for $2/a\in \mathbb {N}$ , the generalized Fourier transform decreases rapidly at infinity and we arrive at the following result.

Let

$$\begin{align*}\mathcal{S}_{org}(\mathbb{R})=\{f\in\mathcal{S}(\mathbb{R})\colon \partial^{k}f(0)=0,\ k\in\mathbb{N}\}. \end{align*}$$

The class $\mathcal {S}_{org}(\mathbb {R})$ is dense in $L^{2}(\mathbb {R},d\mu _{\kappa ,a})$ . Suppose $a>0$ , $f(x)\in \mathcal {S}_{org}(\mathbb {R})$ , then the functions $g_e(u),u^{-2/a}g_o(u)$ belong to $\mathcal {S}(\mathbb {R}_+)$ , cf. (5.3). Therefore, the even functions $H_{\lambda }(g_e)(v)$ , $H_{\lambda +\frac {2}{a}}(u^{-2/a}g_o)(v)$ also belong to $\mathcal {S}(\mathbb {R})$ .

Thus, we obtain the following

5.2 The case of irrational a

We will show that if a is irrational, any nontrivial Schwartz function possesses similar properties to the Gaussian function in Example 5.1. To see this, we need auxiliary properties of the kernel of the generalized Fourier transform.

Let $\mathbb {S}^{d-1}$ be the unit sphere in $\mathbb {R}^{d}$ , $x=\rho x'$ , $\rho =|x|\in \mathbb {R}_+$ , $x'\in \mathbb {S}^{d-1}$ , and $dx'$ be the Lebesgue measure on the sphere. If $a_{\kappa }^{-1}=\int _{\mathbb {S}^{d-1}}v_{\kappa }(x')\,dx'$ , $d\sigma _{\kappa }(x')=a_{\kappa }v_{\kappa }(x')\,dx'$ , then $d\mu _{\kappa ,a}(x)=d\nu _{\lambda _{\kappa },a}(\rho )\,d\sigma _{\kappa }(x')$ and $c_{\kappa ,a}=b_{\kappa ,a}a_{\kappa }$ .

Denote by $\mathcal {H}_{n}^{d}(v_{\kappa })$ the subspace of $\kappa $ -spherical harmonics of degree $n\in \mathbb {Z}_{+}$ in $L^{2} (\mathbb {S}^{d-1},d\sigma _{\kappa })$ (see [Reference Dunkl and Xu13, Chapter 5]). Let $\mathcal {P}_{n}^{d}$ be the space of homogeneous polynomials of degree n in $\mathbb {R}^{d}$ . Then $\mathcal {H}_{n}^{d}(v_{\kappa })$ is the restriction of $\ker \Delta _{\kappa }\cap \mathcal {P}_{n}^{d}$ to the sphere $\mathbb {S}^{d-1}$ .

If $l_{n}$ is the dimension of $\mathcal {H}_{n}^{d}(v_{\kappa })$ , we denote by $\{Y_{n}^{j}\colon j=1,\ldots ,l_{n}\}$ the real-valued orthonormal basis of $\mathcal {H}_{n}^{d}(v_{\kappa })$ in $L^{2}(\mathbb {S}^{d-1},d\sigma _{\kappa })$ . A union of these bases forms an orthonormal basis in $L^{2}(\mathbb {S}^{d-1},d\sigma _{\kappa })$ consisting of k-spherical harmonics.

Let us rewrite (4.3) as follows

$$\begin{align*}B_{\kappa,a}(x,y) =\sum_{j=0}^{\infty}e^{-\frac{i\pi j}{a}}\,\frac{\lambda_{\kappa}+j}{\lambda_{\kappa}}\,\frac{\Gamma(2\lambda_{\kappa}/a+1)(|x||y|)^{j}}{a^{2j/a}\,\Gamma(2(\lambda_{\kappa}+j)/a+1)}\, j_{\frac{2(\lambda_{\kappa}+j)}{a}}\Bigl(\frac{2}{a}\,(|x||y|)^{a/2}\Bigr)V_{\kappa}C_{j}^{\lambda_{\kappa}}(\langle x',{\cdot}\,\rangle)(y'). \end{align*}$$

Integrating this and using orthogonality of $\kappa $ -spherical harmonics, as in the case of the Dunkl kernel (see [Reference Dunkl and Xu13, Theorem 5.3.4], [Reference Rösler21, Corollary 2.5]), we obtain the following crucial property of the kernel of the generalized Fourier transform.

Proposition 5.7. If $x,y\in \mathbb {R}^d$ , $x=\rho x'$ , $y=vy'$ , then

$$\begin{align*}\int_{\mathbb{S}^{d-1}}B_{\kappa,a}(x,vy')Y_{n}^{j}(y')\,d\sigma_{\kappa}(y')=\frac{e^{-\frac{i\pi n}{a}}\Gamma(2\lambda_{\kappa}/a+1)}{a^{2n/a} \,\Gamma(2(\lambda_{\kappa}+n)/a+1)}\,v^n j_{\frac{2(\lambda_{\kappa}+n)}{a}}\Bigl(\frac{2}{a}\,(\rho v)^{a/2}\Bigr)Y_n^j(x). \end{align*}$$

Proof. 1. First, assume that $f(x)=\rho ^n\psi (\rho )Y_n^j(x')$ , where $n\in \mathbb {Z}_+$ , $x=\rho x'$ , $|x'|=1$ , and $\psi \in \mathcal {S}(\mathbb {R}_+)$ . Since $\rho ^nY_n^j(x')=Y_n^j(x)$ is a homogeneous polynomial of degree n, then $f(x)=\psi (|x|)Y_n^j(x)\in \mathcal {S}(\mathbb {R}^d)$ . If $y=vy'$ , $\rho =(a/2)^{1/a}u^{2/a}$ , then by [Reference Gorbachev, Ivanov and Tikhonov15]

$$ \begin{align*} \mathcal{F}_{\kappa,a}(f)(y)=e^{-\frac{i\pi n}{a}}Y_n^j(y)&H_{\lambda_{\kappa}+n,a}(\psi)(v) =e^{-\frac{i\pi n}{a}}Y_n^j(y)\int_{0}^{\infty}\psi(\rho) j_{\frac{2(\lambda_{\kappa}+n)}{a}} \Bigl(\frac{2}{a}\,(v\rho)^{a/2} \Bigr)\,d\nu_{\lambda_{\kappa},a}(\rho)\\ & =e^{-\frac{i\pi n}{a}}Y_n^j(y)\int_{0}^{\infty}\psi\Bigl(\Bigl(\frac{a}{2}\Bigr)^{1/a}u^{2/a}\Bigr) j_{\frac{2(\lambda_{\kappa}+n)}{a}}\Bigl(\Bigl(\frac{2}{a}\Bigr)^{1/2}\,v^{a/2}u \Bigr)\,d\nu_{\frac{2(\lambda_{\kappa}+n)}{a}}(u). \end{align*} $$

Moreover, following the ideas used in Example 5.1, we obtain that the function

$$\begin{align*}g_a(v)=\int_{0}^{\infty}\psi\Bigl(\Bigl(\frac{a}{2}\Bigr)^{1/a}u^{2/a}\Bigr)j_{\frac{2(\lambda_{\kappa}+n)}{a}}\Bigl(\Bigl(\frac{2}{a}\Bigr)^{1/2}\,v^{a/2}u \Bigr)\,d\nu_{\frac{2(\lambda_{\kappa}+n)}{a}}(u) \end{align*}$$

with irrational a cannot decrease rapidly as $v\to \infty $ , and it has finite smoothness at the origin. Therefore, it follows that $\mathcal {F}_{\kappa ,a}(f)\notin \mathcal {S}(\mathbb {R}^d)$ .

2. Let now $f\in \mathcal {S}(\mathbb {R}^d)$ be any nonzero function. Then its spherical $\kappa $ -harmonic expansion is given by

$$\begin{align*}f(\rho x')=\sum_{n=0}^{\infty}\sum_{j=1}^{l_{n}}f_{nj}(\rho)Y_{n}^{j}(x'),\quad f_{nj}(\rho)=\int_{\mathbb{S}^{d-1}}f(\rho x')Y_{n}^{j}(x')\,d\sigma_{\kappa}(x') \end{align*}$$

(see [Reference Gorbachev, Ivanov and Tikhonov15]). Since the subspaces $\mathcal {H}_n^d(v_{\kappa })$ are orthogonal, then $f_{nj}^{(j)}(0)=0$ , $j=0,1,\dots ,n-1$ . Changing variables $x'\to -x'$ implies $f_{nj}(-\rho ) =(-1)^nf_{nj}(\rho ). $ Hence, setting a nonzero function $g_{nj}(f)(x)=f_{nj}(\rho )Y_{n}^{j}(x')\in \mathcal {S}(\mathbb {R}^d)$ , we have $g_{nj}(x)=\rho ^n\psi (\rho )Y_{n}^{j}(x')$ with some $\psi \in \mathcal {S}(\mathbb {R}_+)$ .

In order to apply the results obtained in case 1, we need to show that

$$ \begin{align*}\mathcal{F}_{\kappa,a}(g_{nj}(f))(y)=g_{nj}(\mathcal{F}_{\kappa,a}(f))(y).\end{align*} $$

Indeed, we note that

$$\begin{align*}\mathcal{F}_{\kappa,a}(g_{nj}(f))(y)=e^{-i\pi n/a}Y_n^j(y)H_{\lambda_{\kappa}+n,a}(\psi)(v). \end{align*}$$

On the other hand, in view of Proposition 5.7, we deduce that

$$ \begin{align*} g_{nj}(\mathcal{F}_{\kappa,a}(f))(y)&=Y_{n}^{j}(y')\int_{\mathbb{S}^{d-1}}\mathcal{F}_{\kappa,a}(f)(vy')Y_{n}^{j}(y')\,d\sigma_{\kappa,a}(y')\\ &=e^{-\frac{i\pi n}{a}}Y_{n}^{j}(y) \int_{\mathbb{R}^d}\frac{f(x)\Gamma(2\lambda_{\kappa}/a+1)}{a^{2n/a} \,\Gamma(2(\lambda_{\kappa}+n)/a+1)} j_{\frac{2(\lambda_{\kappa}+n)}{a}}\Bigl(\frac{2}{a}\,(\rho v)^{a/2}\Bigr)Y_n^j(x)\,d\mu_{\kappa,a}(x) \\ &=e^{-\frac{i\pi n}{a}}Y_{n}^{j}(y)\int_{0}^{\infty}\psi(\rho) j_{\frac{2(\lambda_{\kappa}+n)}{a}}\Bigl(\frac{2}{a}\,(\rho v)^{a/2}\Bigr)\,d\nu_{\lambda_{\kappa}+n,a}(\rho)\\ &=e^{-\frac{i\pi n}{a}}Y_n^j(y)H_{\lambda_{\kappa}+n,a}(\psi)(v). \end{align*} $$

Assuming here that $f,\mathcal {F}_{\kappa ,a}(f)\in \mathcal {S}(\mathbb {R}^d)$ yields $g_{nj}(f),\mathcal {F}_{\kappa ,a}(g_{nj}(f))\in \mathcal {S}(\mathbb {R}^d)$ , which contradicts the first case.

Summarizing, the generalized Fourier transform for irrational a drastically deforms even very smooth functions. It was mentioned in [Reference Ben Saïd, Kobayashi and Ørsted4, Chapter 5] that the generalized Fourier transform has a finite order only for rational a. Therefore, the case of irrational a is of little interest in harmonic analysis.