Finite transitive groups having many suborbits of cardinality at most two and an application to the enumeration of Cayley graphs

Let $G$ be a finite transitive group on a set $\Omega$, let $\alpha\in \Omega$ and let $G_\alpha$ be the stabilizer of the point $\alpha$ in $G$. In this paper, we are interested in the proportion $$\frac{|\{\omega\in \Omega\mid \omega \textrm{ lies in a }G_\alpha\textrm{-orbit of cardinality at most two}\}|}{|\Omega|},$$ that is, the proportion of elements of $\Omega$ lying in a suborbit of cardinality at most two. We show that, if this proportion is greater than $5/6$, then each element of $\Omega$ lies in a suborbit of cardinality at most two and hence $G$ is classified by a result of Bergman and Lenstra. We also classify the permutation groups attaining the bound $5/6$. We use these results to answer a question concerning the enumeration of Cayley graphs. Given a transitive group $G$ containing a regular subgroup $R$, we determine an upper bound on the number of Cayley graphs on $R$ containing $G$ in their automorphism groups.


Introduction
This paper is part of a series [17,19,24] aiming to obtain an asymptotic enumeration of finite Cayley graphs.However, the main players in this paper are not finite Cayley graphs, but finite transitive groups.Our results on finite transitive groups can then be used to make a considerable step towards the enumeration problem of Cayley graphs and thus getting closer to solving an outstanding question of Babai and Godsil, see [2] or [7,Conjecture 3.13].
Let G be a finite transitive group on Ω, let α ∈ Ω and let G α be the stabilizer in G of the point α.The orbits of G α on Ω are said to be the suborbits of G and their cardinalities are said to be the subdegrees of G.In this paper, we are concerned in finite transitive groups having many subdegrees equal to 1 or 2. In particular, we are interested in the ratio I Ω (G) := |{ω ∈ Ω | ω lies in a G α -orbit of cardinality at most two}| |Ω| .
As G is transitive on Ω, the value of I Ω (G) does not depend on α.Clearly, 0 < I Ω (G) ≤ 1.
Theorem 1.1.Let G be a finite transitive group on Ω, let α ∈ Ω and let G α be the stabilizer in G of the point α.If I Ω (G) > 5  6 , then I G (G α ) = 1, that is, each suborbit of G has cardinality at most 2. It turns out that finite transitive groups G with I Ω (G) = 1 are classified by a classical result of Bergman and Lenstra [3].The result of Bergman and Lenstra is rather general and applies to arbitrary (i.e. not necessarily finite) groups.The proof of [3,Theorem 1] is very beautiful and it is based on certain equivalence relations; also the strengthening of Isaacs [14] of the theorem of Bergman and Lenstra has a remarkably ingenious proof.
From [3, Theorem 1], finite transitive groups with I Ω (G) = 1 can be partitioned in three families In the first family, each suborbit of G has cardinality 1, that is, G acts regularly on Ω.In the second family, since G α has cardinality 2, each orbit of G α has cardinality at most 2. In the third family, since N G, the orbits of N on Ω form a system of imprimitivity for the action of G; as |N : G α | = 2, the blocks of this system of imprimitivity have cardinality 2 and hence all orbits of G α have cardinality at most 2. Theorem 1.1 shows that, with respect to the operator I Ω (G), there is a gap between 5/6 and 1.The value 5/6 is special: there exist finite transitive groups attaining the value 5/6.Theorem 1.2.Let G be a finite transitive group on Ω, let α ∈ Ω and let G α be the stabilizer in G of the point α.If I Ω (G) = 5  6 , then there exists an elementary abelian normal 2-subgroup N of G with |V : G α | = |G α | = 4.Moreover, let e 1 , e 2 , e 3 , e 4 be a basis of V , regarded as a 4-dimensional vector space over the field with 2 elements, with G α = e 1 , e 2 , let H := G/C G (V ) where C G (V ) is the centralizer of V in G, and let K be the stabilizer of the subspace Then, H is K-conjugate to one of the following two groups: The first group has order 12 and is isomorphic to the alternating group of degree 4 and the second group has order 24 and is isomorphic to the symmetric group of degree 4.
Conversely, if G is a finite group containing an elementary abelian normal 2-subgroup V := e 1 , e 2 , e 3 , e 4 of order 16 and H := G/C G (V ) is as above, then the action of G on the set Ω of the right cosets of e 1 , e 2 gives rise to a finite permutation group of degree 4|G : V | with I Ω (G) = 5/6.Theorem 1.2 classifies the finite transitive groups attaining the bound 5/6.Before discussing our motivation for proving Theorems 1.1 and 1.2, we make some speculations.A computer search among the transitive groups G of degree at most 48 with the computer algebra system magma [4] reveals that, if I Ω (G) > 1/2, then I Ω (G) = (q + 1)/2q, for some q ∈ Q with 2q ∈ N. We pose this as a conjecture.
If true, Conjecture 1.3 establishes a permutation analogue with a classical problem in finite group theory.Let G be a finite group and let I(G) := {x ∈ G | x has order at most 2}.
Miller [16] in 1905 has shown that, if I(G) > 3/4, then each element of G has order at most 2 and hence G is an elementary abelian 2-group.In this regard, Theorem 1.1 can be seen as a permutation analogue of the theorem of Miller, with the only difference that the ratio 3/4 in the context of abstract groups has to bump up to 5/6 in the context of permutation groups.Miller has also classified the finite groups G with I(G) = 3/4.Therefore, Theorem 1.2 can be seen as a permutation analogue of the classification of Miller.The theorem of Miller has stimulated a lot of research; for instance, Wall [25] has classified all finite groups G with I(G) > 1/2.In his proof, Wall uses the Frobenius-Schur formula for counting involutions.An application of this classification shows that, if I(G) > 1/2, then I(G) = (q + 1)/2q, for some positive integer q.Therefore, in Conjecture 1.3, we believe that the same type of result holds for the permutation analogue I Ω (G), but allowing q to be an element of {x/2 | x ∈ N}.As a wishful thinking, we also pose the following problem.
Liebeck and MacHale [15] have generalized the results of Miller and Wall in yet another direction.Indeed, Liebeck and MacHale have classified the finite groups G admitting an automorphism inverting more than half of the elements of G. (The classical results of Miller and Wall can be recovered by considering the identity automorphism.)Then, this classification has been pushed even further by Fitzpatrick [6] and Hegarty and MacHale [9], by classifying the finite groups G admitting an automorphism inverting exactly half of the elements of G.An application of this classification shows that, if α is an automorphism of G inverting more than half of the elements of G, then the proportion of elements inverted by α is (q + 1)/2q, for some positive integer q.Yet again, another analogue with Theorems 1.1 and 1.2, with Conjecture 1.3 and with Problem 1.4.We observe that a partial generalization of this type of results in the context of association schemes is in [20].
We now discuss our original motivation for proving Theorems 1.1 and 1.2.A digraph Γ is an ordered pair (V, E) with V a finite non-empty set of vertices, and E is a subset of V × V , representing the arcs.A graph Γ is a digraph (V, E), where the binary relation E is symmetric.An automorphism of a (di)graph is a permutation on V that preserves the set E.
Definition 1.5.Let R be a group and let S be a subset of R. The Cayley digraph Γ(R, S) is the digraph with V = R and (r, t) ∈ E if and only if tr −1 ∈ S.
The Cayley digraph is a graph if and only if S = S −1 , that is, S is an inverse-closed subset of R.
The problem of finding graphical regular representations (GRRs) for groups has a long history.Mathematicians have studied graphs with specified automorphism groups at least as far back as the 1930s, and in the 1970s there were many papers devoted to the topic of finding GRRs (see for example [1,10,11,12,13,21,22,23,26]), although the "GRR" terminology was coined somewhat later.Definition 1.6.A digraphical regular representation (DRR) for a group R is a digraph whose full automorphism group is the group R acting regularly on the vertices of the digraph.
Similarly, a graphical regular representation (GRR) for a group R is a graph whose full automorphism group is the group R acting regularly on the vertices of the graph.
It is an easy observation that when Γ(R, S) is a Cayley (di)graph, the group R acts regularly on the vertices as a group of graph automorphisms.A DRR (or GRR) for R is therefore a Cayley (di)graph on R that admits no other automorphisms.
The main thrust of much of the work through the 1970s was to determine which groups admit GRRs.This question was ultimately answered by Godsil in [8].The corresponding result for DRRs was proved by a much simpler argument by Babai [1].
Babai and Godsil made the following conjecture.(Given a finite group R, 2 c(R) denotes the number of inverse-closed subsets of R. See Definition 1.10 for the definition of generalized dicyclic group.)Conjecture 1.7 ([2]; Conjecture 3.13, [7]).If R is not generalised dicyclic or abelian of exponent greater than 2, then for almost all inverse-closed subsets S of R, Γ(R, S) is a GRR.In other words, lim : R admits a GRR| = 1.
From Godsil's theorem [8], as |R| → ∞, the condition "R admits a GRR" is equivalent to "R is neither a generalised dicyclic group, nor abelian of exponent greater than 2." The corresponding conjecture for Cayley digraphs (which does not require any families of groups to be excluded) was proved by Morris and the author in [19].Our current strategy for proving the conjecture of Babai and Godsil is to use the proof of the corresponding conjecture for Cayley digraphs as a template and extend the work in [19] in the context of undirected Cayley graphs.This strategy so far has been rather successful and in [17,24] the authors have already adapted some of the arguments in [19] for undirected graphs.
One key tool in [19] is an elementary observation of Babai.
Lemma 1.8.Let G be a finite transitive group properly containing a regular subgroup R. Then there are at most 2 The proof of this fact is elementary, see for instance [19,Lemma 1.8].Observe that the number of Cayley digraphs on R is the number of subsets of R, that is, 2 |R| .Therefore, Lemma 1.8 says that, given G properly containing R, only at most 2 |R|− |R| 4 of these Cayley digraphs admit G as a group of automorphisms.This gain of |R|/4 is one of the tools in [19] for proving the Babai-Godsil conjecture on Cayley digraphs.
To continue our project of proving the Babai-Godsil conjecture for Cayley graphs, we need an analogue of Lemma 1.8 for Cayley graphs.Observe that the number of Cayley graphs on R is the number of inverse-closed subsets of R. We denote this number with 2 c(R) .It is not hard to prove (see for instance [17,Lemma 1.12]) that where To obtain this analogue one needs to investigate finite transitive groups having many suborbits of cardinality at most 2. Therefore, our investigation leads to the following result.
Theorem 1.9.Let G be a finite transitive group properly containing a regular subgroup R. Then one of the following holds (a) the number of Cayley graphs 1.1.Notation.In this section, we establish some notation that we use throughout the rest of the paper.
Given a subset X of permutations from Ω, we use an exponential notation for the action on Ω and hence, in particular, given ω ∈ Ω, we let where ω x is the image of ω under the permutation x.Similarly, we let Let G be a transitive permutation group on Ω.For each positive integer i and for each ω ∈ Ω, we let and the non-empty sets in this union form a partition of Ω.
that is, Ω ω,1 is the set of fixed points of G ω on Ω.It is well-known that Ω ω,1 is a block of imprimitivity for the action of G on Ω, see for instance [5, 1.6.5].Since this fact will play a role in what follows, we prove it here; this will also be helpful for setting up some additional notation.
We let As G is transitive on Ω, d does not depend on ω.From the previous paragraph, we deduce that d divides |Ω ω,i |, for each positive integer i.We define (3) In particular, x 1 := 1 and, from ( 2) and ( 3), we have Definition 1.10.Let A be an abelian group of even order and of exponent greater than 2, and let y be an involution of A.
The generalised dicyclic group Dic(A, y, x) is the group ) is called a dicyclic or generalised quaternion group.

Lemmata
In this section we use the notation established in Section 1.1.
Lemma 2.1.Let G be a finite permutation group on a set Ω and let α ∈ Ω.
From this we deduce The proof for all other equalities in ( 5) is similar.From (5), we obtain Recall that, by hypothesis, Using this together with (6), we get However, this contradicts (4).Therefore, G α ∩ G ω and G β ∩ G ω are two distinct subgroups of G ω having index 2.This yields From ( 8) and from the fact that |G ω : We have summarized this paragraph in Figure 2. In other words, β ∈ Ω α,4 .
We now need two rather technical lemmas, at first they seem out of context, but their relevance is pivotal in the proof of Lemma 2.5.We could phrase Lemma 2.3 in a purely group theoretic terminology, but it is easier to state in our opinion using some terminology from graph theory.
be the collection of the proper subgroups of X and Y , respectively, properly containing Z.
Let Γ be the bipartite graph having vertex set Λ X ∪ Λ Y , where a pair {X i , Y j } is declared to be adjacent if X i Y j is a subgroup of G conjugate to X via an element of G.If Γ has at least 6 edges, then X commutes with Y .
Proof.Suppose that ( * ) there exist two distinct vertices of Γ having valency at least 2.
By symmetry, without loss of generality, we suppose that these two vertices are in Λ X .Thus suppose that X i , X j ∈ Λ X have valency at least 2 in Γ.
Let Y i1 and Y i2 be two neighbours of X i in Γ.Then, by definition, X i Y i1 and X i Y i2 are both subgroups of G conjugate to X. Therefore, X i Y i1 and X i Y i2 are elementary abelian 2-groups and hence X i commutes with both Y i1 and Y i2 .Since Y i1 , Y i2 = Y , we deduce that X i commutes with Y .
Arguing as in the paragraph above with X i replaced by X j , we deduce that X j commutes with Y .Therefore, X = X i , X j commutes with Y .Now, it is elementary to see that every bipartite graph on six vertices, with parts having cardinality 3 and having at least 6 edges has the property ( * ).
Recally that a graph Γ is said to be vertex-transitive if its automorphism group acts transitively on the vertices of Γ.Given a vertex ω of Γ, we denote by Γ(ω) the neighbourhood of ω in Γ.
Lemma 2.4.Let Γ be a finite vertex-transitive graph having valency 2, let V be the set of vertices of Γ, let ω 1 , ω 2 be two adjacent vertices of Γ and let W be a subset of V containing ω 1 and ω 2 and with the property that, for any two distinct vertices Proof.Since Γ is vertex-transitive of valency 2, Γ is a disjoint union s of cycles of the same length ℓ.If ℓ ≥ 7 or if Γ is disconnected, that is, s ≥ 2, it can be easily checked that W = V .

Lemma 2.5. Let G be a finite permutation group on a set Ω and let
Clearly, this 2-group P must be either cyclic of order 4, or elementary abelian of order 4, or dihedral of order 8.
We have drawn in Figure 3 the lattice of subgroups of the cyclic group of order 4, the elementary abelian group of order 4 and the dihedral group of order 8: the dark colored nodes indicate the lattice of subgroups between the whole group and the stabilizer of a point.
Figure 3 shows that, given G α and G α ∩ G β , we only have one choice for G α ∩ G ω when P is cyclic of order 4 or dihedral of order 8, whereas we have at most three choices for G α ∩ G ω when P is elementary abelian of order 4.

Given
Observe that in the set S α,β we are collecting point stabilizers and not elements of Ω and hence different elements ω 1 , ω 2 of Ω can give rise to the same element of S α,β when G ω1 = G ω2 .
We claim that ( 9) 3 when the permutation group induced by G α on β Gα or by G β on α G β is not an elementary abelian 2-group of order 4, 9 otherwise.This claim follows from the paragraphs above and from Figure 3. Indeed, from Lemma 2.1 part (c), for each X ∈ S α,β , there exists a proper subgroup A of G α and a proper subgroup Observe that we have at most 3 choices for A and at most 3 choices for B and hence at most 9 choices for X.Moreover, as long as the permutation group induced on the corresponding orbit is not elementary abelian, we actually have only one choice for either A or B yielding at most 3 choices for X.
We now suppose that part (a) does not hold and we show that part (bi), (bii), (biii) and (biv) are satisfied.In particular, we work under the assumption that 9), we deduce that x 4 ≤ 2. This already shows part (bi).When x 4 = 2, we deduce |S α,β | ≥ 7 and hence (9) yields that the permutation groups induced by G α on β Gα and by G β on α G β are both elementary abelian 2-groups of order 4. Since this argument does not depend upon β ∈ Ω α,4 , we have shown that G α acts as an elementary abelian group on each of its orbits of cadinality 4. Since all other orbits of G α have cardinality 1 or 2, we deduce that G α acts as an elementary abelian 2-group on each of its orbits and hence G α is an elementary abelian 2-group.This shows part (bii), under the additional assumption that x 4 = 2.Moreover, as |S α,β | ≥ 7, Lemma 2.3 applied with X := G α and Y := G β gives that G α and G β commute with each other.This shows that part (biii) is satisfied.To prove part (biv) we use Lemma 2.4.Let Γ be the graph having vertex set V , the set of conjugates of G α in G, that is, We declare two vertices G ω1 and G ω2 of Γ adjacent if G ω1 ∩ G ω2 has index 4 in G ω1 (and hence also in G ω2 ).Clearly, the action of G by conjugation gives rise to a vertex-transitive action of G on Γ.As x 4 = 2, Γ has valency 2. Let W be the collection of all vertices G ω of Γ with G α ∩ G β ≤ G ω .Clearly, G α , G β ∈ W and, from Lemma 2.1 part (c), for any two distinct vertices G δ1 and G δ2 of Γ contained in W , we have that From this, Lemma 2.4 gives that either W = V or |V | ≤ 6.The second alternative gives is an elementary abelian 2-group of order 16.To prove that G α × G β G it suffices to apply again this argument to the collection W of all vertices In particular, in the rest of the proof we work under the assumption x 4 = 1.
From Lemma 2.1 part (c), Observe also that when x 4 = 1, the hypothesis of Lemma 2.2 are satisfied and hence and G α × G β has order 16.Thus part (biv) is satisfied.
We need one final preliminary lemma, with a somehow different flavour.We denote by C 2 and C 4 the cyclic groups of order 2 and 4, respectively, we denote by Q 8 the quaternion group of order 8 and we denote by D 8 the dihedral group of order 4.
Lemma 2.6.Let R be a finite group, let U be a proper subgroup of R and let r ∈ U be a central involution of R. Let τ : R → R be the permutation defined by

Then one of the following holds (a) the number of inverse-closed subsets
2 , for some non-negative integer ℓ.Proof.Let ι : R → R be the permutation defined by x ι = x −1 , for every x ∈ R, and let T := ι, τ .Observe that a subset S of R is inverse-closed and τ -invariant if and only if S is T -invariant.In particular, the number of inverse-closed subsets S of R with S τ = S is 2 κ , where κ is the number of orbits of T on R. To compute κ we use the orbit-counting lemma, which says that Observe that ιτ = τ ι and ι 2 = τ 2 = 1.Therefore T is an elementary abelian 2-group of order at most 4.
Observe that τ = 1, because U is a proper subgroup of R and r = 1.If ι = 1, then R is an elementary abelian 2-group and T = τ .Thus ( 13) and ( 14) yield Therefore, part (a) holds and the proof follows in this case.Suppose now ι = τ .This means that U is an elementary abelian 2-subgroup of R and x −1 = xr, for every x ∈ R \ U .In other words, all elements in U square to 1 and all elements in R \ U square to r.Let R := R/ r and let us use the "bar" notation for the subgroups and for the elements of R. Consider the function (•, •) : R × R → r defined by (x r , y r ) = x −1 y −1 xy, for every x, y ∈ R. Similarly, consider the function q : R → r defined by q(x r ) = x 2 .It is not hard to see that, regarding R as a vector space over the field with 2 elements, (•, •) is a bilinear form and q is a quadratic form polarizing to (•, •), that is, q(xȳ)q(x)q(ȳ) = (x, ȳ), for every x, ȳ ∈ R. Using this terminology, we have that each element of Ū is totally singular and each element of R \ Ū is non-degenerate.From the classification of the quadratic forms over finite fields, we have | R : Ū | ∈ {2, 4}.When | R : Ū | = 2, we deduce that R is an abelian group isomorphic to the direct product C 4 × C ℓ 2 , for some ℓ ≥ 0. In particular, part (c) holds.When | R : Ū | = 4, we deduce that R ∼ = Q 8 × C ℓ 2 , for some ℓ ≥ 0. In particular, R is generalized dicyclic and part (b) holds.For the rest our our argument, we may suppose that τ = ι = 1.
The paragraph above can be summarized by saying that T = ι, τ has order 4 and hence, from ( 14), ( 13) becomes If S = ∅, then the proof follows immediately from (15), indeed, part (a) holds true.Therefore, for the rest of the proof we suppose S = ∅.
To conclude we divide the proof in various cases.Suppose first that |R : U | = 2. Let x ∈ S and observe that R = U ∪ U x. Now, a computation yields When S = U x, the action of x on U by conjugation is an automorphism of U inverting each element of U .Therefore U is abelian and R is generalized dicyclic.Hence part (b) holds.When S U x, the result of Liebeck and MacHale [15] shows that the automorphism  15) and ( 16), we deduce and the proof follows.Therefore, for the rest of the proof, we suppose Let u be an arbitrary element of U .Then uS ⊆ R \ U and hence S ∪ uS ⊆ R \ U .Therefore Now, let ux ∈ S ∩ uS.Then x ∈ S and hence r = (ux) 2 = uxux = uu x x 2 = uu x r.
Therefore u x = u −1 .Now, repeating the argument above with y ∈ S ∩ uS, we deduce u y = u −1 and hence xy −1 ∈ C R (u).Since we have |S ∩ uS| choices for y, (17) implies |C R (u)| > |R|/2 and hence R = C R (u).Since u is an arbitrary element of U , we deduce that U is a central subgroup of R.
Since u x = u −1 , for every u ∈ U and for every ux ∈ S ∩ uS, and since U is contained in the center of R, we deduce that U has exponent 2. Since U is a central subgroup of R of exponent 2, we now have an easier description of for S, that is, Now that we know that U has exponent 2, we consider the quotient group R := R/ r .Observe that each element of Ū is an involution.Assume that R is not an elementary abelian 2-group.Then, the theorem of Miller [16]  defined by (x r , y r ) = x −1 y −1 xy, for every x, y ∈ R, and its quadratic form q : R → r defined by q(x r ) = x 2 .Again, we use the classification of the quadratic forms over finite fields.Using this terminology, Ū is totally isotropic and contained in the kernel of the the bilinear form (•, •) and the elements of S are non-degenerate.Using this information we obtain that R is isomorphic to one of the following groups 2 , for some ℓ ≥ 0 and t ≥ 1, ×C ℓ 2 , for some ℓ ≥ 0 and some t ≥ 1, ×C ℓ 2 , for some ℓ ≥ 0 and some t ≥ 1.
In Thus, we obtain that part (a) holds.In the forth (and last) case, an explicit computation gives |S| = |R|/2.Therefore we may argue as in the first case and we obtain that part (a) holds.
We have shown that S ⊆ R is G 1 -invariant if and only if S is τ -invariant.Therefore, the proof of this case follows from Lemma 2.6.
To conclude the proof of Theorem 1.9, it remains to consider the case that Let ι : R → R be the permutation defined by x ι := x −1 , for every x ∈ R, and let T := ι, G 1 .Now, the number of G 1 -invariant inverse-closed subsets of R is exactly the number of T -invariant subsets of R.Moreover, the number of T -invariant subsets of R is 2 κ , where κ is the number of orbits of T on R.
(a) finite transitive groups G where the stabilizer G α has order 1, (b) finite transitive groups G where the stabilizer G α has order 2, (c) finite transitive groups G admitting an elementary abelian normal 2-subgroup N with |N : G α | = 2.