1. Introduction
Much of modern Schubert calculus centers on Schubert varieties in the complete flag variety. These Schubert varieties are closely related to affine varieties called matrix Schubert varieties, introduced by Fulton [Reference Fulton11]. Specifically, Schubert polynomials, introduced by Lascoux and Schützenberger [Reference Lascoux and Schützenberger23], form an additive basis of the cohomology ring of the complete flag variety. Expanding products of Schubert polynomials in this basis is arguably the major open problem in Schubert calculus today. Knutson and Miller [Reference Knutson and Miller17] showed that Schubert polynomials are also the multidegrees of matrix Schubert varieties. Moreover, a Schubert variety and its associated matrix Schubert variety share a codimension [Reference Fulton11]; the Cohen–Macaulay property of all Schubert varieties can be inferred from the Cohen–Macaulay property of all matrix Schubert varieties [Reference Fulton11, Reference Knutson and Miller17]; a Schubert variety is Gorenstein if and only if its associated matrix Schubert variety is [Reference Woo and Yong36].
In the course of studying matrix Schubert varieties, one is naturally led to study intersections of several of them, for example, as they arise in Gröbner degenerations and in inductive computations [Reference Klein and Weigandt22]. Weigandt [Reference Weigandt35] showed that arbitrary intersections of matrix Schubert varieties are indexed by what are called alternating sign matrices (ASMs), a generalization of permutation matrices. Permutation matrices index Schubert varieties and matrix Schubert varieties. ASMs were not invented in order to perform this indexing. Rather, they are combinatorial objects with their own rich history: In 1983, Mills, Robbins, and Rumsey [Reference Mills, Robbins and Rumsey26] gave a conjecture for a closed form for the number of
$n \times n$
ASMs. The original proof was given by Zeilberger [Reference Zeilberger37], and a second proof was given by Kuperberg [Reference Kuperberg21] using the six-vertex model of statistical mechanics.
Matrix Schubert varieties are Cohen–Macaulay and admit a codimension formula that can be read easily from their indexing permutation [Reference Fulton11]. These facts are regularly exploited in studying the Schubert and Grothendieck polynomials of their associated permutations and in recent computations of Castelnuovo–Mumford regularity (see, e.g., [RRR+21, Reference Pechenik, Speyer and Weigandt30]). ASM varieties do not, in general, enjoy either of these advantages.
The purpose of this paper is to study the codimension and potential Cohen–Macaulayness of ASM varieties. Let
$X_A$
denote the ASM variety associated to
$A$
. We focus on the embeddings
$\tt ASM(n) \hookrightarrow \tt ASM(n+1)$
determined by
$A \mapsto 1 \oplus A$
and, more generally, on ASMs obtained as direct sums of other ASMs. This embedding is natural in light of the role its restriction to
$S_n$
plays in the theory of Stanley symmetric functions [Reference Stanley33, Reference MacDonald25].
Let
$X_A$
denote the ASM variety determined by the ASM
$A$
. We show the following:
Theorem 1.1. (Theorem 3.19).
-
(1)
$\texttt {codim}(X_{A_1 \oplus A_2}) = \texttt {codim}(X_{A_1}) + \texttt {codim}(X_{A_2})$ .
-
(2)
$X_{A_1 \oplus A_2}$ is equidimensional if and only if
$X_{A_1}$ and
$X_{A_2}$ are both equidimensional.
We conjecture that (2) holds if equidimensional is replaced with Cohen–Macaulay (Conjecture 3.15). We prove one direction of the conjecture and provide empirical evidence for the other (Theorem 3.19).
We give examples to show that the method used by Knutson and Miller [Reference Knutson and Miller17] to prove that matrix Schubert varieties are Cohen–Macaulay, recovering a result of Fulton [Reference Fulton11], does not extend to the Cohen–Macaulay ASMs (Example 3.6 and Figure 1). We also give data suggesting that the property of being Cohen–Macaulay becomes increasingly rare as the size of the ASM grows (Figure 2). These data motivate the study of ASM pattern avoidance from an algebro-geometric perspective. We argue that the naive notion of pattern avoidance does not explain the eventual rarity of Cohen–Macaulay ASMs (Example 4.2 and Proposition 4.3) but that there is still some algebro-geometric control enforced by this notion (Proposition 4.6). We hope that this will provide context for future studies of pattern avoidance of ASMs from an algebro-geometric perspective.
2. Background
Throughout this document,
$\kappa$
will denote an arbitrary field. For
$k \leq n \in \mathbb{Z}_+$
, let
$[n] = \{1, 2, \ldots , n\}$
and
$[k,n] = \{k, k+1, \ldots , n\}$
. Let
$S_n$
denote the symmetric group on
$[n]$
. We identify
$w \in S_n$
with the
$n \times n$
permutation matrix with
$1$
s in positions
$(i,w(i))$
,
$i \in [n]$
, and
$0$
s elsewhere.
Definition 2.1. An alternating sign matrix (ASM) is a square matrix with the following properties:
-
(1) Each entry is taken from the set
$\{-1,0,1\}$ .
-
(2) The entries in each row (resp. column) sum to 1.
-
(3) The nonzero entries in a row (resp. column) alternate between 1 and
$-1$ .
Let
$\tt ASM(n)$
denote the set of
$n \times n$
ASMs. The ASMs whose entries all lie in
$\{0,1\}$
are exactly the permutation matrices. For
$A \in \tt ASM(n)$
, note that the sum of all entries of
$A$
is
$n$
.
2.1. ASM varieties and ideals
We now outline the process for associating a variety and an ideal to an ASM.
Fix
$A = (A_{a,b}) \in \tt ASM(n)$
. Define a rank function
${rk}_A$
on
$[n] \times [n]$
by

The rank matrix of
$A$
is the
$n \times n$
matrix whose
$(i,j)$
entry is
${rk}_A(i,j)$
. (Note that
${rk}_A(i,j)$
is typically not the rank of the submatrix of
$A$
consisting of the first
$i$
rows and first
$j$
columns, unless
$A \in S_n$
.)
Throughout this paper, we will let
$Z = (z_{i,j})$
denote an
$n \times n$
generic matrix, that is, a matrix of distinct variables, and we will let
$R$
denote the polynomial ring over
$\kappa$
in the entries of
$Z$
. For
$i,j \in [n]$
, let
$Z_{[i],[j]}$
be the submatrix of
$Z$
consisting of the first
$i$
rows and
$j$
columns. For
$k \in \mathbb{Z}_+$
, let
$I_k(Z_{[i],[j]})$
denote the ideal of
$R$
generated by the
$k \times k$
minors of
$Z_{[i],[j]}$
.
Definition 2.2. For
$A \in \tt ASM(n)$
, let

We call
$I_A$
the ASM ideal of
$A$
. Let
$X_A$
denote the variety of
$I_A$
, which we call the ASM variety of
$A$
.
Every ASM ideal
$I_A$
is radical ([Reference Weigandt35, Proposition 5.4]). See also [Reference Fulton11, Proposition 3.3] for the case
$A \in S_n$
. When
$A \in S_n$
,
$I_A$
is called a Schubert determinantal ideal and
$X_A$
a matrix Schubert variety. An ASM variety is irreducible if and only if it is a matrix Schubert variety [Reference Fulton11, Proposition 3.3], [Reference Weigandt35, Proposition 5.4]. The codimension of
$X_w$
is equal to the Coxeter length of
$w$
, denoted
$\ell (w)$
([Reference Fulton11, Proposition 3.3]), also known as the inversion number of
$w$
. See [Reference Fulton11] for further information on matrix Schubert varieties, including their close connection to Schubert varieties in the complete flag variety.
We call the generators described in Definition 2.2 the natural generators of
$I_A$
. It is sometimes more convenient to work with a smaller set of generators called the Fulton generators, which we describe below, after giving some required auxiliary constructions. Fulton generators of ASM ideals were introduced by Weigandt [Reference Weigandt35], generalizing the description given by Fulton [Reference Fulton11] for Schubert determinantal ideals.
For
$A \in \tt ASM(n)$
, let

We call
$D(A)$
the Rothe diagram of
$A$
. (Some authors refer to
$D(A)$
as the inversion set of
$A$
.) Note that the definition of ASM forces
$\sum _{k = 1}^i A_{k,j}, \sum _{\ell = 1}^j A_{i,\ell } \in \{0,1\}$
for all
$i,j \in [n]$
.
Example 2.3. We give an example of a method for visualizing the Rothe diagram of an ASM. If
$A=\begin{pmatrix} 0 &\quad 0 &\quad 1 &\quad 0\\ 1 &\quad 0 &\quad -1 &\quad 1 \\ 0 &\quad 1 &\quad 0 &\quad 0\\ 0 &\quad 0 &\quad 1 &\quad 0 \end{pmatrix}$
, we may visualize
$D(A)$
by drawing

In an
$n \times n$
grid, we have placed a solid dot for every
$1$
in
$A$
and an open dot for every
$-1$
. Lines emanate down and to the right out of each solid dot, stopping before entering the cell of an open dot. Then
$D(A) = \{(1,1), (1,2), (2,3)\}$
consists of the indices of cells without a line intersecting their interior.
Let
$\texttt {Ess}(A) = \{(i,j) \in D(A) \mid (i,j+1), (i+1,j) \notin D(A)\}$
. We call
$\texttt {Ess}(A)$
the essential set of
$A$
and elements of
$\texttt {Ess}(A)$
the essential cells of
$A$
.
Returning to Example 2.3, we have
$\texttt {Ess}(A) = \{(1,2),(2,3)\}$
. We may visualize
$\texttt {Ess}(A)$
as the maximally southeast elements of connected components of
$D(A)$
(where two cells are considered adjacent if they share an edge).
It is a theorem due to Fulton [Reference Fulton11, Lemma 3.10] for
$A \in S_n$
and generalized for all
$A \in \tt ASM(n)$
by Weigandt [Reference Weigandt35, Lemma 5.9] that the ASM ideal
$I_A$
may be generated solely by considering rank conditions at essential cells:

We call this set of generators of
$I_A$
the Fulton generators.
Continuing with Example 2.3, the rank matrix of
$A$
is
${rk}_A=\begin{pmatrix} 0 & \fbox {0} & 1 & 1\\ 1 & 1 & \fbox {1} & 2 \\ 1 & 2 & 2 & 3\\ 1 & 2 & 3 & 4 \end{pmatrix}$
, where the ranks in essential cells appear in boxes. The Fulton generators of
$I_A$
are

The degree
$1$
generators are determined by the essential cell
$(1,2)$
, and the degree
$2$
generators are determined by the essential cell
$(2,3)$
.
For
$A \in \tt ASM(n)$
, let
$\texttt {Dom}(A) = \{(i,j) \in [n] \times [n] \mid {rk}_A(i,j) = 0\}$
, and call
$\texttt { Dom}(A)$
the dominant part of
$A$
. Note that
$\texttt {Dom}(A) \subseteq D(A)$
and that
$\texttt {Dom}(A)$
consists exactly of the indices of the degree
$1$
Fulton generators of
$I_A$
, that is, the generators which are single variables. In Example 2.3,
$\texttt {Dom}(A) = \{(1,1),(1,2)\}$
.
3. Stability under embeddings and limitations
It is of central import in Schubert calculus that core constructions (e.g., the (double) Schubert polynomial, the defining equations of a matrix Schubert variety) depending on the permutation
$w \in S_n$
do not change if we instead view
$w$
as an element of
$S_{n+1}$
under the embedding
$S_n \hookrightarrow S_{n+1}$
where each function is extended by a fixed point at
$n+1$
. Constructions and invariants vary, though predictably, if instead we consider the embedding
$S_n \hookrightarrow S_{n+1}$
where
$w \in S_n$
is mapped to the function
$w' \in S_{n+1}$
where
$w'(1) = 1$
and
$w'(i) = w(i-1)+1$
for
$2 \leq i \leq n+1$
. For example, the assignment
$(i,j) \mapsto (i+1,j+1)$
gives a bijection between
$D(w)$
and
$D(w')$
;
${rk}_w(i,j) = {rk}_{w'}(i+1,j+1)-1$
for all
$(i,j) \in D(w)$
;
$\texttt {codim}(X_w) = \texttt{codim}(X_{w'})$
; and the Castelnuovo–Mumford regularity (an important algebraic invariant) of one may be inferred easily from the other via [Reference Pechenik, Speyer and Weigandt30]. This embedding plays an important role in the rich theory of Stanley symmetric functions, introduced by Stanley [Reference Stanley33], which MacDonald [Reference MacDonald25] showed to be the stable limit of Schubert polynomials.
For
$A \in \tt ASM(n)$
, let
$1 \oplus A$
denote the direct sum of the
$1 \times 1$
identity matrix with
$A$
and
$A \oplus 1$
the direct sum of
$A$
with the
$1 \times 1$
identity matrix, both of which are easily seen to be elements of
$\tt ASM(n+1)$
. As with permutations, we note that
$I_A$
and
$I_{A \oplus 1}$
have the same set of Fulton generators (in different polynomial rings). Thus,
$X_A$
and
$X_{A \oplus 1}$
differ only by products with affine factors. This is the extension of the first embedding of
$S_n \hookrightarrow S_{n+1}$
discussed above.
Motivated by the similarities between
$I_w$
and
$I_{1 \oplus w}$
,
$w \in S_n$
, we discuss the relationship between the ASM ideals
$I_A$
and
$I_{1 \oplus A}$
. That is, we consider the natural extension of the second embedding of
$S_n \hookrightarrow S_{n+1}$
discussed above. Specifically, we give a codimension-preserving bijection between the components of
$X_A$
and the components of
$X_{1 \oplus A}$
(Proposition 3.3), which can be viewed as an extension of the elementary fact that
$\ell (w) = \ell (1 \oplus w)$
for
$w \in S_n$
.
By contrast, we present data showing that a vertex decomposition of a simplicial complex naturally associated to
$A$
does not necessarily give rise to a vertex decomposition of the corresponding simplicial complex of
$1\oplus A$
(Figure 1). This failure is interesting because vertex decompositions of these simplicial complexes have algebro-geometric interpretations via [Reference Klein and Rajchgot20]. Specifically, had that implication held, it would have allowed one to lift certain proofs that certain
$X_A$
belong to the Gorenstein linkage class of a complete intersection to proofs that the corresponding
$X_{1 \oplus A}$
do. For the relevant Gorenstein linkage background, we direct the reader to [Reference Nagel and Römer28, Reference Klein and Rajchgot20].
This contrast motivates our interest in the relationship between Cohen–Macaulayness of
$A$
and
$1 \oplus A$
. Every variety in the Gorenstein linkage class of a complete intersection is Cohen–Macaulay, and every Cohen–Macaulay variety is equidimensional. In this sense, the Cohen–Macaulay property is a “middle ground” between the two previously discussed conditions. The equidimensionality result (Proposition 3.3) together with computer data lead us to conjecture that
$A$
is Cohen–Macaulay if and only if
$1 \oplus A$
is (Conjecture 3.15).
3.1. Equidimensionality
If
$A,B$
are
$n \times n$
ASMs, define
$A \ge B$
if
${rk}_A(i,j) \le {rk}_B(i,j)$
for all
$1 \le i,j \le n$
. Restricted to permutation matrices, this is the (strong) Bruhat order on
$S_n$
. Let

Proposition 3.1. [Reference Weigandt35, Proposition 5.4].
$I_A$
has the irredundant prime decomposition

The codimension of
$X_A$
is
$\min \{\ell (w) \mid w \in \textrm{Perm}(A)\}$
.
Definition 3.2. Call
$A$
equidimensional if all elements of
$\textrm{Perm}(A)$
have the same Coxeter length, or, equivalently, if
$X_A$
is equidimensional.
Proposition 3.3. Let
$A \in \tt ASM(n)$
and
$\textrm{Perm}(A) =\{w_1,\ldots ,w_k\}$
. For
$w \in S_n$
, the assignment
$w \mapsto 1 \oplus w$
gives a bijection between
$\textrm{Perm}(A)$
and
$\textrm{Perm}(1 \oplus A)$
. In particular,
$ \texttt {codim}(A)=\texttt {codim}(1 \oplus A)$
, and
$A$
is equidimensional if and only if
$1 \oplus A$
is equidimensional.
Proof.
If
$A \in S_n$
, then the claim is trivial, and so we assume
$A \notin S_n$
.
Suppose
$w \in \textrm{Perm}(A)$
. We will first show that
$1 \oplus w \in \textrm{Perm}(1 \oplus A)$
. From the definition of
$1 \oplus A$
, we compute

Thus,

Suppose there exists a
$v$
such that
$1 \oplus w \ge v \gt 1 \oplus A$
. From the equations

and

for all
$j \in [n+1]$
, we see that the first row of
$v$
is the same as the first row of
$1 \oplus A$
, which is the same as the first row of
$1 \oplus w$
—specifically, the
$1^{st}$
unit row vector. Similarly, the first column of all three matrices
$1 \oplus w$
,
$v$
, and
$1 \oplus A$
must be the
$1^{st}$
unit column vector.
Thus
$v = 1 \oplus v'$
for some
$v' \in S_n$
. By (1),
$w \geq v'\gt A$
. By definition of
$\textrm{Perm}(A)$
, the inequality
$w \geq v' \gt A$
implies
$w=v'$
, and so
$1 \oplus w = 1 \oplus v'=v$
. Hence,
$1 \oplus w \in \textrm{Perm}(1 \oplus A)$
.
In the other direction, fix some
$u \in \textrm{Perm}(1 \oplus A)$
. We claim that
$u = 1 \oplus u'$
for some
$u' \in S_n$
. It suffices to show that
$u(1) = 1$
.
Suppose for contradiction that
$u(1) = j$
for some
$j\gt 1$
and that
$i$
is the least index so that
$u(i)\lt j$
. Let
$\nu = ut_{1,i}$
, where
$t_{1,i}$
is the transposition
$(1i)$
. That is, in matrix form,
$\nu$
is the matrix obtained from
$u$
by exchanging rows
$1$
and
$i$
.
We claim that
$1 \oplus A \leq \nu \lt u$
, which will contradict the assumption
$u \in \textrm{Perm}(1 \oplus A)$
. Note that
${rk}_\nu (\alpha ,\beta ) = {rk}_u(\alpha ,\beta )$
unless
$\alpha \in [i-1]$
and
$\beta \in [u(i), j-1]$
, in which case
${rk}_u(\alpha ,\beta ) = {rk}_\nu (\alpha ,\beta )-1$
(see Example 3.4 for a visualization). Thus,
$\nu \lt u$
. Because
$i$
was chosen minimally,
${rk}_u(\alpha ,\beta ) = 0$
for all
$\alpha \in [i-1]$
and
$\beta \in [u(i), j-1]$
, and so
${rk}_\nu (\alpha ,\beta ) = 1$
for all such
$(\alpha , \beta )$
. But
${rk}_{1 \oplus A}(\alpha , \beta ) \ge 1$
for all
$\alpha ,\beta$
. It follows from
${rk}_\nu (\alpha ,\beta ) = {rk}_u(\alpha ,\beta ) \leq {rk}_{1 \oplus A}(\alpha , \beta )$
unless
$\alpha \in [i-1]$
and
$\beta \in [u(i), j-1]$
together with
${rk}_\nu (\alpha ,\beta ) = 1 \leq {rk}_{1 \oplus A}(\alpha ,\beta )$
for
$\alpha \in [i-1]$
and
$\beta \in [u(i), j-1]$
that
$1 \oplus A \leq \nu$
, completing the claim.
Hence, our arbitrary
$u \in \textrm{Perm}(1 \oplus A)$
has the form
$u = 1 \oplus u'$
for some
$u' \in S_n$
. It remains to show that
$u' \in \textrm{Perm}(A)$
. By (1),
$u' \gt A$
. Hence, if
$u' \notin \textrm{Perm}(A)$
, then there exists some
$\tilde {u} \in S_n$
satisfying
$A \lt \tilde {u}\lt u'$
. But then, by (1),
$1 \oplus A \lt 1 \oplus \tilde {u}\lt u$
, contradicting the assumption
$u \in \textrm{Perm}(1 \oplus A)$
.
Example 3.4. In order to help internalize the argument of Proposition 3.3, we will give a visualization of the region in which
${rk}_u$
and
${rk}_\nu$
differ for some
$\nu = ut_{1,i}$
. Consider
$u = 45213$
, in which case
$j =4$
,
$i = 3$
,
$u(i) = 2$
, and
$\nu = 25413$
. The region
$[3-1] \times [2,4-1]$
, in which
${rk}_u$
and
${rk}_\nu$
differ, is shaded in yellow.

Example 3.5. Proposition 3.3 involves inserting into an ASM a row and column whose entries are
$0$
except where they intersect, where the value is
$1$
. In the case of Proposition 3.3, it is specifically the first row and first column. It is worth noting that equidimensionality is not preserved by an arbitrary insertion of this type. For example, consider
$A = \begin{pmatrix} 0 &\quad 1 &\quad 0 \\ 1 &\quad -1 &\quad 1 \\ 0 &\quad 1 &\quad 0 \end{pmatrix}$
and
$B = \begin{pmatrix} 0 &\quad 1 &\quad 0 &\quad 0 \\ 0 &\quad 0 &\quad 1 &\quad 0 \\ 1 &\quad -1 &\quad 0 &\quad 1 \\ 0 &\quad 1 &\quad 0 &\quad 0 \end{pmatrix}$
, in which case
$B$
may be obtained from
$A$
by inserting
$(0,1,0,0)^T$
to become column
$3$
and
$(0,0,1,0)$
to become row
$2$
.
Then
$I_A = (z_{1,1}, z_{1,2}z_{2,1}) = I_{312} \cap I_{231}$
not only defines an equidimensional ASM variety but also a complete intersection while
$I_B = (z_{1,1}, z_{2,1}, z_{1,2}z_{3,1}, z_{2,2}z_{3,1}) = I_{3412} \cap I_{2341}$
has one component of codimension
$4$
and one of codimension
$3$
.
3.2. Vertex decomposition
We now review some basic definitions from simplicial complex theory. A simplicial complex
$\Delta$
on
$[n]$
is a set of subsets of
$[n]$
such that, if
$\sigma \in \Delta$
and
$\tau \subseteq \sigma$
, then
$\tau \in \Delta$
. We call the elements of
$\Delta$
faces and the maximal faces (by inclusion) facets. The dimension of a face
$\sigma \in \Delta$
is
$|\sigma |-1$
, and the dimension of
$\Delta$
is the maximum among dimensions of its faces. Faces of dimension
$0$
are called
$vertices$
. We call
$\Delta$
pure if all of its facets have the same dimension.
Now, we discuss how these ideas from simplicial complex theory relate to the objects we want to study. The Stanley–Reisner correspondence is a bijection between simplicial complexes on
$[n]$
and squarefree monomial ideals. For a subset
$\sigma$
of
$[n]$
, let
$\mathbf{x}_\sigma = \prod _{i \in \sigma } x_i$
. The Stanley–Reisner ideal of the simplicial complex
$\Delta$
is

The complements of the facets of
$\Delta$
correspond to the associated primes of
$I_\Delta$
. Hence,
$\Delta$
is pure if and only if the variety defined by
$I_\Delta$
is equidimensional. We call
$\Delta$
Cohen–Macaulay whenever
$I_\Delta$
defines a Cohen–Macaulay quotient ring.
Given a simplicial complex
$\Delta$
, we define the link of
$\Delta$
at a face
$\sigma$
of
$\Delta$
by

and the deletion of
$\Delta$
at a face
$\sigma$
of
$\Delta$
by

Notice that
$\texttt {lk}_\sigma (\Delta )$
is a subcomplex of
$\texttt {del}_{\sigma }(\Delta )$
. When
$\sigma = \{v\}$
is a vertex of
$\Delta$
, we will often write
$\texttt {lk}_{v}(\Delta )$
and
$\texttt {del}_{v}(\Delta )$
for
$\texttt { lk}_{\sigma }(\Delta )$
and
$\texttt {del}_{\sigma }(\Delta )$
, respectively.
A simplicial complex
$\Delta$
is vertex decomposable if
$\Delta$
is pure and if either
-
(1)
$\Delta = \{\emptyset \}$ , or if
-
(2) for some vertex
$v$ in
$\Delta$ , both
$\texttt {del}_{v}(\Delta )$ and
$\texttt {lk}_{v}(\Delta )$ are vertex decomposable.
Vertex decomposable simplicial complexes were introduced by Provan and Billera [Reference Provan and Billera29], who showed that vertex decomposable simplicial complexes are shellable, known more classically to be a virtue of a simplicial complex. Reisner [Reference Reisner31] gave a topological criterion characterizing Cohen–Macaulayness of
$\Delta$
from which it follows that every shellable simplicial complex is Cohen–Macaulay, a desirable algebraic property we discuss further in Section 3.3.
We will use Stanley–Reisner theory to study initial ideals of ASM ideals. We refer the reader to [Reference Miller and Sturmfels27, Chapter 1] for more information on Stanley–Reisner theory and to [Reference Eisenbud10, Chapter 15] for general background on term orders and Gröbner bases.
Recall that
$Z$
is a fixed
$n \times n$
generic matrix. We call a term order antidiagonal if the lead term of the determinant of any submatrix
$Y$
of
$Z$
is the product of the entries along the antidiagonal of
$Y$
. For each
$A \in \tt ASM(n)$
, the Fulton generators of
$I_A$
form a Gröbner basis with respect to any antidiagonal term order ([Reference Knutson and Miller17, Reference Knutson19, Reference Weigandt35]). (For
$w \in S_n$
, Gao and Yong [Reference Gao and Yong12] recently determined a minimal generating set of
$I_w$
which is already a Gröbner basis under any antidiagonal term order.) In particular, for any
$A \in \tt ASM(n)$
, there is only one initial ideal arising from an antidiagonal term order. We call that initial ideal the antidiagonal initial ideal of
$I_A$
, which we denote
$\texttt {in} I_A$
.
Knutson and Miller [Reference Knutson and Miller17] showed that, for
$w \in S_n$
,
$\texttt {in} I_w$
is not only Cohen–Macaulay but, moreover, also the Stanley–Reisner ideal of a vertex decomposable simplicial complex. In fact, for a fixed
$n$
, there is an ordering of the vertices (labeling the vertices by their associated variable)

by which the Stanley–Reisner complex of each
$\texttt {in} I_w$
is vertex decomposable, independent of
$w$
. We may hope that the same term order would show that every Stanley–Reisner ideal of every Cohen–Macaulay
$\texttt {in} I_A$
is vertex decomposable. It turns out that that is false.
Example 3.6. For example, let

The ideal of the deletion at that vertex corresponding to
$z_{13}$
is

which has associated primes of different heights, which shows that the deletion of the Stanley–Reisner complex of
$\texttt { in} I_A$
at the vertex corresponding to
$z_{13}$
is not pure. Hence, the Stanley–Reisner complex of
$\texttt{in} I_A$
is not vertex decomposable under the order used by Knutson and Miller.
If the Stanley–Reisner complex of
$\texttt {in} I_A$
is vertex decomposable by the order dictated by [Reference Knutson and Miller17], we will call
$A$
Knutson–Miller vertex decomposable, a definition which is local to this paper. We use the abbreviation CM for Cohen–Macaulay. The middle column of Figure 1 records the number of elements of
$\tt ASM(n)$
that are obtained as
$1 \oplus A$
from some
$A \in \tt ASM(n-1)$
(which is equivalent to satisfying
$A_{1,1} = 1$
) and that are not Knutson–Miller vertex decomposable. Note, for example, that there are two such elements of
$\tt ASM(5)$
but only one element of
$\tt ASM(4)$
that is not Knutson–Miller vertex decomposable. Hence, there exists at least one element of
$\tt ASM(5)$
of the form
$1 \oplus A$
that is not Knutson–Miller vertex decomposable though
$A$
is. Indeed, there is exactly one. The example is

This example shows that the operation
$1 \oplus -$
does not preserve Knutson–Miller vertex decomposability. This failure also appears to become more frequent as
$n$
grows. It is in part due to the frequency of this failure for small
$n$
that we are meaningfully encouraged by the preservation of Cohen–Macaulayness under
$1 \oplus -$
applied to all
$A \in \tt ASM(n)$
for
$n \leq 6$
, which we discuss further in Conjecture 3.15.

Figure 1. Count of Cohen–Macaulay elements of
$\tt ASM(n)$
that are and are not vertex decomposable by the Knutson–Miller ordering of the variables.
3.3. Cohen–Macaulayness
Cohen–Macaulayness of rings and varieties is a central consideration in commutative algebra and algebraic geometry, especially in the context of intersection theory. As Hochster said, “Life is really worth living in a Noetherian ring
$R$
when all the local rings have the property that every [system of parameters] is an
$R$
-sequence. Such a ring is called Cohen-Macaulay” [Reference Hochster13, p. 887]. For background on Cohen–Macaulay rings, we direct the reader to [Reference Bruns and Herzog5] and, especially for their import in combinatorial settings, to [Reference Hochster14].
Definition 3.7. We will say that the ASM
$A$
is Cohen–Macaulay if the ring
$R/I_A$
is Cohen–Macaulay.
In light of recent work of Conca and Varbaro [Reference Conca and Varbaro8], we will be able to study the Cohen–Macaulayness of each
$R/I_A$
via the Cohen–Macaulayness of a suitable Gröbner degeneration.
Proposition 3.8. [Reference Weigandt35, Reference Knutson19, Reference Klein and Weigandt22]. If
$w_1, \ldots , w_r, u_1, \ldots , u_k \in S_n$
and
$A \in \tt ASM(n)$
are such that

then

Proposition 3.9. [Reference Knutson and Miller17, Reference Bergeron and Billey3]. Fix
$w \in S_n$
. If
$P$
is an associated prime of
$\texttt {in} I_w$
, then the generators of
$P$
determine a reduced word for
$w$
. Hence,
$P$
is not an associated prime of any
$\texttt {in} I_u$
with
$w \neq u \in S_n$
.
Proposition 3.10. Fix
$A \in \tt ASM(n)$
. Then
$ R/I_A$
is Cohen–Macaulay if and only if
$R/\texttt {in} I_A$
is Cohen–Macaulay.
Proof.
The ideal
$I_A$
is homogeneous. Its initial ideal
$\texttt {in} I_A$
is radical by Proposition 3.8 because each
$\texttt {in} I_w$
is radical for
$w \in S_n$
[Reference Knutson and Miller17, Theorem B]. The main theorem of [Reference Conca and Varbaro8] states that a homogeneous ideal with a radical initial ideal defines a Cohen–Macaulay quotient if and only if the radical initial ideal does.
Our primary interest in what follows will be to infer Cohen–Macaulayness of
$A$
from Cohen–Macaulayness of
$1 \oplus A$
. Because the argument is essentially the same, we will prove something slightly stronger using the construction below. In the notation that follows,
$1 \oplus A = \widetilde {A}(1,1)$
.
Fix
$A \in \tt ASM(n)$
and
$i, j \in [n]$
. We will build an element of
$\tt ASM(n+1)$
, which we denote
$\widetilde {A} = \widetilde {A}(i,j)$
, by the following rules:

Example 3.11. Returning to Example 3.5, consider
$A = \begin{pmatrix} 0 &\quad 1 &\quad 0\\ 1 &\quad -1 &\quad 1\\ 0 &\quad 1 &\quad 0 \end{pmatrix}$
and then
$\widetilde {A} = \widetilde {A}(2,3)= \begin{pmatrix} 0 &\quad 1 &\quad {\color {blue}0} &\quad 0\\ {\color {blue}0} &\quad {\color {blue}0} &\quad {\color {blue}1} &\quad {\color {blue}0}\\ 1 &\quad -1 &\quad {\color {blue}0} &\quad 1\\ 0 &\quad 1 &\quad {\color {blue}0} &\quad 0 \end{pmatrix}$
(which in Example 3.5 was named
$B$
). The entries of
$\widetilde {A}$
inherited from
$A$
appear in black, and the new entries appear in blue.
The reader may verify that the Rothe diagram of
$A$
embeds into the Rothe diagram of
$\widetilde {A}$
. This close relationship may cause one to hope that the geometry of
$X_A$
and
$X_{\widetilde {A}}$
would be very closely related. However, as we discussed in Example 3.5,
$X_A$
may be Cohen–Macaulay while
$X_{\widetilde {A}}$
is not even unmixed. We now investigate special cases when
$X_A$
and
$X_{\widetilde {A}}$
are closely related.
As before, for
$\tau \subseteq [n]$
, let
$\mathbf{x}_\tau = \prod _{i \in \tau } x_i$
. Recall that, for an ideal
$I$
and element
$f$
of
$R$
, we define the colon ideal
$(I:f) = (r \in R \mid rf \in I)$
.
Lemma 3.12. Let
$\Delta$
be a simplicial complex on
$[n]$
. If
$\sigma$
is a face of
$\Delta$
, then

Proof.
Fix a squarefree monomial
$\mathbf{x}_\tau$
corresponding to the subset
$\tau$
of
$[n]$
. If
$\tau \cap \sigma \neq \emptyset$
, then
$\mathbf{x}_\tau \in (x_i \mid i \in \sigma )$
. Also, by the definition of link,
$\tau \notin \texttt{lk}_\sigma (\Delta )$
, and so
$\mathbf{x}_\tau \in I_{\texttt {lk}_\sigma (\Delta )}$
.
Now assume
$\tau \cap \sigma = \emptyset$
. Then

For
$A \in \tt ASM(n)$
, we write
$\Delta (A)$
for the Stanley–Reisner complex of
$\texttt {in} I_A$
.
Theorem 3.13. Let
$A \in \tt ASM(n)$
, fix
$j \in [n+1]$
, and set
$\widetilde {A} = \widetilde {A}(1,j)$
. If
$\widetilde {A}$
is Cohen–Macaulay then
$A$
is Cohen–Macaulay.
Proof. Recall that the antidiagonal initial ideals of ASM ideals are radical. Recall also that if a homogeneous ideal
$I$
possesses a radical initial ideal
$J$
then
$I$
is a Cohen–Macaulay ideal if and only if
$J$
is a Cohen–Macaulay ideal [Reference Conca and Varbaro8]. Hence, it suffices to show that, if
$\texttt {in} I_{\widetilde {A}}$
is a Cohen–Macaulay ideal, then
$\texttt {in} I_A$
is a Cohen–Macaulay ideal. We will show that, after a relabeling of vertices and deconing,
$\Delta (A)$
is obtained as a link of
$\Delta (\widetilde {A})$
, from which the result will follow because links of Cohen–Macaulay simplicial complexes are Cohen–Macaulay [Reference Reisner31].
For convenience, we will choose a non-standard indexing on the generic matrix from which equations for
$I_{\widetilde {A}}$
are defined. Let
$Z' = (z_{a,b})$
be an
$(n+1) \times (n+1)$
generic matrix with
$0 \leq a \leq n$
and
$1 \leq b \leq n+1$
. Correspondingly, index the rows of
$\widetilde {A}$
with the set
$\{0,1,\ldots , n\}$
and the columns with
$\{1,\ldots ,n+1\}$
, and consider
$\Delta (\widetilde {A})$
as a complex on
$\{0,\ldots ,n\} \times [n+1]$
with vertex
$(a,b)$
corresponding to variable
$z_{a,b}$
in the usual way. We retain the usual indexing for
$A$
and use
$Z = (z_{a,b})$
with
$a,b \in [n]$
to define
$I_A$
.
Example 3.14. If
$A = \begin{pmatrix} \fbox {0} &\quad 1 &\quad 0\\ 1 &\quad \fbox {$-$1} &\quad 1\\ 0 &\quad 1 &\quad 0 \end{pmatrix}$
and
$j=2$
, then
$\widetilde {A} = \begin{pmatrix} {\color {blue}0} &\quad {\color {blue}1} &\quad {\color {blue}0} &\quad {\color {blue}0}\\ \fbox {0} &\quad {\color {blue}0} &\quad 1 &\quad 0\\ 1 &\quad {\color {blue}0} &\quad \fbox {$-$1} &\quad 1\\ 0 &\quad {\color {blue}0} &\quad 1 &\quad 0 \end{pmatrix}$
. With the prescribed indexing,
$\texttt {Ess}(A) = \{(1,1),(2,2)\}$
and
$\texttt {Ess}(\widetilde {A}) = \{(1,1),(2,3)\}$
. Note also that
${rk}_{\widetilde {A}}(1,1) ={rk}_A(1,1)$
(because column
$1$
is to the left of column
$j=2$
) and that
${rk}_{\widetilde {A}}(2,3) = {rk}_A(2,2)+1$
(because column
$3$
is to the right of column
$j=2$
). Then

where the matrix entries corresponding to essential cells are boxed.
We claim that, with this choice of indexing,

Because
$I_A$
has a generating set that does not involve the variables
$z_{0,1}, \ldots , z_{0,n+1}$
,
$\texttt {in} I_A$
is a Cohen–Macaulay ideal if and only if
$\texttt {in} I_A+(z_{0,1}, \ldots z_{0,j-1}, z_{0,j+1}, \ldots , z_{0,n+1})$
is. Similarly,
$\texttt {in} I_{\widetilde {A}}:z_{0,j+1}\cdots z_{0,n+1}$
has a generating set that does not involve the variables
$z_{0,j+1}, \ldots , z_{0,n+1}$
, and so
$(\texttt {in} I_{\widetilde {A}}:z_{0,j+1}\cdots z_{0,n+1})+(z_{0,j+1}, \ldots , z_{0,n+1})$
is a Cohen–Macaulay ideal because
$\texttt {in} I_{\widetilde {A}}:z_{0,j+1}\cdots z_{0,n+1}$
is. Hence, the desired result will follow from establishing Equation (2).
The latter equality of Equation (2) follows from Lemma 3.12. In order to establish the first equality of Equation (2), we will first show

We note that the containments
$(z_{0,1}, \ldots , z_{0,j-1}) \subseteq \texttt {in} I_{\widetilde {A}} \subseteq \texttt { in} I_{\widetilde {A}}:z_{0,j+1}\cdots z_{0,n+1}$
are immediate from the definitions of rank function, ASM ideal, and colon ideal. Fix
$\mu \in \texttt {in} I_A$
. We may assume that
$\mu$
is the leading term of some Fulton generator
$f$
of
$I_A$
determined by essential cell
$(a,b)$
of
$A$
.
If
$b\lt j$
, then
$(a,b)$
is also an essential cell of
$\widetilde {A}$
and
${rk}_A(a,b) = {rk}_{\widetilde {A}}(a,b)$
. Hence,
$f$
is a Fulton generator of
$I_{\widetilde {A}}$
, and so
$\mu \in \texttt {in} I_{\widetilde {A}} \subseteq \texttt {in} I_{\widetilde {A}}:z_{0,j+1}\cdots z_{0,n+1}$
.
If
$b \geq j$
, then
$(a,b+1)$
is an essential cell of
$\widetilde {A}$
and
${rk}_A(a,b) = {rk}_{\widetilde {A}}(a,b+1)-1$
. Let
$M$
be the
$({rk}_A(a,b)+1) \times ({rk}_A(a,b)+1)$
submatrix of
$Z$
so that
$f = \det (M)$
. Form a submatrix
$M'$
of
$Z'$
by augmenting the set of rows of
$M$
by
$0$
and the set of columns of
$M$
by
$b+1$
. Then,
$M'$
is a
$({rk}_{\widetilde {A}}(a,b+1)+1) \times ({rk}_{\widetilde {A}}(a,b+1)+1)$
submatrix of
$Z'$
weakly northwest of
$(a,b+1)$
. Hence,
$\det (M')$
is a Fulton generator of
$I_{\widetilde {A}}$
. But
$z_{0,b+1}\mu = \texttt {in} \det (M')$
, and so
$\mu \in \texttt {in} I_{\widetilde {A}}:z_{0,j+1}\cdots z_{0,n+1}$
, as desired.
It remains to show

for which it suffices to show
$\texttt {in} I_{\widetilde {A}}:z_{0,j+1}\cdots z_{0,n+1} \subseteq \texttt {in} I_A+(z_{0,1}, \ldots ,z_{0,j-1}, z_{0,j+1}, \ldots , z_{0,n+1})$
. Fix a monomial
$\nu \in \texttt {in} I_{\widetilde {A}}:z_{0,j+1}\cdots z_{0,n+1}$
. First suppose
$\nu \in \texttt {in} I_{\widetilde {A}}$
. We may assume that
$\nu$
is the leading term of some Fulton generator
$g$
of
$I_{\widetilde {A}}$
. Then, there exist
$(c,d) \in \texttt {Ess}(\widetilde {A})$
and
$N'$
an
$({rk}_{\widetilde {A}}(c,d)+1) \times ({rk}_{\widetilde {A}}(c,d)+1)$
submatrix of
$Z'$
weakly northwest of
$(c,d)$
so that
$g = \det (N')$
.
If
$d\lt j$
, then, either
$\nu \in (z_{0,1}, \ldots , z_{0,j-1})$
, which occurs if
$N'$
involves row
$0$
, or
$g$
is a Fulton generator of
$I_A$
belonging to essential cell
$(c,d)$
, which occurs if
$N'$
does not involve row
$0$
. If
$d \gt j$
, then
$\nu$
is equal to the product of the northeast entry of
$N'$
with the lead term of Fulton generators of
$A$
belonging to essential cell
$(c,d-1)$
of
$A$
. Note that
$d \neq j$
because
$\widetilde {A}$
has no essential cells in column
$j$
.
Finally, suppose
$\nu \in (\texttt {in} I_{\widetilde {A}}:z_{0,j+1}\cdots z_{0,n+1})\setminus \texttt {in} I_{\widetilde {A}}$
. Because the variables
$z_{0,k}$
,
$k \in [j+1,n+1]$
, all belong to the same row, each minimal generator of
$\texttt {in} I_{\widetilde {A}}$
is divisible by at most one such
$z_{0,k}$
. We may assume there exists
$k\in [j+1,n+1]$
so that
$z_{0,k}\nu$
is the product of antidiagonal entries of a submatrix of
$Z'$
pertaining to an essential cell
$(c,d)$
of
$\widetilde {A}$
with
$d\gt j$
. Then,
$\nu$
is the product of antidiagonal entries of a submatrix of
$Z$
pertaining to essential cell
$(c,d-1)$
, which is to say that
$\nu \in \texttt {in} I_A$
, completing the proof.
We have now seen that
$A$
is equidimensional if and only if
$1 \oplus A$
is equidimensional and that
$1 \oplus A$
Cohen–Macaulay implies
$A$
Cohen–Macaulay. A Macaulay2 computation confirms that, over the field of rational numbers, if
$A \in \tt ASM(n)$
with
$n \leq 6$
and
$A$
is Cohen–Macaulay, then
$1 \oplus A$
is Cohen–Macaulay. Based on this evidence, we make the following conjecture:
Conjecture 3.15. The ASM
$A$
is Cohen–Macaulay if and only if
$1 \oplus A$
is Cohen–Macaulay.
The truth of Conjecture 3.15 would have implications for any ASM obtained as a diagonal block sum of others. Our next goal is to state and prove that implication, for which we require a couple of lemmas.
The following is a routine exercise, which can be found, for example, in [Reference Villarreal34, Chapter 3].
Lemma 3.16. Let
$S_1 = \mathbb{C}[x_1, \ldots , x_n]$
,
$S_2 = \mathbb{C}[y_1, \ldots , y_m]$
, and
$R = \mathbb{C}[x_1, \ldots , x_n,y_1,\ldots ,y_m]$
. Suppose that
$I_1$
is a proper homogeneous ideal of
$S_1$
and that
$I_2$
is a proper homogeneous ideal of
$S_2$
. Then,
$R/(I_1R+I_2R)$
is Cohen–Macaulay if and only if
$S_1/I_1$
and
$S_2/I_2$
are both Cohen–Macaulay. The associated primes of
$R/(I_1R+I_2R)$
are exactly those ideals of the form
$P_1R+P_2R$
where
$P_i$
is an associated prime of
$I_i$
. In particular,
$R/(I_1R+I_2R)$
is equidimensional if and only if
$S_1/I_1$
and
$S_2/I_2$
are both equidimensional.
Recall that the support of a monomial ideal
$J$
, denoted
$\texttt {Supp}(J)$
, is the set of variables dividing some minimal monomial generator of
$J$
. If
$J = \texttt {in} I_w$
for some
$w \in S_n$
, then
$\texttt {Supp}(J)$
is sometimes called the core of
$w$
.
The following lemma is known to experts. We record it below for completeness.
Lemma 3.17. If
$A \in \tt ASM(n)$
, then
$\texttt {Supp}(\texttt {in} I_A) \subseteq \{z_{i,j} \mid i+j \leq n\}$
.
Proof.
By [Reference Weigandt35, Proposition 3.11], there exist
$u_1, \ldots , u_k \in S_n$
such that
$I_A = I_{u_1}+\cdots +I_{u_k}$
. By Proposition 3.8,
$\texttt {in} I_A = \texttt {in} I_{u_1}+\cdots +\texttt {in} I_{u_k}$
. Hence, it suffices to show that
$\texttt {Supp}(\texttt {in} I_w) \subseteq \{z_{i,j} \mid i+j \leq n\}$
for arbitrary
$w \in S_n$
. This follows from the pipe dream description of
$\texttt {in} I_w$
in [Reference Knutson and Miller17].
Example 3.18. Consider
$w = 31542$
, whose Rothe diagram is below. The cells
$(i,j)$
with
$z_{i,j} \in \texttt {Supp}(\texttt {in} I_w)$
are those with an orange line running across their antidiagonal.

Theorem 3.19. (a) Fix
$A_1 \in \tt ASM(m)$
and
$A_2 \in \tt ASM(n)$
, and set
$A = A_1 \oplus A_2$
. For
$u \in S_m$
and
$v \in S_n$
, the assignment
$(u,v) \mapsto u \oplus v$
gives a bijection between
$\textrm{Perm}(A_1) \times \textrm{Perm}(A_2)$
and
$\textrm{Perm}(A)$
. In particular,
$\texttt {codim}(X_A) = \texttt {codim}(X_{A_1})+\texttt {codim}(X_{A_2})$
, and
$A$
is equidimensional if and only if
$A_1$
and
$A_2$
are both equidimensional.
(b) If
$A$
is Cohen–Macaulay, then
$A_1$
and
$A_2$
are both Cohen–Macaulay. If Conjecture 3.15 is true, then the converse also holds.
Proof.
Consider
$R = \mathbb{C}[z_{i,j} \mid i, j \in [m+n]]$
as the ambient polynomial ring of
$I_A$
. Consider the subrings
$S_1 = \mathbb{C}[z_{i,j} \mid i+j \leq m]$
and
$S_2 = \mathbb{C}[z_{i,j} \mid i+j \gt m]$
of
$R$
. Then
$R = S_1 \otimes _{\mathbb{C}} S_2$
.
For
$k \geq 1$
, let
$\mathbb{I}_k$
denote the
$k \times k$
identity matrix, and observe that

By Proposition 3.8,

By Lemma 3.17,
$\texttt {in} I_{A_1+\mathbb{I}_n}$
is supported only on variables
$z_{i,j}$
with
$i+j\leq m$
. Note that
$\texttt {in} I_{\mathbb{I}_m \oplus A_2}$
is supported only on
$z_{i,j}$
with
$i+j\gt m+1$
. Specifically,
$\texttt {in} I_{A_1 \oplus \mathbb{I}_n}$
and
$\texttt {in} I_{\mathbb{I}_m \oplus A_2}$
are supported on disjoint sets of variables; the former on a subset of the variables of
$S_1$
and the latter on a subset of the variables of
$S_2$
.
(a) By Lemma 3.16, the associated primes
$P$
of
$\texttt {in} I_A$
are exactly those ideals of the form
$Q_1+Q_2$
where
$Q_1$
is an associated prime of
$\texttt {in} I_{A_1 \oplus \mathbb{I}_n}$
and
$Q_2$
is an associated prime of
$\texttt {in} I_{\mathbb{I}_m \oplus A_2}$
.
Combining Proposition 3.8 and Proposition 3.9, the associated primes of
$\texttt {in} I_{A_1 \oplus \mathbb{I}_m}$
are exactly those ideals of the form
$\texttt {in} I_{u'}$
for some
$u' \in \textrm{Perm}(A_1 \oplus \mathbb{I}_n)$
, and the associated primes of
$\texttt {in} I_{\mathbb{I}_m \oplus A_2}$
are exactly those of the form
$\texttt {in} I_{v'}$
for some
$v' \in \textrm{Perm}(\mathbb{I}_m \oplus A_2)$
. Because
$I_{A_1 \oplus \mathbb{I}_n} = I_{A_1}R$
, those permutations
$u'$
are exactly those of the form
$u \oplus \mathbb{I}_m$
for some
$u \in \textrm{Perm}(A_1)$
. By Proposition 3.3, those permutations
$v'$
are exactly those of the form
$\mathbb{I}_m \oplus v$
for some
$v \in \textrm{Perm}(A_2)$
. Then
$(u,v) \in \textrm{Perm}(A_1) \times \textrm{Perm}(A_2)$
if and only if
$I_{u \oplus v} = I_{u \oplus \mathbb{I}_n}+I_{\mathbb{I}_m \oplus v}$
is an associated prime of
$I_A$
, which is equivalent to
$u \oplus v \in \textrm{Perm}(A)$
. This completes the proof of the bijection between
$\textrm{Perm}(A_1) \times \textrm{Perm}(A_2)$
and
$\textrm{Perm}(A)$
.
Because, for any
$u \in S_m$
and
$v \in S_n$
,
$\ell (u \oplus v) = \ell (u)+\ell (v)$
, the codimension and equidimensionality claims now follow from Proposition 3.1.
(b) We turn to the Cohen–Macaulayness claim. By Theorem 3.10,
$I_A$
,
$I_{A_1 \oplus \mathbb{I}_n}$
, and
$I_{\mathbb{I}_m\oplus A_2}$
define Cohen–Macaulay quotient rings if and only if
$\texttt {in} I_A$
,
$\texttt {in} I_{A_1 \oplus \mathbb{I}_n}$
, and
$\texttt {in} I_{\mathbb{I}_m\oplus A_2}$
do, respectively. By Proposition 3.16,
$\texttt {in} I_{A_1 \oplus \mathbb{I}_n}+\texttt {in} I_{\mathbb{I}_m\oplus A_2}$
defines a Cohen–Macaulay quotient if and only if both
$\texttt {in} I_{A_1 \oplus \mathbb{I}_n}$
and
$\texttt {in} I_{\mathbb{I}_m\oplus A_2}$
do. Because
$I_{A_1 \oplus \mathbb{I}_n} = I_{A_1}R$
,
$I_{A_1 \oplus \mathbb{I}_n}$
and
$I_{A_1}$
define Cohen–Macaulay quotients or not alike. By Theorem 3.13, if
$I_{\mathbb{I}_m\oplus A_2}$
defines a Cohen–Macaulay quotient ring, then
$I_{A_2}$
does. If Conjecture 3.15 is true, the converse is true as well.
In Proposition 3.19, one may alternatively establish the codimension and equidimensionality claims by appealing to Lemma 3.16 together with the fact that codimension does not change under Gröbner degeneration (see, e.g., [Reference Eisenbud10, Chapter 15]).
Proposition 3.20. If
$A$
admits a block-matrix decomposition of the form

where both
$A_1 \in \tt ASM(m)$
and
$A_2 \in \tt ASM(n)$
are ASMs, then
$A$
is unmixed (respectively, Cohen–Macaulay) if and only if
$A_1$
and
$A_2$
are both unmixed (respectively, Cohen–Macaulay).
Proof.
Let
$J = (z_{i,j} \mid i \in [m], j \in [n])$
. For a suitable choice of indexing of the variables,
$I_A = J+I_{A_1}+I_{A_2}$
, where
$J$
,
$I_{A_1}$
, and
$I_{A_2}$
involve pairwise disjoint sets of variables. Hence, the result follows from Lemma 3.16.
In the next section, we will see through Proposition 4.3 that the special case of a block sum decomposition is not quite so specialized as it may seem in the sense that ASMs do not easily contain one another.
4. Pattern avoidance for ASMs
In this section, we will consider a natural extension of the notion of pattern avoidance in
$S_n$
to
$\tt ASM(n)$
. Permutation pattern avoidance, its own area of study within combinatorics, has been shown to govern desirable algebro-geometric properties within Schubert calculus (see, e.g., [Reference Woo and Yong36, Reference Knutson, Miller and Yong18, Reference Klein16] and quite importantly [Reference Lakshmibai and Song24] together with a survey of its consequences [Reference Abe and Billey1]). Properties that are governed by pattern avoidance occur asymptotically with probability
$0$
. As we saw in Figure 1, the property of being Knutson–Miller vertex decomposable becomes increasingly rare. So, too, it turns out does Cohen–Macaulayness also becomes increasingly rare, which we see now in Figure 2.

Figure 2. Counts of Cohen–Macaulay and non-Cohen–Macaulay elements of
$\tt ASM(n)$
.
It is for this reason that we are motivated to consider pattern avoidance in
$\tt ASM(n)$
.
Definition 4.1. Let
$A$
and
$A'$
be ASMs. If
$A'$
is a submatrix of
$A$
, then we will say that
$A$
contains
$A'$
. Otherwise, we will say that
$A$
avoids
$A'$
.
These definitions coincide with the usual uses of “contains” and “avoids” in the sense of permutation pattern avoidance when
$A$
and
$A'$
are permutation matrices.
We will first use an example to show that, perhaps surprisingly, this extension of pattern avoidance to
$\tt ASM(n)$
given above does not govern Cohen–Macaulayness of ASM varieties. We will then prove that, nevertheless, there is still some behavior that is correctly understood via pattern avoidance. Specifically, we will describe configurations whose containment is an obstruction to unmixedness. Our goal in presenting this information is to provide motivation and context for future work to consider other extensions of pattern avoidance from
$S_n$
to
$\tt ASM(n)$
that could do more work to capture the phenomena documented here, perhaps those beginning from a more geometric perspective as in [Reference Billey and Braden4] or [Reference Bergeron and Sottile6].
There are other notions of pattern avoidance for ASMs that have emerged in the literature on enumerative combinatorics (e.g., [Reference Johansson and Linusson15, Reference Ayyer, Cori and Gouyou-Beauchamps2, Reference Bouvel, Smith and Striker7]). To the best of the authors’ knowledge, these have not been shown to have algebro-geometric interpretations.
We first consider the relationship between Cohen–Macaulayness and ASM pattern containment. We have previously seen, in Example 3.5 and Example 3.11, that a non-Cohen–Macaulay ASM can contain a Cohen–Macaulay ASM even when the larger ASM is obtained via the construction
$\widetilde {A}(i,j)$
. What is more surprising is that, as Example 4.2 shows, a Cohen–Macaulay ASM may contain a non-Cohen–Macaulay ASM.
Example 4.2. Let
$A = \begin{pmatrix} 0 &\quad 0 &\quad {\color {blue}0} &\quad 1 &\quad 0 &\quad 0\\ 0 &\quad 0 &\quad {\color {blue}0} &\quad 0 &\quad 1 &\quad 0\\ 0 &\quad 1 &\quad {\color {blue}0} &\quad 0 &\quad -1 &\quad 1\\ {\color {blue}0} &\quad {\color {blue}0} &\quad {\color {blue}1} &\quad {\color {blue}0} &\quad {\color {blue}0} &\quad {\color {blue}0}\\ 1 &\quad 0 &\quad {\color {blue}0} &\quad -1 &\quad 1 &\quad 0\\ 0 &\quad 0 &\quad {\color {blue}0} &\quad 1 &\quad 0 &\quad 0 \end{pmatrix}$
and
$B = \begin{pmatrix} 0 &\quad 0 &\quad 1 &\quad 0 &\quad 0\\ 0 &\quad 0 &\quad 0 &\quad 1 &\quad 0\\ 0 &\quad 1 &\quad 0 &\quad -1 &\quad 1\\ 1 &\quad 0 &\quad -1 &\quad 1 &\quad 0\\ 0 &\quad 0 &\quad 1 &\quad 0 &\quad 0 \end{pmatrix}$
. Then
$A$
is Cohen–Macaulay while
$B$
is not even unmixed, even though
$B$
is obtained from
$A$
by deletion of row
$4$
and column
$3$
. That is,
$A = \widetilde {B}(4,3)$
. Indeed,
$\textrm{Perm}(B) = \{45213, 34512, 35241\}$
, whose first and third elements have length
$7$
while the second has length
$6$
. In contrast to the situation of Proposition 3.13, some elements of
$\textrm{Perm}(A) = \{562314,462513,456213,462351\}$
fail to have a
$1$
in row
$4$
and column
$3$
, for example,
$462513$
. Moreover,
$|\textrm{Perm}(A)| = 4 \neq 3 = |\textrm{Perm}(B)|$
.
We provide one more piece of evidence of a shortcoming of this notion of pattern avoidance. We will show that the conditions implied by the containment of one ASM in another are rather restrictive. From that standpoint, one may worry that ASMs do not contain each other frequently enough to do the heavy lifting of recording obstructions to important algebro-geometric properties that become increasingly rare.

Figure 3. A visualization of the various regions discussed in Proposition 4.3 with
$k=3$
. The matrix
$A'$
is the submatrix of
$A$
consisting of its entries in the pink regions. The rows and columns that are removed from
$A$
to obtain
$A'$
are shaded in yellow. The entries in the yellow strips outside of the inner box are all
$0$
. The sum, for example, of the three entries marked
$\ast$
is
$0 = \sum _{j \in [3]} A_{r,c_j}$
for some
$r \notin W$
, i.e., for some non-yellow row
$r$
. The sum of the nine entries marked with a
$\star$
is
$\sum _{r \in W, c \in C} A_{r,c} = k=3$
.
Proposition 4.3. Suppose
$A \in \tt ASM(n)$
has a submatrix
$A' \in \tt ASM(n-k)$
. Let
$W = \{r_1,\ldots , r_k\}$
be the ordered set of indices of rows of
$A$
that do not intersect
$A'$
and
$C = \{c_1, \ldots , c_k\}$
be the ordered set of indices of columns of
$A$
that do not intersect
$A'$
. Then, the following hold:
-
(1) For all
$i \in [k]$ ,
$A_{r_i, c}=0$ if
$c \lt c_1$ or
$c\gt c_k$ and
$A_{r,c_i}=0$ if
$r \lt r_1$ or
$r \gt r_k$ .
-
(2)
$\sum _{r \in W, c \in C} A_{r,c} = k$ . In particular, there are at least
$k$ pairs
$(r_i,c_j)$ such that
$A_{r_i,c_j} = 1$ .
Proof. Throughout this proof, for a visualization of the various regions discussed, see Figure 3.
(1) Consider first the case
$c \lt c_1$
. For the sake of contradiction, suppose there exists some
$i \in [k]$
and
$c\lt c_1$
with
$A_{r_i,c} \neq 0$
. Assume that
$c$
has been chosen minimally. Because the first nonzero entry in each row of an ASM must be
$1$
,
$A_{r_i,c} = 1$
. The sum of the entries in column
$c$
of
$A$
is
$1$
, and also the sum of the entries in column
$c$
of
$A'$
is
$1$
because both
$A$
and
$A'$
are ASMs. Thus,
$\sum _{j \in [k]} A_{r_j, c} = 0$
, and so there must be some
$A_{r_j,c} = -1$
. But, by minimality of
$c$
,
$A_{r_j,c}$
must be the first nonzero entry in row
$r_j$
and, therefore, cannot be
$-1$
.
The arguments for the other cases of (1) are symmetric, and so we omit them.
(2) Because
$A'$
is an ASM,
$\sum _{j \in [k]} A_{r_j,c} = 0$
for each
$c \notin C$
and
$\sum _{i \in [k]} A_{r,c_i} = 0$
for each
$r \notin W$
. Note also that, because
$A \in \tt ASM(n)$
and
$A' \in \tt ASM(n-k)$
, we have, respectively,

Hence,

from which the result follows.
Corollary 4.4. If
$A' \in \tt ASM(n-k)$
embeds in
$A \in \tt ASM(n)$
as a block in the northwest corner, that is,
$A = \left (\begin{array}{@{}c|c@{}} A' & * \\ \hline * & * \end{array}\right )$
, then
$A = A' \oplus B$
for some
$B \in \tt ASM(k)$
.
Proof.
We use Proposition 4.3, where
$W = C = [n-k+1,n]$
, to see that
$A = A' \oplus B$
for some
$k \times k$
matrix
$B$
with entries in
$\{0, 1, -1\}$
. From the direct sum decomposition, we see that all nonzero entries of
$A$
in each row (resp., column)
$i \in [n-k+1,n]$
occur in columns (resp., rows)
$[n-k+1,n]$
. Hence, the nonzero entries of each row (resp., column) of
$B$
alternate in sign and sum to
$1$
. Therefore,
$B \in \tt ASM(k)$
.
Although the notion of pattern avoidance given in Definition 4.1 has significant limitations, it should not be entirely ignored. As an example of its capacity to encode some valuable information, we will show in Proposition 4.6 that containing an ASM belonging to a particular family is adequate to prevent unmixedness. We present this as evidence that pattern avoidance should not be entirely abandoned as a means to understand the algebra and geometry of ASM varieties. We begin with a lemma.
Lemma 4.5. Let
$I$
be a squarefree monomial ideal, and let
$S$
be a subset of the support of
$I$
. Let
$P = (S)$
. Then
$P$
is a minimal prime of
$I$
if and only if both of the following two conditions hold:
-
(i) For each monomial
$\mu \in I$ , at least one element of the support of
$\mu$ is an element of
$S$ .
-
(ii) For every
$x \in S$ , there exists a monomial
$\nu \in I$ so that
$x$ is in the support of
$\nu$ and
$x$ is the only element of
$S$ in the support of
$\nu$ .
Proof.
Clearly
$(i)$
holds if and only if
$I \subseteq P$
. It is also clear that
$P$
is prime. It remains to show
$(ii)$
holds if and only if that
$I \subseteq P' \subseteq P$
for a prime ideal
$P'$
implies
$P = P'$
.
Because the minimal primes of monomial ideals are monomial ideals, it suffices to show that
$(ii)$
holds if and only if that
$I \subseteq (S') \subseteq P$
for a subset
$S'$
of
$S$
implies
$P = (S')$
. But
$(ii)$
is false if and only if there is a proper subset
$S'$
of
$S$
so that condition
$(i)$
holds for
$S'$
, which is true if and only if
$I \subseteq (S') \subset (S)$
. Specifically, if
$(ii)$
is false and
$x \in S$
is the violating element, set
$S' = S \setminus \{x\}$
. Conversely, if such an
$S' \subset S$
exists, take
$x$
to be any element of
$S \setminus S'$
.
Using Lemma 4.5, we can show that various families of ASMs are not equidimensional (and therefore, not Cohen–Macaulay) by finding two minimal primes which we demonstrate to be of different heights. We give such a class of ASMs now.

Figure 4. (*) Pick a
$132$
-avoiding permutation whose bottom left boxes are
$0$
. Then replace the bottom left entry with
$-1$
and the bottom right with any entry allowable by the definition of ASM.
In the proof of the following proposition, we will use the total order
$\lt$
on
$[n] \times [n]$
given by
$(i,j) \lt (i',j')$
if
$i \lt i'$
or if
$i = i'$
and
$j \lt j'$
. That is,

We will use
$\lt$
as the total order for an inductive argument. Note that
$\lt$
is not related to a total order on the monomials determining the antidiagonal initial ideal
$\texttt {in} I_A$
of the ASM
$A$
.
Proposition 4.6. Suppose that
$A = (a_{ij}) \in \tt ASM(n)$
satisfies the following properties:
-
(1) There exists
$(r,c) \in [n-1] \times [n-2]$ so that the submatrix
$B$ of
$A$ consisting of rows
$\{r-1,r\}$ and columns
$\{c,c+1\}$ has the form
.
-
(2) If
$i \leq r$ and
$j \leq c$ and
$(i,j) \neq (r,c)$ , then
$a_{ij} = 0$ .
-
(3) If
$(i,j) \in \texttt {Ess}(A)$ and
$(i,j) \neq (r,c+1)$ , then
${rk}_A(i,j) = 0$ or
${rk}_A(i,j) \geq r-1$ .
-
(4)
$A$ has no essential cell in column
$c$ . Then
$A$ is not equidimensional.
Before beginning the proof, the reader may prefer to see Figure 4 for an illustration of the ASMs satisfying the conditions of Proposition 4.6.
Proof.
Recall that
$A$
is equidimensional if and only if all minimal primes of
$\texttt {in} I_A$
are of the same height. We will construct two different minimal primes of
$\texttt {in} I_A$
and show that they have different heights.
For
$(i,j) \in [n] \times [n]$
, let

Let
$F_{i,j}$
be the set of Fulton generators of
$I_A$
satisfying
$\texttt {Supp}(\texttt {in} F_{i,j}) \subseteq Z_{i,j}$
. Set
$I_{i,j} = (\texttt {in} f \mid f \in F_{i,j})$
.
Then
$\texttt {in} I_A = I_{n,n}+(z_{a,b} \mid (a,b) \in \texttt {Dom}(A))$
, and no term of any generator of
$I_{n,n}$
is divisible by any
$z_{a,b}$
with
$(a,b) \in \texttt {Dom}(A)$
. Hence, every minimal prime
$P$
of
$\texttt {in} I_A$
will be the sum of a minimal prime
$Q$
of
$I_{n,n}$
with
$(z_{a,b} \mid (a,b) \in \texttt {Dom}(A))$
, and
$\texttt {codim}(P) = \texttt {codim}(Q)+|\texttt {Dom}(A)|$
. Thus, it suffices to show that
$I_{n,n}$
has two or more minimal primes of different heights.
We will show that, for all
$(r,n) \leq (i,j) \leq (n,n)$
,
$I_{i,j}$
has two or more minimal primes of different heights. We will proceed by induction. We begin by finding two different minimal primes of
$I_{r,n}$
, which have different heights.
Before giving the formal argument, we recommend a visualization: Draw the grid of points representing the variables
$z_{i,j}$
in a generic
$n \times n$
matrix. For each minimal generator
$\mu$
of
$I_{r,n}$
, we draw a wire connecting the set of points corresponding to variables in the support of
$\mu$
. Figure 5 shows such a drawing for a possible
$I_{r,n}$
.

Figure 5. The
$r \times n = 5 \times 11$
grid corresponding to the variables
$z_{i,j}$
for
$(i,j) \in [r] \times [n]$
. Elements of
$\texttt {Dom}(A)$
are denoted by stars. The elements of
$O_{r,n}$
are circled in orange, and those of
$Y_{r,n}$
are circled in yellow. Minimal generators of
$I_{r,n}$
are indicated with blue and green wires connecting elements of
$Y_{r,n}$
to elements of
$O_{r,n}$
. In this example,
$c=2$
. The submatrix
$B$
is boxed.
Given such a drawing for
$I_{r,n}$
, we draw an orange circle around the northeast vertex of each minimal monomial generator of
$I_{r,n}$
. Call this set of vertices
$O_{r,n}$
. Now, we draw a yellow circle around the southwest vertex of each minimal monomial generator of
$I_{r,n}$
, and call this set of vertices
$Y_{r,n}$
.
More formally, if
$\mu = \texttt {in} f$
for some
$f \in F_{r,n}$
, then, with respect to the order
$\lt$
on
$[n] \times [n]$
, let

and

Similarly, let

and

Because each minimal generator
$\mu$
of
$\texttt {in} I_A$
of degree
$k$
is formed by the product of variables along the antidiagonal of a generic matrix, the
$k$
variables in the support of
$\mu$
have distinct row indices and distinct column indices. The generators of
$I_{r,n}$
are supported only on variables in the first
$r$
rows of a generic matrix, and so every generator of degree
$r$
in
$I_{r,n}$
is supported on some variable in row
$1$
and some variable in row
$r$
(and on some variable in each row in between).
By property (3) of the hypotheses, each
$f \in F_{r,n}$
is either of degree
$r$
or is determined by the essential cell
$(r,c+1)$
. By property (2) of the hypotheses, any
$f \in F_{r,n}$
determined by the essential cell
$(r,c+1)$
must be the determinant of a
$2 \times 2$
submatrix whose bottom row is
$[z_{r,c}$
$z_{r,c+1}]$
. Thus, all minimal generators of
$I_{r,n}$
are divisible by some variable in row
$r$
. Hence, we have
$Y_{r,n} = \{z_{r,c}, \ldots , z_{r, c+i}\}$
for some
$i \geq 0$
.
Similarly,
$O_{r,n}$
will consist of the set of variables
$\{z_{1,c+r},\ldots , z_{1,c+i+r-1}\}$
together with at least two more variables in column
$c+1$
, determined by the degree
$2$
minimal generators of
$I_{r,n}$
. Hence,
$|O_{r,n}| \geq |Y_{r,n}|+1$
.
By construction, both
$Y_{r,n}$
and
$O_{r,n}$
satisfy the conditions of Lemma 4.5, and so both
$(Y_{r,n})$
and
$(O_{r,n})$
are minimal primes of
$I_{r,n}$
. Now
$\texttt {codim}((O_{r,n})) = |O_{r,n}| \gt |Y_{r,n}| = \texttt {codim}((Y_{r,n}))$
, and so
$I_{r,n}$
is not height unmixed.
For
$(i,j)$
satisfying
$(r,n)\lt (i,j) \leq (n,n)$
, we will define sets
$Y_{i,j}$
and
$O_{i,j}$
inductively. We will then argue, for all such
$(i,j)$
, that
$|Y_{i,j}|\lt |O_{i,j}|$
and that both
$(Y_{i,j})$
and
$(O_{i,j})$
are minimal primes of
$I_{i,j}$
.
Suppose that
$(r,n)\leq (i,j) \lt (i',j')\leq (n,n)$
and that
$(i',j')$
covers
$(i,j)$
in the order
$\lt$
, that is, either
$j\lt n$
and
$(i',j') = (i,j+1)$
or
$j = n$
and
$(i',j') = (i+1,1)$
.
Define

and

Note that the minimal generators of
$I_{i',j'}$
that are not elements of
$I_{i,j}$
are exactly those that are divisible by
$z_{i',j'}$
. Thus,
$I_{i',j'} \not \subseteq (O_{i,j})$
(respectively,
$I_{i',j'} \not \subseteq (Y_{i,j})$
) if and only if there exists some minimal generator
$\mu$
of
$I_{i',j'}$
divisible by
$z_{i',j'}$
that is not divisible by any element of
$O_{i,j}$
(respectively, of
$Y_{i,j}$
). Hence, arguing by induction, it follows from Lemma 4.5 that both
$(O_{i',j'})$
and
$(Y_{i',j'})$
are minimal primes of
$I_{i',j'}$
We now claim that, for all
$(i',j')\geq (r,n)$
,
$z_{i',j'} \in Y_{i',j'}$
if and only if
$z_{i',j'} \in O_{i',j'}$
. From this claim, it will be immediate that
$|Y_{i',j'}|\gt |O_{i',j'}|$
and, in particular, that
$|Y_{n,n}|\gt |O_{n,n}|$
.
Noting that the claim is true for
$(r,n)$
itself, we fix some
$(i,j)$
satisfying
$(r,n) \leq (i,j) \lt (n,n)$
and assume the truth of the claim for all
$(a,b)$
satisfying
$(r,n) \leq (a,b) \leq (i,j)$
. It then suffices to prove the claim for
$(i',j')$
covering
$(i,j)$
.

Figure 6. In this example,
$r = 4$
,
$c=2$
, and
$n=8$
. Notice that
$\texttt {Ess}(A) \setminus \texttt {Dom}(A) = \{(4,3), (4,6), (7,6)\}$
, whose locations are indicated with a black box. Locations of elements of
$Y_{6,8} = \{(4,2),(4,3)\}$
are indicated with a yellow box, and locations of elements of
$O_{6,8} = \{(1,6),(2,3),(3,3)\}$
are indicated with an orange box. Consider
$(r',c') = (7,6)$
, and note
${rk}_A(7,6) = 5$
. Then
$\mu = (z_{1,6}z_{2,5}z_{3,4})(z_{5,3}z_{6,2}z_{7,1})$
witnesses
$I_{7,1} \not \subseteq (Y_{6,8})$
but does not witness
$I_{7,1} \not \subseteq (O_{6,8})$
. The modification
$\mu ' = (z_{2,6}z_{3,5}z_{4,4})(z_{5,3}z_{6,2}z_{7,1})$
witnesses
$I_{7,1} \not \subseteq (O_{6,8})$
.
Suppose that
$z_{i',j'} \in Y_{i',j'}$
, equivalently
$I_{i',j'} \not \subseteq (Y_{i,j})$
. Then, there exists some generator
$\mu$
of
$I_{i',j'}$
so that
$z_{i',j'} \mid \mu$
and
$\mu \notin (Y_{i,j})$
. If
$\mu$
is the initial term of a Fulton generator determined by an essential cell whose column index is
$\lt c$
, then
$\deg (\mu ) = 0$
, and so
$\mu \in \texttt {Dom}(A)$
, a contradiction. Hence, by property (4) of the hypotheses,
$\mu$
is the initial term of a Fulton generator determined by an essential cell whose column index is
$\gt c$
. Call that essential cell
$(r',c')$
.
Let
$\nu$
be the product of the variables dividing
$\mu$
whose row index is at most
$r$
. We consider two cases:
$\deg (\nu ) = r$
and
$\deg (\nu )\lt r$
.
First suppose that
$\deg (\nu ) = r$
. By property (2) of the hypotheses, we know that
$\nu$
is divisible only by variables in columns with index at least
$c$
. By construction of
$Y_{r,n}$
and
$O_{r,n}$
, an antidiagonal of length
$r$
in the first
$r$
rows of
$Z$
and columns weakly east of
$c$
that is disjoint from
$\texttt {Dom}(A)$
either contains both an element of
$Y_{r,n}$
and an element of
$O_{r,n}$
or neither. Thus,
$\mu \notin (Y_{i,j})$
implies
$\nu \notin (Y_{r,n})$
implies
$\nu \notin (O_{r,n})$
. Because
$Y_{i,j}$
and
$O_{i,j}$
agree below row
$r$
by induction, it follows that
$\mu \notin (O_{i,j})$
.
Alternatively, suppose
$\deg (\nu )\lt r$
. Let
$Z'$
be the
$\deg (\nu ) \times \deg (\nu )$
submatrix of
$Z$
with row indices
$\{r-\deg (\nu )+1, \ldots , r\}$
and column indices
$\{c'-\deg (\nu )+1, \ldots , c'\}$
. Then set
$\mu ' = (z_{r,c'-\deg (\nu )+1}z_{r-1,c'-\deg (\nu )+2} \cdots z_{r-\deg (\nu )+1,c'})\mu /\nu$
, and note that
$\mu ' \in I_{i',j'}$
because it is the product of the antidiagonal entries of
$Z'$
with the antidiagonal entries of the submatrix of
$Z$
below row
$r$
giving rise to
$\nu$
. That is, it is the product of antidiagonal entries of a
$\deg (\mu ) \times \deg (\mu )$
submatrix of
$Z$
weakly northwest of
$(r',c')$
and so the lead term of an element of
$F_{i',j'}$
. Said otherwise, we are replacing the submatrix giving rise to
$\mu$
with a submatrix whose first
$\deg (\nu )$
rows and final
$\deg (\nu )$
columns are maximally southeast subject to the condition of remaining weakly north of row
$r$
and weakly west of the essential box giving rise to
$\mu$
. See Figure 6 for an illustration of the replacement of
$\mu$
by
$\mu '$
.
Because
$\deg (\nu )\lt r$
,
$\mu '$
is not divisible by any element in row
$1$
or any element in column
$c+1$
except possibly
$z_{r,c+1}$
. Hence, using that
$O_{i,j}$
and
$Y_{i,j}$
agree by induction below row
$r$
,
$\mu ' \notin (O_{i,j})$
.
Thus, in all cases,
$I_{i',j'} \not \subseteq (O_{i,j})$
, and so
$z_{i',j'} \in O_{i',j'}$
, as desired.
The proof that
$z_{i',j'} \in O_{i',j'}$
implies
$z_{i',j'} \in Y_{i',j'}$
is analogous. We conclude by induction that
$|O_{n,n}|\lt |Y_{n,n}|$
and, having shown that both
$(O_{n,n})$
and
$(Y_{n,n})$
are minimal primes for
$I_{n,n}$
, that
$I_{n,n}$
and so also
$\texttt {in} I_A$
are not height unmixed, which implies that
$A$
is not equidimensional.
Acknowledgments
This project began as part of the University of Minnesota School of Mathematics Summer 2022 REU program. The authors would like to thank John O’Brien for his assistance as a TA during that program as well as all of the REU organizers, mentors, and participants for valuable feedback throughout the summer. They particularly thank Sterling Saint Rain for very regular and engaged participation. In addition, they would like to thank Mike Cummings and Adam Van Tuyl for sharing an early version of their new Macaulay2 package GeometricDecomposability, now described in [Reference Cummings and Van Tuyl9], and for additional coding advice. The authors thank Anna Weigandt for helpful conversations and Frank Sottile for communication concerning [Reference Billey and Braden4] and [Reference Bergeron and Sottile6]. Finally, the authors thank the anonymous referee, whose careful reading and helpful comments have helped them to improve this document.